Tfs Extensibility - Filtering Lab Management Test Agents by Test Configuration

Original Problem
As previously mentioned there's no built-in way to pick a test agent automatically based on the selected test configuration, after much digging I have found a partial work around.

Theory
It turns out that like a lot of tfs components the test controller supports plugins.  These inherit from TcmRunControllerPlugin and have 3 methods - Init, TestRunStarting and TestRunCompleted.  Unlike other TFS Plugins they are not automatically detected and need to be manually added to the qtcontroller.config (in c:\program files (x86)\microsoft visual studio 11.0\common7\ide).

I found that on the TestRunConfiguration object there is a StringDictionary called AgentProperties, upon further inspection I noticed that it appeared to be used to filter test agents to find a suitable one to run the tests on.  By default it has a filter for environment name, as a small test I added another hardcoded filter by overriding the Init method in a very simple plugin.  To my surprise whenever tests were run they would now complain that no suitable agent could be found, by changing my filter it quickly came apparent that this method worked for machine tags.

By overriding the Init method and a bit more digging I managed to get a list of properties from the Test Configurations and put them as filters for the agent selected.

Limitations/Issues
I created a Test Configuration with values "IE:9" and another with "IE:10" and respectively labeled the web client machines with the same tags.  Running the IE9 tests from MTM would always pick the machine tagged IE:9 and the IE10 ones would pick the machine tagged IE:10.  Running both together would look for a machine labelled IE:9,10 which it wouldn't find and would fail.  Unfortunately I have no way around this limitation currently.

When running coded ui tests via a lab build you can only select one Test Configuration anyway so this limitation is only a problem when manually running the tests in MTM.

The test controller plugins don't appear to be documented, which may mean this breaks in a future update.

Cross Browser Support
This idea can be extended to apply to other browsers too using the cross browser feature from update 1 simply create an extra machine for each browser you wish to test and label correctly.  I'd probably change my labelling scheme from IE:9 to Browser:IE9 to be bit more consistant.  On each machine add a system environment variable called browser set to the correct name, then in your coded UI test make sure you always grab this value and set BrowserWindow.CurrentBrowser to its value.

Advantages over other methods

• Tests can be run from MTM without needing to manually select a different environment to run it in (as long as you only run one Test Configuration at a time)
• Machines other than the test agents can be reused (eg database/webserver) meaning less hyper-v resources are needed
• Works fine with TFS2012's auto config of test clients which causes issues for the multiple test client role method

Potential Extensions (not sure if they're possible)

• A way to disable/enable this filtering (an extra property in a test configuration would easily do it)
• I'm still looking for ways to allow multiple Test Configurations to be run in one test run from MTM.
• Overriding the browser environment variable automatically so I can use the same virtual machine for IE, Chrome and Firefox (obviously still needing a second machine for IE9 vs IE10).
• Check if any other types of filters are available, I did see some xml floating around suggesting you may be able to filter based on Ram available.

Tfs Automation - keep user images in sync with active directory

In a previous post I shows how to set user images to be same as active directory in a console application.

I have updated this code to run as a recurring ITeamFoundationJobExtension .
 The first file creates the job in TFS to run every 24 hours, and runs it once right now.

The second file is the updated code that is now a TFS Agent Job, this required a few changes to the code to run against the server api instead of the client one. They're fairly similar but the client api tends to be a bit more friendly.
Installation
Copy the compiled job to c:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\TFSJobAgent\plugins and restart the agent service. Then run the console application to register the job. Finally check the state of the job by looking in the tfs database

select * from tfs_configuration.dbo.tbl_JobHistory where jobid = 'fa60c04e-c996-413e-8151-15933f5a2bac'

Tfs Automation - Automatically creating queries when new areas are created

Our testers got sick of setting up new queries whenever they created a new area in TFS so asked me if I could automate it.
The solution is broken into 2 parts:

AreaCreatedSubscriber

This is ISubscriber that watches for StructureChangedNotification events and queues a one time job to create the queries.
The reason I queued a job instead of just doing the work in this plugin was that when creating the query TFS was throwing an exception on validation (TF51011: The specified area path does not exist.) It seems that there is a slight delay between adding areas and them becoming valid for use in queries. I thought it was a caching issue but even when explicitly clearing the cache on the workitemstore it still happens, I found a post (New Area Path values are not available in work items) that suggested that tfs subscribes to its own events which then triggers a process of making the area available. This meant I needed to delay the creation of the queries until after all the events had completed, hence the job queue.
Also it is fairly good practice to push any event semi long running tasks into a job as it can delay actions for end users.
This gets installed to c:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\Web Services\bin\Plugins

CreateDefaultQueriesJob

The second part is the actual job, its also fairly simple (although crudely written). It reads the Area path out of the event data and creates a folder hierarchy for the queries to live in. It then checks if the queries already exist and if not creates them.
This job should probably be extended/cleanedup to read the queries from an external source, whether it be a WIQL file on the disk, some metadata stored on the team project, or just a special folder in source control.
This is installed into c:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\TFSJobAgent\plugins

Debugging

I had a bunch of issues trying to get this to work.
Firstly the TF51011 error - folders were appearing in the shared queries so I knew the plugin was doing something, luckily the exception was being logged to the event logs so it was really easy to find.
After moving the logic to a one time job I tried testing it only to find nothing happened. No folders created and nothing logged in the event logs. I figured that the event was still being fired so it was probably queueing the job, after a quick google I came across a blog article from Martin Hinshelwood debugging the TFS Active directory sync job this gave me a bunch of helpful debugging tips.
This lead me to the tfs project collection database
SELECT * FROM [Tfs_Enlighten].[dbo].[tbl_JobDefinition] where ExtensionName like '%CreateDefaultQueriesJob'
Returned rows but they weren't much help, jobs were being queued but I had no idea about if they were successful or not. I eventually discovered another table in the tfs_configuration database which contained the jobHistory
select * from tfs_configuration.dbo.tbl_JobHistory where jobid in (SELECT [JobId] FROM [Tfs_Enlighten].[dbo].[tbl_JobDefinition] where ExtensionName like '%CreateDefaultQueriesJob')
Also had one row for every time I queued the job, with the status result of 6, this time I needed to do some diving with ILSpy, which lead me to this enum

ExtensionNotFound = 6 - TFS couldn't find my job! I double and triple checked all my code, and started diving through the JobRunner code in TFS trying to figure out why it wasn't loading. I noticed a few hints of MEF in the code so tried adding an [Export] attribute to the class, still no luck. Enabling the trace log by editing c:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\TFSJobAgent\TfsJobAgent.exe.config I started seeing a curious exception about a plugin being added to a dictionary twice.

Especially interesting as it appears to check if the plugin is already registered in the dictionary before adding it, which makes me think they have a threading bug.
After removing the [Export] attribute and restarting the tfs job service a few times it seemed to come right.

Update: Apparently there's some new web access interfaces to view the status of TFS Jobs (and more), may have meant less database diving had I known at the time

Tfs Automation - Create Builds Automatically for New Solutions

I recently wrote a tfs server plugin that creates new builds automatically whenever you add or branch a solution file.  It's fairly simple, but worth sharing.

It takes advantage of another piece of code I had previously written (or perhaps found online, I honestly can't remember now) to create builds using the TFS api.

Installing it is very simple, just build the class into an assembly and copy it to c:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\Web Services\bin\Plugins you may also need to copy a couple of the referenced dlls to the same folder.