Tfs Extensibility - Filtering Lab Management Test Agents by Test Configuration

Original Problem
As previously mentioned there's no built-in way to pick a test agent automatically based on the selected test configuration, after much digging I have found a partial work around.

Theory
It turns out that like a lot of tfs components the test controller supports plugins.  These inherit from TcmRunControllerPlugin and have 3 methods - Init, TestRunStarting and TestRunCompleted.  Unlike other TFS Plugins they are not automatically detected and need to be manually added to the qtcontroller.config (in c:\program files (x86)\microsoft visual studio 11.0\common7\ide).

I found that on the TestRunConfiguration object there is a StringDictionary called AgentProperties, upon further inspection I noticed that it appeared to be used to filter test agents to find a suitable one to run the tests on.  By default it has a filter for environment name, as a small test I added another hardcoded filter by overriding the Init method in a very simple plugin.  To my surprise whenever tests were run they would now complain that no suitable agent could be found, by changing my filter it quickly came apparent that this method worked for machine tags.

By overriding the Init method and a bit more digging I managed to get a list of properties from the Test Configurations and put them as filters for the agent selected.

Limitations/Issues
I created a Test Configuration with values "IE:9" and another with "IE:10" and respectively labeled the web client machines with the same tags.  Running the IE9 tests from MTM would always pick the machine tagged IE:9 and the IE10 ones would pick the machine tagged IE:10.  Running both together would look for a machine labelled IE:9,10 which it wouldn't find and would fail.  Unfortunately I have no way around this limitation currently.

When running coded ui tests via a lab build you can only select one Test Configuration anyway so this limitation is only a problem when manually running the tests in MTM.

The test controller plugins don't appear to be documented, which may mean this breaks in a future update.

Cross Browser Support
This idea can be extended to apply to other browsers too using the cross browser feature from update 1 simply create an extra machine for each browser you wish to test and label correctly.  I'd probably change my labelling scheme from IE:9 to Browser:IE9 to be bit more consistant.  On each machine add a system environment variable called browser set to the correct name, then in your coded UI test make sure you always grab this value and set BrowserWindow.CurrentBrowser to its value.

Advantages over other methods

• Tests can be run from MTM without needing to manually select a different environment to run it in (as long as you only run one Test Configuration at a time)
• Machines other than the test agents can be reused (eg database/webserver) meaning less hyper-v resources are needed
• Works fine with TFS2012's auto config of test clients which causes issues for the multiple test client role method

Potential Extensions (not sure if they're possible)

• A way to disable/enable this filtering (an extra property in a test configuration would easily do it)
• I'm still looking for ways to allow multiple Test Configurations to be run in one test run from MTM.
• Overriding the browser environment variable automatically so I can use the same virtual machine for IE, Chrome and Firefox (obviously still needing a second machine for IE9 vs IE10).
• Check if any other types of filters are available, I did see some xml floating around suggesting you may be able to filter based on Ram available.

Microsoft Test Manager / Lab Management Misconceptions

As we were not lucky enough to have Visual Studio 2010 Ultimate my knowledge of MTM and LM were fairly piecemeal, this meant when I finally got to install it as part of 2012 (available to Premium as of 2012) I found out a lot of things didn't work like I had expected or were simply completely missing.

Test Configurations and Machine Tags are not related
While at first glance it appeared I could label a machine with "OS: WindowsXP" and create a respective test configuration with the same key value pair, MTM doesn't seem to actually do anything with them.

Machine Tags appear to have no function at all, except to help organize machines.

Test Configurations seem mainly for manual testing, so you can manually test each software configuration you feel is important. If you record your manual test it seems to be available to every configuration and can therefore be overwritten. I'm presuming this to make it easy to rerun the same test on different machines which means you can use it to manually test a windows app in different versions of windows, or multiple versions of IE.  The same restriction applies to coded ui tests, you can only apply one coded ui test to a test case which must apply for all test configurations.

It should be possible to specify that the machine in the environment that the test is run must have x, y and z machine tags letting you test all sorts of combinations like Windows XP + IE 7 + No Flash or Windows 8 with UAC enabled.  Instead it appears that tests are split into groups of about 100 and dished out between the available test agents in the environment.

UserVoice Suggestion

Microsoft recommended Work Around - Configuration matrix testing using Visual Studio Lab Management although it's limited to 2 configs (eg IE / Firefox) and would need to be manually extended to support more and would need to be upgraded to 2012.  It also means you need extra capacity for virtual machines as you can't have IE and Firefox on the same machine.  There are issues with this approach in 2012 as the environments will have their agents automatically deployed (or removed) depending on their roles.  Basically you can't use MTM to update the environment, or use snapshots if you want to use this approach.

Another potential work around that I've come up with is to use a Test Controller plugin to filter available test agents.

Test Playback in browsers other than IE is only supported for Visual Studio Coded UI Tests
Ok, no big deal, I'll just convert my test to a Coded UI one (which seem to be more reliable anyway) then associate it back to the testcase.


MTM Recorded Tests don't support assertions
Unless you convert to VS CUIT you can't automatically perform checks at the end to ensure the desired behaviour occurred.  Again no big deal, CUIT seem the better option in the long run anyway.


Coded UI Tests aren't passed the Test Configuration
While it's easily possible to switch between IE/Firefox/Chrome in code, you can't seem to do this based on the Test Configuration.  You can use a csv DataSource attribute to run an automated test multiple times and switch the browser for each run, but that doesn't seem as nice as simply adding a new Test Configuration.

Now I'm confused as to what the point is of test configurations is for automated tests, as it appears to basically just run the same test 4 times in the same environment on the same test agent.

The Lab Management Build Template can only run one configuration
If you want to run the tests in multiple configurations it appears you need to setup multiple lab builds, an advantage of this does mean you can set a different environment per build which does mean you can test different versions of IE.

Lab Environments cannot have machines added (or removed)
You can add/remove machines from templates, but not environments once they've been deployed.  This just means you need to be careful that you don't need to customise machines heavily after deploy to get your environment up as you may need to redeploy on occasion.


Auto running tests on build in VS starts coded ui tests

Couldn't find a way to disable this, although I heart the latest CTP (2012.2) may be improving something in this area.

Conclusion
I don't mean for this to be an overly negative post, MTM & LM do seem to be very powerful but I am struggling to figure out how to best utilize them for my needs.

Tfs Automation - keep user images in sync with active directory

In a previous post I shows how to set user images to be same as active directory in a console application.

I have updated this code to run as a recurring ITeamFoundationJobExtension .
 The first file creates the job in TFS to run every 24 hours, and runs it once right now.

The second file is the updated code that is now a TFS Agent Job, this required a few changes to the code to run against the server api instead of the client one. They're fairly similar but the client api tends to be a bit more friendly.
Installation
Copy the compiled job to c:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\TFSJobAgent\plugins and restart the agent service. Then run the console application to register the job. Finally check the state of the job by looking in the tfs database

select * from tfs_configuration.dbo.tbl_JobHistory where jobid = 'fa60c04e-c996-413e-8151-15933f5a2bac'