Tfs Extensibility - Automatically Create Code Reviews on Checkin

I created a small plugin that has a percentage chance to create code review requests on checkin. You could enhance this pretty easily to create reviews using a more complex condition (based on how long its been since they last had a review, or the size of the checkin etc) however I've found the 5% rule to be fairly successful mainly because people have gotten used to the review feature and have started requesting code reviews manually for things they feel are important.

The major gotcha I found while coding this was the need to impersonate the user performing the checkin as the creator is the only user who can close the review (which isn't very useful when its the service account).

Tfs Extensibility - Filtering Lab Management Test Agents by Test Configuration

Original Problem
As previously mentioned there's no built-in way to pick a test agent automatically based on the selected test configuration, after much digging I have found a partial work around.

Theory
It turns out that like a lot of tfs components the test controller supports plugins.  These inherit from TcmRunControllerPlugin and have 3 methods - Init, TestRunStarting and TestRunCompleted.  Unlike other TFS Plugins they are not automatically detected and need to be manually added to the qtcontroller.config (in c:\program files (x86)\microsoft visual studio 11.0\common7\ide).

I found that on the TestRunConfiguration object there is a StringDictionary called AgentProperties, upon further inspection I noticed that it appeared to be used to filter test agents to find a suitable one to run the tests on.  By default it has a filter for environment name, as a small test I added another hardcoded filter by overriding the Init method in a very simple plugin.  To my surprise whenever tests were run they would now complain that no suitable agent could be found, by changing my filter it quickly came apparent that this method worked for machine tags.

By overriding the Init method and a bit more digging I managed to get a list of properties from the Test Configurations and put them as filters for the agent selected.

Limitations/Issues
I created a Test Configuration with values "IE:9" and another with "IE:10" and respectively labeled the web client machines with the same tags.  Running the IE9 tests from MTM would always pick the machine tagged IE:9 and the IE10 ones would pick the machine tagged IE:10.  Running both together would look for a machine labelled IE:9,10 which it wouldn't find and would fail.  Unfortunately I have no way around this limitation currently.

When running coded ui tests via a lab build you can only select one Test Configuration anyway so this limitation is only a problem when manually running the tests in MTM.

The test controller plugins don't appear to be documented, which may mean this breaks in a future update.

Cross Browser Support
This idea can be extended to apply to other browsers too using the cross browser feature from update 1 simply create an extra machine for each browser you wish to test and label correctly.  I'd probably change my labelling scheme from IE:9 to Browser:IE9 to be bit more consistant.  On each machine add a system environment variable called browser set to the correct name, then in your coded UI test make sure you always grab this value and set BrowserWindow.CurrentBrowser to its value.

Advantages over other methods

• Tests can be run from MTM without needing to manually select a different environment to run it in (as long as you only run one Test Configuration at a time)
• Machines other than the test agents can be reused (eg database/webserver) meaning less hyper-v resources are needed
• Works fine with TFS2012's auto config of test clients which causes issues for the multiple test client role method

Potential Extensions (not sure if they're possible)

• A way to disable/enable this filtering (an extra property in a test configuration would easily do it)
• I'm still looking for ways to allow multiple Test Configurations to be run in one test run from MTM.
• Overriding the browser environment variable automatically so I can use the same virtual machine for IE, Chrome and Firefox (obviously still needing a second machine for IE9 vs IE10).
• Check if any other types of filters are available, I did see some xml floating around suggesting you may be able to filter based on Ram available.

Microsoft Test Manager / Lab Management Misconceptions

As we were not lucky enough to have Visual Studio 2010 Ultimate my knowledge of MTM and LM were fairly piecemeal, this meant when I finally got to install it as part of 2012 (available to Premium as of 2012) I found out a lot of things didn't work like I had expected or were simply completely missing.

Test Configurations and Machine Tags are not related
While at first glance it appeared I could label a machine with "OS: WindowsXP" and create a respective test configuration with the same key value pair, MTM doesn't seem to actually do anything with them.

Machine Tags appear to have no function at all, except to help organize machines.

Test Configurations seem mainly for manual testing, so you can manually test each software configuration you feel is important. If you record your manual test it seems to be available to every configuration and can therefore be overwritten. I'm presuming this to make it easy to rerun the same test on different machines which means you can use it to manually test a windows app in different versions of windows, or multiple versions of IE.  The same restriction applies to coded ui tests, you can only apply one coded ui test to a test case which must apply for all test configurations.

It should be possible to specify that the machine in the environment that the test is run must have x, y and z machine tags letting you test all sorts of combinations like Windows XP + IE 7 + No Flash or Windows 8 with UAC enabled.  Instead it appears that tests are split into groups of about 100 and dished out between the available test agents in the environment.

UserVoice Suggestion

Microsoft recommended Work Around - Configuration matrix testing using Visual Studio Lab Management although it's limited to 2 configs (eg IE / Firefox) and would need to be manually extended to support more and would need to be upgraded to 2012.  It also means you need extra capacity for virtual machines as you can't have IE and Firefox on the same machine.  There are issues with this approach in 2012 as the environments will have their agents automatically deployed (or removed) depending on their roles.  Basically you can't use MTM to update the environment, or use snapshots if you want to use this approach.

Another potential work around that I've come up with is to use a Test Controller plugin to filter available test agents.

Test Playback in browsers other than IE is only supported for Visual Studio Coded UI Tests
Ok, no big deal, I'll just convert my test to a Coded UI one (which seem to be more reliable anyway) then associate it back to the testcase.


MTM Recorded Tests don't support assertions
Unless you convert to VS CUIT you can't automatically perform checks at the end to ensure the desired behaviour occurred.  Again no big deal, CUIT seem the better option in the long run anyway.


Coded UI Tests aren't passed the Test Configuration
While it's easily possible to switch between IE/Firefox/Chrome in code, you can't seem to do this based on the Test Configuration.  You can use a csv DataSource attribute to run an automated test multiple times and switch the browser for each run, but that doesn't seem as nice as simply adding a new Test Configuration.

Now I'm confused as to what the point is of test configurations is for automated tests, as it appears to basically just run the same test 4 times in the same environment on the same test agent.

The Lab Management Build Template can only run one configuration
If you want to run the tests in multiple configurations it appears you need to setup multiple lab builds, an advantage of this does mean you can set a different environment per build which does mean you can test different versions of IE.

Lab Environments cannot have machines added (or removed)
You can add/remove machines from templates, but not environments once they've been deployed.  This just means you need to be careful that you don't need to customise machines heavily after deploy to get your environment up as you may need to redeploy on occasion.


Auto running tests on build in VS starts coded ui tests

Couldn't find a way to disable this, although I heart the latest CTP (2012.2) may be improving something in this area.

Conclusion
I don't mean for this to be an overly negative post, MTM & LM do seem to be very powerful but I am struggling to figure out how to best utilize them for my needs.

Tfs Automation - keep user images in sync with active directory

In a previous post I shows how to set user images to be same as active directory in a console application.

I have updated this code to run as a recurring ITeamFoundationJobExtension .
 The first file creates the job in TFS to run every 24 hours, and runs it once right now.

The second file is the updated code that is now a TFS Agent Job, this required a few changes to the code to run against the server api instead of the client one. They're fairly similar but the client api tends to be a bit more friendly.
Installation
Copy the compiled job to c:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\TFSJobAgent\plugins and restart the agent service. Then run the console application to register the job. Finally check the state of the job by looking in the tfs database

select * from tfs_configuration.dbo.tbl_JobHistory where jobid = 'fa60c04e-c996-413e-8151-15933f5a2bac'

Tfs Automation - Automatically creating queries when new areas are created

Our testers got sick of setting up new queries whenever they created a new area in TFS so asked me if I could automate it.
The solution is broken into 2 parts:

AreaCreatedSubscriber

This is ISubscriber that watches for StructureChangedNotification events and queues a one time job to create the queries.
The reason I queued a job instead of just doing the work in this plugin was that when creating the query TFS was throwing an exception on validation (TF51011: The specified area path does not exist.) It seems that there is a slight delay between adding areas and them becoming valid for use in queries. I thought it was a caching issue but even when explicitly clearing the cache on the workitemstore it still happens, I found a post (New Area Path values are not available in work items) that suggested that tfs subscribes to its own events which then triggers a process of making the area available. This meant I needed to delay the creation of the queries until after all the events had completed, hence the job queue.
Also it is fairly good practice to push any event semi long running tasks into a job as it can delay actions for end users.
This gets installed to c:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\Web Services\bin\Plugins

CreateDefaultQueriesJob

The second part is the actual job, its also fairly simple (although crudely written). It reads the Area path out of the event data and creates a folder hierarchy for the queries to live in. It then checks if the queries already exist and if not creates them.
This job should probably be extended/cleanedup to read the queries from an external source, whether it be a WIQL file on the disk, some metadata stored on the team project, or just a special folder in source control.
This is installed into c:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\TFSJobAgent\plugins

Debugging

I had a bunch of issues trying to get this to work.
Firstly the TF51011 error - folders were appearing in the shared queries so I knew the plugin was doing something, luckily the exception was being logged to the event logs so it was really easy to find.
After moving the logic to a one time job I tried testing it only to find nothing happened. No folders created and nothing logged in the event logs. I figured that the event was still being fired so it was probably queueing the job, after a quick google I came across a blog article from Martin Hinshelwood debugging the TFS Active directory sync job this gave me a bunch of helpful debugging tips.
This lead me to the tfs project collection database
SELECT * FROM [Tfs_Enlighten].[dbo].[tbl_JobDefinition] where ExtensionName like '%CreateDefaultQueriesJob'
Returned rows but they weren't much help, jobs were being queued but I had no idea about if they were successful or not. I eventually discovered another table in the tfs_configuration database which contained the jobHistory
select * from tfs_configuration.dbo.tbl_JobHistory where jobid in (SELECT [JobId] FROM [Tfs_Enlighten].[dbo].[tbl_JobDefinition] where ExtensionName like '%CreateDefaultQueriesJob')
Also had one row for every time I queued the job, with the status result of 6, this time I needed to do some diving with ILSpy, which lead me to this enum

ExtensionNotFound = 6 - TFS couldn't find my job! I double and triple checked all my code, and started diving through the JobRunner code in TFS trying to figure out why it wasn't loading. I noticed a few hints of MEF in the code so tried adding an [Export] attribute to the class, still no luck. Enabling the trace log by editing c:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\TFSJobAgent\TfsJobAgent.exe.config I started seeing a curious exception about a plugin being added to a dictionary twice.

Especially interesting as it appears to check if the plugin is already registered in the dictionary before adding it, which makes me think they have a threading bug.
After removing the [Export] attribute and restarting the tfs job service a few times it seemed to come right.

Update: Apparently there's some new web access interfaces to view the status of TFS Jobs (and more), may have meant less database diving had I known at the time

Tfs Automation - Create Builds Automatically for New Solutions

I recently wrote a tfs server plugin that creates new builds automatically whenever you add or branch a solution file.  It's fairly simple, but worth sharing.

It takes advantage of another piece of code I had previously written (or perhaps found online, I honestly can't remember now) to create builds using the TFS api.

Installing it is very simple, just build the class into an assembly and copy it to c:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\Web Services\bin\Plugins you may also need to copy a couple of the referenced dlls to the same folder.

Creating a build using the Tfs API

Here is a snippet of code to create tfs builds using the API with a few sensible default settings.

Web Code Coverage for Manual Tests

Found a great little tool which I think may be new in VS/TFS 2012 (it was possible in previous versions but seemed much harder to setup)

 C:\Program Files (x86)\Microsoft Visual Studio 11.0\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe collect /iis /output:c:\outputfile.coverage

 It will restart IIS and record a coverage file which can be later opened in visual studio. Great to set going when testers are doing any manual testing in a development environment

Very Simple TFS 2012 Web Access Plugin

Made a very basic tfs plugin to make the tfs logo a link back to the homepage.

Manifest.xml
clickable.min.js
Then zip them up together and upload it in the tfs admin section.

Obviously not exactly the best plugin in the world, considering we actually bypass the whole plugin system, but it works and may be of help to someone.

TFS Extensibility

In this post I hope to outline all the extensibility points in TFS 2012 and come up with a reason I might use it for the company I work for.  I would define this as both a major strength and weakness in TFS, there's a lot of powerful stuff exposed to developers but there's not a lot of documentation and a fair bit of it seems half complete which all results in very little adoption.  The community around customizing TFS seems fairly small in my opinion, but could be a major win for Microsoft.

Tfs Client API - .NET
A set of .NET classes to interact with the various webservices exposed by TFS, this is the main interaction point from which Visual studio and all components that don't need to be installed on the TFS server directly, for this reason it is extremely powerful.

We mainly use it to build a custom TFS web frontend for our clients for logging bugs which is a bit more cut down than the TFS web access. The need for this has reduced drastically with the release of TFS 2012 however it does mean we can integrate other systems giving our clients a one stop site for all their needs (rather than requiring them to login to 3-4 different systems).  We are careful that users can only see any work items that they create so that they don't require a special TFS CAL. The built in boards and backlog in TFS 2012 require a higher level CAL, but you can build your own backlog/board that works under the limited CAL.

We also make use of the API in a few ways to help manage the TFS Server. For example automatically granting permissions to TFS teams for any areas/iterations they're associated with, that way I can just assign people to project teams and they'll have the permissions they need.  Another example is a set of sanity checks which do things like making sure every solution in source control has an associated build.

Tarun Arora, Neno Loje, and Shai Raiten all blog extensively on this subject.

TFS Build Templates - XAML, .NET
TFS Builds in 2010 and higher are controlled by .NET Workflows, this includes a lot of the Lab management functionality.  There are lots of builtin activities you can add to your templates and you can write your own activities in Visual studio and include you (or use one of the many activities that are part of the community tfs build extensions).

I'm not a fan of the editor and using custom assemblies causes all kinds of pain just to get the custom activities into Visual Studio so you can add them.  You can't easily just grab a component off someones blog, add it to your template, test it and deploy it.  This gets even more complex when you've already customized your template so can't just replace the entire template and have to add their changes manually.

For the above reasons I typically try keep template customizations to a minimum.  I believe the only customization I have currently is to set the default build quality of a successful build so that I can make use of TFS Deployer to deploy the same binaries in turn to each environment for testing.  TFS Deployer then makes use of powershell scripts, which I find much easier to customise and test.

TFS Soap Subscriptions - SOAP
The tfs api (mentioned above) allows you to subscribe to a number of events via soap webservices, this allows your app to get notified when any of the following events occurs in TFS.

• Work item changes
• Check ins
• Build completes
• Build quality changes (this is what TFS Deployer uses)
• Project is created or deleted
• Branch moved
• Security Acl changes
• Iteration/Area changes

Note: The last 4 may be client events only, meaning only the client that triggers the event gets notified.

Back in 2005/2008 we used to use this notification service to send out emails when a workitem was assigned (instead of getting every developer to create a subscription themselves), in 2010 we migrated this to use the server events instead and finally replaced it with a single out of the box team subscription in tfs 2012.

These days I don't make use of any of these events myself (except obviously build quality change). I can imagine subscribing to area/iteration changes to automatically create folders in SharePoint, much like the proposed TFS Automation Platform from Martin Hishelwood, sadly this project never seemed to get off the ground.

TFS Server-side Event Handlers - .NET
Location: %Program Files%\Microsoft Team Foundation Server 11.0\Application Tier\Web Services\bin\Plugins

TFS 2010 introduced the ability to create server side event subscribers by implementing the ISubscriber interface and dropping the resulting assembly into the plugin directory and access the same events as the soap subscriptions.

These seemed a lot more reliable than the soap subscriptions, but had to process fairly quickly as it blocked TFS from finishing what it was doing. As mentioned earlier we used this to send email alerts for assigned work items, if you ever did a bulk update (100 work items) TFS would lock up and not respond for up to a minute.  The solution was either to spin up a background thread (which I believe had a chance to be killed) or queue a custom TFS Job.

Because TFS waits for you to process it means you can use events as Decision Points to allow or deny certain actions from occurring.  This means you can add extra custom validation to workitems and checkins, there are better ways to do both but sometimes you need access to code to do more advanced checks.

Apart from emailing notifications, I've seen a plugin that can associate workitems to checkins based on the comments (and heard of someone doing the reverse - setting comments based on the associated workitems).

Update: I created a few sample server side event plugins that automatically creates builds for new solution files, automatically creates workitem queries for new areas

MTM/TestCase server notifications are also available in 2012.

Check In Policies - .NET
Location: Developer machines

Code that can ensure particular requirements are met before a checkin can occur, examples:

• Project Compiles
• Tests pass
• Comments added
• WorkItem associated
• Passes StyleCop checks
• WorkItems associated are in particular areas/queries
• Regex patterns on filenames/contents

New policies can be created by extending the PolicyBase class, and deploying them to your development machines.

These are annoying to deploy as they get installed on each developers machine, not the TFS server. They became easier in VS2010 with the ability to install using a vsix, in Visual Studio 2012 it became possible to create private extension galleries to help deploy extensions within your enterprise.  However as far as I'm aware there is still no automated way to deploy one of these when your developers connect to a TFS server.

I try not discourage my developers from checking in so I keep the policies fairly light.  We currently require a comment, but encourage other practices just by word of mouth.  I think it would be possible to prompt the user if they want to create a code review request in 2012 on checkin, which may be a good way to introduce users to code review, however it would likely be annoying fast and probably still harder than just informing developers manually.

TFS Job Agent - .NET
Location: %ProgramFiles%\Microsoft Team Foundation Server 11.0\Application Tier\TFSJobAgent\plugins\

TFS runs a number of jobs in the background using its job agent and service (Processing the cube, email subscriptions, AD sync).  You can either write your own custom tasks by implementing the  ITeamFoundationJobExtension interface, or fire off existing tasks when you feel like it.

Custom Jobs are a good choice when you need to do a lot of work in the background, the problem I mentioned earlier when TFS hanging when trying to update 100+ workitems due to the email plugin could be solved by writing an event handler that queues a custom job.
Update: I created a sample server side job plugin that automatically creates workitem queries for new areas

Warehouse/Cube Customization - .NET
Location: %ProgramFiles%\Microsoft Team Foundation Server 11.0\Application Tier\TFSJobAgent\plugins\

TFS builds its own data warehouse from scratch, this is likely because work item fields can be configured to push information through to the warehouse/cube as either a dimension or measure.  TFS supports the interface IWarehouseAdapter which allows the various parts of TFS to add fields to the warehouse as it is getting built.  Developers can also take advantage of this interface and write their own custom data warehouse adapter.

The above example is regarding TFS 2008 which I have heard does not work in 2010, unfortunately there seems to be no real documentation from Microsoft and very few people trying to build a custom adapter.  I would suggest that maybe a tool like ILSpy and looking at the built in adapters would be the way to go.

In 2010 it looks like the warehouse processing was merged into the job service, and any IWarehouseAdapter have been converted to extend the abstract class WarehouseAdapter.  There's also a notion of WebhouseJobExtension which are the ITeamFoundationJobExtension jobs that presumably kick off the schema and data updates int the adapters.

Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter looks to be a good example of a warehouse adapter.

Deployment Services - REST, .NET
TFS 2012 update 1 adds in a new build definition and services that allow you to deploy websites to azure. The code behind this actually looks a lot more powerful and gives me the impression that they're going to open it up for others to start publishing not only deployment methods but other services that can interact with TFS using OAuth.

Using WCF Storm I discovered the following webservices.

http://tfs:8080/tfs/{TeamProjectCollection}/services/v1.0/ConnectedServicesService.asmx

• CreateConnectedService
• QueryConnectedServices
• GetConnectedService

http://tfs:8080/tfs/{TeamProjectCollection}/Build/v4.0/BuildDeploymentService.asmx

• CreateDeploymentEnvironment
• GetDeploymentEnvironments

http://tfs:8080/tfs/{TeamProjectCollection}/services/v1.0/StrongBoxService.asmx

• GetDrawerContents
• GetString

Once you register a deployment service (eg Azure, or a custom built one) you can add environments to it with a custom set of properties (any keyvalue pair). You can then query all enviroments for a given service and pull properties out of its strong box drawer.

I haven't looked at the workflow activities at all to see if they would support calling a custom service.


Custom WorkItem Controls - .NET, Javascript
It's fairly well known that you can customize the fields for any workitem, however you can also write your own custom controls to display on workitems.

With the totally new Web Acess in TFS 2012 there's a new way to write custom controls.


There's a codeplex project that contains many work item tracking custom controls along with their source, and look like they may be updated for TFS 2012 (the visual studio integration side anyway).

Web Access Extensibility - Javascript
The Web access allows you to upload new javascript based plugins, apparently the team were not happy with the state of this at the time of release and are currently reworking them before they make many details public.

It also appears that the different sections of Web Access are built using the above plugins, two controllers (one for the view, one for json) and an AreaRegistration to register them. It may be possible to create new sections just by creating a new AreaRegistration and dropping it in the bin directory.


Update - below are some extensibility points I was either unaware of at the time of writing, or didn't think worth mentioning (visual studio extensions) as everyone already knows about them.

Test Controller Plugins

I've never seen anyone else use one of these, but they appear to exist and work well. I've written one to help filter available test agents by their machine tags for the current test config.

Diagnostic Data Adapters
Custom Adapter – Part 1


MTM Extensions
Example Extension TestScribe for MTM

Visual Studio Extensions
Developing Visual Studio Extensions

Summary
There's lots of extensibility points in TFS but a number of them seem half finished, or not documented.  It could be that most of them are only designed for internal Microsoft use, but it wouldn't take too much to make them very useful for other developers.