Thursday, January 30, 2014

Building Xamarin Applications on a TFS Build Server

Enabling Android Builds


  • Install the android SDK to a common path
  • Edit the TFS Build definition and in advanced settings find the "MSBuild Arguments" property and set it to "/p:AndroidSdkDirectory=E:\Android\android-sdk" obviously with the correct path
  • Edit the *.csproj file for the android project and change the first line so that DefaultTargets="Build" become DefaultTargets="SignAndroidPackage".
  • Queue a build in tfs - the APK should be put in the drop folder along with the other binaries.


Enabling iOS Builds

  • Open up the iPhone *.csproj file in a text editor, change the first line so that DefaultTargets="Build" become DefaultTargets="Build;_TfsRemoteBuild". And add a conditional propertygroup and targets
<PropertyGroup Condition=" '$(BuildingInsideVisualStudio)'!='true' ">
          <ServerAddress>Name or IP of Mac Build Server</ServerAddress>
          <BSAT>NjY3MjY5</BSAT>
         <HttpPort>5000</HttpPort>
  </PropertyGroup>
  <Target Name="_TfsRemoteBuild" Condition=" '$(BuildingInsideVisualStudio)'!='true' " DependsOnTargets="_RemoteBuild">

BSAT is the base64 encoded Pin you used to connect to the iOS build agent from Visual Studio



The server committed a protocol violation. Section=ResponseStatusLine
Unknown solution.

http://forums.xamarin.com/discussion/3696/how-to-build-from-msbuild-command-line-trig-build-on-osx-from-windows

Failed to fetch manifest from the build server

  • Ensure the build server and the Mac build server are running the same version of Xamarin 

Other Tips

  • Remember to use build server tags to mark any build controller you setup to build Xamarin builds so that you can filter by just compatible build servers when creating a build definition
  • You will need to login to your build server setup your Xamarin license in visual studio - the licenses are stored in a common place so it shouldn't matter who you login as to set this up.

Friday, September 13, 2013

Automatically create NuGet packages for TFS/SharePoint dlls

I write a number of TFS Plugins, meaning I end up referencing all sorts of TFS assemblies. This entails trawling through the various client and server installation directories trying to find the particular dll I'm looking for.

As I like my projects to build without needing to install any extra dependencies manually, this means either I include the files when I check into source control, or more recently I manually create a NuGet package for each dll and put it in my private repository.

Since I also have a habit of updating to the latest version of TFS often I also find that I need to repeat this process quite often - finding all these assemblies, creating NuGet packages, then updating my plugins and redeploying

I finally got sick of it enough to write a small powershell script to find any assembly starting with Microsoft.TeamFoundationServer.* and create packages, including package references to other packages.

This script should also work for other products like SharePoint with a few tweaks

Sunday, June 23, 2013

Using Mvc 4 Display Modes for Feature Toggles

It occurred to me that it's possible to use the new Mvc 4 feature Display Mode (meant for switching views for mobile versions) as a complement for Feature Toggles.

Obviously this can only switch out an entire view (or partial view) but it can be a nice alternative to putting lots of @if(featureEnabled) {} tags through your view.

Add the following code to your global.asax Application_Start (or create an App_Start module).

 DisplayModeProvider.Instance.Modes.Insert(0, new DefaultDisplayMode("{NameOfFeature}")
{
     ContextCondition = (context => {true/false logic})
});

Then create 2 views - one for the feature being disabled and one for it being enabled - eg Index.cshtml and Index.{NameOfFeature}.cshtml.

This could also be used for similar concepts like permissions and a/b testing.

Thursday, June 20, 2013

Tfs Extensibility - Automatically Create Code Reviews on Checkin

I created a small plugin that has a percentage chance to create code review requests on checkin. You could enhance this pretty easily to create reviews using a more complex condition (based on how long its been since they last had a review, or the size of the checkin etc) however I've found the 5% rule to be fairly successful mainly because people have gotten used to the review feature and have started requesting code reviews manually for things they feel are important.

The major gotcha I found while coding this was the need to impersonate the user performing the checkin as the creator is the only user who can close the review (which isn't very useful when its the service account).

Friday, February 15, 2013

Tfs Extensibility - Filtering Lab Management Test Agents by Test Configuration

Original Problem
As previously mentioned there's no built-in way to pick a test agent automatically based on the selected test configuration, after much digging I have found a partial work around.

Theory
It turns out that like a lot of tfs components the test controller supports plugins.  These inherit from TcmRunControllerPlugin and have 3 methods - Init, TestRunStarting and TestRunCompleted.  Unlike other TFS Plugins they are not automatically detected and need to be manually added to the qtcontroller.config (in c:\program files (x86)\microsoft visual studio 11.0\common7\ide).

I found that on the TestRunConfiguration object there is a StringDictionary called AgentProperties, upon further inspection I noticed that it appeared to be used to filter test agents to find a suitable one to run the tests on.  By default it has a filter for environment name, as a small test I added another hardcoded filter by overriding the Init method in a very simple plugin.  To my surprise whenever tests were run they would now complain that no suitable agent could be found, by changing my filter it quickly came apparent that this method worked for machine tags.

By overriding the Init method and a bit more digging I managed to get a list of properties from the Test Configurations and put them as filters for the agent selected.

Limitations/Issues
I created a Test Configuration with values "IE:9" and another with "IE:10" and respectively labeled the web client machines with the same tags.  Running the IE9 tests from MTM would always pick the machine tagged IE:9 and the IE10 ones would pick the machine tagged IE:10.  Running both together would look for a machine labelled IE:9,10 which it wouldn't find and would fail.  Unfortunately I have no way around this limitation currently.

When running coded ui tests via a lab build you can only select one Test Configuration anyway so this limitation is only a problem when manually running the tests in MTM.

The test controller plugins don't appear to be documented, which may mean this breaks in a future update.

Cross Browser Support
This idea can be extended to apply to other browsers too using the cross browser feature from update 1 simply create an extra machine for each browser you wish to test and label correctly.  I'd probably change my labelling scheme from IE:9 to Browser:IE9 to be bit more consistant.  On each machine add a system environment variable called browser set to the correct name, then in your coded UI test make sure you always grab this value and set BrowserWindow.CurrentBrowser to its value.

Advantages over other methods
  • Tests can be run from MTM without needing to manually select a different environment to run it in (as long as you only run one Test Configuration at a time)
  • Machines other than the test agents can be reused (eg database/webserver) meaning less hyper-v resources are needed
  • Works fine with TFS2012's auto config of test clients which causes issues for the multiple test client role method

Potential Extensions (not sure if they're possible)
  • A way to disable/enable this filtering (an extra property in a test configuration would easily do it)
  • I'm still looking for ways to allow multiple Test Configurations to be run in one test run from MTM.
  • Overriding the browser environment variable automatically so I can use the same virtual machine for IE, Chrome and Firefox (obviously still needing a second machine for IE9 vs IE10).
  • Check if any other types of filters are available, I did see some xml floating around suggesting you may be able to filter based on Ram available.


Tuesday, February 12, 2013

Microsoft Test Manager / Lab Management Misconceptions

As we were not lucky enough to have Visual Studio 2010 Ultimate my knowledge of MTM and LM were fairly piecemeal, this meant when I finally got to install it as part of 2012 (available to Premium as of 2012) I found out a lot of things didn't work like I had expected or were simply completely missing.

Test Configurations and Machine Tags are not related
While at first glance it appeared I could label a machine with "OS: WindowsXP" and create a respective test configuration with the same key value pair, MTM doesn't seem to actually do anything with them.

Machine Tags appear to have no function at all, except to help organize machines.

Test Configurations seem mainly for manual testing, so you can manually test each software configuration you feel is important. If you record your manual test it seems to be available to every configuration and can therefore be overwritten. I'm presuming this to make it easy to rerun the same test on different machines which means you can use it to manually test a windows app in different versions of windows, or multiple versions of IE.  The same restriction applies to coded ui tests, you can only apply one coded ui test to a test case which must apply for all test configurations.

It should be possible to specify that the machine in the environment that the test is run must have x, y and z machine tags letting you test all sorts of combinations like Windows XP + IE 7 + No Flash or Windows 8 with UAC enabled.  Instead it appears that tests are split into groups of about 100 and dished out between the available test agents in the environment.

UserVoice Suggestion

Microsoft recommended Work Around - Configuration matrix testing using Visual Studio Lab Management although it's limited to 2 configs (eg IE / Firefox) and would need to be manually extended to support more and would need to be upgraded to 2012.  It also means you need extra capacity for virtual machines as you can't have IE and Firefox on the same machine.  There are issues with this approach in 2012 as the environments will have their agents automatically deployed (or removed) depending on their roles.  Basically you can't use MTM to update the environment, or use snapshots if you want to use this approach.

Another potential work around that I've come up with is to use a Test Controller plugin to filter available test agents.

Test Playback in browsers other than IE is only supported for Visual Studio Coded UI Tests
Ok, no big deal, I'll just convert my test to a Coded UI one (which seem to be more reliable anyway) then associate it back to the testcase.


MTM Recorded Tests don't support assertions
Unless you convert to VS CUIT you can't automatically perform checks at the end to ensure the desired behaviour occurred.  Again no big deal, CUIT seem the better option in the long run anyway.


Coded UI Tests aren't passed the Test Configuration
While it's easily possible to switch between IE/Firefox/Chrome in code, you can't seem to do this based on the Test Configuration.  You can use a csv DataSource attribute to run an automated test multiple times and switch the browser for each run, but that doesn't seem as nice as simply adding a new Test Configuration.

Now I'm confused as to what the point is of test configurations is for automated tests, as it appears to basically just run the same test 4 times in the same environment on the same test agent.

The Lab Management Build Template can only run one configuration
If you want to run the tests in multiple configurations it appears you need to setup multiple lab builds, an advantage of this does mean you can set a different environment per build which does mean you can test different versions of IE.

Lab Environments cannot have machines added (or removed)
You can add/remove machines from templates, but not environments once they've been deployed.  This just means you need to be careful that you don't need to customise machines heavily after deploy to get your environment up as you may need to redeploy on occasion.


Auto running tests on build in VS starts coded ui tests
Couldn't find a way to disable this, although I heart the latest CTP (2012.2) may be improving something in this area.


Conclusion
I don't mean for this to be an overly negative post, MTM & LM do seem to be very powerful but I am struggling to figure out how to best utilize them for my needs.

Thursday, February 7, 2013

Tfs Automation - keep user images in sync with active directory

In a previous post I shows how to set user images to be same as active directory in a console application.

I have updated this code to run as a recurring ITeamFoundationJobExtension .
 The first file creates the job in TFS to run every 24 hours, and runs it once right now.

The second file is the updated code that is now a TFS Agent Job, this required a few changes to the code to run against the server api instead of the client one. They're fairly similar but the client api tends to be a bit more friendly.
Installation
Copy the compiled job to c:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\TFSJobAgent\plugins and restart the agent service. Then run the console application to register the job. Finally check the state of the job by looking in the tfs database

select * from tfs_configuration.dbo.tbl_JobHistory where jobid = 'fa60c04e-c996-413e-8151-15933f5a2bac'

Wednesday, December 19, 2012

Tfs Automation - Automatically creating queries when new areas are created

Our testers got sick of setting up new queries whenever they created a new area in TFS so asked me if I could automate it.
The solution is broken into 2 parts:

AreaCreatedSubscriber

This is ISubscriber that watches for StructureChangedNotification events and queues a one time job to create the queries.
The reason I queued a job instead of just doing the work in this plugin was that when creating the query TFS was throwing an exception on validation (TF51011: The specified area path does not exist.) It seems that there is a slight delay between adding areas and them becoming valid for use in queries. I thought it was a caching issue but even when explicitly clearing the cache on the workitemstore it still happens, I found a post (New Area Path values are not available in work items) that suggested that tfs subscribes to its own events which then triggers a process of making the area available. This meant I needed to delay the creation of the queries until after all the events had completed, hence the job queue.
Also it is fairly good practice to push any event semi long running tasks into a job as it can delay actions for end users.
This gets installed to c:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\Web Services\bin\Plugins

CreateDefaultQueriesJob

The second part is the actual job, its also fairly simple (although crudely written). It reads the Area path out of the event data and creates a folder hierarchy for the queries to live in. It then checks if the queries already exist and if not creates them.
This job should probably be extended/cleanedup to read the queries from an external source, whether it be a WIQL file on the disk, some metadata stored on the team project, or just a special folder in source control.
This is installed into c:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\TFSJobAgent\plugins

Debugging

I had a bunch of issues trying to get this to work.
Firstly the TF51011 error - folders were appearing in the shared queries so I knew the plugin was doing something, luckily the exception was being logged to the event logs so it was really easy to find.
After moving the logic to a one time job I tried testing it only to find nothing happened. No folders created and nothing logged in the event logs. I figured that the event was still being fired so it was probably queueing the job, after a quick google I came across a blog article from Martin Hinshelwood debugging the TFS Active directory sync job this gave me a bunch of helpful debugging tips.
This lead me to the tfs project collection database
SELECT * FROM [Tfs_Enlighten].[dbo].[tbl_JobDefinition] where ExtensionName like '%CreateDefaultQueriesJob'
Returned rows but they weren't much help, jobs were being queued but I had no idea about if they were successful or not. I eventually discovered another table in the tfs_configuration database which contained the jobHistory
select * from tfs_configuration.dbo.tbl_JobHistory where jobid in (SELECT [JobId] FROM [Tfs_Enlighten].[dbo].[tbl_JobDefinition] where ExtensionName like '%CreateDefaultQueriesJob')
Also had one row for every time I queued the job, with the status result of 6, this time I needed to do some diving with ILSpy, which lead me to this enum

ExtensionNotFound = 6 - TFS couldn't find my job! I double and triple checked all my code, and started diving through the JobRunner code in TFS trying to figure out why it wasn't loading. I noticed a few hints of MEF in the code so tried adding an [Export] attribute to the class, still no luck. Enabling the trace log by editing c:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\TFSJobAgent\TfsJobAgent.exe.config I started seeing a curious exception about a plugin being added to a dictionary twice.

Especially interesting as it appears to check if the plugin is already registered in the dictionary before adding it, which makes me think they have a threading bug.
After removing the [Export] attribute and restarting the tfs job service a few times it seemed to come right.

Update: Apparently there's some new web access interfaces to view the status of TFS Jobs (and more), may have meant less database diving had I known at the time

Friday, December 7, 2012

Tfs Automation - Create Builds Automatically for New Solutions

I recently wrote a tfs server plugin that creates new builds automatically whenever you add or branch a solution file.  It's fairly simple, but worth sharing.

It takes advantage of another piece of code I had previously written (or perhaps found online, I honestly can't remember now) to create builds using the TFS api.

Installing it is very simple, just build the class into an assembly and copy it to c:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\Web Services\bin\Plugins you may also need to copy a couple of the referenced dlls to the same folder.

Creating a build using the Tfs API

Here is a snippet of code to create tfs builds using the API with a few sensible default settings.