Test Configurations and Machine Tags are not related
While at first glance it appeared I could label a machine with "OS: WindowsXP" and create a respective test configuration with the same key value pair, MTM doesn't seem to actually do anything with them.
Machine Tags appear to have no function at all, except to help organize machines.
Test Configurations seem mainly for manual testing, so you can manually test each software configuration you feel is important. If you record your manual test it seems to be available to every configuration and can therefore be overwritten. I'm presuming this to make it easy to rerun the same test on different machines which means you can use it to manually test a windows app in different versions of windows, or multiple versions of IE. The same restriction applies to coded ui tests, you can only apply one coded ui test to a test case which must apply for all test configurations.
It should be possible to specify that the machine in the environment that the test is run must have x, y and z machine tags letting you test all sorts of combinations like Windows XP + IE 7 + No Flash or Windows 8 with UAC enabled. Instead it appears that tests are split into groups of about 100 and dished out between the available test agents in the environment.
UserVoice Suggestion
Microsoft recommended Work Around - Configuration matrix testing using Visual Studio Lab Management although it's limited to 2 configs (eg IE / Firefox) and would need to be manually extended to support more and would need to be upgraded to 2012. It also means you need extra capacity for virtual machines as you can't have IE and Firefox on the same machine. There are issues with this approach in 2012 as the environments will have their agents automatically deployed (or removed) depending on their roles. Basically you can't use MTM to update the environment, or use snapshots if you want to use this approach.
Another potential work around that I've come up with is to use a Test Controller plugin to filter available test agents.
Test Playback in browsers other than IE is only supported for Visual Studio Coded UI Tests
Ok, no big deal, I'll just convert my test to a Coded UI one (which seem to be more reliable anyway) then associate it back to the testcase.
MTM Recorded Tests don't support assertions
Unless you convert to VS CUIT you can't automatically perform checks at the end to ensure the desired behaviour occurred. Again no big deal, CUIT seem the better option in the long run anyway.
Coded UI Tests aren't passed the Test Configuration
While it's easily possible to switch between IE/Firefox/Chrome in code, you can't seem to do this based on the Test Configuration. You can use a csv DataSource attribute to run an automated test multiple times and switch the browser for each run, but that doesn't seem as nice as simply adding a new Test Configuration.
Now I'm confused as to what the point is of test configurations is for automated tests, as it appears to basically just run the same test 4 times in the same environment on the same test agent.
The Lab Management Build Template can only run one configuration
If you want to run the tests in multiple configurations it appears you need to setup multiple lab builds, an advantage of this does mean you can set a different environment per build which does mean you can test different versions of IE.
Lab Environments cannot have machines added (or removed)
You can add/remove machines from templates, but not environments once they've been deployed. This just means you need to be careful that you don't need to customise machines heavily after deploy to get your environment up as you may need to redeploy on occasion.
Auto running tests on build in VS starts coded ui tests
Couldn't find a way to disable this, although I heart the latest CTP (2012.2) may be improving something in this area.
Conclusion
I don't mean for this to be an overly negative post, MTM & LM do seem to be very powerful but I am struggling to figure out how to best utilize them for my needs.
