A few days ago I bit the bullet and upgraded our on premises Team Foundation Server from TFS 2015 to TFS 2015 Update 4. As I sat there, after hours, watching the update spinner with the IT department on speed dial, it made me reflect on whether putting in on premises TFS was the best idea. If we had gone for Visual Studio Online (now Visual Studio Team Services) I wouldn’t have to do upgrades – ever.

Flashback 2 years: we had a mishmash of ALM tools that didn’t talk to each other, a git repository that worked at a snail’s pace and a deployment process that involved manually editing csproj files but never checking them in. We had to change. We discussed online vs on premises and after some pen chewing, fingernail biting and some proper pondering we went for on premises. Did we do the right thing? Here is why we took the decision to go on premises and on reflection here’s if I think we did the right thing.

1. Code stored offsite

With Visual Studio Team Services (VSTS) your code is clearly stored offsite on someone else’s machine – your crown jewels are stored under someone else’s bed. This shouldn’t be a problem. Microsoft bunkers are secure, more secure than what I could come up with. Surprisingly this was an issue and various stakeholders where very much happier with code being stored by us. I’ve had very experienced IT staff visit us since then and they were also happy to see we had complete control over our code. So on-premises won out on that score even if it was perception over reality.

Was I right?

Yes and no. Our source code is probably better off in Microsoft’s super secure bunkers but if the decisions makers want the code with us then that’s fine. I didn’t push for it to be stored offsite and I wouldn’t push for it now.

2. High levels of Customisation

This wasn’t a Greenfield project and we had pieces that needed to be integrated into TFS.  With on premises we could break it open and code against it. At the time with Team Services you got what you were given. Our two pain points were

1. MSpec tests. We could only get these under continuous integration with a lot of on-premises custom tinkering
2. Labelling build versions. I put a piece in to code against the TFS object model to get the build versions automatically populating and integrating this into continuous build. Again custom on premise tinkering but well worth it.

Was I right?

Yes and No – we ultimately retired MSpec but for a year about a third of our test coverage was using this so needed to be included. Labelling build versions is still really important to us and works well. But VSTS has come on a lot and would now do most of this if not all.

3. Infrastructure

With on-premises you are responsible for your own servers – you are masters of you own bits of tin. You want more infrastructure then you will need to find it. I didn’t think this would be a problem

Was I right?

No. Infrastructure became a big issue. Disk space ran out and I had to keep scavenging around for more. Our builds proliferated and our build server became very under powered. It took a long time to get another – we just didn’t have the rack space. Don’t underestimate the infrastructure – if the TFS implementation is successful them people will want more and more of it.

4. Quality of Internet access

2 years ago our internet was powered by a very tired hamster on a wheel. It wasn’t good – our existing online tools were causing a headache. No-one had much appetite for further reliance on our poor internet connectivity. On-premises seemed the better option.

Was I right?

Yes but not for long. We got a better Internet connection after 6 months so we could have ultimately used Team Services. But I think an initial 6 months of a poor user experience would have led to a rejection of the implementation. Not everyone in the world has great Internet so Team Services is always going to battle against that.

5. Upgrading

With Team Services your upgrade is done for you. You are always on the latest version. With on premises you do your own upgrades when you have the required levels of emotional resilience to embark on it. I didn’t think this would be a problem.

Was I right?

No. Upgrading is a pain. I’ve done it twice and it worked well both times but there is a lot of thought that needs to go into rollback plans and minimising outages. Now I’ve gone from TFS 2013 to 2015 R4 the next is 2017 and that’s going to take an upgrade to SQL Server as well. It will be much more difficult next time.

6. Amending Bug  and Work Item definitions

At the time Team Services didn’t offer customisation of the process templates i.e. you couldn’t put in extra field, dropdowns and process flows into your project tracking and defect management. This was a deal breaker for us – we had absolute requirements around this. Over two years we have done a lot of customisation of the templates and we really benefit. Once again customisation is king.

Was I right?

Yes and No. Team Services has added some customisation but it is still work in progress. We couldn’t and still wouldn’t wait for a feature that may or may not happen. We needed that level of customisation – it wasn’t just a nice to have. That said this feature is definitely on the road map for VSTS so any concerns I have will continue to evaporate.

7. Availability

Team Services promises 99.9% availability.  On-premises is going to struggle to complete with this particularly when factoring in downtime for upgrades. I didn’t think this would be an issue for on-premises.

Was I right?

Yes – it wasn’t an issue. Over 2 years we didn’t get anywhere near 99.9% for availability but it didn’t cause problems. Outages were planned and developers could keep developing, testers keep testing and project managers could keep doing whatever it is that project managers do. It was fine.

8. Other Issues

There are a few other issues that didn’t come up at the time but I would probably consider if I was to take the decision today

Costing

The costing model for on premises and VSTS is comparable currently.  You can get monthly users of on premise TFS with an online subscription. From the costing page

Each paid user in your Team Services account also gets a TFS client access license (CAL). This lets you buy monthly access to TFS for your team.

This really helps out when you are ramping up testing and need a whole bunch of new users. I’m just a little wary if this will always be the case – monthly users are baked into VSTS but I could imagine them being quietly dropped from on-premises. Paranoid maybe (maybe not).

Support for extensions

I was interested in a decent wiki for TFS but the extension is just for VSTS. The extension ecosystem seems better supported for VSTS. I suspect it always will be.

The future

I don’t need to climb inside the mind of the Microsoft Board to understand that there is a big focus on cloud offerings. VSTS fits nicely within this. TFS on-premises feels a little bit like a relic now. Perhaps not but I would be wary about ongoing support for the on premise version for small shops. I wouldn’t want to be chopping and changing my ALM solution every couple of years so I would have an eye on what the future may hold.

So … on-premise or online?

From my own experience I would say that for

Greenfield project with flexible ALM requirements
Pick VSTS – it’s easier to get going

Established project with requirements for high levels of customisation
Consider on premises but do review VSTS – it might do everything you need. It probably does by now.

For large shops with big IT departments and deep pockets
Go for on premises if it suits and you feel better with your code in house. Infrastructure and upgrading will be no problem. You’ll eat it up

For small shops with a small/non-existent IT department
Be realistic about your IT capacity and willingness to upgrade. Even if stakeholders are screaming for on-premises be certain about your own capacities.

And if I was to implement a Microsoft ALM today?
I would go for Visual Studio Team Services though on-premise was the right decision at the time.

As ever this is just my opinions. Feel free to disagree and if you post a comment telling me why I’m terribly wrong – that would be great.

Useful Links

Visual Studio Online home page
https://www.visualstudio.com/vso/

Customising process templates
https://www.visualstudio.com/en-us/docs/work/reference/process-templates/customize-process

Team Foundation Server object model – which I used to do a fair bit of customisation to our on-premises installation
https://www.nuget.org/packages/Microsoft.TeamFoundationServer.ExtendedClient/14.89.0