In development it’s quite often that someone will commit something bad to source control, and as a consequence everyone else will “pull” the bad changes and have the infected code on their machines too. Before you know it, the bad code has spread to most developer pc’s and development grinds to a halt. It’s a pain most people will have experienced and can be costly to the business. So, how to stop it…here are 7 steps.

1) Build Sound

A very simple, effective, and usually quite easy step to implement. A build sound tells the whole office if the build has passed or failed. You don’t need to be at your desk to know either.

Set up a dedicated pc, with a small speaker, and using something like CCTray (if you use CruiseControl.Net) you can select sounds for when the build passes or fails (and continues to fail). We currently use a Homer Simpson “Woohoo” and “D’oh!” for pass and fail.

2) Jabber Notifications

For those of you that like to develop with headphones on, so are oblivious to the build noise, there are Jabber notifications. We use Jabber chat rooms as part of our development, mainly as a way of broadcasting something to the team, asking questions etc… We still actively encourage face to face communication but in some instances a jabber message is more appropriate (especially if some people haven’t had a recent shot of coffee/nicotine).

Anyway, Jabber notifications can be set up in a number of ways. If you’re using Team City then it’s inbuilt. We don’t, so instead use a NAnt task for sending notifications to our “Development” chat room.

We’ve basically wrapped Jabber-Net into a NAnt task, that fires should the build fail. You can then use the nant.onfailure/nant.onsuccess events to trigger the jabber task. I can’t take credit for it, I had wanted to do it for ages, but thanks to Matt we know have it implemented. It’s definitely something worth setting up, and does make quite a difference.

3) A Quick Build

If a build takes 10 minutes to run, a number of commits will have been made in that time, which makes it harder to figure out who broke it. Also people are less inclined to hang around and wait for it to pass. Keeping the build quick (a minute or two) ensures you know as soon as possible when the build has broken. With regards to speeding up the build, there are a few pointers here.

4) A Separated Build

This relates to speeding up the build, but we’ve broken our build into a number of CruiseControl.Net build projects. The typical build does an msbuild, aspnet_compile and runs the unit tests. We also have separate build projects for integration tests (as they take quite a while) and coding standards (ie StyleCop).

This keeps things more granular, and doesn’t completely stop the flow of work. For example, if the coding standards build has broken, it shouldn’t halt all development. It should still be fixed, but has less importance compared to if the msbuild/aspnet_compile/unit-tests failed.

5) Prioritise Breakages

If the build breaks, it is a priority to get it fixed. Whoever broke it should ideally fix it, but if they are not about, or are stuck, the rest of the team should make it their priority to get it fixed as soon as possible.

There’s no real way of enforcing this. It’s just good practise and something a team should discipline themselves to do. The quicker you fix it the quicker development can continue, and the less likely someone else is to commit a breaking change while the build is already broken.

6) Stop Commits Once Broken

This isn’t something we’ve implemented yet, but are looking into. Using svn-hooks it is possible to stop further commits if the build is already broken. This stops bad code being committed on top of bad code (and an already broken build), and makes fixing the build much easier.

There’s nothing more annoying than fixing some bad code, waiting for a green light, only to get a fail because someone else has committed more bad code.

7) Move to Git?

OK, so it doesn’t directly stop the build breaking as such. But it’s a superior source control system to Subversion, and merging/cloning/pushing/pulling is much better so chances of bad code being committed are reduced purely because a better system is in place. No more having to commit some code so someone else can pull it down to work on it, even though the code isn’t quite complete or stable yet. You can just use Git to pull from the persons pc, rather than having to do so via a central repository a la Subversion.

Unfortunately Git isn’t quite there yet for Windows development. It does work, and reasonably well (I use it for personal projects) but as a solution, it’s lacking mature tools like TortoiseSvn and VisualSvn. There is TortoiseGit but it’s still relatively young, but there is no VisualGit which is a shame. Plus SSH is a bit painful on Windows, or at least I find it so (being a Linux user at home). I’ll hopefully be “spiking” trying it out within our environment in the near future.

I’ve already written an article about trying Git as an alternative to Subversion that explains the pains and shows you how to initially get set up.

Colin Grossman

Request a Demo

© 2006 - 2019. All Rights Reserved.