The Seven Deadly Sins of Build Systems

Build systems tend to be among the messiest code around. Arguably, many many coding sins are being perpetrated there, so it's kind of hard to pare it down to seven. 

I do think the following are the most common, and also the simplest to fix.

1. Changing Files In Place

As far as sins go, this one is rather minor. There may even be some good reasons to do this, but in most cases where I've seen it happen, it was due to a reluctance to admit that there actually is a proper, full scale code generation problem at hand.

Best is to avoid doing it: rename the file, for example by appending a .in  or a .template suffix, and generate the target file via the build system.

Situations where this may be inconvenient are rare, but do exist. For example if the generation step is not useful for developers, developers might want a way to skip it - especially if the code generation is to generate files used in an IDE. Better of course would be to teach the IDE how to generate the files themselves, but that may be impractical in some cases.

If you must do it:

• ensure your modification is "idem-potent", i.e. it doesn't care if has been run before
• ensure it's OK for the modified files to be checked in as such. 

2. Mixing up Version Control with Builds

This sin is fortunately getting rarer, as people are adopting saner version control systems like mercurial or git. Nevertheless, it is still common that a build system will either access multiple branches of the source tree at the same time, or even perform checkouts.

Remedy is to collect your sources prior to starting the build.

If you are using a version control system that conflates branches with source trees (e.g. perforce and subversion), don't let branches appear in the source tree. Use the version control system to prepare your source tree from the right branches for you and only then let the build system loose.


3. Build Systems Performing Version Control Checkins

Unfortunately very common. Most common is abusing the version control system to perform duties that should be performed by a different service, for example an artifact repository  (to store built binaries) or registry system (to store generated configurations or logs). If you do this, the obvious next question becomes how you would resolve merge conflicts stemming from those checkins. Assume they never happen?


4. Mixing up Build Configuration with Runtime Configuration

This is very common. This likely stems from the way most open source software is packaged and deployed:

• Unpack source
• configure
• make install

This works brilliantly as long as you don't actually modify the sources (and you're willing to live with having a compiler installed on your production machines), and if your runtime environment is stable.

The important thing people miss is that this is a packaging and installation model, not a development model - and as a distribution model it might not even be good for you, as many shops wouldn't dream of shipping their sources and expecting their customers to build it locally.

Unfortunately, the lore has won, and many build systems continue to be modeled like open source packages, with two main effects:

• Your build system spends a lot of time moving files around, aka installing them (where?)
• Any change in the runtime environment (install location, names of other services, host names etc) requires a rebuild

The latter point introduces delays and risks during testing, as your software migrates between different environments to eventually end up in production.


5. Mixing up Build with Packaging and with Deploy

This is mostly just sloppy coding. At this day and age, it shouldn't require convincing that encapsulation, separation of concerns and clean APIs are good things. When designing build systems, it is helpful to consider a couple of little disaster scenarios:

• Pretend you must change your version control system, now!
• Pretend you must switch to a different packaging system, now of course!
• Pretend your switching deploy strategies (e.g. from an incremental update to wholesale re-imaging)

None of these should require any surgery in the build system itself.


6. Builds Spill Outside Their Designated Area

Otherwise known as "works on my machine". Three sources of evil:

• Uncatalogued dependencies on outside files
• Undocumented and unvalidated dependencies on the environment
• Writing to locations shared by others

The antidote is to ensure to never write anywhere you don't have exclusive access to, thereby insuring you do not interfere with any other activities on the build host, to always clean out and establish a well defined environment and to specifically test and validate any toolchain elements assumed pre-installed on the build host.


7. Labeling Builds

This one epitomizes the strong belief in rituals often found in build engineering.  All version control systems have ways to identify a unique source code configuration. If they didn't, they wouldn't be source code control systems. It is sufficient to register the parameters required to recreate the source code configuration. You need some sort of registry no matter what you do - and using labels for that job can actually be a lot more complicated than one might think at first:

• You need to generate a label name. The complication here is about what to do when multiple builds are done at the same time, using the same source code configuration (e.g. multi-platform builds): use the same label? use multiple labels?
• You need to ensure the labeling completes. In some systems, labeling is not an atomic operation. Instead, every file is labeled individually.

The Mercurial version control system exposes the complications of making labels work correctly in a decentralized environment by forcing you to check in and merge labels.

In the end, you will be producing many thousands of builds each year, and the vast majority of them are useless, never to be referenced again. Keeping track of all those labels  can be quite a burden.

Instead: label the important builds, after you know that they really are important (e.g. when the products built from those sources are released). Then the presence of a label actually means something, and may help answer questions like "Where did you want me to clone from?".