header

Combining SCons and Ninja builds

After considering several software build systems for Swift a few years ago, we ended up settling on SCons for its correctness, power, and maintainability. The main thing we traded this for was speed: although we could get a no-op build down to only a few seconds by using the right flags, those few seconds can still get in the way when doing very short build/test/debug cycles during daily development. To shorten the turnaround time during development, I created a script to use SCons to generate a build file for Ninja, a small but very fast build system, which gets our no-op builds down to a fraction of a second.

Ninja’s intent is to act as the low-level ‘assembler’ back end for other, higher level build systems such as CMake and GYP. Since SCons is a complete build tool instead of a build file generator, there’s no natural way to plug Ninja in as a different back end inside SCons itself. However, it is possible to automatically generate a Ninja build file by reverse engineering a log of all the commands SCons executes for a (clean) build, which is exactly what the scons2ninja.py script does.

How to use it

The procedure to use Ninja with SCons is bootstrapped by calling the scons2ninja.py script once, with the parameters you would pass to SCons for doing a normal build, for example:

BuildTools/scons2ninja.py debug=1 optimize=0

This will generate the build.ninja file, which you can use from then on to build the project using ninja:

ninja swift
ninja check
ninja -t clean

The build.ninja file will regenerate itself when one of the SConscript files change.

An extra bonus of having a Ninja build file is that Ninja can generate a JSON CLang Compilation Database, which can be used for various CLang-based analysis and transformation tools.

How it works

The script calls SCons, and makes it simulate a dry run (--dry-run) of a full clean build, dumping all the commands it executes. In addition to that, it asks SCons to print the dependency tree of all the files (--tree=all,prune). The script then parses the commands from the log, tries to recognize the exact tool and command line flags, and then generates build rules for each command. It uses the dependency tree from SCons to add dependencies for the build rules (except for C/C++ files, where all dependencies are computed by Ninja from compiler output). For files it does not have commands for (e.g. those generated by a Python function in SCons), it calls SCons to generate the files in question.

Caveats

Unfortunately, scons2ninja isn’t a fully generic SCons/Ninja solution (yet), as it has been tested only with Swift. However, all the Swift-specific stuff is separated out in the .scons2ninja.conf configuration file, which should be generic enough to extend for your own needs.

Apart from this, there are some limitations, such as:

  • It currently only works with the tools and flags that Swift builds use (gcc, clang, Microsoft Visual Studio, Qt, …). Since the script requires full knowledge of all tools run by SCons, it will fail if it finds an unsupported tool. Feel free to send a patch with support for your favorite tools.
  • It does not take into account environment variables etc. set from SCons. Ninja will always use values from the environment from which it’s called.
  • Files that have dynamic content (i.e. content depending on other things than files, such as Values), will not be regenerated automatically. This is a general problem with generator-based build systems. Delete the file if you want it to be regenerated.
  • Files generated using SCons are built one by one, which means you have to pay the SCons startup time for each one. This isn’t too bad in practice, since the number of SCons-generated files is typically small. However, there is some discussion on implementing a ‘batching’ feature in Ninja, which would allow to group all of these together into one SCons invocation.

Published by

Remko Tronçon

Software Engineer · Hobby musician · BookWidgets