The human urge to replace Makefiles with something more modern
Make is showing its age as it nears 50. What would it take to replace it with a program without so many pitfalls?
Modern web software projects have code written in a bunch of languages. If you have an API and power a web service and the 2 major apps, you might have Go, Javascript, Kotlin, and Swift running in production. These languages will need to be compiled or bundled, which means that each of them needs a build. Managing all of these builds by hand is tedious and error-prone. So we use build systems to make this job easier. Build systems integrate with the operating system and the compiler to provide a simplified interface for the user. Some build systems target one language, but many build systems can build anything. It saves a huge amount of time.
Note: I’m going to capitalize program names like Make, Just. and Bazel so that the post is more legible. Typically they would be stylized like make
and just
but I didn’t find it very legible.
Today’s newsletter is about Make. In the absence of a language-specific build system, Make is the best option simply because it is ubiquitous. It’s almost 50 years old, and is the flavor of build system that can build anything. It also comes from a time when programs were built like bear traps1. The user would periodically lose their hand and some guy would poke his head in and say, “I am sorry about your hand. But as you can clearly see in the documentation, hand removal is clearly the defined behavior for that case. Please read the manual next time.” By the time you’re yelling at the guy to shut up and give you medical attention, you can hear him in the next room explaining that it should be called “GNU plus Linux and not simply Linux.” Nobody likes living this way, and nobody likes this sanctimonious guy. But to truly live the Unix lifestyle, you need all of it. The powerful tool. The bloody stump. The reminder that you had not memorized the 10,000 word manual and every single one of its implications. And Make is the cornerstone of the Unix lifestyle.
Some developers will absolutely hate that I called Make the “best option” because quite honestly there are probably a dozen superior options. But in practice Make comes preinstalled on almost everything but Windows2. Do you want people around the world to build your project in a variety of environments? Then use Make. They have it already and it already works. Look no further than the Kubernetes project, which itself exclusively uses Make after having deleted a separate Google-specific build system. Google loves reinventing the wheel and trying to force their better tool upon the unwashed masses. And even they fell back to Make after their build tool proved to be too much of a maintenance burden.
It’s not all sunshine and roses. Make is showing its age. By design, Make wants to deeply understand the entire dependency chain in your project so that it can efficiently do rebuilds. If it knows that a file and all of its dependencies haven’t changed, it could potentially save a multiple-minute rebuild. But in practice, large Makefiles are often difficult to write and maintain, which is why programs like autotools and CMake exist.
But most modern language ecosystems already provide their own single-purpose build systems for their language. If you’re using Javascript, you probably have a dozen to choose from by now. These single-purpose build systems are superior at handling their targeted language. Developers still use Make’s dependency features to enforce behaviors in the build like “this command may only be run from the main branch,” but in practice Make nowadays is often more like a pile of scripts than a true build system.
Given this reality, there have been efforts to replace full-fledged build systems like Make. These tend to fall into three categories.
In the first category, you have the “next generation program that can also build everything imaginable.” One example of this is Bazel, the build system that was stripped out of Kubernetes. Bazel is an open-source version of Google’s internal build system, Blaze. It requires that the user very carefully specify every single dependency and output and side effect and program output and build file, and in return it will give you unparalleled speed, control, and reproducible builds. But in practice, Bazel is best when it can solve problems that large companies have like “wow we are wasting so much time compiling everything all the time; it would be great if developers could share the compiled versions of things.” If you try using Bazel for a distributed project like Kubernetes, you’re going to be eaten alive by problems like “versions of Bazel that are compiled years apart from each other are wildly incompatible.”
In the second category, you have “Scripts. I only use scripts.” The idea here is that since every programming language is starting to have its own single-purpose build system, you write your own scripts that defer to them. Push the client bundles to a CDN or the respective app store, containerize and run the Go binaries, and then you have a fully-fledged website!
In practice, scripts start to run into the opposite problem of Make. They are underpowered unless you spend a tremendous amount of time on them, and when you add more scripts for complex projects used by hundreds of developers, the pile of ad-hoc scripts you’ve assembled start to resemble a build system. But this is a build system without 50 years of bugfixes, and without a community of thousands of developers that are tirekicking new changes. On the plus side, if you’re a shop with scripting language expertise like Ruby or Python, then writing and managing maintainable scripts is within your power. But in practice, most people will be writing Bash scripts and dealing with its arcane syntax.
In the third category, you have command runners. Just is one such command runner, and learning about it recently was the impetus for this whole newsletter. Just threads the needle nicely between scripts and Make. It is actually inspired by the syntax of Make, but is not a fully-fledged build system. Instead, it is a command runner that allows you to set dependencies between the commands, and additionally use your preferred scripting language for writing the commands and deferring to the language-specific build tools. The implication is that each tool is the expert in running and building its own code, so you should leave the expertise to the experts.
Just has a nice mental model. You’re still able to do neat things like “define an easy way to prevent commands from being run on branches besides main” or “split out commands that build parts of your project, and roll them up into your development and production builds,” but it drops a bunch of complexity on the floor by not even trying to be a build system.
Make obviously isn’t going anywhere. But I like the implication that we’re still exploring the space, both far from Make (in tools like Bazel), and in the Make extended universe (with tools like Just). But we all know that there is one single best build system for all purposes. Coincidentally this is the one that you, dear reader, happen to like, that I might not have even mentioned.
My most recent example: I accidentally named an environment variable the same thing as a Makefile variable last month and the documented behavior in this case was so unexpected that I wasted an hour of debugging and documentation hunting.
And on Windows, you can use Windows Subsystem for Linux (WSL) to install it.