How to fuck up software releases

Published 2019-10-12 on Drew DeVault's blog

I manage releases for a bunch of free & open-source software. Just about every time I ship a release, I find a novel way to fuck it up. Enough of these fuck-ups have accumulated now that I wanted to share some of my mistakes and how I (try to) prevent them from happening twice.

At first, I did everything manually. This is fine enough for stuff with simple release processes - stuff that basically amounts to tagging a commit, pushing it, and calling it a day. But even this gets tedious, and I’d often make a mistake when picking the correct version number. So, I wrote a small script: semver. semver patch bumps the patch version, semver minor bumps the minor version, and semver major bumps the major version, based on semantic versioning. I got into the habit of using this script instead of making the tags manually. The next fuckup soon presented itself: when preparing the shortlog, I would often feed it the wrong commits, and the changelog would be messed up. So, I updated the script to run the appropriate shortlog command and pre-populate the annotated tag with it, launching the editor to adjust the changelog as necessary.

Soon I wanted to apply this script to other projects, but not all of them used semantic versioning. I updated it to work for projects which just use major.minor versions as well. However, another problem arose: some projects have the version number specified in the Makefile or I would frequently fuck this up in many creative ways: forgetting it entirely; updating it but not committing it; updating it and committing it, but tagging the wrong commit; etc. wlroots in particular was difficult because I also had to update the soversion, which had special requirements. To address these issues, I added a custom .git/_incr_version script which can add additional logic on a per-repo basis, and updated semver to call this script if present.1

Eventually, I went on vacation and shipped a release while I was there. The _incr_version script I had put into .git on my home workstation wasn’t checked into version control and didn’t come with me on vacation, leading to yet another fucked up release. I moved it from .git/_incr_version to contrib/_incr_version. I made the mistake, however, of leaving the old path in as a fallback, which meant that I never noticed that another project’s script was still in .git until I went on another vacation and fucked up another release. Add a warning which detects if the script is at the old path…

Some of my projects don’t use semantic versioning at all, but still have all of these other gotchas, so I added an option to just override the automatic version increment with a user-specified override. For a while, this worked well. But, inevitably, no matter how much I scripted away my mistakes I would always find a new and novel way of screwing up. The next one came when I shipped a release while on an Alpine Linux machine, which ships Busybox instead of GNU tools. Turns out Busybox gzip produces output which does not match the GNU output, which means the tarballs I signed locally differed from the ones generated by Github. Update the signing script to save the tarball to disk (previously, it lived in a pipe) and upload these alongside the releases…2

Surely, there are no additional ways to fuck it up at this point. I must have every base covered, right? Wrong. Dead wrong. On the very next release I shipped, I mistakenly did everything from a feature branch, and shipped experimental, incomplete code in a stable release. Update the script to warn if the master branch isn’t checked out… Then, of course, another fuckup: I tagged a release without pulling first, and when I pushed, git happily rejected my branch and accepted the tag - shipping an outdated commit as the release. Update the script to git pull first…

I am doomed to creatively outsmart my tools in releases. If you’d like to save yourself from some of the mistakes I’ve made, you can find my semver script here.

  1. Each of these _incr_version scripts proved to have many bugs of their own, of course. 

  2. Eli Schwartz of Arch Linux also sent a patch to Busybox which made their gzip implementation consistent with GNU’s. 

Have a comment on one of my posts? Start a discussion in my public inbox by sending an email to ~sircmpwn/ [mailing list etiquette]

Are you a free software maintainer who is struggling with stress, demanding users, overwork, or any other social problems in the course of your work? Please email me — I know how you feel, and I can lend a sympathetic ear and share some veteran advice.

Articles from blogs I follow around the net

Updates in March 2020

This post gives an overview of the recent updates to the Writing an OS in Rust blog and the corresponding libraries and tools. I focused my time this month on finishing the long-planned post about Async/Await. In addition to that, there were a few updates …

via Writing an OS in Rust April 1, 2020

Wayland clipboard and drag & drop

Clipboard and drag & drop are arguably one of the most complicated parts of the core Wayland protocol. They involve a lot of back-and-forth communication between three processes: the application where some content has been copied, the compositor, and …

via emersion March 26, 2020

Go, the Go Community, and the Pandemic

Go always comes second to more basic concerns like personal and family health and safety. Around the world, the past couple months have been terrible, and we are still at the start of this awful pandemic. There are days when it seems like wo…

via The Go Programming Language Blog March 25, 2020

Generated by openring