Drew DeVault's blog

tar: a slop-free alternative to rsync

Sat, 28 Mar 2026 00:00:00 +0000

So apparently rsync is slop now. When I heard, I wanted to drop a quick note on my blog to give an alternative: tar. It doesn’t do everything that rsync does, in particular identifying and skipping up-to-date files, but tar + ssh can definitely accomodate the use case of “transmit all of these files over an SSH connection to another host”.

Consider the following:

tar -cz public | ssh example.org tar -C /var/www -xz

This will transfer the contents of ./public/ to example.org:/var/www/public/, preserving file ownership and permissions and so on, with gzip compression. This is roughly the equivalent of:

A eulogy for Vim

Wed, 25 Mar 2026 00:00:00 +0000

Vim is important to me. I’m using it to write the words you’re reading right now. In fact, almost every word I have ever committed to posterity, through this blog, in my code, all of the docs I’ve written, emails I’ve sent, and more, almost all of it has passed through Vim.

My relationship with the software is intimate, almost as if it were an extra limb. I don’t think about what I’m doing when I use it. All of Vim’s modes and keybindings are deeply ingrained in my muscle memory. Using it just feels like my thoughts flowing from my head, into my fingers, into a Vim-shaped extension of my body, and out into the world. The unique and profound nature of my relationship with this software is not lost on me.

The cults of TDD and GenAI

Thu, 29 Jan 2026 00:00:00 +0000

I’ve gotten a lot of flack throughout my career over my disdain towards test-driven development (TDD). I have met a lot of people who swear by it! And, I have also met a lot of people who insisted that I adopt it, too, often with the implied threat of appealing to my boss if appealing to me didn’t work.

The basic premise of TDD, for those unaware, is that one first writes a unit test that verifies the expected behavior for some code they want to write, observes the new test fail, and then one writes the implementation, iterating on it until the test passes. The advantage of this approach is, first, to ensure that your codebase is adequately covered by testing, and, second, to provide you a rapid feedback loop to assist in your work.

Redesigning my microkernel from the ground up

Mon, 12 Jan 2026 00:00:00 +0000

As you may recall, circa 2022-2023 I was working on a microkernel written in Hare named Helios. Helios was largely inspired by and modelled after the design of seL4 and was my first major foray into modern OS development that was serious enough to get to a somewhat useful state of functionality, with drives for some real hardware, filesystems, and an environment for running user programs of a reasonable level of sophistication.

OpenAI employees… are you okay?

Sat, 08 Nov 2025 00:00:00 +0000

You might have seen an article making the rounds this week, about a young man who ended his life after ChatGPT encouraged him to do so. The chat logs are really upsetting.

Someone two degrees removed from me took their life a few weeks ago. A close friend related the story to me, about how this person had approached their neighbor one evening to catch up, make small talk, and casually discussed their suicidal ideation at some length. At the end of the conversation, they asked to borrow a rope, and their neighbor agreed without giving the request any critical thought. The neighbor found them the next morning.

What's up with FUTO?

Wed, 22 Oct 2025 00:00:00 +0000

Some time ago, I noticed some new organization called FUTO popping up here and there. I’m always interested in seeing new organizations that fund open source popping up, and seeing as they claim several notable projects on their roster, I explored their website with interest and gratitude. I was first confused, and then annoyed by what I found. Confused, because their website is littered with bizzare manifestos,¹ and ultimately annoyed because they were playing fast and loose with the term “open source”, using it to describe commercial source-available software.

Cloudflare bankrolls fascists

Wed, 24 Sep 2025 00:00:00 +0000

US politics has been pretty fascist lately. The state is filling up concentration camps, engaging in mass state violence against people on the basis of racialized traits, deporting them to random countries without any respect for habeas corpus, exerting state pressure on the free press to censor speech critical of the current administration, and Trump is openly floating the idea of an unconstitutional third term.

Fascism is clearly on the rise, and they’re winning more and more power. None of this is far removed from us in the FOSS community – there are a number of fascists working in FOSS, same as the rest of society. I don’t call them fascists baselessly – someone who speaks out in support of and expresses solidarity with fascists, or who uses fascists dog-whistles or promotes fascist ideology and talking points, or boosts fascist conspiracy theories – well, they’re a fascist.

A better future for JavaScript that won't happen

Wed, 17 Sep 2025 00:00:00 +0000

In the wake of the largest supply-chain attack in history, the JavaScript community could have a moment of reckoning and decide: never again. As the panic and shame subsides, after compromised developers finish re-provisioning their workstations and rotating their keys, the ecosystem might re-orient itself towards solving the fundamental flaws that allowed this to happen.

After all, people have been sounding the alarm for years that this approach to dependency management is reckless and dangerous and broken by design. Maybe this is the moment when the JavaScript ecosystem begins to understand the importance and urgency of this problem, and begins its course correction. It could leave behind its sprawling dependency trees full of micro-libraries, establish software distribution based on relationships of trust, and incorporate the decades of research and innovation established by more serious dependency management systems.

Embedding Wren in Hare

Wed, 20 Aug 2025 00:00:00 +0000

I’ve been on the lookout for a scripting language which can be neatly embedded into Hare programs. Perhaps the obvious candidate is Lua – but I’m not particularly enthusiastic about it. When I was evaluating the landscape of tools which are “like Lua, but not Lua”, I found an interesting contender: Wren.

I found that Wren punches far above its weight for such a simple language. It’s object oriented, which, you know, take it or leave it depending on your use-case, but it’s very straightforwardly interesting for what it is. I found a few things to complain about, of course – its scope rules are silly, the C API has some odd limitations here and there, and in my opinion the “standard library” provided by wren CLI is poorly designed. But, surprisingly, my list of complaints more or less ends there, and I was excited to build a nice interface to it from Hare.

What's new with Himitsu 0.9?

Fri, 08 Aug 2025 00:00:00 +0000

Last week, Armin and I worked together on the latest release of Himitsu, a “secret storage manager” for Linux. I haven’t blogged about Himitsu since I announced it three years ago, and I thought it would be nice to give you a closer look at the latest release, both for users eager to see the latest features and for those who haven’t been following along.¹

A brief introduction: Himitsu is like a password manager, but more general: it stores any kind of secret in its database, including passwords but also SSH keys, credit card numbers, your full disk encryption key, answers to those annoying “security questions” your bank obliged you to fill in, and so on. It can also enrich your secrets with arbitrary metadata, so instead of just storing, say, your IMAP password, it can also store the host, port, TLS configuration, and username, storing the complete information necessary to establish an IMAP session.

Just speak the truth

Mon, 30 Jun 2025 00:00:00 +0000

Today, we’re looking at two case studies in how to respond when reactionaries appear in your free software community.

Exhibit A

It is a technical decision.

The technical reason is that the security team does not have the bandwidth to provide lifecycle maintenance for multiple X server implementations. Part of the reason for moving X from main to community was to reduce the burden on the security team for long-term maintenance of X. Additionally, nobody so far on the security team has expressed any interest in collaborating with xxxxxx on security concerns.

Unionize or die

Mon, 09 Jun 2025 00:00:00 +0000

Tech workers have long resisted the suggestion that we should be organized into unions. The topic is consistently met with a cold reception by tech workers when it is raised, and no big tech workforce is meaningfully organized. This is a fatal mistake – and I don’t mean “fatal” in the figurative sense. Tech workers, it’s time for you to unionize, and strike, or you and your loved ones are literally going to die.

The British Airways position on various border disputes

Mon, 05 May 2025 00:00:00 +0000

My spouse and I are on vacation in Japan, spending half our time seeing the sights and the other half working remotely and enjoying the experience of living in a different place for a while. To get here, we flew on British Airways from London to Tokyo, and I entertained myself on the long flight by browsing the interactive flight map on the back of my neighbor’s seat and trying to figure out how the poor developer who implemented this map solved the thorny problems that displaying a world map implies.

Resistance from the tech sector

Sun, 20 Apr 2025 00:00:00 +0000

As of late, most of us have been reading the news with a sense of anxious trepidation. At least, those of us who read from a position of relative comfort and privilege. Many more read the news with fear. Some of us are already no longer in a position to read the news at all, having become the unfortunate subjects of the news. Fascism is on the rise worldwide and in the United States the news is particularly alarming. The time has arrived to act.

A Firefox addon for putting prices into perspective

Fri, 04 Apr 2025 00:00:00 +0000

I had a fun idea for a small project this weekend, and so I quickly put it together over the couple of days. The result is Price Perspective.

Humor me: have you ever bought something, considered the price, and wondered how that price would look to someone else? Someone in the developing world, or a billionaire, or just your friend in Australia? In other words, can we develop an intuition for purchasing power?

Using linkhut to signal-boost my bookmarks

Thu, 27 Mar 2025 00:00:00 +0000

Notice: linkhut has started to use generative AI tools upstream. Consequently, I have withdrawn my endorsement of the project.

It must have been at least a year ago that I first noticed linkhut, and its flagship instance at ln.ht, appear on SourceHut, where it immediately caught my attention for its good taste in inspirations. Once upon a time, I had a Pinboard account, which is a similar concept, but I never used it for anything in the end. When I saw linkhut I had a similar experience: I signed up and played with it for a few minutes before moving on.

Please stop externalizing your costs directly into my face

Mon, 17 Mar 2025 00:00:00 +0000

This blog post is expressing personal experiences and opinions and doesn’t reflect any official policies of SourceHut.

Over the past few months, instead of working on our priorities at SourceHut, I have spent anywhere from 20-100% of my time in any given week mitigating hyper-aggressive LLM crawlers at scale. This isn’t the first time SourceHut has been at the wrong end of some malicious bullshit or paid someone else’s externalized costs – every couple of years someone invents a new way of ruining my day.

A holistic perspective on intellectual property, part 1

Thu, 13 Feb 2025 00:00:00 +0000

I’d like to write about intellectual property in depth, in this first of a series of blog posts on the subject. I’m not a philosopher, but philosophy is the basis of reasonable politics so buckle up for a healthy Friday afternoon serving of it.

To understand intellectual property, we must first establish at least a shallow understanding of property generally. What is property?¹ An incomplete answer might state that a material object I have power over is my property. An apple I have held in my hand is mine, insofar as nothing prevents me from using it (and, in the process, destroying it), or giving it away, or planting it in the ground. However, you might not agree that this apple is necessarily mine if I took it from a fruit stand without permission. This act is called “theft” — one of many possible transgressions upon property.

Join us to discuss transparency and governance at FOSDEM '25

Thu, 23 Jan 2025 00:00:00 +0000

Good news: it appears that Jack Dorsey’s FOSDEM talk has been cancelled!

This is a follow up to two earlier posts, which you can read here: one and two.

I say it “appears” so, because there has been no official statement from anyone to that effect. There has also been no communication from staff to the protest organizers, including to our email reaching out as requested to discuss fire safety and crowd control concerns with the staff. The situation is a bit unclear, but… we’ll extend FOSDEM the benefit of the doubt, and with it our gratitude. From all of the volunteers who have been organizing this protest action, we extend our heartfelt thanks to the staff for reconsidering the decision to platform Dorsey and Block, Inc. at FOSDEM. All of us – long-time FOSDEM volunteers, speakers, devroom organizers, and attendees – are relieved to know that FOSDEM stands for our community’s interests.

FOSDEM '25 protest

Mon, 20 Jan 2025 00:00:00 +0000

Update: Dorsey’s talk was cancelled! See the update here.

Last week, I wrote to object to Jack Dorsey and his company, Block, Inc., being accepted as main track speakers at FOSDEM, and proposed a protest action in response. FOSDEM issued a statement about our plans on Thursday.

Today, I have some updates for you regarding the planned action.

I would like to emphasize that we are not protesting FOSDEM or its organizers. We are protesting Jack Dorsey and his company, first and foremost, from promoting their business at FOSDEM. We are members of the FOSDEM community. We have variously been speakers, devroom organizers, volunteers, and attendees for years — in other words, we are not activism tourists. We have a deep appreciation for the organizers and all of the work that they have done over the years to make FOSDEM such a success.

No billionaires at FOSDEM

Thu, 16 Jan 2025 00:00:00 +0000

Update: Dorsey’s talk was cancelled! See the update here.

Jack Dorsey, former CEO of Twitter, ousted board member of BlueSky, and grifter extraordinaire to the tune of a $5.6B net worth, is giving a keynote at FOSDEM.

The FOSDEM keynote stage is one of the biggest platforms in the free software community. Janson is the biggest venue in the event – its huge auditorium can accommodate over 1,500 of FOSDEM’s 8,000 odd attendees, and it is live streamed to a worldwide audience as the face of one of the free and open source software community’s biggest events of the year. We’ve platformed Red Hat, the NLNet Foundation, NASA, numerous illustrious community leaders, and many smaller projects that embody our values and spirit at this location to talk about their work or important challenges our community faces.

Neurodivergence and accountability in free software

Wed, 25 Sep 2024 00:00:00 +0000

In November of last year, I wrote Richard Stallman’s political discourse on sex, which argues that Richard Stallman, the founder of and present-day voting member of the board of directors of the Free Software Foundation (FSF), endorses and advocates for a harmful political agenda which legitimizes adult attraction to minors, consistently defends adults accused of and convicted of sexual crimes with respect to minors, and more generally erodes norms of consent and manipulates language regarding sexual harassment and sexual assault in his broader political program.

Rust for Linux revisited

Fri, 30 Aug 2024 00:00:00 +0000

Ugh. Drew’s blogging about Rust again.

– You

I promise to be nice.

Two years ago, seeing the Rust-for-Linux project starting to get the ball rolling, I wrote “Does Rust belong in the Linux kernel?”, penning a conclusion consistent with Betteridge’s law of headlines. Two years on we have a lot of experience to draw on to see how Rust-for-Linux is actually playing out, and I’d like to renew my thoughts with some hindsight – and more compassion. If you’re one of the Rust-for-Linux participants burned out or burning out on this project, I want to help. Burnout sucks – I’ve been there.

So you want to compete with or replace open source

Tue, 16 Jul 2024 00:00:00 +0000

We are living through an interesting moment in source-available software.¹ The open source movement has always had, and continues to have, a solid grounding in grassroots programmers building tools for themselves and forming communities around them. Some looming giants brought on large sums of money – Linux, Mozilla, Apache, and so on – and other giants made do without, like GNU, but for the most part if anyone thought about open source 15 years ago they were mostly thinking about grassroots communities who built software together for fun. With the rise of GitHub and in particular the explosion of web development as an open platform, commercial stakeholders in software caught on to the compelling economics of open source. The open source boom that followed caused open source software to have an enormous impact on everyone working in the software industry, and, in one way or another, on everyone living on planet Earth.

Writing a Unix clone in about a month

Fri, 24 May 2024 00:00:00 +0000

I needed a bit of a break from “real work” recently, so I started a new programming project that was low-stakes and purely recreational. On April 21st, I set out to see how much of a Unix-like operating system for x86_64 targets that I could put together in about a month. The result is Bunnix. Not including days I didn’t work on Bunnix for one reason or another, I spent 27 days on this project.

Copyleft licenses are not “restrictive”

Fri, 19 Apr 2024 00:00:00 +0000

One may observe an axis, or a “spectrum”, along which free and open source software licenses can be organized, where one end is “permissive” and the other end is “copyleft”. It is important to acknowledge, however, that though copyleft can be found at the opposite end of an axis with respect to permissive, it is not synonymous with the linguistic antonym of permissive – that is, copyleft licenses are not “restrictive” by comparison with permissive licenses.

FDO's conduct enforcement actions regarding Vaxry

Tue, 09 Apr 2024 00:00:00 +0000

freedesktop(.org), aka FDO, recently banned Hyprland maintainer Vaxry from the FDO community, and in response Vaxry has taken his case to the court of public opinion, publishing their email exchanges and writing about it on his blog.

It saddens me to bear witness to these events today. I wrote in September of last year about problems with toxicity in the Hyprland community. I initially reached out to Vaxry to discuss these problems in private in February of last year. I failed to get through to him, leading to that blog post in September. I spent some time in the following weeks talking with Vaxry on his behavior and his community’s social norms, again in private, but again, I was unable to get through to him. Unfortunately, we find ourselves again leaving the private sphere and discussing Vaxry’s behavior and the problem posed by the Hyprland community once again.

Why Prusa is floundering, and how you can avoid their fate

Tue, 26 Dec 2023 00:00:00 +0000

Prusa is a 3D printer manufacturer which has a long history of being admired by the 3D printing community for high quality, open source printers. They have been struggling as of late, and came under criticism for making the firmware of their Mk4 printer non-free.¹

Armin Ronacher uses Prusa as a case-study in why open source companies fail, and uses this example to underline his argument that open source needs to adapt for commercial needs, namely by adding commercial exclusivity clauses to its licenses – Armin is one of the principal proponents of the non-free Functional Source License. Armin cites his experience with a Chinese manufactured 3D printer as evidence that intellectual property is at the heart of Prusa’s decline, and goes on to discuss how this dynamic applies to his own work in developing a non-free license for use with Sentry. I find this work pretty interesting – FSL is a novel entry into the non-free license compendium, and it’s certainly a better way to do software than proprietary models, assuming that it’s not characterized as free or open source. But, allow me to use the same case study to draw different conclusions.

Richard Stallman's political discourse on sex

Sat, 25 Nov 2023 00:00:00 +0000

Richard Stallman, the founder of the Free Software Foundation, has been subject to numerous allegations of misconduct. He stepped down in 2019, and following his re-instatement in 2021, a famous open letter was published in which numerous organizations and individuals from throughout the Free Software ecosystem called for his removal from the Free Software Foundation. The letter had no effect; Stallman remains a voting member of the FSF’s board of directors to this day and continues to receive numerous speaking engagements.

Can I be on your podcast?

Thu, 09 Nov 2023 00:00:00 +0000

I am working on rousing the Hare community to get the word out about our work. I have drafted the Hare evangelism guidelines to this effect, which summarizes how we want to see our community bringing Hare to more people.

We’d like to spread the word in a way which is respectful of the attention of others – we’re explicitly eschewing unsolicited prompts for projects to consider writing/rewriting in Hare, as well as any paid sponsorships or advertising. Blog posts about Hare, videos, participating in (organic) online discussions – much better! And one idea we have is to talk about Hare on podcasts which might be interested in the project.

On "real name" policies

Tue, 31 Oct 2023 00:00:00 +0000

Some free software projects reject anonymous or pseudonymous contributions, requiring you to author patches using your “real name”. Such projects have a so-called “real name” policy; Linux is one well-known example.¹

The root motivations behind such policies vary, but in my experience the most often cited rationale is that it’s important to establish the provenance of the contribution for copyright reasons. In the case of Linux, contributors are asked to “sign-off” their commits to indicate their agreement to the terms of the Developer Certificate of Origin (DCO), which includes clauses like the following:

Going off-script

Fri, 13 Oct 2023 00:00:00 +0000

There is a phenomenon in society which I find quite bizarre. Upon our entry to this mortal coil, we are endowed with self-awareness, agency, and free will. Each of the 8 billion members of this human race represents a unique person, a unique worldview, and a unique agency. Yet, many of us have the same fundamental goals and strive to live the same life.

I think of such a life experiences as “following the script”. Society lays down for us a framework for living out our lives. Everyone deviates from the script to some extent, but most people hit the important beats. In Western society, these beats are something like, go to school, go to college, get a degree, build a career, get married, have 1.5 children, retire to Florida, die.

The forbidden topics

Fri, 29 Sep 2023 00:00:00 +0000

There are forbidden topics in the hacker community. One is sternly reprimanded for bringing them up, by their peers, their leaders, and the community at large. In private, one can expect threats and intimidation; in public, outcry and censorship. The forbidden topics are enforced by the moderators of our spaces, taken off of forums, purged from chat rooms, and cleaned up from GitHub issues and mailing lists; the ban-hammers fall swiftly and resolutely. My last article to touch these subjects was removed from Hacker News by the moderators within 30 minutes and landed several death threats in my inbox. The forbidden topics, when raised, are met with a resounding, aggressive dismissal and unconditional condemnation.

Hyprland is a toxic community

Sun, 17 Sep 2023 00:00:00 +0000

Hyprland is an open source Wayland compositor based on wlroots, a project I started back in 2017 to make it easier to build good Wayland compositors. It’s a project which is loved by its users for its emphasis on customization and “eye candy” – beautiful graphics and animations, each configuration tailored to the unique look and feel imagined by the user who creates it. It’s a very exciting project!

Unfortunately, the effect is spoilt by an incredibly toxic and hateful community. I cannot recommend Hyprland to anyone who is not prepared to steer well clear of its community spaces. Imagine a high school boys’ locker room come to life on Discord and GitHub and you’ll get an idea of what it’s like.

AI crap

Tue, 29 Aug 2023 00:00:00 +0000

There is a machine learning bubble, but the technology is here to stay. Once the bubble pops, the world will be changed by machine learning. But it will probably be crappier, not better.

Contrary to the AI doomer’s expectations, the world isn’t going to go down in flames any faster thanks to AI. Contemporary advances in machine learning aren’t really getting us any closer to AGI, and as Randall Monroe pointed out back in 2018:

Hello from Ares!

Wed, 09 Aug 2023 00:00:00 +0000

I am pleased to be writing today’s blog post from a laptop running Ares OS. I am writing into an ed(1) session, on a file on an ext4 filesystem on its hard drive. That’s pretty cool! It seems that a lot of interesting stuff has happened since I gave that talk on Helios at FOSDEM in February.

The talk I gave at FOSDEM was no doubt impressive, but it was a bit of a party trick. The system was running on a Raspberry Pi with one process which included both the slide deck as a series of raster images baked into the ELF file, as well as the GPU driver and drawing code necessary to display them, all in one package. This was quite necessary, as it turns out, given that the very idea of “processes” was absent from the system at this stage.

The rc shell and its excellent handling of whitespace

Mon, 31 Jul 2023 00:00:00 +0000

This blog post is a response to Mark Dominus’ “The shell and its crappy handling of whitespace”.

I’ve been working on a shell for Unix-like systems called rc, which draws heavily from the Plan 9 shell of the same name. When I saw Mark’s post about the perils of whitespace in POSIX shells (or derived shells, like bash), I thought it prudent to see if any of the problems he outlines are present in the shell I’m working on myself. Good news: they aren’t!

Alpine Linux does not make the news

Tue, 25 Jul 2023 00:00:00 +0000

My Linux distribution of choice for several years has been Alpine Linux. It’s a small, efficient distribution which ships a number of tools I appreciate for their simplicity, such as musl libc. It has a very nice package manager, apk, which is fast and maintainable. The development community is professional and focuses on diligent maintenance of the distribution and little else. Over the years I have used it, very little of note has happened.

Seriously, don't sign a CLA

Tue, 04 Jul 2023 00:00:00 +0000

SourceGraph is making their product closed source, abandoning the Apache 2.0 license it was originally distributed under, so once again we convene in the ritual condemnation we offer to commercial products that piss in the pool of open source. Invoking Bryan Cantrill once more:

Bryan Cantrill on OpenSolaris — YouTube

A contributor license agreement, or CLA, usually (but not always) includes an important clause: a copyright assignment. These agreements are provided by upstream maintainers to contributors to open source software projects, and they demand a signature before the contributor’s work is incorporated into the upstream project. The copyright assignment clause that is usually included serves to offer the upstream maintainers more rights over the contributor’s work than the contributor was offered by upstream, generally in the form of ownership or effective ownership over the contributor’s copyright and the right to license it in any manner they choose in the future, including proprietary distributions.

Social media and "parasocial media"

Fri, 30 Jun 2023 00:00:00 +0000

A few months ago, as Elon Musk took over Twitter and instituted polices that alienated many people, some of these people fled towards federated, free software platforms like Mastodon. Many people found a new home here, but there is a certain class of refugee who has not found it to their liking.

I got to chatting with one such “refugee” on Mastodon today. NotJustBikes is a creator I enjoy watching on ~~YouTube~~ Invidious, who makes excellent content on urbanism and the design of cities. He’s based in my home town of Amsterdam and his videos do a great job of explaining many of the things I love about this place for general audiences. He’s working on building an audience, expanding his reach, and bringing his message to as many people as possible in the interest of bringing better infrastructure to everyone.

Burnout and the quiet failures of the hacker community

Thu, 29 Jun 2023 00:00:00 +0000

This has been a very challenging year for me. You probably read that I suffered from burnout earlier in the year. In some respects, things have improved, and in many other respects, I am still haunted.

You might not care to read this, and so be it, take your leave if you must. But writing is healing for me. Maybe this is a moment for solidarity, sympathy, for reflecting on your own communities. Maybe it’s a vain and needlessly public demonstration of my slow descent into madness. I don’t know, but here we go.

Reforming the free software message

Mon, 19 Jun 2023 00:00:00 +0000

Several weeks ago, I wrote The Free Software Foundation is dying, wherein I enumerated a number of problems with the Free Software Foundation. Some of my criticisms focused on the message: fsf.org and gnu.org together suffer from no small degree of incomprehensibility and inaccessibility which makes it difficult for new participants to learn about the movement and apply it in practice to their own projects.

This is something which is relatively easily fixed! I have a background in writing documentation and a thorough understanding of free software philosophy and practice. Enter writefreesoftware.org: a comprehensive introduction to free software philosophy and implementation.

Throwing in the towel on mobile Linux

Fri, 16 Jun 2023 00:00:00 +0000

I have been tinkering with mobile Linux – a phrase I will use here to describe any Linux distribution other than Android running on a mobile device – as my daily driver since about 2019, when I first picked up the PinePhone. For about 3 years I have run mobile Linux as my daily driver on my phone, and as of a few weeks ago, I’ve thrown in the towel and switched to Android.

How to go to war with your employer

Mon, 12 Jun 2023 00:00:00 +0000

There is a power differential between you and your employer, but that doesn’t mean you can’t improve your working conditions. Today I’d like to offer a little bit of advice on how to frame your relationship with your employer in terms which empower you and afford you more agency. I’m going to talk about the typical working conditions of the average white-collar job in a neo-liberal political environment where you are mostly happy to begin with and financially stable enough to take risks, and I’m specifically going to talk about individual action or the actions of small groups rather than large-scale collective action (e.g. unions).

Burnout

Mon, 01 May 2023 00:00:00 +0000

It kind of crept up on me. One day, sitting at my workstation, I stopped typing, stared blankly at the screen for a few seconds, and a switch flipped in my head.

On the night of New Year’s Eve, my backpack was stolen from me on the train from Berlin to Amsterdam, and with it about $2000 worth of equipment, clothes, and so on. A portent for the year that was to come. I generally keep my private and public lives carefully separated, but perhaps I will offer you a peek behind the curtain today.

Who should lead us?

Mon, 24 Apr 2023 00:00:00 +0000

Consider these two people, each captured in the midst of delivering a technical talk.

Based on appearances alone, what do you think of them?

The person on the left is a woman. She’s also pretty young, one might infer something about her level of experience accordingly. I imagine that she has led a much different life than I have, and may have a much different perspective, worldview, identity, and politics than I. Does she complain about sexism and discrimination in her work? Is she a feminist? Does she lean left or right on the political spectrum?

rc: a new shell for Unix

Tue, 18 Apr 2023 00:00:00 +0000

rc is a Unix shell I’ve been working on over the past couple of weeks, though it’s been in the design stages for a while longer than that. It’s not done or ready for general use yet, but it is interesting, so let’s talk about it.

As the name (which is subject to change) implies, rc is inspired by the Plan 9 rc shell. It’s not an implementation of Plan 9 rc, however: it departs in many notable ways. I’ll assume most readers are more familiar with POSIX shell or Bash and skip many of the direct comparisons to Plan 9. Also, though most of the features work as described, the shell is a work-in-progress and some of the design I’m going over today has not been implemented yet.

The Free Software Foundation is dying

Tue, 11 Apr 2023 00:00:00 +0000

The Free Software Foundation is one of the longest-running missions in the free software movement, effectively defining it. It provides a legal foundation for the movement and organizes activism around software freedom. The GNU project, closely related, has its own long story in our movement as the coding arm of the Free Software Foundation, taking these principles and philosophy into practice by developing free software; notably the GNU operating system that famously rests atop GNU/Linux.

Writing Helios drivers in the Mercury driver environment

Sat, 08 Apr 2023 00:00:00 +0000

Helios is a microkernel written in the Hare programming language and is part of the larger Ares operating system. You can watch my FOSDEM 2023 talk introducing Helios on PeerTube.

Let’s take a look at the new Mercury driver development environment for Helios.

As you may remember from my FOSDEM talk, the Ares operating system is built out of several layers which provide progressively higher-level environments for an operating system. At the bottom is the Helios microkernel, and today we’re going to talk about the second layer: the Mercury environment, which is used for writing and running device drivers in userspace. Let’s take a look at a serial driver written against Mercury and introduce some of the primitives used by driver authors in the Mercury environment.

When to comment that code

Thu, 09 Mar 2023 00:00:00 +0000

My software tends to have a surprisingly low number of comments. One of my projects, scdoc, has 25 comments among its 1,133 lines of C code, or 2%, compared to the average of 19%.¹ Naturally, I insist that my code is well-written in spite of this divergence from the norm. Allow me to explain.

The philosophy and implementation of code comments varies widely in the industry, and some view comment density as a proxy for code quality.² I’ll state my views here, but will note that yours may differ and I find that acceptable; I am not here to suggest that your strategy is wrong and I will happily adopt it when I write a patch for your codebase.

Porting Helios to aarch64 for my FOSDEM talk, part one

Mon, 20 Feb 2023 00:00:00 +0000

Helios is a microkernel written in the Hare programming language, and the subject of a talk I did at FOSDEM earlier this month. You can watch the talk here if you like:

A while ago I promised someone that I would not do any talks on Helios until I could present them from Helios itself, and at FOSDEM I made good on that promise: my talk was presented from a Raspberry Pi 4 running Helios. The kernel was originally designed for x86_64 (though we were careful to avoid painting ourselves into any corners so that we could port it to more architectures later on), and I initially planned to write an Intel HD Graphics driver so that I could drive the projector from my laptop. But, after a few days spent trying to comprehend the IHD manuals, I decided it would be much easier to port the entire system to aarch64 and write a driver for the much-simpler RPi GPU instead. 42 days later the port was complete, and a week or so after that I successfully presented the talk at FOSDEM. In a series of blog posts, I will take a look at those 42 days of work and explain how the aarch64 port works. Today’s post focuses on the bootloader.

Should private platforms engage in censorship?

Mon, 30 Jan 2023 00:00:00 +0000

Private service providers are entitled to do business with whom they please, or not to. Occasionally, a platform will take advantage of this to deny service to a particular entity on any number of grounds, often igniting a flood of debate online regarding whether or not censorship in this form is just. Recently, CloudFlare pulled the plug on a certain forum devoted to the coordinated harassment of its victims. Earlier examples include the same service blocking a far-right imageboard, or Namecheap cancelling service for a neo-Nazi news site.

My plans at FOSDEM: SourceHut, Hare, and Helios

Tue, 24 Jan 2023 00:00:00 +0000

FOSDEM is right around the corner, and finally in person after long years of dealing with COVID. I’ll be there again this year, and I’m looking forward to it! I have four slots on the schedule (wow! Thanks for arranging these, FOSDEM team) and I’ll be talking about several projects. There is a quick lightning talk on Saturday to introduce Helios and tease a full-length talk on Sunday, a meetup for the Hare community, and a meetup for the SourceHut community. I hope to see you there!

Setting a new focus for my blog

Sun, 22 Jan 2023 00:00:00 +0000

Just shy of two months ago, I published I shall toil at a reduced volume, which addressed the fact that I’m not getting what I want from my blog anymore, and I would be taking an indefinite break. Well, I am ready to resume my writing, albeit with a different tone and focus than before.

Well, that was fast.

– Everyone

Since writing this, I have been considering what exactly the essential subject of my dissatisfaction with my writing has been. I may have found the answer: I lost sight of my goals. I got so used to writing that I would often think to myself, “I want to write a blog post!”, then dig a topic out of my backlog (which is 264 items long) and write something about it. This is not the way; much of the effort expended on writing in this manner is not spent on the subjects I care about most, or those which most urgently demand an expenditure of words.

I shall toil at a reduced volume

Thu, 01 Dec 2022 00:00:00 +0000

Over the last nine years I have written 300,000 words for this blog on the topics which are important to me. I am not certain that I have much left to say.

I can keep revisiting these topics for years, each time adding a couple more years of wisdom and improvements to my writing skills to present my arguments more effectively. However, I am starting to feel diminishing returns from my writing. It does not seem like my words are connecting with readers anymore. And, though the returns on my work seem to be diminishing, the costs are not. Each new article spurs less discussion than the last, but provides an unwavering supply of spiteful responses.

Codegen in Hare v2

Sat, 26 Nov 2022 00:00:00 +0000

I spoke about code generation in Hare back in May when I wrote a tool for generating ioctl numbers. I wrote another code generator over the past few weeks, and it seems like a good time to revisit the topic on my blog to showcase another approach, and the improvements we’ve made for this use-case.

In this case, I wanted to generate code to implement IPC (inter-process communication) interfaces for my operating system. I have designed a DSL for describing these interfaces — you can read the grammar here. This calls for a parser, which is another interesting topic for Hare, but I’ll set that aside for now and focus on the code gen. Assume that, given a file like the following, we can parse it and produce an AST:

In praise of Plan 9

Sat, 12 Nov 2022 00:00:00 +0000

Plan 9 is an operating system designed by Bell Labs. It’s the OS they wrote after Unix, with the benefit of hindsight. It is the most interesting operating system that you’ve never heard of, and, in my opinion, the best operating system design to date. Even if you haven’t heard of Plan 9, the designers of whatever OS you do use have heard of it, and have incorporated some of its ideas into your OS.

Notes from kernel hacking in Hare, part 3: serial driver

Thu, 27 Oct 2022 00:00:00 +0000

Today I would like to show you the implementation of the first userspace driver for Helios: a simple serial driver. All of the code we’re going to look at today runs in userspace, not in the kernel, so strictly speaking this should be “notes from OS hacking in Hare”, but I won’t snitch if you don’t.

Note: In the previous entry to this series, I promised to cover the userspace threading API in this post. I felt like covering this instead. Sorry!

TOTP for 2FA is incredibly easy to implement. So what's your excuse?

Tue, 18 Oct 2022 00:00:00 +0000

Time-based one-time passwords are one of the more secure approaches to 2FA — certainly much better than SMS. And it’s much easier to implement than SMS as well. The algorithm is as follows:

Divide the current Unix timestamp by 30
Encode it as a 64-bit big endian integer
Write the encoded bytes to a SHA-1 HMAC initialized with the TOTP shared key
Let offs = hmac[-1] & 0xF
Let hash = decode hmac[offs .. offs + 4] as a 32-bit big-endian integer
Let code = (hash & 0x7FFFFFFF) % 1000000
Compare this code with the user’s code

You’ll need a little dependency to generate QR codes with the otpauth:// URL scheme, a little UI to present the QR code and store the shared secret in your database, and a quick update to your login flow, and then you’re good to go.

Status update, October 2022

Sat, 15 Oct 2022 00:00:00 +0000

After a few busy and stressful months, I decided to set aside October to rest. Of course, for me, rest does not mean a cessation of programming, but rather a shift in priorities towards more fun and experimental projects. Consequently, it has been a great month for Helios!

Hare upstream has enjoyed some minor improvements, such as from Pierre Curto’s patch to support parsing IPv6 addresses with a port (e.g. “[::1]:80”) and Kirill Primak’s improvements to the UTF-8 decoder. On the whole, improvements have been conservative. However, queued up for integration once qbe upstream support is merged is support for @threadlocal variables, which are useful for Helios and for ABI compatibility with C. I also drafted up a proof-of-concept for @inline functions, but it still needs work.

In praise of ffmpeg

Wed, 12 Oct 2022 00:00:00 +0000

My last “In praise of” article covered qemu, a project founded by Fabrice Bellard, and today I want to take a look at another work by Bellard: ffmpeg. Bellard has a knack for building high-quality software which solves a problem so well that every other solution becomes obsolete shortly thereafter, and ffmpeg is no exception.

ffmpeg has been described as the Swiss army knife of multimedia. It incorporates hundreds of video, audio, and image decoders and encoders, muxers and demuxers, filters and devices. It provides a CLI and a set of libraries for working with its tools, and is the core component of many video and audio players as a result (including my preferred multimedia player, mpv). If you want to do almost anything with multimedia files — re-encode them, re-mux them, live stream it, whatever — ffmpeg can handle it with ease.

Does Rust belong in the Linux kernel?

Mon, 03 Oct 2022 00:00:00 +0000

I am known to be a bit of a polemic when it comes to Rust. I will be forthright with the fact that I don’t particularly care for Rust, and that my public criticisms of it might set up many readers with a reluctance to endure yet another Rust Hot Take from my blog. My answer to the question posed in the title is, of course, “no”. However, let me assuage some of your fears by answering a different question first: does Hare belong in the Linux kernel?

Notes from kernel hacking in Hare, part 2: multi-threading

Sun, 02 Oct 2022 00:00:00 +0000

I have long promised that Hare would not have multi-threading, and it seems that I have broken that promise. However, I have remained true to the not-invented-here approach which is typical of my style by introducing it only after designing an entire kernel to implement it on top of.¹

For some background, Helios is a micro-kernel written in Hare. In addition to the project, the Vulcan system is a small userspace designed to test the kernel.

The phrase "open source" (still) matters

Fri, 16 Sep 2022 00:00:00 +0000

In 1988, “Resin Identification Codes” were introduced by the plastic industry. These look exactly like the recycling symbol ♺, which is not trademarked or regulated, except that a number is enclosed within the triangle. These symbols simply identify what kind of plastic was used. The vast majority of plastic is non-recyclable, but has one of these symbols on it to suggest otherwise. This is a deceptive business practice which exploits the consumer’s understanding of the recycling symbol to trick them into buying more plastic products.

Status update, September 2022

Thu, 15 Sep 2022 00:00:00 +0000

I have COVID-19 and I am halfway through my stockpile of tissues, so I’m gonna keep this status update brief.

In Hare news, I finally put the last pieces into place to make cross compiling as easy as possible. Nothing else particularly world-shattering going on here. I have a bunch of new stuff in my patch queue to review once I’m feeling better, however, including bigint stuff — a big step towards TLS support. Unrelatedly, TLS support seems to be progressing upstream in qbe. (See what I did there?)

Notes from kernel hacking in Hare, part 1

Wed, 07 Sep 2022 00:00:00 +0000

One of the goals for the Hare programming language is to be able to write kernels, such as my Helios project. Kernels are complex beasts which exist in a somewhat unique problem space and have constraints that many userspace programs are not accustomed to. To illustrate this, I’m going to highlight a scenario where Hare’s low-level types and manual memory management approach shines to enable a difficult use-case.

Helios is a micro-kernel. During system initialization, its job is to load the initial task into memory, prepare the initial set of kernel objects for its use, provide it with information about the system, then jump to userspace and fuck off until someone needs it again. I’m going to focus on the “providing information” step here.

In praise of qemu

Fri, 02 Sep 2022 00:00:00 +0000

qemu is another in a long line of great software started by Fabrice Bellard. It provides virtual machines for a wide variety of software architectures. Combined with KVM, it forms the foundation of nearly all cloud services, and it runs SourceHut in our self-hosted datacenters. Much like Bellard’s ffmpeg revolutionized the multimedia software industry, qemu revolutionized virtualisation.

qemu comes with a large variety of studiously implemented virtual devices, from standard real-world hardware like e1000 network interfaces to accelerated virtual hardware like virtio drives. One can, with the right combination of command line arguments, produce a virtual machine of essentially any configuration, either for testing novel configurations or for running production-ready virtual machines. Network adapters, mouse & keyboard, IDE or SCSI or SATA drives, sound cards, graphics cards, serial ports — the works. Lower level, often arch-specific features, such as AHCI devices, SMP, NUMA, and so on, are also available and invaluable for testing any conceivable system configurations. And these configurations work, and work reliably.

powerctl: A small case study in Hare for systems programming

Sun, 28 Aug 2022 00:00:00 +0000

powerctl is a little weekend project I put together to provide a simple tool for managing power states on Linux. I had previously put my laptop into suspend with a basic “echo mem | doas tee /sys/power/state”, but this leaves a lot to be desired. I have to use doas to become root, and it’s annoying to enter my password — not to mention difficult to use in a script or to attach to a key binding. powerctl is the solution: a small 500-line Hare program which provides comprehensive support for managing power states on Linux for non-privileged users.

A review of postmarketOS on the Xiaomi Poco F1

Thu, 25 Aug 2022 00:00:00 +0000

I have recently had cause to start looking into mainline Linux phones which fall outside of the common range of grassroots phones like the PinePhone (which was my daily driver for the past year). The postmarketOS wiki is a great place to research candidate phones for this purpose, and the phone I landed on is the Xiaomi Poco F1, which I picked up on Amazon.nl (for ease of return in case it didn’t work out) for 270 Euro. Phones of this nature have a wide range of support from Linux distros like postmarketOS, from “not working at all” to “mostly working”. The essential features I require in a daily driver phone are (1) a working modem and telephony support, (2) mobile data, and (3) reasonably good performance and battery life; plus of course some sane baseline expectations like a working display and touchscreen driver.

PINE64 has let its community down

Thu, 18 Aug 2022 00:00:00 +0000

Context for this post:

I know that apologising and taking responsibility for your mistakes is difficult. It seems especially difficult for commercial endeavours, which have fostered a culture of cold disassociation from responsibility for their actions, where admitting to wrongdoing is absolutely off the table. I disagree with this culture, but I understand where it comes from, and I can empathise with those who find themselves in the position of having to reconsider their actions in the light of the harm they have done. It’s not easy.

Status update, August 2022

Tue, 16 Aug 2022 00:00:00 +0000

It is a blessedly cool morning here in Amsterdam. I was busy moving house earlier this month, so this update is a bit quieter than most.

For a fun off-beat project this month, I started working on a GameBoy emulator written in Hare. No promises on when it will be functional or how much I plan on working on it – just doing it for fun. In more serious Hare news, I have implemented Thread-Local Storage (TLS) for qbe, our compiler backend. Hare’s standard library does not support multi-threading, but I needed this for Helios, whose driver library does support threads. It will also presumably be of use for cproc once it lands upstream.

How I wish I could organize my thoughts

Wed, 10 Aug 2022 00:00:00 +0000

I keep a pen & notebook on my desk, which I make liberal use of to jot down my thoughts. It works pretty well: ad-hoc todo lists, notes on problems I’m working on, tables, flowcharts, etc. It has some limitations, though. Sharing anything out of my notebook online is an awful pain in the ass. I can’t draw a straight line to save my life, so tables and flowcharts are a challenge. No edits, either, so lots of crossed-out words and redrawn or rewritten pages. And of course, my handwriting sucks and I can type much more efficiently than I can write. I wish this was a digital medium, but there are not any applications available which can support the note-taking paradigm that I wish I could have. What would that look like?

Conciseness

Tue, 26 Jul 2022 00:00:00 +0000

Conciseness is often considered a virtue among hackers and software engineers. FOSS maintainers in particular generally prefer to keep bug reports, questions on mailing lists, discussions in IRC channels, and so on, close to the point and with minimal faff. It’s not considered impolite to skip the formalities — quite the opposite. So: keep your faffery to a minimum. A quick “thanks!” at the end of a discussion will generally suffice. And, when someone is being direct with you, don’t interpret it as a slight: simply indulge in the blissful freedom of a discussion absent of faffery.

Code review at the speed of email

Mon, 25 Jul 2022 00:00:00 +0000

I’m a big proponent of the email workflow for patch submission and code review. I have previously published some content (How to use git.sr.ht’s send-email feature, Forks & pull requests vs email, git-send-email.io) which demonstrates the contributor side of this workflow, but it’s nice to illustrate the advantages of the maintainer workflow as well. For this purpose, I’ve recorded a short video demonstrating how I manage code review as an email-oriented maintainer.

The past and future of open hardware

Mon, 25 Jul 2022 00:00:00 +0000

They say a sucker is born every day, and at least on the day of my birth, that certainly may have been true. I have a bad habit of spending money on open hardware projects that ultimately become vaporware or seriously under-deliver on their expectations. In my ledger are EOMA68, DragonBox Pyra, the Jolla Tablet — which always had significant non-free components — and the Mudita Pure, though I did successfully receive a refund for the latter two.¹

Status update, July 2022

Mon, 18 Jul 2022 00:00:00 +0000

Hello there! It’s been a hot July week in Amsterdam, and I expect hotter days are still to come. I wish air conditioning was more popular in Europe, but alas. This month of FOSS development enjoyed a lot of small improvements in a lot of different projects.

For Hare, I have introduced a number of improvements. I wrote a new standard library module for string templates, strings::template, and a new third-party library for working with pixel buffers, pixbuf. The templating is pretty simple — as is typical for the standard library — but allows a fairly wide range of formatting options. We’ll be extending this a little bit more in the future, but it will not be a complete solution like you see in things like Jinja2. Nevertheless, it makes some use-cases, like code generation, a lot cleaner, without introducing a weighty or complex dependency.

The Fediverse can be pretty toxic

Sat, 09 Jul 2022 00:00:00 +0000

Mastodon, inspired by GNU social, together with Pleroma, form the most popular components of what we know as the “Fediverse” today. All of them are, in essence, federated, free software Twitter clones, interoperable with each other via the ActivityPub protocol.

In many respects, the Fediverse is a liberating force for good. Its federated design distributes governance and costs across many independent entities, something I view as a very strong design choice. Its moderation tools also do a pretty good job of keeping neo-nazis out of your feeds and providing a comfortable space to express yourself in, especially if your form of expression is maligned by society. Large groups of Fediverse members have found in it a home for self-expression which is denied to them elsewhere on the basis of their sexuality, gender expression, politics, or other characteristics. It’s also essentially entirely free from commercial propaganda.

Porting Doom to Helios

Fri, 01 Jul 2022 00:00:00 +0000

Doom was an incredibly popular video game by Id software which, six years following its release, was made open source under the GPLv2 license. Thanks to this release, combined with the solid software design and lasting legacy of backwards compatibility in C, Doom has been ported to countless platforms by countless programmers. And I recently added myself to this number :)

I’m working on a new kernel called Helios, and I thought that porting Doom would present a good opportunity for proving the kernel design — you never know if you have a good design until you try to use it for real. Doom is a good target because it does not require much to get working, but it is a useful (and fun) program to port. It calls for the following features:

GitHub Copilot and open source laundering

Thu, 23 Jun 2022 00:00:00 +0000

Disclaimer: I am the founder of a company which competes with GitHub. I am also a long-time advocate for and developer of free and open source software, with a broad understanding of free and open source software licensing and philosophy. I will not name my company in this post to reduce the scope of my conflict of interest.

We have seen an explosion in machine learning in the past decade, alongside an explosion in the popularity of free software. At the same time as FOSS has come to dominate software and found its place in almost all new software products, machine learning has increased dramatically in sophistication, facilitating more natural interactions between humans and computers. However, despite their parallel rise in computing, these two domains remain philosophically distant.

Introducing the Himitsu keyring & password manager for Unix

Mon, 20 Jun 2022 00:00:00 +0000

Himitsu is a new approach to storing secret information on Unix systems, such as passwords or private keys, and I released version 0.1 this morning. It’s available on Alpine Linux community and the Arch User Repository, with more distributions hopefully on the way soon.

So, what is Himitsu and what makes it special? The following video introduces the essential concepts and gives you an idea of what’s possible:

If you prefer reading to watching, this blog post includes everything that’s in the video.

Status update, June 2022

Wed, 15 Jun 2022 00:00:00 +0000

Hello again! I would like to open this post by acknowledging the response to my earlier post, “bleh”. Since it was published, I have received several hundred emails expressing support and kindness. I initially tried to provide these with thoughtful replies, then shorter replies, then I had to stop replying at all, but I did read every one. Thank you, everyone, for sending these. I appreciate it very much, and it means a lot to me.

The Helios microkernel

Mon, 13 Jun 2022 00:00:00 +0000

I’ve been working on a cool project lately that I’d like to introduce you to: the Helios microkernel. Helios is written in Hare and currently targets x86_64, and riscv64 and aarch64 are on the way. It’s very much a work-in-progress: don’t expect to pick this up and start building anything with it today.

Drawing some inspiration from seL4, Helios uses a capability-based design for isolation and security. The kernel offers primitives for allocating physical pages, mapping them into address spaces, and managing tasks, plus features like platform-specific I/O (e.g. reading and writing x86 ports). The entire system is written in Hare, plus some necessary assembly for the platform bits (e.g. configuring the GDT or IDT).

bleh

Mon, 30 May 2022 00:00:00 +0000

A few weeks ago, the maintainer of a project on SourceHut stepped down from their work, citing harassment over using SourceHut as their platform of choice. It was a difficult day when I heard about that.

Over the past few weeks, I have been enduring a bit of a depressive episode. It’s a complex issue rooted in several different problems, but I think a major source of it is the seemingly constant deluge of hate I find myself at the receiving end of online. I had to grow a thick skin a long time ago, but lately it has not been thick enough. I am finding it increasingly difficult to keep up with my work.

Google has been DDoSing SourceHut for over a year

Wed, 25 May 2022 00:00:00 +0000

Just now, I took a look at the HTTP logs on git.sr.ht. Of the past 100,000 HTTP requests received by git.sr.ht (representing about 2½ hours of logs), 4,774 have been requested by GoModuleProxy — 5% of all traffic. And their requests are not cheap: every one is a complete git clone. They come in bursts, so every few minutes we get a big spike from Go, along with a constant murmur of Go traffic.

Status update, May 2022

Mon, 16 May 2022 00:00:00 +0000

This was an exciting month: the Hare programming language is a secret no more! You can now try out the programming language I first teased over a year ago and tell me what you think. I hope you like it! I’m quite pleased with it so far.

One thing Hare has done is allow me to unshelve several projects which were blocked pending the availability of a suitable language to write them in. I have actually been working on several of these for a while now — and several more are to come later — but I couldn’t share them thanks to Hare’s policy of secrecy early in its development. Allow me to introduce you to a few projects!

A Hare code generator for finding ioctl numbers

Sat, 14 May 2022 00:00:00 +0000

Modern Unix derivatives have this really bad idea called ioctl. It’s a function which performs arbitrary operations on a file descriptor. It is essentially the kitchen sink of modern Unix derivatives, particularly Linux, in which they act almost like a second set of extra syscalls. For example, to get the size of the terminal window, you use an ioctl specific to TTY file descriptors:

let wsz = rt::winsize { ... };
match (rt::ioctl(fd, rt::TIOCGWINSZ, &wsz: *void)) {
case let e: rt::errno =>
	switch (e: int) {
	case rt::EBADFD =>
		return errors::invalid;
	case rt::ENOTTY =>
		return errors::unsupported;
	case =>
		abort("Unexpected error from ioctl");
	};
case int =>
	return ttysize {
		rows = wsz.ws_row,
		columns = wsz.ws_col,
	};
};

This code performs the ioctl syscall against the provided file descriptor “fd”, using the “TIOCGWINSZ” operation, and setting the parameter to a pointer to a winsize structure. There are thousands of ioctls provided by Linux, and each of them is assigned a constant like TIOCGWINSZ (0x5413). Some constants, including this one, are assigned somewhat arbitrarily. However, some are assigned with some degree of structure.

When will we learn?

Thu, 12 May 2022 00:00:00 +0000

Congratulations to Rust for its first (but not its last) supply-chain attack this week! They join a growing club of broken-by-design package managers which publish packages uploaded by vendors directly, with no review step, and ship those packages directly to users with no further scrutiny.

Timeline of major incidents on npm/Crates/PyPI/etc

2022-05-10: Cargo: rustdecimal ships with malicious code
2022-05-09: npm: foreach is taken over via an expired email domain
2022-03-17: npm: node-ipc ships malware targeting Russia and Belarus
2022-01-09: npm: colors and faker are deliberately sabotaged
2021-11-19: PyPI: 11 malicious packages discovered
2021-11-04: npm: rc ships malicious code
2021-11-04: npm: coa steals your passwords
2021-10-22: npm: ua-parser-js ships malicious code
2021-10-11: PyPI: mitmproxy2 typo-squats mitmproxy with an added RCE
2021-07-30: PyPI: 8 malicious packages discovered
2020-12-16: RubyGems: pretty_color (and one other) steals bitcoin from victims
2020-09-11: npm: dozens of packages steal your user’s credit card number
2020-09-03: npm: bb-builder steals your password
2020-04-16: RubyGems: 760+ malicious packages found stealing bitcoin
2018-11-28: npm: event-stream ships with a bitcoin theft kit
2018-10-21: PyPI: colourama sneaks bitcoin addresses into your clipboard
2018-10-13: PyPI: more typo-squatting malware attempts various attacks
2018-07-12: npm: eslint-scope ships with malicious code
2018-07-08: AUR: acroread is compromised
2018-05-11: Snap: a 2048 clone ships a cryptocurrency miner
2017-09-09: PyPI: typo-squatted packages published by researchers
2016-07-22: npm: left-pad incident

There are hundreds of additional examples. I had to leave many of them out. Here’s a good source if you want to find more.

Implementing an SSH agent in Hare

Mon, 09 May 2022 00:00:00 +0000

Cross-posted from the Hare blog

In the process of writing an SSH agent for Himitsu, I needed to implement many SSH primitives from the ground up in Hare, now available via hare-ssh. Today, I’m going to show you how it works!

Important: This blog post deals with cryptography-related code. The code you’re going to see today is incomplete, unaudited, and largely hasn’t even seen any code review. Let me begin with a quote from the “crypto” module’s documentation in the Hare standard library:

Announcing the Hare programming language

Mon, 25 Apr 2022 00:00:00 +0000

The “secret programming language” I have been teasing for several months now is finally here! It is called Hare, and you can read about it on the Hare blog:

https://harelang.org/blog/2022-04-25-announcing-hare/

Check it out!

Status update, April 2022

Fri, 15 Apr 2022 00:00:00 +0000

This month marked my first time filing taxes in two countries, and I can assure you it is the worst. I am now a single-issue voter in the US: stop taxing expats! You can get some insight into the financials of SourceHut in the recently-published financial report. But let’s get right into the fun stuff: free software development news.

There was some slowdown from me this month thanks to all of the business and financial crap I had to put up with, but I was able to get some cool stuff done and many other contributors have been keeping things moving. I’ll start by introducing a new/old project: Himitsu.

Announcing git snail-mail

Fri, 01 Apr 2022 00:00:00 +0000

You’ve heard of git-over-email thanks to git send-email — now you can enjoy git snail-mail: a new tool making it easier than ever to print out git commits on paper and mail them to your maintainers.

Running git snail-mail HEAD~2.. prepares the last two commits for post and sends them directly to the system’s default printer. Configuration options are available for changing printer settings, paper size, and options for faxing or printing envelopes automatically addressed to the maintainers based on address info stored in your git config. Be sure to help the maintainers review your work by including a return envelope and a stamp!

It is important for free software to use free software infrastructure

Tue, 29 Mar 2022 00:00:00 +0000

Disclaimer: I founded a project and a company that focuses on free software infrastructure. I will elect not to name them in this post, and will only recommend solutions I do not have a vested interest in.

Free and open source software (FOSS) projects need infrastructure. Somewhere to host the code, to facilitate things like code review, end-user support, bug tracking, marketing, and so on. A common example of this is the “forge” platform: infrastructure which pitches itself as a one-stop shop for many of the needs of FOSS projects in one place, such as code hosting and review, bug tracking, discussions, and so on. Many projects will also reach for additional platforms to provide other kinds of infrastructure: chat rooms, forums, social media, and more.

The Netherlands so far

Thu, 24 Mar 2022 00:00:00 +0000

I moved to Amsterdam in July 2021, and now that I’ve had some time to settle in I thought I’d share my thoughts on how it’s been so far. In short: I love it here!

I did end up finding housing through the hacker community thanks to my earlier post, which was a great blessing. I am renting an apartment from a member of the Techinc hacker space, which I have joined as a member myself. One of my biggest fears was establishing a new social network here in the Netherlands, but making friends here has been easy. Through this hacker space and through other connections besides, I have quickly met many wonderful, friendly, and welcoming people, and I have never felt like a stranger in a strange land. For this I am very grateful.

Status update, March 2022

Tue, 15 Mar 2022 00:00:00 +0000

Greetings! The weather is starting to warm up again, eh? I’m a bit disappointed that we didn’t get any snow this winter. Yadda yadda insert intro text here. Let’s get down to brass tacks. What’s new this month?

I mainly focused on the programming language this month. I started writing a kernel, which you can see a screenshot of below. This screenshot shows a simulated page fault, demonstrating that we have a working interrupt handler, and also shows something mildly interesting: backtraces. I need to incorporate this approach into the standard library as well, so that we can dump useful stack traces on assertion failures and such. I understand that someone is working on DWARF support as well, so perhaps we’ll soon be able to translate function name + offset into a file name and line number.

It takes a village

Mon, 14 Mar 2022 00:00:00 +0000

As a prolific maintainer of several dozen FOSS projects, I’m often asked how I can get so much done, being just one person. The answer is: I’m not just one person. I have enjoyed the help of thousands of talented people who have contributed to these works. Without them, none of the projects I work on would be successful.

I’d like to take a moment to recognize and thank all of the people who have participated in these endeavours. If you’ve enjoyed any of the projects I’ve worked on, you owe thanks to some of these wonderful people. The following is an incomplete list of authors who have contributed to one or more of the projects I have started:

Why am I building a programming language in private?

Sun, 13 Mar 2022 00:00:00 +0000

As many readers are aware, I have been working on designing and implementing a systems programming language. This weekend, I’ve been writing a PNG file decoder in it, and over the past week, I have been working on a simple kernel with it as well. I’m very pleased with our progress so far — I recently remarked that this language feels like the language I always wanted, and that’s mission accomplished by any definition I care to consider.

Open Source is defined by the OSI's Open Source Definition

Tue, 01 Mar 2022 00:00:00 +0000

The Open Source Initiative (OSI) publishes a document called the Open Source Definition (OSD), which defines the term “open source”. However, there is a small minority of viewpoints within the software community which wishes that this were not so. The most concerning among them are those who wish open source was more commercially favorable to themselves, and themselves alone, such as companies like Elastic.

I disagree with this perspective, and I’d like take a few minutes today to explore several of the most common arguments in favor of this view, and explain why I don’t agree with them. One of the most frustrating complications in this discussion is the context of motivated reasoning (relevant xkcd): most people arguing in favor of an unorthodox definition of “open source” have a vested interest in their alternative view.¹ This makes it difficult to presume good faith. For example, say someone wants to portray their software as open source even if it prohibits commercial use by third parties, which would normally disqualify it as such. Their interpretation serves to re-enforce their commercialization plans, providing a direct financial incentive not only for them to promote this definition of “open source”, but also for them to convince you that their interpretation is valid.

Plaid is an evil nightmare product from Security Hell

Sat, 19 Feb 2022 00:00:00 +0000

Plaid is a business that has built a widget that can be embedded in any of their customer’s websites which allows their customers to configure integrations with a list of third-party service providers. To facilitate this, Plaid pops up a widget on their customer’s domain which asks the end-user to type in their username and password for the third-party service provider. If necessary, they will ask for a 2FA code. This is done without the third party’s permission, presumably through a browser emulator and a provider-specific munging shim, and collects the user’s credentials on a domain which is operated by neither the third party nor by Plaid.

Status update, February 2022

Tue, 15 Feb 2022 00:00:00 +0000

Hello once again! Another month of free software development goes by with lots of progress in all respects.

I will open with some news about godocs.io: version 1.0 of our fork of gddo has been released! Big thanks to Adnan Maolood for his work on this. I’m very pleased that, following our fork, we were not only able to provide continuity for godoc.org, but also to simplify, refactor, and improve the underlying software considerably. Check out Adnan’s blog post for more details.

Framing accessibility in broader terms

Sun, 13 Feb 2022 00:00:00 +0000

Upon hearing the term “accessibility”, many developers call to mind the HTML ARIA attributes and little else. Those who have done some real accessibility work may think of the WCAG guidelines. Some FOSS developers¹ may think of AT-SPI. The typical user of these accessibility features is, in the minds of many naive developers, a blind person. Perhaps for those who have worked with WCAG, a slightly more sophisticated understanding of the audience for accessibility tools may include users with a greater variety of vision-related problems, motor impairments, or similar needs.

Free software licenses explained: MIT

Mon, 07 Feb 2022 00:00:00 +0000

This is the first in a series of posts I intend to write explaining how various free and open source software licenses work, and what that means for you as a user or developer of that software. Today we’ll look at the MIT license, also sometimes referred to as the X11 or Expat license.

The MIT license is:

Both free software and open source
Permissive (and thus non-copyleft and non-viral)

This means that the license upholds the four essential freedoms of free software (the right to run, copy, distribute, study, change and improve the software) and all of the terms of the open source definition (largely the same). Further more, it is classified on the permissive/copyleft spectrum as a permissive license, meaning that it imposes relatively few obligations on the recipient of the license.

Implementing a MIME database in XXXX

Fri, 28 Jan 2022 00:00:00 +0000

This is a (redacted) post from the internal blog of a new systems programming language we’re developing. The project is being kept under wraps until we’re done with it, so for this post I’ll be calling it XXXX. If you are interested in participating, send me an email with some details about your background and I’ll get back to you.

Recently, I have been working on implementing a parser for media types (commonly called MIME types) and a database which maps media types to file extensions and vice-versa. I thought this would be an interesting module to blog about, given that it’s only about 250 lines of code, does something useful and interesting, and demonstrates a few interesting xxxx concepts.

Pine64 should re-evaluate their community priorities

Tue, 18 Jan 2022 00:00:00 +0000

Pine64 has a really interesting idea: make cheap hardware with low margins, get it into the hands of the FOSS community, and let them come up with the software. No one has ever done this before, at least not on this scale, and it’s a really neat idea! Pine64 is doing a lot to support the FOSS community bringing up its hardware, but I’m afraid that I have to ask them to do a bit more.

Status update, January 2022

Mon, 17 Jan 2022 00:00:00 +0000

Happy New Year! I had a lovely time in Amsterdam. No one had prepared me for the (apparently infamous) fireworks culture of the Netherlands. I thought it was really cool.

Our programming language continues to improve apace. Our cryptography suite now includes Argon2, Salsa20/XSalsa20, ChaCha20/XChaCha20, and Poly1305, and based on these functions we have added libsodium-style high-level cryptographic utilities for AEAD and key derivation, with stream encryption, message signing and verification, and key exchange coming soon. We have also laid out the priorities for future crypto work towards supporting TLS, and on the way we expect to have ed25519/x25519 and Diffie-Hellman added soon. Perhaps enough to implement an SSH client?

The RISC-V experience

Sat, 15 Jan 2022 00:00:00 +0000

I’m writing to you from a Sway session on Alpine Linux, which is to say from a setup quite similar to the one I usually write blog posts on, save for one important factor: a RISC-V CPU.

I’ll state upfront that what I’m using is not a very practical system. What I’m going to describe is all of the impractical hacks and workarounds I have used to build a “useful” RISC-V system on which I can mostly conduct my usual work. It has been an interesting exercise, and it bodes well for the future of RISC-V, but for all practical purposes the promise of RISC-V still lives in tomorrow, not today.

Breaking down a small language design proposal

Thu, 30 Dec 2021 00:00:00 +0000

We are developing a new systems programming language. The name is a secret, so we’ll call it xxxx instead. In xxxx, we have a general requirement that all variables must be initialized. This is fine for the simple case, such as “let x: int = 10”. But, it does not always work well. Let’s say that you want to set aside a large buffer for I/O:

Please don't use Discord for FOSS projects

Tue, 28 Dec 2021 00:00:00 +0000

Six years ago, I wrote a post speaking out against the use of Slack for the instant messaging needs of FOSS projects. In retrospect, this article is not very good, and in the years since, another proprietary chat fad has stepped up to bat: Discord. It’s time to revisit this discussion.

In short, using Discord for your free software/open source (FOSS) software project is a very bad idea. Free software matters — that’s why you’re writing it, after all. Using Discord partitions your community on either side of a walled garden, with one side that’s willing to use the proprietary Discord client, and one side that isn’t. It sets up users who are passionate about free software — i.e. your most passionate contributors or potential contributors — as second-class citizens.

Please use me as a resource

Sat, 25 Dec 2021 00:00:00 +0000

I write a lot of blog posts about my ideas,¹ some of which are even good ideas. Some of these ideas stick, and many readers have attempted to put them into practice, taking on challenges like starting a business in FOSS or stepping up to be leaders in their communities. It makes me proud to see the difference you’re making, and I’m honored to have inspired many of you.

I’m sitting here on my soapbox shouting into the void, but I also want to work with you one-on-one. Here are some things people have reached out to me for:

Sustainable creativity in a world without copyright

Thu, 23 Dec 2021 00:00:00 +0000

I don’t believe in copyright. I argue that we need to get rid of copyright, or at least dramatically reform it. The public domain has been stolen from us, and I want it back. Everyone reading this post has grown up in a creative world defined by capitalism, in which adapting and remixing works — a fundamental part of the creative process — is illegal. The commons is dead, and we suffer for it. But, this is all we’ve ever known. It can be difficult to imagine a world without copyright.

On commercial forks of FOSS projects

Sat, 18 Dec 2021 00:00:00 +0000

The gaming and live streaming industry is a lucrative and rapidly growing commercial sector with a unique understanding of copyright and intellectual property, and many parties with conflicting interests and access to different economic resources.

The understanding of intellectual property among gamers and the companies which serve them differs substantailly from that of free software, and literacy in the values and philosophy of free software among this community is very low. It is then of little surprise that we see abuse of free software from this community, namely in the recent (and illegal) commercial forks of a popular FOSS streaming platform called OBS Studio by companies like TikTok.

Status update, December 2021

Wed, 15 Dec 2021 00:00:00 +0000

Greetings! It has been a cold and wet month here in Amsterdam, much like the rest of them, as another period of FOSS progress rolls on by. I have been taking it a little bit easier this month, and may continue to take some time off in the coming weeks, so I can have a bit of a rest for the holidays. However, I do have some progress to report, so let’s get to it.

Impressions of Linux Mint & elementary OS

Tue, 14 Dec 2021 00:00:00 +0000

In a recent post, I spoke about some things that Linux distros need to do better to accommodate end-users. I was reminded that there are some Linux distros which are, at least to some extent, following my recommended playbook, and have been re-evaluating two of them over the past couple of weeks: Linux Mint and elementary OS. I installed these on one of my laptops and used it as my daily driver for a day or two each.

How new Linux users can increase their odds of success

Sun, 05 Dec 2021 00:00:00 +0000

The Linus Tech Tips YouTube channel has been putting out a series of videos called the Switching to Linux Challenge that has been causing a bit of a stir in the Linux community. I’ve been keeping an eye on these developments, and thought it was a good time to weigh in with my thoughts. This article focuses on how new Linux users can increase their odds for success — I have also written a companion article, “What desktop Linux needs to succeed in the mainstream”, which looks at the other side of the problem.

What desktop Linux needs to succeed in the mainstream

Sun, 05 Dec 2021 00:00:00 +0000

The Linus Tech Tips YouTube channel has been putting out a series of videos called the Switching to Linux Challenge that has been causing a bit of a stir in the Linux community. I’ve been keeping an eye on these developments, and thought it was a good time to weigh in with my thoughts. This article focuses on what Linux needs to do better — I have also written a companion article, “How new Linux users can increase their odds of success”, which looks at the other side of the problem.

postmarketOS revolutionizes smartphone hacking

Fri, 26 Nov 2021 00:00:00 +0000

I briefly mentioned postmarketOS in my Pinephone review two years ago, but after getting my Dutch SIM card set up in my Pinephone and having another go at using postmarketOS, I reckon they deserve special attention.

Let’s first consider the kind of ecosystem into which postmarketOS emerged: smartphone hacking in the XDA Forums era. This era was dominated by amateur hackers working independently for personal prestige, with little to no regard for the values of free software or collaboration. It was common to see hacked-together binary images shipped behind adfly links in XDA forum threads in blatant disregard of the GPL, with pages and pages of users asking redundant questions and receiving poor answers to the endless problems caused by this arrangement.

My philosophy for productive instant messaging

Wed, 24 Nov 2021 00:00:00 +0000

We use Internet Relay Chat (IRC) extensively at sourcehut for real-time group chats and one-on-one messaging. The IRC protocol is quite familiar to hackers, who have been using it since the late 80’s. As chat rooms have become more and more popular among teams of both hackers and non-hackers in recent years, I would like to offer a few bites of greybeard wisdom to those trying to figure out how to effectively use instant messaging for their own work.

I will pay you cash to delete your npm module

Tue, 16 Nov 2021 00:00:00 +0000

npm’s culture presents a major problem for global software security. It’s grossly irresponsible to let dependency trees grow to thousands of dependencies, from vendors you may have never heard of and likely have not critically evaluated, to solve trivial tasks which could have been done from scratch in mere seconds, or, if properly considered, might not even be needed in the first place.

We need to figure out a way to curb this reckless behavior, but how?

Python: Please stop screwing over Linux distros

Tue, 16 Nov 2021 00:00:00 +0000

Linux distributions? Oh, those things we use to bootstrap our Docker containers? Yeah, those are annoying. What were you complaining about again?

The Python community is obsessed with reinventing the wheel, over and over and over and over and over and over again. distutils, setuptools, pip, pipenv, tox, flit, conda, poetry, virtualenv, requirements.txt, setup.py, setup.cfg, pyproject.toml… I honestly can’t even list all of the things you have to deal with. It’s a disaster.

Status update, November 2021

Mon, 15 Nov 2021 00:00:00 +0000

Hello again! Following a spooky month, we find ourselves again considering the progress of our eternal march towards FOSS world domination.

I’ll first address SourceHut briefly: today is the third anniversary of the opening of the public alpha! I have written a longer post for sourcehut.org which I encourage you to read for all of the details.

In other news, we have decided to delay the release of our new programming language, perhaps by as much as a year. We were aiming for February ‘22, but slow progress on some key areas such as cryptography and the self-hosting compiler, plus the looming necessity of the full-scale acceptance testing of the whole language and standard library, compound to make us unsure about meeting the original release plans. However, progress is slow but moving. We have incorporated the first parts of AES support in our cryptography library, and ported the language to FreeBSD. A good start on date/time support has been under development and I’m pretty optimistic about the API design we’ve come up with. Things are looking good, but it will take longer than expected.

Breaking down Apollo Federation's anti-FOSS corporate gaslighting

Fri, 05 Nov 2021 00:00:00 +0000

Gather around, my friends, for there is another company which thinks we are stupid and we enjoy having our faces spat in. Apollo Federation¹ has announced that they will switch to a non-free license. Let’s find out just how much the Elastic license really is going to “protect the community” like they want you to believe.

Let’s start by asking ourselves, objectively, what practical changes can we expect from a switch from the MIT license to the Elastic License? Both licenses are pretty short, so I recommend quickly reading them yourself before we move on.

GitHub stale bot considered harmful

Tue, 26 Oct 2021 00:00:00 +0000

Disclaimer: I work for a GitHub competitor.

One of GitHub’s “recommended” marketplace features is the “stale” bot. The purpose of this bot is to automatically close GitHub issues after a period of inactivity, 60 days by default. You have probably encountered it yourself in the course of your work.

This is a terrible, horrible, no good, very bad idea.

I’m not sure what motivates maintainers to install this on their repository, other than the fact that GitHub recommends it to them. Perhaps it’s motivated by a feeling of shame for having a lot of unanswered issues? If so, this might stem from a misunderstanding of the responsibilities a maintainer has to their project. You are not obligated to respond to every issue, implement every feature request, or fix every bug, or even acknowledge them in any way.

How SmarterEveryDay's 4privacy can, and cannot, meet its goals

Fri, 22 Oct 2021 00:00:00 +0000

I don’t particularly find myself to be a fan of the SmarterEveryDay YouTube channel, simply for being outside of Destin’s target audience most of the time. I understand that Destin, the channel’s host, is a friendly person and a great asset to his peers, and that he generally strives to do good. When I saw that he was involved in a Kickstarter to develop a privacy product, it piqued my interest. As a privacy advocate and jaded software engineer, I set out to find out what it’s all about.

Software developers have stopped caring about reliability

Sun, 17 Oct 2021 00:00:00 +0000

Of all the principles of software engineering which has fallen by the wayside in the modern “move fast and break things” mentality of ~~assholes~~ modern software developers, reliability is perhaps the most neglected, along with its cousin, robustness. Almost all software that users encounter in $CURRENTYEAR is straight-up broken, and often badly.

Honestly, it’s pretty embarassing. Consider all of the stupid little things you’ve learned how to do in order to work around broken software. Often something as simple as refreshing the page or rebooting the program to knock some sense back into it — most users can handle that. There are much stupider problems, however, and they are everywhere. Every morning, I boot, then immediately hard-reboot, my workstation, because it seems to jigger my monitors into waking up properly to do their job. On many occasions, I have used the browser dev tools to inspect a broken web page to figure out how to make it do the thing I want to do, usually something complicated like submitting a form properly (a solved problem since 1993).¹

Status update, October 2021

Fri, 15 Oct 2021 00:00:00 +0000

On this dreary morning here in Amsterdam, I’ve made my cup of coffee and snuggled my cat, and so I’m pleased to share some FOSS news with you. Some cool news today! We’re preparing for a new core product launch at sr.ht, cool updates for our secret programming language, plus news for visurf.

Simon Ser has been hard at work on expanding his soju and gamja projects for the purpose of creating a new core sourcehut product: chat.sr.ht. We’re rolling this out in a private beta at first, to seek a fuller understanding of the system’s performance characteristics, to make sure everything is well-tested and reliable, and to make plans for scaling, maintenance, and general availability. In short, chat.sr.ht is a hosted IRC bouncer which is being made available to all paid sr.ht users, and a kind of webchat gateway which will be offered to unpaid and anonymous users. I’m pretty excited about it, and looking forward to posting a more detailed announcement in a couple of weeks. In other sourcehut news, work on GraphQL continues, with paste.sr.ht landing and todo.sr.ht’s writable API in progress.

How reflection works in ****

Tue, 05 Oct 2021 00:00:00 +0000

Note: this is a redacted copy of a blog post published on the internal development blog of a new systems programming language. The name of the project and further details are deliberately being kept in confidence until the initial release. You may be able to find it if you look hard enough — you have my thanks in advance for keeping it to yourself. For more information, see “We are building a new systems programming language”.

Developers: Let distros do their job

Mon, 27 Sep 2021 00:00:00 +0000

I wrote a post some time ago titled Developers shouldn’t distribute their own software, and after a discussion on the sr.ht IRC channel today, the topic seems worthy of renewed mention. Let’s start with this: what exactly is a software distribution, anyway?

I use “software distribution” here, rather than “Linux distribution”, because it generalizes better. For example, all of the major BSD systems, plus Illumos and others besides, are software distributions, but don’t involve Linux. Some differ further still, sitting on top of another operating system, such as Nix or pkgsrc. What these systems all have in common is that they concern themselves with the distribution of software, and thus are a software distribution.

Nitter and other Internet reclamation projects

Thu, 23 Sep 2021 00:00:00 +0000

The world wide web has become an annoying, ultra-commercialized space. Many websites today are prioritizing the interests of the company behind the domain, at the expense of the user’s experience and well-being. This has been a frustrating problem for several years, but lately there’s been a heartwarming trend of users fighting back against the corporate web and stepping up to help and serve each other’s needs in spite of them, through what I’ve come to think of as Internet reclamation projects.

Status update, September 2021

Wed, 15 Sep 2021 00:00:00 +0000

It’s a quiet, foggy morning here in Amsterdam, and here with my fresh mug of coffee and a cuddly cat in my lap, I’d like to share the latest news on my FOSS efforts with you. Grab yourself a warm drink and a cat of your own and let’s get started.

First, a new project: visurf. I announced this a few days ago, but the short of it is that I am building a minimal Wayland-only frontend for the NetSurf web browser which uses vi-inspired keybindings. Since the announcement there has been some good progress: touch support, nsvirc, tabs, key repeat, and so on. Some notable medium-to-large efforts ahead of us include a context menu on right click, command completion and history, kinetic scrolling via touch, pinch-to-zoom, clipboard support, and a readability mode. Please help! It’s pretty easy to get involved: join the IRC channel at #netsurf on libera.chat and ask for something to do.

visurf, a web browser based on NetSurf

Sat, 11 Sep 2021 00:00:00 +0000

I’ve started a new side project that I would like to share with you: visurf. visurf, or nsvi, is a NetSurf frontend which provides vi-inspired key bindings and a lightweight Wayland UI with few dependencies. It’s still a work-in-progress, and is not ready for general use yet. I’m letting you know about it today in case you find it interesting and want to help.

NetSurf is a project which has been on my radar for some time. It is a small web browser engine, developed in C independently of the lineage of WebKit and Gecko which defines the modern web today. It mostly supports HTML4 and CSS2, plus only a small amount of HTML5 and CSS3. Its JavaScript support, while present, is very limited. Given the epidemic of complexity in the modern web, I am pleased by the idea of a small browser, more limited in scope, which perhaps requires the cooperation of like-minded websites to support a pleasant experience.

Status update, August 2021

Sun, 15 Aug 2021 00:00:00 +0000

Greetings! It’s shaping up to be a beautiful day here in Amsterdam, and I have found the city much to my liking so far. If you’re in Amsterdam and want to grab a beer sometime, send me an email! I’ve been making a lot of new friends here. Meanwhile, I’ve also enjoyed a noticable increase in my productivity levels. Let’s go over the month’s accomplishments.

First, I have spent most of my time on the programming language project. I mentioned in the last update that we broke ground on a codegen rewrite, and yesterday all of our tests finally passed and I merged it. The new design is much better, and we should be able to simplify it even further still when we write the hosted compiler in the near future. This will also give us a better basis for a small number of experiments we’d like to do before finalizing the language design. Some other improvements include fleshing out our floating point math support library, a base64 module, a poll module, and parallel DNS resolution.

Tips for debugging your new programming language

Wed, 11 Aug 2021 00:00:00 +0000

Say you’re building a new (compiled) programming language from scratch. You’ll inevitably have to debug programs written in it, and worse, many of these problems will lead you into deep magic, as you uncover problems with your compiler or runtime. And as you find yourself diving into the arcane arts, your tools may be painfully lacking: how do you debug code written in a language for which debuggers and other tooling simply has not been written yet?

Police to begin regular, warrant-free searches of homes for child abuse material

Tue, 10 Aug 2021 00:00:00 +0000

The Federal Bureau of Investigations announced a new initiative today to combat the proliferation of child sexual abuse materials (CSAM) in the United States. Starting next year, police will be conducting regular searches of US homes, as often as once or twice per week per home, to find child sexual abuse materials. This initiative will bring more child abusers to justice and help abuse victims to find solace in the knowledge that records of their abuse are not being shared in perpetuity.

proxy.golang.org allows many Go packages to be silently broken

Fri, 06 Aug 2021 00:00:00 +0000

GOPROXY (or proxy.golang.org) is a service through which all “go get” commands (and other module downloads) are routed. It may speed up some operations by providing a cache, and it publishes checksums and an “index” of all Go packages; but this is done at the cost of sending details of all of your module downloads to Google and imposing extra steps when using Go packages from an intranet.

This cache never expires, which can cause some problems: you can keep fetching a module from proxy.golang.org long after the upstream version has disappeared. The upstream author probably had a good reason for removing a version! Because I set GOPROXY=direct in my environment,¹ which bypasses the proxy, I’ve been made aware of a great number of Go packages which have broken dependencies and are none the wiser. They generally can’t reproduce the problem without GOPROXY=direct, which can make it a challenge to rouse up the enthusiasm for upstream to actually fix the issue. Caching modules forever can encourage bitrot.

In praise of PostgreSQL

Thu, 05 Aug 2021 00:00:00 +0000

After writing Praise for Alpine Linux, I have decided to continue writing more articles in praise of good software. Today, I’d like to tell you a bit about PostgreSQL.

Many people don’t understand how old Postgres truly is: the first release¹ was in July of 1996. It used this logo:

After 25 years of persistence, and a better logo design, Postgres stands today as one of the most significant pillars of profound achievement in free software, alongside the likes of Linux and Firefox. PostgreSQL has taken a complex problem and solved it to such an effective degree that all of its competitors are essentially obsolete, perhaps with the exception of SQLite.

My wish-list for the next YAML

Wed, 28 Jul 2021 00:00:00 +0000

YAML is both universally used, and universally reviled. It has a lot of problems, but it also is so useful in solving specific tasks that it’s hard to replace. Some new kids on the block (such as TOML) have successfully taken over a portion of its market share, but it remains in force in places where those alternatives show their weaknesses.

I think it’s clear to most that YAML is in dire need of replacement, which is why many have tried. But many have also failed. So what are the key features of YAML which demonstrate its strengths, and key weaknesses that could be improved upon?

Status update, July 2021

Thu, 15 Jul 2021 00:00:00 +0000

Hallo uit Nederland! I’m writing to you from a temporary workstation in Amsterdam, pending the installation of a better one that I’ll put together after I visit a furniture store today. I’ve had to slow a few things down somewhat while I prepare for this move, and I’ll continue to be slower for some time following it, but things are moving along regardless.

One point of note is that the maintainer for aerc, Reto Brunner, has stepped down from his role. I’m looking for someone new to fill his shoes; please let me know if you are interested.

Is GitHub a derivative work of GPL'd software?

Sun, 04 Jul 2021 00:00:00 +0000

GitHub recently announced a tool called Copilot, a tool which uses machine learning to provide code suggestions, inciting no small degree of controversy. One particular facet of the ensuing discussion piques my curiosity: what happens if the model was trained using software licensed with the GNU General Public License?

Disclaimer: I am the founder of a company which competes with GitHub.

The GPL is among a family of licenses considered “copyleft”, which are characterized by their “viral” nature. In particular, the trait common to copyleft works is the requirement that “derivative works” are required to publish their new work under the same terms as the original copyleft license. Some weak copyleft licenses, like the Mozilla Public License, only apply to any changes to specific files from the original code. Stronger licenses like the GPL family affect the broader work that any GPL’d code has been incorporated into.

How does IRC's federation model compare to ActivityPub?

Sat, 03 Jul 2021 00:00:00 +0000

Today’s federated revolution is led by ActivityPub, leading to the rise of services like Mastodon, PeerTube, PixelFed, and more. These new technologies have a particular approach to federation, which is coloring perceptions on what it actually means for a system to be federated at all. Today’s post will explain how Internet Relay Chat (IRC), a technology first introduced in the late 1980’s, does federation differently, and why.

As IRC has aged, many users today have only ever used a few networks, such as Liberachat (or Freenode, up until several weeks ago), which use a particular IRC model which does not, at first glance, appear to utilize federation. After all, everyone types “irc.libera.chat” into their client and they all end up on the same network and in the same namespace. However, this domain name is backed by a round-robin resolver which will connect you to any of several dozen servers, which are connected to each other¹ and exchange messages on behalf of the users who reside on each. This is why we call them IRC networks — each is composed of a network of servers that work together.

You can't capture the nuance of my form fields

Sun, 27 Jun 2021 00:00:00 +0000

Check out this text box:

Here are some of the nuances of using this text box on my operating system (Linux) and web browser (Firefox):

A finger client

Thu, 24 Jun 2021 00:00:00 +0000

This is a short follow-up to the io_uring finger server article posted about a month ago. In the time since, we have expanded our language with a more complete networking stack, most importantly by adding a DNS resolver. I have used these improvements to write a small client implementation of the finger protocol.

use fmt;
use io;
use net::dial;
use os;
use strings;

@init fn registersvc() void = dial::registersvc("tcp", "finger", [], 79);
@noreturn fn usage() void = fmt::fatal("Usage: {} <user>[@<host>]", os::args[0]);

export fn main() void = {
	if (len(os::args) != 2) usage();

	const items = strings::split(os::args[1], "@");
	defer free(items);
	if (len(items) == 0) usage();

	const user = items[0];
	const host = if (len(items) == 1) "localhost"
		else if (len(items) == 2) items[1]
		else usage();

	match (execute(user, host)) {
		err: dial::error => fmt::fatal(dial::strerror(err)),
		err: io::error => fmt::fatal(io::strerror(err)),
		void => void,
	};
};

fn execute(user: str, host: str) (void | dial::error | io::error) = {
	const conn = dial::dial("tcp", host, "finger")?;
	defer io::close(conn);
	fmt::fprintf(conn, "{}\r\n", user)?;
	io::copy(os::stdout, conn)?;
};

Technically, we could do more, but I chose to just address the most common use-case for finger servers in active use today: querying a specific user. Expanding this with full support for all finger requests would probably only grow this code by 2 or 3 times.

Status update, June 2021

Tue, 15 Jun 2021 00:00:00 +0000

Hiya! Got another status update for you. First, let me share this picture that my dad and I took on our recent astronomy trip (click for full res):

Bonus Venus:

So, what’s new? With SourceHut, there are a few neat goings-on. For one, thanks to Michael Forney putting the finishing touches on the patchset, the long-awaited NetBSD image is now available for builds.sr.ht. Also, the initial lists.sr.ht GraphQL API design is in place, and Simon Ser is working on a new and improved implementation of email discussion parsing for us to use. I’ve also redesigned the registration & onboarding flow based on a maintainer/contributor distinction, which should help people understand how sourcehut works a bit better. Also, as promised, the writable GraphQL API for builds.sr.ht is now available.

Provided "as is", without warranty of any kind

Mon, 14 Jun 2021 00:00:00 +0000

The MIT license contains the following text, in all uppercase no less:

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.

The BSD licenses, GPL family of licenses, Apache 2.0, Mozilla Public License, and likely any other license you’d care to name, have similar clauses of their own. It’s worth taking a moment to consider the implications of this statement and what it says about the social aspects of free and open source software.

I will be moving to the Netherlands

Mon, 07 Jun 2021 00:00:00 +0000

I had been planning a move to the Netherlands for a while, at least until a large COVID-shaped wrench was thrown into the gears. However, I was fully vaccinated by early April, and there are signs of the border opening up now, so my plans have been slowly getting back on track. I sent off my visa application today, and assuming I can navigate the pandemic-modified procedures, I should be able to make my move fairly soon. It’s a little bit intimidating, but I am looking forward to it!

Build your project in our new language

Sun, 30 May 2021 00:00:00 +0000

Do you have a new systems programming project on your todo list? If you’re feeling adventurous, I would like you to give it a crack in our new systems programming language, and to use it to drive improvements in the less-developed areas of our standard library.

Note: we have enough projects on board now. Keep an eye on the blog, I’ll publish another announcement when we’re ready for more.

Are you making a new coreutils implementation? A little OS kernel? A new shell? A GUI toolkit? Database system? Web server? Whatever your systems programming use-case, we think that our language is likely to be a good fit for you, and your help in proving that, and spurring development to rise to meet your needs, would be quite welcome.

Using io_uring to make a high-performance... finger server

Mon, 24 May 2021 00:00:00 +0000

I’m working on adding a wrapper for the Linux io_uring interface to my secret programming language project. To help learn more about io_uring and to test out the interface I was designing, I needed a small project whose design was well-suited for the value-add of io_uring. The Finger protocol is perfect for this! After being designed in the 70’s and then completely forgotten about for 50 years, it’s the perfect small and simple network protocol to test drive this new interface with.

How to write release notes

Wed, 19 May 2021 00:00:00 +0000

Release notes are a concept most of us are familiar with. When a new software release is prepared, the release notes tell you what changed, so you understand what you can expect and how to prepare for the update. They are also occasionally used to facilitate conversations:

Many of the people tasked with writing release notes have never found themselves on that side of the screen before. If that describes you, I would like to offer some advice on how to nail it. Note that this mostly applies to free and open source software, which is the only kind of software which is valid.

aerc, mbsync, and postfix for maximum comfy offline email

Mon, 17 May 2021 00:00:00 +0000

I am the original author of the aerc mail client, though my official relationship with it today is marginal at best. I think that, with hindsight, I’ve come to understand that the “always online” approach of aerc’s IMAP implementation is less than ideal. The next email client (which will exist at some point!) will improve on this design, but, since it’s still my favorite email client despite these flaws, they will have to be worked around.

Status update, May 2021

Sun, 16 May 2021 00:00:00 +0000

Hello! This update is a bit late. I was travelling all day yesterday without internet, so I could not prepare these. After my sister and I got vaccinated, I took a trip to visit her at her home in beautiful Hawaii — it felt great after a year of being trapped within these same four walls. I hope you get that vaccine and things start to improve for you, too!

Pinebook Pro review

Fri, 14 May 2021 00:00:00 +0000

I received the original Pinebook for free from the good folks at Pine64 a few years ago, when I visited Berlin to work with the KDE developers. Honestly, I was underwhelmed. The performance was abysmal and ARM is a nightmare to work with. For these reasons, I was skeptical when I bought the Pinebook Pro. I have also spoken of my disdain for modern laptops in general before: the state of laptops in $CURRENTYEAR is abysmal. As such, I have been using a ThinkPad X200, an 11 year old laptop, as my sole laptop for several years now.

I try not to make unlikable software (and features)

Sat, 08 May 2021 00:00:00 +0000

I am writing to you from The Sky. On my flight today, I noticed an example of “unlikable” software — something I’ve been increasingly aware of recently — inspiring me to pull out my laptop and write. On this plane, there are displays in the back of each seat which provides entertainment for the person seated one row back. Newer planes no longer include these, given that in $CURRENTYEAR everyone would just prefer some power for their phone or laptop. Nevertheless, you can still end up a plane with this design. You can shut the thing off by repeatedly pressing the “☀️ -” button, though that button is rated for half the cycles it will have already received by the time you press it.

godocs.io six months later

Fri, 07 May 2021 00:00:00 +0000

We’re six months on from forking godoc.org following its upstream deprecation, and we’ve made a lot of great improvements since. For those unaware, the original godoc.org was replaced with pkg.go.dev, and a redirect was set up. The new website isn’t right for many projects — one of the most glaring issues is the narrow list of software licenses pkg.go.dev will display documentation for. To continue serving the needs of projects which preferred the old website, we forked the project and set up godocs.io.

In praise of Alpine Linux

Thu, 06 May 2021 00:00:00 +0000

Note: this blog post was originally only available via Gemini, but has been re-formatted for the web.

The traits I prize most in an operating system are the following:

Simplicity
Stability
Reliability
Robustness

As a bonus, I’d also like to have:

Documentation
Professionalism
Performance
Access to up-to-date software

Alpine meets all of the essential criteria and most of the optional criteria (documentation is the weakest link), and far better than any other Linux distribution.

Cryptocurrency is an abject disaster

Mon, 26 Apr 2021 00:00:00 +0000

This post is long overdue. Let’s get it over with.

🛑 Hey! If you write a comment about this article online, disclose your stake in cryptocurrency. I will explain why later in this post. For my part, I held <$10,000 USD worth of Bitcoin prior to 2016, plus small amounts of altcoins. I made a modest profit on my holdings. Today my stake in all cryptocurrency is $0.

Starting on May 1st, users of sourcehut’s CI service will be required to be on a paid account, a change which will affect about half of all builds.sr.ht users.¹ Over the past several months, everyone in the industry who provides any kind of free CPU resources has been dealing with a massive outbreak of abuse for cryptocurrency mining. The industry has been setting up informal working groups to pool knowledge of mitigations, communicate when our platforms are being leveraged against one another, and cumulatively wasting thousands of hours of engineering time implementing measures to deal with this abuse, and responding as attackers find new ways to circumvent them.

Parsers all the way down: writing a self-hosting parser

Thu, 22 Apr 2021 00:00:00 +0000

One of the things we’re working on in my new programming language is a self-hosting compiler. Having a self-hosted compiler is a critical step in the development of (some) programming languages: it signals that the language is mature enough to be comfortably used to implement itself. While this isn’t right for some languages (e.g. shell scripts), for a systems programming language like ours, this is a crucial step in our bootstrapping plan. Our self-hosted parser design was completed this week, and today I’ll share some details about how it works and how it came to be.

Status update, April 2021

Thu, 15 Apr 2021 00:00:00 +0000

Another month goes by! I’m afraid that I have very little to share this month. You can check out the sourcehut “what’s cooking” post for sourcehut news, but outside of that I have focused almost entirely on the programming language project this month, for which the details are kept private.

The post calling for contributors led to a lot of answers and we’ve brought several new people on board — thanks for answering the call! I’d like to narrow the range of problems we still need help with. If you’re interested in (and experienced in) the following problems, we need your help:

The Developer Certificate of Origin is a great alternative to a CLA

Mon, 12 Apr 2021 00:00:00 +0000

Today Amazon released their fork of ElasticSearch, OpenSearch, and I want to take a moment to draw your attention to one good decision in particular: its use of the Developer Certificate of Origin (or “DCO”).

Previously:

Elastic betrayed its community when they changed to a proprietary license. We could have seen it coming because of a particular trait of their contribution process: the use of a Contributor License Agreement, or CLA. In principle, a CLA aims to address legitimate concerns of ownership and copyright, but in practice, they are a promise that one day the stewards of the codebase will take your work and relicense it under a nonfree license. And, ultimately, this is exactly what Elastic did, and exactly what most other projects which ask you to sign a CLA are planning to do. If you ask me, that’s a crappy deal, and I refrain from contributing to those projects as a result.

What should the next chat app look like?

Wed, 07 Apr 2021 00:00:00 +0000

As you’re surely aware, Signal has officially jumped the shark with the introduction of cryptocurrency to their chat app. Back in 2018, I wrote about my concerns with Signal, and those concerns were unfortunately validated by this week’s announcement. Moxie’s insistence on centralized ownership, governance, and servers for Signal puts him in a position of power which is easily, and inevitably, abused. In that 2018 article, and in articles since, I have spoken about the important of federation to address these problems. In addition to federation, what else does a chat app need?

Go is a great programming language

Fri, 02 Apr 2021 00:00:00 +0000

No software is perfect, and thus even for software I find very pleasant, I can usually identify some problems in it — often using my blog to do so. Even my all-time favorite software project, Plan 9, has some painful flaws! For some projects, it may be my fondness for them that drives me to criticise them even more, in the hope that they’ll live up to the level of respect I feel for them.

The world's stupidest IRC bot

Mon, 29 Mar 2021 00:00:00 +0000

I’m an IRC power user, having been hanging out in 200+ channels on 10+ networks 24/7 for the past 10 years or so. Because IRC is standardized and simple, a common pastime for IRC enthusiasts is the creation of bots. In one of the social channels I hang out in, we’ve spent the past 6 years gradually building the world’s stupidest IRC bot: wormy.

For a start, wormy is highly schizophrenic. Though it presents itself as a single bot, it is in fact a bouncer which combines the connections of 7 independent bots. At one point, this number was higher — as many as 11 — but some bots were consolidated.

The complete guide for open sourcing video games

Tue, 23 Mar 2021 00:00:00 +0000

Video games are an interesting class of software. Unlike most software, they are a creative endeavour, rather than a practical utility. Where most software calls for new features to address practical needs of their users, video games call for new features to serve the creative vision of their makers. Similarly, matters like refactoring and paying down tech debt are often heavily de-prioritized in favor of shipping something ASAP. Many of the collaborative benefits of open source are less applicable to video games. It is perhaps for these reasons that there are very few commercial open source games.

We are building a new systems programming language

Fri, 19 Mar 2021 00:00:00 +0000

It’s an open secret: the “secret project” I’ve been talking about is a new systems programming language. It’s been underway since December ‘19, and we hope to release the first version in early 2022. The language is pretty small — we have a mostly complete specification which clocks in at 60 pages. It has manual memory management, no runtime, and it uses a superset of the C ABI, making it easy to link with libraries and C code. It should be suitable almost anywhere C is useful: compilers, system utilities, operating systems, network servers and clients, and so on.

Status update, March 2021

Mon, 15 Mar 2021 00:00:00 +0000

After the brief illusion of spring, this morning meets us with a cold apartment indoors and fierce winds outdoors. Today concludes a productive month, mainly for the secret project and for sourcehut, but also marked by progress in some smaller projects as well. I’ll start with those smaller projects.

I have written a feed reader for Gemini, which is (1) free software, and (2) available as a free hosted service. Big thanks to adnano, the author of the go-gemini library, which has been very helpful for many of my Gemini-related exploits, and who has been a great collaborator. I also used it to provide Gemini support for the new pages.sr.ht, which offers static web and gemini hosting for sr.ht users. I also updated gmni to use BearSSL instead of OpenSSL this month.

The corporate surveillance machine is killing people

Sat, 06 Mar 2021 00:00:00 +0000

I have never been angrier about the corporate surveillance complex, which I have rallied against for years, than I am today. Buying and selling user’s private information on the open market is bad enough for the obvious reasons, but today, I learned that the depths of depravity this market will descend to are without limit. Today I am more angry and ashamed at this industry than I have ever been. Corporate surveillance and adtech has turned your phone into an informant against you and brought about the actual murder of the user.

To make money in FOSS, build a business first

Wed, 03 Mar 2021 00:00:00 +0000

I’ve written about making money in free and open source software before, but it’s a deep topic that merits additional discussion. While previously I focused on what an individual can do in order to build a career in FOSS, but today I want to talk about how you can build a sustainable business in FOSS.

It’s a common mistake to do this the wrong way around: build the software, then the business. Because FOSS requires you to surrender your sole monetization rights, building the software first and worrying about the money later puts you at a huge risk of losing your first-mover advantage. If you’re just making a project which is useful to you and you don’t want the overhead of running a business, then that may be totally okay — you can just build the software without sweating the business issues. If you choose this path, however, be aware that the promise of free and open source software entitles anyone else to build that business without you. If you lapse in your business-building efforts and your software project starts making someone else money, then they’re not at fault for “taking your work” — you gave it to them.

Gmail is a huge source of spam

Thu, 25 Feb 2021 00:00:00 +0000

5× as many spam registrations on sourcehut are from gmail than from the second-largest offender.

# SELECT
  SPLIT_PART(email, '@', 2) as domain, count(*) as count
  FROM "user"
  WHERE user_type = 'suspended'
  GROUP BY domain
  ORDER BY count DESC;
          domain           | count
---------------------------+-------
 gmail.com                 |   119
 qq.com                    |    26
 mail.ru                   |    17
 mailinator.com            |    10
 yopmail.com               |     6
 aol.com                   |     6
 yahoo.com                 |     6
[...more omitted...]

This is just the ones which got through: most spam registrations are detected and ignored before they make it to the database.

A great alternative is rarely fatter than what it aims to replace

Sun, 21 Feb 2021 00:00:00 +0000

This is not always true, but in my experience, it tends to hold up. We often build or evaluate tools which aim to replace something kludgy^Wvenerable. Common examples include shells, programming languages, system utilities, and so on. Rust, Zig, etc, are taking on C in this manner; so too does zsh, fish, and oil take on bash, which in turn takes on the Bourne shell. There are many examples.

All of these tools are fine in their own respects, but they have all failed to completely supplant the software they’re seeking to improve upon.¹ What these projects have in common is that they expand on the ideas of their predecessors, rather than refining them. A truly great alternative finds the nugget of truth at the center of the idea, cuts out the cruft, and solves the same problem with less.

Status update, February 2021

Mon, 15 Feb 2021 00:00:00 +0000

Salutations! It's officially a year of pandemic life. I hear the vaccine distribution is going well, so hopefully there won't be another year of this. In the meanwhile, I've been working hard on free software, what with having little else to do. However, I'm afraid I cannot tell you about most of it!

I've been working on todo.sr.ht's GraphQL API, and it's going quite well. I hope to ship a working read-only version later this month. There have been a number of bug fixes and rote maintenance work on sr.ht as well, but nothing particularly exciting. We did upgrade everything for Alpine 3.13, which went off without a hitch. Anyway, I'll go over the minor details in the sr.ht "what's cooking" post later today.

The rest of the progress was made in secret. Secret! You will have to live in ignorance for now. Sorry!

(unless you click this)

Here's a peek at our progress:

How to make your downstream users happy

Tue, 09 Feb 2021 00:00:00 +0000

There are a number of things that your FOSS project can be doing which will make the lives of your downstream users easier, particularly if you’re writing a library or programmer-facing tooling. Many of your downstreams (Linux distros, pkgsrc, corporate users, etc) are dealing with lots of packages, and some minor tweaks to your workflow will help them out a lot.

The first thing to do is avoid using any build system or packaging system which is not the norm for your language. Also avoid incorporating information into your build which relies on being in your git repo — most packagers prefer to work with tarball snapshots, or to fetch your package from e.g. PyPI. These two issues are definitely the worst offenders. If you do have to use a custom build system, take your time to document it thoroughly, so that users who run into problems are well-equipped to address them. The typical build system or packaging process in use for your language already addressed most of those edge cases long ago, which is why we like it better. If you must fetch, say, version information from git, then please add a fallback, such as an environment variable.

Use open platforms — or else

Thu, 28 Jan 2021 00:00:00 +0000

The ongoing events around /r/wallstreetbets teaches us, once again, about the value of open platforms, and the tremendous risk involved in using proprietary platforms. The economic elites who control those proprietary platforms, backed by their venture capital interests, will shut us down if we threaten them. We’re taking serious risk by casting our lot with them.

Discord, a proprietary instant messaging and VoIP platform, kicked out the /r/WSB community yesterday. They claimed it was due to spam and abuse from bots. These are convenient excuses when considered in the broader context of Discord’s conflict of interest, between its retail investor users and its wall-street investor backers. However, even if we take their explanation at face value, we can easily question Discord’s draconian policies about its proprietary chat protocol. They have a history of cracking down on third-party bots and clients with the same excuses of preventing spam and abuse. If Discord accepts responsibility for preventing spam and abuse, then why are they deplatforming users when they, Discord, failed to prevent it?

Open source means surrendering your monopoly over commercial exploitation

Wed, 20 Jan 2021 00:00:00 +0000

Participation in open source requires you to surrender your monopoly over commercial exploitation. This is a profound point about free and open source software which seems to be causing a lot of companies to struggle with their understanding of the philosophy of FOSS, and it’s worth addressing on its own. It has been apparent for some years now that FOSS is eating the software world, and corporations are trying to figure out their relationship with it. One fact that you will have to confront in this position is that you cannot monopolize the commercial potential of free and open source software.

Elasticsearch does not belong to Elastic

Tue, 19 Jan 2021 00:00:00 +0000

Elasticsearch belongs to its 1,573 contributors, who retain their copyright, and granted Elastic a license to distribute their work without restriction. This is the loophole which Elastic exploited when they decided that Elasticsearch would no longer be open source, a loophole that they introduced with this very intention from the start. When you read their announcement, don’t be gaslit by their deceptive language: Elastic is no longer open source, and this is a move against open source. It is not “doubling down on open”. Elastic has spit in the face of every single one of 1,573 contributors, and everyone who gave Elastic their trust, loyalty, and patronage. This is an Oracle-level move.

Spooky action at a distance

Tue, 19 Jan 2021 00:00:00 +0000

Einstein famously characterized the strangeness of quantum mechanics as “spooky action at a distance”, which, if I had to pick one phrase about physics to be my favorite, would be a strong contender. I like to relate this to programming language design: there are some language features which are similarly spooky. Perhaps the most infamous of these is operator overloading. Consider the following:

x + y

If this were written in C, without knowing anything other than the fact that this code compiles correctly, I can tell you that x and y are numeric types, and the result is their sum. I can even make an educated guess about the CPU instructions which will be generated to perform this task. However, if this were a language with operator overloading… who knows? What if x and y are some kind of some Vector class? It could compile to this:

Status update, January 2021

Fri, 15 Jan 2021 00:00:00 +0000

Hello from the future! My previous status update was last year, but it feels like it was only a month ago. I hope you didn't miss my crappy jokes too much during the long wait.

One of the advancements that I would like to mention this month is the general availability of godocs.io, which is a replacement for the soon-to-be-obsolete godoc.org, based on a fork of their original codebase. Our fork has already attracted interest from many contributors who wanted to work on godoc.org, but found the Google CLA distasteful. We've been hard at work excising lots of Google crap, rewriting the indexer to use PostgreSQL instead of GCP, and making the little JavaScript bits more optional & more conservative in their implementation. We also plan to update it with first-class support for Go modules, which was never added to the upstream gddo codebase. Beyond this, we do not plan on making any large-scale changes: we just want godoc.org to keep being a thing. Enjoy!

On SourceHut, the first point of note is the new dark theme, which is automatically enabled when your user-agent configures prefers-color-scheme: dark. It has gone through a couple of iterations of refinement, and I have a few more changes queued up for my next round of improvements. Please let me know if you notice anything unusual! Additionally, I broke ground on the todo.sr.ht API 2.0 implementation this month. It required some minor changes to our underlying GraphQL approach, but in general it should be fairly straightforward — albeit time consuming — to implement. Ludovic has also started working on an API 2.0 branch for hg.sr.ht, which I plan on reviewing shortly.

Small projects have enjoyed some improvements as well. mkproof grew multi-processor support and had its default difficulty tweaked accordingly — thanks, Tom! Zach DeCook and Nolan Prescott also sent some bugfixes for gmnisrv, and René Wagner and Giuseppe Lumia both helped fix some issues with gmni as well. Jason Phan sent an improvement for dowork which adds random jitter to the exponential backoff calculation. Thanks to all of these folks for their help!

That's all for today. Thanks again for your support and attention, and I'll see you again soon!

...

I have actually been working on this a lot this month. Progress is good.

fn measurements() void = {
	const x = "Hello!";
	assert(len(x) == 6z);
	assert(size(str) == size(*u8) + size(size) * 2z);
	const align: size =
		if (size(*u8) > size(size)) size(*u8)
		else size(size);
	assert(&x: uintptr: size % align == 0z);
};

fn charptr() void = {
	const x = "Hello!";
	const y = x: *const char;
	const z = y: *[*]u8;
	const expected = ['H', 'e', 'l', 'l', 'o', '!', '\0'];
	for (let i = 0z; i < len(expected); i += 1z) {
		assert(z[i] == expected[i]: u32: u8);
	};
};

fn storage() void = {
	const string = "こんにちは";
	const ptr = &string: *struct {
		data: *[*]u8,
		length: size,
		capacity: size,
	};
	assert(ptr.length == 15z && ptr.capacity == 15z);

	// UTF-8 encoded
	const expected = [
		0xE3u8, 0x81u8, 0x93u8, 0xE3u8, 0x82u8, 0x93u8, 0xE3u8, 0x81u8,
		0xABu8, 0xE3u8, 0x81u8, 0xA1u8, 0xE3u8, 0x81u8, 0xAFu8, 0x00u8,
	];
	for (let i = 0z; i < len(expected); i += 1z) {
		assert(ptr.data[i] == expected[i]);
	};
};

export fn main() void = {
	measurements();
	charptr();
	storage();
};

History will not remember us fondly

Thu, 07 Jan 2021 00:00:00 +0000

Today, we recall the Middle Ages as an unenlightened time (quite literally, in fact). We view the Middle Ages with a critical eye towards its brutality, lack of individual freedoms, and societal and technological regression. But we rarely turn that same critical lens on ourselves to consider how we’ll be perceived by future generations. I expect the answer, upsetting as it may be, is this: the future will think poorly of us.

Fostering a culture that values stability and reliability

Mon, 04 Jan 2021 00:00:00 +0000

There’s an idea which encounters a bizarre level of resistance from the broader software community: that software can be completed. This resistance manifests in several forms, perhaps the most common being the notion that a git repository which doesn’t receive many commits is abandoned or less worthwhile. For my part, I consider software that aims to be completed to be more worthwhile most of the time.

There are two sources of change which projects are affected by: external and internal. An internal source of change is, for example, a planned feature, or a discovered bug. External sources of change are, say, when a dependency makes a breaking change and your software has to be updated accordingly. Some projects will necessarily have an indefinite source of external change to consider, often as part of their value proposition. youtube-dl will always evolve to add new sites and workarounds, wlroots will continue to grow to take advantage of new graphics and input hardware features, and so on.

A megacorp is not your dream job

Fri, 01 Jan 2021 00:00:00 +0000

Megacorporations¹ do not care about you. You’re worth nothing to them. Google made $66 billion in 2014 — even if you made an exorbitant $500K salary, you only cost them .00075% of that revenue. They are not invested in you. Why should you invest in them? Why should you give a company that isn’t invested in you 40+ hours of your week, half your waking life, the only life you get?

How to design a new programming language from scratch

Fri, 25 Dec 2020 00:00:00 +0000

There is a long, difficult road from vague, pie-in-the-sky ideas about what would be cool to have in a new programming language, to a robust, self-consistent, practical implementation of those ideas. Designing and implementing a new programming language from scratch is one of the most challenging tasks a programmer can undertake.

Note: this post is targeted at motivated programmers who want to make a serious attempt at designing a useful programming language. If you just want to make a language as a fun side project, then you can totally just wing it. Taking on an unserious project of that nature is also a good way to develop some expertise which will be useful for a serious project later on.

godocs.io is now available

Fri, 18 Dec 2020 00:00:00 +0000

Due to the coming sunsetting of godoc.org in favor of pkg.go.dev, I’m happy to announce that godocs.io is now available as a replacement. We have forked the codebase and cleaned things up quite a bit, removing lots of dead or obsolete features, cleaning out a bunch of Google-specific code and analytics, reducing the JavaScript requirements, and rewriting the search index for Postgres. We will commit to its maintenance going forward for anyone who prefers the original godoc.org experience over the new website.

Status update, December 2020

Tue, 15 Dec 2020 00:00:00 +0000

Happy holidays! I hope everyone’s having a great time staying at home and not spending any time with your families. It’s time for another summary of the month’s advances in FOSS development. Let’s get to it!

One of my main focuses has been on sourcehut’s API 2.0 planning. This month, the meta.sr.ht and git.sr.ht GraphQL APIs have shipped feature parity with the REST APIs, and the RFC 6749 compatible OAuth 2.0 implementation has shipped. I’ve broken ground on the todo.sr.ht GraphQL API — it’ll be next. Check out the GraphQL docs on man.sr.ht if you want to kick the tires.

Become shell literate

Sat, 12 Dec 2020 00:00:00 +0000

Shell literacy is one of the most important skills you ought to possess as a programmer. The Unix shell is one of the most powerful ideas ever put to code, and should be second nature to you as a programmer. No other tool is nearly as effective at commanding your computer to perform complex tasks quickly — or at storing them as scripts you can use later.

In my workflow, I use Vim as my editor, and Unix as my “IDE”. I don’t trick out my vimrc to add a bunch of IDE-like features — the most substantial plugin I use on a daily basis is Ctrl+P, and that just makes it easier to open files. Being Vim literate is a valuable skill, but an important detail is knowing when to drop it. My daily workflow involves several open terminals, generally one with Vim, another to run builds or daemons, and a third which just keeps a shell handy for anything I might ask of it.

Web analytics should at least meet the standards of informed consent

Fri, 04 Dec 2020 00:00:00 +0000

Research conducted on human beings, at least outside of the domain of technology, has to meet a minimum standard of ethical reasoning called informed consent. Details vary, but the general elements of informed consent are:

Disclosure of the nature and purpose of the research and its implications (risks and benefits) for the participant, and the confidentiality of the collected information.
An adequate understanding of these facts on the part of the participant, requiring an accessible explanation in lay terms and an assessment of understanding.
The participant must exercise voluntary agreement, without coercion or fear of repercussions (e.g. not being allowed to use your website).

So, I pose the following question: if your analytics script wouldn’t pass muster at your university’s ethics board, then what the hell is it doing on your website? Can we not meet this basic minimum standard of ethical decency and respect for our users?

A few ways to make money in FOSS

Fri, 20 Nov 2020 00:00:00 +0000

I work on free and open-source software full time, and I make a comfortable living doing it. And I don’t half-ass it: 100% of my code is free and open-source. There’s no proprietary add-ons, no periodic code dumps, just 100% bona-fide free and open source software. Others have often sought my advice — how can they, too, make a living doing open source?

Well, there’s more than one way to skin a cat. There are many varieties of software, each with different needs, and many kinds of people, each with different needs. The exact approach which works for you and your project will vary quite a bit depending on the nature of your project.

We can do better than DuckDuckGo

Tue, 17 Nov 2020 00:00:00 +0000

DuckDuckGo is one of the long-time darlings of the technophile’s pro-privacy recommendations, and in fact the search engine that I use myself on the daily. They certainly present a more compelling option than many of the incumbents, like Google or Bing. Even so, DuckDuckGo is not good enough, and we ought to do better.

I have three grievances with DuckDuckGo:

It’s not open source. Almost all of DDG’s software is proprietary, and they’ve demonstrated gross incompetence in privacy in what little software they have made open source. Who knows what else is going on in the proprietary code?
DuckDuckGo is not a search engine. It’s more aptly described as a search engine frontend. They do handle features like bangs and instant answers internally, but their actual search results come from third-parties like Bing. They don’t operate a crawler for their search results, and are not independent.
The search results suck! The authoritative sources for anything I want to find are almost always buried beneath 2-5 results from content scrapers and blogspam. This is also true of other search engines like Google. Search engines are highly vulnerable to abuse and they aren’t doing enough to address it.

There are some FOSS attempts to do better here, but they all fall flat. searX is also a false search engine — that is, they serve someone else’s results. YaCy has their own crawler, but the distributed design makes results untolerably slow, poor quality, and vulnerable to abuse, and it’s missing strong central project leadership.

Status update, November 2020

Sun, 15 Nov 2020 00:00:00 +0000

Greetings, humanoids! Our fleshy vessels have aged by 2.678×10⁶ seconds, and you know what that means: time for another status update! Pour a cup of your favorite beverage stimulant and gather ‘round for some news.

First off, today is the second anniversary of SourceHut’s alpha being opened to the public, and as such, I’ve prepared a special blog post for you to read. I’ll leave the sr.ht details out of this post and just send you off to read about it there.

Utility vs usability

Fri, 06 Nov 2020 00:00:00 +0000

In many fields, professional-grade tooling requires a high degree of knowledge and training to use properly, usually more than is available to the amateur. The typical mechanic’s tool chest makes my (rather well-stocked, in my opinion) tool bag look quite silly. A racecar driver is using a vehicle which is much more complex than, say, the soccer mom’s mini-van. Professional-grade tools are, necessarily, more complex and require skill to use.

What is this Gemini thing anyway, and why am I excited about it?

Sun, 01 Nov 2020 00:00:00 +0000

I’ve been writing about some specific topics in the realm of Gemini on my blog over the past two months or so, but I still haven’t written a broader introduction to Gemini, what I’m doing with it, and why you should be excited about it, too. Let’s do that today!

Gemini is a network protocol for exchanging hypertext documents — “hypertext” in the general sense of the word, not with respect to the hypertext markup language (HTML) that web browsers understand. It’s a simple network protocol which allows clients to request hypertext documents (in its own document format, gemtext). It is, in some respects, an evolution of Gopher, but more modernized and streamlined.

I'm handing over maintenance of wlroots and sway to Simon Ser

Fri, 23 Oct 2020 00:00:00 +0000

Over the past several months, I’ve been gradually weaning down my role in both projects, and as a contributor to Wayland in general. I feel that I’ve already accomplished everything I set out to do with Wayland — and more! I have been happily using sway as my daily driver for well over a year with no complaints or conspicuously absent features. For me, there’s little reason to stay involved. This will likely come as no surprise to many who’ve kept their ear to the ground in these communities.

Firefox: The Jewel^WEmbarassment of Open Source

Thu, 22 Oct 2020 00:00:00 +0000

Circa 2006, the consensus on Firefox was concisely stated by this classic xkcd:

Status update, October 2020

Thu, 15 Oct 2020 00:00:00 +0000

I’m writing this month’s status update from a brand-new desktop workstation (well, I re-used the GPU), my first new workstation in about 10 years. I hope this new one lasts for another decade! I aimed for something smaller and lightweight this time — it’s a Mini-ITX build. I’ve only been running this for a few days, so let me tell you about the last few accomplishments which are accountable to my venerable workstation’s final days of life.

Four principles of software engineering

Fri, 09 Oct 2020 00:00:00 +0000

Software should be robust. It should be designed to accommodate all known edge cases. In practice, this means predicting and handling all known error cases, enumerating and addressing all classes of user inputs, reasoning about and planning for the performance characteristics of your program, and so on.

Software should be reliable. It should be expected to work for an extended length of time under design conditions without failures. Ideally, it should work outside of design conditions up to some threshold.

Spamtoberfest

Thu, 01 Oct 2020 00:00:00 +0000

As I’ve written before, the best contributors to a FOSS project are intrinsically motivated to solve problems in your software. This sort of contribution is often fixing an important problem and places a smaller burden on maintainers to spend their time working with the contributor. I’ve previously contrasted this with the “I want to help out!” contributions, where a person just has a vague desire to help out. Those contributions are, generally, less valuable and place a greater burden on the maintainer. Now, DigitalOcean has lowered the bar even further with Hacktoberfest.

A tale of two libcs

Fri, 25 Sep 2020 00:00:00 +0000

I received a bug report from Debian today, who had fed some garbage into scdoc, and it gave them a SIGSEGV back. Diving into this problem gave me a good opportunity to draw a comparison between musl libc and glibc. Let’s start with the stack trace:

==26267==ERROR: AddressSanitizer: SEGV on unknown address 0x7f9925764184
(pc 0x0000004c5d4d bp 0x000000000002 sp 0x7ffe7f8574d0 T0)
==26267==The signal is caused by a READ memory access.
    0 0x4c5d4d in parse_text /scdoc/src/main.c:223:61
    1 0x4c476c in parse_document /scdoc/src/main.c
    2 0x4c3544 in main /scdoc/src/main.c:763:2
    3 0x7f99252ab0b2 in __libc_start_main
/build/glibc-YYA7BZ/glibc-2.31/csu/../csu/libc-start.c:308:16
    4 0x41b3fd in _start (/scdoc/scdoc+0x41b3fd)

And if we pull up that line of code, we find…

TOFU recommendations for Gemini

Mon, 21 Sep 2020 00:00:00 +0000

I will have more to say about Gemini in the future, but for now, I wanted to write up some details about one thing in particular: the trust-on-first-use algorithm I implemented for my client, gmni. I think you should implement this algorithm, too!

First of all, it’s important to note that the Gemini specification explicitly mentions TOFU and the role of self-signed certificates: they are the norm in Geminiland, and if your client does not support them then you’re going to be unable to browse many sites. However, the exact details are left up to the implementation. Here’s what mine does:

The unrealized potential of federation

Sun, 20 Sep 2020 00:00:00 +0000

There are some major problems on the internet which may seem intractable. How do we prevent centralization of our communication tools under the authority of a few, whose motivations may not align with our interests? How do we build internet-scale infrastructure without a megacorp-scale budget? Can we make our systems reliable and fault-tolerant — in the face of technical and social problems?

Federation is an idea which takes a swing at all of these problems.

Status update, September 2020

Tue, 15 Sep 2020 00:00:00 +0000

A mercifully cool September is upon us, and after years of searching, I finally was able to secure Club Mate in the US. Let’s decant a bottle and recant the story of this month’s progress in free software development.

First of all, I’ve been able to put a pin on operations work on SourceHut for the time being, and focus again on its software development. The GraphQL APIs are a major focus area here, and I’ve made a lot of progress towards OAuth 2.0 support and writable GraphQL APIs. Additionally, I’ve laid out a number of prioritized tickets for the beta — with the “beta” label on todo.sr.ht — and have been picking items off of the list one at a time, mainly focusing on meta.sr.ht improvements at first. I’ll go into more detail in the What’s Cooking post for SourceHut later today, stay tuned.

Linux development is distributed - profoundly so

Wed, 02 Sep 2020 00:00:00 +0000

The standard introduction to git starts with an explanation of what it means to use a “distributed” version control system. It’s pointed out that every developer has a complete local copy of the repository and can work independently and offline, often contrasting this design with systems like SVN and CVS. The explanation usually stops here. If you want to learn more, consider git’s roots: it is the version control system purpose-built for Linux, the largest and most active open source project in the world. To learn more about the true nature of distributed development, we should observe Linux.

Embrace, extend, and finally extinguish - Microsoft plays their hand

Thu, 27 Aug 2020 00:00:00 +0000

GitHub took a note out of the Microsoft “EEE” playbook when designing their git services. They embraced git, and then rather than building an interface on top of email — the collaboration mechanism that git was designed to use, and which is still used for Linux kernel development¹ — they built their “pull requests” mechanism.

They took terminology which already had meaning — “fork”, meaning the creation a separate governing body and development upstream for a codebase, a rather large task; and “pull request”, a git workflow which prepares an email asking a receipient to pull a large branch of changes from a non-centralized source — and replaced these decentralized, open systems with a completely incompatible system designed to keep you on GitHub and to teach you to collaborate using GitHub’s proprietary tools. They extended git in a proprietary way.

Alice in Wonderland and the theft of the public domain

Mon, 24 Aug 2020 00:00:00 +0000

Disney’s Alice in Wonderland is one of my favorite movies and an undisputed classic. After its release in 1951, Alice holds a fond place in billions of children’s hearts, over almost four generations. And it has been stolen from those generations, as part of the theft of one of these generations’ greatest treasures: the public domain.

I often use this film as an example when arguing about copyright. Almost everyone I speak to was born well after the film’s release (in fact, this is true of almost everyone alive today), but they remember it fondly regardless. Many people I’ve spoken to would agree that it even played a formative role in their childhoods; it’s a film dear to many hearts. My mom is very fond of the Cheshire Cat in particular, and owns quite a bit of relevant merchandise.¹

Software engineers solve problems

Mon, 17 Aug 2020 00:00:00 +0000

Software engineers solve problems. A problem you may have encountered is, for example, “this function has a bug”, and you’re probably already more or less comfortable solving these problems. Here are some other problems you might encounter on the way:

Actually, the bug ultimately comes from a third-party program
Hm, it uses a programming language I don’t know
Oh, the bug is in that programming language’s compiler
This subsystem of the compiler would have to be overhauled
And the problem is overlooked by the language specification

I’ve met many engineers who, when standing at the base of this mountain, conclude that the summit is too far away and clearly not their responsibility, and subsequently give up. But remember: as an engineer, your job is to apply creativity to solving problems. Are these not themselves problems to which the engineering process may be applied?

Status update, August 2020

Sun, 16 Aug 2020 00:00:00 +0000

Greetings! Today is another rainy day here in Philadelphia, which rather sours my plans of walking over to the nearby cafe to order some breakfast to-go. But I am tired, and if I’m going to make it to the end of this blog post in one piece, I’m gonna need a coffee. brb.

Hey, that was actually pretty refreshing. It’s just drizzling, and the rain is nice and cool. Alright, here goes! What’s new? I’ll leave the Wayland news for Simon Ser’s blog this month - he’s been working on some exciting stuff. The BARE encoding announced last month has received some great feedback and refinements, and there are now six projects providing BARE support for their author’s favorite programming language¹. There have also been some improvements to the Go implementation which should help with some SourceHut plans later on.

Web browsers need to stop

Thu, 13 Aug 2020 00:00:00 +0000

Enough is enough.

The web and web browsers have become Lovecraftian horrors of an unprecedented scale. They’ve long since left “scope creep” territory and entered “oh my god please just stop” territory, and are trucking on through to hitherto unexplored degrees of obscene scope. And we don’t want what they’re selling. Google pitches garbage like AMP¹ and pushing dubious half-assed specs like Web Components. Mozilla just fired everyone relevant² to focus on crap no one asked for like Pocket, and fad nonsense like a paid VPN service and ~~virtual reality tech~~.³ [2020-08-14: It has been pointed out that the VR team was also fired.]

I want to contribute to your project, how do I start?

Mon, 10 Aug 2020 00:00:00 +0000

I get this question a lot! The answer is usually… don’t. If you already know what you want to do, then the question doesn’t need to be asked.¹ But, if you don’t already know what you want to do, then your time might be better spent elsewhere!

The best contributors are always intrinsically motivated. Some contributors show up every now and then who appreciate the value the project gives to them and want to give something back. Their gratitude is definitely appreciated², but these kinds of contributions tend to require more effort from the maintainers, and don’t generally lead to recurring contributions. Projects you already like are less likely to need help when compared to incomplete projects that you don’t already depend on — so this model leaves newer projects with fewer contributors and encourages established projects to grow in complexity.

pkg.go.dev is more concerned with Google's interests than good engineering

Sat, 01 Aug 2020 00:00:00 +0000

pkg.go.dev sucks. It’s certainly prettier than godoc.org, but under the covers, it’s a failure of engineering characteristic of the Google approach.

Go is a pretty good programming language. I have long held that this is not attributable to Google’s stewardship, but rather to a small number of language designers and a clear line of influences which is drawn entirely from outside of Google — mostly from Bell Labs. pkg.go.dev provides renewed support for my argument: it has all the hallmarks of Google crapware and none of the deliberate, good engineering work that went into Go’s design.

The falsehoods of anti-AGPL propaganda

Mon, 27 Jul 2020 00:00:00 +0000

Google is well-known for forbidding the use of software using the GNU Affero General Public License, commonly known as “AGPL”. Google is also well-known for being the subject of cargo-culting by fad startups. Unfortunately, this means that they are susceptible to what is ultimately anti-AGPL propaganda from Google, with little to no basis in fact.

Obligatory: I’m not a lawyer; this is for informational purposes only.

In truth, the terms of the AGPL are pretty easy to comply with. The basic obligations of the AGPL which set it apart from other licenses are as follows:

Status update, July 2020

Wed, 15 Jul 2020 00:00:00 +0000

Hello again! Another month of FOSS development behind us, and we’re back again to share the results. I took a week off at the end of June, so my progress this month is somewhat less than usual. Regardless, I have some updates for you, mainly in the domain of SourceHut work.

But before we get to that, let’s go over this month’s small victories. One was the invention of the BARE message format, which I wrote a blog post about if you want to learn more. Since that article, five new implementations have appeared from various authors: Rust, Python, JavaScript, D, and Zig.

March 2nd, 1943

Tue, 14 Jul 2020 00:00:00 +0000

It’s March 2nd, 1943. The user asks your software to schedule a meeting with Acmecorp at “9 AM on the first Monday of next month”.


[6:17:45] homura ~ $ cal -3 2 March 1943
    February 1943          March 1943            April 1943
Su Mo Tu We Th Fr Sa  Su Mo Tu We Th Fr Sa  Su Mo Tu We Th Fr Sa
    1  2  3  4  5  6      1  2 3  4  5  6               1  2  3
 7  8  9 10 11 12 13   7  8  9 10 11 12 13   4  5  6  7  8  9 10
14 15 16 17 18 19 20  14 15 16 17 18 19 20  11 12 13 14 15 16 17
21 22 23 24 25 26 27  21 22 23 24 25 26 27  18 19 20 21 22 23 24
28                    28 29 30 31           25 26 27 28 29 30

Right now, California is on Pacific Standard Time (PST) and Arizona is on Mountain Standard Time (MST). On March 8th, California will transition to Pacific Daylight Time (PDT), one hour ahead. Arizona does not observe DST, so they’ll stay behind.

General-purpose OS, special-purpose OS, and now: vendor-purpose OS

Fri, 26 Jun 2020 00:00:00 +0000

There have, historically, been two kinds of operating systems: general-purpose, and special-purpose. These roles are defined by the function they serve for the user. Examples of general-purpose operating systems include Unix (Linux, BSD, etc), Solaris, Haiku, Plan 9, and so on. These are well-suited to general computing tasks, and are optimized to solve the most problems possible, perhaps at the expense of those in some niche domains. Special-purpose operating systems serve those niche domains, and are less suitable for general computing. Examples of these include FreeRTOS, Rockbox, Genode, and so on.

Introducing the BARE message encoding

Sun, 21 Jun 2020 00:00:00 +0000

I like stateless tokens. We started with stateful tokens: where a generated string acts as a unique identifier for a resource, and the resource itself is looked up separately. For example, your sr.ht OAuth token is a stateful token: we just generate a random number and hand it to you, something like “a97c4aeeec705f81539aa”. To find the information associated with this token, we query the database — our local state — to find it.

Status update, June 2020

Mon, 15 Jun 2020 00:00:00 +0000

Like last month, I am writing to you from the past, preparing this status update a day earlier than usual. This time it’s because I expect to be busy with planned sr.ht maintenance tomorrow, so I’m getting the status updates written ahead of time.

aerc has seen lots of patches merged recently thanks to the hard work of co-maintainer Reto Brunner and the many contributors who sent patches, ranging from a scrollable folder list to improvements and bugfixes for PGP support. We wrapped all of this up in the aerc 0.4.0 release in late May. Thanks to Reto and all of the other contributors for their hard work on aerc!

Can we talk about client-side certificates?

Fri, 12 Jun 2020 00:00:00 +0000

I’m working on improving the means by which API users authenticate with the SourceHut API. Today, I was reading RFC 6749 (OAuth2) for this purpose, and it got me thinking about the original OAuth spec. I recalled vaguely that it had the API clients actually sign every request, and… yep, indeed it does. This also got me thinking: what else signs requests? TLS!

OAuth is very complicated. The RFC is 76 pages long, the separate bearer token RFC (6750) is another 18, and no one has ever read either of them. Add JSON Web Tokens (RFC 7519, 30 pages), too. The process is complicated and everyone implements it themselves — a sure way to make mistakes in a security-critical component. Not all of the data is authenticated, no cryptography is involved at any step, and it’s easy for either party to end up in an unexpected state. The server has to deal with problems of revocation and generating a secure token itself. Have you ever met anyone who feels positively about OAuth?

Add a "contrib" directory to your projects

Sat, 06 Jun 2020 00:00:00 +0000

There’s a common pattern among free- and open-source software projects to include a “contrib” directory at the top of their source code tree. I’ve seen this in many projects for many years, but I’ve seen it discussed only rarely — so here we are!

The contrib directory is used as an unorganized (or, at best, lightly organized) bin of various useful things contributed by the community around the software, but which is not necessarily a good candidate for being a proper part of the software. Things in contrib should not be wired into your build system, shouldn’t be part of your automated testing, shouldn’t be included in your documentation, and should not be installed with your packages. contrib entries are not supported by the maintainers, and are given only a light code review at the most. There is no guarantee whatsoever of workitude or maintenance for anything found in contrib.

Status update, May 2020

Fri, 15 May 2020 00:00:00 +0000

Hello, future readers! I am writing to you from one day in the past. I finished my plans for today early and thought I’d get a head start on writing the status updates for tomorrow, or rather, for today. From your reference frame, that is.

Let’s start with Wayland. First, as you might have heard, The Wayland Protocol is now free for anyone to read, and has been relicensed as CC-BY-SA. Enjoy! It’s still not quite done, but most of it’s there. In development news, wlroots continues to enjoy incremental improvements, and is being refined further and further towards a perfect citizen of the ecosystem in which it resides. Sway as well has seen many small bugfixes and improvements. Both have been been stable for a while now: the only meaningful changes will be, for the most part, a steady stream of bug fixes and performance improvements.

We are complicit in our employer's deeds

Tue, 05 May 2020 00:00:00 +0000

Tim Bray’s excellent “Bye Amazon” post inspired me to take this article off of my backlog, where it has been sitting for a few weeks. I applaud Tim for stepping down from a company that has demonstrated itself incompatible with his sense of right and wrong, and I want to take a moment to remind you that the rest of us in the tech industry have the same opportunity — no, the same obligation as Tim did.

How to store data forever

Wed, 22 Apr 2020 00:00:00 +0000

As someone who has been often maligned by the disappearance of my data for various reasons — companies going under, hard drive failure, etc — and as someone who is responsible for the safekeeping of other people’s data, I’ve put a lot of thought into solutions for long-term data retention.

There are two kinds of long-term storage, with different concerns: cold storage and hot storage. The former is like a hard drive in your safe — it stores your data, but you’re not actively using it or putting wear on the storage medium. By contrast, hot storage is storage which is available immediately and undergoing frequent reads and writes.

Configuring aerc for git via email

Mon, 20 Apr 2020 00:00:00 +0000

I use aerc as my email client (naturally — I wrote it, after all), and I use git send-email to receive patches to many of my projects. I designed aerc specifically to be productive for this workflow, but there are a few extra things that I use in my personal aerc configuration that I thought were worth sharing briefly. This blog post will be boring and clerical, feel free to skip it unless it’s something you’re interested in.

Status update, April 2020

Wed, 15 Apr 2020 00:00:00 +0000

Wow, it’s already time for another status update? I’m starting to lose track of the days stuck inside. I have it easier than many - I was already used to working from home before any of this began. But, weeks and weeks of not spending IRL time with anyone else is starting to get to me. Remember to call your friends and family and let them know how you’re doing. Meanwhile, I’ve had a productive month - let’s get you up to date!

My unorthodox, branchless git workflow

Mon, 06 Apr 2020 00:00:00 +0000

I have been using git for a while, and I took the time to learn about it in great detail. Equipped with an understanding of its internals and a comfortable familiarity with tools like git rebase — and a personal, intrinsic desire to strive for minimal and lightweight solutions — I have organically developed a workflow which is, admittedly, somewhat unorthodox.

In short, I use git branches very rarely, preferring to work on my local master branch almost every time. When I want to work on multiple tasks in the same repository (i.e. often), I just… work on all of them on master. I waste no time creating a new branch, or switching to another branch to change contexts; I just start writing code and committing changes, all directly on master, intermixing different workstreams freely.¹ This reduces my startup time to zero, both for starting new tasks and revisiting old work.

Designing and 3D printing a new part for my truck

Wed, 25 Mar 2020 00:00:00 +0000

I drove a car daily for many years while I was living in Colorado, California, and New Jersey, but since I moved to Philadelphia I have not needed a car. The public transit here is not great, but it’s good enough to get where I need to be and it’s a lot easier than worrying about parking a car. However, in the past couple of years, I have been moving more and more large server parts back and forth to the datacenter for SourceHut. I’ve also developed an interest in astronomy, which benefits from being able to carry large equipment to remote places. These reasons, among others, put me into the market for a vehicle once again.

The reckless, infinite scope of web browsers

Wed, 18 Mar 2020 00:00:00 +0000

Since the first browser war between Netscape and Internet Explorer, web browsers have been using features as their primary means of competing with each other. This strategy of unlimited scope and perpetual feature creep is reckless, and has been allowed to go on for far too long.

I used wget to download all 1,217 of the W3C specifications which have been published at the time of writing¹, of which web browsers need to implement a substantial subset in order to provide a modern web experience. I ran a word count on all of these specifications. How complex would you guess the web is?

GitHub's new notifications: a case of regressive design

Fri, 13 Mar 2020 00:00:00 +0000

Disclaimer: I am the founder of a company which competes with GitHub. However, I still use tools like GitHub, GitLab, and so on, as part of regular contributions to projects all over the FOSS ecosystem. I don’t dislike GitHub, and I use it frequently in my daily workflow.

GitHub is rolling out a new notifications UI. A few weeks ago, I started seeing the option to try it. Yesterday, I received a warning that the old UI will soon be deprecated. At this pace, I would not be surprised to see the new UI become mandatory in a week or two. I’m usually optimistic about trying out new features, but this change worried me right away. I still maintain a few projects on GitHub, and I frequently contribute to many projects there. Using the notification page to review these projects is a ritual I usually conduct several times throughout the workday. So, I held my breath and tried it out.

An open letter to Senator Bob Casey on end-to-end encryption

Sat, 07 Mar 2020 00:00:00 +0000

To Senator Bob Casey, I’m writing this open letter.

As your constituent, someone who voted for you in 2018, and an expert in software technology, I am disappointed in your support of the EARN IT Act. I am aware that encryption is a challenging technology to understand, even for us software engineers, and that it raises difficult problems for the legislature. The EARN IT Act does not protect our children, and it has grave implications for the freedoms of our citizens.

The Abiopause

Tue, 03 Mar 2020 00:00:00 +0000

The sun has an influence on its surroundings. One of these is in the form of small particles that are constantly ejected from the sun in all directions, which exerts an outward pressure, creating an expanding sphere of particles that moves away from the sun. These particles are the solar wind. As the shell of particles expands, the density (and pressure) falls. Eventually the solar wind reaches the interstellar medium — the space between the stars — which, despite not being very dense, is not empty. It exerts a pressure that pushes inwards, towards the sun.

Thoughts on performance & optimization

Fri, 21 Feb 2020 00:00:00 +0000

The idea that programmers ought to or ought not to be called “software engineers” is a contentious one. How you approach optimization and performance is one metric which can definitely push my evaluation of a developer towards the engineering side. Unfortunately, I think that a huge number of software developers today, even senior ones, are approaching this problem poorly.

Centrally, I believe that you cannot effectively optimize a system which you do not understand. Say, for example, that you’re searching for a member of a linked list, which is an O(n) operation. You know this can be improved by switching from a linked list to a sorted array and using a binary search. So, you spend a few hours refactoring, commit the implementation, and… the application is no faster. What you failed to consider is that the lists are populated from data received over the network, whose latency and bandwidth constraints make the process much slower than any difference made by the kind of list you’re using. If you’re not optimizing your bottleneck, then you’re wasting your time.

Fucking laptops

Tue, 18 Feb 2020 00:00:00 +0000

The best laptop ever made is the ThinkPad X200, and I have two of them. The caveats are: I get only 2-3 hours of battery life even with conservative use; and it struggles to deal with 1080p videos.

The integrated GPU, Bluetooth and WiFi, internal sensors, and even the fingerprint reader can all be driven by the upstream Linux kernel. In fact, the hardware is so well understood that I have successfully used almost all of the laptop’s features on Linux, FreeBSD, NetBSD, Minix, Haiku, and Plan 9. Plan fucking 9. It can run coreboot, too. The back of the laptop has all of the screws (Phillips head) labelled so you know which to remove to service which parts. User replacable parts include the screen, keyboard (multiple layouts are available and are interchangeable), the RAM, hard drive (I put a new SSD in one of mine a few weeks ago, and it took about 30 seconds) — actually, there are a total of 26 replacable parts in this laptop.¹ There is a detailed 278-page service manual to assist you or your local repair tech in addressing any problems that arise.

Status update, February 2020

Sat, 15 Feb 2020 00:00:00 +0000

Today I thought it’d try out something new: I have an old family recipe simmering on the stove right now, but instead of beef I’m trying out impossible beef. It cooked up a bit weird — it doesn’t brown up in the same way I expect of ground beef, and it made a lot more fond than I expected. Perhaps the temperature is too high? We’ll see how it fares when it’s done. In the meanwhile, let’s get you up to speed on my free software projects.

Dependencies and maintainers

Thu, 06 Feb 2020 00:00:00 +0000

I’m 34,018 feet over the Atlantic at the moment, on my way home from FOSDEM. It was as always a lovely event, with far too many events of interest for any single person to consume. One of the few talks I was able to attend¹ left a persistent worm of thought in my brain. This talk was put on by representatives of Microsoft and GitHub and discusses whether or not there is a sustainability problem in open source (link). The content of the talk, interpreted within the framework in which it was presented, was moderately interesting. It was more fascinating to me, however, as a lens for interpreting GitHub’s (and, indirectly, Microsoft’s) approach to open source, and of the mindset of developers who approach problems in the same ways.

KnightOS was an interesting operating system

Mon, 27 Jan 2020 00:00:00 +0000

KnightOS is an operating system I started writing about 10 years ago, for Texas Instruments line of z80 calculators — the TI-73, TI-83+, TI-84+, and similar calculators are supported. It still gets the rare improvements, but these days myself and most of the major contributors are just left with starry eyed empty promises to themselves that one day they’ll do one of those big refactorings we’ve been planning… for 4 or 5 years now.

The happinesses and stresses of full-time FOSS work

Tue, 21 Jan 2020 00:00:00 +0000

In the past few days, several free software maintainers have come out to discuss the stresses of their work. Though the timing was suggestive, my article last week on the philosophy of project governance was, at best, only tangentially related to this topic - I had been working on that article for a while. I do have some thoughts that I’d like to share about what kind of stresses I’ve dealt with as a FOSS maintainer, and how I’ve managed (or often mismanaged) it.

A philosophy of project governance

Fri, 17 Jan 2020 00:00:00 +0000

I’ve been in the maintainer role for dozens of projects for a while now, and have moderated my fair share of conflicts. I’ve also been on the other side, many times, as a minor contributor watching or participating in conflict within other projects. Over the years, I’ve developed an approach to project governance which I believe is lightweight, effective, and inclusive.

I hold the following axioms to be true:

Computer projects are organized by humans, creating a social system.
Social systems are fundamentally different from computer systems.
Objective rules cannot be programmed into a social system.

And the following is true of individuals within those systems:

Status update, January 2020

Wed, 15 Jan 2020 00:00:00 +0000

I forgot to write this post this morning, and I’m on cup 3 of coffee while knee-deep in some arcane work with tarballs in Python. Forgive the brevity of this introduction. Let’s get right into the status update.

First of all, FOSDEM 2020 is taking place on February 1st and 2nd, and I’m planning on being there again this year. I hope to see you there! I’ll be hosting another small session for SourceHut and aerc users where I’ll take questions, demo some new stuff, and give out stickers.

Following up on "Hello world"

Wed, 08 Jan 2020 00:00:00 +0000

This is a follow-up to my last article, Hello world, which is easily the most negatively received article I’ve written — a remarkable feat for someone who’s written as much flame bait as me. Naturally, the fault lies with the readers.

All jokes aside, I’ll try to state my point better. The “Hello world” article was a lot of work to put together — frustrating work — by the time I had finished collecting numbers, I was exhausted and didn’t pay much mind to putting context to them. This left a lot of it open to interpretation, and a lot of those interpretations didn’t give the benefit of the doubt.

Hello world

Sat, 04 Jan 2020 00:00:00 +0000

Let’s say you ask your programming language to do the simplest possible task: print out “hello world”. Generally this takes two syscalls: write and exit. The following assembly program is the ideal Linux x86_64 program for this purpose. A perfect compiler would emit this hello world program for any language.

bits 64
section .text
global _start
_start:
	mov rdx, len
	mov rsi, msg
	mov rdi, 1
	mov rax, 1
	syscall

	mov rdi, 0
	mov rax, 60
	syscall

section .rodata
msg: db "hello world", 10
len: equ $-msg

Most languages do a whole lot of other crap other than printing out “hello world”, even if that’s all you asked for.

Managing my dotfiles as a git repository

Mon, 30 Dec 2019 00:00:00 +0000

There are many tools for managing your dotfiles - user-specific configuration files. GNU stow is an example. I’ve tried a few solutions over the years, but I settled on a very simple system several years ago which has served me very well in the time since: my $HOME is a git repository. This repository, in fact. This isn’t an original idea, but I’m not sure where I first heard it from either, and I’ve extended upon it somewhat since.

PinePhone review

Wed, 18 Dec 2019 00:00:00 +0000

tl;dr: Holy shit! This is the phone I have always wanted. I have never been this excited about the mobile sector before. However: the software side is totally absent — phone calls are very dubious, SMS is somewhat dubious, LTE requires some hacks, and everything will have to be written from the ground up.

I have a PinePhone developer edition model, which I paid for out of pocket¹ and which took an excruciatingly long time to arrive. When it finally arrived, it came with no SIM or microSD card (expected), and the eMMC had some half-assed version of Android on it which just boot looped without POSTing to anything useful². This didn’t bother me in the slightest — like any other computer I’ve purchased, I planned on immediately flashing my own OS on it. My Linux distribution of choice for it is postmarketOS, which is basically the mobile OS I’d build if I wanted to build a mobile OS.

Status update, December 2019

Sun, 15 Dec 2019 00:00:00 +0000

It’s December 15th and it still hasn’t snowed here. Why did I move to this godforsaken den of unholy heat and rain? I think I have chosen a latitude just southerly enough to deprive me of the climate I yearn for. I take some comfort in the knowledge that I’m travelling home to see the family in a couple of weeks, and sure enough Colorado has been covered in snow for some time now. Anyway, none of this is relevant to my work, which is what you came here for. Let’s take a look at this past month.

Developers shouldn't distribute their own software

Mon, 09 Dec 2019 00:00:00 +0000

An oft-heard complaint about Linux is that software distribution often takes several forms: a Windows version, a macOS version, and… a Debian version, an Ubuntu version, a Fedora version, a CentOS version, an openSUSE version… but these complaints miss the point. The true distributable form for Linux software, and rather for Unix software in general, is a .tar.gz file containing the source code.

Note: This article presumes that proprietary/nonfree software is irrelevant, and so should you.

Take action to save .org and prosecute those who sold out the internet

Fri, 29 Nov 2019 00:00:00 +0000

As many of you have no doubt heard, control of the .org registry has been sold to private interests. There have been attempts to call them to reason, like Save .ORG, but let’s be realistic: they knew what they’re doing is wrong, the whole time. If they were a commercial entity, our appeals would fall on deaf ears and that would be the end of it. But, they’re not a commercial entity - so our appeals may fall on deaf ears, but that doesn’t have to be the end of it.

Software developers should avoid traumatic changes

Tue, 26 Nov 2019 00:00:00 +0000

A lot of software has gone through changes which, in retrospect, I would describe as “traumatic” to their communities. I recognize these sorts of changes by their effect: we might have pulled through in the end, but only after a lot of heartbreak, struggle, and hours of wasted hacking; but the change left a scar on the community.

There are two common cases in which a change risks introducing this kind of trauma:

China

Wed, 20 Nov 2019 00:00:00 +0000

This article will be difficult to read and was difficult to write. I hope that you can stomach the uncomfortable nature of this topic and read my thoughts in earnest. I usually focus on technology-related content, but at the end of the day, this is my personal blog and I feel that it would betray my personal principles to remain silent. I’ve made an effort to provide citations for all of my assertions.

Status update, November 2019

Fri, 15 Nov 2019 00:00:00 +0000

Today’s update is especially exciting, because today marks the 1 year anniversary of Sourcehut opening it’s alpha to public registration. I wrote a nice long article which goes into detail about what Sourcehut accomplished in 2019, what’s to come for 2020, and it lays out the entire master plan for your consideration. Be sure to give that a look if you have the time. I haven’t slowed down on my other projects, though, so here’re some more updates!

An old-school shell hack on a line printer

Wed, 30 Oct 2019 00:00:00 +0000

It’s been too long since I last did a good hack, for no practical reason other than great hack value. In my case, these often amount to a nostalgia for an age of computing I wasn’t present for. In a recent bid to capture more of this nostalgia, I recently picked up a dot matrix line printer, specifically the Epson LX-350 printer. This one is nice because it has a USB port, so I don’t have to break out my pile of serial cable hacks to get it talking to Linux 😁

Status update, October 2019

Tue, 15 Oct 2019 00:00:00 +0000

Last month, I gave you an update at the conclusion of a long series of travels. But, I wasn’t done yet - this month, I spent a week in Montreal for XDC. Simon Ser put up a great write-up which goes over a lot of the important things we discussed there. It was a wonderful conference and well worth the trip - but I truly am sick of travelling. Now, I can enjoy some time at home, working on free and open source software.

How to fuck up software releases

Sat, 12 Oct 2019 00:00:00 +0000

I manage releases for a bunch of free & open-source software. Just about every time I ship a release, I find a novel way to fuck it up. Enough of these fuck-ups have accumulated now that I wanted to share some of my mistakes and how I (try to) prevent them from happening twice.

At first, I did everything manually. This is fine enough for stuff with simple release processes - stuff that basically amounts to tagging a commit, pushing it, and calling it a day. But even this gets tedious, and I’d often make a mistake when picking the correct version number. So, I wrote a small script: semver. semver patch bumps the patch version, semver minor bumps the minor version, and semver major bumps the major version, based on semantic versioning. I got into the habit of using this script instead of making the tags manually. The next fuckup soon presented itself: when preparing the shortlog, I would often feed it the wrong commits, and the changelog would be messed up. So, I updated the script to run the appropriate shortlog command and pre-populate the annotated tag with it, launching the editor to adjust the changelog as necessary.

RaptorCS's redemption: the POWER9 machine works

Thu, 10 Oct 2019 00:00:00 +0000

This is a follow-up to my earlier article, “RaptorCS POWER9 Blackbird PC: An expensive mistake”. Since I published that article, I’ve been in touch with Raptor and they’ve been much more communicative and helpful. I now have a working machine!

Update Feb. 2024: Seems the improvements I asked for may not have stuck. Buyer beware.

After I published my article, Raptor reached out and apologised for my experience. They offered a full refund, but I agreed to work on further diagnosis now that we had opened a dialogue¹. They identified that my CPU was defective and sent me a replacement, then we found the mainboard to be defective, too, and the whole thing was shipped back and replaced. I installed the new hardware into the datacenter today and it was quite pleasant to get up and running. Raptor assures me that my nightmares with the old board are atypical, and if the new board is representative of the usual user experience, I would have to agree. The installation was completely painless.²

Why Collabora really added Digital Restrictions Management to Weston

Mon, 07 Oct 2019 00:00:00 +0000

A recent article from Collabora, Why HDCP support in Weston is a good thing, proports to offer a lot of insight into why HDCP - a Digital Restrictions Management (DRM) related technology - was added to Weston - a well known basic Wayland compositor which was once the reference compositor for Wayland. But this article is gaslighting you. There is one reason and one reason alone that explains why HDCP support landed in Weston.

RaptorCS POWER9 Blackbird PC review

Mon, 23 Sep 2019 00:00:00 +0000

November 2018: Ordered Basic Blackbird Bundle w/32 GB RAM: $1,935.64

Update 2019-12-23: This article was originally titled “RaptorCS POWER9 Blackbird PC: An expensive mistake”. Please read the follow-up article, published 2019-10-10: RaptorCS’s redemption: the POWER9 machine works

June 2019

Order ships, and arrives without RAM. It had been long enough that I didn’t realize the order had only been partially fulfilled, so I order some RAM from the list of recommended chips ($338.40), along with the other necessities that I didn’t purchase from Raptor: a case ($97.99) and a PSU ($68.49), and grab some hard drives I have lying around. Total cost: about $2,440. Worth it to get POWER9 builds working on builds.sr.ht!

Don't sacrifice the right ideas to win the right words

Tue, 17 Sep 2019 00:00:00 +0000

There is a difference between free software and open-source software. But you have to squint to see it. Software licenses which qualify for one title but not the other are exceptionally rare.

A fascination with linguistics is common among hackers, and I encourage and participate in language hacking myself. Unfortunately, that seems to seep into the Free Software Foundation’s message a bit too much. Let’s see if any of this rings familiar:

Status update, September 2019

Sun, 15 Sep 2019 00:00:00 +0000

Finally home again after a long series of travels! I spent almost a month in Japan, then visited my sister’s new home in Hawaii on the way eastwards, then some old friends in Seattle, and finally after 5½ long weeks, it’s home sweet home here in Philadelphia. At least until I leave for XDC in Montreal 2 weeks from now. Someday I’ll have some rest… throughout all of these wild travels, I’ve been hard at work on my free software projects. Let’s get started with this month’s status update!

How I decide between many programming languages

Sun, 08 Sep 2019 00:00:00 +0000

I have a few old standards in my toolbelt that I find myself calling upon most often, but I try to learn enough about many programming languages to reason about whether or not they’re suitable to any use-case I’m thinking about. The best way is to learn by doing, so getting a general impression of the utility of many languages helps equip you with the knowledge of whether or not they’d be useful for a particular problem even if you don’t know them yet.

Building interactive SSH applications

Mon, 02 Sep 2019 00:00:00 +0000

After the announcement of shell access for builds.sr.ht jobs, a few people sent me some questions, wondering how this sort of thing is done. Writing interactive SSH applications is actually pretty easy, but it does require some knowledge of the pieces involved and a little bit of general Unix literacy.

On the server, there are three steps which you can meddle with using OpenSSH: authentication, the shell session, and the command. The shell is pretty easily manipulated. For example, if you set the user’s login shell to /usr/bin/nethack, then nethack will run when they log in. Editing this is pretty straightforward, just pop open /etc/passwd as root and set their shell to your desired binary. If the user SSHes into your server with a TTY allocated (which is done by default), then you’ll be able to run a curses application or something interactive.

Shell access for builds.sr.ht CI

Mon, 19 Aug 2019 00:00:00 +0000

Have you ever found yourself staring at a failed CI build, wondering desperately what happened? Or, have you ever needed a fresh machine on-demand to test out an idea in? Have you been working on Linux, but need to test something on OpenBSD? Starting this week, builds.sr.ht can help with all of these problems, because you can now SSH into the build environment.

If you didn't know, Sourcehut is the 100% open/libre software forge for hackers, complete with git and Mercurial hosting, CI, mailing lists, and more - with no JavaScript. Try it out!

The next time your build fails on builds.sr.ht, you’ll probably notice the following message:

Status update, August 2019

Thu, 15 Aug 2019 00:00:00 +0000

Outside my window, the morning sun can be seen rising over the land of the rising sun, as I sip from a coffee purchased at the konbini down the street. I almost forgot to order it, as the staffer behind the counter pointed out with a smile and a joke that, having been told in Japanese, mostly went over my head. It’s on this quiet Osaka morning I write today’s status update - there are lots of existing developments to share!

DRM leasing: VR for Wayland

Fri, 09 Aug 2019 00:00:00 +0000

As those who read my status updates have been aware, recently I’ve been working on bringing VR to Wayland (and vice versa). The deepest and most technical part of this work is DRM leasing (Direct Rendering Manager, not Digital Restrictions Management), and I think it’d be good to write in detail about what’s involved in this part of the effort. This work has been sponsored by Status.im, as part of an effort to build a comprehensive Wayland-driven VR workspace. When we got started, most of the plumbing was missing for VR headsets to be useful on Wayland, so this has been my focus for a while. The result of this work is summed up in this crappy handheld video:

FOSS contributor tracks

Mon, 29 Jul 2019 00:00:00 +0000

Just like many companies have different advancement tracks for their employees (for example, a management track and an engineering track), similar concepts exist in free software projects. One of the roles of a maintainer is to help contributors develop into the roles which best suit them. I’d like to explain what this means to me in my role as a maintainer of several projects, though I should mention upfront that I’m just some guy and, while I can explain what has and hasn’t worked for me, I can’t claim to have all of the answers. People are hard.

Status update, July 2019

Mon, 15 Jul 2019 00:00:00 +0000

Today I received the keys to my new apartment, which by way of not being directly in the middle of the city¹ saves me a decent chunk of money - and allows me to proudly announce that I have officially broken even on doing free software full time! I owe a great deal of thanks to all of you who have donated to support my work or purchased a paid SourceHut account. I’ve dreamed of sustainably working on free software for a long, long time, and I’m very grateful for all of your support in helping realize that dream. Now let me share with you what your money has bought over the past month!

Announcing code annotations for SourceHut

Mon, 08 Jul 2019 00:00:00 +0000

Today I’m happy to announce that code annotations are now available for SourceHut! These allow you to decorate your code with arbitrary links and markdown. The end result looks something like this:

NOTICE: Annotations were ultimately removed from sourcehut.

SourceHut is the "hacker's forge", a 100% open-source platform for hosting Git & Mercurial repos, bug trackers, mailing lists, continuous integration, and more. No JavaScript required!

The annotations shown here are sourced from a JSON file which you can generate and upload during your CI process. It looks something like this:

Absence of certain features in IRC considered a feature

Mon, 01 Jul 2019 00:00:00 +0000

The other day a friend of mine (an oper on Freenode) wanted to talk about IRC compared to its peers, such as Matrix, Slack, Discord, etc. The ensuing discussion deserves summarization here. In short: I’m glad that IRC doesn’t have the features that are “showstoppers” for people choosing other platforms, and I’m worried that attempts to bring these showstopping “features” to IRC will worsen the platform for the people who use it now.

Status update, June 2019

Sat, 15 Jun 2019 00:00:00 +0000

Summer is in full swing here in Philadelphia. Last night I got great views of Jupiter and a nearly-full Moon, and my first Saturn observation of the year. I love astronomy on clear Friday nights, there’s always plenty of people coming through the city. And today, on a relaxing lazy Saturday, waiting for friends for dinner later, I have the privilege of sharing another status report with you.

First, I want to talk about some work I’ve done with blogs lately. On the bottom of this article you’ll find a few blog posts from around the net. This is populated with openring, a small Go tool I made to fetch a few articles from a list of RSS feeds. A couple of other people have added this to their own sites as well, and I hope to use this to encourage the growth of a network of bloggers supporting each other without any nonfree or centralized software. I’ll write about this in its own article in time. I’ve also made an open offer to give $20 to anyone who wants to make their own blog, and so far 5 new blogs have taken me up on the offer. Maybe you’ll be the next?

My personal journey from MIT to GPL

Thu, 13 Jun 2019 00:00:00 +0000

As I got started writing open source software, I generally preferred the MIT license. I actually made fun of the “copyleft” GPL licenses, on the grounds that they are less free. I still hold this opinion today: the GPL license is less free than the MIT license - but today, I believe this in a good way.

If you haven’t yet, I suggest reading the MIT license - it’s very short. It satisfies the four essential freedoms guaranteed of free software:

Initial pre-release of aerc: an email client for your terminal

Mon, 03 Jun 2019 00:00:00 +0000

After years of painfully slow development, the aerc email client has seen a huge boost in its pace of development recently. This leads to today’s announcement: aerc 0.1.0 is now available! After my transition to working on free software full time allowed me to spend more time on more projects, I was able to invest considerably more time into aerc. Your support led us here: thank you to all of the people who donate to my work!

What is a fork, really, and how GitHub changed its meaning

Fri, 24 May 2019 00:00:00 +0000

The fork button on GitHub - with the little number next to it for depositing dopamine into your brain - is a bit misleading. GitHub co-opted the meaning of “fork” to trick you into participating in their platform more. They did this in a well-intentioned way, for the sake of their pull requests feature, but ultimately this design is self-serving and causes some friction when contributors venture out of their GitHub sandbox and into the rest of the software development ecosystem. Let’s clarify what “fork” really means, and what we do without GitHub’s concept of one - for it is in this difference that we truly discover how git is a distributed version control system.

Status update, May 2019

Wed, 15 May 2019 00:00:00 +0000

This month, it seems the most exciting developments again come from the realm of email. I’ve got cool email-related news to share for aerc, lists.sr.ht, and todo.sr.ht, and many cool developments in my other projects to share.

Let’s start with lists.sr.ht: I have broken ground on the web-based patch review tools! I promised these features when I started working on sourcehut, to make the email-based workflow more enticing to those who would rather work on the web. Basically, this gives us a Github or Gerrit-esque review UI for patches which arrive on the mailing list. Thanks to a cool library Simon Ser wrote for me… almost a year ago… I’m able to take a thread of emails discussing a patch and organically convert them into inline feedback on the web.

Webcast: Reviewing git & mercurial patches with email

Mon, 13 May 2019 00:00:00 +0000

With the availability of new resources like git-send-email.io, I’ve been working on making the email-based workflow more understandable and accessible to the world. One thing that’s notably missing from this tutorial, however, is the maintainer side of the work. I intend to do a full write-up in the future, but for now I thought it’d be helpful to clarify my workflow a bit with a short webcast. In this video, I narrate my workflow as I review a few sourcehut patches and participate in some dicsussions.

Calculating your donation's value following Patreon's fee changes

Mon, 06 May 2019 00:00:00 +0000

In January 2018, I wrote a blog post which included a fee calculator. Patreon changes their fee model tomorrow, and it’s time for an updated calculator. I’m grandfathered into the old fees, so not much has changed for me, but I want to equip Patreon users - creators and supporters - with more knowledge of how their money is moving through the platform.

Patreon makes money by siphoning some money off the top of a donation flow between supporters and creators. Because of the nature of its business (a private, VC-backed corporation), the siphon’s size and semantics are prone to change in undesirable ways, since VC’s expect infinite growth and a private business generally puts profit first. For this reason, I diversify my income, so that when Patreon makes these changes it limits their impact on my financial well-being. Even so, Patreon is the biggest of my donation platforms, representing over $500/month at the time of writing (full breakdown here)¹.

Announcing Wio: A clone of Plan 9's Rio for Wayland

Wed, 01 May 2019 00:00:00 +0000

For a few hours here and there over the past few months, I’ve been working on a side project: Wio. I’ll just let the (3 minute) screencast do the talking first:

Note: this video begins with several seconds of grey video. This is normal.

In short, Wio is a Wayland compositor based on wlroots which has a similar look and feel to Plan 9’s Rio desktop. It works by running each application in its own nested Wayland compositor, based on Cage - yet another wlroots-based Wayland compositor. I used Cage in last week’s RDP article, but here’s another cool use-case for it.

The "shut up and get back to work" coding style guide

Mon, 29 Apr 2019 00:00:00 +0000

So you’re starting a new website, and you open the first CSS file. What style do you use? Well, you hate indenting with spaces passionately. You know tabs are right because they’re literally made for this, and they’re only one byte, and these god damn spaces people with their bloody spacebars…

Shut up and use spaces. That’s how CSS is written¹. And you, mister web programmer, coming out of your shell and dipping your toes into the world of Real Programming, writing your first Golang program: use tabs, jerk. There’s only one principle that matters in coding style: don’t rock the boat. Just do whatever the most common thing is in the language you’re working in. Write your commit messages the same way as everyone else, too. Then shut up and get back to work. This hill isn’t worth dying on.

Using Cage for a seamless remote Wayland session

Tue, 23 Apr 2019 00:00:00 +0000

Congratulations to Jente Hidskes on the first release of Cage! Cage is a Wayland compositor designed for kiosks - though, as you’ll shortly find out, is useful in many unexpected ways. It launches a single application, in fullscreen, and exits the compositor when that application exits. This lets you basically add a DRM+KMS+libinput session to any Wayland-compatible application (or X application via XWayland) and run it in a tiny wlroots compositor.

Choosing a VPN service is a serious decision

Fri, 19 Apr 2019 00:00:00 +0000

There’s a disturbing trend in the past year or so of various VPN companies advertising to the general, non-technical public. It’s great that the general public is starting to become more aware of their privacy online, but I’m not a fan of these companies exploiting public paranoia to peddle their wares. Using a VPN in the first place has potentially grave consequences for your privacy - and can often be worse than not using one in the first place.

Announcing first-class Mercurial support on Sourcehut

Mon, 15 Apr 2019 00:00:00 +0000

I’m pleased to announce that the final pieces have fallen into place for Mercurial support on SourceHut, which is now on-par with our git offering. Special thanks are owed to SourceHut contributor Ludovic Chabant, who has been instrumental in adding Mercurial support to SourceHut. You may have heard about it while this was still experimental - but I’m happy to tell you that we have now completely integrated Mercurial support into SourceHut! Want to try it out? Check out the tutorial.

Status update, April 2019

Mon, 15 Apr 2019 00:00:00 +0000

Spring is here, and I’m already miserable in the heat. Crazy weather here in Philadelphia - I was woken up at 3 AM by my phone buzzing, telling me to take immediate shelter from a tornado. But with my A/C cranked up and the tornado safely passed, I’ve been able to get a lot of work done.

The project with the most impressive progress is aerc2. It can now read emails, including filtering them through arbitrary commands for highlighting diffs or coloring quotes, or even rendering HTML email with a TUI browser like w3m.

NewPipe represents the best of FOSS

Tue, 02 Apr 2019 00:00:00 +0000

NewPipe is a free and open-source Android application for browsing & watching YouTube. In my opinion, NewPipe is a perfect case-study in why free & open source software is great and how our values differ from proprietary software in important ways. There’s one simple reason: it’s better than the proprietary YouTube app, in every conceivable way, for free.

NewPipe is better because it’s user-centric software. It exists to make its users lives better, not to enrich its overseers. Because of this, NewPipe has many features which are deliberately omitted from the proprietary app, such as:

Rust is not a good C replacement

Mon, 25 Mar 2019 00:00:00 +0000

I have a saying that summarizes my opinion of Rust compared to Go: “Go is the result of C programmers designing a new programming language, and Rust is the result of C++ programmers designing a new programming language”. This isn’t just a metaphor - Go was designed by plan9 alumni, an operating system written in C and the source of inspiration for many of Go’s features, and Rust was designed by the folks at Mozilla - whose flagship product is one of the largest C++ codebases in the world.

Status update, March 2019

Fri, 15 Mar 2019 00:00:00 +0000

My todo list is getting completed at a pace it’s never seen before, and growing at a new pace, too. This full-time FOSS gig is really killing it! As the good weather finally starts to roll in, it’s time for March’s status update. Note: I posted updates on Patreon before, but will start posting here instead. This medium doesn’t depend on a proprietary service, allows for richer content, and is useful for my supporters who support my work via other donation platforms.

Announcing the release of sway 1.0

Mon, 11 Mar 2019 00:00:00 +0000

1,315 days after I started the sway project, it’s finally time for sway 1.0! I had no idea at the time how much work I was in for, or how many talented people would join and support the project with me. In order to complete this project, we have had to rewrite the entire Linux desktop nearly from scratch. Nearly 300 people worked together, together writing over 9,000 commits and almost 100,000 lines of code, to bring you this release.

Sourcehut's spartan approach to web design

Mon, 04 Mar 2019 00:00:00 +0000

Sourcehut is known for its brutalist design, with its mostly shades-of-gray appearance, conservative color usage, and minimal distractions throughout. This article aims to share some insights into the philosophy that guides this design, both for the curious reader and for the new contributor to the open-source project.

The most important principle is that sr.ht is an engineering tool first and foremost, and when you’re there it’s probably because you’re in engineering mode. Therefore, it’s important to bring the information you’re there for to the forefront, and minimize distractions. In practice, this means that the first thing on any page to grab your attention should be the thing that brought you there. Consider the source file view on git.sr.ht. For reference, here are similar pages on GitHub and Gitlab.

Tips for a disciplined git workflow

Mon, 25 Feb 2019 00:00:00 +0000

Basic git usage involves typing a few stock commands to “sync everyone up”. Many people who are frustrated with git become so because they never progress beyond this surface-level understanding of how it works. However, mastering git is easily worth your time. How much of your day is spent using git? I would guess that there are many tools in your belt that you use half as often and have spent twice the time studying.

Generics aren't ready for Go

Mon, 18 Feb 2019 00:00:00 +0000

In the distance, a gradual roar begins to grow in volume. A dust cloud is visible over the horizon. As it nears, the shouts of the oncoming angry mob can be heard. Suddenly, it stops, and a brief silence ensues. Then the air is filled with the clackings of hundreds of keyboards, angrily typing the owner’s opinion about generics and Go. The clans of Java, C#, Rust, C++, TypeScript, Haskell, and more - usually mortal enemies - have combined forces to fight in what may become one of the greatest flamewars of our time. And none of them read more than the title of this article before writing their comment.

Wayland misconceptions debunked

Sun, 10 Feb 2019 00:00:00 +0000

This article has been on my backburner for a while, but it seems Wayland FUD is making the news again recently, so I’ve bumped up the priority a bit. For those new to my blog, I am the maintainer of wlroots, a library which implements much of the functionality required of a Wayland compositor and is arguably the single most influential project in Wayland right now; and sway, a popular Wayland compositor which is nearing version 1.0. Let’s go over some of the common misconceptions I hear about Wayland and why they’re wrong. Feel free to pick and choose the misconceptions you believe to read and disregard the rest.

My experiences at FOSDEM 2019

Tue, 05 Feb 2019 00:00:00 +0000

Currently in a plane on my way home from FOSDEM and, as seems to be a recurring pattern when I fly long distances home after attending a conference, a recap is readily flowing from my fingertips. This was my first year at FOSDEM, and I’m glad that I came. I’m already excited for next year! It was also my first year volunteering, which was equally great and another thing I expect to repeat.

Why I chose Flask to build sr.ht's mini-services

Wed, 30 Jan 2019 00:00:00 +0000

sr.ht is a large, production-scale suite of web applications (I call them “mini-services”, as they strike a balance between microservices and monolithic applications) which are built in Python with Flask. David Lord, one of the maintainers of Flask, reached out to me when he heard about sr.ht and saw that it was built with Flask. At his urging, I’d like to share the rationale behind the decision and how it’s turned out in the long run.

Why I use old hardware

Wed, 23 Jan 2019 00:00:00 +0000

Recently I was making sure my main laptop is ready for travel¹, which mostly just entails syncing up the latest version of my music collection. This laptop is a Thinkpad X200, which turns 11 years old in July and is my main workstation away from home (though I bring a second monitor and an external keyboard for long trips). This laptop is a great piece of hardware. 100% of the hardware is supported by the upstream Linux kernel, including the usual offenders like WiFi and Bluetooth. Niche operating systems like 9front and Minix work great, too. Even coreboot works! It’s durable, user-serviceable, light, and still looks brand new after all of these years. I love all of these things, but there’s no denying that it’s 11 years behind on performance innovations.

I'm going to work full-time on free software

Tue, 15 Jan 2019 00:00:00 +0000

Sorry for posting two articles so close to each other - but this is important! As I’m certain many of you know, I maintain a large collection of free software projects, including sway, wlroots, sr.ht, scdoc, aerc, and many, many more. I contribute to more still, working on projects like Alpine Linux, mrsh, musl libc, and anything else I can. Until now, I’ve been working on these in my spare time, but just under a year ago I wrote “The path to sustainably working on FOSS full-time” laying out my future plans. Today I’m proud to tell you that, thanks to everyone’s support, I’ll be working on free software full-time starting in February.

Backups & redundancy at sr.ht

Sun, 13 Jan 2019 00:00:00 +0000

sr.ht¹ is 100% open source and I encourage people to install it on their own infrastructure, especially if they’ll be sending patches upstream. However, I am equally thrilled to host sr.ht for you on the “official” instance, and most users find this useful because the maintenance burden is non-trivial. Today I’ll give you an idea of what your subscription fee pays for. In this first post on ops at sr.ht, I’ll talk about backups and redundancy. In future posts, I’ll talk about security, high availability, automation, and more.

Patches welcome

Tue, 01 Jan 2019 00:00:00 +0000

Happy new year! This is always a weird “holiday” for me, since all of the fun happened last night. Today is just kind of… I guess a chance for everyone to sober up before work tomorrow? It does tend to invite a sense of reflection and is the ideal time to plan for the year ahead. One of my goals in 2019 is to change more people’s thinking about the open source community and what it means to count among their number.

Anatomy of a shell

Fri, 28 Dec 2018 00:00:00 +0000

I’ve been contributing where I can to Simon Ser’s mrsh project, a work-in-progress strictly POSIX shell implementation. I worked on some small mrsh features during my holiday travels and it’s in the forefront of my mind, so I’d like to share some of its design details with you.

There are two main components to a shell: parsing and execution. mrsh uses a simple recursive descent parser to generate an AST (Abstract Syntax Tree, or an in-memory model of the structure of the parsed source). This design was chosen to simplify the code and avoid dependencies like flex/bison, and is a good choice given that performance isn’t critical for parsing shell scripts. Here’s an example of the input source and output AST:

Porting Alpine Linux to RISC-V

Thu, 20 Dec 2018 00:00:00 +0000

I recently received my HiFive Unleashed, after several excruciating months of waiting, and it’s incredibly cool. For those unaware, the HiFive Unleashed is the first consumer-facing Linux-capable RISC-V hardware. For anyone who’s still lost, RISC-V is an open, royalty-free instruction set architecture, and the HiFive is an open CPU implementing it. And here it is on my dining room table:

This board is cool. I’m working on making this hardware available to builds.sr.ht users in the next few months, where I intend to use it to automate the remainder of the Alpine Linux port and make it available to any other operating systems (including non-Linux) and userspace software which are interested in working on a RISC-V port. I’m fairly certain that this will be the first time hardware-backed RISC-V cycles are being made available to the public.

How to abandon a FLOSS project

Tue, 04 Dec 2018 00:00:00 +0000

It’s no secret that maintaining free and open source software is often a burdensome and thankless job. I empathise with maintainers who lost interest in a project, became demotivated by the endless demands of users, or are no longer blessed with enough free time. Whatever the reason, FLOSS work is volunteer work, and you’re free to stop volunteering at any time.

In my opinion, there are two good ways to abandon a project: the fork it option and the hand-off option. The former is faster and easier, and you can pick this if you want to wash your hands of the project ASAP, but has a larger effect on the community. The latter is not always possible, requires more work on your part, and takes longer, but it has a minimal impact on the community.

sr.ht, the hacker's forge, now open for public alpha

Thu, 15 Nov 2018 00:00:00 +0000

I’m happy to announce today that I’m opening sr.ht (pronounced “sir hat”, or any other way you want) to the general public for the remainder of the alpha period. Though it’s missing some of the features which will be available when it’s completed, sr.ht today represents a very capable software forge which is already serving the needs of many projects in the free & open source software community. If you’re familiar with the project and ready to register your account, you can head straight to the sign up page.

It's not okay to pretend your software is open source

Tue, 30 Oct 2018 00:00:00 +0000

Unfortunately, I find myself writing about the Commons Clause again. For those not in the know, the Commons Clause is an addendum designed to be added to free software licenses. The restrictions it imposes (you cannot sell the software) makes the resulting franken-license nonfree. I’m not going to link to the project which brought this subject back into the discussion - they don’t deserve the referral - but the continued proliferation of software using the Commons Clause gives me reason to speak out against it some more.

How does virtual memory work?

Mon, 29 Oct 2018 00:00:00 +0000

Virtual memory is an essential part of your computer, and has been for several decades. In my earlier article on pointers, I compared memory to a giant array of octets (bytes), and explained some of the abstractions we make on top of that. In actual fact, memory is more complicated than a flat array of bytes, and in this article I’ll explain how.

An astute reader of my earlier article may have considered that pointers on, say, an x86_64 system, are 64 bits long¹. With this, we can address up to 18,446,744,073,709,551,616 bytes (16 exbibytes²) of memory. I only have 16 GiB of RAM on this computer, so what gives? What’s the rest of the address space for? The answer: all kinds of things! Only a small subset of your address space is mapped to physical RAM. A system on your computer called the MMU, or Memory Management Unit, is responsible for managing the abstraction that enables this and other uses of your address space. This abstraction is called virtual memory.

Sway 1.0-beta.1 release highlights

Sat, 20 Oct 2018 00:00:00 +0000

1,173 days ago, I wrote sway’s initial commit, and 8,269 commits followed¹, written by hundreds of contributors. What started as a side project became the most fully featured and stable Wayland desktop available, and drove the development of what has become the dominant solution for building Wayland compositors - wlroots, now the basis of 10 Wayland compositors.

Sway 1.0-beta.1 was just released and is 100% compatible with the i3 X11 window manager. It’s faster, prettier, sips your battery, and supports Wayland clients. When we started, I honestly didn’t think we’d get here. When I decided we’d rewrite our internals and build wlroots over a year ago, I didn’t think we’d get here. It’s only thanks to an amazing team of talented contributors that we did. So what can users expect from this release? The difference between sway 0.15 and sway 1.0 is like night and day. The annoying bugs which plauged sway 0.15 are gone, and in their place is a rock solid Wayland compositor with loads of features you’ve been asking after for years. The official release notes are a bit thick, so let me give you a guided tour.

Go 1.11 got me to stop ignoring Go

Mon, 08 Oct 2018 00:00:00 +0000

I took a few looks at Go over the years, starting who knows when. My first serious attempt to sit down and learn some damn Go was in 2014, when I set a new personal best at almost 200 lines of code before I got sick of it. I kept returning to Go because I could see how much potential it had, but every time I was turned off for the same reason: GOPATH.

Don't sign a CLA

Fri, 05 Oct 2018 00:00:00 +0000

A large minority of open-source projects come with a CLA, or Contributor License Agreement, and require you to sign one before they’ll merge your patch. These agreements typically ask you to go above and beyond the rights you afford the project by contributing under the license the software is distributed with. And you should never sign one.

Free and open source software licenses grant explicit freedoms to three groups: the maintainers, the users, and the contributors. An important freedom is the freedom to make changes to the software and to distribute these changes to the public. The natural place to do so is by contributing to the upstream project, something a project should be thankful for. A CLA replaces this gratitude with an attempt to weaken these freedoms in a manner which may stand up to the letter of the license, but is far from the spirit.

Sway & wlroots at XDC 2018

Sun, 30 Sep 2018 00:00:00 +0000

Just got my first full night of sleep after the return flight from Spain after attending XDC 2018. It was a lot of fun! I attended along with four other major wlroots contributors. Joining me were Simon Ser (emersion) (a volunteer) and Scott Anderson (ascent12) of Collabora, who work on both wlroots and sway. ongy works on wlroots, hsroots, and waymonad, and joined us on behalf of IGEL. Finally, we were joined by Guido Günther (agx) of Purism, who works with us on wlroots and on the Librem 5. This was my first time meeting most of them face-to-face!

Getting started with qemu

Mon, 10 Sep 2018 00:00:00 +0000

I often get asked questions about using my software, particularly sway, on hypervisors like VirtualBox and VMWare, as well as for general advice on which hypervisor to choose. My answer is always the same: qemu. There’s no excuse to use anything other than qemu, in my books. But I can admit that it might be a bit obtuse to understand at first. qemu’s greatest strength is also its greatest weakness: it has so many options that it’s hard to know which ones you need just to get started.

Conservative web development

Tue, 04 Sep 2018 00:00:00 +0000

Today I turned off my ad blocker, enabled JavaScript, opened my network monitor, and clicked the first link on Hacker News - a New York Times article. It started by downloading a megabyte of data as it rendered the page over the course of eight full seconds. The page opens with an advertisement 281 pixels tall, placed before even the title of the article. As I scrolled down, more and more requests were made, downloading a total of 2.8 MB of data with 748 HTTP requests. An article was weaved between a grand total of 1419 vertical pixels of ad space, greater than the vertical resolution of my display. Another 153-pixel ad is shown at the bottom, after the article. Four of the ads were identical.

How to make a self-hosted video livestream

Sun, 26 Aug 2018 00:00:00 +0000

I have seen some articles in the past which explain how to build the ecosystem around your video streaming, such as live chat and forums, but which leave the actual video streaming to Twitch.tv. I made a note the last time I saw one of these articles to write one of my own explaining the video bit. As is often the case with video, we’ll be using the excellent ffmpeg tool for this. If it’s A/V-related, ffmpeg can probably do it.

The Commons Clause will destroy open source

Wed, 22 Aug 2018 00:00:00 +0000

An alarmist title, I know, but it’s true. If the Commons clause were to be adopted by all open source projects, they would cease to be open source¹, and therefore the Commons clause is trying to destroy open source. When this first appeared I spoke out about it in discussion threads around the net, but didn’t think anyone would take it seriously. Well, yesterday, some parts of Redis became proprietary software.

The Commons Clause promoted by Kevin Wang presents one of the greatest existential threats to open source I’ve ever seen. It preys on a vulnerability open source maintainers all suffer from, and one I can strongly relate to. It sucks to not be able to make money from your open source work. It really sucks when companies are using your work to make money for themselves. If a solution presents itself, it’s tempting to jump at it. But the Commons Clause doesn’t present a solution for supporting open source software. It presents a framework for turning open source software into proprietary software.

I don't trust Signal

Wed, 08 Aug 2018 00:00:00 +0000

Occasionally when Signal is in the press and getting a lot of favorable discussion, I feel the need to step into various forums, IRC channels, and so on, and explain why I don’t trust Signal. Let’s do a blog post instead.

Off the bat, let me explain that I expect a tool which claims to be secure to actually be secure. I don’t view “but that makes it harder for the average person” as an acceptable excuse. If Edward Snowden and Bruce Schneier are going to spout the virtues of the app, I expect it to actually be secure when it matters - when vulnerable people using it to encrypt sensitive communications are targeted by smart and powerful adversaries.

Setting up a local dev mail server

Sun, 05 Aug 2018 00:00:00 +0000

As part of my work on lists.sr.ht, it was necessary for me to configure a self-contained mail system on localhost that I could test with. I hope that others will go through a similar process in the future when they set up the code for hacking on locally or when working on other email related software, so here’s a guide on how you can set it up.

There are lots of things you can set up on a mail server, like virtual mail accounts backed by a relational database, IMAP access, spam filtering, and so on. We’re not going to do any of that in this article - we’re just interested in something we can test our email code with. To start, install your distribution of postfix and pop open that /etc/postfix/main.cf file.

Writing a Wayland compositor with wlroots: shells

Sun, 29 Jul 2018 00:00:00 +0000

I apologise for not writing about wlroots more frequently. I don’t really enjoy working on the McWayface codebase this series of blog posts was originally about, so we’re just going to dismiss that and talk about the various pieces of a Wayland compositor in a more free-form style. I hope you still find it useful!

Today, we’re going to talk about shells. But to make sure we’re on the same page first, a quick refresher on surfaces. A basic primitive of the Wayland protocol is the concept of a “surface”. A surface is a rectangular box of pixels sent from the client to the compositor to display on-screen. A surface can source its pixels from a number of places, including raw pixel data in memory, or opaque handles to GPU resources that can be rendered without copying pixels on the CPU. These surfaces can also evolve over time, using “damage” to indicate which parts have changed to reduce the workload of the compositor when re-rendering them. However, making a surface and filling it with pixels is not enough to get the compositor to show them.

Git is already federated & decentralized

Mon, 23 Jul 2018 00:00:00 +0000

There have always been murmurs about “replacing GitHub with something decentralized!”, but in the wake of the Microsoft acquisition these murmurs have become conversations. In particular, this blog post is a direct response to forge-net (formerly known as GitPub). They want to federate and decentralize git using ActivityPub, the same technology leveraged by Mastodon and PeerTube. But get this: git is already federated and decentralized!

I already spoke at length about how a large minority of the git community uses email for collaboration in my previous article on the subject. Definitely give it a read if you haven’t already. In this article I want to focus on comparing this model with the possibilities afforded by ActivityPub and provide direction for new forge¹ projects to work towards embracing and improving git’s email-based collaboration tools.

Input handling in wlroots

Tue, 17 Jul 2018 00:00:00 +0000

I’ve said before that wlroots is a “batteries not included” kind of library, and one of the places where that is most apparent is with our approach to input handling. We implemented a very hands-off design for input, in order to support many use-cases: desktop input, phones with and without USB-OTG HIDs plugged in, multiple mice bound to a single cursor, multiple keyboards per seat, simulated input from fake input devices, on-screen keyboards, input which is processed by the compositor but not sent to clients… we support all of these use-cases and even more. However, the drawback of our powerful design is that it’s confusing. Very confusing.

Simple, correct, fast: in that order

Mon, 09 Jul 2018 00:00:00 +0000

The single most important quality in a piece of software is simplicity. It’s more important than doing the task you set out to achieve. It’s more important than performance. The reason is straightforward: if your solution is not simple, it will not be correct or fast.

Given enough time, you’ll find that all software which solves sufficiently complex problems is going to (1) have bugs and (2) have performance problems. Software with bugs is incorrect. Software with performance problems is not fast. We will face this fact as surely as we will face death and taxes, and we should prepare ourselves accordingly. Let’s consider correctness first.

The advantages of an email-driven git workflow

Mon, 02 Jul 2018 00:00:00 +0000

git 2.18.0 has been released, and with it my first contribution to git has shipped! My patch was for a git feature which remains disappointingly obscure: git send-email. I want to introduce my readers to this feature and speak to the benefits of using an email-driven git workflow - the workflow git was originally designed for.

Email isn’t as sexy as GitHub (and its imitators), but it has several advantages over the latter. Email is standardized, federated, well-understood, and venerable. A very large body of email-related software exists and is equally reliable and well-understood. You can interact with email using only open source software and customize your workflow at every level of the stack - filtering, organizing, forwarding, replying, and so on; in any manner you choose.

A quick review of my Let's Encrypt setup

Wed, 27 Jun 2018 00:00:00 +0000

Let’s Encrypt makes TLS much easier for pretty much everyone, but can still be annoying to use. It took me a while to smooth over the cracks in my Let’s Encrypt configuration across my (large) fleet of different TLS-enabled services. I wanted to take a quick moment to share setup with you.

2020-01-02 update: acme-client is unmaintained and caught the BSD disease anyway. I use uacme and my current procedure is documented on my new server checklist. It might not be exactly applicable to your circumstances, YMMV.

Should you move from GitHub to sr.ht

Tue, 05 Jun 2018 00:00:00 +0000

I’m not terribly concerned about Microsoft’s acquisition of GitHub, but I don’t fault those who are worried. I’ve been working on my alternative platform, sr.ht, for quite a while. I’m not about to leave GitHub because of Microsoft alone. I do have some political disagreements with GitHub and Microsoft, but those are also not the main reason that I’m building sr.ht. I simply think I can do it better. If my approach aligns with your needs, then sr.ht may be the platform for you.

How I maintain FOSS projects

Fri, 01 Jun 2018 00:00:00 +0000

Today’s is another blog post which has been on my to-write list for a while. I have hesitated a bit to write about this, because I’m certain that my approach isn’t perfect. I think it’s pretty good, though, and people who work with me in FOSS agreed after a quick survey. So! Let’s at least put it out there and discuss it.

There are a few central principles I use to guide my maintainership work:

Embedding files in C programs with koio

Tue, 29 May 2018 00:00:00 +0000

Quick blog post today to introduce a new tool I wrote: koio. This is a small tool which takes a list of files and embeds them in a C file. A library provides an fopen shim which checks the list of embedded files before resorting to the real filesystem.

I made this tool for chopsui, where I eventually want to be able to bundle up sui markup, stylesheets, images, and so on in a statically linked chopsui program. Many projects have small tools which serve a similar purpose, but it was simple enough and useful enough that I chose to make something generic so it could be used on several projects.

Why did we replace wlc?

Sun, 27 May 2018 00:00:00 +0000

For a little over a year, I’ve been working with a bunch of talented C developers to build a replacement for the wlc library. The result is wlroots, and we’re still working on completing it and updating our software to use it. The conventional wisdom suggests that rewriting your code from scratch is almost never the right idea. So why did we do it, and how is it working out? I have spoken a little about this in the past, but we’ll answer this question in detail today.

Introducing scdoc, a man page generator

Sun, 13 May 2018 00:00:00 +0000

A man page generator is one of those tools that I’ve said I would write for a long time, being displeased with most of the other options. For a while I used asciidoc, but was never fond of it. There are a few things I want to see in a man page generator:

A syntax which is easy to read and write
Small and with minimal dependencies
Designed with man pages as a first-class target

All of the existing tools failed some of these criteria. asciidoc hits #1, but fails #2 and #3 by being written in XSLT+Python and targetting man pages as a second-class citizen. mdocml fails #1 (it’s not much better than writing raw roff), and to a lesser extent also fails criteria #2¹. Another option, ronn meets criteria #1 and #3, but it’s written in Ruby and fails #2. All of these are fine for the niches they fill, but not what I’m looking for. And as for GNU info… ugh.

Redirecting stderr of a running process

Fri, 04 May 2018 00:00:00 +0000

During the KDE sprint in Berlin, Roman Gilg leaned over to me and asked if I knew how to redirect the stderr of an already-running process to a file. I Googled it and found underwhelming answers using strace and trying to decipher the output by reading the write syscalls. Instead, I thought a gdb based approach would work better, and after putting the pieces together Roman insisted I wrote a blog post on the topic.

Google embraces, extends, and extinguishes

Thu, 03 May 2018 00:00:00 +0000

Microsoft infamously coined the euphemism “embrace, extend, extinguish” to describe their strategy for disrupting markets dominated by open standards. These days, Microsoft seems to have turned the other leaf, contributing to a huge amount of open source and supporting open standards, and is becoming a good citizen of the technology community. It’s time to turn our concerns to Google.

Google famously “embraced” email on April Fool’s day, 2004, which is of course based on an open standard and federates with the rest of the world. If you’ve read the news lately, you might have seen that Google is shipping a big update to GMail soon, which adds “self-destructing” emails that vanish from the recipient’s inbox after a time. Leaving aside that this promise is impossible to deliver, look at the implementation - Google emails a link to a webpage with the actual email content, and does magic in their client to make it look seamless. Thus, they “extend” email. The “extinguish” with GMail is also well underway - it’s infamous for having an extremely strict spam filter for incoming emails from people who run personal or niche mail servers.

Sway reporting in from KDE's Berlin development sprint

Sat, 28 Apr 2018 00:00:00 +0000

I’m writing to you from an airplane on my way back to Philadelphia, after spending a week in Berlin working with the KDE team. It was great to meet those folks and work with them for a while. It’ll take me some time to get the taste of C++ out of my mouth, though! In all seriousness, it was a very productive week and I feel like we have learned a lot about each other’s projects and have a strengthened interest in collaborating more in the future.

Achtung! Decentralize, decentralize, decentralize!

Sat, 24 Mar 2018 00:00:00 +0000

I can hardly believe it, but the media is finally putting Facebook’s feet to the fire! No longer is it just the weird paranoid kids shouting at everyone to stop giving all of their information to these companies. We need to take this bull by the horns and drive it in a productive direction, and for that reason, it’s time to talk about decentralization, federation, and open source.

This article has been translated into Russian by Get Colorings.

Hack everything without fear

Sat, 17 Mar 2018 00:00:00 +0000

We live in a golden age of open source, and it can sometimes be easy to forget the privileges that this affords us. I’m writing this article with vim, in a terminal emulator called urxvt, listening to music with mpv, in a Sway desktop session, on the Linux kernel. Supporting this are libraries like glibc or musl, harfbuzz, and mesa. I also have the support of the AMDGPU video driver, libinput and udev, alsa and pulseaudio.

How to write an IRC bot

Sat, 10 Mar 2018 00:00:00 +0000

My disdain for Slack and many other Silicon Valley chat clients is well known, as is my undying love for IRC. With Slack making the news lately after their recent decision to disable the IRC and XMPP gateways in a classic Embrace Extend Extinguish move, they’ve been on my mind and I feel like writing about them more. Let’s compare writing a bot for Slack with writing an IRC bot.

First of all, let’s summarize the process for making a Slack bot. Full details are available in their documentation. The basic steps are:

Writing a Wayland Compositor, Part 3: Rendering a window

Wed, 28 Feb 2018 00:00:00 +0000

This is the third in a series of articles on the subject of writing a Wayland compositor from scratch using wlroots. Check out the first article if you haven’t already. We left off with a Wayland server which accepts client connections and exposes a handful of globals, but does not do anything particularly interesting yet. Our goal today is to do something interesting - render a window!

The commit that this article dissects is 342b7b6.

The path to sustainably working on FOSS full-time

Sat, 24 Feb 2018 00:00:00 +0000

This is an article I didn’t think I’d be writing any time soon. I’ve aspired to work full-time on my free and open source software projects for a long time now, but I have never expected that it could work. However, as of this week, I finally have enough recurring donation revenue to break even on FOSS, and I’ve started to put the extra cash away. I needed to set the next donation goal and ran the numbers to figure out what it takes to work on FOSS full-time.

Writing a Wayland Compositor, Part 2: Rigging up the server

Thu, 22 Feb 2018 00:00:00 +0000

This is the second in a series of articles on the subject of writing a Wayland compositor from scratch using wlroots. Check out the first article if you haven’t already. Last time, we ended up with an application which fired up a wlroots backend, enumerated output devices, and drew some pretty colors on the screen. Today, we’re going to start accepting Wayland client connections, though we aren’t going to be doing much with them yet.

Writing a Wayland Compositor, Part 1: Hello wlroots

Sat, 17 Feb 2018 00:00:00 +0000

This is the first in a series of many articles I’m writing on the subject of building a functional Wayland compositor from scratch. As you may know, I am the lead maintainer of sway, a reasonably popular Wayland compositor. Along with many other talented developers, we’ve been working on wlroots over the past few months. This is a powerful tool for creating new Wayland compositors, but it is very dense and difficult to understand. Do not despair! The intention of these articles is to make you understand and feel comfortable using it.

The last years

Tue, 13 Feb 2018 00:00:00 +0000

August 14th, 2019 PYONGYANG IN CHAOS AS PANDEMIC DECIMATES LEADERSHIP. Sources within the country have reported that a fast-acting and deadly infectious disease has suddenly infected the population of Pyongyang, the capital city of North Korea, where most of the country’s political elite live. Unconfirmed reports suggest that a significant fraction of the leadership has been affected.

The reclusive country has appealed for immediate aid from the international community and it is reported that a group of medical experts from Seoul have been permitted to enter via the Joint Security Area. Representatives from the United States Center for Disease Control and the Chinese Center for Disease Control and Prevention have also agreed to send representatives into the country to help control the outbreak.

Introduction to POSIX shell

Mon, 05 Feb 2018 00:00:00 +0000

What the heck is the POSIX shell anyway? Well, the POSIX (the Portable Operating System Interface) shell is the standard Unix shell - standard meaning it was formally defined and shipped in a published standard. This makes shell scripts written for it portable, something no other shell can lay claim to. The POSIX shell is basically a formalized version of the venerable Bourne shell, and on your system it lives at /bin/sh, unless you’re one of the unlucky masses for whom this is a symlink to bash.

Sway and client side decorations

Sat, 27 Jan 2018 00:00:00 +0000

You may have recently seen an article from GNOME on the subject of client side decorations (CSD) titled Introducing the CSD Initiative. It states some invalid assumptions which I want to clarify, and I want to tell you Sway’s stance on the subject. I also speak for the rest of the projects involved in wlroots on this matter, including Way Cooler, waymonad, and bspwc.

The subject of which party is responsible for window decorations on Wayland (the client or the server) has been a subject of much debate. I want to clarify that though GNOME may imply that a consensus has been reached, this is not the case. CSD have real problems that have long been waved away by its supporters:

Fee breakdown for various donation platforms

Tue, 16 Jan 2018 00:00:00 +0000

Understanding fees are a really confusing part of supporting creators of things you like. I provide a few ways for people to support my work, and my supporters can struggle to understand the differences between them. It comes down to fees, of which there are several kinds (note: I just made these terms up):

Transaction fees are charged by the payment processor (the company that takes down your card number and runs the transaction with your bank). These are typically in the form of a percentage of the transaction plus a few cents.
Platform fees are charged by the platform (e.g. Patreon) to run their operation, typically in the form of a fixed percentage of the transaction.
Withdrawal fees are charged to move money from the platform to the creator’s bank account. These vary depending on the withdrawal processor.
Taxes are also implicated, depending on how much the creator makes.

All of this adds up to a very confusing picture. I’ve made a calculator to help you sort it out.

Learn about your package manager

Wed, 10 Jan 2018 00:00:00 +0000

Tools like virtualenv, rbenv, and to a lesser extent npm and pip, are occasionally useful in development but encourage bad practices in production. Many people forget that their distro already has a package manager! And there’s more– you, the user, can write packages for it!

Your distro’s package repositories probably already have a lot of your dependencies, and can conveniently update your software alongside the rest of your system. On the whole you can expect your distro packages to be much better citizens on your system than a language-specific package manager will be. Additionally, pretty much all distros provide a means for you to host your own package repositories, from which you can install and update any packages you choose to make.

fork is not my favorite syscall

Tue, 02 Jan 2018 00:00:00 +0000

This article has been on my to-write list for a while now. In my opinion, fork is one of the most questionable design choices of Unix. I don’t understand the circumstances that led to its creation, and I grieve over the legacy rationale that keeps it alive to this day.

Let’s set the scene. It’s 1971 and you’re a fly on the wall in Bell Labs, watching the first edition of Unix being designed for the PDP-11/20. This machine has a 16-bit address space with no more than 248 kilobytes of memory. They’re discussing how they’re going to support programs that spawn new programs, and someone has a brilliant idea. “What if we copied the entire address space of the program into a new process running from the same spot, then let them overwrite themselves with the new program?” This got a rousing laugh out of everyone present, then they moved on to a better design which would become immortalized in the most popular and influential operating system of all time.

wlroots whitepaper available

Thu, 28 Dec 2017 00:00:00 +0000

View PDF

Firefox is on a slippery slope

Sat, 16 Dec 2017 00:00:00 +0000

For a long time, it was just setting the default search provider to Google in exchange for a beefy stipend. Later, paid links in your new tab page were added. Then, a proprietary service, Pocket, was bundled into the browser - not as an addon, but a hardcoded feature. In the past few days, we’ve discovered an advertisement in the form of browser extension was sideloaded into user browsers. Whoever is leading these decisions at Mozilla needs to be stopped.

A history of emergent intelligence

Sat, 02 Dec 2017 00:00:00 +0000

As you all know, the simulation of universe 2813/9301 is now coming to a close. This simulation is notable for being the first simulated universe suitable for hosting intelligent life, but yesterday the simulation reached a state where we believe no additional intelligences will emerge. It seems the final state of this set of physical laws is a dark and empty universe of slowly evaporating black holes. Though, given the historical significance of this simulation, it’s unlikely we we’ll be turning it off any time soon!

On taking good care of your phone

Fri, 24 Nov 2017 00:00:00 +0000

I just finished replacing the micro-USB daughterboard on my Samsung Galaxy S5, which involved taking the phone most of the way apart, doing the replacement, and putting it back together. This inspired me to write about my approach to maintaining my cell phone. I’ve had this phone for a while and I have no plans to upgrade - I backed the upcoming Purism phone, but I expect to spend months/years on the software before I’ll be using that as my daily driver.

Portability matters

Mon, 13 Nov 2017 00:00:00 +0000

There are many kinds of “portability” in software. Portability refers to the relative ease of “porting” a piece of software to another system. That platform might be another operating system, another CPU architecture, another web browser, another filesystem… and so on. More portable software uses the limited subset of interfaces that are common between systems, and less portable software leverages interfaces specific to a particular system.

Some people think that portability isn’t very important, or don’t understand the degree to which it’s important. Some people might call their software portable if it works on Windows and macOS - they’re wrong. They might call their software portable if it works on Windows, macOS, and Linux - but they’re wrong, too. Supporting multiple systems does not necessarily make your software portable. What makes your software portable is standards.

Nvidia sucks and I'm sick of it

Thu, 26 Oct 2017 00:00:00 +0000

There’s something I need to make clear about Nvidia. Sway 1.0, which is the release after next, is not going to support the Nvidia proprietary driver, EGLStreams, or any other proprietary graphics APIs. The only supported driver for Nvidia cards will be the open source nouveau driver. I will explain why.

Today, Sway is able to run on the Nvidia proprietary driver. This is not and has never been an officially supported feature - we’ve added a few things to try and make it easier but my stance has always been that Nvidia users are on their own for support. In fact, Nvidia support was added to Sway without my approval. It comes from a library we depend on called wlc - had I’d made the decision on whether or not to support EGLStreams in wlc, I would have said no.

The future of Wayland, and sway's role in it

Mon, 09 Oct 2017 00:00:00 +0000

Today I’ve released sway 0.15-rc1, the first release candidate for the final 0.x release of sway. That’s right - after sway 0.15 will be sway 1.0. After today, no new features are being added to sway until we complete the migration to our new plumbing library, wlroots. This has been a long time coming, and I would love to introduce you to wlroots and tell you what to expect from sway 1.0.

Analyzing HN moderation & censorship

Wed, 13 Sep 2017 00:00:00 +0000

Hacker News is a popular “hacker” news board. One thing I love about HN is that the moderation generally does an excellent job. The site is free of spam and the conversations are usually respectful and meaningful (if pessimistic at times). However, there is always room for improvement, and moderation on Hacker News is no exception.

Notice: on 2017-10-19 this article was updated to incorporate feedback the Hacker News moderators sent to me to clarify some of the points herein. You may view a diff of these changes here.

Killing ants with nuclear weapons

Fri, 08 Sep 2017 00:00:00 +0000

Complexity is quickly becoming an epidemic. In this developer’s opinion, complexity is the ultimate enemy - the final boss - of good software design. Complicated software generally has complicated bugs. Simple software generally has simple bugs. It’s as easy as that.

It’s for this reason that I strongly dislike many of the tools and architectures that have been proliferating over the past few years, particularly in web development. When I look at a tool like Gulp, I wonder if its success is largely attributable to people not bothering to learn how Makefiles work. Tools like Docker make me wonder if they’re an excuse to avoid learning how to do ops or how to use your distribution’s package manager. Chef makes me wonder if its users forgot that shell scripts can use SSH, too.

When not to use a regex

Sun, 13 Aug 2017 00:00:00 +0000

The other day, I saw Learn regex the easy way. This is a great resource, but I felt the need to pen a post explaining that regexes are usually not the right approach.

Let’s do a little exercise. I googled “URL regex” and here’s the first Stack Overflow result:

https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,6}\b([-a-zA-Z0-9@:%_\+.~#?&//=]*)

source

This is a bad regex. Here are some valid URLs that this regex fails to match:

http://x.org
http://nic.science
http://名がドメイン.com (warning: this is a parked domain)
http://example.org/url,with,commas
https://en.wikipedia.org/wiki/Harry_Potter_(film_series)
http://127.0.0.1
http://[::1] (ipv6 loopback)

Here are some invalid URLs the regex is fine with:

State of Sway August 2017

Wed, 09 Aug 2017 00:00:00 +0000

Is it already time to write another one of these? Phew, time flies. Sway marches ever forward. Sway 0.14.0 was recently released, adding much asked-after support for tray icons and fixing some long-standing bugs. As usual, we already have some exciting features slated for 0.15.0 as well, notably some cool improvements to clipboard support. Look forward to it!

Today Sway has 24,123 lines of C (and 4,489 lines of header files) written by 94 authors across 2,345 commits. These were written through 689 pull requests and 624 issues. Sway packages are available today in the repos of almost every Linux distribution.

Archive it or you will miss it

Mon, 19 Jun 2017 00:00:00 +0000

Let’s open with some quotes from the Wikipedia article on link rot:

In 2014, bookmarking site Pinboard’s owner Maciej Cegłowski reported a “pretty steady rate” of 5% link rot per year… approximately 50% of the URLs in U.S. Supreme Court opinions no longer link to the original information… (analysis of) more than 180,000 links from references in… three major open access publishers… found that overall 24.5% of links cited were no longer available.

An introduction to Wayland

Sat, 10 Jun 2017 00:00:00 +0000

Wayland is the new hotness on the Linux graphics stack. There are plenty of introductions to Wayland that give you the high level details on how the stack is laid out how applications talk directly to the kernel with EGL and so on, but that doesn’t give you much practical knowledge. I’d like to instead share with you details about how the protocol actually works and how you can use it.

Limited "generics" in C without macros or UB

Mon, 05 Jun 2017 00:00:00 +0000

I should start this post off by clarifying that what I have to show you today is not, in fact, generics. However, it’s useful in some situations to solve the same problems that generics might. This is a pattern I’ve started using to reduce the number of void* pointers floating around in my code: multiple definitions of a struct.

Errata: we rolled this approach back in wlroots because it causes problems with LTO. I no longer recommend it.

Rotating passwords in bulk in the wake of security events

Thu, 11 May 2017 00:00:00 +0000

I’ve been putting this post off for a while. Do you remember the CloudFlare security problem that happened a few months ago? This is the one that disclosed huge amounts of sensitive information for huge numbers websites. When this happened, your accounts on thousands of websites were potentially compromised.

Updating passwords for all of these services at once was a major source of frustration for users. Updating a single password can take 5 minutes, and changing dozens of them might take hours. I decided that I wanted to make this process easier.

Building a "real" Linux distro

Fri, 05 May 2017 00:00:00 +0000

I recently saw a post on Hacker News: “Build yourself a Linux”, a cool project that guides you through building a simple Linux system. It’s similar to Linux from Scratch in that it helps you build a simple Linux system for personal use. I’d like to supplement this with some insight into my experience with a more difficult task: building a full blown Linux distribution. The result is agunix, the “silver unix” system.

State of Sway April 2017

Sat, 29 Apr 2017 00:00:00 +0000

Development on Sway continues. I thought we would have slowed down a lot more by now, but every release still comes with new features - Sway 0.12 added redshift support and binary space partitioning layouts. Sway 0.13.0 is coming soon and includes, among other things, nvidia proprietary driver support. We already have some interesting features slated for Sway 0.14.0, too!

Today Sway has 21,446 lines of C (and 4,261 lines of header files) written by 81 authors across 2,263 commits. These were written through 653 pull requests and 529 issues. Sway packages are available today in the official repos of pretty much every distribution except for Debian derivatives, and a PPA is available for those guys.

MSG_PEEK is pretty common, CVE-2016-10229 is worse than you think

Thu, 13 Apr 2017 00:00:00 +0000

I heard about CVE-2016-10229 earlier today. In a nutshell, it allows for arbitrary code execution via UDP traffic if userspace programs are using MSG_PEEK in their recv calls. I quickly updated my kernels and rebooted any boxes where necessary, but when I read the discussions on this matter I saw people downplaying this issue by claiming MSG_PEEK is an obscure feature.

I don’t want to be a fear monger and I’m by no means a security expert but I suspect that this is a deeply incorrect conclusion. If I understand this vulnerability right you need to drop everything and update any servers running a kernel <4.5 immediately. MSG_PEEK allows a programmer using UDP to read from the kernel’s UDP buffer without consuming the data (so subsequent reads will continue to read the same data). This immediately sounds to me like a pretty useful feature that a lot of software might use, not an obscure one.

Principles for C programming

Wed, 15 Mar 2017 00:00:00 +0000

In the words of Doug Gwyn, “Unix was not designed to stop you from doing stupid things, because that would also stop you from doing clever things”. C is a very powerful tool, but it is to be used with care and discipline. Learning this discipline is well worth the effort, because C is one of the best programming languages ever made. A disciplined C programmer will…

Prefer maintainability. Do not be clever where cleverness is not required. Instead, seek out the simplest and most understandable solution that meets the requirements. Most concerns, including performance, are secondary to maintainability. You should have a performance budget for your code, and you should be comfortable spending it.

Compiler devnotes: Machine specs

Wed, 22 Feb 2017 00:00:00 +0000

I have a number of long-term projects that I plan for on long timelines, on the order of decades or more. One of these projects is cozy, a C toolchain. I haven’t talked about this project in public before, so I’ll start by introducing you to the project. The main C toolchains in the “actually usable” category are GNU and LLVM, but I’m satisfied with neither and I want to build my own toolchain. I see no reason why compilers should be deep magic. Here are my goals for cozy:

Lessons to learn from C

Mon, 30 Jan 2017 00:00:00 +0000

C is my favorite language, though I acknowledge that it has its warts. I’ve tried looking at languages people hope will replace C (Rust, Go, etc), and though they’ve improved on some things they won’t be supplanting C in my life any time soon. I’ll share with you what makes C a great language to me. Take some of these things as inspiration for the next C replacement you write.

The only problem with Python 3's str is that you don't grok it

Fri, 13 Jan 2017 00:00:00 +0000

I’ve found myself explaining Python 3’s str to people online more and more often lately. There’s this ridiculous claim about that Python 3’s string handling is broken or somehow worse than Python 2, and today I intend to put that myth to rest. Python 2 strings are broken, and Python 3 strings are sane. The only problem is that you don’t grok strings.

The basic problem many people seem to have with Python 3’s strings arises when they write code that treats bytes like a string, because that’s how it was in Python 2. Let me make this as clear as possible:

Actually, you CAN do it

Fri, 06 Jan 2017 00:00:00 +0000

I maintain a lot of open source projects. In order to do so, I have to effectively manage my time. Most of my projects follow this philosophy: if you want something changed, send a patch. If you are running into an annoying bug, fix it and send a patch. If you want a new feature, implement it and send a patch. It’s definitely a good idea to talk about it beforehand on the issue tracker or IRC, but don’t make the mistake of thinking this processes ends with someone else doing it for you.

State of Sway December 2016 - secure your Wayland desktop, get paid to work on Sway

Tue, 27 Dec 2016 00:00:00 +0000

Earlier today I released sway 0.11, which (along with lots of the usual new features and bug fixes) introduces support for security policies that can help realize the promise of a secure Wayland desktop. We also just started a bounty program that lets you sponsor the things you want done and rewards contributors for working on them.

Today sway has 19,371 lines of C (and 3,761 lines of header files) written by 70 authors across 2,067 commits. These were written through 589 pull requests and 425 issues. Sway packages are available today in the official repos of Arch, Gentoo, Fedora, NixOS, openSUSE, Void Linux, and more. Sway looks like this:

A broad intro to networking

Tue, 06 Dec 2016 00:00:00 +0000

Disclaimer: I am not a network engineer. That’s the point of this blog post, though - I want to share with non-networking people enough information about networking to get by. Hopefully by the end of this post you’ll know enough about networking to keep up with a conversation on networking, or know what to search for when something breaks, or know what tech to research more in-depth when you are putting together something new.

Electron considered harmful

Thu, 24 Nov 2016 00:00:00 +0000

Yeah, I know that “considered harmful” essays are allegedly considered harmful. If it surprises you that I’m writing one, though, you must be a new reader. Welcome! Let’s get started. If you’re unfamiliar with Electron, it’s some hot new tech that lets you make desktop applications with HTML+CSS+JavaScript. It’s basically a chromeless web browser with a Node.js backend and a Chromium-based frontend. What follows is the rant of a pissed off Unix hacker, you’ve been warned.

Getting on without Google

Wed, 16 Nov 2016 00:00:00 +0000

I used Google for a long time, but have waned myself off of it over the past few years, and I finally deleted my account a little over a month ago. I feel so much better about my privacy now that I’ve removed Google from the equation, and self hosting my things affords me a lot of flexibility and useful customizations.

mail.cmpwn.com

This one was the most difficult and time consuming to set up, but it was very worth it. I’ve intended for a while to make a new mail server software suite that’s less terrible to set up, so hopefully that situation will improve in the future. I want to flesh out aerc some more first. A personal mail server was one of the earliest things I set up in my post-Google life - I’ve operated it for about two years now.

I'm losing faith in America

Sat, 05 Nov 2016 00:00:00 +0000

I recently quit my job at Linode and started looking for something else to do. For the first time in my career, I’m seriously considering opportunities abroad. Sorry for the politically charged post - I promise to get back to tech stuff right away.

On November 8th, I’m going to step into the voting booth and will be presented with the following options:

A criminal who cheated her way into a spot on the ballot
An egotistical racist maniac

The next president of the United States will probably be Hillary Clinton. I’m sure I don’t have to tell you how ridiculous this is. This is a person who has pulled all of the stops to get her name on the ballot, including voter fraud and disturbing amounts of corruption within the Democratic party. Not to mention that she’s probably going to start a war with Syria, mess with the already fragile geopolitical relationship we have with Russia, and likely deserves to be incarcerated for mishandling classified information. Say what you will about the Republican party - at least Trump won his nomination fair and square. Bonus: not voting for Hillary is sexist.

Using the right tool for the job

Sat, 17 Sep 2016 00:00:00 +0000

One of the most important choices you’ll make for the software you write is what you write it in, what frameworks you use, the design methodologies to subscribe to, and so on. This choice doesn’t seem to get the respect it’s due. These are some of the only choices you’ll make that you cannot change. Or, at least, these choices are among the most difficult ones to change.

People often question why TrueCraft is written in C# next to projects like Sway in C, alongside KnightOS in Assembly or sr.ht in Python. It would certainly be easier from the outset if I made every project in a language I’m comfortable with, using tools and libraries I’m comfortable with, and there’s certainly something to be had for that. That’s far from being the only concern, though.

What motivates the authors of the software you use?

Fri, 09 Sep 2016 00:00:00 +0000

We face an important choice in our lives as technophiles, hackers, geeks: the choice between proprietary software and free/open source software. What platforms we choose to use are important. We have a choice between Windows, OS X, and Linux (not to mention the several less popular choices). We choose between Android or iOS. We choose hardware that requires nonfree drivers or ones that don’t. We choose to store our data in someone else’s cloud or in our own. How do we make the right choice?

[VIDEO] Arch Linux with full disk encryption in (about) 15 minutes

Thu, 18 Aug 2016 00:00:00 +0000

After my blog post emphasizing the importance of taking control of your privacy, I’ve decided to make a few more posts going over detailed instructions on how to actually do so. Today we have a video that goes over the process of installing Arch Linux with full disk encryption.

This is my first go at publishing videos on my blog, so please provide some feedback in the comments of this article. I’d prefer to use my blog instead of YouTube for publishing technical videos, since it’s all open source, ad-free, and DRM-free. Let me know if you’d like to see more content like this on my blog and which topics you’d like covered - I intend to at least release another video going over this process for Ubuntu as well.

Sway 0.9 & One year of Sway

Tue, 02 Aug 2016 00:00:00 +0000

Today marks one year since the initial commit of Sway. Over the year since, we’ve written 1,823 commits by 54 authors, totalling 16,601 lines of C (and 1,866 lines of header files). This was written over the course of 515 pull requests and 300 issues. Today, most i3 features are supported. In fact, as of last week, all of the features from the i3 configuration I used before I started working on Sway are now supported by Sway. Today, Sway looks like this (click to expand):

Using -Wl,--wrap for mocking in C

Tue, 19 Jul 2016 00:00:00 +0000

One of the comforts I’ve grown used to in higher level languages when testing my code is mocking. The idea is that in order to test some code in isolation, you should “mock” the behavior of things it depends on. Let’s see a (contrived) example:

int read_to_end(FILE *f, char *buf) {
    int r = 0, l;
    while (!feof(f)) {
        l = fread(buf, 1, 256, f);
        r += l;
        buf += l;
    }
    return r;
}

If we want to test this function without mocking, we would need to actually open a specially crafted file and provide a FILE* to the function. However, with the linker --wrap flag, we can define a wrapper function. Using -Wl,[flag] in your C compiler command line will pass [flag] to the linker. Gold (GNU) and lld (LLVM) both support the wrap flag, which specifies a function to be “wrapped”. If I use -Wl,--wrap=fread, then the code above will be compiled like so:

Life, liberty, and the pursuit of privacy

Wed, 29 Jun 2016 00:00:00 +0000

Privacy is my hobby, and should be a hobby of every technically competent American. Within the eyes of the law I have a right to secure the privacy of my information. At least that’s the current law - many officials are trying to subvert that right. I figure that we’d better exercise that right while we have it, so that we know how to keep exercising it once it’s illegal and all the information about it dries up.

Understanding pointers

Sat, 28 May 2016 00:00:00 +0000

I was recently chatting with a new contributor to Sway who is using the project as a means of learning C, and he had some questions about what void** meant when he found some in the code. It became apparent that this guy only has a basic grasp on pointers at this point in his learning curve, and I figured it was time for another blog post - so today, I’ll explain pointers.

In Memoriam - Mozilla

Wed, 11 May 2016 00:00:00 +0000

Today we look back to the life of Mozilla, a company that was best known for creating the Firefox web browser. I remember a company that made the web better and more open by providing a browser that was faster and more customizable than anyone had ever seen, and by making that browser free and open source.

I expect many of my readers will be older than I am, but my first memories of Firefox are back in high school with Firefox 3. I fondly remember my discovery of it. Mozilla gave us a faster and more powerful web browser to use on school computers. The other choice was Internet Explorer 6 - but with a flash drive we could run a “portable” version of Firefox instead. Using tabbed web browsing was a clear improvement for usability and I loved installing all sorts of cool add-ons and I’m sure I’ve spent at least a few hours of my life browsing persona themes.

State of Sway - April 2016

Wed, 20 Apr 2016 00:00:00 +0000

Since the previous State of Sway, we have accomplished quite a bit. We are now shipping versioned releases of sway, which include support for window borders, input device configuration, more new features, and many bug fixes and stability improvements. I’m also happy to say that Sway 0.5 has landed in the Arch Linux community repository and I’m starting to hear rumors of it landing in other Linux distros as well. Here’s a quick rundown of what’s happened in the past four months:

How to write a better bloom filter in C

Tue, 12 Apr 2016 00:00:00 +0000

This is in response to How to write a bloom filter in C++, which has good intentions, but is ultimately a less than ideal bloom filter implementation. I put together a better one in C in a few minutes, and I’ll explain the advantages of it.

The important differences are:

You bring your own hashing functions
You can add arbitrary data types, not just bytes
It uses bits directly instead of relying on the std::vector<bool> being space effecient

I chose C because (1) I prefer it over C++ and (2) I just think it’s a better choice for implementing low level data types, and C++ is better used in high level code.

Please use text/plain for email

Mon, 11 Apr 2016 00:00:00 +0000

A lot of people have come to hate email, and not without good reason. I don’t hate using email, and I attribute this to better email habits. Unfortunately, most email clients these days lead users into bad habits that probably contribute to the sad state of email in 2016. The biggest problem with email is the widespread use of HTML email.

Compare email to snail mail. You probably throw out most of the mail you get - it’s all junk, ads. Think about the difference between snail mail you read and snail mail you throw out. Chances are, the mail you throw out is flashy flyers and spam that’s carefully laid out by a designer and full of eye candy (kind of like many HTML emails). However, if you receive a letter from a friend it’s probably going to be a lot less flashy - just text on a page. Reading letters like this is pleasant and welcome. Emails should be more like this.

Integrating a VT220 into my life

Tue, 22 Mar 2016 00:00:00 +0000

I bought a DEC VT220 terminal a while ago, and put it next to my desk at work. I use it to read emails on mutt now, and it’s actually quite pleasant. There was some setup involved in making it as comfortable as possible, though.

Here’s the terminal up close:

Hardware

First, I have several pieces of hardware involved in this:

VT220 terminal
LK201 keyboard (later made obsolete)
USB to serial adapter
DB9->DB29 null modem cable

It took a while to get all of these things, but I was able to get a nice refurbished terminal and a couple of crappy LK201 keyboards. Luckily I was able to eventually remove the need for the keyboard.

State of Sway - December 2015

Sun, 20 Dec 2015 00:00:00 +0000

I wrote sway’s initial commit 4 months ago, on August 4th. At the time of writing, there are now 1,070 commits from 29 different authors, totalling 10,682 lines of C (and 1,176 lines of header files). This has been done over the course of 256 pull requests and 118 issues. Of the 73 i3 features we’re tracking, 51 are now supported, and I’ve been using sway as my daily driver for a while now. Today, sway looks like this:

Bring more Tor into your life

Wed, 11 Nov 2015 00:00:00 +0000

Tor is a project that improves your privacy online by encrypting and bouncing your connection through several nodes before leaving for the outside world. It makes it much more difficult for someone spying on you to know who you’re talking to online and what you’re saying to them. Many people use it with the Tor Browser (a fork of Firefox) and only use it with HTTP.

What some people do not know is that Tor works at the TCP level, and can be used for any kind of traffic. There is a glaring issue with using Tor for your daily browsing - it’s significantly slower. That being said, there are several things you run on your computer where speed is not quite as important. I am personally using Tor for several things (this list is incomplete):

Please don't use Slack for FOSS projects

Sun, 01 Nov 2015 00:00:00 +0000

I’ve noticed that more and more projects are using things like Slack as the chat medium for their open source projects. In the past couple of days alone, I’ve been directed to Slack for Babel and Bootstrap. I’d like to try and curb this phenomenon before it takes off any more.

Problems with Slack

Slack…

is closed source
has only one client (update: errata at the bottom of this article)
is a walled garden
requires users to have a different tab open for each project they want to be involved in
requires that Heroku hack to get open registration

The last one is a real stinker. Slack is not a tool built for open source projects to use for communication with their userbase. It’s a tool built for teams and it is ill-suited to this use-case. In fact, Slack has gone on record as saying that it cannot support this sort of use-case: “it’s great that people are putting Slack to good use” but unfortunately “these communities are not something we have the capacity to support given the growth in our existing business.” ¹

A practical understanding of Flux

Mon, 20 Jul 2015 00:00:00 +0000

React.js and the Flux are shaping up to be some of the most important tools for web development in the coming years. The MVC model was strong on the server when we decided to take the frontend seriously, and it was shoehorned into the frontend since we didn’t know any better. React and Flux challenge that and I like where it’s going very much. That being said, it was very difficult for me to get into. I put together this blog post to serve as a more practical guide - the upstream documentation tells you a lot of concepts and expects you to put them together yourself. Hopefully at the end of this blog post you can confidently start writing things with React+Flux instead of reading brain-melting docs for a few hours like I did.

osu!web - WebGL & Web Audio

Sun, 14 Jun 2015 00:00:00 +0000

I’ve taken a liking to a video game called osu! over the past few months. It’s a rhythm game where you use move your mouse to circles that appear with the beat, and click (or press a key) at the right time. It looks something like this:

The key of this game is that the “beatmaps” (a song plus notes to hit) are user-submitted. There are thousands of them, and the difficulty curve is very long - I’ve been playing for 10 months and I’m only maybe 70% of the way up the difficulty curve. It’s also a competitive game, which leads to a lot more fun, where each user tries to complete maps a little bit better than everyone else can. You can see on the left in that video - this is a very good player who earned the #1 rank during this play.

Hooks - running stuff on Github hooks

Sun, 19 Apr 2015 00:00:00 +0000

I found myself in need of a simple tool for deploying a project on every git commit, but I didn’t have a build server set up. This led to Hooks - a very simple tool that allows you to run arbitrary commands when Github’s hooks execute.

The configuration is very simple. In /etc/hooks.conf, write:

[truecraft]
repository=SirCmpwn/TrueCraft
branch=master
command=systemctl restart hooks
valid_ips=204.232.175.64/27,192.30.252.0/22,127.0.0.1

You may include any number of hooks. The valid_ips entry in that example allows you to accept hooks from Github and from localhost. Then you run Hooks itself, it will execute your command when you push a commit to your repository.

Porting an assembler, debugger, and more to WebAssembly

Sun, 30 Nov 2014 00:00:00 +0000

WebAssembly is pretty cool! It lets you write portable C and cross-compile it to JavaScript so it’ll run in a web browser. As the maintainer of KnightOS, I looked to WASM as a potential means of reducing the cost of entry for new developers hoping to target the OS.

Rationale for WASM

There are several pieces of software in the toolchain that are required to write and test software for KnightOS:

On the profitability of image hosting websites

Fri, 10 Oct 2014 00:00:00 +0000

I’ve been doing a lot of thought about whether or not it’s even possible to both run a simple website and turn a profit from it and maintain a high quality of service. In particular, I’m thinking about image hosts, considering that I run one (a rather unprofitable one, too), but I would think that my thoughts on this matter apply to more kinds of websites. That being said, I’ll just talk about media hosting because that’s where I have tangible expertise.

Process scheduling and multitasking in KnightOS

Tue, 02 Sep 2014 00:00:00 +0000

I’m going to do some blogging about technical decisions made with KnightOS. It’s an open-source project I’ve been working on for the past four years to build an open-source Unix-like kernel for TI calculators (in assembly). It’s been a cool platform on top of which I can research low level systems concepts and I thought I’d share some of my findings with the world.

So, first of all, what is scheduling? For those who are completely out of the loop, I’ll explain what exactly it is and why it’s neccessary. Computers run on a CPU, which executes a series of instructions in order. Each core is not capable of running several instructions concurrently. However, you can run hundreds of processes at once on your computer (and you probably are doing so as you read this article). There are a number of ways of accomplishing, but the one that suits the most situations is preemtive multitasking. This is what KnightOS uses. You see, a CPU can only execute one instruction after another, but you can “raise an interrupt”. This will halt execution and move to some other bit of code for a moment. This can be used to handle various events (for example, the GameBoy raises an interrupt when a button is pressed). One of these events is often a timer, which raises an interrupt at a fixed interval. This is the mechanism by which preemptive multitasking is accomplished.

Python's datetime sucks

Sat, 28 Jun 2014 00:00:00 +0000

I’ve been playing with Python for about a year now, and I like pretty much everything about it. There’s one thing that’s really rather bad and really should not be that bad, however - date & time support. It’s ridiculous how bad it is in Python. This is what you get with the standard datetime module:

The current time and strftime, with a reasonable set of properties
Time deltas with days, seconds, and microseconds and nothing else
Acceptable support for parsing dates and times

What you don’t get is:

Go's error handling doesn't sit right with me

Sat, 07 Jun 2014 00:00:00 +0000

I’ll open up by saying that I am not a language designer, and I do like a lot of things about Go. I just recently figured out how to describe why Go’s error handling mechanics don’t sit right with me.

If you aren’t familiar with Go, here’s an example of how Go programmers might do error handling:

result, err := SomethingThatMightGoWrong()
if err != nil {
    // Handle error
}
// Proceed

Let’s extrapolate this:

Hacking on your TI calculator

Tue, 25 Feb 2014 00:00:00 +0000

I’ve built the KnightOS kernel, an open-source OS that runs on several TI calculator models, including the popular TI-83+ family, and recently the new TI-84+ Color Silver Edition. I have published some information on how to build your own operating sytsems for these devices, but I’ve learned a lot since then and I’m writing this blog post to include the lessons I’ve learned from other attempts.

Prerequisites

Coming into this, you should be comforable with z80 assembly. It’s possible to write an OS for these devices in C (and perhaps other high-level languages), but proficiency in z80 assembly is still required. Additionally, I don’t consider C a viable choice for osdev on these devices when you consider that the available compliers do not optimize the result very well, and these devices have very limited resources.

The bug that hides from breakpoints

Sun, 02 Feb 2014 00:00:00 +0000

This is the story of the most difficult bug I ever had to solve. See if you can figure it out before the conclusion.

Background

For some years now, I’ve worked on a kernel for Texas Instruments calculators called KnightOS. This kernel is written entirely in assembly, and targets the old-school z80 processor from back in 1976. This classic processor was built without any concept of protection rings. It’s an 8-bit processor, with 150-some instructions and (in this application) 32K of RAM and 32K of Flash. This stuff is so old, I ended up writing most of the KnightOS toolchain from scratch rather than try to get archaic assemblers and compilers running on modern systems.

Custom Music Syncing on Android

Sat, 24 Aug 2013 00:00:00 +0000

I have an HTC One, with CyanogenMod installed. I usually use Spotify, but I’ve been wanting to move away from it for a while. The biggest thing keeping me there was the ease of syncing up with my phone - I added music on my PC and it just showed up on my phone.

So, I finally decided to make it work on my phone without Spotify. You might have success if you aren’t using CyanogenMod, but you definitely need to be rooted and you need to access a root shell on your phone. I was using adb shell to start with, but it has poor terminal emulation. Instead, I ended up installing an SSH daemon on the phone and just using that. Easier to use vim in such an enviornment.

You don't need jQuery

Mon, 19 Aug 2013 00:00:00 +0000

It’s true. You really don’t need jQuery. Modern web browsers can do most of what you want from jQuery, without jQuery.

For example, take MediaCrush. It’s a website I spent some time working on with a friend. It’s actually quite sophisticated - drag-and-drop uploading, uploading via a hidden form, events wired up to links and dynamically generated content, and ajax requests/file uploads, the whole she-bang. It does all of that without jQuery. It’s open source, if you’re looking for a good example of how all of this can be used in the wild.

Mon, 01 Jan 0001 00:00:00 +0000

Dynamic linking

Do your installed programs share dynamic libraries?

Findings: not really

Over half of your libraries are used by fewer than 0.1% of your executables. Number of times each dynamic library is required by a program

libs.awk

/\t.*\.so.*/ {
	n=split($1, p, "/")
	split(p[n], l, ".")
	lib=l[1]
	if (libs[lib] == "") {
		libs[lib] = 0
	}
	libs[lib] += 1
}
END {
	for (lib in libs) {
		print libs[lib] "\t" lib
	}
}

Usage

$ find /usr/bin -type f -executable -print \
  | xargs ldd 2>/dev/null \
  | awk -f libs.awk \
  | sort -rn > results.txt
$ awk '{ print NR "\t" $1 }' < results.txt > nresults.txt
$ gnuplot
gnuplot> plot 'nresults.txt'

my results

$ find /usr/bin -type f -executable -print | wc -l
5688
$ head -n20 < results.txt
4496	libc
4484	linux-vdso
4483	ld-linux-x86-64
2654	libm
2301	libdl
2216	libpthread
1419	libgcc_s
1301	libz
1144	libstdc++
805	liblzma
785	librt
771	libXdmcp
771	libxcb
771	libXau
755	libX11
703	libpcre
667	libglib-2
658	libffi
578	libresolv
559	libXext

Is loading dynamically linked programs faster?

Findings: definitely not

Linkage	Avg. startup time
Dynamic	137263 ns
Static	64048 ns

ex.c

#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <unistd.h>

int main(int argc, char *argv[]) {
	struct timespec ts;
	clock_gettime(CLOCK_MONOTONIC, &ts);
	fprintf(stdout, "%ld\t", ts.tv_nsec);
	fflush(stdout);
	if (argc == 1) {
		char *args[] = { "", "", NULL };
		execvp(argv[0], args);
	} else {
		fprintf(stdout, "\n");
	}
	return 0;
}

test.sh

#!/bin/sh
i=0
while [ $i -lt 1000 ]
do
	./ex
	i=$((i+1))
done

My results

$ musl-gcc -o ex ex.c
$ ./test.sh | awk 'BEGIN { sum = 0 } { sum += $2-$1 } END { print sum / NR }'
137263
$ musl-gcc -static -o ex ex.c
$ ./test.sh | awk 'BEGIN { sum = 0 } { sum += $2-$1 } END { print sum / NR }'
64048

Wouldn't statically linked executables be huge?

Findings: not really

On average, dynamically linked executables use only 4.6% of the symbols on offer from their dependencies. A good linker will remove unused symbols. % of symbols requested by dynamically linked programs from the libraries that it depends on

nsyms.go

package main

import (
	"bufio"
	"fmt"
	"os"
	"os/exec"
	"path/filepath"
	"strings"
)

func main() {
	ldd := exec.Command("ldd", os.Args[1])
	rc, err := ldd.StdoutPipe()
	if err != nil {
		panic(err)
	}
	ldd.Start()

	var libpaths []string
	scan := bufio.NewScanner(rc)
	for scan.Scan() {
		line := scan.Text()[1:] /* \t */
		sp := strings.Split(line, " ")
		var lib string
		if strings.Contains(line, "=>") {
			lib = sp[2]
		} else {
			lib = sp[0]
		}
		if !filepath.IsAbs(lib) {
			lib = "/usr/lib/" + lib
		}
		libpaths = append(libpaths, lib)
	}
	ldd.Wait()
	rc.Close()

	syms := make(map[string]interface{})
	for _, path := range libpaths {
		objdump := exec.Command("objdump", "-T", path)
		rc, err := objdump.StdoutPipe()
		if err != nil {
			panic(err)
		}
		objdump.Start()
		scan := bufio.NewScanner(rc)
		for i := 0; scan.Scan(); i++ {
			if i < 4 {
				continue
			}
			line := scan.Text()
			sp := strings.Split(line, " ")
			if len(sp) < 5 {
				continue
			}
			sym := sp[len(sp)-1]
			syms[sym] = nil
		}
		objdump.Wait()
		rc.Close()
	}

	objdump := exec.Command("objdump", "-R", os.Args[1])
	rc, err = objdump.StdoutPipe()
	if err != nil {
		panic(err)
	}
	objdump.Start()
	used := make(map[string]interface{})
	scan = bufio.NewScanner(rc)
	for i := 0; scan.Scan(); i++ {
		if i < 5 {
			continue
		}
		sp := strings.Split(scan.Text(), " ")
		if len(sp) < 3 {
			continue
		}
		sym := sp[len(sp)-1]
		used[sym] = nil
	}
	objdump.Wait()
	rc.Close()

	if len(syms) != 0 && len(used) != 0 && len(used) <= len(syms) {
		fmt.Printf("%50s\t%d\t%d\t%f\n", os.Args[1], len(syms), len(used),
			float64(len(used)) / float64(len(syms)))
	}
}

Usage

$ find /usr/bin -type f -executable -print | xargs -n1 ./nsyms > results.txt
$ awk '{ n += $4 } END { print n / NR }' < results.txt

my results

Will security vulnerabilities in libraries that have been statically linked cause large or unmanagable updates?

Findings: not really

Not including libc, the only libraries which had "critical" or "high" severity vulnerabilities in 2019 which affected over 100 binaries on my system were dbus, gnutls, cairo, libssh2, and curl. 265 binaries were affected by the rest.

The total download cost to upgrade all binaries on my system which were affected by CVEs in 2019 is 3.8 GiB. This is reduced to 1.0 GiB if you eliminate glibc.

It is also unknown if any of these vulnerabilities would have been introduced after the last build date for a given statically linked binary; if so that binary would not need to be updated. Many vulnerabilities are also limited to a specific code path or use-case, and binaries which do not invoke that code path in their dependencies will not be affected. A process to ascertain this information in the wake of a vulnerability could be automated.

arch-security

extractcves.py

import email.utils
import mailbox
import re
import shlex
import time

pacman_re = re.compile(r'pacman -Syu .*')
severity_re = re.compile(r'Severity: (.*)')

mbox = mailbox.mbox("arch-security.mbox")
for m in mbox.items():
    m = m[1]
    date = m["Date"]
    for part in m.walk():
        if part.is_multipart():
            continue
        content_type = part.get_content_type()
        [charset] = part.get_charsets("utf-8")
        if content_type == 'text/plain':
            body = part.get_payload(decode=True).decode(charset)
            break
    pkgs = pacman_re.findall(body)
    severity = severity_re.findall(body)
    date = email.utils.parsedate(date)
    if len(pkgs) == 0 or date is None:
        continue
    if date[0] <= 2018 or date[0] > 2019:
        continue
    severity = severity[0]
    args = shlex.split(pkgs[0])
    pkg = args[2].split(">=")[0]
    print(pkg, severity)

$ python3 extractcves.py | grep Critical > cves.txt
$ xargs pacman -Ql < cves.txt | grep \\.so | awk '{print $1}' | sort -u>affected.txt
# Manually remove packages like Firefox, Thunderbird, etc; write remainder.txt
$ xargs pacman -Ql < remainder.txt | grep '/usr/lib/.*.so$' | awk '{ print $2 }' > libs.txt
$ ldd /usr/bin/* >ldd.txt
$ ./scope.sh <libs.txt | sort -nr >sobjects.txt

sobjects.txt is a sorted list of shared objects and the number of executables that link to them. To find the total size of affected binaries, I ran the following command:

# With libc
$ egrep -la 'libc.so|libm.so|libdl.so|libpthread.so|librt.so|libresolv.so|libdbus-1.so|libgnutls.so|libcairo.so|libutil.so|libssh2.so|libcurl.so|libcairo-gobject.so|libcrypt.so|libspice-server.so|libarchive.so|libSDL2-2.0.so|libmvec.so|libmagic.so|libtextstyle.so|libgettextlib-0.20.2.so|libgettextsrc-0.20.2.so|libMagickWand-7.Q16HDRI.so|libMagickCore-7.Q16HDRI.so|libbfd-2.34.0.so|libpolkit-gobject-1.so|libwebkit2gtk-4.0.so|libjavascriptcoregtk-4.0.so|libpolkit-agent-1.so|libgs.so|libctf.so|libSDL.so|libopcodes-2.34.0.so|libQt5WebEngine.so|libQt5WebEngineCore.so|libctf-nobfd.so|libcairo-script-interpreter.so' /usr/bin/* | xargs wc -c
# Without libc
$ egrep -la 'libdbus-1.so|libgnutls.so|libcairo.so|libssh2.so|libcurl.so|libcairo-gobject.so|libcrypt.so|libspice-server.so|libarchive.so|libSDL2-2.0.so|libmvec.so|libmagic.so|libtextstyle.so|libgettextlib-0.20.2.so|libgettextsrc-0.20.2.so|libMagickWand-7.Q16HDRI.so|libMagickCore-7.Q16HDRI.so|libbfd-2.34.0.so|libpolkit-gobject-1.so|libwebkit2gtk-4.0.so|libjavascriptcoregtk-4.0.so|libpolkit-agent-1.so|libgs.so|libctf.so|libSDL.so|libopcodes-2.34.0.so|libQt5WebEngine.so|libQt5WebEngineCore.so|libctf-nobfd.so|libcairo-script-interpreter.so' /usr/bin/* | xargs wc -c

Doesn't static linking prevent ASLR from working?

No.

We've had ASLR for statically linked binaries for some time now. It's called static PIE.

Test environment

Arch Linux, up-to-date as of 2020-06-25
2188 packages installed
gcc 10.1.0

Mon, 01 Jan 0001 00:00:00 +0000

Videos of people editing text

Videos of people editing text, explaining the shortcuts they use, and recordings of their keyboards as they work. The purpose of this page is to provide a reference for people writing new software which involves text editing tasks.

These videos are licensed with CC-BY-SA, which permits re-distribution and modification so long as attribution is provided and derivative works use the same license.

Can you contribute your own?

Resources

Videos

Your browser must support HTML5 video and webm to play this video.

Source: Drew DeVault
Language: English
Keyboard: Kinesis Freestyle Pro; ANSI layout; QWERTY
Operating system: Linux 5.7.6
Software: gedit 3.36.2 (GTK+ 3.24.20), alacritty 0.5.0.1564.g3233ede0, vim 8.2.0814
Tags: #english #en #en-us #ansi #qwerty #modal-editing

Contribute

I would be interested in adding your video to this page, provided that it demonstrates one of the following qualities which is not yet represented by another video:

A natural language not yet represented
An operating system not yet represented
A keyboard layout or design not yet represented
Software with novel editing features
A novel workflow, e.g. screen reader or dictation software

When preparing your video, please use a camera to capture a video of your keyboard. I recommend OBS to capture a recording of both your screen and keyboard, as well as audio from your microphone. Please narrate your workflow in English, and include details which seem "obvious" to you; they might not be obvious to someone else. Contact Drew DeVault <sir@cmpwn.com> to list your video. Please include the following details:

Your name, for attribution
The natural language you are demonstrating input for
Your keyboard model and layout
Your operating system and the names & version numbers of your software

Thank you!

Mon, 01 Jan 0001 00:00:00 +0000

New Server Checklist

Set root password Generate fresh sshd host keys Create admin user Add admin to doas group and test doas SSH key added to admin's authorized_keys Disable root login via ssh Disable password login via ssh Set hostname Run system updates Reboot Install & test postfix

doas.conf

# see doas.conf(5) for configuration details

# Uncomment to allow group "admin" to become root
# permit :admin
permit nopass :admin
permit nopass deploy cmd apk args upgrade -U
permit nopass deploy cmd service args SERVICE restart
permit nopass acme cmd nginx args -s reload

acme setup

TODO: a package could be made to automate many of these steps

Mon, 01 Jan 0001 00:00:00 +0000

ThinkPad T420 vs Pinebook Pro

	ThinkPad T420	Pinebook Pro
Upstream kernel	✓	✗
Open BIOS	✓¹	✓
Sane bootloader	BIOS	uBoot
Open display firmware	✓	✗
Open wireless firmware	✗	✗
Wireless kill switch	✓	✓
Bluetooth 5.0	✗	✓
Storage	500G	64G
Upgradable storage	✓	✓
RAM	16G	4G
Upgradable memory	✓	✗
Weight	2.2kg	1.3kg
Battery life	3.5-5.8 hours	6-8 hours
Replacable battery	✓	requires screwdriver
User-serviceable parts	✓	some
HD video playback	✓	good luck
Usable web browsing	✓	good luck
Price	$150 (eBay)	$200 (Pine store)
Release year	2011	2019²

The Pinebook Pro is a decent laptop. It is lightweight, has a good display, and a good battery life. I am happy with Pine64 as a company and think that it is a good product. There is a place for it in my life. Pine64 has their heart in the right place and I support them and buy their products because I want them to succeed.

But, is this the best the modern laptop industry can come up with? As a workstation, it doesn't compete. The performance of the ARM CPU is absolutely put to shame by the 8 year old T420, even for basic tasks. Is this 8 years of advancement in the state of the art?

Get your shit together, laptop vendors. Fuck's sake.

1: Aftermarket via coreboot

2: Limited release, full release in 2020

Mon, 01 Jan 0001 00:00:00 +0000

Human history

Human history Scroll down ↓

Disclaimer: I have made no attempt to make this look correct on screens <1,000 pixels wide.

10 years 1,000 years

Stone tools

Homo sapiens

Art

Agriculture

Civilization

Industry World War II

Building and scaling Hare’s community governance

Mon, 01 Jan 0001 00:00:00 +0000

Abstract

How the Hare programming language community grew from one BDFL and a ragtag group of early hackers to a productive, sociable, and egalitarian community of happy hackers with a lightweight and effective model of participatory governance.

Resources

Events

FOSS Backstage 2026

2026-03-17 11:55 - 12:25

Details

Can free software be made sustainable?

Mon, 01 Jan 0001 00:00:00 +0000

Abstract

Money is a tool, and it can be used to make free software work better. It can also be used to make free software worse. In this talk, we’ll discuss strategies for managing the economic incentives of free software such that you can put food on your table without compromising on your values.

Resources

Download slides

Events

Hackmeeting 0x19

CSOA Gabrio, Via Millio 42, Torino, Italia

2022-09-10 @ 19:00 local time

Cryptocurrency sucks

Mon, 01 Jan 0001 00:00:00 +0000

Abstract

We will examine cryptocurrency in detail to understand how it works, how it fails to achieve its stated goals, how useful it (isn’t) as a currency, and the consequences for society.

Resources

Events

Techinc

2022-04-16 @ 17:00

Louwesweg 1
1066 EA Amsterdam
Netherlands

Call +31852016320 to get someone to let you in.

Drew DeVault's Japanese Learning Resources

Mon, 01 Jan 0001 00:00:00 +0000

Note: this is a work in progress. I last updated this page on 2019-11-12. If that was a while ago, feel free to ping me and ask me to write some more.

Introduction

I’ve been studying Japanese for a few years now, and while I still have a lot to learn I feel comfortable having conversations in Japanese, without feeling like the other participant is having to be concious of the complexity of their language use. Accomplishing that had been my goal from the start; yours may be different. What follows is what I believe is a reasonably good approach to self-directed Japanese language study.

Himitsu: A novel keyring & password manager for Unix

Mon, 01 Jan 0001 00:00:00 +0000

Abstract

Himitsu is a novel design for a key/value store to securely store your passwords, private keys, and other secrets. It is a simple core upon which a highly extensible ecosystem of integrations can be built to handle any kind of secret. This talk will introduce the Himitsu key store, demonstrate many of the things it is capable of, and explain its current level of sophistication and what we aspire for in the future.

Introducing the Hare programming langauge

Mon, 01 Jan 0001 00:00:00 +0000

Abstract

Hare is a systems programming language designed to be simple, stable, and robust. Hare uses a static type system, manual memory management, and a minimal runtime. It is well-suited to writing operating systems, system tools, compilers, networking software, and other low-level, high performance tasks.

Resources

Events

Techinc

2022-05-04 @ 19:00 AMS

Live stream/recording:

Introduction to qbe

Mon, 01 Jan 0001 00:00:00 +0000

Abstract

qbe is an optimizing compiler backend which consumes programs in a simple intermediate language, optimizes them, and emits assembly for x86_64, aarch64, or riscv64, aiming to achieve “70% of the performance” of advanced compilers like LLVM in “10% of the code”. This talk will briefly introduce qbe and its intermediate language, explain how it works and what it’s capable of, and go over some sample programs which can be written in it.

Miscellaneous links

Mon, 01 Jan 0001 00:00:00 +0000

Services

Talks

New sysadmin lecture

Mon, 01 Jan 0001 00:00:00 +0000

You’re a production sysadmin now. That comes with certain responsibilities. In short:

Respect the user’s privacy, and look at only what you must.
Think before you type.
With great power comes great responsibility.

Assorted tips:

Practice your changes on localhost first.
Ask for help if you need it.
Always run your SQL queries in a transaction.
SELECT things, you, want FROM x; is generally better than SELECT * FROM x; when considering the user’s privacy.
Share information on a need-to-know basis, both with people and with computers.
Avoid doing things that cannot be undone.

Spear Phishing

Because you now have access to production systems, you may be a target for spear phishing. A bad actor may target you directly in a social engineering attack in an attempt to get you to leverage your access to mistakenly compromise the system. For example, someone may impersonate another admin and ask you to add an SSH key to a server. You need to be aware of this risk.

Opening up the SourceHut CI engine

Mon, 01 Jan 0001 00:00:00 +0000

Abstract

An overview of the architecture of SourceHut’s CI engine, builds.sr.ht. We’ll be looking at how the system is laid out and at directions for future improvement.

Events

LOSTilburg

2022-05-24 @ 19:30

Hendrikx ITC (conference room Spoorzicht013)
Spoorlaan 21K
5038 CB Tilburg

The FOSS contributor's mindset

Mon, 01 Jan 0001 00:00:00 +0000

Abstract

Many people use free software, and a fraction of them have dipped their toes into contributing - publishing their own projects, or contributing to one or two. Too few, however, have completely grokked the opportunities the expansive free software ecosystem offers to us.

This talk will discuss how to contribute to a wider variety of projects, introduce useful tools for quickly coming to grips with a foreign codebase, and explain the mindset that these tools help to cultivate.

What is Gemini?

Mon, 01 Jan 0001 00:00:00 +0000

The Gemini protocol is an alternative to HTTP, and Gemtext is an alternative document format to HTML. This blog is available over both Gemini and HTTP, and some content is exclusive to each. To connect with Gemini, you will need a Gemini client. I wrote one called gmnlm, and there are more listed here.

Once you have your client working, direct it to gemini://drewdevault.com and you’ll be on your way.

You should make a blog!

Mon, 01 Jan 0001 00:00:00 +0000

I would like to see more people writing about their passions and thoughts in a permanent and public medium. You should make a blog!

This is an open offer to pay anyone $20 for starting a tech blog and writing their first article. If you write another 3 articles within 6 months, I’ll give you another $20. You can also choose to have me donate your reward to one of the following charitable organizations: