Matthew's Journal

Systemd is in many ways an improvement over System V init (sysvinit), but I worry about the direction it seems to be going in (and how upstream interact with folk), and I think distributions need to work a bit harder to preserve user choice in this area. ISTM there is space for a more tightly-focused new init system.

A little background - the first process started at boot is process 1, often called init. It is the parent of every other process on the system, and is responsible for bringing the system up (and shutting it down again). For a long time, on Linux this was sysvinit. It is pretty simple, but could be slow to start up a system[0] because it started services up in serial, and it didn't have much in the way of machinery for tracking the state of processes - it couldn't, for example, notice your sshd had crashed and restart it for you. There are plenty of people still running sysvinit (I run it on a system at home, because I'm too lazy to re-write the firewall script), since it remains by and large good enough for a lot of applications.

There have been a number of attempts to improve on sysvinit, the first of which to get significant traction was Upstart, but I think Debian choosing systemd over it (and thus causing Canonical to drop it in favour of systemd) killed it off. Fairly rapidly, systemd has become the default in most distributions. It has a number of key advantages over sysvinit:

• parallel start-up of services
• distro-agnostic service files
• tracks processes belonging to services
• socket management for daemons (facilitating parallel startup, lossless reload, etc)
• service environment control

It has some useful secondary features (starting services on-demand, dealing with changing hardware) which to my mind are great for portables, but less necessary for servers or desktop systems. And while the more-automatic handling of dependencies is a nice idea, it doesn't seem to get all the edge cases right[1].

So far, so good. And, if you look at Lennart Poettering's initial blog post, those are mostly the things he talks about. Since then, though, systemd has considerably extended its scope to include networking, DNS, time, session management, and so on. You can see from Poettering's blog post a year later quite how much systemd is doing compared to other init systems. And there he talks about systemd becoming a comprehensive platform, and driving standardisation.

And while standards are generally a good thing (e.g. POSIX, the FHS), I don't think "the systemd authors decide what linux looks like and call that a standard" is such a good thing. I say that for 3 main reasons: the init system should be a component of a unix system, not the whole system; the systemd authors seem to replace existing unix components without entirely understanding what they did; and the systemd authors are too "my way or the highway".

A traditional unix system is a series of interoperating components, each of which does one or a few things well. I think that's a reasonably successful model, and it gives our users freedom to switch out particular components to suit their needs. One consequence of systemd's mission creep is that it is proving increasingly difficult to ship a distribution where systemd and any other init are both supported. Debian will do so in bullseye, but it's taken quite a lot of work, and I think everyone agrees that there will continue to be tension between supporting alternative init systems and keeping up with systemd's new features and the tighter integration required to take advantage of them.

Systemd has started re-implementing existing unix services - ntp, a local resolver, cron, etc. Its replacements seem sometimes to suffer from a Chesterton's fence problem; the new implementation seems to not fully understand or value what the traditional one did leading to surprising or incorrect behaviour. And yet they become the default almost de facto as part of the advance of systemd. We had to restore ntp on our ceph servers, for example, because systemd-timesyncd didn't provide a good enough time source. We've had some problems with systemd-resolved just giving up, and it has had a number of interop problems (e.g. with dnsmasq). I'm not sure systemd should be replacing extant utilities conventionally understood to have little to do with init (e.g. ntp) rather than fixing them (or starting a new separate project for next-gen-ntp or whatever), but if you're going to, you need to (IMO) understand fully what those things were doing and why. The somewhat cavalier approach to existing utilities is particularly concerning from people who want to make the new standard.

Which brings me to "my way or the highway". I think it's fair to say that Poettering is opinionated about what a linux system should look like, and that he's not that interested in systemd supporting different setups. That results in both some surprising behaviours and often pretty aggressive responses to bug reports[2] (there's even a "lennart-doesnt-like-it" label in the issue tracker). Standards processes can be time-consuming and far from ideal, but they at least try and get a broad feeling for what use-cases are out there. Having everyone standardise on the systemd way where that seems so hostile to alternative models doesn't seem ideal; and some way for people working on alternatives to have a stable interface to work with would be good.

So what? It seems to me like there ought to be space for an init that does a bunch of the good stuff systemd does, but without the endless scope creep; it would also need to understand systemd service files (at least). OpenRC and runit do some but not all of what you'd need (and I think GNU Shepherd may do, too, but that seems quite Guix-specific); both are usable in Debian bullseye at least. I think you'd want a starting feature set something like

• support for systemd service files and sysvinit scripts
• parallelised start-up with dependency management
• tracking of services' processes
• control of services' environments/resource usage
• optional socket / dbus management for services
• tight focus on service (?+boot) management

It strikes me, though, that this is the sort of application that Rust might be good for, which would give you some useful safety features compared to C while still being performant. Maybe someone will be inspired to write one?

In any case, I think it remains important that users continue to have a choice of init system (not least so when the next big thing does come along, it's possible to transition painlessly), and this is not just about supporting a bunch of dinosaurs who won't be parted from sysvinit. I think that's going to mean a continuing process of working out what new systemd features to incorporate and which to refrain from in the interests of supporting a range of init alternatives.

[0] I think the speed argument isn't all that interesting - my laptop boots up in seconds regardless of what init it's running, and my servers at work spend so long doing POST that how long init takes to do its thing is irrelevant
[1] We had some fun with sssd and trying to make sure services that needed it to be actually running (rather than the process having started but not being able to answer queries) worked OK, and then a mistake meant that restarting sssd also restarted the other services unexpectedly.
[2] as a for-instance: if you try and schedule a delayed shutdown and logind is unhappy for some reason, systemd will immediately shutdown (rather than, say, giving the user a choice). Despite a number of people saying this was really not what you wanted, Poettering declared it was clearly the correct behaviour and closed a PR to fix it, leaving downstream maintainers to apply the fix for their users