The history of systemd, and why it won

Why systemd won

The history of systemd starts with two things SysV init could not do. SysV init (System V init, the startup system Linux inherited from AT&T Unix) had no way to describe what a service depends on, and no way to know which processes belong to a service once it was running. systemd answered both with kernel features that a shell script cannot reach: control groups for tracking processes, and pre-opened listening sockets for ordering. The rest of the story is how those two answers spread into the rest of userland, which is where the objections start, and several of the objections were right.

What SysV init actually did

On a SysV system, PID 1 (process ID 1, the first process the kernel starts) read /etc/inittab, picked a runlevel, and ran the scripts for that runlevel. The scripts lived in /etc/init.d/. Symbolic links in /etc/rc3.d/ decided which ones ran and in what order, so /etc/rc3.d/S20nginx pointed at /etc/init.d/nginx and was called with the argument start.

#!/bin/sh
### BEGIN INIT INFO
# Provides:          nginx
# Required-Start:    $local_fs $remote_fs $network $syslog
# Required-Stop:     $local_fs $remote_fs $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
### END INIT INFO

case "$1" in
  start)
    start-stop-daemon --start --quiet --pidfile /run/nginx.pid \
      --exec /usr/sbin/nginx
    ;;
  stop)
    start-stop-daemon --stop --quiet --retry TERM/30/KILL/5 \
      --pidfile /run/nginx.pid
    ;;
esac

The 20 in S20nginx is a position, not a dependency. It says this script runs after S19 and before S21. It does not say why, so nothing can check it, and nothing can safely run two unrelated scripts at the same time without a human deciding it is safe.

The rc program ran each script in turn and waited for it to exit. A script that blocked for thirty seconds waiting for a network address blocked the whole boot for thirty seconds, even for services that never touch the network.

The LSB (Linux Standard Base) header at the top of that script was an attempt to fix this from the inside. Debian 6.0 in 2011 made insserv the default: it read Required-Start from every script, built a graph, and renumbered the symlinks. Debian could then run independent scripts at the same time with startpar. That helped, and it did not reach the deeper problem. The dependency was still on a script exiting. S20nginx returning 0 means a shell function returned. It does not mean nginx is accepting connections.

The five things no init script could fix

• Parallel startup. Ordering by filename is a total order over every service on the machine, so the boot is as slow as the sum of its parts.
• Readiness. A start script exits when it has forked the daemon, not when the daemon can serve a request, so the next script often starts too early.
• Supervision. A daemon forks twice and its parent exits, which detaches it from the terminal and reparents it to PID 1. init sees a child exit and has no reliable link to the process that survived.
• On-demand start. inetd (the internet super-server) could launch a daemon when a connection arrived, but it was a separate system with its own configuration file, and it did nothing about the ordering of everything else at boot.
• Resource control. Nothing in an init script could bound a service's memory or its share of the CPU. ulimit applied to one process and nice only touched the scheduler, so a runaway child of a service looked like any other process on the box.

The supervision gap is the one that hurt daily. The PID file was the workaround: the daemon wrote its process ID to /run/nginx.pid, and the stop function read the file back. If the daemon was killed hard, the file stayed behind. The kernel then reused that number for something else, and start-stop-daemon --stop --pidfile sent a signal to whatever now owned it. A stale PID file is how an init script kills the wrong process.

launchd solved the socket problem first

Apple shipped launchd in Mac OS X 10.4 in 2005, written by Dave Zarzycki. One process replaced init, rc, xinetd, crond and watchdogd.

The idea worth copying was socket activation. launchd creates every listening socket first, then starts the daemons. A client that connects to a daemon which has not started yet does not get a connection refused, because the kernel holds the connection in that socket's backlog queue until the daemon calls accept(). Ordering between two daemons stops being something a human declares. The socket handles it.

launchd was built on Mach IPC (inter-process communication), which belongs to Apple's XNU kernel and has no Linux equivalent. Porting the code was never realistic. The idea crossed anyway.

Upstart made events the unit of work

Canonical's Upstart, written by Scott James Remnant, shipped in Ubuntu 6.10 in October 2006. Fedora 9 through Fedora 14 used it, as did RHEL 6 and Chrome OS. It replaced the runlevel with an event, and a job said which events should start and stop it.

# /etc/init/example.conf
start on filesystem and net-device-up IFACE!=lo
stop on runlevel [!2345]
respawn
respawn limit 10 5
exec /usr/local/bin/exampled

Two problems showed up as the job count grew. The first is direction. A job says "start me when this happens", so the knowledge of what depends on what sits in the wrong file: a service knows what it needs, and it cannot know who will need it next year. Adding a service often meant editing an existing job so it emitted a new event.

The second is tracking. Upstart followed a forking daemon by counting fork() calls with ptrace, which you configured as expect fork or expect daemon. Guess the number of forks wrong and Upstart supervises a process that has already exited, or waits for a fork that already happened. The symptom is initctl start hanging with no error, which the job file gives you no way to explain.

Upstart also required contributors to sign Canonical's contributor agreement. That was not an engineering fault, and it did shape who worked on it.

Rethinking PID 1, April 2010

On 30 April 2010 Lennart Poettering published a post called "Rethinking PID 1". Kay Sievers worked on the project with him. The argument had four parts.

• Start less. Many services can wait until something actually asks for them.
• Stop declaring order where a socket can imply it. Open all the sockets in one pass, then start everything at once.
• Track processes with control groups instead of PID files.
• Describe a service in a declarative file, so one description works on every distribution.

The first release followed that year. Fedora 14 shipped systemd as an option in November 2010, and Fedora 15 made it the default in May 2011.

Why cgroups made supervision reliable

A cgroup (control group) is a kernel feature for grouping processes, merged into Linux 2.6.24 in 2008. systemd puts every service in its own cgroup. A child inherits the cgroup of its parent, and an unprivileged process cannot move itself out of one. Double forking therefore hides nothing: PID 1 holds the exact set of processes that belong to a unit, at all times. Stopping a service means killing everything in its cgroup, which is what the default KillMode=control-group does.

systemctl status prints that group:

● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-08-11 09:14:22 UTC; 3min ago
   Main PID: 1042 (nginx)
      Tasks: 3 (limit: 4653)
     Memory: 6.1M (peak: 7.4M)
     CGroup: /system.slice/nginx.service
             ├─1042 "nginx: master process /usr/sbin/nginx"
             ├─1043 "nginx: worker process"
             └─1044 "nginx: worker process"

That block is the whole answer to the stale PID file. There is no file to go stale, because the list is kernel state.

The same tree carries limits, because cgroups were built for accounting before anyone used them for tracking. MemoryMax=, CPUQuota= and TasksMax= are one line each. Putting a hard memory and CPU cap on a service is a drop-in file today, and in 2009 it was a patch to a shell script that nobody wrote.

Why every distribution switched between 2011 and 2015

• Fedora 15, May 2011.
• openSUSE 12.1, November 2011.
• Mageia 2, May 2012.
• Arch Linux, default for new installations from October 2012.
• RHEL 7, June 2014.
• SLES 12, October 2014.
• Debian 8, April 2015.
• Ubuntu 15.04, April 2015.

The reasons were mostly boring, which is why the switch was fast.

• One unit file works on every distribution, so upstream projects started shipping a .service file and distributions stopped maintaining a shell script per package per release.
• Desktop session tracking moved to systemd-logind after ConsoleKit stopped being maintained around 2012. GNOME needed logind, so a distribution without systemd had to find a replacement. That replacement, elogind, is systemd's logind extracted and maintained separately.
• udev, the device manager, was merged into the systemd source tree in April 2012. Distributions shipping udev were now tracking systemd's repository. Gentoo forked eudev in response.
• Containers made reliable process tracking and per-service limits matter more, since both are cgroup features. The question of which supervisor owns a container process is still live whenever you make a Docker Compose stack come back after a reboot.

Debian's decision was the loud one. The Technical Committee voted in February 2014, the vote was tied, and the chair, Bdale Garbee, cast the deciding vote for systemd. Ubuntu announced days later that it would follow Debian rather than continue with Upstart. A group of Debian developers forked the distribution as Devuan in November 2014 and released Devuan 1.0 in May 2017.

The objections, stated fairly

Scope. One project now ships PID 1, the logging daemon, login session management, the device manager, a network configuration daemon, a DNS (domain name system) resolver, an NTP (network time protocol) client, a container runner and a boot loader. The usual defence, that these are separate binaries you do not have to install, is true and it does not answer the objection. Once a desktop needs logind, and logind is released from systemd's tree, the choice is no longer free. That is what coupling meant in the argument, and it happened.

The binary journal. journald writes an indexed binary format instead of plain text. You get things text never gave you: per-unit and per-priority filtering, structured fields, and metadata that the sending program cannot forge, because journald records the unit and cgroup itself. journalctl -u nginx -p err --since "-1h" replaces a grep with a date regular expression. The cost is real too. On a machine that will not boot, you cannot read the log with less from a rescue shell. You point journalctl at the mounted disk instead:

sudo journalctl --directory /mnt/var/log/journal --boot -1 --priority err

There is a second trap here that catches people once. journald keeps logs in /run/log/journal, which is memory, unless /var/log/journal exists. On a box where it does not, journalctl -b -1 has nothing to show after a reboot, which is the exact moment you wanted it. Check and fix it:

journalctl --disk-usage
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

journalctl --disk-usage should now report archived journals under /var/log/journal. If you want plain text as well, set ForwardToSyslog=yes in /etc/systemd/journald.conf and keep rsyslog installed.

Debuggability of the boot. When a unit hangs, the console shows one line and nothing else:

[  *** ] A start job is running for Wait for Network to be Configured (1min 32s / no limit)

The tools to go further do exist: systemctl list-jobs while it is stuck, systemd-analyze blame and systemd-analyze critical-chain afterwards, and systemd.log_level=debug on the kernel command line. The fair version of the complaint is that an init script could be read from top to bottom by anyone who knew sh, while a stuck unit requires knowing which of a dozen commands to reach for. That is a real cost. It is paid once per administrator, and it was paid by a lot of administrators at the same time.

A default that changes for everybody. systemd 230 in 2016 changed logind's default so leftover user processes were killed at logout. Detached tmux and screen sessions died when the session that started them ended. Distributions shipped KillUserProcesses=no in /etc/systemd/logind.conf, and the supported answer is loginctl enable-linger <user>. One default in one project changed a habit that millions of people relied on, which is what "too much of userland in one place" means in practice.

A default dependency is a security surface. In March 2024 the backdoor in xz-utils targeted sshd on Debian and Ubuntu. Upstream OpenSSH does not link libsystemd. Those distributions patched it in so sshd could report readiness to systemd, and libsystemd pulled in liblzma, where the backdoor lived. The readiness protocol itself is a single datagram sent to the socket named in $NOTIFY_SOCKET, so no library was ever required for it. systemd's response was to load compression libraries with dlopen, so they are no longer linked in by default. A related class of bug shows the same shape: in 2017 a User= value starting with a digit was treated as invalid and the unit ran as root instead of failing, so a typo became a privilege escalation. Later versions refuse to start the unit.

History at your own systemctl prompt

Every problem above is now one directive in a file you can read.

• Serial boot became After= and Wants=, and systemd-analyze critical-chain shows what actually held your boot.
• Readiness became Type=notify, where the service writes READY=1 to $NOTIFY_SOCKET when it can serve. Type=forking with PIDFile= still exists for old daemons, and it is the type that fails with start operation timed out. Terminating. when the PID file never appears.
• Supervision became the cgroup, so Restart=on-failure with RestartSec= replaces a wrapper script, and StartLimitBurst= stops a crash loop from running forever.
• inetd became a .socket unit sitting next to the .service unit.
• ulimit became MemoryMax=, CPUQuota= and TasksMax=.
• The su - appuser -c line in an init script became User=, NoNewPrivileges=yes and ProtectSystem=strict, so running a service as an unprivileged user is the default shape of a unit rather than extra work.

[Unit]
Description=Example API
Wants=network-online.target
After=network-online.target postgresql.service
Requires=postgresql.service

[Service]
Type=notify
ExecStart=/usr/local/bin/exampled
Restart=on-failure
RestartSec=2
MemoryMax=512M
CPUQuota=50%
TasksMax=128
User=exampled
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
StateDirectory=exampled

[Install]
WantedBy=multi-user.target

One line in that file is the mistake everybody makes once. Requires=postgresql.service is a requirement, not an ordering: it says your unit fails if Postgres fails, and it does not say start Postgres first. Without After=postgresql.service both start at the same moment, and your service connects to a port nothing is listening on yet. The two are separate on purpose, because you sometimes want one without the other. ProtectSystem=strict mounts the file system read-only for this service, which is why StateDirectory= is there: it gives the service one writable path under /var/lib.

The clearest place to see 2005 on a 2026 server is SSH on Ubuntu 24.04, which ships systemd 255 as of August 2026. ssh.service is socket activated by default: ssh.socket holds the listening socket, and sshd starts when a connection arrives. So Port 2222 in /etc/ssh/sshd_config has no effect, because sshd is not the process that opened the port. The change belongs in the socket unit.

sudo systemctl edit ssh.socket

[Socket]
ListenStream=
ListenStream=2222

The empty ListenStream= clears the value inherited from the packaged unit. Leave it out and you get both ports, because systemd appends to a list rather than replacing it. Then apply and check, keeping a second SSH session open the whole time:

sudo systemctl daemon-reload
sudo systemctl restart ssh.socket
sudo ss -lntp | grep 2222

ss should list one socket on port 2222 owned by systemd, not by sshd. That is the launchd design, twenty years later, on your VPS. If you prefer the old behaviour, sudo systemctl disable --now ssh.socket followed by sudo systemctl enable --now ssh.service gives you a long running sshd that reads Port from its own config again.

Which of these details you meet depends on the release you run, so it is worth knowing the difference between an LTS and an interim Ubuntu release before you plan an upgrade. Across more than one machine, the fact that a unit file is identical everywhere is the reason managing several servers from one place is a configuration problem now instead of a shell scripting problem. And when you write your own units, the service and timer pair does the job you would have split between an init script and a cron line in 2009.

FAQ

Why did Linux distributions replace SysV init with systemd?

For two engineering reasons and one maintenance reason. SysV init ordered services by filename, which is a position rather than a dependency, and it lost track of any daemon that forked away from its parent, which is why stale PID files could kill the wrong process. systemd solved ordering with socket activation and dependency directives, and solved tracking with control groups. The maintenance reason decided the speed: one unit file works on every distribution, so upstream projects shipped a .service file and distribution maintainers stopped writing a shell script per package. Fedora 15 switched in May 2011 and Ubuntu 15.04 was the last big holdout, in April 2015.

Is systemd one giant binary?

No. The source tree builds many separate programs. PID 1 is /usr/lib/systemd/systemd, while journald, logind and udevd are separate processes with their own binaries; run ls /usr/lib/systemd/ to see them on your own box. The criticism that survives is about release coupling rather than binary size: these programs are released together and share private interfaces, so distributions tend to take them as a set, and software such as GNOME came to expect logind specifically.

Can I still run Linux without systemd?

Yes. Devuan ships sysvinit, Gentoo defaults to OpenRC, Void uses runit, Alpine uses busybox init with OpenRC, and Slackware keeps BSD style scripts. The cost is compatibility work. Desktop software that expects logind needs elogind, which is systemd's logind maintained as a standalone package, and a growing amount of server software now ships only a .service file, so you write and maintain the startup script yourself.

Why is the journal binary instead of a plain text file?

Because journald stores structured fields with an index, which gives per-unit filtering, priority filtering, and metadata the sending program cannot forge: journald records the unit, the cgroup and the real UID itself rather than trusting the log line. The price is that you need journalctl to read it, including from a rescue system, where you point it at the mounted disk with journalctl --directory /mnt/var/log/journal. If you want text as well, set ForwardToSyslog=yes in /etc/systemd/journald.conf.

What replaced editing my /etc/init.d script?

Drop-in files. Do not edit the unit in /usr/lib/systemd/system/, because a package upgrade overwrites it. Run sudo systemctl edit nginx.service and systemd creates /etc/systemd/system/nginx.service.d/override.conf, which is merged over the packaged unit. systemctl cat nginx.service shows the merged result, and systemd-delta lists every override on the machine. After any edit made by hand, run sudo systemctl daemon-reload, or the next command prints Warning: The unit file, source configuration file or drop-ins of nginx.service changed on disk.