Why systemd Win and SysV init No Fit Compete

Why systemd win

The history of systemd start with two things wey SysV init no fit do. SysV init (System V init, the startup system wey Linux inherit from AT&T Unix) no get way to describe wetin service depend on, and no get way to know which processes belong to service after e don start. systemd answer both with kernel features wey shell script no fit reach: control groups to track processes, and pre-opened listening sockets to arrange startup order. The rest of the story na how those two answers spread enter the rest of userland, and na there the objections start. Several of those objections correct.

Wetin SysV init really dey do

For SysV system, PID 1 (process ID 1, the first process wey kernel start) read /etc/inittab, pick a runlevel, then run the scripts for that runlevel. The scripts dey inside /etc/init.d/. Symbolic links for /etc/rc3.d/ decide which ones go run and the order wey dem go run, so /etc/rc3.d/S20nginx point to /etc/init.d/nginx and dem call am with argument start.

#!/bin/sh
### BEGIN INIT INFO
# Provides:          nginx
# Required-Start:    $local_fs $remote_fs $network $syslog
# Required-Stop:     $local_fs $remote_fs $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
### END INIT INFO

case "$1" in
  start)
    start-stop-daemon --start --quiet --pidfile /run/nginx.pid \
      --exec /usr/sbin/nginx
    ;;
  stop)
    start-stop-daemon --stop --quiet --retry TERM/30/KILL/5 \
      --pidfile /run/nginx.pid
    ;;
esac

The 20 for S20nginx na position, e no be dependency. E mean say this script go run after S19 and before S21. E no explain why, so nothing fit check am. Nothing fit safely run two unrelated scripts at the same time unless person first decide say e safe.

The rc program run each script one after another and wait for am to exit. If script block for thirty seconds because e dey wait for network address, e go block the whole boot for thirty seconds, even for services wey no dey use network.

The LSB (Linux Standard Base) header for top of that script na attempt to fix this problem from inside. Debian 6.0 for 2011 make insserv the default: e read Required-Start from every script, build a graph, then renumber the symlinks. Debian fit then run independent scripts at the same time with startpar. That one help, but e no solve the deeper problem. The dependency still depend on script to exit. S20nginx returning 0 mean say shell function return successfully. E no mean say nginx dey accept connections.

Wetin be the five things init script no fit fix

• Parallel startup. When you order every service for the machine by filename, you create one complete order, so boot time na the total time of all the services together.
• Readiness. Start script dey finish once e fork the daemon, no be when daemon don ready to serve request. So the next script often dey start too early.
• Supervision. Daemon fit fork two times, then the parent go exit. This one detach am from the terminal and make PID 1 become e new parent. init go see say child don exit, but e no get reliable link to the process wey remain alive.
• On-demand start. inetd (the internet super-server) fit launch daemon when connection arrive. But na separate system with e own configuration file, and e no solve the startup order for other services.
• Resource control. Nothing inside init script fit limit service memory or how much CPU e dey use. ulimit apply to one process, while nice only affect the scheduler. So if service child process run out of control, e look like any other process for the machine.

Na the supervision gap cause the daily wahala pass. PID file na the workaround: daemon write e process ID to /run/nginx.pid, then stop function read the file back. If daemon get hard-kill, the file remain. Kernel fit reuse that number for another process, then start-stop-daemon --stop --pidfile go send signal to the process wey don get the number. Stale PID file na how init script fit kill the wrong process.

launchd solve socket problem first

Apple release launchd for Mac OS X 10.4 for 2005, and Dave Zarzycki write am. One process replace init, rc, xinetd, crond and watchdogd.

The idea wey make sense to copy na socket activation. launchd dey create every listening socket first, then e start the daemons. If client connect to daemon wey never start yet, e no get connection refused, because kernel hold the connection for that socket backlog queue until daemon call accept(). Ordering between two daemons no longer depend on wetin human declare. The socket handle am.

launchd build on Mach IPC (inter-process communication), wey belong to Apple's XNU kernel and no get Linux equivalent. Porting the code no ever realistic. But dem still carry the idea go elsewhere.

Upstart make events the main unit of work

Canonical’s Upstart, wey Scott James Remnant write, release for Ubuntu 6.10 for October 2006. Fedora 9 reach Fedora 14 use am too, and RHEL 6 plus Chrome OS use am. E replace runlevel with event, and job talk which events suppose start and stop am.

# /etc/init/example.conf
start on filesystem and net-device-up IFACE!=lo
stop on runlevel [!2345]
respawn
respawn limit 10 5
exec /usr/local/bin/exampled

Two problems show as the number of jobs grow. The first one na direction. Job dey talk say “start me when this happen”, so the information about wetin depend on wetin dey the wrong file: service know wetin e need, but e no fit know who go need am next year. To add service, you often need edit existing job so e go emit new event.

The second one na tracking. Upstart follow forking daemon by counting fork() calls with ptrace, wey you configure as expect fork or expect daemon. If you guess the fork number wrong, Upstart go supervise process wey don already exit, or wait for fork wey don already happen. The symptom na initctl start hanging without error, and the job file no give you any way to explain am.

Upstart also require contributors to sign Canonical’s contributor agreement. That one no be engineering fault, but e affect who work on am.

Rethink PID 1, April 2010

On 30 April 2010 Lennart Poettering publish one post wey dem call "Rethinking PID 1". Kay Sievers work with am for the project. The argument get four parts.

• Start fewer things. Plenty services fit wait until something really ask for dem.
• Stop declaring order when socket fit imply am. Open all the sockets for one pass, then start everything together.
• Track processes with control groups instead of PID files.
• Describe service inside one declarative file, so one description fit work for every distribution.

The first release follow that year. Fedora 14 release systemd as an option for November 2010, and Fedora 15 make am the default for May 2011.

Why cgroups make supervision reliable

A cgroup (control group) na kernel feature wey dey group processes. Dem merge am into Linux 2.6.24 for 2008. systemd dey put every service for im own cgroup. Child process dey inherit the cgroup of im parent, and unprivileged process no fit move itself comot from one. So double forking no fit hide anything: PID 1 dey hold the exact set of processes wey belong to one unit, all the time. To stop service mean say kill everything wey dey inside im cgroup. Na this the default KillMode=control-group dey do.

systemctl status dey print that group:

● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-08-11 09:14:22 UTC; 3min ago
   Main PID: 1042 (nginx)
      Tasks: 3 (limit: 4653)
     Memory: 6.1M (peak: 7.4M)
     CGroup: /system.slice/nginx.service
             ├─1042 "nginx: master process /usr/sbin/nginx"
             ├─1043 "nginx: worker process"
             └─1044 "nginx: worker process"

That block na the complete answer to stale PID file. No file dey wey fit become stale, because the list na kernel state.

The same tree dey carry limits too, because dem build cgroups first for accounting before anybody start to use dem for tracking. MemoryMax=, CPUQuota= and TasksMax= each dey take one line. How to put hard memory and CPU limit for service na drop-in file today. For 2009, e for be patch to shell script wey nobody write.

Why every distribution switched between 2011 and 2015

• Fedora 15, May 2011.
• openSUSE 12.1, November 2011.
• Mageia 2, May 2012.
• Arch Linux, default for new installations from October 2012.
• RHEL 7, June 2014.
• SLES 12, October 2014.
• Debian 8, April 2015.
• Ubuntu 15.04, April 2015.

The reasons no too interesting, na why the switch happen quick.

• One unit file dey work for every distribution, so upstream projects start shipping one .service file, and distributions stop maintaining one shell script for every package and every release.
• Desktop session tracking move go systemd-logind after ConsoleKit stop being maintained around 2012. GNOME need logind, so any distribution wey no use systemd need find replacement. That replacement, elogind, na systemd logind wey dem extract and maintain separately.
• udev, wey be the device manager, join the systemd source tree for April 2012. Distributions wey ship udev now dey track systemd's repository. Gentoo fork eudev because of this.
• Containers make reliable process tracking and per-service limits more important, because both na cgroup features. The question of which supervisor dey own container process still dey open anytime you make Docker Compose stack come back after reboot.

Debian decision na the one wey cause plenty discussion. The Technical Committee vote for February 2014, the vote tie, and the chair, Bdale Garbee, cast the deciding vote for systemd. Ubuntu announce some days later say e go follow Debian instead of continuing with Upstart. One group of Debian developers fork the distribution as Devuan for November 2014 and release Devuan 1.0 for May 2017.

Objection dem talk about, fairly

Scope. One project now dey ship PID 1, logging daemon, login session management, device manager, network configuration daemon, DNS (domain name system) resolver, NTP (network time protocol) client, container runner and boot loader. The usual defence say these na separate binaries wey you no need install dey true, but e no answer the objection. Once desktop need logind, and logind come from systemd's tree, the choice no longer free. Na this coupling mean for the argument, and e happen.

The binary journal. journald dey write indexed binary format instead of plain text. You get things wey text never give you: filtering by unit and priority, structured fields, and metadata wey the sending program no fit forge, because journald dey record the unit and cgroup by itself. journalctl -u nginx -p err --since "-1h" dey replace grep with date regular expression. The cost dey real too. For machine wey no go boot, you no fit read the log with less from rescue shell. Point journalctl to the mounted disk instead:

sudo journalctl --directory /mnt/var/log/journal --boot -1 --priority err

Another trap dey here wey fit catch person once. journald dey keep logs for /run/log/journal, wey na memory, unless /var/log/journal dey exist. For machine wey this no dey, journalctl -b -1 get nothing to show after reboot, exactly when you need am. Check and fix am:

journalctl --disk-usage
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

journalctl --disk-usage suppose now report archived journals under /var/log/journal. If you want plain text too, set ForwardToSyslog=yes inside /etc/systemd/journald.conf and keep rsyslog installed.

Debuggability of the boot. When unit hang, console dey show one line and nothing else:

[  *** ] A start job is running for Wait for Network to be Configured (1min 32s / no limit)

Tools to investigate further dey exist: systemctl list-jobs while e still dey stuck, systemd-analyze blame and systemd-analyze critical-chain afterwards, and systemd.log_level=debug for the kernel command line. The fair way to state the complaint be say anybody wey know sh fit read init script from top to bottom, but stuck unit require you know which one among dozen commands to use. This na real cost. Each administrator pay am once, and plenty administrators pay am at the same time.

A default wey change for everybody. systemd 230 for 2016 change logind default, so leftover user processes go die when person logout. Detached tmux and screen sessions die when the session wey start dem end. Distributions ship KillUserProcesses=no inside /etc/systemd/logind.conf, and the supported answer na loginctl enable-linger <user>. One default for one project change habit wey millions of people rely on, and na this "too much of userland in one place" mean for practice.

A default dependency na security surface. For March 2024, the backdoor inside xz-utils target sshd for Debian and Ubuntu. Upstream OpenSSH no link libsystemd. Those distributions patch am so sshd fit report readiness to systemd, and libsystemd pull in liblzma, where the backdoor dey. The readiness protocol itself na one datagram wey dem send to the socket named inside $NOTIFY_SOCKET, so library never really necessary for am. systemd response na to load compression libraries with dlopen, so dem no longer link in by default. Related bug class show the same pattern: for 2017, a User= value wey start with digit dey treated as invalid, and the unit run as root instead of failing, so typo turn privilege escalation. Later versions no gree start the unit.

History for your own systemctl prompt

Every problem wey dey above don become one directive for file wey you fit read.

• Serial boot become After= and Wants=, and systemd-analyze critical-chain dey show wetin really hold your boot.
• Readiness become Type=notify, where service dey write READY=1 to $NOTIFY_SOCKET when e fit serve requests. Type=forking with PIDFile= still dey for old daemons, and na this type dey fail with start operation timed out. Terminating. when PID file no ever show.
• Supervision become the cgroup, so Restart=on-failure with RestartSec= replace wrapper script, and StartLimitBurst= stop crash loop from running forever.
• inetd become a .socket unit wey dey beside .service unit.
• ulimit become MemoryMax=, CPUQuota= and TasksMax=.
• The su - appuser -c line for init script become User=, NoNewPrivileges=yes and ProtectSystem=strict, so running service as unprivileged user na the normal unit structure, no be extra work.

[Unit]
Description=Example API
Wants=network-online.target
After=network-online.target postgresql.service
Requires=postgresql.service

[Service]
Type=notify
ExecStart=/usr/local/bin/exampled
Restart=on-failure
RestartSec=2
MemoryMax=512M
CPUQuota=50%
TasksMax=128
User=exampled
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
StateDirectory=exampled

[Install]
WantedBy=multi-user.target

One line for that file na the mistake wey almost everybody go make once. Requires=postgresql.service na requirement, no be ordering: e mean say your unit go fail if Postgres fail, but e no mean say systemd go start Postgres first. Without After=postgresql.service, both go start for the same time, and your service go try connect to port wey nothing dey listen on yet. Dem separate on purpose, because sometimes you want one without the other. ProtectSystem=strict mount the file system as read-only for this service, na why StateDirectory= dey there: e give the service one writable path under /var/lib.

The clearest place to see 2005 for 2026 server na SSH on Ubuntu 24.04, wey ships systemd 255 as of August 2026. ssh.service dey socket activated by default: ssh.socket hold the listening socket, and sshd start when connection arrive. So Port 2222 for /etc/ssh/sshd_config no get effect, because sshd no be the process wey open the port. The change belong inside socket unit.

sudo systemctl edit ssh.socket

[Socket]
ListenStream=
ListenStream=2222

The empty ListenStream= clear the value wey packaged unit inherit. If you leave am out, you go get both ports, because systemd dey append to list instead of replacing am. Then apply the change and check am, while you keep second SSH session open throughout:

sudo systemctl daemon-reload
sudo systemctl restart ssh.socket
sudo ss -lntp | grep 2222

ss suppose list one socket for port 2222 wey systemd own, not sshd. Na the launchd design be this, twenty years later, for your VPS. If you prefer the old behaviour, sudo systemctl disable --now ssh.socket followed by sudo systemctl enable --now ssh.service go give you long-running sshd wey read Port from its own config again.

Which details you go meet depend on the release wey you dey run, so e good make you know the difference between LTS and interim Ubuntu release before you plan upgrade. For more than one machine, the fact say unit file identical everywhere na why managing several servers from one place don become configuration problem instead of shell scripting problem. And when you write your own units, the service and timer pair go do the work wey you for split between init script and cron line for 2009.

FAQ

Why Linux distributions replace SysV init with systemd?

Na two engineering reasons and one maintenance reason. SysV init arrange services by filename. That na position, no be dependency. E also lose track of daemon wey fork away from im parent. Na why stale PID files fit kill wrong process. systemd solve ordering with socket activation and dependency directives. E solve process tracking with control groups. The maintenance reason decide the speed: one unit file work for every distribution. So upstream projects ship a .service file, and distribution maintainers stop writing one shell script for each package. Fedora 15 switch for May 2011, and Ubuntu 15.04 be the last big holdout for April 2015.

systemd na one giant binary?

No. The source tree build plenty separate programs. PID 1 na /usr/lib/systemd/systemd, while journald, logind, and udevd dey run as separate processes with their own binaries. Run ls /usr/lib/systemd/ to see dem for your own box. The criticism wey remain concern release coupling, no be binary size. Dem release these programs together and dem share private interfaces. Because of this, distributions normally take dem as one set. Software like GNOME come to expect logind specifically.

I still fit run Linux without systemd?

Yes. Devuan ships sysvinit, Gentoo default to OpenRC, Void use runit, Alpine use busybox init with OpenRC, and Slackware keep BSD style scripts. The cost na compatibility work. Desktop software wey expect logind need elogind. This na systemd's logind wey dem maintain as standalone package. More server software now ships only a .service file, so you go write and maintain the startup script by yourself.

Why the journal be binary instead of plain text file?

Because journald store structured fields with an index. This give per-unit filtering, priority filtering, and metadata wey the sending program no fit forge. journald record the unit, the cgroup, and the real UID by itself instead of trusting the log line. The price be say you need journalctl to read am, including from rescue system. For rescue system, point am to the mounted disk with journalctl --directory /mnt/var/log/journal. If you want text too, set ForwardToSyslog=yes inside /etc/systemd/journald.conf.

Wetin replace editing my /etc/init.d script?

Drop-in files. No edit the unit inside /usr/lib/systemd/system/, because package upgrade go overwrite am. Run sudo systemctl edit nginx.service and systemd go create /etc/systemd/system/nginx.service.d/override.conf. systemd go merge am over the packaged unit. systemctl cat nginx.service show the merged result, while systemd-delta list every override for the machine. After any edit wey you make by hand, run sudo systemctl daemon-reload, otherwise the next command go print Warning: The unit file, source configuration file or drop-ins of nginx.service changed on disk.