Why systemd did not restart your service

The short answer: systemd restart policies watch one process

systemd restart policies watch one process per unit: the main process. Restart= reads the exit status of that one process and nothing else. A unit's control group can hold twenty processes, one of them can die, and the unit stays active (running) because the main process is still there. Nothing failed as far as systemd is concerned, so nothing is restarted.

systemd does know about the other processes. It kills them when the unit stops, it counts their memory against the unit's limits, it applies the unit's CPU quota to them, and it prints them in systemctl status. It just never reads their exit status. The restart logic and the cgroup are two different things, and most of this guide is about the gap between them.

What the cgroup holds, and what the restart logic reads

A cgroup (control group) is a kernel object that owns a set of processes. Every service unit gets one, named after the unit. A process cannot leave it. Children inherit the cgroup of their parent, and an unprivileged process cannot move itself somewhere else. That is why systemd can clean up a daemon that forks twice, which old init scripts could never do reliably.

Look at both facts side by side:

systemd-cgls --unit myapp.service
systemctl show -p MainPID -p NRestarts -p Restart -p RestartUSec myapp.service

systemd-cgls lists every process in the unit. MainPID is the single number the restart policy reads. When those two disagree with your mental model, the disagreement is the bug. MainPID=0 is worse than a wrong PID: it means systemd is tracking nothing at all, so no Restart= value can ever fire.

There is one real exception to the main-process rule. If the kernel out-of-memory killer kills any process inside the unit's cgroup, systemd sees it, because it watches the cgroup's memory.events file. OOMPolicy= decides what happens next, and its default is stop: the whole unit is stopped, the result is recorded as oom-kill, and that counts as a failure, so Restart=on-failure fires. The journal says it plainly.

myapp.service: A process of this unit has been killed by the OOM killer.
myapp.service: Failed with result 'oom-kill'.

So a child killed for memory does take the unit down, while the same child dying from a segmentation fault does not. If you set memory limits on a unit, read how MemoryMax and CPUQuota apply to a unit's cgroup before tuning the restart policy, because those two features meet here and nowhere else.

How Type= picks the main process

Type= in the [Service] section is not only about start-up ordering. It is the rule that decides which PID (process ID) becomes MainPID, which is the same as deciding what Restart= is able to see.

• Type=simple is the default. The process systemd forks from ExecStart= is the main process. systemd marks the unit as started immediately, before it knows whether the exec even worked. A typo in the binary path gives you a start job that succeeds, and then Main process exited, code=exited, status=203/EXEC a moment later.
• Type=exec behaves like simple, except the start job waits until the exec has succeeded. That turns the typo above into an honest start failure. It needs systemd 240 or newer, which every supported distribution has. Prefer it to simple.
• Type=forking expects the process from ExecStart= to fork a background daemon and then exit. systemd waits for the parent to exit, then looks for the real daemon. Give it PIDFile=. Without one, GuessMainPID= (on by default) works only when exactly one process is left in the cgroup. Leave two behind and MainPID stays 0.
• Type=notify means the service calls sd_notify(3) and sends READY=1 when it can serve traffic. It may also send MAINPID= to hand systemd a different process to track. NotifyAccess= defaults to main, so a notification sent by a child is ignored and the journal names the PID it came from.
• Type=oneshot has no lasting main process. The unit goes inactive as soon as ExecStart= finishes, unless you set RemainAfterExit=yes. Restart=always and Restart=on-success are refused here, with the message Service has Restart= set to either always or on-success, which isn't allowed for Type=oneshot services. Refusing. The other values, including on-failure, are accepted.

Two Type=forking errors are worth memorising, because each leaves you with a unit that looks broken for no visible reason:

myapp.service: Can't open PID file /run/myapp.pid (yet?) after start: No such file or directory
myapp.service: New main PID 4711 does not belong to service, and PID file is not owned by root. Refusing.

The first means the daemon writes its PID file somewhere else, or writes it later than systemd looks. The second means the PID file names a process outside the unit's cgroup, which systemd refuses to adopt, because a writable PID file would otherwise become a way to make systemd send signals to any process on the box.

Why a wrapper script hides the death of its children

Here is the shape that produces the question in the title.

#!/bin/bash
/usr/local/bin/myapp-web &
/usr/local/bin/myapp-worker &
wait

The unit is Type=simple, so the main process is the shell. wait with no arguments returns only after every child has exited. Kill the worker and the shell keeps waiting for the web process, so the shell does not exit, so MainPID does not exit, so Restart= is never consulted. The cgroup now holds one process fewer, systemctl status prints the shorter tree, and the unit is still active (running). Nothing in systemd watches that tree for changes.

A second version of the same mistake is quieter:

ExecStart=/bin/sh -c 'export APP_ENV=production; /usr/local/bin/myapp'

The main process is the shell, not myapp. On systemctl stop, systemd sends SIGTERM to the main process, and a shell waiting on a foreground child does not pass the signal on. The stop then takes the full TimeoutStopSec, 90 seconds by default, and ends like this:

myapp.service: State 'stop-sigterm' timed out. Killing.
myapp.service: Killing process 4711 (myapp) with signal SIGKILL.

The fix is exec. Write exec /usr/local/bin/myapp and the shell is replaced by the program, so MainPID is the program and signals reach it. Better still, delete the shell and use Environment= or EnvironmentFile= in the unit. Note that this bug hides itself when the -c string holds a single command, because bash and dash both optimise that case into a direct exec. Add a second command to the string and the shell stays alive in front of your program.

The full Restart= vocabulary, and when on-failure beats always

Restart= takes one of seven values, and the distinction that separates them is what counts as clean. systemd treats exit code 0, any code listed in SuccessExitStatus=, and the signals SIGHUP, SIGINT, SIGTERM and SIGPIPE as a clean exit. Everything else, SIGKILL and SIGSEGV included, is unclean.

• no is the default. The unit never restarts itself, which is why a unit with no Restart= line dies at the first crash and stays dead.
• on-success restarts only after a clean exit.
• on-failure restarts on a non-zero exit code, an unclean signal, a start or stop timeout, or a watchdog expiry.
• on-abnormal restarts on an unclean signal, a timeout or a watchdog expiry, but never on a plain non-zero exit code.
• on-abort restarts only on an unclean signal, which means a crash.
• on-watchdog restarts only when WatchdogSec= expires.
• always restarts after every case above, including a clean exit with status 0.

on-failure is the right default for a long-running daemon. It brings back a crash and it leaves a deliberate exit 0 alone. always belongs to a program that exits cleanly for reasons outside its control, such as a tunnel client that returns 0 when the far end disconnects. The cost of always is that it hides bugs: a service that starts, reads a broken config file, logs the error and exits 0 will loop forever, and the only sign is a restart counter climbing.

SuccessExitStatus= moves the line between clean and unclean. Borg exits 1 for warnings and 2 for errors, so a backup unit without SuccessExitStatus=1 is marked failed every time it skips one unreadable file. RestartPreventExitStatus= lists codes that block a restart even under always, which is the clean way for a program to say it should not come back. RestartForceExitStatus= does the opposite. A backup job belongs in a Type=oneshot unit driven by a timer rather than in a restart loop, and the service and timer pair that runs a job on a schedule is the shape to copy there.

One warning about testing. Killing your service with plain kill <pid> sends SIGTERM, which is on the clean list, so Restart=on-failure correctly does nothing and you conclude your config is broken. Use kill -9 <pid> or systemctl kill -s SIGKILL myapp.service instead. Also remember that no value of Restart= fires after systemctl stop, or when the unit was stopped because a BindsTo= or PartOf= dependency went away. A stop job is not a failure.

RestartSec, and the 100 millisecond default

RestartSec= is the pause between the unit stopping and systemd starting it again, and the default is 100 milliseconds. Check what your unit actually loaded:

systemctl show -p RestartUSec -p StartLimitIntervalUSec -p StartLimitBurst myapp.service

A unit that has not set it prints RestartUSec=100ms. That default is fine for a service that crashes once and comes back. It is wrong for a service that cannot start at all, because five restarts then happen inside half a second, which is exactly what trips the rate limit described next. For anything that waits on a database, a mount or a network route, set RestartSec=5s or more.

As of August 2026, systemd 254 and newer also offer RestartSteps= and RestartMaxDelaySec=, which grow the delay from RestartSec= up to a ceiling over that many attempts. Ubuntu 24.04 ships systemd 255 and has them. Debian 12 ships systemd 252 and does not. Growing delays are the right answer when the dependency may be down for a long time.

What "start request repeated too quickly" really means

This is the state readers take for systemd giving up arbitrarily. It is a counter. The rule: if a unit is started more than StartLimitBurst= times inside StartLimitIntervalSec=, systemd refuses to start it again and puts it in the failed state. The defaults are 5 starts in 10 seconds.

The journal shows the sequence:

myapp.service: Scheduled restart job, restart counter is at 5.
myapp.service: Start request repeated too quickly.
myapp.service: Failed with result 'start-limit-hit'.
Failed to start myapp.service - My application.

and systemctl start answers with the fix already written out:

Job for myapp.service failed because start of the service was attempted too often. See "systemctl status myapp.service" and "journalctl -xeu myapp.service" for details. To force a start use "systemctl reset-failed myapp.service" followed by "systemctl start myapp.service" again.

systemctl reset-failed myapp.service clears the counter and the failed state. Nothing else does, so a plain systemctl start keeps being refused until you run it. Manual starts count towards the limit too, so a few impatient systemctl restart runs while you edit a config file can trip it with no crash involved at all.

The part that misleads people: start-limit-hit never says why the service was failing. It only says that it failed repeatedly and fast. The real reason is in the journal lines above it.

Both settings belong in the [Unit] section. You will find examples that put them in [Service], which older systemd accepted, and that is where the confusion starts. Write them in [Unit], then ask systemd what it loaded with systemctl show, because the loaded value is the only one that counts.

[Unit]
Description=My application
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
Type=exec
ExecStart=/usr/local/bin/myapp
Restart=on-failure
RestartSec=10s

That gives the unit five attempts inside a five minute window before it gives up. StartLimitIntervalSec=0 turns the limit off completely, and you should know what you are choosing: a service that can never start will now retry forever and write to the journal every time. The machine-wide defaults live in /etc/systemd/system.conf as DefaultStartLimitIntervalSec= and DefaultStartLimitBurst=.

One neighbouring setting deserves a warning. StartLimitAction= decides what happens when the limit is hit, and it accepts values including reboot, reboot-force and poweroff. The default is none, which fails the unit and leaves the machine alone. On a remote VPS, poweroff means a box that stays off until you open the provider's console.

Fix one: one process per unit

This is the answer in almost every case. If two programs must run, write two units. Each one then has a real main process, a real exit status and its own restart policy. You also get separate logs, separate resource limits and separate restart counters, which is what you want at three in the morning.

Express the relationship between the units in the unit files, not in a shell script.

• After= orders start-up only. It says nothing about failures.
• Requires= starts the other unit alongside this one, and stops this one if the other is stopped explicitly.
• BindsTo= is Requires= plus the case you care about: this unit stops when the other stops for any reason, including a crash. Pair it with After=, or the ordering is undefined.
• PartOf= propagates stop and restart downwards, so systemctl restart myapp.target reaches every unit that is PartOf= it.
• Upholds= (systemd 249 and newer, so Ubuntu 22.04 and later) keeps the named unit running: if it stops, systemd starts it again. It is subject to the same start rate limit as everything else.

A worker that must never run without its API server, and that systemd keeps alive whenever the API is up:

# /etc/systemd/system/myapp-api.service
[Unit]
Description=myapp API server
Wants=network-online.target
After=network-online.target
Upholds=myapp-worker.service

[Service]
Type=exec
User=myapp
ExecStart=/usr/local/bin/myapp serve
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/myapp-worker.service
[Unit]
Description=myapp background worker
BindsTo=myapp-api.service
After=myapp-api.service
StartLimitIntervalSec=120
StartLimitBurst=5

[Service]
Type=exec
User=myapp
ExecStart=/usr/local/bin/myapp worker
Restart=on-failure
RestartSec=5s

The worker has no [Install] section and is never enabled by hand. The API unit pulls it in with Upholds=, so systemctl enable --now myapp-api.service is the only command you run. Reload and check what systemd made of the pair:

sudo systemctl daemon-reload
systemd-analyze verify /etc/systemd/system/myapp-worker.service
systemctl list-dependencies myapp-api.service

systemd-analyze verify prints nothing at all when the file is clean. Any output is a problem, usually a key systemd does not recognise in the section where you wrote it, or a dependency on a unit that does not exist.

Fix two: Type=notify, so systemd knows more than a PID

If the program speaks the systemd notification protocol, use it. With Type=notify the service tells systemd when it is ready, which makes ordering real instead of hopeful, and it can send MAINPID= to point systemd at the process that matters rather than at a launcher.

WatchdogSec= is the part worth the effort. Set it, and the service must send WATCHDOG=1 through sd_notify(3) at least that often. When the messages stop, systemd terminates the service with SIGABRT and marks it failed, so Restart=on-failure or Restart=on-watchdog brings it back. This is the only built-in way to restart a process that is alive but stuck, which no exit-status policy can ever catch.

[Service]
Type=notify
NotifyAccess=main
ExecStart=/usr/local/bin/myapp serve
WatchdogSec=30s
Restart=on-failure
RestartSec=5s

A watchdog trip appears in the journal as myapp.service: Watchdog timeout (limit 30s)! followed by the kill. If instead the unit sits in activating (start) until TimeoutStartSec runs out, READY=1 never arrived: either the program does not speak the protocol, or NotifyAccess=main is rejecting a notification that came from a child process, which the journal reports with both PIDs.

For software that exposes an HTTP health endpoint but has no sd_notify support, the honest choices are a small timer unit that probes the endpoint and calls systemctl restart, or letting a container runtime do the probing, which is what Compose healthchecks and their restart behaviour exist for.

Fix three: a supervisor inside the unit, only when there is no choice

Some software genuinely ships as a bundle of processes behind a launcher you cannot split apart. Then you run a supervisor inside the unit and you accept the consequence: systemd watches the supervisor, the supervisor watches everything else, and your restart policy now lives in two files.

The common form of this is a container runtime. A docker compose or podman unit is exactly this pattern, with the per-container restart policy expressed in the Compose file and the systemd unit only keeping the runtime up. If that is your shape, the unit that brings a Compose stack up at boot shows the working version, including why Type=oneshot with RemainAfterExit=yes is usually correct there.

The cgroup still works in your favour. Everything the supervisor starts stays inside the unit's cgroup, so MemoryMax=, CPUQuota= and the cleanup at stop time all still cover the whole tree. Only the restart decision is delegated.

Whatever supervisor you choose, do not set Restart=always on the outer unit and an aggressive restart policy inside it without thinking. Two layers of restart logic, each with its own backoff, produce a service that flaps for minutes and a journal that does not explain why.

ExitType=cgroup does not mean "restart when any process dies"

ExitType= (systemd 250 and newer, so Ubuntu 24.04 and Debian 12 both have it) is the setting people find when they search for this problem, and it does the opposite of what the name suggests. The default, ExitType=main, means the service is considered stopped when the main process exits. ExitType=cgroup means the service is considered running until the last process in the cgroup exits.

So ExitType=cgroup makes a unit less sensitive to one process dying, not more. It is the right setting for a program that forks its real worker and exits the parent without writing a PID file, where Type=forking cannot find the daemon. It is the wrong setting for the failure described here.

There is no Restart= value meaning "restart the unit when any process in the cgroup dies". If you need that behaviour, you need one process per unit. If you cannot split the program and you control the wrapper script, the nearest thing is wait -n, which returns as soon as the first child exits:

#!/bin/bash
/usr/local/bin/myapp-web &
/usr/local/bin/myapp-worker &
wait -n
exit 1

Any child dying now takes the wrapper down with a non-zero status, so Restart=on-failure acts. This is a compromise, not a fix. You still get one restart counter for two programs, one log stream, and no way to restart the failing half on its own.

How to inspect what actually happened

Four commands, in this order.

systemctl status myapp.service
systemd-cgls --unit myapp.service
systemctl show -p MainPID -p NRestarts -p Result -p ExecMainStatus myapp.service
journalctl -u myapp.service -b -o short-precise

systemctl status gives the state, the main PID and the cgroup tree on one screen. A healthy unit reads Active: active (running) with a Main PID: line naming the process you expect. If the tree at the bottom lists processes you do not recognise, or is missing one you do, you already have your answer.

systemd-cgls --unit prints the same tree without truncation, which starts to matter once a unit holds more than a handful of processes.

systemctl show gives machine-readable facts. NRestarts= is the restart counter, and it is the fastest way to tell a service that has restarted forty times from one that has been up since boot. Result= holds the last failure reason: exit-code, signal, timeout, oom-kill, watchdog or start-limit-hit. ExecMainStatus= is the raw exit status of the last main process.

The journal holds the sequence. These are the three lines to search for:

myapp.service: Main process exited, code=exited, status=1/FAILURE
myapp.service: Failed with result 'exit-code'.
myapp.service: Scheduled restart job, restart counter is at 1.

code=exited, status=N means the program chose to return N, so the fault is in the program or its configuration. code=killed, signal=SEGV means it crashed. code=killed, signal=TERM usually means something else asked it to stop, which is not a failure and will not trigger Restart=on-failure. code=dumped means it left a core file, which coredumpctl list will show you when systemd-coredump is installed.

Across more than one machine, NRestarts is the number worth collecting on a schedule. A unit whose counter climbs every day is failing every day, whether or not anybody noticed. Once you are past two or three boxes, a consistent way to run one command on every server turns that from a guess into a report.

FAQ

Why does systemctl say my service is active when the process died?

systemd tracks one process per service unit, the main process, and Restart= reads only that process's exit status. Everything else the unit starts lives in the same cgroup, and systemd will kill those processes when the unit stops, but it never watches them for exit. Run systemctl show -p MainPID myapp.service and compare the number with systemd-cgls --unit myapp.service. If the process that died appears in the tree but is not MainPID, systemd behaved exactly as designed. The fix is one process per unit, with the relationship written as BindsTo= and Upholds= between the units.

What does "start request repeated too quickly" mean?

It means the unit was started more than StartLimitBurst= times inside StartLimitIntervalSec=, which defaults to 5 starts in 10 seconds, so systemd stopped trying. It is a rate limit and it never says why the service was failing, so read the journal lines above it. Clear the state with systemctl reset-failed myapp.service, then fix the underlying failure. If the service waits on something slow to come up, raise RestartSec=, because the default gap of 100 milliseconds burns all five attempts in under a second.

Should I use Restart=always or Restart=on-failure?

Use on-failure for almost everything. It restarts a crash, a non-zero exit, a timeout and a watchdog trip, and it leaves a deliberate exit 0 alone. Use always only when the program exits cleanly for reasons outside its control, such as a client that returns 0 when its peer disconnects. The cost of always is that a service which reads a broken config, logs one error and exits 0 will loop forever, and the only visible symptom is NRestarts climbing in systemctl show.

Why does killing my process by hand not trigger a restart?

Because systemd counts SIGHUP, SIGINT, SIGTERM and SIGPIPE as clean exits, and a plain kill <pid> sends SIGTERM. Under Restart=on-failure a clean exit is not a failure, so nothing restarts and the configuration looks broken when it is not. Test with kill -9 <pid> or systemctl kill -s SIGKILL myapp.service, which is an unclean termination and does trigger the policy. The same rule explains why systemctl stop never fights your restart policy.

Where do StartLimitIntervalSec and StartLimitBurst go?

In the [Unit] section. Older material and older systemd versions put them in [Service], so copied examples disagree with each other. Do not guess which one your version honours. After systemctl daemon-reload, ask systemd what it loaded with systemctl show -p StartLimitBurst -p StartLimitIntervalUSec myapp.service, and treat those numbers as the truth. systemd-analyze verify /etc/systemd/system/myapp.service catches keys systemd does not recognise at all, and prints nothing when the file is clean.