Systemd unit won't start: read the exit code

Why a systemd unit will not start

A systemd unit that will not start tells you the reason in one field. Run systemctl status <unit> and look for code= and status= on the line that reports the failure. A status number in the 200s means systemd never reached your program: it failed while building the environment your unit file asked for. A status below 200 means your program did run and exited on its own, so the unit file is probably right and the application is not.

That split is the decision path. Everything below follows from it, in the order the numbers show up.

Which three commands answer the question, in order

systemctl status myapp.service
journalctl -u myapp.service -b --no-pager
systemd-analyze verify /etc/systemd/system/myapp.service

systemctl status gives the verdict. Read the Loaded: line first, because it names the file systemd actually parsed and says whether the unit is enabled, masked, or not found at all. Then read the Active: line and the code= and status= pair below it.

journalctl -u myapp.service -b --no-pager gives the detail. -u filters to that one unit, -b limits the output to the current boot so you are not reading a failure from last week, and --no-pager prints straight to the terminal so you can pipe it into grep. status shows only the last handful of log lines and cuts long ones short. The journal shows everything the program printed before it died, which is usually the real error. Add -n 100 for more history, or run it with -f in a second terminal while you restart the unit.

systemd-analyze verify loads a unit file without running it. It warns about unknown sections and directives, and it flags commands in ExecStart= that it cannot execute. That catches the two quiet classes of mistake: a misspelled key, which systemd ignores at load time with a warning most people never read, and a path that is not there.

After editing any unit file, run sudo systemctl daemon-reload. Until you do, systemd keeps using the copy it loaded earlier, and systemctl status adds a warning that the file on disk has changed. A fix that "did nothing" is often a fix systemd has not read yet.

Two more commands earn their place. systemctl cat myapp.service prints the effective unit, meaning the main file plus every drop-in under /etc/systemd/system/myapp.service.d/. systemctl show myapp.service -p ExecStart -p User -p WorkingDirectory prints those values as systemd parsed them, which is what will actually run.

What does status=203/EXEC mean?

203/EXEC means systemd finished the setup, called execve(), and the kernel refused. Your program never ran a line of its own code. Four causes cover almost every case.

1. The path in ExecStart= is wrong, or it is not absolute. Check it with ls -l against the exact string in the unit file.
2. The file does not have the execute bit. sudo chmod +x /opt/myapp/run.sh fixes it. A file unpacked from an archive or copied from another machine often loses that bit.
3. The shebang line is broken. The kernel reads the first line of a script and runs the interpreter named there, so #!/usr/bin/env python3 fails when the service PATH holds no python3, and a file saved with Windows line endings asks for an interpreter called /bin/bash\r, which does not exist.
4. The file is not something this machine can run: the wrong architecture, or a text file with no shebang at all.

Reproduce it by hand, as the service user, before you change anything.

sudo -u appuser /opt/myapp/run.sh
file /opt/myapp/run.sh
head -1 /opt/myapp/run.sh | cat -A

file names the architecture and reports "with CRLF line terminators" when the line endings are the problem. cat -A shows the same thing as a trailing ^M. Strip them with sed -i 's/\r$//' /opt/myapp/run.sh.

One honest caveat about the range: 200 and up is a convention, not a guarantee. Your own program is free to exit with 203, and systemd cannot tell the two apart. systemd-analyze exit-status 203 prints the name and the class of any code, which helps you read the table, but if your application picks exit codes above 199, change them.

Why do I get 217/USER or 216/GROUP?

217/USER means the account named in User= does not exist at the moment the service starts. 216/GROUP is the same failure for Group= or SupplementaryGroups=. Confirm it with one command each.

getent passwd appuser
getent group appgroup

Each one either prints a line, or prints nothing and returns non-zero. Nothing means the name is unknown to the system, so systemd cannot switch to it and stops before exec. The fix is to create the account, not to set User=root. Running one service under a dedicated system account with least privilege is the whole point of that directive.

sudo useradd --system --no-create-home --shell /usr/sbin/nologin appuser

DynamicUser=yes sidesteps the problem by having systemd allocate a temporary account for each start. It suits a service that keeps no state. Anything that writes files needs StateDirectory= alongside it, because the user ID changes between starts and files under a plain path end up owned by an account that no longer exists.

What is 226/NAMESPACE?

226/NAMESPACE comes from the sandboxing directives. When a unit sets ProtectSystem=, ProtectHome=, PrivateTmp=, ReadWritePaths= or anything similar, systemd builds a private mount namespace for that service before it execs the program. A namespace here is a private view of the filesystem for one process. If any mount in that plan fails, the start fails with 226 and your program never runs.

The usual cause is a path in ReadWritePaths= that does not exist. ProtectSystem=strict mounts the whole filesystem read only, and ReadWritePaths= reopens named paths for writing. systemd cannot reopen a directory that is not there. Two fixes are good. Let systemd create the directory with StateDirectory=, which makes /var/lib/<name> at every start and gives it to the service user, or prefix the path with -, which tells systemd to ignore that entry when the source is missing. The bad fix is deleting the hardening, which trades a five minute problem for a permanent one.

[Service]
ProtectSystem=strict
ProtectHome=yes
StateDirectory=myapp
ReadWritePaths=-/srv/uploads

When you cannot tell which line is responsible, remove the whole hardening block, reload, and start. If the service comes up, add the lines back one at a time and restart after each one. Two neighbours in this family are 233/RUNTIME_DIRECTORY and 238/STATE_DIRECTORY. They mean systemd could not create or take ownership of the directory named in RuntimeDirectory= or StateDirectory=, usually because that path already exists and belongs to a different user.

Why does 200/CHDIR appear when WorkingDirectory looks correct?

200/CHDIR means the chdir() into WorkingDirectory= failed. The directory is missing, or the service user cannot enter it. Entering a directory needs execute permission on that directory and on every parent above it, so a perfectly readable /home/deploy/app is unreachable when /home/deploy is mode 700 and the service runs as appuser.

sudo -u appuser test -x /srv/myapp && echo ok
namei -l /srv/myapp

namei -l prints the owner and the mode of every component of the path, which is the fastest way to find the one directory that blocks the rest. Writing WorkingDirectory=-/srv/myapp makes a missing directory non-fatal. That is right for a program that does not care where it starts, and wrong for one that opens files by relative path.

Why does the service start and then stop one second later?

Here there is no 200-series code, and often no error text at all. The unit shows inactive (dead) straight after a start, or it cycles through activating (auto-restart). systemd built the environment correctly. The mismatch is between what your program does and what Type= promised it would do.

Type=simple, the default, says the program stays in the foreground. Give it a daemon that forks into the background and exits, and systemd sees the main process finish and calls the service done. Most daemons have a flag to stay in front, such as nginx -g 'daemon off;'.

Type=forking says the first process exits once its child is ready. Give it a foreground program and the start job waits until TimeoutStartSec= runs out, 90 seconds by default, then systemd kills it and logs a timeout.

Type=notify says the program calls sd_notify() to announce readiness. A program without that support announces nothing, so the start times out and the journal records the result as a protocol failure.

Pick the type from what the program actually does. How simple, forking, oneshot and notify differ is the one decision that settles this whole class of failure.

When a service exits again and again, systemd stops trying and reports that the start request repeated too quickly. The unit then stays failed until the rate limit window passes or you run sudo systemctl reset-failed myapp.service. Raising the limit only hides the symptom. Read the journal from the first failure instead of the last, and see what Restart=on-failure really retries before you change it.

Why is the unit inactive with no error at all?

A unit can be skipped instead of started. Condition* directives are silent by design: when the check fails, systemd marks the job successful and does nothing. A unit carrying ConditionPathExists=/etc/myapp/config.yml will never start while that file is missing, and it will never report an error either.

systemctl show myapp.service -p ConditionResult -p ConditionTimestamp
journalctl -u myapp.service -b --no-pager | grep -i condition

ConditionResult=no confirms the skip, and the journal names the check that did not hold. Use an Assert* directive instead when a missing prerequisite should fail loudly. Conditions, asserts and unit ordering covers which check belongs in which place.

A few other quiet cases live nearby. A "could not be found" error usually means the file sits in the wrong directory or you have not reloaded: unit files you write belong in /etc/systemd/system/. A masked unit refuses every start until sudo systemctl unmask myapp.service clears it. And systemctl enable fails on a unit with no [Install] section, so give it WantedBy=multi-user.target.

What if it was killed rather than failed?

code=killed is a different story from code=exited. Something ended the process from outside it. status=9/KILL points at the out of memory (OOM) killer, and the journal names the process it picked. A limit you set yourself does the same thing inside the cgroup (control group), so check the free memory on the host with free -m and check the unit for a MemoryMax=. MemoryMax, CPUQuota and the other cgroup limits explains which limit kills a process and which one only slows it down.

status=15/TERM right after a start attempt usually means systemd timed the start out and terminated the process, which sends you back to Type=.

Two habits that prevent most of these failures

Use absolute paths everywhere. systemd does not run your login shell, so there is no .bashrc, no .profile, and no activated virtual environment. $PATH for a system service is a short built in list that will not hold /opt or the shims of a language version manager. Write /usr/bin/python3 or /opt/myapp/venv/bin/python in full. command -v myapp in your shell prints the path to paste. The same rule covers WorkingDirectory=, EnvironmentFile= and every path in ReadWritePaths=.

ExecStart= is not a shell. systemd splits the line into words and calls execve() itself. Pipes, redirections, globs, &&, backticks and ~ carry no meaning: they reach your program as literal arguments. ExecStart=/usr/bin/myapp --flag > /tmp/out.log hands > and /tmp/out.log to myapp, which then exits with a usage error that looks nothing like a systemd problem. When you need shell features, ask for a shell.

ExecStart=/bin/sh -c '/usr/bin/myapp --flag | /usr/bin/tee -a /var/log/myapp.log'

For output alone you do not need that. Service output goes to the journal by default, and StandardOutput=append:/var/log/myapp.log writes to a file with no shell involved.

Variable expansion is limited in the same way. $MYVAR and ${MYVAR} are replaced from Environment= and EnvironmentFile=, and nothing else expands. $HOME is not set for a system service unless you set it. An EnvironmentFile= is not a shell script either: export does not belong in it, its quoting rules differ from bash, and a missing file is fatal unless you prefix the path with -.

Working through it on a live server

Read the code, prove the cause, change one thing, restart. That order matters more than knowing every number, because it stops you from stacking three speculative edits and losing track of which one helped. The same path works for units you did not write. A timer that never fires is a service that never started, so debug the service first: a systemd timer and the service it triggers fails in exactly the ways above, with the timer hiding the output until you ask the journal for it.

FAQ

What does status=203/EXEC mean in systemctl status?

systemd set up everything the unit asked for, then the execve() call failed, so your program never started. Check four things in order: the path in ExecStart= exists and is absolute, the file carries the execute bit, the shebang names an interpreter that exists on the service PATH, and the file uses Unix line endings. file reports "with CRLF line terminators" for the last one, which turns the interpreter name into /bin/bash\r and makes the kernel refuse.

Why does my service start and then stop right away?

The unit file promises behaviour the program does not have. With Type=simple systemd expects the program to stay in the foreground, so a daemon that forks into the background looks finished the moment it forks. With Type=forking systemd waits for the first process to exit, so a foreground program makes the start job hang until TimeoutStartSec= runs out. Match Type= to the program, and where the program offers a foreground flag, use that flag with the default Type=simple.

How do I see the real error instead of the short status output?

systemctl status prints only the last few journal lines and cuts long ones short. Run journalctl -u myapp.service -b --no-pager to get everything the unit logged during this boot, add -n 200 for a bigger window, or pipe it into grep. If the application writes its own log file, read that as well, because systemd captures only what the program sends to standard output and standard error.

Why is my unit inactive with no error message?

Most often a Condition* directive skipped it. Those checks are silent: a failed condition marks the start job successful. Run systemctl show myapp.service -p ConditionResult and look for ConditionResult=no, then read the journal line that names the check. The other common cause is a masked unit, which refuses every start until sudo systemctl unmask clears it.

Do I need daemon-reload after every unit file change?

Yes, for any edit to a unit file or a drop-in. sudo systemctl daemon-reload makes systemd re-read the files from disk, then sudo systemctl restart myapp.service applies them to the running service. You do not need it after systemctl edit, which reloads for you, and you do not need it after changing a configuration file that belongs to the application rather than to systemd.