Fix 'A stop job is running' timeout wahala

Wetin "A stop job is running" mean for your VPS

"A stop job is running" mean say systemd dey wait for one service to stop before e fit continue di shutdown or reboot. E no mean say your server don die. Every service get two clocks, one for start and one for stop. When di clock finish, systemd go kill di service by itself and di reboot go continue. So wait small, copy di unit name wey dey di line, and fix dat unit after di box come back.

Dis guide na for person wey just install kernel update, run sudo reboot, and SSH no answer for one minute plus. Or you open your provider console come see line like A stop job is running for myapp.service. Or systemctl start return timeout error give you. All three na di same family of wahala: systemd timeout.

The reboot no go finish until di stop job finish or di clock expire. Di box never come back up, so SSH no dey answer for dat time.

Why you no suppose press hard reset for panel

When reboot dey hang, e dey tempt person to press "Reset" or "Power off" for provider panel. No do am quick quick. Systemd dey count. When di stop clock finish, systemd go send SIGKILL to di service wey refuse to stop, and di reboot go continue by itself.

Hard reset cut power in di middle of shutdown. Data wey never reach disk fit lost, and di file system go need check for next boot. Plus, di line for console dey tell you which unit dey hold you. Na dat name you need to fix di matter, so write am down before anything.

Di line itself come from di systemd manager. For di systemd source code, di format na %sA %s job is running for %s (%s / %s). Di second %s na di job type. So during boot you fit see A start job is running for ..., and during shutdown you go see A stop job is running for .... Di two times wey dey inside di bracket na how long e don wait and di limit.

Di two clocks: TimeoutStartSec and TimeoutStopSec

TimeoutStartSec= na di start clock. If daemon service no signal say e don start finish inside dis time, systemd go mark am as failed and shut am down again. Wetin happen after dat depend on TimeoutStartFailureMode=. If unit no set TimeoutStartSec=, di value come from di manager setting DefaultTimeoutStartSec=. One exception dey: for Type=oneshot, di start timeout dey off by default. Di start clock also apply to reload.

Wetin "signal say e don start" mean depend on di service type. For Type=notify, di app must send READY=1 to systemd. If you no sure which type your unit be, di guide on systemd service types and how each one report ready explain am well.

TimeoutStopSec= na di stop clock, and e dey do two jobs. First job: na how long systemd go wait for each ExecStop= command. If one of dem time out, systemd skip di remaining ExecStop= lines and send SIGTERM. Second job: na how long systemd go wait for di service itself to stop. If di service never end by den, systemd go force am to die with SIGKILL.

TimeoutSec= na shorthand. E set both TimeoutStartSec= and TimeoutStopSec= to di same value.

Di systemd.kill man page describe di stop sequence like dis:

1. Systemd send KillSignal=, wey default to SIGTERM. E also send SIGCONT right after am.
2. E wait for di TimeoutStopSec= time.
3. If any process still dey alive, e send SIGKILL, because SendSIGKILL= default to yes.

KillMode= default to control-group. Dat one mean say di signal go reach every process inside di service control group, no be only di main process. If you set SendSIGKILL=no, systemd no go send SIGKILL at all. For control-group or mixed service, di service then no go restart if processes from di old run still dey inside di control group. Dat setting fit hold you for long, so leave am for yes unless you get strong reason.

How to write di values: number without unit na seconds, or you fit write time span like 5min 20s. infinity switch di clock off. For 0, di man page no talk anything. Di systemd source code turn 0 into infinity for TimeoutSec= and TimeoutStartSec=, and di comment there talk say these options used to accept 0 to disable di timeouts. TimeoutStopSec= dey pass through another parser, parse_sec_fix_0(), and dat one also turn 0 into infinity. So for all three, 0 no mean "time out sharp sharp". E mean "no timeout at all". If you need no limit, infinity clear pass, so write infinity straight and confirm am with systemctl show.

Read di timeout defaults wey your server dey use

Di upstream docs talk say DefaultTimeoutStartSec= and DefaultTimeoutStopSec= default to 90 s for di system manager. But distro build fit change am, and somebody fit don drop config file for your box before. So no carry 90 for head. Ask di manager:

systemctl --version
systemctl show --property=DefaultTimeoutStartUSec --property=DefaultTimeoutStopUSec
systemctl show -p TimeoutStartUSec -p TimeoutStopUSec -p KillSignal -p SendSIGKILL myapp.service

The second command show di manager-wide default. The third command show di real value wey one unit dey use, after systemd don merge all di drop-ins. Change myapp.service to your own unit name. Ubuntu 24.04 ship systemd 255, and Ubuntu 26.04 ship systemd 259. Small things fit differ between versions, so always note wetin systemctl --version print before you compare your output with another person own.

Build two test units and watch di timeout happen

Best way to understand dis thing na to cause am yourself for safe place. Use test VPS or test container wey get systemd, no be production box.

Di first unit na Type=notify. But sleep no dey ever send READY=1, so di start no go ever finish, and di five-second start clock go expire:

sudo tee /etc/systemd/system/lab-slowstart.service >/dev/null <<'EOF'
[Unit]
Description=Lab unit whose start never finishes

[Service]
Type=notify
ExecStart=/usr/bin/sleep infinity
TimeoutStartSec=5
EOF
sudo systemctl daemon-reload
time sudo systemctl start lab-slowstart.service
systemctl status lab-slowstart.service --no-pager
journalctl -u lab-slowstart.service --no-pager -n 20

systemctl start go block for about five seconds, then return error. systemctl status go show di unit as failed. journalctl -u go show di timeout and di kill wey follow am. Read di exact words wey your own version print, because di wording dey change between systemd versions.

Di second unit start fine, but di script ignore SIGTERM. trap "" TERM tell bash to ignore di signal, and di sleep wey bash start go inherit dat ignore:

sudo tee /etc/systemd/system/lab-trapterm.service >/dev/null <<'EOF'
[Unit]
Description=Lab unit that ignores SIGTERM

[Service]
Type=simple
ExecStart=/usr/bin/bash -c 'trap "" TERM; while true; do sleep 1; done'
TimeoutStopSec=5
EOF
sudo systemctl daemon-reload
sudo systemctl start lab-trapterm.service
time sudo systemctl stop lab-trapterm.service
systemctl status lab-trapterm.service --no-pager
journalctl -u lab-trapterm.service --no-pager -n 20

Now systemctl stop go wait di full five seconds. Systemd send SIGTERM, nothing happen, di clock finish, and SIGKILL end di process. For di systemd source code, di message for dis case na State 'stop-sigterm' timed out. Killing.. Check your journal to see di exact text wey your version print. Dat stop-sigterm state na di same one wey di guide on systemd restart policies explain when restart dey slow, so I no go repeat am here.

Na exactly dis thing dey happen during your reboot. Change five seconds to di default wey your box report, and you get A stop job is running for dat whole time.

Clean up when you finish:

sudo systemctl stop lab-slowstart.service lab-trapterm.service
sudo rm /etc/systemd/system/lab-slowstart.service /etc/systemd/system/lab-trapterm.service
sudo systemctl daemon-reload
sudo systemctl reset-failed

Fix one service with drop-in file

No edit di unit file inside /usr/lib/systemd/system, because package update go overwrite am. Use drop-in instead. For unit foo.service, folder foo.service.d/ fit hold .conf files wey add or change settings. Systemd merge dem in alphanumeric order, and drop-in files win over di unit file. Write am with tee:

sudo mkdir -p /etc/systemd/system/myapp.service.d
sudo tee /etc/systemd/system/myapp.service.d/10-timeout.conf >/dev/null <<'EOF'
[Service]
TimeoutStopSec=30s
EOF
sudo systemctl daemon-reload
systemctl show -p TimeoutStopUSec myapp.service

daemon-reload tell di manager to read di unit config again. After am, systemctl show must show di new value. If e still show di old value, check di folder name. E must match di unit name exactly, with .d for di end.

Use di same method to give slow-starting app more start time. Example: database wey dey replay log after crash fit need TimeoutStartSec=5min. If di unit fail for another reason (bad path, wrong user, exit code wey no be zero), di timeout no be your problem, and di guide on why a systemd unit won't start and wetin di exit codes mean go help you more.

Change di default timeout for all services

Di manager-wide defaults dey for system.conf, and you fit override dem with drop-in under /etc/systemd/system.conf.d/:

sudo mkdir -p /etc/systemd/system.conf.d
sudo tee /etc/systemd/system.conf.d/60-timeouts.conf >/dev/null <<'EOF'
[Manager]
DefaultTimeoutStopSec=30s
EOF

Systemd sort these files by name, and for one-value setting, di last file win. Di docs suggest two-digit prefix, and 60 to 90 for files under /etc. Di man page no talk which command make di manager pick up di change. Run sudo systemctl daemon-reload, then read systemctl show --property=DefaultTimeoutStopUSec. If di value never change, e go apply after di next reboot.

Think well before you shorten di default. Dis one touch every service wey no set its own timeout, including databases wey need time to write data to disk before dem stop. Per-unit drop-in for di one bad service na di safer fix.

Longer timeout na stopgap. Handling SIGTERM na di cure

When app no gree stop, longer timeout only mean say you go wait longer before SIGKILL land. SIGKILL no give di app any chance to close file, close database connection, or finish di request wey e dey handle. Di real fix: di app must catch SIGTERM, finish wetin e dey do, and exit by itself.

For bash script, e look like dis:

#!/usr/bin/env bash
cleanup() { echo "SIGTERM don land, I dey close"; exit 0; }
trap cleanup TERM
while true; do sleep 1 & wait $!; done

Di sleep 1 & wait $! pattern dey important. Bash run di trap after di foreground command finish, but wait return as soon as signal wey get trap land. So di script exit fast instead of waiting for di sleep.

If di app dey run inside Docker, di container get its own grace period before Docker send SIGKILL. Di guide on docker compose down vs stop and di stop grace period cover dat side. Long systemd timeout no go help container wey Docker don already kill.

Find which unit hold your last reboot

Dis part na for your real VPS, because fresh container no get previous boot. First check whether di journal keep old boots at all:

journalctl --list-boots

If e show only one line, di journal no keep di last boot. Check di Storage= setting with systemd-analyze cat-config systemd/journald.conf. With Storage=auto, journald dey save to disk only if /var/log/journal exist. Otherwise e keep di journal for memory only, and memory clear on reboot. To start keeping boots:

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald

Next reboot go dey saved. Once --list-boots show more than one boot, read di end of di last one. -b -1 mean di boot before di current one:

journalctl -b -1 -n 200 --no-pager
journalctl -b -1 --no-pager | grep -iE 'timed out|stop job|killing'
journalctl -b -1 -u myapp.service --no-pager

Di first command show di last 200 lines, which na di shutdown. Di second one filter for timeout lines. When you see di unit name, di third command show only dat unit story. Then use di drop-in or di SIGTERM fix above.

Kernel updates na di main reason VPS dey reboot, and old kernels dey pile for /boot with every update. If dat partition dey fill, clean up old kernels on Ubuntu before di next update.

FAQ

Why my VPS show "A stop job is running" for long time after I reboot?

One service no stop when systemd send SIGTERM, so systemd dey wait for di stop clock, TimeoutStopSec=, to finish. When di clock finish, systemd send SIGKILL and di reboot continue by itself. Di unit name dey inside di line. Check di default wey your box use with systemctl show --property=DefaultTimeoutStopUSec, because no be every image get di same value.

Wetin TimeoutStopSec=0 or infinity mean?

infinity switch di timeout off, so systemd go wait forever. Di man page no talk anything about 0. For di systemd source, 0 for TimeoutSec=, TimeoutStartSec= and TimeoutStopSec= become infinity, so e mean "no timeout", no be "kill am now". If you want no limit, infinity clear pass, so write am straight and confirm with systemctl show -p TimeoutStopUSec yourunit.service.

How I go know which service slow down my last shutdown?

Run journalctl --list-boots first. If e show more than one boot, run journalctl -b -1 --no-pager | grep -iE 'timed out|stop job|killing' to see di timeout lines from di previous boot. If e show only one boot, di journal dey memory only. Create /var/log/journal, restart systemd-journald, and di next shutdown go dey saved.

Make I just set DefaultTimeoutStopSec to small number?

No be di best fix. Di default touch every service wey no set its own timeout, including databases wey need time to write to disk. Set TimeoutStopSec= for di one bad service with drop-in under /etc/systemd/system/yourunit.service.d/, run daemon-reload, and fix di app to handle SIGTERM.

Why systemctl start fail with timeout even though my app dey run?

Systemd wait for di app to report say e ready, and di way e report depend on Type=. For Type=notify, di app must send READY=1. If di app no ever send am, di start clock expire, systemd mark di unit as failed and stop di app, even if di app itself dey work fine. Fix di Type= to match how di app really start, or make di app send di ready signal.