Zombie process for your Linux server: e get wahala?
You see Z or defunct for ps or top for your VPS? Learn wetin zombie process be, how to make one, how to find di parent wey cause am, and why reboot no be di first fix.
Wetin zombie process be?
Zombie process for Linux server na process wey don finish run, but im parent never collect im exit status. E no dey run any code again. E just dey wait make di parent come read how e take end. One or two zombie for ps or top no mean say your VPS dey for trouble. If di number dey grow every hour, one program for di server get bug, and na di parent you go find.
Di Linux man page for wait(2) talk am straight:
A child that terminates, but has not been waited for becomes a "zombie".
"Waited for" mean say di parent call wait() or waitpid() to collect di exit status of im child. Dat call na im dem dey call "reap". Until di parent do am, wait(2) talk say di zombie go use one slot for di kernel process table. If dat table full, di same page say e no go possible to create new process again. Na dat one be di cost wey di man page name: one slot, and every slot carry one PID (process ID).
Wetin Z and defunct mean for ps output?
For ps, di STAT column na wetin di man page call "multi-character process state". Di first character na di state itself. Di characters wey follow na extra flags wey di same man page explain. Di ps man page (procps-ng) define Z like dis:
Z defunct (“zombie”) process, terminated but not reaped by its parentSo any STAT wey start with Z na zombie, whatever flag follow am. Di word <defunct> wey you fit don see near one command name na di same word wey di man page use for Z.
Run dis command make you see only di zombie lines, plus di header line:
ps -eo pid,ppid,stat,cmd | awk 'NR==1 || $3 ~ /^Z/'pid na di process ID. ppid na di parent process ID. Na ppid be di column wey matter pass, because na di parent get di bug, no be di zombie. cmd na di command with all im arguments. If you only want count dem:
ps -eo stat= | grep -c '^Z'Make one zombie on purpose make you see how e dey behave
Di best way to stop to fear zombie na to make one yourself. Run dis one for normal shell. E no need root.
bash -c 'sleep 1 & exec sleep 300' &
sleep 3
ps -eo pid,ppid,stat,cmd | grep '[s]leep'Wetin happen here: bash start sleep 1 for background. Then exec replace bash with sleep 300, inside di same PID. So sleep 300 don become di parent of sleep 1. But sleep na small program wey no dey call wait(), so when sleep 1 finish after one second, nobody collect im exit status. Di [s]leep pattern stop grep from matching im own line.
Now look di output well and answer dese questions yourself:
- Which line carry Z for di STAT column, and wetin di CMD column show for am?
- Di PPID for dat line, e match di PID of
sleep 300?
Write down di two PIDs. You go need dem for di next two tests.
Kill -9 on di zombie PID: e dey change anything?
kill -9 ZOMBIE_PID
ps -eo pid,ppid,stat,cmd | grep '[s]leep'Put di number wey carry Z for di place of ZOMBIE_PID. Then look di list again. Di Z line still dey there? Before you run am, ask yourself one question: wetin signal go do to process wey ps don already talk say e "terminated"? Then compare wetin you expect with wetin your screen show.
Kill di parent: wetin happen to di zombie?
kill PARENT_PID
sleep 1
ps -eo pid,ppid,stat,cmd | grep '[s]leep'PARENT_PID na di PID of sleep 300. wait(2) explain say when parent terminate without to wait, im zombie children go move to di nearest ancestor subreaper wey still dey alive, or init(1) if no such subreaper dey. Init dey reap di zombies wey e adopt; if subreaper adopt dem, na dat subreaper must collect dem. Check whether your output agree with dat sentence. Na dis one be di lesson for real server: di fix wey wait(2) describe dey pass through di parent. Di parent collect di status, or di parent end and di new parent must reap di zombie.
Zombie inside systemd service: wetin systemctl go show?
For real server, your programs dey run as systemd units, no be from your shell. So make we put di same pattern inside throwaway unit. Create di file /etc/systemd/system/zombie-demo.service:
[Unit]
Description=Zombie demo, delete after test
[Service]
ExecStart=/bin/bash -c 'sleep 1 & exec sleep 300'I no put Type= line for here. If you wan sabi how systemd decide which PID na di main process of service, read how systemd service types decide di main process before you continue.
sudo systemctl daemon-reload
sudo systemctl start zombie-demo
sleep 3
systemctl status zombie-demo --no-pager
systemctl show -p MainPID zombie-demo
systemctl --version
uname -rWrite down your systemd version and your kernel version. Wetin you see fit depend on dem, so if you share your result with person, give dem dat two lines too. Di systemctl man page say status dey show "runtime status information" for di unit plus recent journal lines, and di example output for dat page carry Tasks line and CGroup line. Now answer:
- Di zombie show inside di CGroup tree for di status output?
- Wetin di Tasks line talk?
- For
ps -eo pid,ppid,stat,cmd | grep '[s]leep', di PPID of di Z line na di MainPID weysystemctl showprint? - Di CMD of di Z line, e get
<defunct>?
Di zombie still count for pids.current?
Di kernel cgroup v2 docs define pids.current as "The number of processes currently in the cgroup and its descendants." Di docs no talk whether zombie dey inside dat number. So make your own server answer am. Dis step work only if your system expose di unit cgroup under /sys/fs/cgroup:
CG=$(systemctl show -p ControlGroup --value zombie-demo)
cat "/sys/fs/cgroup${CG}/pids.current"If cat reply No such file or directory, your system no expose di file for dat path, so skip dis step. If e print number, compare am with how many sleep lines ps show for di unit, including di Z line. Dis answer matter later, because TasksMax= dey work on top of dis same counter.
Wetin systemctl restart do to di zombie?
sudo systemctl restart zombie-demo
sleep 3
ps -eo pid,ppid,stat,cmd | grep '[s]leep'Di systemctl man page describe restart as "Stop and then start" di unit. Di systemd.kill man page say KillMode= default na control-group, and with dat mode, all di processes wey remain for di unit control group go die when di unit stop. Di restart also start new copy of di same pattern. So compare PIDs carefully. Di old zombie PID wey you write down, e still dey di list? If one Z line dey, im PPID na di old MainPID or di new one?
When you finish, clean up:
sudo systemctl stop zombie-demo
sudo rm /etc/systemd/system/zombie-demo.service
sudo systemctl daemon-reloadIf di real service wey dey born zombie also dey crash and come back by itself, how Restart= and systemd restart policies work go help you see how often e dey restart. If e no gree start again after you restart am, read di exit code for systemd unit wey no gree start before you do anything else.
Why zombies dey gather inside Docker containers?
Container get im own PID namespace by default. Di pid_namespaces(7) man page say when child process become orphan, e go move to di nearest ancestor wey register as subreaper with PR_SET_CHILD_SUBREAPER, if one dey; otherwise e go move to di init for di PID namespace of im parent. Inside container, dat "init" na whatever dey run as PID 1 there. E fit be your Node app or your Python script. On di host, init na real init system wey dey call wait. Inside container, if PID 1 na normal app wey no dey call wait(), adopted children wey die go remain zombie unless di app explicitly set SIGCHLD to SIG_IGN or enable SA_NOCLDWAIT. With either setting, children wey terminate no dey become zombies.
Docker get answer for dis. Di docker run reference describe --init like dis: "Run an init inside the container that forwards signals and reaps processes". Di same page say di default init na di docker-init binary, and dat binary "is backed by tini". For Compose, di services reference say "init runs an init process (PID 1) inside the container that forwards signals and reaps processes." You turn am on for each service:
services:
web:
image: alpine:latest
init: trueTest am yourself, but dis time with orphan. Dis pattern different from di one before. Di subshell ( ... ) start sleep 1 and exit at once, so sleep 1 become orphan, and di kernel give am to PID 1 of di container for dis example, where no nearer ancestor subreaper dey.
docker run -d --name zdemo alpine sh -c '(sleep 1 &); exec sleep 300'
sleep 3
ps -eo pid,ppid,stat,cmd | grep '[s]leep'
docker rm -f zdemoNow run di same thing with --init:
docker run -d --init --name zdemo-init alpine sh -c '(sleep 1 &); exec sleep 300'
sleep 3
ps -eo pid,ppid,stat,cmd | grep '[s]leep'
docker rm -f zdemo-initYou fit see container processes from ps on di host, because dem be normal host processes wey dey inside another namespace. Compare di two outputs. Z line dey for di first run? E dey for di second run?
One thing --init no fit do. Init dey reap children wey e create directly and orphans wey e adopt. If your app still dey run and e no dey wait for im own children, di zombies still belong to di app. Init no go touch dem. Dat one na bug for di app itself, and init: true no go cover am.
Headless Chrome workers dey show for here often. Puppeteer troubleshooting page get comment inside im Docker example: "If running Docker >= 1.13.0 use docker run's --init arg to reap zombie processes". Di Cloud Run tips for di same page also mention dumb-init if you see plenty zombie Chrome processes wey no gree go. So if you dey run headless browser for AI agents on a VPS, put init: true for dat service. To look inside container wey dey run, open shell inside di container with docker compose exec. After you change di compose file, run docker compose up -d make Compose create di container again with di new setting. Di difference between restart and recreate for Compose explain why plain restart no be di same thing.
When zombie process go start to cause real wahala?
Each zombie hold one PID until somebody reap am. Di kernel docs call kernel.pid_max di "PID allocation wrap value", and say "PIDs of value pid_max or larger are not allocated." Di proc_sys_kernel(5) man page give 32768 as di default for dat file, and say on 64-bit system you fit set am up to 2^22, about 4 million. Your server fit carry different value, so check your own. PID numbers dey wrap back to small number when dem reach di top. New process go fail when no free PID remain under di limit, but fork() also fit fail before dat because of RLIMIT_NPROC, kernel.threads-max, cgroup PID limits, SCHED_DEADLINE without reset-on-fork, or memory wey no reach.
systemd get second limit for each unit. Di systemd.resource-control man page say TasksMax= dey "Specify the maximum number of tasks that may be created in the unit", and e control di cgroup pids.max value. Di kernel cgroup v2 docs say once new process go pass dat limit, fork() or clone() go fail with -EAGAIN. Di systemd-system.conf man page say DefaultTasksMax= default na 15% of di smallest of kernel.pid_max, kernel.threads-max and di root cgroup pids.max. Your distro systemd fit differ, so ask your own server:
sysctl kernel.pid_max
systemctl show -p DefaultTasksMax
systemctl show -p TasksMax your-app.serviceIf your pids.current test above show say zombie dey inside di count, then a service wey dey leak zombie fit reach im own TasksMax= before di whole server reach pid_max, depending on di limits wey dem set. Dat service go stop to fit start new process, while other services still dey fine.
To know whether you get real problem, watch di trend, no be one number:
watch -n 60 "ps -eo stat= | grep -c '^Z'"Number wey stay small and steady no be emergency. Number wey dey climb every minute mean say one parent dey born children and e no dey collect dem. If you leave am, dat service, or di whole server, go reach di point where e no fit create new process again.
How to fix zombie processes, step by step
- Find di parent. Run
ps -eo ppid=,stat= | awk '$2 ~ /^Z/ {print $1}' | sort | uniq -c. E go show each PPID and how many zombies e get. - Find out who be dat parent. Run
ps -o pid,stat,cmd -p PPIDto see di command, andsystemctl status PPIDto see which unit di process belong to. - Restart only dat service. Use
sudo systemctl restart name.service, ordocker compose restart serviceif di parent dey inside container. When di parent end, di nearest ancestor subreaper wey still dey alive adopt di zombies, or init adopt dem if no such subreaper dey; di adopter must reap dem. - Report di bug to di people wey maintain di program. Give dem di
psoutput, di program version, and how fast di count dey grow. - Reboot no be di first move. E go clear di zombies, but e go also clear di evidence wey show which parent cause am, and di bug go come back with di next boot.
FAQ
Zombie process dey chop memory or CPU for my server?
Di wait(2) man page name di cost of zombie as one slot for di kernel process table, and every slot hold one PID. You fit check di rest yourself with ps -o pid,stat,rss,time,cmd -p PID, where rss show memory and time show CPU time. Di real danger na PIDs: plenty zombies fit stop di server from creating new process.
How I go remove zombie process wey no gree go?
Find di parent with ps -eo pid,ppid,stat,cmd, look di PPID column, then restart di service wey own dat parent. According to wait(2), when parent terminate, im zombie children move to di nearest ancestor subreaper wey still dey alive, or init(1) if no such subreaper dey; di adopter must reap dem. If di zombie dey inside Docker container, set init: true for Compose or use docker run --init, so a real init dey PID 1 to collect orphans.
Zombie process fit crash my Linux server?
One or two zombies no go crash anything. Wahala start when di count keep growing, because each zombie hold one PID. Process creation fit stop at kernel.pid_max, kernel.threads-max, RLIMIT_NPROC, or a cgroup PID limit like di one TasksMax= set, with fork() returning EAGAIN. Memory wey no reach also fit make fork() fail with ENOMEM. Check yours with sysctl kernel.pid_max and systemctl show -p TasksMax your-app.service.
Why my Docker container dey gather zombie processes?
Inside container, orphan process dey go to di nearest ancestor subreaper wey still dey alive, or PID 1 if no such subreaper dey; PID 1 fit be your app or a dedicated init. If di app adopt di children and e no dey call wait(), dem go remain zombie when dem die unless e explicitly ignore SIGCHLD or enable SA_NOCLDWAIT. Docker --init and Compose init: true put small init (by default docker-init, backed by tini) as PID 1 to forward signals and reap processes. If di app itself be di parent and e still dey run, init no fit help, and you must report di bug to di app maintainers.