Protect services from the OOM killer

How to protect services from the OOM killer

To protect services from the OOM killer on a Linux VPS, you do two things. You lower the score the kernel gives the services you cannot lose, such as sshd and your database. Then you put a memory limit on the service that actually leaks, so the kernel never has to choose between them.

The OOM (out of memory) killer is the part of the kernel that ends a process when the system has no memory left to give. It does not know which process matters to you. It only knows a score. This guide shows how that score works, how to read it, how to change it with systemd, and why a memory cap on the leaky service is the better fix. It starts where reading the kernel log on a VPS stops: you have already found an OOM kill in the log, and now you want it to happen to the right process.

Every command here is one you run on your own server. Container OOM behaviour is a different case, so test on the VPS itself, not inside a Docker container on your laptop.

How does the kernel pick a victim?

The kernel gives every process a "badness" score. The base score is the share of memory the process may use, on a scale of 0 to 1000. A process that uses all of the memory it is allowed scores 1000. A process that uses half scores 500. The process with the highest score is killed first.

You can move that score with oom_score_adj. The kernel documentation says it accepts values from -1000 to +1000, and that the value "is added to the badness score before it is used to determine which task to kill."

In the current kernel source (mm/oom_kill.c on the master branch, which is newer than the kernel on your VPS), the adjustment is scaled by the total memory before it is added: adj *= totalpages / OOM_SCORE_ADJ_MAX; points += adj;. In plain words, oom_score_adj = -500 makes the kernel treat the process as if it used half of the machine's memory less than it really does. A value of +500 does the opposite.

The lowest value is special. The kernel documentation says: "The lowest possible value, -1000, is equivalent to disabling oom killing entirely for that task." In the source, a task at -1000 gets a score of LONG_MIN, so it is never chosen, however much memory it holds.

How to read a process's current OOM score

Two files under /proc tell you what you need:

• /proc/<pid>/oom_score is the score the kernel uses right now. The man page says: "A higher score means that the process is more likely to be selected by the OOM-killer." It already includes the adjustment.
• /proc/<pid>/oom_score_adj is the adjustment, from -1000 to 1000. On most processes it is 0.

Read one process:

pid=$(pgrep -o mariadbd)
cat /proc/$pid/oom_score /proc/$pid/oom_score_adj

Replace mariadbd with your own process name. To see which processes the kernel would kill first, list the 15 highest scores:

for d in /proc/[0-9]*; do
  printf '%s %s %s\n' "$(cat $d/oom_score 2>/dev/null)" "${d#/proc/}" "$(cat $d/comm 2>/dev/null)"
done | sort -rn | head -15

Each line is the score, the PID (process ID) and the process name. The top line is the next victim if memory runs out now. On a database server that line is usually the database, because it holds the most memory. That is the correct choice for the kernel. It is also the reason a blanket -1000 on the database is dangerous, as the next sections show.

The kernel log also records the adjustment of the process that triggered an OOM event. The first line of every OOM report has the form <process> invoked oom-killer: gfp_mask=..., order=..., oom_score_adj=.... Find past events with:

sudo journalctl -k | grep -iE 'invoked oom-killer|out of memory'

A line that says Memory cgroup out of memory means a limit on one service was reached, not the whole machine. That difference matters later in this guide.

Check sshd before you change it

Losing sshd to the OOM killer locks you out of your own server. Before you add any config, check what it already has. Portable OpenSSH, in openbsd-compat/port-linux.c, writes -1000 to /proc/self/oom_score_adj at startup on Linux, with the comment "Tell the kernel's out-of-memory killer to avoid sshd." The same file also restores the saved value later, so not every sshd process has to carry -1000. Your distribution's package may also differ from the upstream code. So check every sshd process:

for p in $(pgrep sshd); do
  printf '%s adj=%s %s\n' "$p" "$(cat /proc/$p/oom_score_adj)" "$(cat /proc/$p/comm)"
done

If the listening sshd already shows adj=-1000, it is protected and you do not need a drop-in for it. If it shows adj=0, read on. To find which systemd unit owns a process, pass its PID to systemctl status:

systemctl status $(pgrep -o sshd)

The first line names the unit. Use that name below, because it is not the same on every distribution.

Shield sshd and a database with an OOMScoreAdjust= drop-in

systemd sets the adjustment for you with OOMScoreAdjust=. The systemd.exec man page says it takes an integer "between -1000 (to disable OOM killing of processes of this unit) and 1000 (to make killing of processes of this unit under memory pressure very likely)." If you do not set it, it "defaults to the OOM score adjustment level of the service manager itself, which is normally at 0."

Put the setting in a drop-in file. A drop-in is a small .conf file in a <unit>.service.d/ directory. systemd merges it into the unit, so a package upgrade that replaces the main unit file does not remove your change. Drop-ins in /etc/ take precedence over the ones in /run/ and /usr/lib/.

The examples use mariadb.service as the database. Find your own unit name with systemctl list-units --type=service.

sudo mkdir -p /etc/systemd/system/mariadb.service.d
printf '[Service]\nOOMScoreAdjust=-500\n' \
  | sudo tee /etc/systemd/system/mariadb.service.d/oom.conf
sudo systemctl daemon-reload
sudo systemctl restart mariadb

The [Service] header is required, because every drop-in needs its section headers. daemon-reload tells systemd to read the new file. The restart is needed because the value applies to the processes the unit starts, and the running ones were started before the file existed.

sudo systemctl edit mariadb does the same job in an editor. It creates a drop-in called override.conf and reloads the configuration when you save.

Now check that it took effect:

systemctl show -p OOMScoreAdjust mariadb
cat /proc/$(pgrep -o mariadbd)/oom_score_adj

Both should print -500. The kernel documentation does not say whether child processes inherit the value, so check every PID you care about, not only the first one.

For sshd, use the same steps with your SSH unit name and OOMScoreAdjust=-1000, but only if the check above showed it unprotected. Keep a second SSH session open while you restart the SSH unit. sshd is small, so excluding it costs almost nothing.

Why -500 and not -1000 for the database

With -500, the database stays killable. It only becomes the victim again when it holds half of the machine's memory more than any other process. That protects it from a small leaky worker. It does not protect it if the database itself is the cause, and that is the behaviour you want. The kernel docs add one rule: to lower the value below the last value set by a process with CAP_SYS_RESOURCE, you need CAP_SYS_RESOURCE yourself. systemd runs as root, so the drop-in can set it.

What can -1000 on a large database lead to?

This section describes kernel source behaviour from mm/oom_kill.c on master. Check your own kernel's source before you depend on the details.

A task at -1000 is never chosen. So when the database grows until the machine runs out of memory, the kernel kills the next highest score instead. Then the next. Each kill frees a little memory, the database takes it, and the cycle repeats. Your web server and your monitoring agent go first. Anything else you also set to -1000, such as sshd, is skipped.

When no killable task is left, the source logs Out of memory and no killable processes... and, for a system-wide OOM, calls panic("System is deadlocked on memory\n"). A panic stops the whole machine. This panic path does not apply to an OOM inside a memory cgroup, which is one more reason to put limits on services, as the next section shows.

The sysctl vm.panic_on_oom is a separate switch. The kernel docs say the default is 0, which means the OOM killer runs. With 1 the kernel panics on a system-wide OOM. With 2 it panics even when only a cgroup is out of memory. The docs say 1 and 2 "are for failover of clustering." Check yours:

sysctl vm.panic_on_oom

On the default 0, a -1000 database does not panic the box at the first OOM. It gets the other services killed one at a time, and then the box panics. Either way, you lose everything except the process that caused the problem.

The better fix: cap the leaky service with MemoryHigh and MemoryMax

Scores only decide who dies. A memory limit stops the leak from reaching the rest of the machine at all. systemd sets two limits through cgroup v2 (the second version of the Linux control groups interface):

• MemoryHigh= is a throttle. The man page says: "Memory usage may go above the limit if unavoidable, but the processes are heavily slowed down". The kernel cgroup docs say memory.high never triggers the OOM killer.
• MemoryMax= is the hard limit. The man page says: "If memory usage cannot be contained under the limit, out-of-memory killer is invoked inside the unit."

The man page recommends both together: "It is recommended to use MemoryHigh= as the main control mechanism and use MemoryMax= as the last line of defense." The leaky service slows down first. If it keeps growing, the OOM killer runs inside that service's cgroup only, and the database and sshd are never candidates.

First confirm the machine runs cgroup v2:

stat -fc %T /sys/fs/cgroup

cgroup2fs means v2. Then set the limits. worker.service stands for the service that leaks:

sudo systemctl set-property worker.service MemoryHigh=1G MemoryMax=1500M
systemctl show -p MemoryHigh -p MemoryMax worker

set-property applies the change at once and stores it on disk for future boots, unless you pass --runtime. Values take a K, M, G or T suffix (base 1024), or a percentage of the physical memory. infinity means no limit. Run the show command before you change anything too, so you know what the unit had.

When the cap is reached and memory cannot be freed, the kernel log shows Memory cgroup out of memory instead of a system-wide OOM. Only processes in worker.service are candidates. What systemd does next is set by OOMPolicy=: continue logs the kill and keeps the unit running, stop stops the unit, and kill kills the rest of its processes too. Check the current policy with systemctl show -p OOMPolicy worker.

Two more settings belong here. MemorySwapMax= limits the unit's swap use, so a leak cannot fill the swap instead. And do not give the processes inside a capped unit -1000. The kernel cgroup docs say tasks at -1000 are never killed, even when the whole group is killed, so a capped unit full of exempt tasks leaves its own OOM killer with nothing to kill.

How to choose the numbers, and how to add CPU limits next to them, is covered in limiting a process's memory and CPU with systemd. If the leaky service runs in a container, set the cap in the Compose file instead, as in setting memory limits in Docker Compose.

Does systemd-oomd help on a server?

systemd-oomd is a userspace OOM killer. It watches PSI (pressure stall information, the kernel's measure of time lost waiting for memory) through cgroup v2, and it acts before the kernel OOM killer does. When it acts, it sends SIGKILL to a whole cgroup. It needs a full unified cgroup hierarchy and a kernel with PSI support (Linux 4.20 or newer). Memory accounting must also be on for every unit it monitors.

Check whether it runs on your VPS. Do not assume, because it differs between images:

systemctl status systemd-oomd

On Ubuntu and Debian it is a separate package called systemd-oomd. On the package search pages, as of 7 November 2026, Ubuntu 24.04 lists version 255.4-1ubuntu8.18 and Debian 13 lists 257.13-1~deb13u1. Install it with:

sudo apt install systemd-oomd

Installed, it still does nothing by itself. The man page says: "Cgroups of units with ManagedOOMSwap= or ManagedOOMMemoryPressure= set to kill will be monitored." It also says the unit with the setting is not itself a candidate. Only its descendant cgroups are, and only leaf cgroups or cgroups with memory.oom.group set to 1. So the setting goes on a parent, such as a slice, and the services under it are the candidates. ManagedOOMMemoryPressure= tends to work better on cgroups below the root slice. ManagedOOMSwap= works with system-wide swap values. To steer it away from the database, ManagedOOMPreference= takes avoid or omit on the database unit. Read that entry in systemd.resource-control(5) before you rely on it.

Check memory accounting with systemctl show -p DefaultMemoryAccounting. The man page says the easiest way to turn it on is to make sure DefaultMemoryAccounting= is true, as described in systemd-system.conf(5). Then sudo oomctl dump lists the cgroups systemd-oomd monitors and their pressure.

For a small VPS with one leaky service, MemoryHigh= and MemoryMax= on that service are simpler and easier to predict. systemd-oomd earns its place when several services share the box and you cannot predict which one will grow.

Swap as headroom

The systemd-oomd man page says: "It is highly recommended for the system to have swap enabled for systemd-oomd to function optimally." Without swap the system can livelock quickly, which means it stays up but makes no progress. The page also says swap-based actions are ignored on a system with no swap.

Swap also gives MemoryHigh= room to work. The throttled service can push rarely used pages out instead of hitting MemoryMax= at once. Check what you have:

swapon --show
free -h
sysctl vm.swappiness

An empty swapon --show means no swap. The current kernel docs give vm.swappiness a range of 0 to 200 and a default of 60. Older kernels used a smaller range, and distributions can ship another value, so trust the sysctl output over this number. The docs also say the best value "will require experimentation and will also be workload-dependent."

Whether to add swap, and how much, is the subject of whether your VPS needs swap. If kswapd0 then sits at the top of top, working out if kswapd0 means memory pressure or a miner tells the two causes apart.

A plan that holds up

1. Find the last OOM event in the kernel log. Note which process triggered it and which one was killed.
2. Check sshd. If it is not already at -1000, add a drop-in.
3. Give the database a moderate adjustment such as -500, not -1000.
4. Cap the service that leaks with MemoryHigh= and MemoryMax=, and check its OOMPolicy=.
5. Make sure the box has some swap, then decide whether systemd-oomd is worth adding.

After each step, read the scores again with the loop above. The top line should now be a process you can afford to lose.

FAQ

How do I stop the OOM killer from killing my database?

Add a systemd drop-in with OOMScoreAdjust=-500 under [Service], run sudo systemctl daemon-reload, and restart the database. Check it with cat /proc/<pid>/oom_score_adj. Avoid -1000, because a database that can never be killed makes the kernel kill every other process when the database itself grows. The safer fix is a MemoryMax= limit on whatever service is leaking.

What is the difference between oom_score and oom_score_adj?

oom_score_adj is the adjustment you set, from -1000 to 1000. oom_score is the result the kernel uses right now, based on the memory the process uses plus that adjustment. A higher oom_score means the process is more likely to be killed. Both files live under /proc/<pid>/.

Is sshd already protected from the OOM killer?

Upstream portable OpenSSH writes -1000 to /proc/self/oom_score_adj when it starts on Linux, but distribution packages can differ. Run cat /proc/<pid>/oom_score_adj for each PID from pgrep sshd. If the listening process shows -1000, it is protected. If it shows 0, add OOMScoreAdjust=-1000 in a drop-in for your SSH unit.

Should I use MemoryHigh or MemoryMax?

Use both. MemoryHigh= throttles the service and never calls the OOM killer. MemoryMax= is the hard limit, and reaching it runs the OOM killer inside that service only. The systemd man page recommends MemoryHigh= as the main control and MemoryMax= as the last line of defense, so set MemoryHigh= a little below MemoryMax=.

Can setting oom_score_adj to -1000 crash the server?

It can. In the kernel source, a -1000 task is never chosen, so the kernel kills other processes instead. When nothing killable is left in a system-wide OOM, it logs Out of memory and no killable processes... and panics with System is deadlocked on memory. Separately, vm.panic_on_oom set to 1 or 2 makes the kernel panic on OOM. Its default is 0.