How to limit VPS CPU and memory with systemd

Limit process memory and CPU with a systemd drop-in

You fit limit process memory and CPU for Linux VPS by adding some lines to the unit wey dey run the process. MemoryMax= na the hard memory limit. CPUQuota= na the limit for processor time. cgroup v2 (control groups, version 2) dey enforce both limits. Na kernel feature wey systemd already dey use to account for every service for the machine.

sudo systemctl edit myapp.service

That one go open drop-in file wey get instructions for comments. Add this one above dem:

[Service]
MemoryHigh=512M
MemoryMax=768M
MemorySwapMax=0
CPUQuota=80%
TasksMax=128

sudo systemctl daemon-reload
sudo systemctl restart myapp.service
systemctl show myapp.service -p MemoryHigh -p MemoryMax -p CPUQuotaPerSecUSec -p TasksMax

systemctl show must repeat your numbers back with the kernel own units: MemoryMax=805306368 and CPUQuotaPerSecUSec=800ms. If e print MemoryMax=infinity, the drop-in no load. Check whether the file land for /etc/systemd/system/myapp.service.d/override.conf, and whether e start with the [Service] header. If settings line no get section above am, systemd go log Assignment outside of section. Ignoring. and start the service without any limit.

The remaining part of this guide explain how to choose those numbers, and wetin fit still go wrong after you set dem.

Why runaway process fit freeze VPS wey e never fill

Process wey hit hard memory cap dey die for about one second, then service restart. Na the good case be that. The bad case na when nothing die: the box answer ping, SSH accept the connection, but shell prompt no ever show. The machine dey alive and busy, but none of that work useful.

Make we explain how e happen, because e no obvious. When free memory low, kernel dey reclaim pages instead of giving out new ones. File-backed pages dey cheapest to reclaim, and page cache dey hold executable code for everything wey dey run. So kernel evict the text pages of sshd, and the next instruction sshd run na page fault wey must read those bytes back from storage. Every process end up waiting for disk instead of running. The same pages dey leave and come back repeatedly. Dem dey call this thrashing.

Two things make this worse for VPS than for laptop. Storage often dey network-attached or shared, so every fault cost more milliseconds than local NVMe device go cost. Also, kernel no dey measure time; e dey measure failure: as long as reclaim dey return one page, no matter how slowly, kernel believe say e dey make progress and e no call the out of memory (OOM) killer. Box fit remain for that state for many minutes before anything get killed.

You fit monitor am as e dey happen. Linux 4.20 and newer dey expose pressure stall information (PSI):

cat /proc/pressure/memory
cat /proc/pressure/io

some avg10=63.72 avg60=41.02 avg300=12.33 total=13729481
full avg10=48.15 avg60=30.44 avg300=8.90 total=9114233

Na full line matter pass. full avg10=48.15 mean say for the last ten seconds, 48% of the time every runnable task for the box dey stalled, waiting for memory work, so nothing run. Healthy server dey read close to zero for full. When e pass 10, human go feel say e slow. When e reach 40 or more, na the state people dey describe as frozen.

Na this one also explain why limit by itself no be promise. Unit wey dey under MemoryHigh= dey throttled instead of killed, so e remain alive and remain slow. Nothing restart am because from systemd point of view, e never fail. Capped unit wey still get permission to swap dey generate reads and writes wey dem charge to that unit, but one shared device dey serve dem. This fit push /proc/pressure/io up for every other service for the box. Limits decide who go pay for shortage; dem no fit create capacity.

Check say your VPS dey run cgroup v2

stat -fc %T /sys/fs/cgroup

cgroup2fs na the unified hierarchy, and every setting wey follow need am. tmpfs mean say the box boot with the older v1 layout, where MemoryHigh= and MemorySwapMax= no dey exist and per-unit OOM behaviour different. Ubuntu 22.04 and later, plus Debian 11 and later, use v2 by default. Old image, or kernel wey boot with systemd.unified_cgroup_hierarchy=0, no use am.

For cgroup v2, systemd dey turn on memory accounting for every unit by default, so the numbers don already dey:

systemd-cgtop -m

That command list cgroups according to memory use, and na the fastest way to answer "wetin dey chop this box" while e still fit answer. If the server new, the account and firewall work for the first ten minutes for new VPS suppose come before this.

MemoryHigh dey throttle. MemoryMax dey kill.

Difference between the two memory settings decide wetin failure go look like.

• MemoryHigh= na soft cap. When usage pass am, kernel dey reclaim memory aggressively from that cgroup and dey intentionally slow down its allocations. Usage fit still pass the number, and nothing go get killed.
• MemoryMax= na hard cap. When allocation no fit satisfy under am, OOM killer dey run inside that cgroup and kill one process wey belong to that unit.

That second part na the main reason to set MemoryMax= for anything wey you no fully trust. If no cap dey, shortage become problem for the whole machine, and global OOM killer choose victim by oom_score, wey mostly mean the largest process. Usually, the largest process na your database, no be the script wey leak memory. With cap, the kill happen inside the unit wey cause am.

Set both, with MemoryHigh= around 20 to 30 percent below MemoryMax=. The gap na warning zone: slow leak go cross High and show as service wey don slow down, while sudden spike go pass straight through Max and die.

Percentage values dey use installed physical memory as reference, so MemoryMax=25% for 4 GB plan na 1 GB and e remain one quarter of the machine after you resize the plan. MemorySwapMax=0 keep that unit completely out of swap, wey turn long crawl into fast, obvious kill.

Some services let you decide their memory appetite ahead of time instead of measuring am: Ollama unit dey size its KV cache from the context window wey you give am, so read wetin raising num_ctx dey cost for RAM before you choose ceiling for one.

Cap need restart policy beside am, otherwise the kill go only leave you with stopped service.

[Unit]
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
Restart=on-failure
RestartSec=5s

StartLimit* belong inside [Unit] and Restart= inside [Service]. If you put either one for wrong section, systemd go ignore am. Five restarts within five minutes mean say na leak, no be temporary blip. After that, systemd gives up and leave the unit failed. Na this state you want find later instead of crash loop wey dey hide the problem.

Use CPUQuota to cap CPU, or CPUWeight to share am

CPUQuota= dey use percentage of time wey one CPU get available. CPUQuota=50% na half of one core. CPUQuota=200% na the same as two cores, and the unit fit spread am across as many threads as e want. For 2 vCPU plan, CPUQuota=200% na the whole machine.

CPUWeight= better default for most services. E dey set relative share from 1 to 10000, and kernel default na 100. E only matter when something dey compete: backup job for CPUWeight=20 go yield to web server for 100 under load, but e still fit use the whole machine when machine dey idle. Hard quota go throw away that idle capacity.

Make you understand wetin CPU limit fit give you. Process wey CPU dey limit am rarely go freeze Linux, because scheduler dey continue give everybody time. Na memory dey usually bring machine down. Use CPUQuota= when you want predictable ceiling, like for build or agent wey for otherwise run flat out for one hour. To size this kind workload na separate matter, and how much RAM and CPU coding agent VPS need cover am.

If CPU show say e busy while none of your processes dey do plenty work, the cause fit dey for the other side of hypervisor. Na CPU steal time from noisy neighbour be this, and no quota wey you set go change am.

TasksMax fork loop dey stop

TasksMax= na number of processes and threads wey one unit fit hold. Threads dey count too, so Java or Go service need more headroom than wetin process list show. Na the cheapest protection against script wey dey fork for loop, because fork go fail inside the unit instead of make the whole box run out of process IDs.

TasksMax=128

When unit reach the limit, kernel go log one line wey name the cgroup:

cgroup: fork rejected by pids controller in /system.slice/myapp.service

The program itself usually report fork: retry: Resource temporarily unavailable. Check wetin manager apply by default with systemctl show -p DefaultTasksMax.

Limit one one-off job with systemd-run

You no need unit file to use any of this. systemd-run dey build one temporary unit around one command.

sudo systemd-run --scope -p MemoryMax=1G -p MemorySwapMax=0 -p CPUQuota=50% -p TasksMax=64 ./import-data.sh

--scope dey run the command for your terminal, after e print Running scope as unit: run-r7c1a....scope. Output go remain for your screen, and the limits go disappear when command finish. Any property from systemd.resource-control go work after -p.

For long job, remove --scope and give am name. E go then run for background as temporary service and log go journal:

sudo systemd-run --unit=nightly-import -p MemoryMax=1G -p CPUWeight=20 ./import-data.sh
journalctl -u nightly-import -f

The same options dey work with --user when you no be root, but your user manager only get controllers wey dem delegate give am, so e fit reject one property there. Run am with sudo if that one happen. When job get permanent place, move the settings unchanged enter real unit: see run script as systemd service and timer.

Swap question, make we answer am honestly

Swap dey change how failure go happen; e no dey prevent failure.

If swap no dey, memory leak go reach limit and something go die within seconds. Outage go loud and short, and you fit understand am easily from journal afterwards. If swap dey, kernel go write cold anonymous pages go disk and buy time. If process suppose settle by itself, swap go save you. But if na runaway process, swap go turn five second outage into twenty minute stall. The stall worse, because dead process still leave working shell, but machine wey dey thrash no go.

swapon --show
free -h

For small VPS, practical middle ground na to keep modest swap file for pages wey system allocate once and never touch again. Set MemorySwapMax=0 on units wey you fit afford to lose. Important services go keep their swap. Unpredictable ones go hit the limit quickly and restart.

Lowering vm.swappiness na weak control, and you need know why. E only change balance between removing page cache and swapping anonymous pages. Both options go cost disk read later. E change which pages dey thrash, but e no change whether the machine go thrash.

Early OOM daemon dey kill before system stall

Kernel dey wait make reclaim fail finish, and for small VPS, na during that wait machine fit loss. Two userspace daemons fit close this gap by monitoring memory by demself and killing process earlier.

earlyoom dey check available memory and free swap up to ten times every second, and e dey kill process wey get highest oom_score only when both fall below dem thresholds. For machine wey no get swap, na memory threshold alone dey decide.

sudo apt install earlyoom
systemctl status earlyoom

Debian and Ubuntu package dey start the service when dem install am. E options dey inside /etc/default/earlyoom:

EARLYOOM_ARGS="-m 5,2 -s 5,2 --avoid ^(sshd|systemd)$ --prefer ^(node|python3)$"

-m PERCENT dey set minimum available memory, while -s PERCENT dey set minimum free swap. Both na 10 percent by default. The second number for each pair na the SIGKILL point: earlyoom dey send SIGTERM once both memory and swap fall below their first values, then SIGKILL once both fall below the second value. By default, the second value na half of the first. Leave --avoid and --prefer patterns without quotes and spaces, because systemd passes quote characters through exactly as dem be, and quoted pattern no go ever match. Apply change with sudo systemctl restart earlyoom, and read journalctl -u earlyoom to see which process e kill and how much memory that process dey use.

systemd-oomd na the other option. Its manual page describe am as "a system service wey dey use cgroups-v2 and pressure stall information (PSI) to monitor and take corrective action before OOM happen for kernel space". E dey act on whole cgroups instead of single processes, so e dey kill a unit, no be stray child. Units fit opt in with ManagedOOMMemoryPressure=kill or ManagedOOMSwap=kill, and thresholds dey inside /etc/systemd/oomd.conf.

systemctl status systemd-oomd
oomctl

oomctl dey print wetin e dey monitor currently. For server image, na often nothing, because setting na opt-in for each unit. Choose one daemon and stop there. If you run both, dem go race to choose victim, and e go harder to reconstruct why any process die.

Which unit na responsible?

Start with the kernel, because e dey record every kill wey e make.

journalctl -k --grep "Killed process" --since "2 hours ago"

Kill from the global OOM killer dey look like this:

Out of memory: Killed process 4127 (node) total-vm:2731084kB, anon-rss:1874232kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:4212kB oom_score_adj:0

anon-rss na the memory wey that process hold for RAM when e die, about 1.8 GB for here. Read the name for brackets with suspicion. Na the victim wey kernel choose be that, and kernel dey choose the biggest process. But that process no always be the one wey cause the shortage.

Kill from cgroup limit get different prefix, and the report wey print above am name the cgroup wey hit its own ceiling:

Memory cgroup out of memory: Killed process 8811 (python3) total-vm:1044320kB, anon-rss:769112kB, file-rss:0kB, shmem-rss:0kB, UID:998 pgtables:1720kB oom_score_adj:0

That prefix na most of the diagnosis. Memory cgroup out of memory mean say one unit hit the MemoryMax= wey you give am, while the rest of the box dey fine. Plain Out of memory mean say the whole machine run out, so your caps either dey missing or dem too generous to add up.

Then ask systemd wetin e see:

systemctl status myapp.service
journalctl -u myapp.service -n 50

myapp.service: A process of this unit has been killed by the OOM killer.
myapp.service: Main process exited, code=killed, status=9/KILL
myapp.service: Failed with result 'oom-kill'.

systemctl status talk the same thing for one line, as Active: failed (Result: oom-kill).

The cgroup counters na the third source, and na the only one wey dey record throttling. Throttling no dey produce log line at all:

cat /sys/fs/cgroup/system.slice/myapp.service/memory.events
cat /sys/fs/cgroup/system.slice/myapp.service/memory.peak

low 0
high 4213
max 118
oom 12
oom_kill 12

high count how many times the unit pass MemoryHigh= and get throttled. max count how often e reach the hard cap, and oom_kill count processes wey actually get killed. Big high together with oom_kill 0 na the silent case wey we talk earlier: the service dey run, e slow reach crawl, and e never report failure to anybody. memory.peak (Linux 5.19 and newer) hold the highest usage wey the cgroup reach. Na this number you go use size MemoryMax= against. Both files reset when the unit restart, because systemd create the cgroup again.

One prerequisite dey under all these things. If /var/log/journal no exist, the journal dey live for RAM, and every line go disappear after the reboot wey you need to recover the box.

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
journalctl --list-boots

journalctl --list-boots wey show more than the current boot mean say history dey survive now, so journalctl -k -b -1 fit show you kernel messages from the boot wey die.

Pikin VPS wey you fit start with

For 2 GB plan, leave 300 to 400 MB for kernel and page cache. No make all the caps add up reach the full 2 GB, because every unit fit reach peak at the same time. Give the service wey matter most the biggest share. Then cap every speculative service around am.

[Service]
MemoryHigh=256M
MemoryMax=384M
MemorySwapMax=0
CPUWeight=20
TasksMax=64
Restart=on-failure
RestartSec=5s

To keep one way to enter the server, add one more setting. OOMScoreAdjust=-500 for a drop-in under ssh.service go make global OOM killer much less likely to pick your SSH daemon as victim. This fit be the difference between fixing the box and rebooting am from control panel. E only change the victim wey kernel choose. E no reduce how long the stall go last.

Containers dey run inside their own cgroups. Na container runtime create dem, no be your unit files. So, limit for docker.service no become limit for one container. The per-container equivalents of MemoryMax= and CPUQuota= dey explained for how to set memory and CPU limits for Docker Compose.

FAQ

Why my VPS freeze instead of e kill the runaway process?

Because kernel dey judge progress by whether reclaim dey return pages, no be by how long e take. When memory short, e dey evict page cache, including executable pages of programs wey dey run, then e read dem back for the next instruction. Everything dey wait for storage and no allocation don technically fail, so OOM killer no dey ever get called. Check /proc/pressure/memory while e dey happen: full avg10 above 40 mean say almost no task run for the last ten seconds. Userspace daemon like earlyoom fit kill before the box reach that state.

Wetin be the difference between MemoryHigh and MemoryMax?

MemoryHigh= na soft cap wey dey throttle. Kernel dey reclaim hard from the unit and slow down its allocations, but usage fit pass the number and nothing go get killed. MemoryMax= na hard cap: allocation wey no fit meet under am go invoke OOM killer inside that unit own cgroup, so na the process wey cause the problem go die instead of the biggest process for the box. Set MemoryHigh= below MemoryMax= and treat the gap between dem as warning zone.

How I fit find which service OOM killer hit?

Run journalctl -k --grep "Killed process" --since "2 hours ago". Line wey start with Memory cgroup out of memory mean say one unit hit its own MemoryMax=, while plain Out of memory mean say the whole machine run out. Then run journalctl -u <unit> -n 50 and look for Failed with result 'oom-kill'. If /var/log/journal no dey your server, journal stay for RAM and the evidence die together with the reboot, so create that directory before the next incident.

I suppose add swap to small VPS?

Small swap file fit help with cold pages wey dem allocate once and never touch again. E no help with runaway process: e dey delay the kill and replace short outage with long stall wey you no fit log in to fix. Keep swap modest, and set MemorySwapMax=0 for the units wey you ready to lose, so dem reach their ceiling and restart quickly while important services keep their swap.

I fit limit command without writing unit file?

Yes. sudo systemd-run --scope -p MemoryMax=1G -p CPUQuota=50% ./script.sh dey run the command for your terminal inside transient scope with those limits, and the limits go disappear when e exit. Every property from systemd.resource-control dey available after -p, so MemorySwapMax=, TasksMax= and CPUWeight= work there too. Remove --scope and add --unit=name to run the job for background with its output inside the journal.