How to Limit VPS CPU and Memory with systemd

Process memory and CPU limit with systemd drop-in

You fit limit process memory and CPU for Linux VPS by adding small lines to the unit wey dey run the process. MemoryMax= na the hard memory limit. CPUQuota= na the limit for processor time. cgroup v2 (control groups, version 2) dey enforce both limits. Na kernel feature systemd already dey use to account for every service for the box.

sudo systemctl edit myapp.service

That one go open drop-in file wey get instructions inside comments. Add this one above dem:

[Service]
MemoryHigh=512M
MemoryMax=768M
MemorySwapMax=0
CPUQuota=80%
TasksMax=128

sudo systemctl daemon-reload
sudo systemctl restart myapp.service
systemctl show myapp.service -p MemoryHigh -p MemoryMax -p CPUQuotaPerSecUSec -p TasksMax

systemctl show must repeat your numbers for the kernel own units: MemoryMax=805306368 and CPUQuotaPerSecUSec=800ms. If e print MemoryMax=infinity, the drop-in no load. Check say the file land for /etc/systemd/system/myapp.service.d/override.conf, and say e start with the [Service] header. If settings line no get section above am, systemd go log Assignment outside of section. Ignoring. and start the service without any limit.

The rest of this guide explain how to choose those numbers, and wetin fit still go wrong after you set dem.

Why runaway process fit freeze VPS without e filling memory

Process wey reach hard memory cap go die for about one second, then service go restart. Na the good case be that. The bad case na when nothing die: box dey answer ping, SSH dey accept connection, but shell prompt no show. Machine dey alive and busy, but none of the work useful.

Make we explain how e happen, because e no obvious. When free memory dey low, kernel go reclaim pages instead of giving out new ones. File-backed pages dey cheapest to reclaim, and page cache dey hold executable code for everything wey dey run. So kernel go evict the text pages of sshd, and the next instruction wey sshd run go be a page fault wey need read those bytes back from storage. Every process go end up dey wait for disk instead of running. The same pages go leave and come back repeatedly. Dem dey call this thrashing.

Two things make this worse for VPS than for laptop. Storage often dey network-attached or shared, so every fault dey cost more milliseconds than local NVMe device go cost. Also, kernel no dey measure time; e dey measure failure. As long as reclaim still dey return one page, no matter how slowly, kernel believe say e dey make progress and e no call out of memory (OOM) killer. Box fit remain for this state for many minutes before anything get killed.

You fit monitor am as e dey happen. Kernel dey export pressure stall information (PSI) for Linux 4.20 and newer:

cat /proc/pressure/memory
cat /proc/pressure/io

some avg10=63.72 avg60=41.02 avg300=12.33 total=13729481
full avg10=48.15 avg60=30.44 avg300=8.90 total=9114233

The full line na the important one. full avg10=48.15 mean say for the last ten seconds, 48% of the time every runnable task for the box dey stalled, waiting for memory work, so nothing run. Healthy server dey show value close to zero for full. Above 10, human go feel say e slow. 40 or more na the state wey people describe as frozen.

Na this one also explain why limit by itself no be promise. Unit wey dey under MemoryHigh= go get throttled instead of getting killed. E go stay alive and stay slow, and nothing go restart am because from systemd point of view e never fail. Capped unit wey still get permission to swap go generate reads and writes wey dem charge to that unit, but one shared device go serve dem. This fit push /proc/pressure/io up for every other service for the box. Limits decide who go pay for shortage; dem no fit create capacity.

Check say your VPS dey run cgroup v2

stat -fc %T /sys/fs/cgroup

cgroup2fs na the unified hierarchy, and na wetin every setting below need. tmpfs mean say the box boot with the older v1 layout, where MemoryHigh= and MemorySwapMax= no dey exist and per-unit OOM behaviour dey different. Ubuntu 22.04 and later, plus Debian 11 and later, dey use v2 by default. Old image, or kernel wey boot with systemd.unified_cgroup_hierarchy=0, no dey use am.

For cgroup v2, systemd dey turn on memory accounting for every unit by default, so the numbers don already dey available:

systemd-cgtop -m

This one go list cgroups according to memory use, and na the fastest way to answer “wetin dey chop this box” while e still fit respond. If the server new, make you first do the account and firewall work for the first ten minutes for new VPS before this one.

MemoryHigh dey throttle. MemoryMax dey kill.

Difference between the two memory settings dey decide how failure go look.

• MemoryHigh= na soft cap. When usage pass am, kernel dey reclaim memory aggressively from that cgroup and deliberately slow down its allocations. Usage fit still pass the number, and nothing go get killed.
• MemoryMax= na hard cap. When allocation no fit satisfy under am, OOM killer dey run inside that cgroup and kill one process wey belong to that unit.

Na that second part be the real reason to set MemoryMax= for anything wey you no fully trust. Without cap, shortage become problem for the whole machine, and global OOM killer dey choose victim based on oom_score, wey mostly mean the biggest process. Your database na usually the biggest process, no be the script wey leak memory. With cap, the kill go happen inside the unit wey cause am.

Set both, with MemoryHigh= about 20 to 30 percent below MemoryMax=. The gap na warning zone: slow memory leak go pass High and show as service wey don slow down, while sudden spike go pass straight through Max and die.

Percentage values dey use installed physical memory as reference, so MemoryMax=25% for 4 GB plan na 1 GB and e go remain one-quarter of the machine after you resize the plan. MemorySwapMax=0 dey keep that unit completely out of swap, so long crawl go turn to fast, obvious kill.

Cap need restart policy beside am. Otherwise, the kill go only leave you with stopped service.

[Unit]
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
Restart=on-failure
RestartSec=5s

StartLimit* belong inside [Unit], while Restart= belong inside [Service]. If you put either one for wrong section, systemd go ignore am. Five restarts within five minutes mean say na leak, no be temporary blip. After that, systemd go give up and leave the unit failed. Na the state you want to find later, instead of crash loop wey dey hide the problem.

Cap CPU with CPUQuota, or share am with CPUWeight

CPUQuota= dey take percentage of the time wey one CPU get available. CPUQuota=50% na half of one core. CPUQuota=200% na equal to two cores, and the unit fit spread am across as many threads as e want. For 2 vCPU plan, CPUQuota=200% na the whole machine.

CPUWeight= na better default for most services. E dey use relative share from 1 to 10000, and kernel default na 100. E only matter when something dey compete: backup job for CPUWeight=20 go yield to web server for 100 under load, but e still fit use the whole machine when machine dey idle. Hard quota go throw away that idle capacity.

Make you understand wetin CPU limit fit give you. CPU-bound process hardly freeze Linux, because scheduler dey continue give everybody CPU time. Na memory dey usually bring machine down. Use CPUQuota= when you want predictable ceiling, for example for build or agent wey otherwise go run full speed for one hour. Sizing that kind workload na separate matter, and how much RAM and CPU coding agent VPS need cover am.

If CPU dey show as busy while none of your processes dey do plenty work, the cause fit dey for the other side of hypervisor. Na CPU steal time from noisy neighbour be that, and no quota wey you set go change am.

TasksMax dey stop fork loop

TasksMax= na the number of processes and threads wey one unit fit hold. Threads count too, so Java or Go service need more headroom than wetin process list dey show. Na the cheapest protection against script wey dey fork for loop, because fork go fail inside the unit instead of make the whole box run out of process IDs.

TasksMax=128

When one unit reach the limit, kernel go log one line wey name the cgroup:

cgroup: fork rejected by pids controller in /system.slice/myapp.service

The program itself usually report fork: retry: Resource temporarily unavailable. Check wetin manager dey apply by default with systemctl show -p DefaultTasksMax.

Limit one one-off job with systemd-run

You no need unit file to use any of dis. systemd-run dey build one transient unit around one command.

sudo systemd-run --scope -p MemoryMax=1G -p MemorySwapMax=0 -p CPUQuota=50% -p TasksMax=64 ./import-data.sh

--scope dey run the command for your terminal, after e print Running scope as unit: run-r7c1a....scope. Output dey remain for your screen, and the limits go disappear when the command exit. Any property from systemd.resource-control go work after -p.

For long job, remove --scope and give am name. E go then run for background as transient service and log go journal:

sudo systemd-run --unit=nightly-import -p MemoryMax=1G -p CPUWeight=20 ./import-data.sh
journalctl -u nightly-import -f

The same options dey work with --user when you no be root, but your user manager only get the controllers wey dem delegate give am, so e fit reject one property there. Run am with sudo if that one happen. When job get permanent place, move the settings unchanged enter real unit: see run script as systemd service and timer.

The swap question, answered honestly

Swap no dey stop failure; e dey change the way failure go happen.

If swap no dey, memory leak go reach limit, then something go die within seconds. Outage go loud, short, and easy to understand for journal afterwards. If swap dey, kernel go write cold anonymous pages go disk and buy time. If process suppose level off, swap go save you. If na runaway process, swap go turn five second outage into twenty minute stall. The stall worse, because dead process still leave working shell for you, but box wey dey thrash no go.

swapon --show
free -h

One workable middle ground for small VPS na to keep modest swap file for pages wey dem allocate once and never touch again. Set MemorySwapMax=0 on the units wey you fit afford to lose. Important services keep their swap. The unpredictable ones go hit the limit quickly and restart.

Lowering vm.swappiness na weak lever, and e good make you know why. E only shift the balance between evicting page cache and swapping anonymous pages. Both options go cost disk read later. E change which pages dey thrash, not whether the box go thrash.

Early OOM daemon dey kill before stall happen

Kernel dey wait make reclaim fail completely. For small VPS, na this wait be the exact time wey machine fit lost. Two userspace daemons fit close this gap by monitoring memory by themselves and killing earlier.

earlyoom dey watch available memory and free swap. E dey kill the process wey get the highest score when either one drop below threshold.

sudo apt install earlyoom
systemctl status earlyoom

Debian and Ubuntu package dey start the service when installation finish. E options dey for /etc/default/earlyoom:

EARLYOOM_ARGS="-m 5,2 -s 5,2 --avoid '^(sshd|systemd)$' --prefer '^(node|python3)$'"

-m PERCENT dey set the minimum available memory, while -s PERCENT dey set the minimum free swap. Both na 10 percent by default. The second number for each pair na the SIGKILL point. earlyoom dey send SIGTERM once value drop below the first number, then SIGKILL below the second number. By default, the second number na half of the first. Apply change with sudo systemctl restart earlyoom. Read journalctl -u earlyoom to see which process e kill and how much memory that process dey hold.

systemd-oomd na the other option. E manual page describe am as "a system service that uses cgroups-v2 and pressure stall information (PSI) to monitor and take corrective action before an OOM occurs in the kernel space". E dey act on complete cgroups instead of individual processes. So e dey kill one unit, not one stray child. Units fit opt in with ManagedOOMMemoryPressure=kill or ManagedOOMSwap=kill. The thresholds dey for /etc/systemd/oomd.conf.

systemctl status systemd-oomd
oomctl

oomctl dey show wetin e dey monitor currently. For server image, this one often na nothing because setting dey opt in for each unit. Choose one daemon and stop there. If you run both, dem go race to choose victim, and e go harder to know the reason for any kill.

Which unit dey responsible?

Start with the kernel, because e dey record every kill wey e make.

journalctl -k --grep "Killed process" --since "2 hours ago"

A kill from the global OOM killer dey look like this:

Out of memory: Killed process 4127 (node) total-vm:2731084kB, anon-rss:1874232kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:4212kB oom_score_adj:0

anon-rss na the memory wey that process hold for RAM when e die, about 1.8 GB for here. Read the name for brackets with suspicion. Na the victim wey kernel choose be that, and kernel dey choose the biggest process. But the biggest process no always be the one wey cause the shortage.

A kill from cgroup limit get different prefix, and the report wey print above am name the cgroup wey reach its own ceiling:

Memory cgroup out of memory: Killed process 8811 (python3) total-vm:1044320kB, anon-rss:769112kB, file-rss:0kB, shmem-rss:0kB, UID:998 pgtables:1720kB oom_score_adj:0

That prefix na most of the diagnosis. Memory cgroup out of memory mean one unit reach the MemoryMax= wey you give am, while the rest of the box dey fine. Plain Out of memory mean the whole machine run out, so your caps either no dey or dem too generous to add up.

Then ask systemd wetin e see:

systemctl status myapp.service
journalctl -u myapp.service -n 50

myapp.service: A process of this unit has been killed by the OOM killer.
myapp.service: Main process exited, code=killed, status=9/KILL
myapp.service: Failed with result 'oom-kill'.

systemctl status talk the same thing for one line, as Active: failed (Result: oom-kill).

The cgroup counters na the third source, and na the only one wey record throttling, wey no dey produce log line at all:

cat /sys/fs/cgroup/system.slice/myapp.service/memory.events
cat /sys/fs/cgroup/system.slice/myapp.service/memory.peak

low 0
high 4213
max 118
oom 12
oom_kill 12

high count how many times the unit pass MemoryHigh= and get throttled. max count how often e reach the hard cap, while oom_kill count processes wey actually get killed. Big high with oom_kill 0 na the silent case from earlier: the service dey run, e slow reach crawl, and e never report failure to anybody. memory.peak (Linux 5.19 and newer) hold the highest usage wey the cgroup reach, and na this number you go use size MemoryMax= against. Both files reset when the unit restart, because systemd create the cgroup again.

One prerequisite dey under all this. If /var/log/journal no exist, the journal dey live for RAM, and every line go disappear after the reboot wey you need to recover the box.

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
journalctl --list-boots

journalctl --list-boots wey show more than the current boot mean history now dey survive, so journalctl -k -b -1 fit show you the kernel messages from the boot wey die.

Pikin VPS wey you fit start with

For 2 GB plan, leave 300 to 400 MB for kernel and page cache. No let all the caps add up reach the full 2 GB, because every unit fit peak for the same time. Give the service wey matter the biggest share, then cap every speculative service around am.

[Service]
MemoryHigh=256M
MemoryMax=384M
MemorySwapMax=0
CPUWeight=20
TasksMax=64
Restart=on-failure
RestartSec=5s

To keep one way to enter the server, add one more setting. OOMScoreAdjust=-500 inside drop-in for ssh.service makes the global OOM killer much less likely to pick your SSH daemon as victim. This one fit determine whether you go fix the box or reboot am from control panel. E only change the kernel choice of victim. E no make the stall shorter.

Containers dey run inside their own cgroups. Na container runtime create dem, no be your unit files. So, limit for docker.service no become limit for one container. The per-container equivalents of MemoryMax= and CPUQuota= dey covered for how to set memory and CPU limits for Docker Compose.

FAQ

Why my VPS freeze instead of killing the runaway process?

Because kernel dey judge progress by whether reclaim return pages, no be how long e take. When memory short, e dey evict page cache, including executable pages of programs wey dey run, then e read dem back for the next instruction. Everything dey wait for storage and no allocation don technically fail, so OOM killer no dey run. Check /proc/pressure/memory while e dey happen: full avg10 wey pass 40 mean say almost no task manage run for the last ten seconds. Userspace daemon like earlyoom fit kill the process before the box reach that state.

Wetin be the difference between MemoryHigh and MemoryMax?

MemoryHigh= na soft cap wey dey throttle. Kernel dey reclaim memory hard from the unit and slow down its allocations, but usage fit pass the number and nothing go die. MemoryMax= na hard cap: allocation wey no fit meet under am go invoke OOM killer inside that unit own cgroup, so the process wey cause the problem na im go die instead of the biggest process for the box. Set MemoryHigh= below MemoryMax= and treat the gap between dem as warning zone.

How I go find which service OOM killer hit?

Run journalctl -k --grep "Killed process" --since "2 hours ago". Line wey start with Memory cgroup out of memory mean say one unit hit its own MemoryMax=, while plain Out of memory mean say the whole machine run out. Then run journalctl -u <unit> -n 50 and look for Failed with result 'oom-kill'. If /var/log/journal no dey exist for your server, journal dey stay for RAM and the evidence die when system reboot, so create that directory before the next incident.

I suppose add swap to small VPS?

Small swap file dey help with cold pages wey system allocate once and never touch again. E no dey help with runaway process: e dey delay the kill and replace short outage with long stall wey you no fit login to fix. Keep swap modest, and set MemorySwapMax=0 for the units wey you ready lose, so dem reach their ceiling and restart quickly while important services keep their swap.

I fit limit command without writing unit file?

Yes. sudo systemd-run --scope -p MemoryMax=1G -p CPUQuota=50% ./script.sh dey run the command for your terminal inside transient scope with those limits, and the limits go disappear when e exit. Every property from systemd.resource-control dey available after -p, so MemorySwapMax=, TasksMax= and CPUWeight= work there too. Remove --scope and add --unit=name to run the job for background with its output for the journal.