Docker Compose memory limits that stop OOM

What a Docker Compose memory limit does

A Docker Compose memory limit is a hard cap the Linux kernel puts on one container's cgroup (control group, the kernel feature that meters resources for a set of processes). Set deploy.resources.limits.memory on a service and that container can never use more than the number you wrote. When it tries, the kernel kills a process inside the container, and the container usually exits with code 137.

This matters most on a VPS, where the RAM is fixed and there is no spare host memory to borrow. One container with a memory leak or a bad query will take every free page on an 8GB box. The kernel then kills whatever process it judges worst, which is often a database or your SSH session rather than the container that caused the problem. Limits turn a whole-server outage into one service that restarts.

services:
  app:
    image: ghcr.io/example/app:1.4
    deploy:
      resources:
        limits:
          cpus: "1.5"
          memory: 1g
        reservations:
          memory: 256m

Apply it and confirm the limit is live:

docker compose up -d
docker stats --no-stream

The MEM USAGE / LIMIT column should read something like 142MiB / 1GiB. If the limit column shows the full host RAM, the setting did not apply, and the rest of this guide will not help until it does. If the compose file itself is new to you, the Docker Compose basics for a VPS cover the file layout this builds on.

deploy.resources.limits or mem_limit: which one applies

Two spellings exist for the same idea, which is why this is confusing.

mem_limit, mem_reservation, memswap_limit, cpus and cpu_shares are top-level service keys inherited from the older Compose file formats. deploy.resources came from the Swarm schema and is now part of the Compose Specification, which is the format docker compose reads today.

Both work on a single host. Docker Compose, the docker compose plugin, applies deploy.resources.limits and deploy.resources.reservations when you run docker compose up, with no Swarm cluster anywhere. The plugin is now on its v5 line (v5.6.0 was the latest stable release when checked on 2026-10-10). It jumped from v2 straight to v5, skipping 3 and 4 to avoid confusion with the old Compose file format versions, and no v5 release has changed how resource limits are handled. The Swarm-oriented parts of the deploy block are the other keys, such as placement, update_config and endpoint_mode, which describe scheduling and rollouts across a cluster of nodes for docker stack deploy. So the common advice that "deploy needs Swarm" is wrong for the resources subsection, and following it leaves your services with no limit at all.

Pick one spelling per project. Writing mem_limit: 512m and deploy.resources.limits.memory: 1g on the same service is more than hard to read: the Compose Specification says that when mem_limit is set it must be consistent with limits.memory, and it does not say which value wins when they disagree. Whichever spelling you use, confirm what the daemon actually applied:

docker inspect --format '{{.HostConfig.Memory}} {{.HostConfig.MemoryReservation}} {{.HostConfig.NanoCpus}}' app-1

Memory values are bytes, so 1g prints as 1073741824. CPU is in nano CPUs, so 1.5 prints as 1500000000. A 0 in any field means no limit was set. The smallest memory limit Docker accepts is 6m, and below that the container refuses to start.

What happens when a container hits the limit

The container does not slow down. It dies.

When a process asks for a page and the cgroup is already at its memory.max, the kernel first reclaims what it can inside that cgroup: clean page cache, then pages it can swap. If reclaim does not free enough, the cgroup OOM (out of memory) killer picks a process inside the container and sends it SIGKILL. Killing the container's PID 1 ends the container. Exit code 137 is simply 128 plus signal 9, so 137 is the fingerprint of any SIGKILL, not proof of an OOM by itself.

docker compose ps -a
docker inspect --format '{{.State.OOMKilled}} {{.State.ExitCode}}' app-1

true 137 is an OOM kill. false 137 usually means something else sent SIGKILL (a program can also exit with 137 on purpose, but that is rare), and the usual cause is docker compose stop hitting its ten second grace period because the app ignored SIGTERM. That distinction saves hours, because the two problems have nothing in common.

Two more places record the event. Watch the daemon live:

docker events --filter event=oom

Then read the kernel log, which is the record that survives a restart:

sudo dmesg -T | grep -i -E 'memory cgroup out of memory|killed process'

A cgroup kill prints a line beginning Memory cgroup out of memory: Killed process 24713 (node). That line means one container reached its own limit, and it can appear while the host still has plenty of free RAM. A line without the Memory cgroup prefix is a host OOM, which means the machine itself ran out of RAM. That is the failure limits are meant to prevent, so seeing it usually means the sum of your limits is too high, that some services have no limit at all, or that something outside Docker is using more than the headroom you left.

With restart: unless-stopped, an OOM loop hides well, because the service looks up in docker compose ps a second after it died. Check the uptime column and the restart count, and pair the limit with a healthcheck that reports the app as unhealthy so a container that keeps dying is visible without you watching it.

Reservation is a hint, the limit is the rule

reservations.memory (the older mem_reservation) is a soft floor. Docker describes it as a soft limit that is activated when the daemon detects contention or low memory on the host. It never stops a container going above it, and it never guarantees the memory will be free when the container asks. It only biases the kernel toward reclaiming from containers that are above their reservation first.

So a reservation protects nothing on its own. Use it to mark a service you want treated kindly under pressure, and rely on the limit for safety. Keep the reservation below the limit, or the container will not start: Docker rejects the config with Minimum memory limit can not be less than memory reservation limit.

Swap accounting, honestly

Most VPS images ship with no swap file at all. Run swapon --show and free -h. If the swap total is zero, every swap-related setting below does nothing, and your memory limit is a pure RAM cap.

memswap_limit is not the amount of swap. It is the total of memory plus swap. With mem_limit: 1g and memswap_limit: 2g, the container gets 1GB of RAM and 1GB of swap. Setting the two values equal gives the container no swap at all. Setting mem_limit and leaving memswap_limit unset lets the container swap up to the size of its memory limit again.

Ubuntu 24.04 and Debian 13 use cgroup v2 by default, where swap is a separate counter (memory.swap.max) and this works with no extra setup. The old message Your kernel does not support swap limit capabilities comes from cgroup v1 hosts booted without swapaccount=1. On those the memory limit still applies while the swap part is ignored.

Be honest about what swap buys you. It makes an OOM kill slower, not less likely, because a leaking process fills swap as happily as it fills RAM. Meanwhile a container thrashing swap on shared VPS storage slows down every other service on the box. For anything latency sensitive, a correct limit with no swap fails faster and more predictably.

Why memory usage looks worse than it is

On Linux, the MEM USAGE figure in docker stats subtracts only inactive page cache (inactive_file on cgroup v2), so recently used cache still counts, and a container that reads large files climbs toward its limit and sits there. That is normal, and it is not a leak, because clean cache is reclaimed before the OOM killer is ever called. A service like a self-hosted Jellyfin media server will look permanently near its ceiling for exactly this reason.

Split the number into cache and real working set from inside the container:

docker compose exec app grep -E '^(anon|file) ' /sys/fs/cgroup/memory.stat
docker compose exec app cat /sys/fs/cgroup/memory.events

anon is anonymous memory, the working set that cannot be dropped. file is page cache, which can. Size your limit against anon plus a margin, not against the total. The memory.events file settles the argument outright: an oom_kill counter above zero means the kernel has killed something in this container since it started, and a max counter climbing means the container is being held at its ceiling right now. Both commands need a shell and coreutils inside the image, so they fail on a distroless or scratch image.

Sizing limits on an 8GB VPS

Start from the host, not from the apps. On an 8GB VPS, leave about 1GB for the kernel, the Docker daemon, sshd, journald and your own login shell. That leaves roughly 7GB to hand out, and the sum of every container limit should stay under it. Overcommitting works right up to the day two services peak together.

A workable split on an 8GB box:

• Reverse proxy: 128m limit. It is a small process, and a limit this tight catches a runaway config reload immediately.
• PostgreSQL: 2g limit, with shared_buffers set to about 512MB in the database config.
• Application container: 1g limit.
• Background worker: 512m limit.
• Media or file service: 2g limit, most of which will be page cache.

Do not copy those numbers into your own stack. Run the services under real load for a day, watch docker stats, take the peak anon value per container and add roughly half again as headroom. A limit set too tight is worse than no limit, because it kills a healthy service during a normal traffic spike.

One trap deserves its own note. Some runtimes ignore the limit unless you tell them about it, and the ones that do read it may not size themselves the way you expect. PostgreSQL will happily size shared_buffers and work_mem past its container limit and get killed. A modern JVM (Java virtual machine) detects the cgroup limit by default, but HotSpot builds such as OpenJDK then give the heap only 25 percent of it, so -XX:MaxRAMPercentage=75 is the usual way to let the heap use more of the limit while leaving room for memory outside the heap. Node.js has sized its default heap from the cgroup limit since version 12, at about half the container limit and levelling off around 2GB, so set --max-old-space-size in megabytes only when you want a different ceiling, and keep it below the container limit so the garbage collector works before the kernel intervenes. Ollama is the same story with a different knob, because raising num_ctx grows the KV cache by hundreds of megabytes and the container dies partway through a long prompt. The cgroup does not negotiate. It kills.

CPU limits behave completely differently

cpus: "1.5" means 150% of one core, enforced as a CFS (completely fair scheduler) quota. The container gets 150ms of CPU time in every 100ms period, shared across all its threads. When it uses that up, the kernel makes it wait for the next period.

That is the important contrast. A container over its memory limit is killed. A container over its CPU limit is throttled and carries on, more slowly. So a CPU limit is safe to set aggressively, while a memory limit needs headroom.

cpu_shares is a different tool: a relative weight that only matters when the CPUs are actually saturated. Two containers with shares of 1024 and 512 split a busy core roughly two to one, and on an idle box neither one is restricted. Use shares to rank services by importance, and use cpus when you need a real ceiling, for example to stop a nightly transcode job from starving your web server.

FAQ

Does deploy.resources.limits work without Docker Swarm?

Yes. Docker Compose, the docker compose plugin (now on its v5 line), applies deploy.resources.limits and deploy.resources.reservations when you run docker compose up on a single host. Confirm it with docker inspect --format '{{.HostConfig.Memory}}' <container>, which prints the limit in bytes and prints 0 when no limit was applied. The keys inside deploy written with Swarm in mind are the cluster ones, such as placement, update_config and endpoint_mode.

What does exit code 137 mean in Docker Compose?

It almost always means the main process received SIGKILL, since 137 is 128 plus signal 9 (a program can also exit with 137 deliberately, but that is rare). The kernel OOM killer is the common cause, but a shutdown timeout produces the same code when an app ignores SIGTERM. Run docker inspect --format '{{.State.OOMKilled}} {{.State.ExitCode}}' <container> to tell them apart. true 137 is a memory kill, and false 137 is not.

Should I set mem_limit or deploy.resources.limits.memory?

Either works with docker compose. deploy.resources.limits.memory is the current Compose Specification form and is the better default for a new file. Keep mem_limit if the rest of your file already uses the older top-level keys. The Compose Specification requires the two to be consistent when both are set, so pick one and verify the result with docker inspect.

Why is my container sitting at its full memory limit without being killed?

The usage figure in docker stats still counts recently used page cache, which the kernel drops under pressure instead of triggering an OOM kill. Run docker compose exec <service> grep -E '^(anon|file) ' /sys/fs/cgroup/memory.stat and read the anon value, which is the working set that cannot be reclaimed. A high file value next to a low anon value is a container doing disk input and output, not a container about to die.

How much RAM should I leave unallocated on an 8GB VPS?

Leave about 1GB for the kernel, the Docker daemon, sshd, journald and your own shell, then keep the sum of all container limits under the remaining 7GB. Watch the peak anon value per container under real load for a day before you commit to numbers, and treat the total as a budget rather than a target to fill.