Sandbox a systemd service with ProtectSystem

What systemd sandboxing does

systemd sandboxing is the second half of writing a unit file. Type= decides how a service starts, and directives like ProtectSystem= and PrivateTmp= decide what it can touch once it is running. They are kernel features, mount namespaces and seccomp filters, applied by systemd before your process gets control. The application never sees them and needs no code changes.

The default is nothing at all. A unit with no sandboxing runs as root, writes anywhere, reads every file on the box, and can load a kernel module. If that unit is a web application reachable from the internet, one file upload bug becomes a whole compromised server. Under the unit below, the same application gets a read-only file system, an empty /home, a /tmp no other process can see, and no route to root even if it finds a setuid binary.

All of this applies to one unit at a time. Hardening a service does nothing for its neighbours, so start with whatever listens on a public port.

The unit file, all at once

notes is a small web service. It listens on localhost, keeps a SQLite database under /var/lib/notes, and sits behind nginx. Type=exec fits because the binary stays in the foreground, and the difference between Type=simple, exec, forking and notify decides how systemd tracks the startup. Every directive below is explained further down, with what it prevents and what it commonly breaks.

[Unit]
Description=Notes web service
After=network-online.target
Wants=network-online.target

[Service]
Type=exec
ExecStart=/opt/notes/bin/notes --listen 127.0.0.1:8080
DynamicUser=yes
StateDirectory=notes
Environment=NOTES_DB=/var/lib/notes/notes.db

ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectProc=invisible

NoNewPrivileges=yes
CapabilityBoundingSet=
AmbientCapabilities=
RestrictSUIDSGID=yes

ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes

RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RestrictNamespaces=yes
LockPersonality=yes

[Install]
WantedBy=multi-user.target

Write it to /etc/systemd/system/notes.service, run sudo systemctl daemon-reload, then sudo systemctl restart notes.service. For a unit that came from a package, do not edit the vendor file. sudo systemctl edit notes.service opens a drop-in at /etc/systemd/system/notes.service.d/override.conf which holds only your [Service] additions and survives package upgrades. systemctl cat notes.service prints the merged result, vendor file first.

Who the service runs as

Without User=, a service runs as root, and every other directive here is damage control. There are two ways out of that.

A static system user. Create it with sudo useradd --system --no-create-home --shell /usr/sbin/nologin notes, then set User=notes in the unit. The UID stays the same across restarts and reboots, which matters when the service owns files outside its own state directory, or when a backup job has to read them. The reasoning behind giving every service its own unprivileged account applies here without change.

DynamicUser=yes. systemd allocates a UID from a reserved range when the service starts and releases it when the service stops. No account exists on disk, so there is nothing to leave behind when you remove the service. While the service runs, getent passwd notes resolves the name through systemd's NSS (name service switch) module. After it stops, the name is gone.

DynamicUser=yes also switches on four other settings for you: RemoveIPC=yes, PrivateTmp=yes, ProtectSystem=strict and ProtectHome=read-only. That is most of a sandbox from one line, which is why the example unit still reads short.

What it breaks. A dynamic UID cannot own files in arbitrary places, because the number is recycled between services. Persistent data has to go through StateDirectory=, CacheDirectory= or LogsDirectory=, which systemd creates and chowns to the current UID at every start. With DynamicUser=yes the real directory is /var/lib/private/notes, and /var/lib/notes is a symlink pointing at it. /var/lib/private is mode 0700 and owned by root, so a backup running as a normal user gets Permission denied on a path that looks perfectly readable to root. Anything else that wants a fixed owner, an SSH key, an NFS export, a file a second service reads, needs a static user instead.

What the file system looks like: ProtectSystem, ProtectHome, PrivateTmp

ProtectSystem= takes three values. yes mounts /usr and the boot directories read-only. full adds /etc. strict mounts the entire hierarchy read-only, apart from the kernel API directories /dev, /proc and /sys, which other directives cover. Start at strict and open holes, because starting loose and tightening later never happens.

Holes are ReadWritePaths=/srv/notes/uploads. You need fewer than you expect: StateDirectory=, LogsDirectory=, CacheDirectory= and RuntimeDirectory= stay writable under ProtectSystem=strict automatically, which is why the example unit has no ReadWritePaths= line at all. Under strict, /tmp is read-only too, unless PrivateTmp=yes hands the unit its own writable one.

What it breaks. Any write outside those paths now fails with Read-only file system. Applications that save their own config back into /etc, drop a PID file straight into /run, or unpack a plugin into /opt all hit this. Read the path out of the error and add that one path, not its parent. A path listed in ReadWritePaths= that does not exist is a start failure rather than a warning, so prefix the optional ones with a hyphen: ReadWritePaths=-/srv/notes/uploads.

Read-only is not the same as hidden. Under ProtectSystem=strict the service can still read /etc/passwd, and any world-readable secret belonging to another application. InaccessiblePaths=/etc/ssh /srv/otherapp removes a subtree from the unit's view completely. For the service's own secrets, LoadCredential=dbpass:/etc/notes/dbpass copies the file into a per-unit directory that only this service can read, and the app finds it under $CREDENTIALS_DIRECTORY.

ProtectHome=yes makes /home, /root and /run/user appear empty. A web service has no business in a home directory, and this stops a path traversal bug from reaching /root/.ssh. read-only and tmpfs are the softer values. It breaks anything whose data really does live in a home directory, which covers a lot of hand-installed applications under /home/app. Move the data to /var/lib, or set ProtectHome=read-only and accept the smaller win.

PrivateTmp=yes gives the service its own /tmp and /var/tmp, created when it starts and deleted when it stops. It ends the entire class of temp file races between services, and a crash can no longer leave secrets in a directory every user on the box can list.

What it breaks. Anything that treats /tmp as a meeting point. A service configured to reach MySQL through /tmp/mysql.sock now reports Can't connect to local MySQL server through socket '/tmp/mysql.sock', because the database created its socket in the host's /tmp while the service is looking inside its own. Point it at 127.0.0.1 or at the real socket path under /run. The same surprise arrives while debugging: files the service writes to /tmp do not show up in your shell's /tmp. To look inside, enter the service's mount namespace.

pid=$(systemctl show --property=MainPID --value notes.service)
sudo nsenter --target "$pid" --mount ls -l /tmp

PrivateDevices=yes replaces /dev with a small set of pseudo devices such as /dev/null, /dev/zero and /dev/urandom. Physical devices are simply not there. Disk nodes, /dev/kvm, /dev/net/tun, sound cards and GPUs all disappear, so a media server doing hardware transcoding fails to open /dev/dri/renderD128 and either falls back to software or exits. When a service genuinely needs one device node, turn PrivateDevices= off for that unit and name the node with DeviceAllow=/dev/dri/renderD128 rw, which is still far narrower than the default of every device on the system.

What the service can become: NoNewPrivileges and capabilities

NoNewPrivileges=yes sets a process flag the kernel never clears. From that moment the process and every child it starts cannot gain privileges through a setuid binary or a file capability. It is the most valuable single line in the unit, because it makes most local privilege escalation chains stop at their first step.

What it breaks. Anything that calls sudo from inside the service, which now prints sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?. PAM (pluggable authentication modules) password checks that shell out to unix_chkpwd fail in the same way, and so do rootless container tools that need newuidmap. If your service depends on one of those, remove the dependency rather than the directive.

CapabilityBoundingSet= limits which capabilities any process in the unit may ever hold, and an empty assignment drops all of them. For a service that already runs as a non-root user this is a second lock rather than the first one, since NoNewPrivileges=yes blocks the usual way of picking a capability up. Keep both, because they fail differently and you want the failure caught twice.

The one capability a web service often does want is CAP_NET_BIND_SERVICE, for a port below 1024. A non-root process needs it granted, not merely permitted, so both lines are required.

CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE

systemd applies ambient capabilities before it drops privileges, so this still works alongside NoNewPrivileges=yes. Without those lines the service starts and then exits with an error naming the port, such as listen tcp :443: bind: permission denied. On most VPS setups the better answer is to bind 127.0.0.1:8080 and let nginx or Caddy own 443, which keeps the capability out of the unit entirely. A leading ~ inverts the list, so CapabilityBoundingSet=~CAP_SYS_ADMIN blocks that one and allows the rest. Prefer the allow form: a deny list ages badly, because new capabilities keep being added.

What the kernel exposes

ProtectKernelTunables=yes makes /proc/sys, /sys and the writable files under them read-only for this unit. It breaks any service that sets a sysctl during startup: the start script prints sysctl: setting key "vm.max_map_count": Read-only file system and gives up. Put the value in /etc/sysctl.d/ instead, where it belongs and where it survives a reboot, then leave the directive on.

ProtectKernelModules=yes blocks module loading. A unit that runs modprobe gets modprobe: ERROR: could not insert 'nf_conntrack': Operation not permitted. Load the module at boot through /etc/modules-load.d/ rather than from the service.

ProtectKernelLogs=yes takes away dmesg. ProtectControlGroups=yes makes /sys/fs/cgroup read-only, which container runtimes and anything managing its own cgroups will notice immediately. ProtectProc=invisible hides other users' processes in /proc, so a compromised service cannot read another daemon's command line and the password somebody passed on it. Monitoring agents that walk /proc are the ones that need this off.

RestrictNamespaces=yes stops the service creating new namespaces, which is what a container runtime needs and what an attacker uses to build an escape. LockPersonality=yes blocks changing the kernel execution domain, and RestrictSUIDSGID=yes stops the service creating setuid files. Both are cheap and rarely break an ordinary application.

What the service can talk to

RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 allows local sockets plus IPv4 and IPv6, and makes socket() fail with EAFNOSUPPORT for everything else. It is a seccomp filter, so it works on x86-64 and arm64, which covers any current VPS.

What it breaks. AF_NETLINK, far more often than people expect. glibc's getifaddrs() uses a netlink socket, and so do the interface enumeration paths in the Go, Java and .NET runtimes, so a service that only wanted to learn its own IP address dies with OSError: [Errno 97] Address family not supported by protocol or the equivalent in its language. When that happens the list becomes AF_UNIX AF_INET AF_INET6 AF_NETLINK, which is still much narrower than the default. AF_PACKET is what raw packet capture needs, and almost nothing else should ever have it.

IPAddressDeny=any with IPAddressAllow=localhost is a different mechanism: a BPF filter attached to the unit's cgroup. It is per unit and invisible to nft list ruleset, which makes it a good fit for a service that should only ever reach a database on the same host, and a confusing one for whoever debugs the box next. On a kernel without cgroup BPF support systemd logs that the unit configures an IP firewall while the local system does not support BPF/cgroup firewalling, and the rules quietly do nothing, so read the journal instead of assuming.

Why does a hardened unit stop starting?

Every directive above can turn a working service into a failing one, and the failure often looks nothing like the directive that caused it. The loop is always the same. Read the journal, relax exactly one directive, retest.

sudo systemctl restart notes.service
systemctl status notes.service
sudo journalctl -u notes.service -n 50 --no-pager

The lines that name the sandbox look like this:

notes.service: Failed to set up mount namespacing: No such file or directory
notes.service: Failed at step NAMESPACE spawning /opt/notes/bin/notes: No such file or directory
notes.service: Main process exited, code=exited, status=226/NAMESPACE

226/NAMESPACE means systemd could not build the file system view, so your binary never ran at all. The usual cause is a path in ReadWritePaths=, BindPaths= or InaccessiblePaths= that does not exist. 228/SECCOMP points at SystemCallFilter= or SystemCallArchitectures= failing to apply. code=killed, status=31/SYS is different again: the process started, then made a system call the filter denied, and the kernel killed it. Keep the systemd exit code reference for units that will not start open while you work through this, because the number is the fastest way to tell a namespace problem from an application problem.

Relax one directive at a time, in a drop-in, so the result tells you something. Run sudo systemctl edit notes.service and put a single line in it:

[Service]
ProtectSystem=full

Restart. If the service now starts, you know which directive to narrow instead of delete. Put it back to strict, add the ReadWritePaths= line for the one path the application actually needs, and restart again. Deleting the block because the service would not start is how a unit ends up with no protection and a comment nobody remembers writing.

To test a sandbox with no service involved, run a shell inside one:

sudo systemd-run --pty -p ProtectSystem=strict -p ProtectHome=yes -p PrivateTmp=yes /bin/bash

Inside that shell, touch /etc/test returns Read-only file system and ls /home shows nothing. It is the quickest way to find out what an application will see, because you can run its commands by hand and watch which one fails.

One failure mode produces no error at all. A misspelled directive is only a warning, and the service starts without it:

/etc/systemd/system/notes.service:14: Unknown key name 'ProtectSytem' in section 'Service', ignoring.

The unit runs, the sandbox is absent, and nothing further complains. Two commands catch this. sudo systemd-analyze verify /etc/systemd/system/notes.service prints the same warning on demand, and systemctl show prints what the running service actually received:

systemctl show notes.service -p User -p ProtectSystem -p PrivateTmp -p NoNewPrivileges

If that returns ProtectSystem=no when you wrote strict, the unit did not load what you think it did.

Use systemd-analyze security as a checklist

sudo systemd-analyze security notes.service prints every sandboxing setting a unit could use, what this unit currently does with each one, and a judgement per row. Run it with no argument to list every loaded service on the box, and add --offline=true with a path to check a unit file before you install it.

Read the output as a to-do list. Go down the rows it flags and answer one question for each: does this service need that access? Most of the time the answer is no, and you add the line. Sometimes the answer is yes. A media server needs a device node. A backup agent needs to read /home. Those rows stay flagged forever, and that is the correct outcome, not a failure.

The summary number at the end is a roll-up of the rows above it. It does not know what your service does, what data it holds, or whether the application has a bug in the first place. A unit can satisfy every row the tool checks and still be the weakest thing on the server, because what it measures is the exposure of the unit file rather than the exposure of the software. Chasing the number leads people to paste in directives they do not understand, and those are exactly the ones that break after a package upgrade, with nobody left who can explain why the line was there.

Where sandboxing ends

These directives control what a service can reach. They say nothing about how much it can consume, so a fully sandboxed unit can still take every core and all the memory on the box. That is a separate set of settings, covered in the guide to CPU and memory limits for a systemd service.

They also do not replace mandatory access control. A namespace is per unit and set by whoever writes the unit file, while SELinux enforces one policy across the whole system. The two work together, and neither removes the need for the other.

Finally, they cover only the processes systemd starts inside this unit. A service that hands work to a helper daemon over a socket has not sandboxed the helper. Write the same block into that unit too, and check the result with systemctl show rather than trusting the file.

FAQ

What does ProtectSystem=strict actually make read-only?

The whole file system hierarchy, apart from /dev, /proc and /sys, which are covered by PrivateDevices=, ProtectKernelTunables= and ProtectControlGroups= instead. That includes /etc, /var, /srv, /opt and /tmp. The exceptions systemd adds back for you are the directories it manages: StateDirectory=, CacheDirectory=, LogsDirectory= and RuntimeDirectory=. Anything else the service must write needs an explicit ReadWritePaths= entry. Note that read-only does not mean unreadable, so a secret file elsewhere on the box is still open to the service unless you list it in InaccessiblePaths=.

Why does my service fail with status=226/NAMESPACE?

systemd could not build the mount namespace, so the executable never started. The journal line above it usually reads Failed to set up mount namespacing: No such file or directory. In almost every case a path in ReadWritePaths=, BindPaths= or InaccessiblePaths= does not exist on disk. Create the directory, or prefix the entry with a hyphen (ReadWritePaths=-/srv/notes/uploads) so systemd skips it when it is missing. If the path does exist, check for a typo in the unit and confirm with systemctl cat notes.service, since a drop-in may be adding a line you are not looking at.

Can a service with PrivateTmp still share files through /tmp?

No, and that is the point. The service gets a fresh /tmp and /var/tmp that exist only for as long as it runs, so a socket or file another process created in the host's /tmp is invisible to it. A database socket at /tmp/mysql.sock is the common casualty, and the fix is to connect over 127.0.0.1 or to point the client at the real socket under /run. To inspect the service's own temporary files, get its main PID from systemctl show --property=MainPID --value and enter its mount namespace with sudo nsenter --target <pid> --mount.

How does a sandboxed service listen on port 443 without root?

Grant one capability rather than the whole root account. Set AmbientCapabilities=CAP_NET_BIND_SERVICE together with CapabilityBoundingSet=CAP_NET_BIND_SERVICE, keep User= or DynamicUser=yes, and the process can bind low ports and nothing else. Setting only the bounding set is the usual mistake: the capability is then permitted but never granted, and the service exits with a permission error naming the port. On a VPS running a reverse proxy already, the cleaner answer is to bind 127.0.0.1:8080 in the service and let nginx hold 443, so the unit needs no capabilities at all.

Should I use DynamicUser instead of creating a system user?

Use it when the service keeps all of its data inside StateDirectory=, CacheDirectory= or LogsDirectory=, which covers most small self-hosted web applications. You get a UID that exists only while the service runs, plus PrivateTmp=, ProtectSystem=strict, ProtectHome=read-only and RemoveIPC= switched on for free. Use a static system user when the UID has to stay stable: files owned outside those directories, an SSH key, an NFS mount, or a second process that reads the same data. Remember that with DynamicUser=yes the data really lives in /var/lib/private/notes, and that directory is mode 0700 and owned by root, which is what makes non-root backup jobs fail.