SELinux basics for a server: what it breaks

Why nginx returns 403 on a file whose permissions are fine

nginx returning 403 on a file whose permission bits are correct is almost always SELinux (security-enhanced Linux) refusing the read. SELinux checks a second set of rules after the normal permissions pass, and the web server is only allowed to read files that carry a web content label. Your file carries a different label, so the open fails and nginx has nothing to send.

Look at the label, not only at the mode:

ls -ldZ /data/www /data/www/index.html

drwxr-xr-x. root root unconfined_u:object_r:default_t:s0 /data/www
-rw-r--r--. root root unconfined_u:object_r:default_t:s0 /data/www/index.html

The dot printed after drwxr-xr-x means the file carries an SELinux label. default_t is what a path gets when policy has never heard of it, and nothing in the web server's rules allows reading that type. The error log shows an ordinary Unix error, which is why this reads as a permissions bug:

2026/08/11 09:14:02 [error] 1183#1183: *1 open() "/data/www/index.html" failed (13: Permission denied), client: 203.0.113.9, server: _, request: "GET / HTTP/1.1"

The kernel returns 13: Permission denied for both kinds of refusal, the ordinary one and the SELinux one. So the first job is to find out which layer said no. Do not start with setenforce 0.

The part of the model you need

SELinux is mandatory access control, usually written MAC. Every process runs in a domain, such as httpd_t for the web server. Every file and every network port carries a type, such as httpd_sys_content_t. The policy is a list of allowed combinations of domain, type and action, and anything not on that list is denied. It runs after the classic Unix check, so the permission bits in drwxr-xr-x still have to allow the access first. Both layers must say yes.

A full context has four fields separated by colons, like system_u:system_r:httpd_t:s0: the SELinux user, the role, the type, and the level. On a server you spend nearly all your time on the third field, the type. Two commands show the live values:

ps -eZ | grep nginx
id -Z

The nginx workers show a context ending in httpd_t. Your login shell shows unconfined_u:unconfined_r:unconfined_t:s0, because the default targeted policy confines services and leaves interactive users alone. That is worth knowing, because SELinux does not replace running services under least-privilege users. It limits what a service can reach after somebody breaks into it.

The three modes, and which images have SELinux

sestatus
getenforce

Enforcing blocks and logs. Permissive allows everything and logs what it would have blocked. Disabled loads no policy at all. getenforce prints the current mode. sestatus also prints the mode from /etc/selinux/config, which is the one that comes back after a reboot.

Rocky Linux, AlmaLinux, Fedora and RHEL ship SELinux enforcing with the targeted policy. Ubuntu and Debian ship AppArmor instead, which does the same job with a different mechanism (the last section covers it). So the same application can install cleanly on one of your servers and return 403 on another.

Install the tools before you need them

sudo dnf install -y policycoreutils-python-utils setroubleshoot-server

semanage: command not found on a minimal image means policycoreutils-python-utils is missing: that package holds semanage and audit2allow. setroubleshoot-server adds sealert and writes a plain-English summary of each denial into the journal. Install both on a fresh server, because the moment you need them is the moment something is already broken.

How to read an SELinux denial in the audit log

Each refusal is recorded by the audit daemon as an AVC (access vector cache) message:

sudo ausearch -m AVC,USER_AVC,SELINUX_ERR -ts recent

type=AVC msg=audit(1754896442.881:412): avc:  denied  { read } for  pid=1183 comm="nginx" name="index.html" dev="vda1" ino=17203 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:default_t:s0 tclass=file permissive=0

Four fields carry the whole story. comm is the program that was blocked. scontext is the source context, the domain the process was running in. tcontext is the target context, the label on the thing it tried to touch. tclass is the kind of object, here a file. Read together: the process in httpd_t tried to read a file labelled default_t, and permissive=0 says the request was really blocked rather than only logged.

If ausearch prints nothing, the audit daemon may not be running. Denials then land in the kernel ring buffer instead:

sudo journalctl -k | grep -i avc

Now turn the record into a sentence of English:

sudo ausearch -m AVC -ts recent | audit2why
sudo journalctl -t setroubleshoot --since today
sudo sealert -a /var/log/audit/audit.log

audit2why reads the same records and names the cause it recognises: a boolean that is switched off, a label that does not match policy, or no rule at all. sealert walks the whole log and prints a suggested command per denial. Treat the suggestion as a hint. The wording changes between releases, and sealert sometimes proposes a custom policy module where a one-line label fix is the correct answer.

One more thing to know. Policy contains dontaudit rules that hide denials considered harmless, so a program can misbehave while the log stays empty. Unhide them for the length of one test:

sudo semodule -DB
# reproduce the problem, then read the log again
sudo semodule -B

Fix a mislabelled path with semanage fcontext and restorecon

Two commands, and the order matters. semanage fcontext -a records what the label for a path should be. restorecon applies that recorded default to the files on disk.

sudo semanage fcontext -a -t httpd_sys_content_t "/data/www(/.*)?"
sudo restorecon -Rv /data/www
ls -ldZ /data/www/index.html

The path is a regular expression. (/.*)? covers the directory itself and everything under it, which is what a document root needs. See what would change before changing it: sudo restorecon -Rvn /data/www prints the planned relabels, because -n means no action. After a real restorecon the label reads httpd_sys_content_t and the 403 is gone with no service restart.

Use chcon only as a test. chcon -t httpd_sys_content_t index.html sets the label directly, and the next restorecon, package update or full relabel resets it, because policy still says the path should be something else. semanage fcontext is the version that survives. List what you have recorded with sudo semanage fcontext -l | grep '^/data'.

Content the service must write to needs a different type. Use httpd_sys_rw_content_t for an upload directory or a cache, and keep it to those paths: a read-only site under a writable type hands out more access than the application needs.

Why was the label wrong at all? Almost always because of how the files arrived. mv keeps a file's existing label, so a site moved out of /root arrives labelled admin_home_t and stays that way. A plain cp gives the new file the default label of the destination directory, which is usually what you want, while cp -a and rsync -X copy the source labels along with the file. A git clone into a new top-level directory produces default_t. When a page loads fine from /usr/share/nginx/html and fails from your own directory, this is the reason.

Fix a class of behaviour with a boolean

Some failures are not a label problem. A reverse proxy on a fresh Rocky or AlmaLinux box returns 502, and the error log says:

2026/08/11 10:02:55 [crit] 1183#1183: *3 connect() to 127.0.0.1:3000 failed (13: Permission denied) while connecting to upstream

Your upstream is fine. The httpd_t domain is not allowed to open outbound network connections by default, so the connect() call is refused before it reaches the loopback interface. One switch controls that whole behaviour:

getsebool -a | grep httpd_can_network
sudo setsebool -P httpd_can_network_connect on

-P is the flag that matters: it writes the value to disk. Without -P the change is lost at the next reboot, which gives you a service that works until the machine restarts. Confirm with semanage boolean -l | grep httpd_can_network_connect, which prints the running value next to the stored one.

Prefer a boolean over a hand-written rule whenever one exists. Booleans ship with the distribution policy, so they are maintained, documented, and easy for the next person to find. getsebool -a lists every one on the system.

Let a service listen on a non-standard port

Ports are labelled too. Move nginx to 8081 and it refuses to start:

nginx: [emerg] bind() to 0.0.0.0:8081 failed (13: Permission denied)

httpd_t may bind ports labelled http_port_t, and 8081 is not one of them. Add it:

sudo semanage port -l | grep -w http_port_t
sudo semanage port -a -t http_port_t -p tcp 8081

Check the list first. Several high ports are already allowed, including 8008 and 8443, and adding one twice fails with ValueError: Port tcp/8081 already defined. If the port already belongs to a different type, change it with semanage port -m -t http_port_t -p tcp 8081 rather than adding it.

The same command is what makes a moved SSH port work. Bind to port 2222 on 0.0.0.0 failed: Permission denied in journalctl -u sshd means 2222 is missing from ssh_port_t, so run sudo semanage port -a -t ssh_port_t -p tcp 2222 before you restart the daemon and close your session. That is the step people skip when they follow a generic guide to hardening SSH on a VPS on a Red Hat family image. SELinux is not a firewall either, so the port still has to be open: sudo firewall-cmd --permanent --add-port=8081/tcp && sudo firewall-cmd --reload here, or ufw on a Debian or Ubuntu image.

When there is no boolean and no label to change

This is rare on a normal server, and it is where people cause damage. audit2allow can build a policy module from the denials in the log:

sudo ausearch -m AVC -ts recent -c nginx | audit2allow -M nginx_local
cat nginx_local.te
sudo semodule -i nginx_local.pp

Read nginx_local.te before you install it. Two habits keep this safe. Filter the input to the one program you are fixing with -c, because piping a week of unrelated denials into audit2allow grants all of them at once. And never install a module built from a denial you cannot explain: a rule that lets httpd_t read every file on the box is easy to generate and hard to spot months later. Remove a module with sudo semodule -r nginx_local.

Permissive is a diagnostic mode, not a fix

sudo setenforce 0
# reproduce the problem once, all the way through
sudo ausearch -m AVC -ts recent
sudo setenforce 1

Permissive mode allows the access and logs it. Its real value is completeness. Under enforcing, the service stops at the first denial, so you fix that one, restart, and meet the second. Under permissive the run continues and the log collects every denial in a single pass, and then you switch back and repair them together.

setenforce does not touch /etc/selinux/config, so a reboot returns the box to enforcing. That is a safety net, and it is also why a "fix" that consisted of setenforce 0 reappears at the worst possible moment. If one service needs room while you work on it, mark that domain instead of the whole machine: sudo semanage permissive -a httpd_t leaves everything else enforcing, and sudo semanage permissive -d httpd_t reverses it.

Why disabling SELinux costs more than fixing the label

Setting SELINUX=disabled in /etc/selinux/config trades a one-line label fix for a permanently weaker server. The difference shows up on the day a web application is compromised. Under enforcing, the attacker's code runs in httpd_t, so it may read web content, while reading /etc/shadow or writing a systemd unit is refused by policy whatever the Unix user would have permitted. With no policy loaded, that same code gets everything the service account has.

Disabling also has a bill you pay later. While no policy is loaded, new files are created with no label, so the filesystem drifts out of step with policy. Turning SELinux back on then needs a full relabel, or a pile of services fail at once:

sudo fixfiles -F onboot
sudo reboot

That writes /.autorelabel and relabels every filesystem during the next boot. On a large disk it takes a long time and the console looks stuck, so start it when you can wait. On Rocky Linux and AlmaLinux 9 the config file no longer switches off the kernel part on its own, and the documented way to disable SELinux fully is a kernel argument (sudo grubby --update-kernel ALL --args selinux=0). Knowing that command helps when you inherit somebody else's server. It is not the fix for a 403.

Containers add one more label

On a Red Hat family host, container processes run in container_t and may only read files labelled container_file_t. A bind mount from the host fails with Permission denied inside the container while ls -l on the host looks perfectly normal. The :Z suffix tells the runtime to relabel the mount:

docker run -d -v /data/appdata:/var/lib/app:Z myimage

:Z labels the directory for this container alone. :z labels it for sharing between containers. Point :Z at a directory other services use and it relabels that directory recursively, which breaks those services, so give containers their own paths. Everything else about the setup matches any other image, covered in running Docker on a VPS.

Ubuntu and Debian give you AppArmor

Same job, different design. AppArmor confines a program by the path to its executable, using a profile under /etc/apparmor.d/, instead of labelling files on disk. There is nothing to relabel and no restorecon. Start here:

sudo aa-status
sudo journalctl -k | grep -i apparmor

A refusal appears as apparmor="DENIED" operation="open" profile="/usr/sbin/nginx" name="/data/www/index.html" requested_mask="r". The workflow is the same shape: read the denial, find the profile, change the rule. sudo apt install apparmor-utils gives you aa-complain (permissive for one profile) and aa-enforce to put it back. Ubuntu confines a selected set of packaged services and leaves the rest unconfined, so read aa-status to see what is really active instead of assuming.

One habit carries across both systems. When a service reports Permission denied on something that looks correct, read the security log before you touch the permissions. The bits are rarely the problem twice.

FAQ

Why does nginx return 403 when the file permissions are correct?

Because SELinux denied the read, not the permission bits. The web server runs in the httpd_t domain and may only read files labelled for web content, so a file labelled default_t or admin_home_t is refused and nginx has nothing to serve. Confirm it with sudo ausearch -m AVC -ts recent, which shows scontext ending in httpd_t and tcontext holding the wrong type. Then record the correct label and apply it: sudo semanage fcontext -a -t httpd_sys_content_t "/data/www(/.*)?" followed by sudo restorecon -Rv /data/www.

Is it safe to run setenforce 0 to get a service working?

setenforce 0 is a diagnostic step, not a fix. Use it to reproduce the problem once so the log collects every denial in a single pass, read them with sudo ausearch -m AVC -ts recent, then run sudo setenforce 1 and repair the causes. A server left permissive logs every denial and blocks none, so you keep the noise and lose the protection. If one service needs room while you work, run sudo semanage permissive -a httpd_t so the rest of the machine stays enforcing.

How do I run a service on a non-standard port with SELinux enforcing?

Add the port to the type that service is allowed to bind. For a web server on 8081: sudo semanage port -a -t http_port_t -p tcp 8081. For SSH on 2222: sudo semanage port -a -t ssh_port_t -p tcp 2222. Check the current list first with sudo semanage port -l | grep -w http_port_t, because a port that is already listed fails with ValueError: Port tcp/8081 already defined. Without this step the daemon exits at startup with bind() ... Permission denied even though no other process holds the port.

Does Ubuntu have SELinux?

No. Ubuntu and Debian ship AppArmor, which enforces a profile tied to the path of an executable rather than labels on files. Check it with sudo aa-status and look for apparmor="DENIED" lines in sudo journalctl -k. Ubuntu confines a selected set of packaged services, so many programs run unconfined by default. Rocky Linux and AlmaLinux are where you meet SELinux enforcing out of the box, along with Fedora and RHEL.