Audit what commands users ran on your server

What actually records the commands users ran on your server

To audit what commands users ran on your server, you need a record the user cannot edit. Shell history is not that record. It is a convenience file, owned by the account that wrote it, and anyone who can type into that shell can switch it off or delete it.

Four layers do keep a real record, and each one costs something. sudo writes a line per command to syslog. sudo I/O logging captures a whole session for one account. A shell hook such as PROMPT_COMMAND logs what an interactive bash user typed. The kernel audit subsystem records the execve syscall itself, which is why it is the only layer that sees every process. This guide climbs that ladder, says where each layer stops, and ends with the part that decides whether any of it is worth anything: getting the records off the machine before the person you are auditing can reach them.

One warning before you start. The audit subsystem is kernel work, so none of it can be tested inside a container that shares a host kernel. Run these commands on a KVM VPS where the kernel is yours.

Why shell history is not an audit trail

~/.bash_history fails as evidence for four ordinary reasons, and none of them need an attacker who is clever.

It belongs to the user. The file is mode 600 and owned by that account, so rm ~/.bash_history needs no privilege at all. Neither does opening it in an editor and removing the twenty lines that matter.

It is written when the shell exits. A session that ends with kill -9 $$, or with a dropped connection, writes nothing. history -c before exit has the same effect and looks like nothing happened.

It is off with one word. unset HISTFILE stops the file being written for that session. set +o history stops the recording immediately. HISTCONTROL=ignorespace hides every command typed with a leading space. All of this is in man bash, because it is meant to be under the user's control.

It records what was typed, not what ran. An alias or a shell function means the text in the file is not the program the kernel executed.

There are no timestamps either, unless HISTTIMEFORMAT was set while the entry was written, because bash writes its #1755043200 marker lines only when that variable is set.

On a shared login it also cannot tell you who. Three people using one deploy account produce one interleaved file under one uid. No logging layer can attribute an action to a human when two humans share a uid, which is the practical argument for one unprivileged account per person instead of a shared login.

Shell history is good at its real job, which is helping you retype yesterday's command. Use it as a hint. Never present it as proof.

What sudo logs, and where it stops

sudo sends a line for every command it runs to the authpriv syslog facility.

sudo grep 'sudo:' /var/log/auth.log | tail -5
journalctl -t sudo -n 5

Each line names the user, the terminal, the working directory, the target user and the command:

sudo:    alice : TTY=pts/0 ; PWD=/home/alice ; USER=root ; COMMAND=/usr/bin/apt update

If /var/log/auth.log does not exist, rsyslog is not installed on that image and the same records are in the journal only. Check that the journal is not volatile before you rely on it:

journalctl --list-boots

Only the current boot listed means /var/log/journal does not exist, so the journal lives in /run and every line dies at the next reboot. Make it persistent:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Now the limit. sudo logs the command it was asked to run. It does not log what that command then does. So one line ends the trail:

sudo -i

The log gets a single record for the shell. Every command typed inside that root shell is invisible to sudo, because sudo is no longer in the path. sudo su -, sudo bash, and sudo vim /etc/shadow followed by :!bash all have the same shape. A sudoers rule that permits any program with a shell escape, such as vim or find, is a rule that grants unlogged root. Read what an account can actually reach before you trust its log lines:

sudo -l -U alice

Recording a full session for one account

First find out which sudo you have, because this feature does not exist in the Rust rewrite:

sudo --version | head -1

If the output names sudo-rs, skip this section and use the audit subsystem. Ubuntu's own documentation for the 25.10 and 26.04 releases lists I/O logging and sudoreplay as not supported, and that was still true as of August 2026. It matters because sudo-rs is the default sudo on those releases, so an upgrade can remove a control you thought you had. The full list of sudo-rs behaviour changes is worth reading before you plan any sudo-based logging.

With the original sudo, which Ubuntu 24.04 LTS still ships, turn on I/O logging for one account:

sudo visudo -f /etc/sudoers.d/iolog

Defaults:deploy log_output
Defaults!/usr/bin/sudoreplay !log_output

Use visudo rather than an editor, because it refuses to save a file that does not parse. A broken sudoers file locks everyone out of sudo. Then replay a session:

sudo sudoreplay -l user=deploy
sudo sudoreplay 000001

sudoreplay -l lists the sessions with their IDs and prints nothing if log_output never applied to that user. What it costs: every byte that crosses the terminal is stored under /var/log/sudo-io, so a verbose session is large. The second sudoers line stops a replay from recording itself. The real cost is secrets, because an I/O log holds whatever was typed and printed, including a password typed into a prompt inside the session, so it needs the same protection as a password store. Coverage is narrow too. It sees commands run through sudo. Someone who logs in and works entirely as themselves is not recorded at all.

Shell hooks, and exactly how they are bypassed

The recipe that circulates for "log every command" is a PROMPT_COMMAND hook dropped into /etc/profile.d/:

# /etc/profile.d/00-cmdlog.sh
PROMPT_COMMAND='logger -p local6.info -t cmdlog "$(whoami) $$ $(history 1 | sed "s/^ *[0-9]* *//")"'

bash runs PROMPT_COMMAND before drawing each prompt, so a line reaches syslog as it is typed rather than at exit, and logger writes through the system log daemon, so the user's own file permissions never come into it. Open a new login shell and check with sudo tail -f /var/log/syslog, or journalctl -t cmdlog -f on an image with no rsyslog.

Then it stops working, in five ways you can each reproduce in a minute.

• Non-interactive shells never draw a prompt. ssh you@server 'id' runs the command and returns, and nothing is logged, because PROMPT_COMMAND was never evaluated.
• It is a variable. unset PROMPT_COMMAND disables it for the rest of the session and needs no privilege.
• The file is read by login shells. bash --noprofile --norc never sources /etc/profile.d/ at all.
• It is bash-specific. zsh, sh, python3 -c 'import os; os.system("id")' and :!id inside vim all run programs that no bash prompt hook will ever see.
• It logs the line as typed, so an alias or a function still hides the command that really ran.

Use a shell hook as a convenience. It answers "what did I run last Tuesday" for cooperative users. Do not let a checklist call it a control.

The kernel audit subsystem sees every execve

The Linux audit subsystem, driven by the auditd daemon, is the only layer here that a user cannot step around, because the record is made inside the kernel at the moment the syscall runs. If a process executes a program, there is an event. The shell, the language and the presence of a terminal make no difference.

sudo apt update && sudo apt install -y auditd audispd-plugins
sudo systemctl enable --now auditd
sudo auditctl -s

auditctl -s prints the daemon state. enabled 1 with a non-zero pid means it is running, and lost 0 means no records have been dropped yet. Remember that lost counter, it comes back later.

auid is the field that makes audit worth the trouble. PAM sets a login uid when a session starts, and the kernel carries it on every child process from then on. Check yours:

cat /proc/self/loginuid

An interactive SSH session prints your uid, because /etc/pam.d/sshd includes pam_loginuid.so. A value of 4294967295 means the loginuid was never set, which is normal for a process started by a system daemon at boot. The important part is that sudo -i does not change it: a root shell opened by alice still carries auid 1000, so every command inside it is attributable to alice. That is exactly the gap sudo leaves open. Changing a loginuid once set needs CAP_AUDIT_CONTROL, which ordinary users do not have, and sudo auditctl --loginuid-immutable closes it for root as well until the next reboot.

Check that /etc/pam.d/sshd, /etc/pam.d/login and /etc/pam.d/cron each include pam_loginuid.so, or events will arrive with nobody attached to them. That is the same file list you touch when hardening SSH access on a VPS, so do the two jobs together.

A starting rule set for auditd

Rules live in /etc/audit/rules.d/*.rules. augenrules concatenates them in filename order into one list, and order decides behaviour, because the kernel stops at the first matching rule. Read what is already there before you add anything, since a -D in a later file wipes everything loaded before it.

ls /etc/audit/rules.d/
cat /etc/audit/rules.d/audit.rules

Then write /etc/audit/rules.d/50-exec.rules:

## Suppressions first: the kernel takes the first matching rule.
-a never,exit -F arch=b64 -S execve -F exe=/usr/bin/dpkg
-a never,exit -F arch=b64 -S execve -F exe=/usr/bin/dpkg-deb
-a never,exit -F arch=b64 -S execve -F exe=/usr/bin/dpkg-query
-a never,exit -F arch=b64 -S execve -F exe=/usr/bin/dpkg-split
-a never,exit -F arch=b64 -S execve -F exe=/usr/bin/dpkg-trigger

## Every program started by a logged-in human.
-a always,exit -F arch=b64 -S execve,execveat -F auid>=1000 -F auid!=unset -k exec
-a always,exit -F arch=b32 -S execve,execveat -F auid>=1000 -F auid!=unset -k exec

## Changes to who may become root.
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/group -p wa -k identity

## Changes to the audit configuration itself.
-w /etc/audit/ -p wa -k auditconfig

Load it and confirm:

sudo augenrules --load
sudo auditctl -l

auditctl -l printing your rules back means they are live. No rules means the load failed, and journalctl -u auditd -n 20 names the file and the line the parser rejected. Older audit userspace does not understand the unset keyword. If the loader complains about that field, write -F auid!=4294967295 instead, which is the same value spelled out.

Now read events back:

sudo ausearch -k exec -ts recent -i | tail -40
sudo ausearch -ul 1000 -ts today -i
sudo aureport -k --summary -i

-i turns uids and syscall numbers into names, and it is not optional in practice. -ts recent covers the last ten minutes. Each execution arrives as a group of records: a SYSCALL record carrying uid, auid, exit status and key, an EXECVE record with the full argument list, plus CWD and PATH records for context.

One honest limit, because it catches people out. audit records syscalls, and a shell builtin makes no syscall of its own. cd /root runs no program. echo evil >> /etc/passwd typed at a bash prompt runs no program either, because both the echo and the redirect happen inside the shell process that is already running. So the execve rules see programs and the -w rules see the writes. Neither one is enough alone.

Finally, lock the configuration:

## /etc/audit/rules.d/99-finalize.rules
-e 2

-e 2 makes the rule set immutable until the next reboot. After it loads, auditctl -s reports enabled 2, and any attempt to add or delete a rule fails with Operation not permitted, root included. Add this file last, and expect a reboot every time you want to change a rule. That trade is the point: a rule set anyone can quietly switch off is not evidence.

An audit log nobody reads is a compliance artefact

The failure mode of auditd is not that it misses things. It is that it records so much that nobody ever looks, and then the log exists to satisfy a checklist instead of answering a question.

Do the arithmetic on your own box before you tune anything:

sudo aureport -k --summary -i
sudo du -sh /var/log/audit

A single sudo apt upgrade runs thousands of short-lived processes, and every one of them carries your auid, so one package update can outweigh a week of human typing. That is why the suppressions above name dpkg and its helpers. Suppress by executable, never by user: an exclusion for /usr/bin/dpkg is a hole you can describe in one sentence, while an exclusion for an account is a hole shaped exactly like the thing you were trying to catch.

The -k key on each rule is what makes the log searchable a month later. ausearch -k sudoers is a question with an answer. ausearch with no filter is a wall of text that trains you to stop reading. If your collector wants JSON instead of the native format, laurel is an auditd plugin that rewrites each event as one JSON object with the arguments decoded. It registers in /etc/audit/plugins.d/ like any other plugin, and auditd picks up plugin changes on sudo pkill -HUP auditd.

What auditd costs, honestly

Every matching syscall becomes a record that the kernel formats and hands to userspace. The cost lands in two places, and both are measurable on your own workload rather than guessable from someone else's published number.

• CPU and latency. A machine that forks constantly, a build host or a CI runner, produces a record per exec. When the kernel backlog fills, --backlog_wait_time makes the kernel pause the process that generated the event until there is room, so audit shows up as slow builds rather than as a CPU percentage. Watch backlog and lost in sudo auditctl -s under real load. A rising lost means records were dropped, and a log with silent gaps is worse than no log, because you will still trust it.
• Disk. Read /etc/audit/auditd.conf and decide deliberately what happens when the disk fills, because the shipped values are opinions. max_log_file, num_logs and max_log_file_action control rotation. space_left_action, admin_space_left_action and disk_full_action control the emergency, and some of the available actions, halt and single among them, take the machine down rather than lose a record.

The -f line in /etc/audit/rules.d/audit.rules is that same decision at kernel level: -f 1 reports an audit failure to syslog, and -f 2 panics the kernel. Choose 2 only if you would genuinely rather lose the server than lose a record. On a VPS running something people depend on, rotate instead, and move the storage problem off the box.

Ship the logs off the box, in near real time

This is the part the incident write-ups keep proving. Logs that stay on the compromised host can be edited by whoever compromised it. Root can rewrite /var/log/auth.log, delete /var/log/audit/audit.log, and stop the daemon. -e 2 prevents the rules being unloaded. It does nothing about rm. Every layer above only produces evidence if a copy leaves the machine first.

Audit's own transport is the audisp-remote plugin from audispd-plugins. Turn it on in /etc/audit/plugins.d/au-remote.conf:

active = yes
direction = out
path = /usr/sbin/audisp-remote
type = always
format = string

Check path against command -v audisp-remote before you reload, because a wrong path produces nothing except one line in the journal. Set remote_server and port in /etc/audit/audisp-remote.conf, and on the collector set tcp_listen_port = 60 in its own auditd.conf. Reload with sudo pkill -HUP auditd. On many images systemctl restart auditd is refused, because the unit file sets RefuseManualStop=yes, so the signal is the reliable route.

The other option puts audit into the syslog stream you already forward. /etc/audit/plugins.d/syslog.conf ships with active = no. Set it to yes, reload, and audit events join sudo's lines and everything else. Then forward the lot with rsyslog over TLS (transport layer security), which needs the rsyslog-gnutls package:

# /etc/rsyslog.d/60-forward.conf
*.* action(type="omfwd"
    target="logs.example.net" port="6514" protocol="tcp"
    StreamDriver="gtls" StreamDriverMode="1"
    StreamDriverAuthMode="x509/name"
    StreamDriverPermittedPeers="logs.example.net"
    action.resumeRetryCount="-1"
    queue.type="linkedList" queue.filename="fwd" queue.saveOnShutdown="on")

The queue settings are the interesting part. action.resumeRetryCount="-1" retries forever, and the disk-assisted queue with queue.saveOnShutdown="on" holds records while the collector is unreachable, then sends them when it comes back. Without those two, a collector reboot leaves a hole in your evidence and nothing to tell you the hole is there. Apply with sudo systemctl restart rsyslog, then confirm the records actually arrive on the collector before you trust any of it.

One loop left to close: the collector must be a machine the audited people cannot log into. If the same admin group holds root on the log server, you have copied the file, not protected it. Separate credentials, separate keys, and ideally a separate provider account. This is the same reasoning that makes a central way to manage many Linux servers worth building before you need it, and it is the difference between a useful and a useless first hour when you are working through a compromised VPS.

Check that a normal user cannot rewrite the record

Test the claim instead of assuming it. From an ordinary account, with no sudo:

echo test >> /var/log/auth.log
cat /var/log/audit/audit.log
auditctl -D
id -nG

Expect, in order: Permission denied, because auth.log is owned by syslog with group adm and mode 640; Permission denied again, because the audit log is mode 600 and owned by root; an error refusing to run, because changing audit rules needs CAP_AUDIT_CONTROL; and a group list holding neither adm nor systemd-journal.

That last check is the one people fail. Membership of adm grants read access to /var/log/auth.log, and membership of systemd-journal grants read access to the whole journal. Neither grants write access, so neither allows tampering. Both let a person read every authentication line on the box, which is a decision to make on purpose rather than by copying a usermod -aG line out of a forum answer.

Finally, confirm the two things that must survive a reboot:

sudo auditctl -s
systemctl is-enabled auditd

enabled 2 means the rule set is locked until the next boot. enabled from the second command means auditd starts again after that boot. A rule set that lasts only until the next kernel update is not an audit trail either.

FAQ

How do I see every command a specific user ran?

Find their uid with id -u alice, then search the audit log by login uid: sudo ausearch -ul 1000 -ts today -i. Add -k exec to limit it to the execve rule. The login uid is set at login and survives su and sudo -i, so this catches commands run inside a root shell that account opened. It only works for commands executed after the rules were loaded, because audit keeps no history of events it was not configured to record. sudo aureport -k --summary -i gives you the per-rule counts if you want to see the shape of the data first.

Can a user delete their bash history to hide what they ran?

Yes, and it needs no privilege. ~/.bash_history is owned by that user with mode 600, so they can edit it, truncate it, or remove it. They can also stop it being written with unset HISTFILE, stop recording mid-session with set +o history, or hide single commands by typing them with a leading space when HISTCONTROL=ignorespace is set. Bash writes the file when the shell exits, so a session killed with kill -9 $$ records nothing. Treat shell history as a hint, never as evidence.

Does sudo log what happens inside sudo -i?

No. sudo logs the command it was asked to run, so sudo -i produces one line for the shell and nothing after it. Every command typed in that root shell is invisible to sudo, because sudo is no longer involved. sudo su -, sudo bash, and any permitted program with a shell escape behave the same way. Two things close the gap: audit rules on execve, which record every program with the original login uid attached, and sudoers rules that do not hand out a shell in the first place.

Will auditd slow my server down?

It depends entirely on how many processes your workload starts, so measure it rather than trusting a figure. A server that mostly answers requests execs very little and will not notice. A build host or CI runner execs constantly and can notice a lot, because when the kernel audit backlog fills, the process that generated the event is paused until there is room. Run sudo auditctl -s under real load and watch backlog and lost. Any lost above zero means records were dropped, which is the worst outcome, since the log now has invisible gaps.

Where should the audit logs be stored?

On another machine, with a delay measured in seconds. Anyone who reaches root on the audited host can delete /var/log/audit/audit.log and rewrite /var/log/auth.log, so local copies answer questions only about incidents nobody tried to hide. Forward with the audisp-remote plugin to a central auditd, or enable the audit syslog plugin and forward the whole syslog stream with rsyslog over TLS. Give the collector its own credentials, and make sure the accounts being audited have no access to it.