How to Audit Commands Users Run for Your Server

What wey really dey record commands wey users run for your server

If you wan audit commands wey users run for your server, you need record wey the user no fit edit. Shell history no be that record. Na convenience file be this, and the account wey write am own am. Anybody wey fit type for that shell fit disable or delete am.

Four layers dey keep real record, and each one get im own cost. sudo dey write one line per command to syslog. sudo I/O logging dey capture complete session for one account. Shell hook like PROMPT_COMMAND dey log wetin interactive bash user type. Kernel audit subsystem dey record the execve syscall itself. Na why e be the only layer wey dey see every process. This guide go climb through these layers, explain where each one stop, and end with the part wey decide whether any of the records get value: move the records comot from the machine before the person wey you dey audit fit reach dem.

One warning before you start. Audit subsystem na kernel work, so you no fit test any of this inside container wey dey share host kernel. Run these commands for KVM VPS where na you get control of the kernel.

Why shell history no be audit trail

~/.bash_history dey fail as evidence for four normal reasons, and none of dem need attacker wey too clever.

Na user own am. The file get mode 600 and na that account own am, so rm ~/.bash_history no need any privilege at all. Opening am for editor and removing the twenty lines wey matter no need privilege too.

Shell dey write am when e exit. Session wey end with kill -9 $$, or connection wey drop, no write anything. history -c before exit get the same effect and e go look like nothing happen.

One word fit off am. unset HISTFILE stop the file from being written for that session. set +o history stop the recording immediately. HISTCONTROL=ignorespace hide every command wey person type with leading space. All this dey inside man bash, because na user suppose control am.

E record wetin person type, no be wetin run. Alias or shell function fit mean say the text for the file no be the program wey kernel execute.

Timestamps no dey there too, unless HISTTIMEFORMAT dey set when dem write the entry, because bash dey write #1755043200 marker lines only when that variable dey set.

For shared login, e still no fit tell you who do am. Three people wey use one deploy account go produce one interleaved file under one uid. No logging layer fit connect action to one human when two humans share one uid. Na this practical reason make one unprivileged account per person instead of shared login better.

Shell history good for the real work wey e suppose do: helping you type yesterday command again. Use am as clue. Never present am as proof.

Wetin sudo dey log, and where e dey stop

sudo dey send one line for every command wey e run go the authpriv syslog facility.

sudo grep 'sudo:' /var/log/auth.log | tail -5
journalctl -t sudo -n 5

Each line dey name the user, the terminal, the working directory, the target user, and the command:

sudo:    alice : TTY=pts/0 ; PWD=/home/alice ; USER=root ; COMMAND=/usr/bin/apt update

If /var/log/auth.log no dey exist, rsyslog no dey installed for that image, and na journal only get the same records. Check say journal no be volatile before you rely on am:

journalctl --list-boots

If na only the current boot dey listed, /var/log/journal no dey exist. This mean say journal dey inside /run, and every line go disappear when system reboot. Make am persistent:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Now the limit. sudo dey log the command wey e receive to run. E no dey log wetin that command do afterwards. So one line fit end the trail:

sudo -i

The log go get only one record for the shell. Every command wey person type inside that root shell no dey visible to sudo, because sudo no dey inside the execution path again. sudo su -, sudo bash, and sudo vim /etc/shadow followed by :!bash all get the same pattern. A sudoers rule wey permits any program with shell escape, like vim or find, na rule wey grants unlogged root access. Read wetin the account fit actually reach before you trust the lines for its log:

sudo -l -U alice

Record one account full session

First, find out which sudo you get, because this feature no dey for the Rust rewrite:

sudo --version | head -1

If the output name sudo-rs, skip this section and use the audit subsystem. Ubuntu documentation for the 25.10 and 26.04 releases say say I/O logging and sudoreplay no dey supported, and e still be so as of August 2026. This one matter because sudo-rs na the default sudo for those releases. So, upgrade fit remove control wey you think say you get. Read the full list of sudo-rs behaviour changes before you plan any sudo-based logging.

With the original sudo, wey Ubuntu 24.04 LTS still dey ship, turn on I/O logging for one account:

sudo visudo -f /etc/sudoers.d/iolog

Defaults:deploy log_output
Defaults!/usr/bin/sudoreplay !log_output

Use visudo instead of editor, because e no go save file wey no parse correctly. If sudoers file spoil, nobody fit use sudo again. Then replay one session:

sudo sudoreplay -l user=deploy
sudo sudoreplay 000001

sudoreplay -l go list the sessions with their IDs, and e no go print anything if log_output never apply to that user. The cost be say every byte wey pass through the terminal dey stored under /var/log/sudo-io, so verbose session fit big. The second sudoers line stop replay from recording itself. The main cost na secrets, because I/O log keep anything wey person type and anything wey terminal print. This include password wey person type into prompt inside the session. So, protect am the same way you protect password store. Coverage still narrow. E only see commands wey person run through sudo. If person log in and work entirely as themselves, the recording no go capture anything.

Shell hooks, and exactly how dem dey bypass am

The recipe wey dey circulate for “log every command” na a PROMPT_COMMAND hook wey dem drop inside /etc/profile.d/:

# /etc/profile.d/00-cmdlog.sh
PROMPT_COMMAND='logger -p local6.info -t cmdlog "$(whoami) $$ $(history 1 | sed "s/^ *[0-9]* *//")"'

bash dey run PROMPT_COMMAND before e draw each prompt, so line dey reach syslog as user type am instead of when e exit. logger dey write through system log daemon, so user’s own file permissions no enter the matter. Open new login shell and check with sudo tail -f /var/log/syslog, or use journalctl -t cmdlog -f for image wey no get rsyslog.

Then e stop to work in five ways wey you fit reproduce each within one minute.

• Non-interactive shells no dey draw prompt. ssh you@server 'id' run the command and return, and nothing dey logged because PROMPT_COMMAND never evaluate.
• Na variable. unset PROMPT_COMMAND disable am for the rest of the session, and e no need privilege.
• Login shells dey read the file. bash --noprofile --norc no source /etc/profile.d/ at all.
• Na bash-specific feature. zsh, sh, python3 -c 'import os; os.system("id")' and :!id inside vim all run programs wey no bash prompt hook go ever see.
• E dey log the line as user type am, so alias or function still hide the command wey really run.

Use shell hook as convenience. E fit answer “wetin I run last Tuesday” for users wey cooperate. No let checklist call am control.

Kernel audit subsystem dey see every execve

Linux audit subsystem, wey auditd daemon dey drive, na the only layer for here wey user no fit bypass. Na because kernel dey create the record when syscall dey run. If process execute program, event go dey. Shell, language, and whether terminal dey no change anything.

sudo apt update && sudo apt install -y auditd audispd-plugins
sudo systemctl enable --now auditd
sudo auditctl -s

auditctl -s dey print daemon state. enabled 1 with non-zero pid mean say e dey run, and lost 0 mean say no record don drop yet. Remember that lost counter; e go show again later.

auid na the field wey make audit worth the work. PAM dey set login uid when session start, and kernel dey carry am for every child process from that point. Check your own:

cat /proc/self/loginuid

Interactive SSH session go print your uid, because /etc/pam.d/sshd include pam_loginuid.so. Value of 4294967295 mean say loginuid never set, and this normal for process wey system daemon start during boot. The important thing be say sudo -i no dey change am: root shell wey alice open still carry auid 1000, so every command inside am still traceable to alice. Na exactly this gap sudo dey leave open. To change loginuid after dem don set am need CAP_AUDIT_CONTROL, wey ordinary users no get, and sudo auditctl --loginuid-immutable close am for root too until next reboot.

Check say /etc/pam.d/sshd, /etc/pam.d/login and /etc/pam.d/cron each include pam_loginuid.so, otherwise events go arrive without anybody attached to dem. Na the same file list wey you touch when you dey harden SSH access for VPS, so do both jobs together.

Rule set wey you fit start with for auditd

Rules dey inside /etc/audit/rules.d/*.rules. augenrules dey join dem together according to filename order into one list. Order dey decide how dem go work, because kernel dey stop for the first rule wey match. Read wetin dey there already before you add anything, because -D for later file go wipe everything wey load before am.

ls /etc/audit/rules.d/
cat /etc/audit/rules.d/audit.rules

Then write /etc/audit/rules.d/50-exec.rules:

## Suppressions first: the kernel takes the first matching rule.
-a never,exit -F arch=b64 -S execve -F exe=/usr/bin/dpkg
-a never,exit -F arch=b64 -S execve -F exe=/usr/bin/dpkg-deb
-a never,exit -F arch=b64 -S execve -F exe=/usr/bin/dpkg-query
-a never,exit -F arch=b64 -S execve -F exe=/usr/bin/dpkg-split
-a never,exit -F arch=b64 -S execve -F exe=/usr/bin/dpkg-trigger

## Every program started by a logged-in human.
-a always,exit -F arch=b64 -S execve,execveat -F auid>=1000 -F auid!=unset -k exec
-a always,exit -F arch=b32 -S execve,execveat -F auid>=1000 -F auid!=unset -k exec

## Changes to who may become root.
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/group -p wa -k identity

## Changes to the audit configuration itself.
-w /etc/audit/ -p wa -k auditconfig

Load am and confirm:

sudo augenrules --load
sudo auditctl -l

When auditctl -l print your rules back, e mean say dem dey active. No rules mean say the load fail, and journalctl -u auditd -n 20 show the file and line wey parser reject. Older audit userspace no understand the unset keyword. If loader complain about that field, write -F auid!=4294967295 instead. Na the same value, but e spell out fully.

Now read the events back:

sudo ausearch -k exec -ts recent -i | tail -40
sudo ausearch -ul 1000 -ts today -i
sudo aureport -k --summary -i

-i dey change uids and syscall numbers to names, and for practical use, you need am. -ts recent cover the last ten minutes. Each execution dey come as one group of records: a SYSCALL record wey carry uid, auid, exit status, and key; a EXECVE record wey get the complete argument list; plus CWD and PATH records for context.

One honest limit dey here, because e dey catch people unaware. audit dey record syscalls, but shell builtin no make syscall by itself. cd /root no dey run any program. echo evil >> /etc/passwd wey you type for bash prompt no dey run any program too, because both the echo and the redirect happen inside the shell process wey already dey run. So the execve rules dey see programs, while the -w rules dey see the writes. You need both; one alone no dey enough.

Finally, lock the configuration:

## /etc/audit/rules.d/99-finalize.rules
-e 2

-e 2 dey make the rule set immutable until the next reboot. After e load, auditctl -s go report enabled 2, and any attempt to add or delete rule go fail with Operation not permitted, root included. Add this file last, and expect to reboot every time you want change rule. Na that trade be the point: rule set wey anybody fit quietly switch off no be evidence.

Audit log wey nobody dey read na compliance artefact

Auditd problem no be say e dey miss things. Problem na say e dey record plenty things reach where nobody go ever check am. Then the log dey exist just to satisfy checklist instead of answering question.

Do the calculation for your own box before you tune anything:

sudo aureport -k --summary -i
sudo du -sh /var/log/audit

One sudo apt upgrade fit run thousands of short-lived processes, and every one of dem carry your auid. So, one package update fit pass one week of human typing. Na why the suppressions above name dpkg and its helpers. Suppress by executable, never by user. Exclusion for /usr/bin/dpkg na hole wey you fit describe with one sentence. But exclusion for account na hole wey get exactly the shape of the thing wey you dey try catch.

The -k key for each rule na wetin make the log searchable one month later. ausearch -k sudoers na question wey get answer. ausearch without filter na wall of text wey dey train you to stop reading. If your collector need JSON instead of the native format, laurel na auditd plugin wey rewrite each event as one JSON object with the arguments decoded. E register for /etc/audit/plugins.d/ like any other plugin, and auditd go pick plugin changes for sudo pkill -HUP auditd.

Wetin auditd dey cost, honestly

Every syscall wey match rule go become record wey kernel format and hand over to userspace. The cost dey show for two places, and you fit measure both for your own workload instead of guessing from another person published number.

• CPU and latency. Machine wey dey fork plenty, like build host or CI runner, go produce one record for every exec. When kernel backlog full, --backlog_wait_time go make kernel pause the process wey generate the event until space dey. So audit fit show as slow builds instead of CPU percentage. Watch backlog and lost for sudo auditctl -s when real load dey run. If lost dey rise, e mean say records don drop. Log wey get silent gaps worse pass no log, because you go still trust am.
• Disk. Read /etc/audit/auditd.conf and decide deliberately wetin suppose happen when disk full, because the values wey ship with am na just opinions. max_log_file, num_logs and max_log_file_action control rotation. space_left_action, admin_space_left_action and disk_full_action control emergency behaviour. Some of the available actions, including halt and single, go shut down the machine instead of losing one record.

The -f line for /etc/audit/rules.d/audit.rules na that same decision for kernel level: -f 1 reports audit failure to syslog, while -f 2 panics the kernel. Choose 2 only if you truly prefer to lose the server instead of losing one record. For VPS wey dey run something people depend on, rotate instead, and move the storage problem comot from the box.

Send the logs comot for the machine, almost in real time

Na this part incident write-ups dey prove again and again. Logs wey remain for the breached host fit be edited by whoever breach am. Root fit rewrite /var/log/auth.log, delete /var/log/audit/audit.log, and stop the daemon. -e 2 dey prevent dem from unloading the rules. E no do anything about rm. Every layer above go only produce evidence if copy first comot from the machine.

Audit own transport na the audisp-remote plugin from audispd-plugins. Turn am on for /etc/audit/plugins.d/au-remote.conf:

active = yes
direction = out
path = /usr/sbin/audisp-remote
type = always
format = string

Check path against command -v audisp-remote before you reload, because wrong path go produce nothing apart from one line for the journal. Set remote_server and port for /etc/audit/audisp-remote.conf, and for the collector set tcp_listen_port = 60 inside its own auditd.conf. Reload with sudo pkill -HUP auditd. For plenty images, systemctl restart auditd dey refused because the unit file set RefuseManualStop=yes, so the signal na the reliable route.

The other option puts audit inside the syslog stream wey you already dey forward. /etc/audit/plugins.d/syslog.conf dey ship with active = no. Set am to yes, reload, and audit events go join sudo lines and everything else. Then forward everything with rsyslog over TLS (transport layer security), wey need the rsyslog-gnutls package:

# /etc/rsyslog.d/60-forward.conf
*.* action(type="omfwd"
    target="logs.example.net" port="6514" protocol="tcp"
    StreamDriver="gtls" StreamDriverMode="1"
    StreamDriverAuthMode="x509/name"
    StreamDriverPermittedPeers="logs.example.net"
    action.resumeRetryCount="-1"
    queue.type="linkedList" queue.filename="fwd" queue.saveOnShutdown="on")

The queue settings na the important part. action.resumeRetryCount="-1" retries forever, and the disk-assisted queue with queue.saveOnShutdown="on" dey hold records while collector no dey reachable, then send dem when e come back. Without those two, collector reboot go leave gap for your evidence, and nothing go tell you say the gap dey there. Apply with sudo systemctl restart rsyslog, then confirm say the records really dey arrive for the collector before you trust anything.

One loop remain to close: the collector must be a machine wey the people under audit no fit log into. If the same admin group get root for the log server, you don copy the file, but you never protect am. Use separate credentials, separate keys, and ideally separate provider account. Na the same reasoning make one central way to manage many Linux servers worth building before you need am, and na this one make the first hour useful or useless when you dey work through a breached VPS.

Check say normal user no fit rewrite the record

Test the claim instead of just assuming am. From ordinary account, without sudo:

echo test >> /var/log/auth.log
cat /var/log/audit/audit.log
auditctl -D
id -nG

Expect these results in this order: Permission denied, because auth.log get ownership by syslog, group adm, and mode 640; Permission denied again, because audit log get mode 600 and root own am; error wey refuse to run, because changing audit rules need CAP_AUDIT_CONTROL; and group list wey no contain adm or systemd-journal.

Na this last check people dey fail. Membership of adm gives read access to /var/log/auth.log, while membership of systemd-journal gives read access to the whole journal. Neither one gives write access, so neither one allows tampering. Both ones let person read every authentication line for the box. Na decision wey you suppose make intentionally, instead of copying usermod -aG line from forum answer.

Finally, confirm the two things wey must remain after reboot:

sudo auditctl -s
systemctl is-enabled auditd

enabled 2 means the rule set lock until the next boot. enabled from the second command means auditd starts again after that boot. Rule set wey only last until the next kernel update no be audit trail either.

FAQ

How I fit see every command wey one specific user run?

Find the user uid with id -u alice, then search the audit log with login uid: sudo ausearch -ul 1000 -ts today -i. Add -k exec to limit am to the execve rule. Dem set the login uid when user log in, and e dey remain through su and sudo -i. So this one catch commands wey dem run inside root shell wey that account open. E only work for commands wey dem execute after dem load the rules, because audit no keep history for events wey e no configure to record. sudo aureport -k --summary -i go show you the count for each rule if you first want see the data pattern.

User fit delete their bash history to hide wetin dem run?

Yes, and dem no need privilege to do am. ~/.bash_history na that user own, with mode 600, so dem fit edit am, truncate am, or remove am. Dem fit also stop the file from writing with unset HISTFILE, stop recording for the middle of session with set +o history, or hide single commands by typing leading space before dem when HISTCONTROL=ignorespace dey set. Bash dey write the file when shell dey exit, so session wey dem kill with kill -9 $$ no record anything. Treat shell history as hint, never as evidence.

sudo dey log wetin happen inside sudo -i?

No. sudo dey log the command wey dem ask am to run, so sudo -i go produce one line for the shell and nothing after that. Every command wey person type inside that root shell no dey visible to sudo, because sudo no dey involved again. sudo su -, sudo bash, and any permitted program wey get shell escape dey behave the same way. Two things fit close this gap: audit rules on execve, wey record every program with the original login uid attached, and sudoers rules wey no hand out shell from the beginning.

auditd go slow down my server?

E depend completely on how many processes your workload dey start, so measure am instead of trusting any figure. Server wey mostly dey answer requests no dey exec much, and e no go notice. Build host or CI runner dey exec constantly, and e fit notice am well, because when kernel audit backlog full, the process wey generate the event go pause until space dey. Run sudo auditctl -s under real load and monitor backlog and lost. Any lost wey pass zero mean say records don drop. Na the worst result be that one, because the log now get gaps wey nobody fit see.

Where audit logs suppose dey stored?

For another machine, with delay wey dem measure in seconds. Anybody wey reach root for the audited host fit delete /var/log/audit/audit.log and rewrite /var/log/auth.log. So local copies fit answer questions only about incidents wey nobody try hide. Forward am with the audisp-remote plugin to central auditd, or enable the audit syslog plugin and forward the complete syslog stream with rsyslog over TLS. Give the collector im own credentials, and make sure the accounts wey you dey audit no get access to am.