Linux server maintenance checklist wey dey work

Wetín Linux server maintenance really mean

Linux server maintenance na short list of checks wey you run for fixed schedule, no be project wey get end date. Every week, confirm say updates install, disk still get space, no service don stop, and backup job finish. Every month, test restore, check when certificate go expire, audit accounts and cryptographic keys, then remove old kernels and logs. Once for each distribution release, plan version upgrade and do the reboot wey you don dey postpone.

Building the server na different work, and the first ten minutes for new VPS cover that part. This page na about the year after that. Every item below name the failure wey e dey prevent, because checklist wey no explain consequence na the kind wey people quietly stop to run.

The commands here na examples, and you suppose read dem before you run dem. Compare wetin dem output with your own server, because healthy free-space or process-count value depend on wetin the server dey do. Where a check different between distributions, the text talk am. Examples use Debian and Ubuntu with apt. For the RHEL family, the tooling na dnf, and some paths different.

How to choose Linux server maintenance schedule wey you go maintain

Weekly checks cover things wey dey change without you: packages, disk usage, service state, scheduled jobs. Dem dey move by demself, so one week na about the longest time you fit leave dem unchecked.

Monthly checks cover slow decay: certificates wey expiry dey near, accounts wey nobody don remove, kernels wey dey pile up for /boot, log files wey don grow pass rotation rule wey stop matching. None of these go break tomorrow. But all of dem go eventually cause problem.

Release checks dey follow calendar. Distribution release na the only maintenance item wey get external deadline, because support for your current version go end whether you ready or not.

Set the run for one fixed time: Monday morning for the weekly pass, and the first day of the month for the monthly one. Checklist wey you run "when I get to it" no be checklist. Once you get more than a few machines, run these checks from one place instead of doing dem by hand. This na the topic for managing multiple Linux servers from one place.

Weekly: updates really install?

Enabling unattended-upgrades no be the same as knowing say e run. The service fit dey masked, the configuration fit limit am to an origin wey you no dey use, and one held package fit spoil every run after that. We don cover installation for automatic security updates for Ubuntu. The weekly job na to prove say wetin you install really do the work.

systemctl status unattended-upgrades
journalctl -u unattended-upgrades --since "8 days ago" --no-pager
sudo tail -n 50 /var/log/unattended-upgrades/unattended-upgrades.log
apt list --upgradable
apt-mark showhold

apt list --upgradable na the honest measure, because e dey report the current state instead of wetin dem intend. If security updates still dey that list, the automation no dey do the work, so read the log before you assume say the machine don get patch. Package wey you pin with apt-mark hold go skip forever and e no go report anything. Na why apt-mark showhold belong for the same check.

The failure wey this one dey prevent: running a known-vulnerable package for months while you believe say updates dey happen automatically.

Weekly: check free disk space and inodes

Root filesystem full fit break things wey no look related to disk. Database go refuse writes, logging go stop, package upgrade fit fail halfway, and for some setups you no fit open new session because e no fit write im own files.

df -h
df -i
sudo du -xh --max-depth=1 / | sort -h | tail -20

df -i na the half wey most people dey skip. Inodes na structures with fixed count wey dey hold file metadata, and filesystem fit run out of dem while df -h still dey show free gigabytes. Writes go then fail with No space left on device beside output wey show say space still dey, and this fit waste one confusing hour the first time. Millions of small files, from mail queue wey get stuck or session directory wey nobody dey prune, na the usual cause.

du -xh dey remain for one filesystem, and na wetin you want for machine wey get bind mounts or attached storage. For Docker host, the answer usually dey inside image layers and dead volumes. Clear dem as e dey described for pruning Docker disk usage for VPS.

Free space dey tell you about capacity. The storage underneath fit fail according to im own schedule, so na separate check wey disk health monitoring for VPS cover.

Weekly: wetin stop without telling you?

systemctl list-units --state=failed
systemctl list-timers --all
journalctl -p err --since "8 days ago" --no-pager

Unit wey crash and reach im restart limit go remain for failed state, silent. Nobody go email you about am. list-timers na the more useful half: e show when each timer last run and when e go fire next, so if LAST value older than the timer own interval, that job no run at all.

Read the journal for the unit before you restart am, with journalctl -u <unit> -n 100 --no-pager. Restart go clear the symptom, then you no get reason to check again until the same thing happen for a worse time.

The failure wey this one prevent: monitoring agent, queue worker, or backup service wey don dey down since memory spike happen three weeks ago.

Weekly: backup job really finish?

Scheduled backup and completed backup no be the same thing, and na only completed backup fit restore. Check whether e finish.

systemctl list-timers --all | grep -i backup
journalctl -u <your-backup-unit> --since "8 days ago" --no-pager
ls -lh /path/to/backup/target | tail

Confirm two things. The last run exit with zero, and the newest archive both recent and roughly the size wey you expect. Backup file wey suddenly become one-tenth of the usual size na failed dump wey still write file. This one na the most dangerous kind of backup failure because everything downstream still look normal.

If your script pipe dump enter compressor, add set -o pipefail for the top. Without am, exit status of the pipeline na the compressor own, and the compressor succeed: e compress the error message. The job then report success every night while e dey write small archive of nothing.

Monthly: restore backup for another place

Na this item plenty people dey skip, and na e dey decide whether the rest of the list get meaning.

Restore am to another machine or fresh container, never on top live data. Then open wetin you restore and confirm say e real. Count rows for one table. Open one document. Log in to the restored application. If extraction finish, e only prove say archive readable; e no prove anything more.

Repository tools get their own verification: restic check --read-data-subset=5% and borg check --verify-data read the data wey dey stored, instead of index. Run dem, and treat dem as smoke test, no be replacement. Verification dey check say bytes survive. Restore dey check say na the bytes wey your application need.

Two details people dey learn through hard experience. Test the decryption passphrase for a machine wey no already hold the key inside an agent, because backup wey you no fit decrypt no be backup. And measure how long restore take, because na that duration be your real recovery time, and outage na usually when you go discover am.

Monthly: which certificates dey expire soon?

Renewal automation fit fail quietly. The certbot timer fit renew the file for disk while the web server still dey serve the old certificate from memory, because the deploy hook wey suppose reload the service no run. So ask the running server which certificate e dey serve, from outside the box.

sudo certbot certificates
systemctl list-timers --all | grep -i certbot
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | openssl x509 -noout -dates

The -servername flag set SNI (server name indication). You need am for any address wey host more than one site. Otherwise, dem go give you the default certificate instead of your own. If you install certbot from a snap, the timer name go different. So match the word instead of assuming one unit name.

Remember certificates wey no get automation at all: a mail server, a VPN, or an internal certificate authority. Na these ones dey expire during weekend. Browsers and clients go reject dem outright instead of showing warning.

Monthly: users, sudo access and SSH keys

awk -F: '$3>=1000 && $3<65534 {print $1}' /etc/passwd
getent group sudo
sudo find /home /root -name authorized_keys -exec ls -l {} +
sudo sshd -T | grep -E 'permitrootlogin|passwordauthentication|pubkeyauthentication|port'
last -n 25

sshd -T go print the effective configuration after e don merge every Include. Na this configuration the daemon go really use. Recent Ubuntu images dey ship drop-in files for /etc/ssh/sshd_config.d/, and dem fit override the main file. So if you read sshd_config alone, e fit show you the opposite of the real configuration. For RHEL family, the administrative group na wheel, no be sudo. So adjust the getent line.

Then read the authorized_keys files themselves. Na key dey grant access, no be account. So key wey contractor leave behind after e finish work six months ago still be working login. No user list go flag am. Keys get comment field. Use am, and delete anything wey you no fit link to one person.

For login history, journalctl -t sshd --since "30 days ago" | grep -i accepted dey match the syslog identifier, no be unit name. This matter because Ubuntu 24.04 dey activate SSH through a socket. So each connection dey log under one generated per-connection unit. Plain journalctl -u ssh fit miss dem.

Monthly: old kernels and a full /boot

/boot na often separate partition wey get some hundred megabytes for standard VPS image. Every kernel update dey add image and initramfs enter am. When e full, the next upgrade go fail halfway and leave packages wey never configure, and na bad state be that to discover suddenly on Friday.

uname -r
df -h /boot
dpkg -l 'linux-image-*' | grep '^ii'
sudo apt autoremove --purge

uname -r first, always: e dey show the kernel wey you dey run right now, and that one must remain safe no matter wetin you remove. apt autoremove dey handle the normal case for Debian and Ubuntu, because system mark kernels as automatically installed and protect the current one. The edge cases — kernel wey person install manually, or a /boot wey don full reach the point wey e dey block apt itself — dey covered for clearing old kernels on Ubuntu.

Monthly: log growth and the systemd journal

journalctl --disk-usage
sudo du -xh --max-depth=1 /var/log | sort -h | tail
sudo logrotate --debug /etc/logrotate.conf

logrotate --debug na dry run and e no write anything, so e safe for live box. E good make you run am because rotation rules dey match paths: if application change where e dey store log during upgrade, e own rule no cover am again, and that file go continue grow without limit until e fill disk.

systemd dey cap the journal, but e dey use fraction of the filesystem instead of specific number wey you choose. Set SystemMaxUse= inside /etc/systemd/journald.conf and restart systemd-journald if you want specific ceiling. sudo journalctl --vacuum-time=14d go reclaim space immediately, and na one-time action instead of policy, so use am together with the config change.

Per release: reboot wey you dey postpone

Updated kernel package wey dey disk no be kernel wey dey run. Until you reboot, machine still dey run the old one. Live patching, where e dey available, fit cover only some fixes.

uname -r
ls -l /var/run/reboot-required
cat /var/run/reboot-required.pkgs
sudo needrestart

That flag file na Debian and Ubuntu convention wey package scripts dey write. RHEL family systems no dey create am. For dem, needs-restarting -r dey answer the equivalent question, and needs-restarting -r come from dnf-utils. needrestart, wey recent Ubuntu server images install by default, answer the level under the kernel. E list processes wey still dey map library wey dem replace for disk. Na why patched OpenSSL no go take effect until the services wey dey use am restart.

Schedule the reboot instead of avoiding am. For /etc/apt/apt.conf.d/50unattended-upgrades, Unattended-Upgrade::Automatic-Reboot "true"; and Unattended-Upgrade::Automatic-Reboot-Time "03:00";, hand the decision over to the time wey you choose. Planned reboot na also the only test to confirm say the box go come back. A broken fstab entry or service wey you never enable go show for boot and nowhere else.

Per release: plan upgrade

Ubuntu LTS releases get five years of standard support, while interim releases get nine months. So this choice go determine the upgrade work wey you go manage for years. See the trade-off for LTS versus interim releases for server.

lsb_release -a
cat /etc/update-manager/release-upgrades

do-release-upgrade dey read that file, while Prompt=lts restrict am to LTS-to-LTS moves. The LTS-to-LTS path normally dey open when the new version first point release come out, instead of release day. So check wetin your machine dey offer, instead of planning around date wey you assume. The jump process dey for upgrading Ubuntu 24.04 to 26.04.

Plan three months of extra time. Take snapshot wey you don test say you fit restore am. List your third-party apt repositories. The upgrade go disable dem, and every one need new target for the new release. Decide how you go roll back before you start. As of August 2026, Ubuntu 24.04 LTS get standard support reach April 2029. So na scheduling matter, no be emergency.

Wetin to automate, and wetin to keep manual

Automate the decisions wey you don already make: security updates, log rotation, certificate renewal, backup jobs. Automate the alerting too, because check wey depend on you to remember am na check wey no go happen for 2am. External monitor, like self-hosted status monitoring with Uptime Kuma, go catch the one thing wey no script for the box fit report: say the box no dey reachable.

Keep two things manual: restore test and account audit. Both need person to decide whether the result correct. If you prefer to read machine state for browser instead of terminal, Cockpit versus Webmin for server management compares the two usual web consoles.

The automation itself need its own check. Na why the first weekly item for this list be to verify the updater. Automation wey fail silently worse pass no automation, because e remove both the failure and the habit of checking at the same time.

The complete checklist for one place

# weekly
systemctl list-units --state=failed
systemctl list-timers --all
journalctl -u unattended-upgrades --since "8 days ago" --no-pager
apt list --upgradable
apt-mark showhold
df -h
df -i

# monthly
sudo certbot certificates
awk -F: '$3>=1000 && $3<65534 {print $1}' /etc/passwd
getent group sudo
sudo sshd -T | grep -E 'permitrootlogin|passwordauthentication'
uname -r
dpkg -l 'linux-image-*' | grep '^ii'
journalctl --disk-usage

FAQ

How often I suppose run Linux server maintenance?

Run am every week for things wey fit change by demself: update status, disk and inode space wey remain, failed units, and whether backup job don finish. Run am every month for slow changes: restore test, certificate expiry, audit of accounts and SSH keys, old kernels, and log growth. Run am once for each distribution release for version upgrade and reboot into the current kernel. Weekly check dey take few minutes for healthy server, and na why you suppose run am weekly instead of waiting until something look wrong.

Why I need test restore if backup job report success?

Because job dey report only its own exit status, and that status fit be successful even when archive no useful. A dump wey pipe enter compressor without set -o pipefail go return compressor status. So, failed dump wey produce only error message fit still exit zero and write small file. Restore am to another machine, open the data, and count something. The restore test go also show how long e take, and that time na your real recovery time.

I need reboot after every kernel update?

You need reboot before the new kernel fit become the running one. For Debian and Ubuntu, if /var/run/reboot-required dey present, e mean say package ask for reboot, while /var/run/reboot-required.pkgs show which package ask. For the RHEL family, that file no dey exist, and needs-restarting -r from dnf-utils answer the same question. Set automatic reboot window for /etc/apt/apt.conf.d/50unattended-upgrades instead of postponing am forever, because machine wey no reboot for one year get untested boot path and old kernel.

Which of these checks I fit automate safely?

Automate actions wey decision don already clear: security updates, log rotation, certificate renewal, and scheduled backups. Automate notification too, so failed unit or disk wey dey fill up fit reach you without person running command. Keep restore test and key audit manual, because person need judge whether result correct. Then add one check for the automation itself, because silent updater failure look exactly like everything dey work.