Read the kernel log on a VPS with dmesg

What the kernel log on a VPS holds

The kernel log on a VPS is the record your guest kernel keeps about its own work, and for some failures it is the only record that exists. Two commands read it. dmesg reads the kernel ring buffer directly, and journalctl -k reads the copy that systemd's journal stored. Go to one of them whenever a process died without leaving anything in its own log.

Most dmesg guides on the web are about hardware you do not have. A VPS has no USB device to unplug and no network card firmware to reload. What a guest kernel reports instead is memory pressure, plus the errors that come back from the host's virtual disk and virtual network devices. Those errors reach your application as timeouts, so they are easy to blame on the application.

Start with the newest complaints and nothing else:

sudo dmesg -T --level=err,warn | tail -40
sudo journalctl -k -p warning --no-pager | tail -40

--level=err,warn and -p warning both drop the routine boot chatter, so what is left is the kernel objecting to something. If both come back empty, skip to the last section, because either the read is blocked or the kernel is not yours to read.

Why did my process disappear with no error?

A process that vanished without writing a shutdown line was almost certainly killed by the kernel out-of-memory killer (OOM killer). The kill is delivered as SIGKILL, which no program can catch or handle, so the application never gets a chance to log anything. The kernel writes the record instead.

sudo dmesg -T | grep -iE 'out of memory|oom-kill|invoked oom-killer'
sudo journalctl -k | grep -iE 'out of memory|oom-kill'
sudo journalctl -k -b -1 | grep -iE 'out of memory|oom-kill'

The third command reads the boot before this one, which matters when the box rebooted after the incident. journalctl -k implies the current boot, so without -b -1 it will look clean no matter how bad the previous boot was.

A match looks like this. The names and numbers will be yours:

mysqld invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mysql.service,task=mysqld,pid=1234,uid=110
Out of memory: Killed process 1234 (mysqld) total-vm:1596768kB, anon-rss:512300kB, file-rss:0kB, shmem-rss:0kB, UID:110 pgtables:1580kB oom_score_adj:0

Read those lines in order:

• The invoked oom-killer line names the process that asked for memory and could not get it. This is often not the process that died.
• The oom-kill: line says where the limit was hit. global_oom means the whole machine ran out of memory. task_memcg= names the control group the victim was sitting in.
• The Out of memory: Killed process line names the victim and the memory it held when it died. anon-rss is the part that could not simply be dropped, so it is the number that made the kernel pick this process.

The kernel does not kill the process that leaked. It kills the process with the highest badness score, and that score is driven mainly by resident memory. A 200 MB script that allocates in a loop can push a 2 GB database over the edge, and the database is what dies, because the database is bigger. That is why the log so often points at software that was behaving perfectly.

Between the header and the kill line, the kernel prints every task it considered under Tasks state (memory values in pages):. It is long, and it is worth keeping, because it is a snapshot of memory one second before the kill.

sudo dmesg -T | grep -A 40 'Tasks state'

If the second line reads Memory cgroup out of memory rather than global_oom, the machine still had free RAM and one control group hit a ceiling of its own. This is the confusing case, because free -h looks fine afterwards. It happens when a systemd unit carries MemoryMax=, which is how you cap memory and CPU for one service, and it happens to containers that were started with memory limits set in a Compose file. The number to change is the limit, not anything in the kernel.

On the service side, the same event appears in the unit's own log:

systemd[1]: app.service: A process of this unit has been killed by the OOM killer.
systemd[1]: app.service: Main process exited, code=killed, status=9/KILL
systemd[1]: app.service: Failed with result 'oom-kill'.

status=9/KILL on its own only says that something sent signal 9. The kernel log is what tells you who sent it and why. Other codes point somewhere else entirely, and reading systemd exit codes is a separate skill worth having.

One case leaves no kernel message at all. systemd-oomd kills processes from userspace using pressure stall information (PSI), so its decisions never touch the kernel log. Ubuntu ships it enabled. Check it before you conclude nothing killed anything:

sudo journalctl -u systemd-oomd --since today
cat /proc/pressure/memory

If /proc/pressure/memory does not exist, PSI is disabled on that kernel and needs psi=1 on the kernel command line. When it does exist, a rising some avg10 value is the pressure that precedes a kill, and it is visible before anything dies.

A 1 GB VPS with no swap has nothing to reclaim when a spike arrives, so the first over-allocation becomes a kill instead of a slowdown. Whether swap is the right answer for your workload, and what it costs on a small plan, is worked through in whether your VPS needs swap.

dmesg or journalctl -k: which one has the data?

They carry the same messages from different storage, and each one holds data the other has lost.

dmesg reads the kernel ring buffer. The buffer lives in RAM, it is a fixed size, and it is emptied by every reboot. Check the size on your kernel:

grep CONFIG_LOG_BUF_SHIFT /boot/config-$(uname -r)

The value is a power of two in bytes, so 18 means a 256 KiB base buffer. When it fills, the oldest lines are overwritten silently. Nothing warns you, and nothing marks the gap.

That matters more than it sounds, because a public VPS generates constant kernel noise. One ufw logging rule facing the internet writes a line for every blocked packet, and port scans never stop:

[UFW BLOCK] IN=ens3 OUT= MAC=... SRC=203.0.113.9 DST=198.51.100.20 PROTO=TCP SPT=52134 DPT=23 WINDOW=1024 RES=0x00 SYN URGP=0

A few thousand of those push this morning's OOM report out of the buffer before lunch. So dmesg coming back clean does not mean the crash was imagined. It can simply mean the evidence was overwritten.

journalctl -k reads what journald captured, and the journal survives reboots when it is stored on disk. Check whether yours is:

journalctl --list-boots

One row means the journal is volatile. It is sitting under /run, in RAM, and it is discarded at every boot. Make it persistent:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
journalctl --disk-usage

--list-boots should now grow a row after the next reboot, and sudo journalctl -k -b -1 becomes the most useful command you have: it is the only way to read what the box said before it went down.

The journal loses in two situations. Journald reads /dev/kmsg like any other process and can fall behind a flood of messages, and when it does it says so plainly:

systemd-journald[318]: Missed 12 kernel messages

Those lines are absent from the journal while they may still be in the ring buffer. The other situation is journald not running at all: in a rescue shell or at an initramfs prompt, dmesg is the only reader available.

Timestamps are the last difference. dmesg prints seconds since boot. dmesg -T converts them by subtracting uptime from the current wall clock, so a clock that jumped after boot shifts every converted timestamp by the size of the jump, and the manual page warns about exactly this. journalctl stamps entries as it receives them and needs no conversion. When the two disagree on a guest, suspect the conversion first, and treat it as one more reason to fix a VPS clock that drifts.

To watch either one live while you reproduce a fault:

sudo dmesg -Tw
sudo journalctl -kf

Avoid dmesg -C. It empties the ring buffer, and anything journald missed is then gone for good.

Disk errors that look like application timeouts

Your VPS disk is a virtual device. The guest kernel hands a request to the host, and if the host does not answer in time the guest kernel gives up and logs it. Your application sees only a slow query or a failed write.

sudo dmesg -T | grep -iE 'I/O error|EXT4-fs error|blocked for more than'

An I/O error on the virtual disk. A line like I/O error, dev vda, sector 2048 op 0x1:(WRITE) names the device and the operation that failed. On a guest this almost never means a dying disk, because the disk is not yours. It means the host storage did not answer, or the volume hit a throughput limit on your plan.

A filesystem that gave up. After enough failed writes, ext4 protects itself by refusing to write at all:

EXT4-fs error (device vda1): ext4_journal_check_start: Detected aborted journal
EXT4-fs (vda1): Remounting filesystem read-only

Your version of that first line will also carry a function name and a line number. From the moment of the remount, every write fails with Read-only file system while reads keep working, so a web application can serve pages normally and fail every upload. Confirm it with mount | grep ' / ' and look for ro, in the options. This is a different failure from a disk that is simply full, which fails writes with No space left on device and is usually a mismatch between what df and du report.

A task stuck waiting on the host.

INFO: task jbd2/vda1-8:214 blocked for more than 120 seconds.
      Not tainted 6.8.0-45-generic #45-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

The kernel prints this when a process has been in uninterruptible sleep for two minutes. On a guest that means it is waiting on I/O the host has not completed. The process is not deadlocked, it is starved. When the blocked task is the journal thread, jbd2/vda1-8 in the example above, every write to that filesystem was stalled behind it, so the whole box appeared to freeze.

One boot line is worth knowing here, because it tells you which virtual disk is which:

virtio_blk virtio2: [vda] 62914560 512-byte logical blocks (32.2 GB/30.0 GiB)

Dropped packets and conntrack pressure

Network failures in the kernel also reach you as application timeouts, because the packet never arrives at the service to be logged.

nf_conntrack: nf_conntrack: table full, dropping packet

Connection tracking (conntrack) stores one entry per tracked flow, and the module loads as soon as you use a stateful firewall rule or start Docker. When the table is full the kernel drops new connections while existing connections keep working, so the service looks healthy from inside the box and unreachable from outside it. Compare the two numbers:

sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max

A count sitting near the maximum is the confirmation. Raise nf_conntrack_max in a file under /etc/sysctl.d/ so the change survives a reboot, and shorten the UDP timeout if the entries are mostly UDP.

A second message gets misread constantly:

TCP: request_sock_TCP: Possible SYN flooding on port 443. Sending cookies.  Check SNMP counters.

This means the accept queue of that listening socket filled up. An attack is one cause. An application too slow to accept the connections it is offered is the other, and the kernel prints the identical line for both. Run ss -lnt and read Recv-Q on the listening socket, which is the current queue depth, against Send-Q, which is the configured backlog. If the queue is full while traffic is normal, the application is the problem.

net_ratelimit: 24 callbacks suppressed means the kernel discarded some of its own messages to keep the log usable, so the number of events was higher than the number of lines you can see. Before reading any of this as a network fault, confirm the basics: checking whether the port is actually open and listening rules out the simpler explanation in about ten seconds.

What the boot messages say after a kernel upgrade

The first forty lines of a boot describe the machine you are really on:

sudo dmesg | head -40
sudo journalctl -k -b -1 | head -40

Linux version 6.8.0-45-generic (buildd@lcy02-amd64-045) ... #45-Ubuntu SMP PREEMPT_DYNAMIC ...
Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.0-45-generic root=UUID=... ro console=tty1 console=ttyS0
Hypervisor detected: KVM

Linux version is the kernel that actually booted, which is not always the one you installed last. Command line is what the bootloader really passed, so it settles any argument you believe you set. Hypervisor detected names the virtualisation the guest found underneath it.

After an upgrade installs a new kernel, compare what booted against what is on disk:

uname -r
ls /boot/vmlinuz-*

If uname -r is older than the newest file in /boot, the machine came back on the previous kernel. Sometimes that is deliberate. Sometimes it is a failed boot that fell back without telling anyone, and a VPS that will not boot after a kernel update needs its own recovery steps. When you want one specific version to stay in charge, pinning which kernel boots is the durable fix.

One more check costs nothing: cat /proc/sys/kernel/tainted returns 0 on a clean kernel. A non-zero value means an out-of-tree module was loaded or the kernel already hit an internal error, and the flags are repeated in the header of every warning it prints afterwards, as Tainted: instead of Not tainted.

Why is the kernel log empty or permission denied?

There are two reasons a reader sees nothing useful, and they are not the same problem.

The read is restricted. Plain dmesg fails like this:

dmesg: read kernel buffer failed: Operation not permitted

sysctl kernel.dmesg_restrict

A value of 1 means only a process holding the CAP_SYSLOG capability may read the buffer, so sudo dmesg works while dmesg does not. Hardened images and several provider templates ship it set to 1. The journal has its own version of this rule, based on group membership: a user outside the adm and systemd-journal groups sees only their own entries, and journalctl prints a hint saying so. Fix it with sudo usermod -aG adm alice and a fresh login.

The kernel is not yours. Ask what you are running on:

systemd-detect-virt

kvm means a full virtual machine with a kernel of its own, and everything above applies to you. lxc or openvz means a container. Your VPS shares the host's kernel with other tenants, dmesg either fails with that same Operation not permitted or shows nothing at all, and journalctl -k comes back with no entries, because the messages belong to a kernel you do not administer.

That is not a defect to fix. It is the product you bought. On a container VPS you cannot load kernel modules, and most sysctl values are read-only because they are shared. An OOM kill that ended your process can even be a result of memory pressure created by a neighbour on the same host. Reading your own kernel log requires having your own kernel, which is the practical difference between KVM, Xen and LXC virtualisation on a VPS.

Keep the record before you need it

The ring buffer dies at the next reboot, and the journal dies when the VPS is rebuilt. If the box you are investigating is also the box holding the evidence, one impatient reboot ends the investigation. Make the journal persistent, then cap it with SystemMaxUse= in /etc/systemd/journald.conf so it can never fill the disk you are trying to debug. For anything you would be sorry to lose, send it somewhere else: a self-hosted log server on a second VPS keeps kernel messages readable long after the guest that wrote them is gone.

FAQ

Why does dmesg say Operation not permitted?

The full error is dmesg: read kernel buffer failed: Operation not permitted. Run sysctl kernel.dmesg_restrict. If it returns 1, only processes with the CAP_SYSLOG capability may read the kernel ring buffer, so sudo dmesg will work where dmesg will not. If it returns 0 and the command still fails, run systemd-detect-virt. An answer of lxc or openvz means your VPS is a container that shares the host's kernel, and that kernel's log is not exposed to you at any privilege level.

How do I find out what the OOM killer killed?

Run sudo dmesg -T | grep -iE 'out of memory|oom-kill', and add sudo journalctl -k -b -1 | grep -i 'out of memory' if the machine has rebooted since. The line beginning Out of memory: Killed process names the victim, its PID and its resident memory. The oom-kill: line above it says whether the whole machine ran out, shown as global_oom, or whether one control group hit its own MemoryMax= ceiling, which prints as Memory cgroup out of memory. The process named on the invoked oom-killer line is the one that requested memory, and it is frequently not the one that was killed.

Should I use dmesg or journalctl -k?

Use both, because each keeps what the other loses. dmesg reads a fixed-size buffer in RAM that covers the current boot only, and firewall logging can overwrite an hour-old OOM report. journalctl -k keeps history across reboots when the journal is persistent, so journalctl -k -b -1 can show the boot that failed. Note that -k implies the current boot, so you must pass -b -1 explicitly. Journald can also fall behind and log Missed 12 kernel messages, and it is not running at all in a rescue shell, where dmesg is the only option.

Why does my VPS have no kernel log at all?

Run systemd-detect-virt. If it prints lxc or openvz, your VPS is a container running on the host's kernel, so there is no guest kernel to log anything and journalctl -k returns no entries. You also cannot load modules or change most sysctl values on such a plan. If it prints kvm, you do have your own kernel, and an empty result points at kernel.dmesg_restrict being set to 1 or at a journal that was never made persistent.

Why do disk errors show up as application timeouts?

A VPS disk is a virtual device backed by host storage. When the host is slow to answer, the guest kernel logs I/O error, dev vda or INFO: task ... blocked for more than 120 seconds, while your application only sees a request that never returned. If ext4 then logs Remounting filesystem read-only, reads keep succeeding and every write fails with Read-only file system, which is why a site can still serve pages while every upload fails.