VPS will not boot after a kernel update

What to do first when a VPS will not boot after a kernel update

A VPS that will not boot after a kernel update is usually recoverable in a few minutes, because the update did not delete the kernel that worked yesterday. Ubuntu installs a new kernel next to the old one and changes only which entry GRUB starts by default. So the first move is not a repair. Select the previous kernel in the boot menu, get a login prompt back, then diagnose from a running system.

Fixing this on a server is different from fixing a laptop, because no keyboard is attached and no monitor shows the panic. SSH will not answer either, since the machine never reached the point where sshd starts. Everything below happens through your provider's console.

Read your own console before you change anything. The text on that screen decides which failure class you are in, and two servers that both "will not boot" can need opposite fixes.

How do I reach the console when SSH is dead?

Open your provider's control panel and look for a console. The common names are VNC console, web console, noVNC, and serial console. Prefer the serial console when both exist, because it gives you real text you can scroll and copy, while a VNC view is only a picture of a screen. Find this control now, while the machine is healthy, and confirm it opens. Hunting for it during an outage costs you the calm you need. That check belongs in the first ten minutes on a new VPS, next to the firewall rules and the SSH keys.

Most panels also offer a rescue mode or recovery image. It boots a small system from the provider's network and attaches your disk as an extra device, so nothing on your disk runs. Rescue mode is the fallback for when GRUB itself is broken, and it is also how you copy data off a server you have decided not to save.

You will usually need a hard reset from the panel to reach the boot menu, since you cannot run sudo reboot on a machine you cannot log into. A hard reset is the same as cutting the power. Filesystems get an unclean shutdown, so expect a filesystem check on the next boot.

How do I pick an older kernel in the GRUB menu?

Watch the console from the moment you press reset. Press Esc repeatedly during the first seconds, or hold Shift on a machine that boots in legacy BIOS mode. The window is short, and the console viewer often takes a second to connect, so start pressing early and keep pressing.

When the menu appears, choose "Advanced options for Ubuntu". That submenu lists every installed kernel, newest first, with a recovery mode entry for each one. Pick the second normal entry, which is the kernel below the newest, and press Enter. Recovery mode is a different thing: it boots to a minimal single user system, and it is for repair work, not for getting your services back online.

If the older kernel boots, you have a running server again. Confirm what you are on and write the numbers down.

uname -r
dpkg -l 'linux-image-*' | grep '^ii'

The dpkg output is your list of installed kernels. If it holds only one line, you have no fallback at all, and that is the first thing to fix.

The GRUB menu never appears. What now?

Cloud images ship a config that hides the menu. Ubuntu images commonly set the timeout to 0 in a file under /etc/default/grub.d/, so the newest kernel starts at once and there is nothing to press.

There is also the opposite case, where the menu is on screen and waiting, and it looks like a hang. GRUB records a failed boot, and on the next start it can hold the menu open until somebody presses a key. On a box with no keyboard, that wait never ends. If your console shows a menu and nothing moves, that is what happened. Select an entry and continue.

Fix both while the machine is healthy. Edit /etc/default/grub:

GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=10
GRUB_RECORDFAIL_TIMEOUT=10
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200"
GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0,115200"

Then apply it and check that your edit survived, because files in /etc/default/grub.d/ are read after /etc/default/grub and can override you.

sudo update-grub
grep -rE 'TIMEOUT|TERMINAL' /etc/default/grub /etc/default/grub.d/

GRUB_TERMINAL="console serial" sends the menu to the graphical console and to the serial port, so it shows up in whichever viewer your panel gives you. The console= kernel arguments do the same for the boot messages that follow. Ten seconds of delay per boot is a cheap price for a menu you can actually reach at 2am.

Which failure class am I looking at?

Read the last twenty lines before the console stops moving. Four patterns cover most of what happens after a kernel update.

GRUB cannot find its own files. You get a grub rescue> prompt, or an error about a partition or file that does not exist, and no kernel message ever appears. The kernel is not involved yet. This follows a disk or partition change, or a bootloader written to the wrong device, rather than a kernel package on its own.

The kernel starts but cannot mount root. Kernel messages scroll, then you land in a busybox shell whose prompt reads (initramfs), or the boot ends in a panic about being unable to mount the root filesystem. The kernel loaded. The initramfs, which is the small temporary root that finds and mounts your real root filesystem, did not find the disk. On Ubuntu this shell is normally preceded by a message about giving up waiting for the root device, and it names the UUID it wanted. Copy that UUID and compare it against blkid output later.

A logical volume never appears. This is the previous class with one specific cause. At the (initramfs) prompt, run ls /dev/mapper. If the only entry is control, then no LVM (logical volume manager) volume was activated, so the root device does not exist yet. Bring the volume groups up by hand:

lvm vgchange -ay
ls /dev/mapper
exit

exit hands control back to the initramfs script, which retries the mount. If the system then boots, the new initramfs is missing the LVM pieces, and the repair is to rebuild that image rather than to touch the kernel.

Nothing from Linux at all. The console shows firmware text, a UEFI (unified extensible firmware interface) shell, a blank screen with no kernel output, or a reset loop. The failure is happening before Linux runs. Check which mode your server actually uses once you are back up, because many VPS instances boot in legacy BIOS mode and never touch the EFI path:

[ -d /sys/firmware/efi ] && echo UEFI || echo BIOS
mountpoint /boot/efi
sudo efibootmgr -v

An unmounted /boot/efi during the upgrade is a common cause on UEFI machines, because the packages that maintain the EFI system partition then wrote to an ordinary empty directory instead. The firmware keeps starting the old boot entry until that entry stops matching what is on disk.

One more pattern is not a boot failure at all. If you reach a root shell that says the system is in emergency mode, the kernel booted and userspace stopped. That usually means a bad line in /etc/fstab or a filesystem that failed its check. Run journalctl -xb in that shell and read the name of the unit that failed.

Is the kernel package broken, or the initramfs?

These two look identical from the console and need different repairs. Boot the old kernel, then compare the files.

ls -l /boot/vmlinuz-* /boot/initrd.img-*
df -h /boot

You want one vmlinuz- and one matching initrd.img- for every installed version, each with a believable size. A missing initrd, or one much smaller than its neighbours, means initramfs generation failed. The usual reason is a full /boot, and the evidence sits in the package logs:

sudo grep -iE 'no space|update-initramfs' /var/log/apt/term.log
sudo tail -n 40 /var/log/apt/history.log

history.log also lists exactly which packages the last runs installed and when, which settles any argument about what changed.

Free space first if /boot is full, then rebuild the image for the version you need and refresh the menu. Take the version string from your own ls output, since the placeholder below is not a real release:

KVER=6.8.0-XX-generic
sudo update-initramfs -c -k "$KVER"
sudo update-grub
ls -l /boot/initrd.img-$KVER

That last ls is the check. A file of normal size means the image is now there. If instead the kernel image itself is damaged, or dpkg -l shows the package in any state other than ii, reinstall the package:

sudo apt install --reinstall linux-image-$KVER
sudo dpkg --configure -a

Repairing from rescue mode when no kernel boots

If every entry in the menu fails, boot the provider's rescue image and repair the disk from outside. Your disk appears as an unmounted device, so nothing on it is running and nothing can fight you.

sudo lsblk -f
sudo vgchange -ay
sudo mount /dev/ubuntu-vg/ubuntu-lv /mnt
sudo mount /dev/vda2 /mnt/boot
sudo mount /dev/vda1 /mnt/boot/efi

for d in dev proc sys run; do sudo mount --rbind /$d /mnt/$d; done
sudo chroot /mnt /bin/bash

df -h /boot
update-initramfs -u -k all
update-grub
grub-install /dev/vda
exit

Test a new kernel without betting the next boot

GRUB can start one entry a single time and then fall back to your chosen default. Point the default at a kernel you trust, then launch the new one for one boot only. If it fails, a hard reset from the panel returns you to the good kernel with no console timing to get right.

Set GRUB_DEFAULT=saved in /etc/default/grub, run sudo update-grub, then list the entry titles so you can name one exactly:

grep -E "(menuentry|submenu) " /boot/grub/grub.cfg | cut -d"'" -f2
sudo grub-set-default "Advanced options for Ubuntu>Ubuntu, with Linux 6.8.0-XX-generic"
sudo grub-editenv list
sudo grub-reboot 0
sudo reboot

grub-editenv list should print your chosen title as saved_entry. That output is the proof the mechanism works, because saving requires a writable /boot/grub/grubenv, and on some layouts it silently is not. Entry 0 is the top of the menu, which is the newest kernel. Titles are safer than numbers here, since numbers shift every time a kernel is installed or removed.

Why autoremove is risky on a headless box

APT keeps a list of kernel packages it must not remove on its own. Read yours:

cat /etc/apt/apt.conf.d/01autoremove-kernels
dpkg -l 'linux-image-*' | grep -c '^ii'

That file is regenerated whenever kernel packages change, and it protects the running kernel and the most recent ones. The trap is timing. Run sudo apt autoremove --purge right after rebooting into a fresh kernel, and the protected list has already moved forward, so the older kernel you were counting on is no longer protected. On a machine with a keyboard that is an inconvenience. On a headless server it is the difference between choosing a menu entry and mounting your disk from a rescue image.

Keep two kernels as a floor, and three when /boot has the room. Remove old ones by name after checking uname -r, so you can never delete the one you are running:

uname -r
sudo apt purge linux-image-6.8.0-XX-generic
dpkg -l 'linux-image-*' | grep '^ii'

Run that last command again afterwards. The count going from three to two is a cleanup. The count going to one is an outage waiting for the next reboot.

Take a snapshot before the upgrade

A snapshot taken before apt upgrade is the one recovery path that does not depend on booting anything. Restoring it puts the disk back to the state where the old kernel was the default, and you can retry the upgrade with the console already open. Snapshots of a running machine are crash consistent, which means they capture the disk as if the power had been cut, so shut the server down first when your provider supports an offline snapshot. A snapshot is also not a backup, because it usually lives on the same infrastructure as the volume it copies. Understanding the difference between VPS snapshots and real backups decides which one saves you when the failure is larger than a kernel.

This matters most on a release upgrade, where the kernel, the initramfs tools, the bootloader, and the GRUB config all change in one run. Take the snapshot immediately before you begin an Ubuntu 24.04 to 26.04 upgrade, not the night before, so the restore point matches the machine you are about to change.

How unattended-upgrades treats kernel packages

Ubuntu's unattended-upgrades installs security updates without asking, and kernel packages arrive through the security pocket like everything else. Two consequences follow.

First, the new kernel is installed but not running. A kernel only takes effect at boot. The file /var/run/reboot-required appears, and /var/run/reboot-required.pkgs names what asked for the reboot, but nothing restarts unless you enabled Unattended-Upgrade::Automatic-Reboot in /etc/apt/apt.conf.d/50unattended-upgrades.

cat /var/run/reboot-required.pkgs
grep -E 'Automatic-Reboot|Blacklist' /etc/apt/apt.conf.d/50unattended-upgrades

Second, that gap hides the cause. A server can install a kernel in March and reboot in June for a completely unrelated reason, then fail to come up. The change that broke the boot is three months old, so nothing you did that day explains it. /var/log/apt/history.log is where you find the run that installed the kernel you are now failing on.

Reboot on purpose, on a day you picked, with the console window already open. That single habit turns a mystery outage into a two minute menu selection. If you want the automation without the surprise, keep automatic installs on and automatic reboots off, and see how to configure unattended-upgrades on Ubuntu for the exact settings. Holding kernel packages with sudo apt-mark hold linux-image-generic stops them completely, and it stops kernel security fixes at the same time, so treat that as a trade you have decided to make rather than as a safety measure.

FAQ

How do I boot an older kernel on a VPS with no keyboard?

Open the provider's console (VNC or serial) and trigger a hard reset from the control panel, because you cannot log in to reboot cleanly. As the machine restarts, press Esc repeatedly, or hold Shift on a legacy BIOS boot, to hold the GRUB menu. Choose "Advanced options for Ubuntu" and select the entry below the newest kernel. Once you have a login prompt, run uname -r to confirm which kernel you are on and dpkg -l 'linux-image-*' to see what else is installed. Diagnose only after the system is running again.

Why does my VPS show no GRUB menu at all?

Cloud images commonly set the GRUB timeout to 0 in a file under /etc/default/grub.d/, so the newest kernel starts with nothing to press. Set GRUB_TIMEOUT=10 and GRUB_TIMEOUT_STYLE=menu in /etc/default/grub, add GRUB_TERMINAL="console serial" so the menu also reaches a serial console, then run sudo update-grub. Verify with grep -r TIMEOUT /etc/default/grub /etc/default/grub.d/, because files in that directory are read after the main file and can override your edit.

Should I remove old kernels to free space in /boot?

Remove the oldest ones and keep at least two. A full /boot is its own failure mode, because initramfs generation then fails and you are left with a kernel that has no working image. Purge by exact package name after checking uname -r, so the running kernel is never a candidate. Avoid a blanket sudo apt autoremove --purge on a headless machine, since the protected-kernel list is regenerated on every kernel change and a badly timed run can leave you with one kernel and no fallback entry in the menu.

Can unattended-upgrades break my boot?

It can install a kernel that later fails to boot, but it does not restart the machine unless Unattended-Upgrade::Automatic-Reboot is set to true in /etc/apt/apt.conf.d/50unattended-upgrades. The usual pattern is a delayed failure: the kernel lands during an automatic run, /var/run/reboot-required appears, and the problem only surfaces at your next reboot weeks later. Reboot deliberately with the console already open, and read /var/log/apt/history.log to find which run installed the kernel you are booting.