VPS no go boot after kernel update: how to fix am

First thing to do when VPS no go boot after kernel update

VPS wey no go boot after kernel update usually fit recover within few minutes, because the update no delete the kernel wey work yesterday. Ubuntu installs new kernel beside the old one and only change which entry GRUB dey start by default. So the first step no be repair. Select the previous kernel for boot menu, make login prompt come back, then diagnose am from running system.

To fix this for server different from fixing laptop, because no keyboard dey attached and no monitor dey show the panic. SSH no go answer too, since the machine never reach the point wey sshd starts. Everything below go happen through your provider console.

Read your own console before you change anything. The text for that screen go show which failure class you dey face, and two servers wey both "no go boot" fit need opposite fixes.

How I go reach the console when SSH no dey work?

Open your provider control panel and find the console option. Common names na VNC console, web console, noVNC, and serial console. If both dey available, prefer serial console because e give you real text wey you fit scroll and copy. VNC view na only picture of the screen. Find this control now while the machine still dey healthy, and confirm say e dey open. If you search for am during outage, e go waste the calm wey you need. Put this check inside the first ten minutes for a new VPS, together with firewall rules and SSH keys.

Most panels also get rescue mode or recovery image. E boot small system from the provider network and attach your disk as an extra device, so nothing for your disk go run. Rescue mode na fallback when GRUB itself don break. E also be the way to copy data from server wey you don decide say you no go save.

You usually need hard reset from the panel to reach the boot menu, because you no fit run sudo reboot for machine wey you no fit log into. Hard reset dey same as cutting the power. Filesystems go get unclean shutdown, so expect filesystem check for the next boot.

How I fit choose older kernel for GRUB menu?

Watch the console from the time you press reset. Press Esc again and again for the first few seconds, or hold Shift for machine wey dey boot with legacy BIOS mode. The time window short, and console viewer often dey take one second to connect, so start pressing early and continue pressing.

When the menu show, choose "Advanced options for Ubuntu". That submenu list every kernel wey install, newest one first, with one recovery mode entry for each. Choose the second normal entry, wey be the kernel below the newest one, then press Enter. Recovery mode na different thing: e dey boot into minimal single user system, and na for repair work, no be to bring your services back online.

If the older kernel boot, your server don dey run again. Confirm the kernel wey you dey use and write the numbers down.

uname -r
dpkg -l 'linux-image-*' | grep '^ii'

The dpkg output na your list of installed kernels. If e get only one line, you no get any fallback at all, and na the first thing you need fix.

GRUB menu no dey show. Wetin you fit do?

Cloud images dey come with config wey hide the menu. Ubuntu images common set the timeout to 0 for file under /etc/default/grub.d/, so newest kernel go start immediately and nothing dey available to press.

The opposite case dey happen too. Menu fit dey screen and dey wait, so e go look like hang. GRUB dey record failed boot, and for next start e fit hold the menu open until person press key. For machine wey no get keyboard, that wait no go ever end. If your console show menu and nothing dey move, na wetin happen be that. Select one entry and continue.

Fix both issues while the machine still dey healthy. Edit /etc/default/grub:

GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=10
GRUB_RECORDFAIL_TIMEOUT=10
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200"
GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0,115200"

Then apply the change and confirm say your edit remain, because files for /etc/default/grub.d/ dey read after /etc/default/grub and dem fit override your setting.

sudo update-grub
grep -rE 'TIMEOUT|TERMINAL' /etc/default/grub /etc/default/grub.d/

GRUB_TERMINAL="console serial" dey send the menu go graphical console and serial port, so e go show for whichever viewer your panel give you. The console= kernel arguments do the same thing for boot messages wey follow. Ten seconds delay for every boot na small price to pay for menu wey you fit reach at 2am.

Moi failure class I dey face?

Read the last twenty lines before the console stop to move. Four patterns cover most things wey fit happen after kernel update.

GRUB no fit find its own files. You go see a grub rescue> prompt, or error about partition or file wey no dey exist, and no kernel message go ever show. Kernel never enter the matter yet. This one dey follow disk or partition change, or bootloader wey dem write to wrong device, instead of kernel package by itself.

Kernel start but e no fit mount root. Kernel messages go scroll, then you go land for busybox shell wey prompt be (initramfs), or boot go end with panic say e no fit mount root filesystem. Kernel load successfully. The initramfs, wey be the small temporary root wey dey find and mount your real root filesystem, no find the disk. For Ubuntu, this shell normally dey come after message say e don give up waiting for root device, and the message go name the UUID wey e dey expect. Copy that UUID and compare am with blkid output later.

Logical volume no show at all. This na the previous class with one specific cause. For the (initramfs) prompt, run ls /dev/mapper. If the only entry na control, no LVM (logical volume manager) volume activate, so root device never exist yet. Bring the volume groups up by hand:

lvm vgchange -ay
ls /dev/mapper
exit

exit go hand control back to the initramfs script, wey go retry the mount. If system boot after that, the new initramfs dey miss the LVM components, and the repair na to rebuild that image instead of touching the kernel.

Nothing from Linux at all. Console go show firmware text, a UEFI (unified extensible firmware interface) shell, blank screen with no kernel output, or reset loop. The failure dey happen before Linux run. Check which mode your server actually dey use after you don come back up, because many VPS instances dey boot for legacy BIOS mode and dem never touch the EFI path:

[ -d /sys/firmware/efi ] && echo UEFI || echo BIOS
mountpoint /boot/efi
sudo efibootmgr -v

An unmounted /boot/efi during the upgrade na common cause for UEFI machines, because the packages wey maintain the EFI system partition go write to ordinary empty directory instead. Firmware go continue to start the old boot entry until that entry stop matching wetin dey on disk.

One more pattern no be boot failure at all. If you reach root shell wey talk say system dey emergency mode, kernel don boot and userspace stop. Most times, na bad line for /etc/fstab or filesystem wey fail its check cause am. Run journalctl -xb for that shell and read the name of the unit wey fail.

Kernel package dey spoil, or na initramfs?

These two fit look the same for console, but repair wey each one need different. Boot the old kernel, then compare the files.

ls -l /boot/vmlinuz-* /boot/initrd.img-*
df -h /boot

You need one vmlinuz- and one matching initrd.img- for every installed version, and each one suppose get believable size. If initrd no dey, or e much smaller pass the ones around am, initramfs generation don fail. The usual reason na full /boot, and package logs get the evidence:

sudo grep -iE 'no space|update-initramfs' /var/log/apt/term.log
sudo tail -n 40 /var/log/apt/history.log

history.log still dey list exactly which packages the last runs install and when dem install am. This go settle any argument about wetin change.

Free space first if /boot full. Then rebuild the image for the version wey you need and refresh the menu. Take the version string from your own ls output, because the placeholder below no be real release:

KVER=6.8.0-XX-generic
sudo update-initramfs -c -k "$KVER"
sudo update-grub
ls -l /boot/initrd.img-$KVER

That last ls na the check. File wey get normal size mean say the image don dey there now. But if the kernel image itself damage, or dpkg -l show the package for any state apart from ii, reinstall the package:

sudo apt install --reinstall linux-image-$KVER
sudo dpkg --configure -a

Repairing from rescue mode when no kernel boots

If every entry for the menu fail, boot the provider rescue image and repair the disk from outside. Your disk go appear as device wey no mount, so nothing for inside am dey run and nothing fit interfere with you.

sudo lsblk -f
sudo vgchange -ay
sudo mount /dev/ubuntu-vg/ubuntu-lv /mnt
sudo mount /dev/vda2 /mnt/boot
sudo mount /dev/vda1 /mnt/boot/efi

for d in dev proc sys run; do sudo mount --rbind /$d /mnt/$d; done
sudo chroot /mnt /bin/bash

df -h /boot
update-initramfs -u -k all
update-grub
grub-install /dev/vda
exit

Test new kernel without betting on the next boot

GRUB fit start one entry one time, then e go fall back to the default wey you choose. Set the default to a kernel wey you trust, then start the new one for only one boot. If e fail, hard reset from the panel go return you to the good kernel, and you no need get console timing right.

Set GRUB_DEFAULT=saved for /etc/default/grub, run sudo update-grub, then list the entry titles so you fit name one exactly:

grep -E "(menuentry|submenu) " /boot/grub/grub.cfg | cut -d"'" -f2
sudo grub-set-default "Advanced options for Ubuntu>Ubuntu, with Linux 6.8.0-XX-generic"
sudo grub-editenv list
sudo grub-reboot 0
sudo reboot

grub-editenv list suppose print the title wey you choose as saved_entry. That output prove say the mechanism dey work, because saving need a writable /boot/grub/grubenv, and for some layouts e no dey writable but no error show. Entry 0 na the top item for the menu, and na the newest kernel. Titles better pass numbers here, because the numbers dey change every time you install or remove kernel.

Why autoremove dey risky for headless box

APT dey keep list of kernel packages wey e no suppose remove by itself. Read your own list:

cat /etc/apt/apt.conf.d/01autoremove-kernels
dpkg -l 'linux-image-*' | grep -c '^ii'

That file dey regenerate anytime kernel packages change, and e dey protect the kernel wey dey run plus the latest ones. The trap na timing. If you run sudo apt autoremove --purge immediately after you reboot into fresh kernel, the protected list don already move forward. So the older kernel wey you dey depend on no longer dey protected. For machine wey get keyboard, na inconvenience. For headless server, na the difference between choosing menu entry and mounting your disk from rescue image.

Keep two kernels as the minimum, and keep three when /boot get enough space. Remove old ones by name after you check uname -r, so you no fit delete the one wey dey run:

uname -r
sudo apt purge linux-image-6.8.0-XX-generic
dpkg -l 'linux-image-*' | grep '^ii'

Run that last command again afterwards. If the count drop from three to two, na cleanup. If the count drop to one, na outage wey dey wait for the next reboot.

Take snapshot before upgrade

Snapshot wey you take before apt upgrade na the one recovery path wey no depend on booting anything. If you restore am, e go put disk back to the state wey old kernel still be the default, and you fit retry the upgrade with console already open. Snapshot of machine wey dey run dey crash consistent. This mean say e capture disk as if power don cut. So shut server down first when your provider support offline snapshot. Snapshot no be backup too, because e dey usually stay for the same infrastructure as the volume wey e copy. Understanding difference between VPS snapshots and real backups go determine which one fit save you when the failure pass kernel matter.

This one matter pass for release upgrade, where kernel, initramfs tools, bootloader, and GRUB config all change for one run. Take the snapshot immediately before you start Ubuntu 24.04 to 26.04 upgrade, no be the night before, so the restore point match the machine wey you dey about to change. If dem never offer that upgrade for your server, scheduling na the cause, no be broken setup, because LTS to LTS jump only open for first point release, 26.04.1.

How unattended-upgrades dey handle kernel packages

Ubuntu's unattended-upgrades dey install security updates without asking, and kernel packages dey come through security pocket like every other package. Two things follow from this.

First, the new kernel dey installed but e no dey run yet. Kernel only takes effect when system boot. File /var/run/reboot-required dey show, and /var/run/reboot-required.pkgs names wetin request the reboot, but nothing go restart unless you enable Unattended-Upgrade::Automatic-Reboot inside /etc/apt/apt.conf.d/50unattended-upgrades.

cat /var/run/reboot-required.pkgs
grep -E 'Automatic-Reboot|Blacklist' /etc/apt/apt.conf.d/50unattended-upgrades

Second, this gap dey hide the cause. Server fit install kernel for March and reboot for June because of completely unrelated reason, then e no come up again. The change wey break the boot don old reach three months, so nothing wey you do that day go explain am. /var/log/apt/history.log na where you go find the run wey install the kernel wey dey cause the current failure.

Reboot deliberately, for the day wey you choose, with the console window already open. This one habit fit turn mystery outage into two-minute menu selection. If you want automation without surprise, leave automatic installs on and automatic reboots off, then see how to configure unattended-upgrades for Ubuntu for the exact settings. Holding kernel packages with sudo apt-mark hold linux-image-generic go stop dem completely, and e go stop kernel security fixes at the same time, so treat am as trade-off wey you decide to accept, not as safety measure.

FAQ

How I fit boot older kernel for VPS wey no get keyboard?

Open provider console (VNC or serial), then trigger hard reset from control panel, because you no fit log in make you reboot cleanly. As machine dey restart, press Esc repeatedly, or hold Shift for legacy BIOS boot, so GRUB menu go stay open. Choose "Advanced options for Ubuntu" and select the entry wey dey under the newest kernel. Once login prompt show, run uname -r to confirm which kernel you dey use and dpkg -l 'linux-image-*' to see other kernels wey dey installed. Diagnose only after system don run again.

Why my VPS no dey show GRUB menu at all?

Cloud images commonly set GRUB timeout to 0 for file under /etc/default/grub.d/, so newest kernel go start without anything wey you fit press. Set GRUB_TIMEOUT=10 and GRUB_TIMEOUT_STYLE=menu inside /etc/default/grub, add GRUB_TERMINAL="console serial" so menu fit reach serial console too, then run sudo update-grub. Verify with grep -r TIMEOUT /etc/default/grub /etc/default/grub.d/, because files for that directory dey read after the main file and fit override your edit.

I suppose remove old kernels to free space for /boot?

Remove the oldest ones and keep at least two. If /boot full, na failure mode by itself, because initramfs generation go fail and you go remain with kernel wey no get working image. Purge with exact package name after you check uname -r, so running kernel no go become candidate. Avoid blanket sudo apt autoremove --purge for headless machine, because protected-kernel list dey regenerate every time kernel change and badly timed run fit leave you with one kernel and no fallback entry for menu.

Unattended-upgrades fit spoil my boot?

E fit install kernel wey later fail to boot, but e no go restart machine unless Unattended-Upgrade::Automatic-Reboot set to true inside /etc/apt/apt.conf.d/50unattended-upgrades. The usual pattern na delayed failure: kernel enter during automatic run, /var/run/reboot-required show, and problem no go appear until your next reboot weeks later. Reboot deliberately with console already open, then read /var/log/apt/history.log to find which run install the kernel wey you dey boot.