VPS no go boot after kernel update: wetin to do

Wetin to do first when VPS no go boot after kernel update

VPS wey no go boot after kernel update usually fit recover within few minutes, because the update no delete the kernel wey work yesterday. Ubuntu dey install new kernel beside the old one and only change which entry GRUB go start by default. So the first step no be repair. Select the previous kernel for the boot menu, make login prompt show again, then diagnose am from system wey dey run.

To fix this for server different from laptop, because keyboard no dey attached and monitor no dey show the panic. SSH no go answer too, since the machine never reach the point where sshd dey start. Everything wey dey below go happen through your provider console.

Read your own console before you change anything. The text for that screen go show which failure class you dey face, and two servers wey both "no go boot" fit need opposite fixes.

How I fit reach console when SSH no dey work?

Open your provider control panel and look for console. Common names na VNC console, web console, noVNC, and serial console. If both dey available, prefer serial console because e gives you real text wey you fit scroll and copy, while VNC view na only picture of screen. Find this control now, while the machine still healthy, and confirm say e opens. If you dey look for am during outage, e go waste the calm wey you need. Put this check for the first ten minutes on a new VPS, together with firewall rules and SSH keys.

Most panels still get rescue mode or recovery image. E boots small system from provider network and attaches your disk as extra device, so nothing for your disk go run. Rescue mode na fallback when GRUB itself don break. Na also the way to copy data from server wey you don decide say you no go save.

You go usually need hard reset from the panel to reach boot menu, because you no fit run sudo reboot for machine wey you no fit log into. Hard reset na the same thing as cutting power. Filesystems go get unclean shutdown, so expect filesystem check for the next boot.

How I fit choose older kernel for GRUB menu?

Watch the console from the moment you press reset. Press Esc repeatedly during the first seconds, or hold Shift for machine wey dey boot with legacy BIOS mode. The window short, and console viewer often need one second to connect, so start pressing early and continue pressing.

When the menu show, choose "Advanced options for Ubuntu". That submenu list every kernel wey dey installed, newest one first, with one recovery mode entry for each. Pick the second normal entry, wey be the kernel under the newest one, then press Enter. Recovery mode na different thing: e boot into minimal single user system, and na for repair work, no be to bring your services back online.

If the older kernel boot, your server don dey run again. Confirm the one wey you dey use and write the numbers down.

uname -r
dpkg -l 'linux-image-*' | grep '^ii'

The output from dpkg na your list of installed kernels. If e get only one line, you no get any fallback at all, and na the first thing you need fix.

GRUB menu no dey show. Wetin I go do?

Cloud images dey ship with config wey hide the menu. Ubuntu images commonly set the timeout to 0 for file under /etc/default/grub.d/, so the newest kernel go start immediately and nothing dey available to press.

The opposite case fit also happen, where the menu dey on screen and dey wait, making e look like system hang. GRUB dey record failed boot, and for the next start e fit keep the menu open until person press key. For machine wey no get keyboard, that wait no go ever end. If your console dey show menu and nothing dey move, na wetin happen be that. Select one entry and continue.

Fix both things while the machine still dey healthy. Edit /etc/default/grub:

GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=10
GRUB_RECORDFAIL_TIMEOUT=10
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200"
GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0,115200"

Then apply the change and check say e survive, because files for /etc/default/grub.d/ dey read after /etc/default/grub and fit override your setting.

sudo update-grub
grep -rE 'TIMEOUT|TERMINAL' /etc/default/grub /etc/default/grub.d/

GRUB_TERMINAL="console serial" go send the menu to the graphical console and serial port, so e go show for any viewer wey your panel give you. The console= kernel arguments dey do the same thing for boot messages wey follow. Ten seconds delay for every boot na small price to pay for menu wey you fit reach at 2am.

Kon failure class I dey look at?

Read the last twenty lines before the console stop dey move. Four patterns cover most things wey fit happen after kernel update.

GRUB no fit find im own files. You go see a grub rescue> prompt, or error about partition or file wey no exist, and no kernel message go ever show. Kernel never enter the matter yet. This one dey follow disk or partition change, or bootloader wey dem write to wrong device, instead of kernel package by itself.

Kernel start but e no fit mount root. Kernel messages go scroll, then you go land for busybox shell wey prompt dey show (initramfs), or boot go end with panic about inability to mount root filesystem. Kernel load successfully. The initramfs, wey be the small temporary root filesystem wey dey find and mount your real root filesystem, no find the disk. For Ubuntu, this shell normally dey come after message about giving up waiting for root device, and e go name the UUID wey e dey expect. Copy that UUID and compare am with blkid output later.

Logical volume no ever show. This na the previous class with one particular cause. For (initramfs) prompt, run ls /dev/mapper. If the only entry na control, then no LVM (logical volume manager) volume activate, so root device never exist yet. Bring the volume groups up by hand:

lvm vgchange -ay
ls /dev/mapper
exit

exit go hand control back to the initramfs script, wey go retry the mount. If system boot after that, the new initramfs no get the LVM components, and the repair na to rebuild that image instead of touching the kernel.

Nothing from Linux at all. Console go show firmware text, UEFI (unified extensible firmware interface) shell, blank screen with no kernel output, or reset loop. The failure dey happen before Linux run. Check which mode your server actually dey use after you bring am back up, because plenty VPS instances dey boot for legacy BIOS mode and never touch EFI path:

[ -d /sys/firmware/efi ] && echo UEFI || echo BIOS
mountpoint /boot/efi
sudo efibootmgr -v

Unmounted /boot/efi during the upgrade na common cause for UEFI machines, because the packages wey dey maintain EFI system partition go write to ordinary empty directory instead. Firmware go continue start the old boot entry until that entry no longer match wetin dey disk.

One more pattern no be boot failure at all. If you reach root shell wey talk say system dey emergency mode, kernel boot, but userspace stop. This usually mean bad line for /etc/fstab or filesystem wey fail its check. Run journalctl -xb for that shell and read the name of the unit wey fail.

Kernel package dey spoil, or na initramfs?

These two fit look the same for console, but dem need different repairs. Boot the old kernel, then compare the files.

ls -l /boot/vmlinuz-* /boot/initrd.img-*
df -h /boot

You need one vmlinuz- and one matching initrd.img- for every installed version, and each one suppose get believable size. If initrd dey miss, or e much smaller pass the others, that mean initramfs generation fail. The usual reason na full /boot, and package logs get the evidence:

sudo grep -iE 'no space|update-initramfs' /var/log/apt/term.log
sudo tail -n 40 /var/log/apt/history.log

history.log still list exactly which packages the last runs install and when dem install am. This settle any argument about wetin change.

If /boot full, free space first. Then rebuild the image for the version wey you need and refresh the menu. Take the version string from your own ls output, because the placeholder below no be real release:

KVER=6.8.0-XX-generic
sudo update-initramfs -c -k "$KVER"
sudo update-grub
ls -l /boot/initrd.img-$KVER

That last ls na the check. If file get normal size, that mean the image don dey there now. But if the kernel image itself damage, or dpkg -l show the package for any state wey no be ii, reinstall the package:

sudo apt install --reinstall linux-image-$KVER
sudo dpkg --configure -a

Repair wey you do from rescue mode when no kernel dey boot

If every entry for the menu fail, boot the provider rescue image and repair the disk from outside. Your disk go show as an unmounted device, so nothing for inside am dey run and nothing fit interfere with you.

sudo lsblk -f
sudo vgchange -ay
sudo mount /dev/ubuntu-vg/ubuntu-lv /mnt
sudo mount /dev/vda2 /mnt/boot
sudo mount /dev/vda1 /mnt/boot/efi

for d in dev proc sys run; do sudo mount --rbind /$d /mnt/$d; done
sudo chroot /mnt /bin/bash

df -h /boot
update-initramfs -u -k all
update-grub
grub-install /dev/vda
exit

Test new kernel without risking the next boot

GRUB fit start one entry for one time, then fall back to the default wey you choose. Point the default to a kernel wey you trust, then launch the new one for only one boot. If e fail, hard reset from the panel go return you to the good kernel, and you no need get console timing right.

Set GRUB_DEFAULT=saved for /etc/default/grub, run sudo update-grub, then list the entry titles so you fit name one exactly:

grep -E "(menuentry|submenu) " /boot/grub/grub.cfg | cut -d"'" -f2
sudo grub-set-default "Advanced options for Ubuntu>Ubuntu, with Linux 6.8.0-XX-generic"
sudo grub-editenv list
sudo grub-reboot 0
sudo reboot

grub-editenv list suppose print the title wey you choose as saved_entry. This output prove say the mechanism dey work, because saving need writable /boot/grub/grubenv, and for some layouts e no be writable without showing any error. Entry 0 na the top menu entry, and na the newest kernel. Titles safer pass numbers here, because numbers dey change every time you install or remove kernel.

Why autoremove risky for headless box

APT dey keep list of kernel packages wey e must not remove by itself. Read your own list:

cat /etc/apt/apt.conf.d/01autoremove-kernels
dpkg -l 'linux-image-*' | grep -c '^ii'

That file dey regenerate anytime kernel packages change, and e dey protect the kernel wey dey run plus the newest ones. The problem na timing. If you run sudo apt autoremove --purge immediately after you reboot into fresh kernel, the protected list don already move forward. So the older kernel wey you dey depend on no longer dey protected. For machine wey get keyboard, na inconvenience. For headless server, na the difference between choosing menu entry and mounting your disk from rescue image.

Keep at least two kernels, and keep three when /boot get enough space. Remove old ones by name after you check uname -r, so you no fit delete the kernel wey you dey run:

uname -r
sudo apt purge linux-image-6.8.0-XX-generic
dpkg -l 'linux-image-*' | grep '^ii'

Run that last command again afterwards. If the count drop from three to two, na cleanup. If e drop to one, outage dey wait for the next reboot.

Upgrade before make snapshot

Snapshot wey you take before apt upgrade na the only recovery path wey no depend on booting anything. If you restore am, e go return the disk to the state where the old kernel still be the default, and you fit retry the upgrade with the console already open. Snapshot of machine wey dey run dey crash consistent. This mean say e capture the disk as if power just cut. So shut down the server first when your provider support offline snapshot. Snapshot no be backup too, because e usually dey for the same infrastructure as the volume wey e copy. Understanding the difference between VPS snapshots and real backups go decide which one fit save you when the failure pass kernel problem.

This one matter pass during release upgrade, where the kernel, initramfs tools, bootloader, and GRUB config all change for one run. Take the snapshot immediately before you start Ubuntu 24.04 to 26.04 upgrade, no be the night before, so the restore point go match the machine wey you dey about to change.

How unattended-upgrades dey handle kernel packages

Ubuntu unattended-upgrades dey install security updates without asking, and kernel packages dey come through security pocket like every other package. Two things follow from this.

First, dem install the new kernel but e no dey run yet. Kernel only takes effect when system boot. The file /var/run/reboot-required go show, and /var/run/reboot-required.pkgs go show wetin request the reboot, but nothing go restart unless you enable Unattended-Upgrade::Automatic-Reboot for /etc/apt/apt.conf.d/50unattended-upgrades.

cat /var/run/reboot-required.pkgs
grep -E 'Automatic-Reboot|Blacklist' /etc/apt/apt.conf.d/50unattended-upgrades

Second, that gap dey hide the cause. Server fit install kernel for March and reboot for June because of completely different reason, then e no come up again. The change wey spoil the boot don old reach three months, so nothing wey you do that day go explain am. /var/log/apt/history.log na where you go find the run wey install the kernel wey dey cause the current failure.

Reboot on purpose, for the day wey you choose, with the console window already open. That one habit fit turn mystery outage into two-minute menu selection. If you want automation without surprise, leave automatic installs on and automatic reboots off, then see how to configure unattended-upgrades for Ubuntu for the exact settings. Holding kernel packages with sudo apt-mark hold linux-image-generic go stop dem completely, and e go stop kernel security fixes at the same time, so treat am as trade-off wey you decide to accept, not as safety measure.

FAQ

How I fit boot older kernel for VPS wey no get keyboard?

Open provider console (VNC or serial), then trigger hard reset from control panel, because you no fit log in make you reboot cleanly. As machine dey restart, press Esc plenty times, or hold Shift for legacy BIOS boot, to keep GRUB menu open. Choose "Advanced options for Ubuntu" and select the entry under the newest kernel. Once login prompt show, run uname -r to confirm which kernel you dey use, and dpkg -l 'linux-image-*' to see wetin else dey installed. Diagnose only after system don run again.

Why my VPS no dey show GRUB menu at all?

Cloud images commonly set GRUB timeout to 0 for file under /etc/default/grub.d/, so newest kernel go start without anything wey you fit press. Set GRUB_TIMEOUT=10 and GRUB_TIMEOUT_STYLE=menu for /etc/default/grub, add GRUB_TERMINAL="console serial" so menu fit reach serial console too, then run sudo update-grub. Verify with grep -r TIMEOUT /etc/default/grub /etc/default/grub.d/, because files for that directory dey read after the main file and fit override your edit.

I suppose remove old kernels to free space for /boot?

Remove the oldest ones and keep at least two. Full /boot na failure mode by itself, because initramfs generation go fail and you go remain with kernel wey no get working image. Purge with exact package name after you check uname -r, so running kernel no go become candidate. Avoid blanket sudo apt autoremove --purge for headless machine, because protected-kernel list dey regenerate every time kernel change, and badly timed run fit leave you with one kernel and no fallback entry for the menu.

Unattended-upgrades fit spoil my boot?

E fit install kernel wey later fail to boot, but e no go restart machine unless Unattended-Upgrade::Automatic-Reboot set to true for /etc/apt/apt.conf.d/50unattended-upgrades. The usual pattern na delayed failure: kernel land during automatic run, /var/run/reboot-required show, and problem only come out when you reboot weeks later. Reboot deliberately with console already open, and read /var/log/apt/history.log to find which run install the kernel wey you dey boot.