Pin which kernel your VPS boots next

What decides which kernel your VPS boots

Which kernel your VPS boots next is decided by one generated file, /boot/grub/grub.cfg. You never edit that file. You edit its inputs and regenerate it. On an Ubuntu cloud image one of those inputs arrives from the image vendor, and it can make menu selection irrelevant, which is why GRUB_DEFAULT=1 followed by update-grub changes nothing on a rented server while the same two steps work on a laptop install.

Work in this order. Confirm the kernel is yours to choose. Read every input file, including the ones the vendor added. Read the generated output and count the entries it really holds. Only then pick a pinning method. Getting this wrong on a machine you reach only over SSH costs you a rescue console, so the safest answers are at the end of this page and they are often the right ones.

First check that the kernel is yours to pin

uname -r
systemd-detect-virt
ls -1 /boot/vmlinuz-*

systemd-detect-virt printing kvm, qemu or xen means you run your own kernel and everything below applies. Printing lxc or openvz means your server shares the host's kernel, so there is no bootloader of yours and nothing to pin. In that case uname -r reports a version that does not appear in /boot/vmlinuz-* at all, because the running kernel belongs to the host and no setting on your disk can change it.

ls -1 /boot/vmlinuz-* is the real list of kernels you can choose between. If it holds one line, the previous kernel is already deleted, and no bootloader setting brings it back. That usually happens during an autoremove, which is worth understanding before you clean up old kernels on Ubuntu on a box you care about.

The file you edit is not the file GRUB reads

/etc/default/grub holds plain shell variable assignments. It is an input. /boot/grub/grub.cfg is the output, and it opens with # DO NOT EDIT THIS FILE and the reason. Anything you write into the output is gone the next time a kernel package is installed or removed, because those package scripts regenerate it.

cat /usr/sbin/update-grub

update-grub is a wrapper. It runs grub-mkconfig -o /boot/grub/grub.cfg, which reads the variables, runs every script in /etc/grub.d/, and writes the result. Two commands, one direction: inputs go in, grub.cfg comes out.

What overrides your setting: /etc/default/grub.d

grep -rn '^[^#]' /etc/default/grub /etc/default/grub.d/

The second path is the part people miss. grub-mkconfig sources /etc/default/grub first, then every *.cfg file in /etc/default/grub.d/ in glob order. Read the code that does it:

grep -n 'default/grub' /usr/sbin/grub-mkconfig

Sourcing is plain shell, so the last assignment wins. Ubuntu cloud images ship files in that directory, and they set things like the timeout and the kernel command line after your file has already been read. Your GRUB_TIMEOUT=10 in /etc/default/grub is overwritten a moment later by a vendor file that sets it to 0. The grep above prints the exact assignments on your image, so read those rather than trusting this sentence.

The practical rule that follows: put your own settings in a file that sorts last, such as /etc/default/grub.d/99-local.cfg, instead of editing /etc/default/grub. Then nothing shipped by the image can land after you.

Why GRUB_FORCE_PARTUUID makes the menu selection irrelevant

grep -rn GRUB_FORCE_PARTUUID /etc/default/grub /etc/default/grub.d/
grep -n GRUB_FORCE_PARTUUID /etc/grub.d/10_linux
sudo grep -n 'root=PARTUUID' /boot/grub/grub.cfg

GRUB_FORCE_PARTUUID tells the generator to find the root filesystem by partition UUID, written straight onto the kernel command line as root=PARTUUID=..., instead of searching for a filesystem UUID while booting. The image vendor sets it because it makes one disk image boot reliably on hardware it was not built on. The second grep shows you the code that acts on the variable, in /etc/grub.d/10_linux. That script is on your own disk, and it is the authority on what your image does.

The consequence is what matters here: on that path the generator writes a direct boot entry rather than a full list of installed kernels. Count what you ended up with.

sudo grep -cE '^\s*(menuentry|submenu) ' /boot/grub/grub.cfg
sudo grep -nE '^\s*(menuentry|submenu) ' /boot/grub/grub.cfg

If the count is 1, there is no second entry to select, so GRUB_DEFAULT=1 names an entry that does not exist. GRUB cannot resolve it, so it boots the first entry, which is the new kernel you were trying to avoid. grub-set-default does not help either, because the default is not the broken part. The menu you are trying to choose from was never generated.

To get a full menu back, move the vendor file aside and preview the result before committing to it. grub-mkconfig with no -o writes to standard output and touches nothing on disk.

sudo grub-mkconfig 2>/dev/null | grep -cE '^\s*(menuentry|submenu) '
sudo mkdir -p /root/grub-backup
sudo mv /etc/default/grub.d/<the file your grep named> /root/grub-backup/
sudo grub-mkconfig 2>/dev/null | grep -cE '^\s*(menuentry|submenu) '

A count that jumps from 1 to several means the entries appear once the forcing is gone. Nothing has been written yet. Put the file back if the second count does not look right, because the forced PARTUUID is how your provider's image locates its root filesystem, and removing it moves the machine onto the search path instead. Take a snapshot before you run update-grub for real.

If your only goal is to survive one bad kernel, stop here and use the safer options further down. Rebuilding the boot menu on a remote server to escape a single upgrade is more risk than the problem is worth.

Why entry numbers are the wrong thing to pin

GRUB_DEFAULT accepts a number, a title or an identifier. Numbers count top level entries from 0. A nested entry uses > as a separator, so GRUB_DEFAULT="1>2" means the entry at index 2 inside the submenu at index 1.

Indices move. 10_linux lists kernels newest first, so installing a kernel pushes every older entry down by one, and removing a kernel pulls them up. Your careful 1>2 still resolves after that. It now names a different kernel. Nothing errors, nothing warns, and you learn about it after a reboot.

Identifiers do not move, because each one contains the kernel version. Read yours:

sudo awk -F"'" '/menuentry_id_option/ {print $2, "==>", $4}' /boot/grub/grub.cfg

Ignore the first few lines of output, which are the variable being defined in the header. After that, the left side is the title a reader sees and the right side is the identifier you pass to the tools. For an entry inside a submenu, join the submenu identifier and the entry identifier with >, in that order, exactly as the numeric form does.

Boot the previous kernel once with grub-reboot

A one time selection is the right move on a remote server, because it undoes itself. grub-reboot writes next_entry into /boot/grub/grubenv. GRUB reads that variable, clears it, and saves the cleared value before it boots anything, so a kernel that panics is not retried on the next boot. You get one attempt, then the machine returns to its normal default on its own.

First confirm your generated config reads that variable at all:

sudo grep -n -B2 -A5 'next_entry' /boot/grub/grub.cfg

You want a load_env line and a block that sets default from next_entry. If the grep prints nothing, your image never reads grubenv at boot, so grub-reboot will be accepted at the shell and then ignored by the bootloader. That is the same forced direct boot path from the previous section showing up in a second place.

sudo grub-reboot '<the identifier you copied>'
sudo grub-editenv list

grub-editenv list should now print a next_entry= line holding exactly what you passed. Open your provider's console in a browser tab, then reboot and check the result.

sudo reboot

uname -r

uname -r reporting the older version means the pin worked. Reporting the new version means either the identifier did not resolve or grubenv is not being read, and the machine is up either way, which is the point of using the one shot form.

Make the choice stick with GRUB_DEFAULT=saved

GRUB_DEFAULT=saved makes the default come from saved_entry in grubenv, and you set that value with grub-set-default. It survives kernel installs, because update-grub rewrites grub.cfg and never touches grubenv.

echo 'GRUB_DEFAULT=saved' | sudo tee /etc/default/grub.d/99-local.cfg
sudo update-grub
sudo grub-set-default '<the identifier you copied>'
sudo grub-editenv list
sudo grep -n 'set default' /boot/grub/grub.cfg

The last command must print set default="${saved_entry}". If it prints set default="0", something sourced after your file set GRUB_DEFAULT back to a literal value, so list /etc/default/grub.d/ again and check that 99-local.cfg really sorts last.

GRUB_SAVEDEFAULT=true is a different setting and it is easy to confuse with this one. It saves whatever you just booted as the new default, so the default follows the last successful boot. On a server that means an unattended reboot can quietly move your pin. Leave it off unless that is what you want.

A pin by identifier still fails in one way. Remove the kernel it names and the identifier stops resolving, which drops you back to the first entry. So hold the package as well, or keep that kernel out of autoremove.

Getting the menu onto a provider console

Choosing interactively needs the menu on screen, and cloud images hide it. Put these in the file that sorts last, then run sudo update-grub.

GRUB_TIMEOUT=10
GRUB_TIMEOUT_STYLE=menu
GRUB_RECORDFAIL_TIMEOUT=10

GRUB_TIMEOUT_STYLE=hidden together with GRUB_TIMEOUT=0 shows nothing at all, so someone watching the console sees kernel messages start immediately and concludes the bootloader was skipped. GRUB_RECORDFAIL_TIMEOUT is the separate timeout used after a boot that did not complete, and cloud images set it to 0 as well, which is why a server that just failed to boot still does not stop and wait for you.

If your provider gives a serial console rather than a graphical one and you still see nothing, GRUB is writing to a terminal you cannot see. Add both lines together, because the first selects the outputs and the second configures the port:

GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

Ten seconds is added to every boot from now on. Set the timeout back to 0 once you are finished.

Safer options than editing the bootloader

Changing bootloader input on a machine you reach only over SSH is the highest risk option on this page. Cheaper answers exist, and they usually solve the real problem.

Hold the kernel packages. If the goal is "do not hand me a newer kernel", say that to the package manager rather than to the bootloader.

apt list --installed 2>/dev/null | grep -E '^linux-(image|headers|generic|virtual|kvm|aws|azure|gcp|oracle)'
sudo apt-mark hold linux-image-virtual linux-headers-virtual
apt-mark showhold

Use whatever names the first command printed, because cloud images commonly install the virtual or kvm flavour rather than generic. A held package is skipped by apt upgrade, which announces it with The following packages have been kept back:, and it is skipped by unattended upgrades on Ubuntu too. The cost is real: a held kernel stops receiving security fixes, so treat it as a pause with a date on it and release it with sudo apt-mark unhold. If you are avoiding kernel updates because reboots cost downtime rather than because one kernel is bad, live kernel patching on a VPS answers that instead.

Snapshot before the upgrade. A snapshot restores in minutes, with no console typing and no chance of a half applied bootloader change. Snapshot, upgrade, reboot, verify. If the new kernel misbehaves, roll back and the boot path is exactly what it was.

Use the console or a rescue image for a box that is already down. Once the server will not boot, the bootloader config is not where you fix it, and that recovery path is its own procedure: what to do when a VPS will not boot after a kernel update.

What breaks, and the message you will see

Your edit to /boot/grub/grub.cfg disappeared. A kernel package was installed or removed, its maintainer script ran update-grub, and the file was regenerated from the inputs. The # DO NOT EDIT THIS FILE header names the two input locations. Edit those.

grub-editenv: error: environment block too small. /boot/grub/grubenv is missing or truncated. Recreate it with sudo grub-editenv /boot/grub/grubenv create, then set your value again and confirm with sudo grub-editenv list.

A pinned kernel panics with VFS: Unable to mount root fs on unknown-block(0,0). The entry you pinned points at a kernel or initrd that is no longer on disk, usually because the package was removed while the identifier stayed in grubenv. Recovery is a console boot of a working entry, then clearing the stale value.

uname -r is unchanged after a reboot you expected to change it. Check three things in order: does grub-editenv list still show your value or was it consumed; does the identifier you set appear in the current grub.cfg; does grub.cfg contain a set default line that reads the variable you set. One of those three explains it every time.

The menu appeared on its own after a crash. GRUB records a failed boot in grubenv as recordfail=1, and that forces the menu on the following boot so a human can intervene. Clear it with sudo grub-editenv /boot/grub/grubenv unset recordfail once the machine is healthy.

The one sentence worth keeping: the file you edit is not the file GRUB reads, and on a cloud image the gap between them is where the confusion lives. Read the generated config first. Every decision on this page follows from what it actually says.

FAQ

Why does GRUB_DEFAULT=1 not change which kernel my VPS boots?

Because on an Ubuntu cloud image the generated /boot/grub/grub.cfg often holds a single boot entry, so index 1 names nothing and GRUB falls back to the first entry. Confirm it with sudo grep -cE '^\s*(menuentry|submenu) ' /boot/grub/grub.cfg. A count of 1 is the answer. The cause is GRUB_FORCE_PARTUUID, set by the image vendor in a file under /etc/default/grub.d/, which puts the generator on a direct boot path instead of building a full list of installed kernels. Find the file with grep -rn GRUB_FORCE_PARTUUID /etc/default/grub /etc/default/grub.d/.

How do I boot the previous kernel just once?

Run sudo grub-reboot '<identifier>' with an identifier copied out of your own grub.cfg, then reboot with the provider console already open. GRUB clears next_entry before it boots, so the choice applies to exactly one attempt and a kernel that panics is not retried. Confirm the value landed with sudo grub-editenv list. Before relying on it, run sudo grep -n next_entry /boot/grub/grub.cfg, because an image whose config never loads grubenv will ignore the command with no error.

Should I pin by entry number or by identifier?

By identifier. Entry numbers are positions in a list that 10_linux rebuilds newest first, so installing or removing any kernel shifts them, and a stale 1>2 still resolves to a real but wrong entry with nothing printed to warn you. Identifiers contain the kernel version, so they either match the kernel you meant or fail to resolve. List them with sudo grep -n menuentry_id_option /boot/grub/grub.cfg and copy the quoted string that follows on each entry line.

Is holding the kernel package safer than changing the bootloader?

For the usual goal, yes. sudo apt-mark hold linux-image-virtual linux-headers-virtual stops a newer kernel from arriving at all, so the boot path never changes and there is nothing to get wrong from a console you may not have. Check the flavour names installed on your own box first with apt list --installed, and verify the hold with apt-mark showhold. The trade is that a held kernel receives no security fixes, so decide when you will run sudo apt-mark unhold before you run the hold.