How to Pin the Kernel Your VPS Go Boot Next

Wetin dey decide which kernel your VPS go boot

Na one generated file, /boot/grub/grub.cfg, dey decide which kernel your VPS go boot next. You no dey edit that file directly. You edit the input files, then regenerate am. For Ubuntu cloud image, image vendor fit provide one of those inputs. E fit make menu selection no matter again. Na why GRUB_DEFAULT=1 followed by update-grub fit change nothing for rented server, even though the same two steps dey work for laptop installation.

Follow this order. Confirm say na you get permission to choose the kernel. Read every input file, including the ones wey vendor add. Read the generated output and count the entries wey e really contain. Na after that you fit choose pinning method. If you make mistake for machine wey you fit reach only through SSH, you go need rescue console. So, the safest answers dey for the end of this page, and dem often be the correct ones.

Ferst check say na your kernel you fit pin

uname -r
systemd-detect-virt
ls -1 /boot/vmlinuz-*

systemd-detect-virt printing kvm, qemu or xen mean say na your own kernel you dey run, and everything wey follow apply. If e print lxc or openvz, e mean say your server dey share host kernel, so no bootloader dey wey belong to you, and nothing dey wey you fit pin. For that case, uname -r go report version wey no dey appear for /boot/vmlinuz-* at all, because na host own the kernel wey dey run, and no setting for your disk fit change am.

ls -1 /boot/vmlinuz-* na the real list of kernels wey you fit choose from. If e get only one line, dem don already delete the previous kernel, and no bootloader setting fit bring am back. This one usually happen during autoremove, so e good make you understand am before you clean up old kernels on Ubuntu for server wey matter to you.

File wey you edit no be the file GRUB dey read

/etc/default/grub hold plain shell variable assignments. Na input be this. /boot/grub/grub.cfg na output, and e dey open with # DO NOT EDIT THIS FILE plus the reason. Anything wey you write inside the output go disappear next time dem install or remove kernel package, because package scripts go regenerate am.

cat /usr/sbin/update-grub

update-grub na wrapper. E dey run grub-mkconfig -o /boot/grub/grub.cfg, wey dey read the variables, run every script for /etc/grub.d/, then write the result. Two commands, one direction: input dey enter, grub.cfg dey come out.

Wetin go override your setting: /etc/default/grub.d

grep -rn '^[^#]' /etc/default/grub /etc/default/grub.d/

The second path na the part wey people dey miss. grub-mkconfig go source /etc/default/grub first, then every *.cfg file wey dey inside /etc/default/grub.d/ according to glob order. Read the code wey dey do am:

grep -n 'default/grub' /usr/sbin/grub-mkconfig

Sourcing na plain shell, so the last assignment go win. Ubuntu cloud images dey ship files for that directory, and dem dey set things like timeout and kernel command line after dem don already read your file. Your GRUB_TIMEOUT=10 for /etc/default/grub go overwrite a moment later by vendor file wey set am to 0. The grep command above go print the exact assignments for your image, so read those ones instead of trusting this sentence.

The practical rule be say: put your own settings for file wey go sort last, like /etc/default/grub.d/99-local.cfg, instead of editing /etc/default/grub. That way, nothing wey the image ship fit come after your own setting.

Wetin make GRUB_FORCE_PARTUUID make menu selection no matter

grep -rn GRUB_FORCE_PARTUUID /etc/default/grub /etc/default/grub.d/
grep -n GRUB_FORCE_PARTUUID /etc/grub.d/10_linux
sudo grep -n 'root=PARTUUID' /boot/grub/grub.cfg

GRUB_FORCE_PARTUUID dey tell the generator make e find root filesystem by partition UUID. E write am direct for kernel command line as root=PARTUUID=..., instead of searching for filesystem UUID during boot. Image vendor set am because e make one disk image boot reliably for hardware wey dem no build am on. The second grep dey show the code wey dey use the variable, for /etc/grub.d/10_linux. That script dey your own disk, and na e be the authority for wetin your image dey do.

Na the result matter here: for that path, generator dey write one direct boot entry instead of complete list of installed kernels. Count wetin you end up with.

sudo grep -cE '^\s*(menuentry|submenu) ' /boot/grub/grub.cfg
sudo grep -nE '^\s*(menuentry|submenu) ' /boot/grub/grub.cfg

If the count na 1, no second entry dey to select. So GRUB_DEFAULT=1 dey name entry wey no exist. GRUB no fit resolve am, so e boot the first entry. Na the new kernel you dey try avoid be that. grub-set-default no go help too, because default no be the broken part. The menu wey you dey try choose from, dem never generate am.

To bring back complete menu, move the vendor file go another place and preview the result before you commit to am. grub-mkconfig without -o dey write to standard output and e no touch anything for disk.

sudo grub-mkconfig 2>/dev/null | grep -cE '^\s*(menuentry|submenu) '
sudo mkdir -p /root/grub-backup
sudo mv /etc/default/grub.d/<the file your grep named> /root/grub-backup/
sudo grub-mkconfig 2>/dev/null | grep -cE '^\s*(menuentry|submenu) '

If the count jump from 1 to several, e mean say entries dey appear once the forcing don comot. Nothing don write yet. Put the file back if the second count no look correct, because the forced PARTUUID na how your provider image dey locate its root filesystem. If you remove am, machine go use the search path instead. Take snapshot before you run update-grub for real.

If na only to survive one bad kernel you want, stop here and use the safer options wey dey further down. Rebuilding boot menu for remote server to escape one upgrade carry more risk than the problem worth.

Why entry numbers no be the correct thing to pin

GRUB_DEFAULT dey accept number, title, or identifier. Numbers dey count top-level entries from 0. Nested entry dey use > as separator, so GRUB_DEFAULT="1>2" mean the entry for index 2 inside submenu for index 1.

Indices dey change. 10_linux dey list kernels with newest one first, so when you install kernel, every older entry go move down by one, and when you remove kernel, dem go move up. Your carefully set 1>2 go still resolve after that. But e go now point to different kernel. Nothing go show error, nothing go warn you, and you go only discover am after reboot.

Identifiers no dey move, because each one contain the kernel version. Read your own:

sudo awk -F"'" '/menuentry_id_option/ {print $2, "==>", $4}' /boot/grub/grub.cfg

Ignore the first few lines for the output; dem na the variable wey dey define for the header. After that, the left side na the title wey reader dey see, while the right side na the identifier wey you pass to the tools. For entry inside submenu, join the submenu identifier and the entry identifier with >, for that order, exactly as the numeric form dey do.

Boot previous kernel one time with grub-reboot

For remote server, one-time selection na the right choice because e go undo itself. grub-reboot dey write next_entry inside /boot/grub/grubenv. GRUB dey read that variable, clear am, then save the cleared value before e boot anything. So, if kernel panic, e no go try that kernel again for next boot. You get one attempt, then the machine go return to the normal default by itself.

First confirm say your generated config dey read that variable at all:

sudo grep -n -B2 -A5 'next_entry' /boot/grub/grub.cfg

You suppose see one load_env line and one block wey dey set default from next_entry. If grep no print anything, your image no dey read grubenv during boot. That means grub-reboot go work for the shell, but bootloader go ignore am. Na the same forced direct boot path from previous section, but e dey show for another place.

sudo grub-reboot '<the identifier you copied>'
sudo grub-editenv list

grub-editenv list suppose now print one next_entry= line wey contain exactly wetin you pass. Open your provider console for browser tab, then reboot and check the result.

sudo reboot

uname -r

If uname -r report the older version, e mean say the pin work. If e report the new version, either the identifier no resolve or grubenv no dey read. The machine still dey up either way, and na that be the reason to use the one-time form.

Make the choice stick with GRUB_DEFAULT=saved

GRUB_DEFAULT=saved dey make the default come from saved_entry for grubenv, and you set that value with grub-set-default. E dey survive kernel installation, because update-grub dey rewrite grub.cfg and e no dey touch grubenv.

echo 'GRUB_DEFAULT=saved' | sudo tee /etc/default/grub.d/99-local.cfg
sudo update-grub
sudo grub-set-default '<the identifier you copied>'
sudo grub-editenv list
sudo grep -n 'set default' /boot/grub/grub.cfg

The last command suppose print set default="${saved_entry}". If e print set default="0", something wey source after your file don set GRUB_DEFAULT back to literal value, so list /etc/default/grub.d/ again and check say 99-local.cfg really dey sort last.

GRUB_SAVEDEFAULT=true na different setting, and e easy to mix am up with this one. E dey save anything wey you just boot as the new default, so the default dey follow the last successful boot. For server, unattended reboot fit quietly change your pin. Leave am off unless na wetin you want.

Pin by identifier still fit fail one way. If you remove the kernel wey e name, the identifier no go resolve again, and e go return you to the first entry. So hold the package too, or make sure autoremove no fit remove that kernel.

Put the menu for provider console

To choose interactively, the menu must show for screen, but cloud images dey hide am. Put these ones inside the file wey get the last name, then run sudo update-grub.

GRUB_TIMEOUT=10
GRUB_TIMEOUT_STYLE=menu
GRUB_RECORDFAIL_TIMEOUT=10

GRUB_TIMEOUT_STYLE=hidden together with GRUB_TIMEOUT=0 no dey show anything, so person wey dey watch the console go see kernel messages start immediately and conclude say bootloader skip. GRUB_RECORDFAIL_TIMEOUT na the separate timeout wey dem use after boot wey no complete. Cloud images set am to 0 too. Na why server wey just fail to boot still no go stop and wait for you.

If your provider dey give serial console instead of graphical one and you still no see anything, GRUB dey write to terminal wey you no fit see. Add both lines together, because the first one selects the outputs and the second one configures the port:

GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

From now on, dem go add ten seconds to every boot. Set the timeout back to 0 when you don finish.

Options wey safer pass editing the bootloader

Changing bootloader input for machine wey you fit reach only through SSH na the riskiest option for this page. Cheaper solutions dey, and most times dem solve the real problem.

Hold the kernel packages. If the goal na “make you no give me newer kernel”, tell package manager instead of bootloader.

apt list --installed 2>/dev/null | grep -E '^linux-(image|headers|generic|virtual|kvm|aws|azure|gcp|oracle)'
sudo apt-mark hold linux-image-virtual linux-headers-virtual
apt-mark showhold

Use the names wey the first command print, because cloud images often install the virtual or kvm flavour instead of generic. If the newer kernel show up around image refresh and you suspect say the release itself move under you, e no move, because a point release na the updates wey you already get bundled into new install media and e no give already patched server anything wey dem no offer am weeks earlier. apt upgrade dey skip held package, and e announce am with The following packages have been kept back:. unattended upgrades on Ubuntu skip am too. The cost dey real: held kernel no dey receive security fixes, so treat am as pause wey get date, then release am with sudo apt-mark unhold. If you dey avoid kernel updates because reboot dey cause downtime, and not because one kernel bad, live kernel patching on a VPS go solve that instead.

Take snapshot before upgrade. Snapshot fit restore within minutes. You no need type for console, and no chance dey for half-applied bootloader change. Take snapshot, upgrade, reboot, then verify. If the new kernel misbehave, roll back. The boot path go remain exactly as e be.

Use console or rescue image for machine wey don already go down. Once server no fit boot, bootloader config no be where you go fix the problem. That recovery path na separate procedure: wetin to do when VPS no fit boot after kernel update.

Wetin fit break, and the message wey you go see

Your edit to /boot/grub/grub.cfg don disappear. Dem install or remove a kernel package, the maintainer script run update-grub, and the file regenerate from the input files. The # DO NOT EDIT THIS FILE header name the two input locations. Edit those ones.

grub-editenv: error: environment block too small. /boot/grub/grubenv dey missing or e don truncate. Recreate am with sudo grub-editenv /boot/grub/grubenv create, then set your value again and confirm am with sudo grub-editenv list.

A pinned kernel panic with VFS: Unable to mount root fs on unknown-block(0,0). The entry wey you pin point to a kernel or initrd wey no dey disk again. E usually happen because dem remove the package while the identifier remain for grubenv. To recover, boot a working entry from console, then clear the stale value.

uname -r no change after reboot wey you expect say go change am. Check three things in order: grub-editenv list still show your value or dem don consume am; the identifier wey you set dey inside the current grub.cfg; and grub.cfg get a set default line wey read the variable wey you set. Any one of these three go explain am every time.

The menu appear by itself after a crash. GRUB record failed boot for grubenv as recordfail=1, and that one force the menu to show for the next boot so person fit intervene. Clear am with sudo grub-editenv /boot/grub/grubenv unset recordfail once the machine don healthy.

The one sentence wey worth keeping be this: the file wey you edit no be the file wey GRUB dey read, and for cloud image, na the gap between dem two dey cause the confusion. Read the generated config first. Everything for this page follow from wetin e actually talk.

FAQ

Why GRUB_DEFAULT=1 no dey change which kernel your VPS dey boot?

Because for Ubuntu cloud image, the generated /boot/grub/grub.cfg often get only one boot entry. So index 1 no point to anything, and GRUB go fall back to the first entry. Confirm am with sudo grep -cE '^\s*(menuentry|submenu) ' /boot/grub/grub.cfg. If the count na 1, na that be the answer. The cause na GRUB_FORCE_PARTUUID, wey image vendor set inside file under /etc/default/grub.d/. This setting make the generator use direct boot path instead of building full list of installed kernels. Find the file with grep -rn GRUB_FORCE_PARTUUID /etc/default/grub /etc/default/grub.d/.

How I fit boot the previous kernel only once?

Run sudo grub-reboot '<identifier>' with identifier wey you copy from your own grub.cfg, then reboot while provider console don already open. GRUB dey clear next_entry before e boot, so the choice apply to exactly one attempt. If kernel panic, GRUB no go retry am. Confirm say the value enter correctly with sudo grub-editenv list. Before you depend on this, run sudo grep -n next_entry /boot/grub/grub.cfg, because image wey config no load grubenv go ignore the command without showing error.

I suppose pin by entry number or by identifier?

Use identifier. Entry numbers na positions inside list wey 10_linux rebuild from newest to oldest. Installing or removing any kernel go shift the numbers. Stale 1>2 fit still resolve to real but wrong entry, and e no go print warning. Identifiers contain the kernel version, so dem either match the kernel you mean or fail to resolve. List dem with sudo grep -n menuentry_id_option /boot/grub/grub.cfg, then copy the quoted string wey follow each entry line.

E safer to hold kernel package or change the bootloader?

For the usual goal, yes. sudo apt-mark hold linux-image-virtual linux-headers-virtual stop newer kernel from arriving at all. So boot path no go change, and you no get anything to misconfigure from console wey you fit no access. First check the flavour names wey dey installed for your own box with apt list --installed. Then verify the hold with apt-mark showhold. The trade-off na say held kernel no go receive security fixes. So decide when you go run sudo apt-mark unhold before you apply the hold.