How to Pin the Kernel Your VPS Go Boot Next

Wetin decide which kernel your VPS go boot

Na one generated file, /boot/grub/grub.cfg, dey decide which kernel your VPS go boot next. You no dey edit that file. You edit the inputs wey generate am, then regenerate am. For Ubuntu cloud image, image vendor fit add one of those inputs. This fit make menu selection no matter, and na why GRUB_DEFAULT=1 followed by update-grub no change anything for rented server, even though the same two steps work for laptop installation.

Follow this order. Confirm say na you get control to choose the kernel. Read every input file, including the ones wey vendor add. Read the generated output and count the entries wey e really contain. Na only after that you choose pinning method. If you make mistake for machine wey you fit reach only through SSH, you go need rescue console. So, the safest answers dey for the end of this page, and dem often na the correct ones.

First check whether na your kernel you fit pin

uname -r
systemd-detect-virt
ls -1 /boot/vmlinuz-*

systemd-detect-virt printing kvm, qemu or xen means say na you dey run your own kernel, so everything wey dey below apply. If e print lxc or openvz, e means say your server dey share host kernel. So you no get your own bootloader, and you no get anything to pin. For that case, uname -r go report version wey no dey appear for /boot/vmlinuz-* at all, because na host own the kernel wey dey run, and no setting for your disk fit change am.

ls -1 /boot/vmlinuz-* na the real list of kernels wey you fit choose from. If e get only one line, e mean say dem don delete the previous kernel, and no bootloader setting fit bring am back. This one usually happen during autoremove. E good make you understand am before you clean up old kernels for Ubuntu on server wey matter to you.

File wey you edit no be the file GRUB dey read

/etc/default/grub hold plain shell variable assignments. E be input. /boot/grub/grub.cfg na output, and e dey start with # DO NOT EDIT THIS FILE plus the reason. Anything wey you write inside the output go disappear the next time dem install or remove a kernel package, because those package scripts dey regenerate am.

cat /usr/sbin/update-grub

update-grub na wrapper. E dey run grub-mkconfig -o /boot/grub/grub.cfg, wey dey read the variables, run every script for /etc/grub.d/, then write the result. Two commands, one direction: inputs dey enter, grub.cfg dey come out.

Wetin go override your setting: /etc/default/grub.d

grep -rn '^[^#]' /etc/default/grub /etc/default/grub.d/

The second path na the part wey people dey miss. grub-mkconfig first dey source /etc/default/grub, then e dey source every *.cfg file for /etc/default/grub.d/ according to glob order. Read the code wey dey do am:

grep -n 'default/grub' /usr/sbin/grub-mkconfig

Sourcing na plain shell, so the last assignment na im dey win. Ubuntu cloud images dey ship files for that directory, and dem dey set things like timeout and kernel command line after dem don read your file. Your GRUB_TIMEOUT=10 for /etc/default/grub go dey overwritten moments later by vendor file wey set am to 0. The grep above go print the exact assignments for your image, so read those ones instead of trusting this sentence.

The practical rule be say: put your own settings for file wey go sort last, like /etc/default/grub.d/99-local.cfg, instead of editing /etc/default/grub. Then nothing wey the image ship fit come after your own setting.

Why GRUB_FORCE_PARTUUID dey make menu selection irrelevant

grep -rn GRUB_FORCE_PARTUUID /etc/default/grub /etc/default/grub.d/
grep -n GRUB_FORCE_PARTUUID /etc/grub.d/10_linux
sudo grep -n 'root=PARTUUID' /boot/grub/grub.cfg

GRUB_FORCE_PARTUUID dey tell the generator make e find root filesystem by partition UUID. E dey write am directly for kernel command line as root=PARTUUID=..., instead of searching for filesystem UUID during boot. Image vendor set am because e dey help one disk image boot reliably for hardware wey dem no build am on. The second grep dey show you the code wey dey act on the variable, for /etc/grub.d/10_linux. That script dey your own disk, and na e be the authority for wetin your image dey do.

Na the consequence matter here: for that path, generator dey write one direct boot entry instead of complete list of installed kernels. Count wetin you end up with.

sudo grep -cE '^\s*(menuentry|submenu) ' /boot/grub/grub.cfg
sudo grep -nE '^\s*(menuentry|submenu) ' /boot/grub/grub.cfg

If the count na 1, no second entry dey to select, so GRUB_DEFAULT=1 dey name entry wey no exist. GRUB no fit resolve am, so e boot the first entry, wey be the new kernel wey you dey try avoid. grub-set-default no go help too, because default no be the broken part. The menu wey you dey try choose from never generate.

To bring back complete menu, move the vendor file aside and preview the result before you commit to am. grub-mkconfig without -o dey write to standard output and e no touch anything for disk.

sudo grub-mkconfig 2>/dev/null | grep -cE '^\s*(menuentry|submenu) '
sudo mkdir -p /root/grub-backup
sudo mv /etc/default/grub.d/<the file your grep named> /root/grub-backup/
sudo grub-mkconfig 2>/dev/null | grep -cE '^\s*(menuentry|submenu) '

Count wey jump from 1 to several mean say entries dey appear once the forcing comot. Nothing don write yet. Put the file back if the second count no look correct, because forced PARTUUID na how your provider image dey locate root filesystem, and removing am go move the machine onto search path instead. Take snapshot before you run update-grub for real.

If your only goal na to survive one bad kernel, stop here and use the safer options further down. Rebuilding boot menu for remote server to escape one upgrade na more risk than the problem worth.

Why entry numbers no be the correct thing to pin

GRUB_DEFAULT dey accept number, title, or identifier. Numbers dey count top-level entries from 0. Nested entry dey use > as separator, so GRUB_DEFAULT="1>2" mean the entry for index 2 inside submenu for index 1.

Indices dey change. 10_linux dey list kernels from newest to oldest, so when you install kernel, every older entry go move down by one, and when you remove kernel, dem go move up. Your carefully saved 1>2 still go resolve after that. But e don dey name another kernel. No error go happen, no warning go show, and you go only discover am after reboot.

Identifiers no dey change, because each one contain the kernel version. Read your own:

sudo awk -F"'" '/menuentry_id_option/ {print $2, "==>", $4}' /boot/grub/grub.cfg

Ignore the first few lines for the output. Dem na the variable wey dey define for the header. After that, the left side na the title wey reader dey see, and the right side na the identifier wey you pass to the tools. For entry inside submenu, join the submenu identifier and the entry identifier with >, for that order, exactly as the numeric form dey do.

Grub-reboot dey boot previous kernel one time

For remote server, one-time selection na the correct move because e go undo itself. grub-reboot dey write next_entry inside /boot/grub/grubenv. GRUB dey read that variable, clear am, then save the cleared value before e boot anything. So if kernel panic, e no go try am again for next boot. You get one attempt, then machine go return to the normal default by itself.

First confirm say your generated config dey read that variable at all:

sudo grep -n -B2 -A5 'next_entry' /boot/grub/grub.cfg

You suppose see one load_env line and one block wey set default from next_entry. If grep no print anything, your image no dey read grubenv during boot. That mean grub-reboot go accept for shell, but bootloader go ignore am. Na the same forced direct boot path from previous section, but e dey show for another place too.

sudo grub-reboot '<the identifier you copied>'
sudo grub-editenv list

grub-editenv list suppose now print one next_entry= line wey hold exactly wetin you pass. Open your provider console for browser tab, then reboot and check the result.

sudo reboot

uname -r

If uname -r report the older version, that mean the pin work. If e report the new version, either the identifier no resolve or grubenv no dey read. The machine still dey up either way, and na the reason why you use the one-shot form.

Faz GRUB_DEFAULT=saved continuar

GRUB_DEFAULT=saved go make the default come from saved_entry for grubenv, and you go set that value with grub-set-default. E dey survive kernel installs, because update-grub dey rewrite grub.cfg and e no dey touch grubenv.

echo 'GRUB_DEFAULT=saved' | sudo tee /etc/default/grub.d/99-local.cfg
sudo update-grub
sudo grub-set-default '<the identifier you copied>'
sudo grub-editenv list
sudo grep -n 'set default' /boot/grub/grub.cfg

The last command suppose print set default="${saved_entry}". If e print set default="0", something wey source after your file don set GRUB_DEFAULT back to literal value. So list /etc/default/grub.d/ again and check say 99-local.cfg really dey last when dem sort am.

GRUB_SAVEDEFAULT=true na different setting, and e easy to mix am up with this one. E dey save anything wey you just boot as the new default, so the default dey follow the last successful boot. For server, unattended reboot fit quietly change your pin because of this. Leave am off unless na wetin you want.

Pin by identifier still fit fail for one case. If you remove the kernel wey e name, the identifier no go resolve again, and system go fall back to the first entry. So hold the package too, or keep that kernel away from autoremove.

Make the menu show for provider console

Interactive selection need the menu to show for screen, but cloud images dey hide am. Put these for the file wey sort last, then run sudo update-grub.

GRUB_TIMEOUT=10
GRUB_TIMEOUT_STYLE=menu
GRUB_RECORDFAIL_TIMEOUT=10

GRUB_TIMEOUT_STYLE=hidden together with GRUB_TIMEOUT=0 no show anything, so person wey dey watch the console go see kernel messages start immediately and conclude say bootloader no run. GRUB_RECORDFAIL_TIMEOUT na the separate timeout wey GRUB use after boot wey no complete. Cloud images set am to 0 too, and na why server wey just fail to boot still no stop and wait for you.

If your provider give you serial console instead of graphical one and you still no see anything, GRUB dey write to terminal wey you no fit see. Add both lines together, because the first one select the outputs and the second one configure the port:

GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

From now, every boot go get extra ten seconds. Set the timeout back to 0 when you finish.

Options wey safer pass editing bootloader

To change bootloader input for machine wey you fit reach only through SSH na the highest-risk option for this page. Cheaper options dey, and most times dem solve the real problem.

Hold the kernel packages. If your goal na “make newer kernel no come give me”, tell package manager instead of telling bootloader.

apt list --installed 2>/dev/null | grep -E '^linux-(image|headers|generic|virtual|kvm|aws|azure|gcp|oracle)'
sudo apt-mark hold linux-image-virtual linux-headers-virtual
apt-mark showhold

Use any names wey the first command print, because cloud images often install the virtual or kvm flavour instead of generic. apt upgrade go skip package wey you hold, and e go show am with The following packages have been kept back:. unattended upgrades for Ubuntu go skip am too. The cost dey real: kernel wey you hold no go receive security fixes, so treat am like temporary pause wey get date, then release am with sudo apt-mark unhold. If you dey avoid kernel updates because reboot dey cause downtime, and no be because one kernel bad, live kernel patching for VPS fit solve that instead.

Take snapshot before upgrade. Snapshot fit restore within minutes. You no need type for console, and no risk say bootloader change go apply halfway. Take snapshot, upgrade, reboot, then verify. If the new kernel get problem, roll back. The boot path go remain exactly as e be before.

Use console or rescue image for machine wey don already go down. Once server no fit boot, bootloader config no be where you go fix the problem. That recovery path get its own procedure: wetin to do when VPS no fit boot after kernel update.

Wetin fit spoil, and the message wey you go see

Your edit to /boot/grub/grub.cfg don disappear. Dem install or remove kernel package, the package maintainer script run update-grub, and e generate the file again from the input files. The # DO NOT EDIT THIS FILE header name the two input locations. Na those ones you suppose edit.

grub-editenv: error: environment block too small. /boot/grub/grubenv no dey, or e don truncate. Create am again with sudo grub-editenv /boot/grub/grubenv create, then set your value again and confirm with sudo grub-editenv list.

Kernel wey you pin dey panic with VFS: Unable to mount root fs on unknown-block(0,0). The entry wey you pin point to kernel or initrd wey no dey for disk again. Most times, dem don remove the package while the identifier still remain for grubenv. To recover, boot from console with working entry, then clear the stale value.

uname -r no change after reboot wey you expect say e go change am. Check these 3 things in order: grub-editenv list still dey show your value, or dem don consume am; the identifier wey you set dey appear for current grub.cfg; and grub.cfg get set default line wey dey read the variable wey you set. One of these 3 go explain wetin happen every time.

The menu show by itself after crash. GRUB record failed boot for grubenv as recordfail=1, and this one force the menu to show for the next boot so person fit intervene. Clear am with sudo grub-editenv /boot/grub/grubenv unset recordfail after the machine don become healthy.

The one sentence wey worth remember be say: the file wey you edit no be the file wey GRUB dey read, and for cloud image, na the gap between both files dey cause the confusion. Read the generated config first. Everything for this page follow from wetin the file actually talk.

FAQ

GRUB_DEFAULT=1 no change which kernel my VPS dey boot because wetin?

Because for Ubuntu cloud image, the generated /boot/grub/grub.cfg often get only one boot entry. So index 1 no name anything, and GRUB fall back to the first entry. Confirm am with sudo grep -cE '^\s*(menuentry|submenu) ' /boot/grub/grub.cfg. If e show count of 1, na the answer be that. The cause na GRUB_FORCE_PARTUUID, wey image vendor set for file inside /etc/default/grub.d/. This one make the generator use direct boot path instead of building complete list of installed kernels. Find the file with grep -rn GRUB_FORCE_PARTUUID /etc/default/grub /etc/default/grub.d/.

How I fit boot the previous kernel only once?

Run sudo grub-reboot '<identifier>' with identifier wey you copy from your own grub.cfg, then reboot while provider console dey already open. GRUB clear next_entry before e boot, so the choice apply to exactly one attempt. If kernel panic, e no go retry am. Confirm say the value enter correctly with sudo grub-editenv list. Before you depend on am, run sudo grep -n next_entry /boot/grub/grub.cfg, because image wey config no load grubenv go ignore the command without any error.

I suppose pin by entry number or by identifier?

Use identifier. Entry numbers na positions for list wey 10_linux rebuild with newest first. So, whenever you install or remove any kernel, the numbers shift. Also, stale 1>2 fit still resolve to real but wrong entry, and nothing go print to warn you. Identifiers contain the kernel version. So dem either match the kernel wey you mean or fail to resolve. List dem with sudo grep -n menuentry_id_option /boot/grub/grub.cfg and copy the quoted string wey follow each entry line.

Holding the kernel package safer pass changing the bootloader?

For the usual goal, yes. sudo apt-mark hold linux-image-virtual linux-headers-virtual stop newer kernel from arriving at all. So boot path no change, and you no get anything to misconfigure from console wey you fit no access. First check the flavour names installed for your own box with apt list --installed, then verify the hold with apt-mark showhold. The trade-off na say held kernel no go receive security fixes. So decide when you go run sudo apt-mark unhold before you apply the hold.