Fix NVIDIA 'key was rejected by service'

What "key was rejected by service" means

modprobe: ERROR: could not insert 'nvidia': Key was rejected by service means the kernel found your module, read it, and refused to load it. The module carries no signature from a key this kernel trusts, and this kernel booted with Secure Boot enforcing, so an unverified module is not allowed in. Nothing is wrong with the driver source and nothing is wrong with the compiler. Building a module and loading a module are separate steps, and only the load step checks signatures.

Secure Boot is a UEFI (unified extensible firmware interface) feature: the firmware runs boot code only when it is signed by a key the firmware holds. Linux extends that promise upward. When the kernel sees that it was booted through a Secure Boot chain, it switches on lockdown in integrity mode, and one thing integrity lockdown blocks is loading a module the kernel cannot verify. The refusal comes back as the errno EKEYREJECTED, which the C library prints as the words you searched for.

Confirm that mechanism instead of trusting the guess:

cat /sys/kernel/security/lockdown
sudo dmesg | grep -i lockdown

The first command prints the lockdown levels with the active one in square brackets, so none [integrity] confidentiality means enforcement is on. The dmesg line names the caller and the restriction, close to Lockdown: modprobe: unsigned module loading is restricted; see man kernel_lockdown.7.

One nearby message means the opposite thing. module verification failed: signature and/or required key missing - tainting kernel is a warning, not a refusal: the module did load, and the kernel only marked itself tainted. If that is the line you have, signing is not your problem, because enforcement was off on that boot.

Why DKMS reported a successful build

DKMS (dynamic kernel module support) rebuilds out-of-tree modules for each kernel you install. Its job ends when the compiled .ko file is in place under /lib/modules. Signing is a separate step, and DKMS performs it only when it has been given a key to sign with. That is why dkms status reporting installed and modprobe failing are both true at the same time.

Look at what DKMS produced, and at whether it is signed:

dkms status
ls -l /lib/modules/$(uname -r)/updates/dkms/
modinfo -F signer nvidia

dkms status should print a line for the driver against your running kernel, ending in installed. modinfo -F signer nvidia prints the common name of the key that signed the module, and prints nothing at all when the module is unsigned. Empty output there, plus [integrity] in the lockdown file, is the whole diagnosis.

Notice how many files the ls printed. The driver is not a single module. Depending on the release you will see nvidia, nvidia-modeset, nvidia-uvm and nvidia-drm, sometimes with nvidia-peermem alongside them. Each one has to be signed, because modprobe nvidia pulls the others in and the kernel verifies every file on its own. Signing only nvidia.ko moves the error to the next module instead of clearing it.

Is Secure Boot actually enforcing on this machine?

sudo apt update && sudo apt install -y mokutil
mokutil --sb-state
cat /proc/cmdline

mokutil --sb-state prints SecureBoot enabled or SecureBoot disabled. If the machine booted in legacy BIOS mode there is no Secure Boot variable to read, and mokutil reports that it cannot read the state rather than printing one. You can check that separately: the directory /sys/firmware/efi exists only after a UEFI boot.

These commands read UEFI variables through /sys/firmware/efi/efivars, which is firmware state belonging to one physical or virtual machine. Run them on the affected machine, over SSH or at its console. They cannot be demonstrated inside a container, because a container has no firmware of its own and borrows the host's kernel, so anything it reports describes the host rather than your problem.

If mokutil --sb-state says disabled and modules are still rejected, read /proc/cmdline. A kernel booted with module.sig_enforce=1 enforces signatures whatever the firmware says. That is the one case where switching Secure Boot off changes nothing at all.

Is a signing key enrolled?

mokutil --list-enrolled
mokutil --list-new
sudo dmesg | grep -i cert

mokutil --list-enrolled lists the keys shim has stored in the machine owner key database. Read the subject line of each entry. On a fresh install you will see whatever your distribution put there and nothing of your own.

mokutil --list-new lists a request you have already filed that has not been completed. That distinction matters, because an import is only a request. The firmware's key manager finishes it during the next boot and asks for the password you set at import time. Boot straight past that step and the key is never enrolled, while mokutil --list-new keeps showing the request. Do not assume the enrolment worked: re-run mokutil --list-enrolled after the reboot and look for your own subject line.

The dmesg output shows which certificates the kernel loaded at boot. Your own key appearing there is the proof that enrolment worked end to end, because it means the kernel, and not only shim, now trusts it.

Path 1: enrol a Machine Owner Key and sign the module

A MOK (machine owner key) is a key you own, stored by shim, and trusted by the kernel for module verification. The work has two halves. Get the key trusted once, then make sure every later build is signed with it.

On Ubuntu and Debian the driver packaging can handle the first half. Find the DKMS package that owns the driver, then reconfigure it:

dpkg -l | grep dkms
sudo dpkg-reconfigure NAME-FROM-THE-LINE-ABOVE

Reconfiguring re-runs the package's Secure Boot setup, which generates a key if none exists and asks you to choose a one-time password. Reboot, then complete the enrolment in the firmware key manager with that password. The screens are not transcribed here on purpose, because their wording differs between shim versions and firmware vendors. Read what is in front of you, then verify the result with mokutil --list-enrolled once the machine is back.

To do it by hand instead, generate the pair yourself:

openssl req -new -x509 -newkey rsa:2048 -nodes -days 3650 \
  -subj "/CN=$(hostname) module signing/" \
  -keyout module-signing.priv -outform DER -out module-signing.der
sudo install -o root -g root -m 600 module-signing.priv /root/module-signing.priv
sudo install -o root -g root -m 644 module-signing.der /root/module-signing.der
sudo mokutil --import /root/module-signing.der
sudo reboot

The file names are yours to choose. What matters is the mode on the private key. Anyone who can read it can sign a module that this kernel will then load without question, which is exactly the power Secure Boot exists to restrict. Keep it readable by root only, and keep a copy off the machine, because losing it means enrolling a new key from the start.

With the key enrolled, sign every module built for the running kernel:

KVER=$(uname -r)
ls /usr/src/linux-headers-$KVER/scripts/sign-file
for m in /lib/modules/$KVER/updates/dkms/*.ko; do
  sudo /usr/src/linux-headers-$KVER/scripts/sign-file sha256 \
    /root/module-signing.priv /root/module-signing.der "$m"
done
sudo depmod -a

sign-file ships inside the kernel headers package, so the ls is there to prove the path before the loop uses it. If that path does not exist, install the headers that match the running kernel and try again.

One detail catches people here. Some releases install DKMS modules compressed, so the file names end in .ko.zst. sign-file appends a signature to an ELF object and cannot work on a compressed file, which means the loop above quietly matches nothing. Decompress first, sign the plain .ko, then refresh the dependency files. An uncompressed module in that directory loads normally:

sudo unzstd --rm /lib/modules/$KVER/updates/dkms/*.ko.zst

Now load the driver and check the result:

sudo modprobe nvidia
modinfo -F signer nvidia
nvidia-smi

modinfo -F signer nvidia should now print the common name you chose, and nvidia-smi should print its table of GPUs. If modprobe still reports a rejected key after signing, the signature exists but the key is not trusted, so go back to mokutil --list-enrolled and confirm the enrolment finished.

Then close the loop so this never comes back by hand. DKMS can sign on every build. Its configuration file /etc/dkms/framework.conf carries mok_signing_key and mok_certificate for the key pair, plus sign_file for the signing binary, and man dkms documents all of them, including the fact that $kernelver may appear inside those paths. Read the shipped file before you edit it. Distributions set defaults there, and your packaged setup may already point at a key you did not know existed.

Path 2: stop enforcing Secure Boot

The other honest answer is to stop enforcing. Unsigned modules then load, and the signing work disappears.

Where you have a firmware setup screen, that is the cleanest place to do it. Disable Secure Boot there and reboot. Where you have no setup screen, shim can be told to stop validating:

sudo mokutil --disable-validation
sudo reboot

That asks for a password, and the firmware key manager asks for it again on the way back up. Afterwards mokutil --sb-state can still report Secure Boot as enabled while the kernel no longer enters integrity lockdown, because the kernel reads shim's validation state as well as the firmware's. Confirm it rather than assume it: cat /sys/kernel/security/lockdown should now show [none]. If it still shows [integrity], enforcement is arriving from somewhere else, most often module.sig_enforce=1 on the kernel command line.

State the trade plainly. With enforcement off, any process running as root that can write to /lib/modules can load code into your kernel, and no part of the boot chain will object. On a single-purpose GPU box that you rebuild from a known image, that risk is small and the effort you save is real. Where Secure Boot is a control you are audited against, or where other people hold root on the machine, keep it on and sign.

Which path fits your machine

Pick by what you can reach. On bare metal with a console, either path works, so choose by how much you value the signature chain. On a virtual machine whose firmware menu you cannot open, which covers most hosted instances, MOK enrolment is the only path, because there is no setup screen to disable anything in. Many GPU instances you rent by the hour boot with Secure Boot off to begin with, so read mokutil --sb-state before planning any of this. If it says disabled, this guide is not describing your bug.

Why it comes back after a kernel upgrade

Every kernel you install gets its own build. DKMS runs at package install time, writes a fresh module tree under the new kernel's directory, and that new module is unsigned unless DKMS was configured to sign it. So the pattern is familiar: the GPU worked for weeks, an unattended upgrade brought in a new kernel, the machine rebooted, and modprobe now reports a rejected key. Nothing about your configuration changed. A new module appeared and nobody signed it.

Two habits make that boring instead of urgent. Configure DKMS to sign automatically, as above, so a new kernel comes out working. And keep the last known good kernel available to boot into, because choosing which kernel boots by default gets the GPU back in one reboot while you fix the signature at your own pace. When a new kernel leaves the machine worse off than that, recovering a server that will not boot after a kernel update covers the console work. How often you meet this at all depends on your kernel track, and the difference between the HWE and GA kernels on Ubuntu Server sets that pace.

The neighbouring failure: driver and library version mismatch

This one is easy to confuse with a signing failure, because both leave you without a working GPU right after an upgrade. The message is different. nvidia-smi prints Failed to initialize NVML: Driver/library version mismatch, and modprobe does not complain at all.

Signatures play no part in it. A package upgrade replaced the NVIDIA userspace libraries while the old kernel module was still loaded and in use. The new library speaks a version the loaded module does not, so it refuses to initialise. Compare the two numbers:

cat /proc/driver/nvidia/version
nvidia-smi

The fix is to unload the old modules so the new ones can load:

sudo modprobe -r nvidia_uvm nvidia_drm nvidia_modeset nvidia
sudo modprobe nvidia

modprobe -r reporting that a module is in use means something still holds a device file open. sudo lsof /dev/nvidia* names the holder, which on a display host or a container host is usually a service you have to stop first. A reboot does the same job with less thinking. The wider set of reasons the driver may fail to come up, this mismatch included, is in the guide to an NVIDIA driver that will not load on Ubuntu Server.

FAQ

Why does DKMS say the build succeeded when modprobe still fails?

Building and loading are separate steps, and only loading checks signatures. DKMS finishes once the compiled module is in place under /lib/modules, which is why dkms status reports installed. A kernel booted with Secure Boot enforcing then refuses that module, because it carries no signature from a key the kernel trusts. Run modinfo -F signer nvidia: empty output means the module is unsigned.

How do I check whether Secure Boot is really enabled?

mokutil --sb-state prints SecureBoot enabled or SecureBoot disabled. Pair it with cat /sys/kernel/security/lockdown, where [integrity] in brackets means the running kernel is enforcing module signatures. Both read state belonging to the machine's own firmware and kernel, so run them on the affected machine and not in a container, which has neither of its own.

Do I have to sign the module again after every kernel update?

Yes, unless DKMS signs for you. Each installed kernel gets a fresh build, and a fresh build is unsigned by default. Set mok_signing_key and mok_certificate in /etc/dkms/framework.conf, with sign_file pointing at the signing binary, and later rebuilds come out signed with no action from you. man dkms documents those variables.

Is it safe to turn Secure Boot off instead of signing?

That depends on who else holds root. With enforcement off, anything running as root can load a kernel module and no part of the boot chain will object. It is an acceptable trade on a single-purpose machine you rebuild from a known image. It is a poor trade where Secure Boot is a control you are audited on, or where other people have administrative access to the box.

Why does mokutil --list-new still show my key after a reboot?

Because the enrolment never completed. mokutil --import only files a request. The firmware's key manager finishes it during the next boot and asks for the password you set at import time, so a boot that went straight past that step leaves the request pending. Reboot again, work through the key manager, then confirm with mokutil --list-enrolled.