uki-secureboot-tooling › NVRM API mismatch

NVRM: API mismatch after an NVIDIA update, and the PC boots to a TTY with Secure Boot

You updated the NVIDIA driver, rebooted, and got a text console instead of a login screen. The kernel log says NVRM: API mismatch, and nothing changed except the driver. If you boot a signed Unified Kernel Image (UKI), the likely cause is that the UKI still carries the old NVIDIA module. This page shows how to confirm that, fix it by hand on Debian/Ubuntu/Mint, Fedora and Arch, and stop it happening again.

What you are seeing

The NVIDIA driver has two halves that must be the same version: the kernel module (nvidia.ko and friends) and the userspace libraries that Xorg, Wayland compositors and nvidia-smi load. When the userspace side talks to a kernel module of a different version, the module logs a line that starts with NVRM: API mismatch and names both versions: the client (userspace) version and the kernel module version. The display server fails to start and the display manager gives up, which leaves you at a TTY.

The usual cause is not having rebooted after an update, so the old module is still in memory. If you did reboot and it still happens, and your system boots a signed UKI, read on.

Why a signed UKI keeps the old driver

A UKI is one EFI binary that contains the kernel, the initramfs and the kernel command line, signed as one unit. Secure Boot verifies the whole file, which is the point: nobody can swap the initramfs or edit the command line. It also means the initramfs inside the UKI changes only when someone rebuilds and re-signs the UKI.

A driver-only update, such as an nvidia-dkms version bump, does this:

On Debian, Ubuntu and Mint, a typical UKI builder is a script in /etc/kernel/postinst.d/. Those scripts run when a kernel package is installed. A driver update installs no kernel package, so the builder never runs. The UKI on the ESP still contains the initramfs from the last kernel install, with the old NVIDIA module in it.

At boot, if the NVIDIA modules are in the initramfs (early loading, often set up so the driver is ready before the display manager starts), the kernel loads the old module from inside the UKI. That module stays loaded when the root filesystem is mounted. Userspace is the new version. The result is NVRM: API mismatch and a TTY.

The UKI is still validly signed, so Secure Boot does not complain and nothing tells you it is stale.

Confirm it on your machine

Log in on the TTY. First compare the version of the module that is loaded with the version of the module on disk for the running kernel:

uname -r
cat /sys/module/nvidia/version
modinfo -F version nvidia
sudo dkms status

/sys/module/nvidia/version is the module in memory. modinfo searches /lib/modules for the running kernel and reports the file on disk. dkms status lists what DKMS built and for which kernel. If the loaded version is older than the on-disk version and you have rebooted since the update, the old module came from somewhere other than /lib/modules. On a UKI system, that is the initramfs inside the UKI.

Check the kernel log for the mismatch itself:

sudo journalctl -k | grep NVRM

Next, find the UKI you booted. bootctl list shows every entry systemd-boot can see, with a source: path for each:

sudo bootctl list

Set UKI to the source path of the entry for your running kernel, for example UKI=/boot/efi/EFI/Linux/linuxmint-$(uname -r).efi. Then compare the UKI's build time with the module file's time:

sudo stat -c '%y  %n' "$UKI" "$(modinfo -F filename nvidia)"
sudo ukify inspect "$UKI" --section=.uname:text

If the UKI is older than the module, it was built before the driver update. ukify inspect prints the embedded .uname section, which should match uname -r; it confirms you are looking at the right file.

Optionally, check that the NVIDIA modules really are in the UKI's initramfs. Recent ukify can save a section to a file:

sudo ukify inspect "$UKI" --section=.initrd:binary@/tmp/uki-initrd.img

Then list it with your initramfs tool: lsinitrd /tmp/uki-initrd.img | grep nvidia on dracut systems (Fedora), or lsinitcpio /tmp/uki-initrd.img | grep nvidia on mkinitcpio systems (Arch). Debian-family systems have an equivalent lister in initramfs-tools. If the lister cannot read the file, skip this step; the version and timestamp checks are enough. If NVIDIA is not in the initramfs at all, a stale UKI cannot be the cause, and you should look for a driver package that did not finish installing.

Last, sbctl:

sudo sbctl status
sudo sbctl verify

Expect the stale UKI to show as signed. sbctl verify checks that files carry a valid signature from your db key; it cannot tell you whether the contents are current. You use it after the rebuild to make sure the new UKI is signed before you reboot.

Fix it by hand

The fix is the same on every distribution: make sure the module is built for your kernel, regenerate the initramfs, rebuild the UKI from it, and sign the new UKI with your db key. Only the tools differ.

First, keep a way back

Before you overwrite anything, make sure you can boot if the new UKI is bad. Two options:

• A BLS fallback entry. If you kept Type #1 entries (*-fallback.conf in loader/entries/) that boot a loose vmlinuz with the standalone initramfs, pick one at the systemd-boot menu. The driver update already regenerated that initramfs, so it carries the new module. Under Secure Boot the vmlinuz must be signed; the initramfs and command line in such an entry are not covered by Secure Boot, so treat it as a rescue path only.
• A copy of the current UKI. A copy keeps its signature, and systemd-boot lists every .efi in EFI/Linux/ as an entry. It still boots to the TTY, but it boots. Check free space first with df -h "$(dirname "$UKI")", because each UKI can be a few hundred megabytes.

sudo cp "$UKI" "${UKI%.efi}-previous.efi"

Debian, Ubuntu and Linux Mint (initramfs-tools + ukify)

KVER=$(uname -r)
sudo dkms status
sudo update-initramfs -u -k "$KVER"

If dkms status does not show the new NVIDIA version as installed for $KVER, fix that first; rebuilding the UKI from an initramfs without the module helps nobody. If you installed the guard hook described below, update-initramfs has already rebuilt and re-signed the UKI. Otherwise, build it from the fresh initramfs and sign it onto the ESP:

sudo ukify build --linux "/boot/vmlinuz-$KVER" --initrd "/boot/initrd.img-$KVER" --cmdline @/etc/kernel/cmdline --uname "$KVER" --output /tmp/uki-new.efi
sudo sbctl sign -o "$UKI" /tmp/uki-new.efi
sudo sbctl verify "$UKI"

Building in /tmp and letting sbctl sign -o write the signed result to the final path means the ESP never holds an unsigned or half-written UKI. /etc/kernel/cmdline must hold your full command line on one line, without initrd= or BOOT_IMAGE=; editing that file changes nothing until the UKI is rebuilt.

Fedora (dracut + ukify)

KVER=$(uname -r)
sudo dracut --force "/boot/initramfs-$KVER.img" "$KVER"
sudo ukify build --linux "/lib/modules/$KVER/vmlinuz" --initrd "/boot/initramfs-$KVER.img" --cmdline @/etc/kernel/cmdline --uname "$KVER" --output /tmp/uki-new.efi
sudo sbctl sign -o "$UKI" /tmp/uki-new.efi
sudo sbctl verify "$UKI"

Make sure the kernel module for $KVER is built before you run dracut. If you build UKIs with dracut itself (uefi="yes" in a dracut.conf.d file) rather than with ukify, dracut writes the UKI directly; sign that output with sbctl in the same way.

Arch, Manjaro, EndeavourOS (mkinitcpio)

sudo dkms status
sudo mkinitcpio -P
sudo sbctl sign-all
sudo sbctl verify

On Arch the UKI is built by mkinitcpio itself from the default_uki= line in each preset under /etc/mkinitcpio.d/, so mkinitcpio -P regenerates the initramfs and the UKI together. sbctl ships a mkinitcpio post hook that signs the result; sbctl sign-all re-signs every file in sbctl's database and is harmless if they are already signed. If your presets also keep a default_image= line, the standalone initramfs is rebuilt too, which is what a BLS fallback entry needs.

Register the path, then reboot

Check that sbctl tracks the UKI path, so future sbctl sign-all runs and package hooks re-sign it:

sudo sbctl list-files

If your UKI path is not listed, sign it in place with -s so the final path is saved to the database. sbctl records a path when it actually signs the file, so give it the unsigned build at the final path:

sudo cp /tmp/uki-new.efi "$UKI"
sudo sbctl sign -s "$UKI"

Do not use sbctl sign -s -o "$UKI" /tmp/uki-new.efi for this. It saves the input path in /tmp to the database, which is useless after a reboot.

Reboot into the new UKI. Then cat /sys/module/nvidia/version and modinfo -F version nvidia should print the same version, and your desktop should be back. Once it is, delete the -previous.efi copy.

Stop it happening again

The gap is a trigger problem: the UKI builder fires on kernel installs, but the initramfs changes on driver installs too. The fix is to rebuild the UKI every time the initramfs is regenerated.

initramfs-tools (Debian, Ubuntu, Mint): a post-update.d guard hook

update-initramfs runs every script in /etc/initramfs/post-update.d/ each time it writes an initramfs, passing the kernel version as $1 and the new initramfs path as $2. One working design, the zz-uki-rebuild hook, works as follows:

1. It exits quietly if no kernel version was passed.
2. It exits if DPKG_MAINTSCRIPT_PACKAGE (set by dpkg for maintainer scripts) matches linux-image-*, linux-image or kfreebsd-image-*. On a kernel-package install, the /etc/kernel/postinst.d/zzz-uki-build builder already builds and signs the UKI as the last postinst.d step. Skipping here avoids a double build, and avoids deleting the loose vmlinuz out from under later postinst.d scripts in the same run.
3. It exits if the builder is not executable.
4. The builder needs the kernel image as its second argument. The hook uses /boot/vmlinuz-$KVER if present. Otherwise it copies it back from /root/kernel-store/$KVER/vmlinuz, an archive the builder keeps on the LUKS-encrypted root because this setup deletes loose kernels after each build. If neither exists, it logs and skips.
5. It runs the builder with the kernel version and image. The builder reads the initramfs that update-initramfs just wrote and /etc/kernel/cmdline, builds an unsigned UKI in /tmp, signs it onto the ESP with sbctl sign -o, then runs sbctl sign -s on the final path so that path lands in sbctl's database. The builder never calls update-initramfs, so the hook cannot recurse.
6. It is best-effort. It must never abort the driver package's maintainer script, so every failure is swallowed and it always exits 0. A failed rebuild prints a warning that the UKI may be stale.

The core of it:

#!/bin/sh
set -u

KVER="${1:-}"
[ -n "$KVER" ] || exit 0

case "${DPKG_MAINTSCRIPT_PACKAGE:-}" in
    linux-image-*|linux-image|kfreebsd-image-*) exit 0 ;;
esac

BUILDER=/etc/kernel/postinst.d/zzz-uki-build
[ -x "$BUILDER" ] || exit 0

KIMG="/boot/vmlinuz-${KVER}"
if [ ! -f "$KIMG" ]; then
    ARCHIVED="/root/kernel-store/${KVER}/vmlinuz"
    if [ -f "$ARCHIVED" ]; then
        cp -f "$ARCHIVED" "$KIMG" || exit 0
    else
        echo "zz-uki-rebuild: no vmlinuz for $KVER (neither loose nor archived) -- skipping UKI refresh" >&2
        exit 0
    fi
fi

echo "zz-uki-rebuild: refreshing signed UKI for $KVER after initrd regeneration"
if ! "$BUILDER" "$KVER" "$KIMG"; then
    echo "zz-uki-rebuild: UKI refresh for $KVER failed (non-fatal); UKI may be stale" >&2
fi

exit 0

Two details carry weight. The builder in postinst.d is named zzz-uki-build because run-parts runs scripts in name order, and the builder must run after initramfs-tools and zz-systemd-boot have written the initramfs. And the builder refuses to copy an unsigned UKI to /boot if sbctl is missing.

Arch: UKI follows the initramfs, modules get signed by pacman hooks

Because mkinitcpio builds the UKI, every initramfs regeneration also rebuilds the UKI, and sbctl's pacman hook re-signs the files in its database after each transaction. If pacman did not regenerate the images after a DKMS update on your system, run mkinitcpio -P as above.

The other Arch problem is module signatures. DKMS-built modules fail to load if the kernel enforces signed modules. For kernels built with your own key in CONFIG_SYSTEM_TRUSTED_KEYS, two pacman hooks keep the modules signed:

• 80-sign-nvidia-modules.hook fires after installs and upgrades of linux*-nvidia, nvidia, nvidia-dkms and nvidia-utils, and on changes to usr/lib/modules/*/extramodules/nvidia*.ko*. It runs a signer that decompresses each .ko.zst in the running kernel's extramodules directory, signs it with the kernel's scripts/sign-file (sha512), recompresses it, and skips modules already signed with that key.
• 81-sign-all-dkms-modules.hook is the broad net. It fires on any .ko* under extramodules/ or updates/dkms/, and on kernel package changes. For every installed kernel it signs .ko, .ko.zst, .ko.gz and .ko.xz modules in both directories, re-runs depmod, and exits 0 if the key is missing so the pacman transaction never fails.

Fedora: no initramfs-only trigger yet

A common way to build UKIs on Fedora is a kernel-install plugin in /etc/kernel/install.d/. It runs on kernel-install add and remove, which means on kernel installs and removals, not on a driver-only initramfs rebuild. Until you have a trigger that fires on initramfs regeneration, re-run the Fedora steps above after every NVIDIA driver update, before you reboot.

FAQ

Sources

• ArchWiki, Unified kernel image: mkinitcpio presets with default_uki=, mkinitcpio -P, sbctl's mkinitcpio post hook and pacman hooks.
• ArchWiki, Secure Boot: sbctl verify, sbctl sign -s, the sbctl pacman hook.
• ArchWiki, Dynamic Kernel Module Support: dkms status, unsigned DKMS modules under enforced module signing.
• ArchWiki, NVIDIA: early loading of the NVIDIA modules from the initramfs.
• ArchWiki, mkinitcpio: presets, -P, post hooks, lsinitcpio.
• sbctl README and sbctl(8): sign (-o, -s), sign-all, verify, list-files.
• ukify(1): build options and inspect --section=NAME:text|binary[@PATH].
• kernel-install(8), bootctl(1), journalctl(1).
• man7.org: modinfo(8) (-F version, -F filename), dracut(8), lsinitrd(1), dpkg(1) (DPKG_MAINTSCRIPT_PACKAGE).