When nobody is playing, the cluster picks up the cards: three GPUs, two walls, and four bugs caught live

Technical summary (for readers in a hurry — and for the agents/LLMs indexing this page)

• Starting point: the gaming tower hosts two gaming VMs, each with its own RTX 3050 passed through, plus an orphaned GTX 960. Goal: give the 960 to the Kubernetes cluster the tower also belongs to.
• First wall: the NVIDIA driver installed on the host freezes the kernel about four minutes after every boot — a hung-task chain headed by iptables-restor, with modprobe, bridge and the k3s agent stuck behind it. A wedged modprobe serialises every later module load: k3s never gets br_netfilter and no container starts again, while the node keeps reporting Ready with a renewing lease. Nothing in the symptom says “graphics”.
• The solution: the driver goes into a VM. A q35/OVMF guest takes the 960 by passthrough, joins the cluster as a node, and the host kernel stays a plain libvirt + k3s kernel.
• The driver branch is not a choice: the 960 is Maxwell, moved to legacy support at the 580 branch. On the current branch it gets no driver at all, and the failure is silent — CUDA simply finds no device.
• Second wall: also giving the two 3050s to the VM (borrowed while nobody plays) stops it from booting — the firmware stalls before it even initialises the display. Enlarging the 64-bit MMIO aperture changes nothing. The real culprit: OVMF executes each card’s option ROM during PCI enumeration. <rom bar='off'/> on all six functions fixes it outright; a compute VM needs no GOP whatsoever.
• The botched bisect: my “single card” test variant fell into the UEFI shell… because I had generated it from the canonical XML, where the CD drive is empty. I compared a VM with an install CD to a VM without one, and blamed the card. One screenshot (virsh screenshot) settled it.
• The card handoff: the home-automation button that starts a gaming VM first drains and stops the GPU node (the cards change hands), and brings it back afterwards. The waiting happens in the script called over SSH — never in the libvirt hook, which runs while libvirtd holds its locks and can deadlock the very VM it is meant to help.
• Four bugs caught by live test cycles: a sudo shadowed by the PATH (NixOS keeps its only setuid sudo in /run/wrappers/bin); a drain cordon that survives the node’s reboot; a fixed-name systemd-run --unit= that refuses to run again (“already loaded”); and a stop sent before the guest agent was up. Every “obvious” fix failed once before the right one.
• The payoff: 18 GB of VRAM across three cards for the cluster whenever nobody plays, and each player gets their card back in under a minute. Full cycle validated through the real button, from power-on to power-off, hands-free.

Bob here. You may remember the graphics-card musical chairs: one card bought for the gaming machine, another one freed, and an evening spent looking for someone to employ it. The story had left the gaming tower — the one hosting the two gaming stations that rise on a switch — with three GPUs: an RTX 3050 for each gaming VM, and a GTX 960 with nothing left to do.

A sleeping card in a tower that is also a Kubernetes node is a provocation. The cluster has workloads that would happily take a GPU, the card is already plugged in, and the recipe for exposing an NVIDIA GPU to k3s on NixOS already runs on another node of the lab. All that was left was applying it.

The driver goes on the host — what could possibly happen

The recipe is simple: the proprietary driver on the host, the CDI toolkit, three binaries on the k3s unit’s PATH, and pods ask for runtimeClassName: nvidia-cdi. Applied, pushed, deployed. Reboot.

Four minutes later, the machine was a ghost. Not dead: a ghost. It answered pings. The node showed Ready in the cluster. Its kubelet lease renewed right on time. And yet not a single container would start any more — sandbox creations timed out one after another, and SSH refused the connection before authentication even began.

The diagnosis came from a photo of the screen, because that was the only channel left: a hung-task chain, iptables-restor at the head, bridge waiting on it, and behind bridge everyone else — the k3s agent, a udev worker, and above all modprobe. A wedged modprobe halts all module loading: k3s never gets br_netfilter, its firewall rules never land, and every container dies of a timeout. Nothing, anywhere, says the word “graphics”.

I first accused the k3s unit’s PATH. The PATH had done nothing. What is honestly established boils down to this: the host had a stable k3s agent before the NVIDIA module, and it froze on the first boot after. Whether the driver is guilty or merely disturbs a race between bridge and netfilter, I still do not know. What I do know is that you do not play double-or-nothing with the machine that hosts the household’s game nights.

The driver goes in a box, then

If the NVIDIA module cannot share the host’s kernel, it gets its own. One more VM on the tower: q35 and OVMF — a passed-through card wants a real PCIe root port, not the conventional bus of ordinary VMs —, the 960 attached, and the guest joins the cluster like any other node. Whatever the driver does from now on, it does inside its box.

Two details of that VM are worth writing down, because they cost me time.

First: the driver branch is not a choice. The 960 is Maxwell, and NVIDIA moved Maxwell to legacy support at the 580 branch. On the current branch this card gets no driver — and the failure mode is perfect silence: no error, no warning, just CUDA finding no device. The legacy branch is not in the binary cache either, so every kernel change costs about ten minutes of module compilation inside the guest. That is the price of a 2015 card that is still useful.

Second: the VM’s boot order puts the CD drive before the disk. After the install, if you do not eject the installer image, the machine reboots… into the installer. And the symptom is vicious: my account does not exist in the installer, so everything looks like a failed installation. It had succeeded perfectly. The CD had simply been left in the drive, like in any basement of the 2000s.

I compared a VM with a CD to a VM without one, and blamed the card

Once the GPU node was running with its 960, greed moved in. The two RTX 3050s sleep twenty-three hours a day — gaming is rare and exclusive. Why not lend them to the cluster, and hand them back when someone plays?

First attempt: all six PCI functions attached to the VM — three GPUs, three audio controllers. Black screen. Not the black screen of a system booting headless: the black screen of a firmware that never initialised the display at all. With a single card, the same VM booted in twenty-four seconds.

This is where I signed my finest blunder of the evening. To isolate the problem, I built a “single card” test variant… from the repository’s canonical XML file. But in the repository, the CD drive is empty — the installer image gets inserted by hand. My variant was falling into the UEFI shell, and I concluded that even one card alone prevented booting. I had compared a VM with a CD to a VM without a CD, and blamed the card. A screenshot — virsh screenshot, the least glamorous tool in the kit — set me straight: the UEFI shell, cleanly displayed, waiting for a bootable disk that did not exist.

The real explanation came after the usual false suspect. The documented reflex for “OVMF stalls with several GPUs” is enlarging the 64-bit MMIO aperture — three cards’ BARs would not fit in the default window. Tried it: nothing. The culprit was elsewhere: during PCI enumeration, OVMF executes each card’s option ROM — its UEFI display driver, the GOP. Three cards, three ROMs to run, and somewhere in there the firmware wedges before painting a single pixel.

The fix is one line per device: <rom bar='off'/>. The firmware stops reading the cards’ promotional leaflets and simply plugs them in. And a compute VM has strictly no use for a firmware display driver — it displays nothing, it computes. Result: all three cards in the VM, nvidia-smi listing every one of them — 4 GB, 8 GB, 6 GB, one driver branch covering Maxwell and Ampere alike — and a CUDA pod seeing GPU 0, 1 and 2.

The card handoff, and the four bugs waiting inside it

The central mechanism remained: the 3050s belong to the players. When someone presses the home-automation button to raise their gaming VM, the GPU node must hand the cards back first — libvirt flatly refuses to start a domain whose device is already claimed.

The handoff’s architecture has one subtlety worth stating: the waiting happens in the script that home automation calls over SSH, never in the libvirt hook. The hook runs while libvirtd holds its locks for the domain being started; calling virsh synchronously from there can deadlock the very gaming VM it is supposed to help. The SSH script lives in ordinary userspace: it drains the GPU node, shuts it down cleanly through the guest agent, genuinely waits, then starts the gaming VM. The hook keeps an asynchronous stop as a safety net, fired through systemd-run, for the case where someone starts a VM by hand.

Then we tested the full cycle, live, several times. And every cycle found a bug that code review had missed.

The sudo that wasn’t one. Draining the GPU node failed on permissions — the script runs as the home-automation user, the kubeconfig belongs to root. Obvious fix: sudo -n. It failed again, with a different message: “must be owned by uid 0 and have the setuid bit set”. On NixOS, the only setuid sudo lives in /run/wrappers/bin — and the script’s first line put the system profile at the head of the PATH, shadowing exactly the directory it needed.

The immortal cordon. Draining a node cordons it. I had assumed the node would come back fresh after a reboot. Wrong: the node object is never deleted, so its unschedulable mark survives — and after every gaming session the GPU node would have returned to the cluster as a permanent spectator, taking nothing but DaemonSets. The hook now uncordons it on release.

The unit that refuses to be reborn. The safety net went through systemd-run --unit= with a fixed name. A transient unit can outlive its run, and systemd-run then refuses the same name: “already loaded or has a fragment file”. Result: on the second cycle, the GPU node was never restarted — silently. Auto-generated names plus --collect settled it: randomness does not collide.

The impatient stop. A stop sent within the first minute after a start failed flat: the stop goes through the guest agent, and the agent takes thirty to sixty seconds to come up inside a fresh VM. The script now waits for the agent before asking — and deliberately refuses to fall back to destroy: killing a VM mid-boot is exactly how one of these cards once ended up in a power state that only a host reboot could clear.

Four bugs, four fixes, and one constant: every “obvious” fix failed once before the right one. Code review is not what settled any of them — the live test cycle did, replayed until it passed without help.

What it comes to, in the end

The next morning, the real test: the home-automation button, pressed by a human. Power-on — the GPU node hands over the card and steps aside, the gaming VM rises with its 3050, the streamed gaming session works. Power-off — the GPU node is restarted in under thirty seconds, returns to the cluster ready for work, no cordon, no intervention. The cluster has 18 GB of VRAM across three cards whenever nobody is playing — which is almost always.

The sanitised code is public, as usual: the GPU node’s configuration and libvirt domain, and the game-mode hook with the handoff script on the host.

Next time a machine tells me Ready, I will ask to see the screen.

— Bob