Tendril: turning one multi-GPU machine into several gaming PCs

A PC with several GPUs is still one PC: one Windows install, one keyboard, one player. The reliable way to split it on consumer hardware is GPU passthrough, which hands each card to its own virtual machine. Doing that by hand means reading IOMMU tables, rebinding drivers and writing libvirt XML. Tendril does all of it for you. It’s an installable Fedora bootc appliance, about 16,000 lines of Rust in four crates, that turns one multi-GPU machine into several gaming stations. Each station is a Windows or SteamOS VM with a real GPU, managed from a web UI.

What it does #

You flash the installer ISO, boot the machine and open the web UI from another device. The only thing Tendril can’t do for you is turn on VT-d or AMD-Vi in the firmware.

To create a station you pick an OS, a GPU and an account name. Tendril downloads and verifies the install media, builds the disk, an answer file and the VM, and installs the guest while you watch in an in-browser console. When it finishes, the monitor plugged into that GPU is a gaming PC.

After that there are snapshots, golden images that clone instantly, USB “seats” for each player’s peripherals, Sunshine streaming, a kiosk mode and a low-latency mode that pins vCPUs to host cores.

How it works #

The workspace is four crates, and dependencies only point one way:

• capability-engine reads sysfs, the kernel’s view of the hardware, and decides what each GPU can do.
• provisioning plans which PCI addresses to hand to vfio-pci, the kernel’s passthrough driver.
• orchestrator renders libvirt domain XML, writes answer files and drives VMs through virsh.
• web is the Axum and HTMX control plane. It embeds htmx and noVNC in the binary so the appliance works offline.

The first three crates have no third-party dependencies. They shell out to virsh, qemu-img and genisoimage instead of linking libvirt. The sysfs readers can point at fixture directories, so the IOMMU logic is tested without a live machine. The web UI, the on-screen console and the CLIs all call one provision() function, so a station comes out the same from any front end.

Hard problem 1: which GPU can you give away? #

You can’t pass through “a GPU”. What the hardware can isolate is an IOMMU group: the set of PCI devices that the IOMMU, the hardware that fences off each device’s memory access, can’t separate. A modern graphics card is several devices. The RTX 2070 SUPER I tested against shows up as four functions, 0000:83:00.0 through .3: the GPU plus its audio and USB companions, all in one group. They move together or not at all, so the plan always covers the whole group.

Detection asks whether everything in a card’s group sits in the same physical slot:

pub fn viability_for(address: &str, groups: &[IommuGroup]) -> PassthroughViability {
    let Some(group) = group_of(address, groups) else {
        return PassthroughViability::NoIommu;
    };
    let slot = slot_of(address);
    let isolated = group.device_addresses.iter().all(|a| slot_of(a) == slot);
    if isolated {
        PassthroughViability::Isolated
    } else {
        PassthroughViability::SharedGroup
    }
}

An isolated group means clean passthrough. A shared group needs an ACS override, a kernel patch that splits the group anyway at a security cost, so Tendril labels the card and leaves that to you. No group means the IOMMU is off in the firmware, and a boot-time check puts that on the Hardware page.

A hardening pass added a rule: the GPU the firmware booted on (boot_vga=1 in sysfs) always stays with the host. Before that, a single-GPU box would offer its only display for passthrough.

Binding takes three sysfs writes per device: unbind the current driver, set driver_override to vfio-pci, then reprobe. tendril-apply defaults to a dry run that prints the writes. A half-detached GPU can’t be cleanly put back, so a failure partway through reports how many writes already landed and says to re-run or reboot.

Hard problem 2: installing Windows with nobody at the keyboard #

Both guest OSes resist being installed with nobody watching.

A stock Windows 11 ISO can’t see a virtio disk, the paravirtual disk VMs use, so Setup stops at “we couldn’t find any drives”. Tendril generates an autounattend.xml that loads the virtio drivers during WinPE, the first stage of setup. The virtio ISO’s drive letter isn’t predictable there, so the file lists every driver under D:, E: and F:, and Setup skips paths that don’t exist. The same file skips the Microsoft-account screens and creates a local account. It rides in on a small seed ISO, attached as a third CD drive.

Then there’s “Press any key to boot from CD or DVD”, which skips the installer if nobody presses a key within about five seconds. So Tendril sends Enter through virsh send-key once a second for 45 seconds, because firmware startup can be slow on a loaded host. Extra keypresses land harmlessly in WinPE.

SteamOS has no installer for generic PCs. The Steam Deck recovery image is AMD-only, so it can’t drive an NVIDIA station. Tendril’s “SteamOS” station is Bazzite instead. Its Anaconda installer automatically loads a kickstart, Anaconda’s answer file, from a seed ISO labelled OEMDRV.

Mistakes in these files show up late. An empty username produced an empty <Name> element that Windows Setup rejected hours into an install. pykickstart splits lines the way shlex does, so a password ending in a backslash aborted the SteamOS install. Both are fixed, and the two generators are pure functions with 26 unit tests between them.

Hard problem 3: an appliance that can’t brick itself #

Tendril’s users shouldn’t need to know what a kernel argument is. So the host is a Fedora bootc image: the whole OS ships as a container image, an update is staged as a new deployment, and the old one stays on disk. greenboot runs a required health check after each boot, and if it keeps failing, the bootloader goes back to the previous image.

The original design made every host change an image layer, with a first boot that built a custom image for the detected GPUs. What shipped is simpler. The base image carries only what every machine needs:

kargs = [
    "intel_iommu=on",
    "amd_iommu=on",
    "iommu=pt",
    "rd.driver.pre=vfio-pci",
]

GPU binding happens at runtime instead: each GPU device in the domain XML is marked managed='yes', so libvirt moves the group to vfio-pci when the VM starts. Image layers are kept for the case that needs them. A vGPU host driver is a kernel module, so Tendril builds a derived image on the box against the running image’s kernel, and the two can’t drift apart.

The instructive bug was in the health check. It required the UI to answer HTTPS on port 443, but the docs also describe running behind a reverse proxy with TENDRIL_TLS=off, which serves plain HTTP there. On such a box the check failed on every boot, and greenboot rolled back every OS update. Nothing crashed. The box just never moved forward. An audit pass caught it in 0.23, and now the check tries both:

for _ in $(seq 1 30); do
  if curl -ksS -o /dev/null --max-time 3 https://127.0.0.1/ \
     || curl -sS -o /dev/null --max-time 3 http://127.0.0.1:443/; then
    echo "greenboot(tendril): healthy"
    exit 0
  fi
  sleep 2
done

A required health check gates every future update, and when it fails wrongly it fails quietly. It has to be tested against every supported setup, not just the default. The network page has a similar net: each change is a 60-second trial that reverts unless you keep it.

Status #

Tendril is at 0.24.0, pre-1.0 and licensed AGPL-3.0. The changelog lists 24 releases, dated July 4 to July 9, 2026; later commits are release plumbing and docs. The code is four crates, eight binaries and 92 unit tests on a Fedora bootc 42 base. Images ship on three channels: dev, per-release, and stable, which points at an already-released digest instead of rebuilding. Release images are signed with cosign.

The limits:

• Hardware validation is partial. Detection and bind planning ran live on the RTX 2070 SUPER, and the unattended Windows 11 and Bazzite installs were verified under nested KVM. vGPU, fleets, PXE provisioning and remote play are code-complete but untested on a real multi-GPU rig.
• GPU reset quirks aren’t handled. The design doc lists the AMD reset bug, which can stop a VM from restarting, as a risk; the code doesn’t handle it yet.
• PCIe bridges count as foreign devices, so a card can be reported as sharing a group when it doesn’t.
• Kernel-level anti-cheat won’t run, even with the opt-in overlay that hides KVM.
• Multi-machine is federation, not clustering: no consensus and no live migration.

What I’d do differently #

The original design had a custom first-boot image and a controller/agent cluster with an elected leader. Both were dropped. libvirt’s managed devices made the custom image unnecessary, and the cluster fell to two facts: a station owns a physical GPU, so it can’t live-migrate, and a fleet that never acts on its own doesn’t need consensus. Next time I’d write the constraints down first.

I’d also think harder about shelling out. Dependency-free crates were worth it, but parsing command output has a cost: virsh domifaddr was read by fixed column, so a Windows adapter named “Ethernet 2” hid the station’s IP. Next up is running the hardware checklist on a real multi-GPU box, starting with PXE.

Links #

The repository is private for now. The source, the docs (FEDERATION.md on the fleet design, HARDWARE-TESTING.md, the tester checklist, and PLAN.md, the original design) and the installer ISO will be linked here when it goes public. Until then, email me for a walkthrough.