Host configuration and WSL 2

Almost everything vmlab needs is in the lab file. What is left is per-host: where it may allocate subnets from, which VNC viewer to launch, how big a chunk to push to a registry, where the config-weave binaries are, how large a file the workspace syncer carries, which UEFI firmware a VM boots. Those live in one optional host configuration file. This chapter explains that file, the directories vmlab uses, what changes when the host is WSL 2, and how vmlab picks between KVM and TCG. Every field is in Host configuration file and every path in Files and directories.

The host config file

vmlab reads ~/.config/vmlab/config.wcl, honouring XDG_CONFIG_HOME. The file is optional and every field in it is an override, so an absent file means all defaults. It must start with import <vmlab-host.wcl> and contain one host { … } block. A malformed value is reported at its line with the same wording a lab file gets, and every mistake in the file is reported in one pass. The errors surface where a lab is loaded, so a broken file fails vmlab up with its lines named; the daemon processes themselves fall back to defaults for a file they cannot read.

import <vmlab-host.wcl>

host {
  subnet_pool        = "10.99.0.0/16"
  dns_suffix         = "lab.local"
  disk_low_percent   = 5
  viewer             = "remote-viewer vnc://{}"
  oci_chunk_size     = 128MiB
  workspace_max_file = 2GiB
}

config_weave_bin_dir is a location knob with one code path behind it rather than a switch: it is the first rung of a three-rung lookup, ahead of the VMLAB_CONFIG_WEAVE_DIR environment variable and the XDG default. workspace_max_file is host config rather than a @dev argument because the cap is about your link to your guest, not about the lab everyone shares, and the refusal message names it so "raise the cap" points somewhere.

Host GPUs

A lab's gpu {} never carries a PCI address: which GPU sits at which address is a fact about one machine, and the lab file travels. Each machine's host config names its own GPUs instead, beside host {}, and marks one the default. A lab names a host GPU with host_gpu, or takes the default. The same lab then runs on two machines with different cards:

import <vmlab-host.wcl>

gpu "AMD1"    { address = "0000:72:00.0" }
gpu "Nvidia1" { address = "0000:01:00.0"  default = true }
gpu "Nvidia2" { address = "0000:05:00.0" }

import <vmlab-host.wcl>

gpu "Nvidia1" { address = "0000:0a:00.0"  default = true }
gpu "Intel1"  { address = "0000:00:02.0" }

vm "gl01" { template = "x86_64/fedora-44" gpu { mode = "vulkan" } }
vm "gl02" { template = "x86_64/fedora-44" gpu { mode = "vulkan" host_gpu = "AMD1" } }

gl01 renders on Nvidia1 on both machines, at whichever address it has there. gl02 renders on AMD1 here; the second machine has no AMD1, so there it warns that AMD1 is not declared on this host and renders on the default, Nvidia1. A passthrough GPU never falls back like that, because handing the guest a different card takes it from the host: it refuses instead. Find a card's address with lspci -D, or under /sys/bus/pci/devices/. Labels match case-insensitively; a lone gpu block is the default without default = true. The fields are in the host configuration file reference.

A host GPU can also carry the QEMU its VMs run and extra variables for that QEMU's environment. This machine needs both for its NVIDIA cards. QEMU 11.1's egl-headless display binds a guest's blob scanout as GL_TEXTURE_2D, and NVIDIA's EGL accepts that LINEAR dma-buf only as GL_TEXTURE_EXTERNAL_OES. A Venus guest on Nvidia1 therefore renders, but its console and screenshots stay black. A patched QEMU fixes this. AMD and Intel accept the 2D binding, so AMD1 keeps the stock QEMU:

import <vmlab-host.wcl>

gpu "AMD1"    { address = "0000:72:00.0" }
gpu "Nvidia1" {
  address     = "0000:01:00.0"
  default     = true
  // egl-headless shows an external-only dma-buf scanout
  qemu_binary = "/opt/qemu-vmlab/bin/qemu-system-x86_64"
  qemu_env    = { VKR_DEBUG: "validate" }  // virglrenderer debugging; optional
}
gpu "Nvidia2" { address = "0000:05:00.0" }

Every VM that resolves to Nvidia1 runs that binary, in every GPU mode, and the lab file does not change. vmlab passes the binary -L with the stock QEMU's data directory, so a minimal build still finds its firmware and ROMs. qemu_env is applied after the Vulkan loader variable vmlab derives, and an explicit host setting wins. qemu.log's vmlab: env: line shows every variable vmlab set, and vmlab status -v shows qemu=<path> next to host_gpu=, then the same variables as env=NAME=value,..., recorded at the start, so a config edited later does not change what a running VM reports. vmlab status --json carries them as the VM's qemu_env object. Template builds never have a gpu {}, so they always run the stock QEMU.

Nvidia1 and Nvidia2 are identical cards, so the *nvidia* loader filter matches both. The Vulkan loader cannot filter by PCI address, and Mesa's device-select layer breaks Venus, so vmlab ships its own Vulkan layer for it. A vulkan VM's QEMU loads VK_LAYER_VMLAB_pci_filter, which hides every physical device but the one at the host GPU's address. A Venus guest on Nvidia2 therefore sees one device, the card at 0000:05:00.0, the same card egl-headless scans out from. The layer is built into vmlab, and vmlab writes it under ~/.local/share/vmlab/vk-layer/ the first time a VM needs it. qemu.log shows what it decided:

vmlab: env: VK_LOADER_DRIVERS_SELECT=*nvidia* VK_ADD_IMPLICIT_LAYER_PATH=/home/you/.local/share/vmlab/vk-layer/e7eb826ffa68b2aa VMLAB_VK_PCI_ADDRESS=0000:05:00.0
vmlab-pci-filter: 0000:05:00.0: kept NVIDIA GeForce RTX 5090, hid 1

To turn the layer off for one host GPU, set qemu_env = { VMLAB_VK_PCI_FILTER_DISABLE: "1" } on it. The guest then sees every card that matches the driver filter.

The viewer

vmlab console and gui = true launch a viewer chosen the same way. An explicit viewer in the host config wins and is dialled at the VNC unix socket directly. Otherwise vmlab takes the first of remote-viewer, gvncviewer and vncviewer found on PATH; all three are driven over a localhost TCP bridge to the socket, held open by a detached helper that exits when the viewer window closes, so neither command ties up the terminal. With no viewer at all, or with --tcp, vmlab console bridges the socket to a localhost port and prints the address for you to point any VNC client at. Closing a viewer only disconnects; the VM keeps running, always headless behind VNC. See vmlab console.

UEFI firmware

vmlab ships its own UEFI firmware. The guest assets carry Debian's edk2 builds, pinned by version and sha256, under firmware/<arch>/:

So a host needs no ovmf or qemu-efi-aarch64 package, and every host boots the same firmware, which is what lets a Windows 11 template built on one host boot under secure boot on another. firmware/VERSION names the Debian packages the set came from. For each UEFI VM, vmlab looks in three places, in order, for plain UEFI and secure boot alike:

1. firmware_dir from the host config, as <dir>/<arch>/ with the file names above. A directory there is authoritative for that architecture; a missing pair is an error.
2. The bundled set, under each guest asset directory in turn: $VMLAB_GUEST_ASSET_DIR, /usr/share/vmlab/guest, ~/.local/share/vmlab/guest.
3. The host's distribution firmware at its well-known paths. riscv64 always lands here, since vmlab bundles no riscv64 firmware.

By default a secure-boot build is only ever taken with the variable store that has keys enrolled for it: a blank store boots in setup mode, where nothing is verified, so vmlab refuses rather than boot one. Each VM copies its store from the template once, at its first start, into .vmlab/vms/<vm>/OVMF_VARS.fd, and keeps it, enrolled keys and boot entries included. A VM created under a firmware of a different flash layout, such as a host's old 2 MiB OVMF, keeps booting a build of that layout. Templates record firmware, secure_boot and secure_boot_keys, never a firmware path, so nothing about the building host travels with them.

Your own secure-boot keys

A guest signed with a key you generated, such as an mkosi image whose UKI is signed with mkosi genkey's key, does not boot under Microsoft's store. Two settings on the VM (or the template) choose another store:

vm "candidate" {
  template         = "x86_64/distro-server"
  firmware         = "ovmf"
  secure_boot      = true
  secure_boot_keys = "setup"   // "microsoft" is the default
}

Setup mode is never chosen for you: without secure_boot_keys = "setup", a host that has a secure-boot CODE but no enrolled store is still refused. Neither setting means anything without secure boot, so validation rejects them where secure_boot resolves off or the firmware is SeaBIOS. The store is copied at the VM's first start only; to change the key store of a VM that has already started, delete its OVMF_VARS.fd, losing its boot entries. A template built with either setting records it, and one built with secure_boot_vars stores the file beside its disk, so every clone starts from the same keys unless it names its own. firmware_dir is unchanged by all of this: it still replaces the whole firmware set for an architecture, host-wide.

Directories

vmlab follows the XDG layout and honours each variable that overrides it:

The runtime directory is a full-privilege interface, since a client that can connect to a lab socket runs scripts in the lab and reads and writes guest files. vmlab creates it 0700, tightens an existing one, and refuses one owned by someone else. VMLAB_WORK_DIR relocates every lab's .vmlab/ under one base, namespaced by lab name and a hash of its root, which keeps the write-heavy working data off a slow filesystem while the lab file stays put.

WSL 2

vmlab was designed to be clean on WSL 2, and the choices that make it so are the same ones that make it need no privileges elsewhere: the network fabric is userspace with no tap, bridge or macvlan, and the only kernel grant KVM needs is /dev/kvm. Four things are specific to WSL 2.

• Nested virtualisation must be enabled in .wslconfig for /dev/kvm to exist inside the distribution. Without it every VM runs under TCG.
• XDG_RUNTIME_DIR may be missing. Some WSL setups do not set it; vmlab falls back to /tmp/vmlab-<uid>, created private and refused if owned by anyone else.
• The viewer lives on the Windows side. Run vmlab console --tcp to get a localhost address and point a Windows VNC client at it; WSL's localhost forwarding carries it across. Host access to guest services works the same way, through port forwards and localhost forwarding.
• The disk-space watchdog matters more. WSL 2's ext4 VHDX grows and does not shrink, and linked clones grow with use, so disk_low_percent and the host.disk_low event are worth handling.

KVM or TCG

For each machine vmlab picks the accelerator once: KVM when /dev/kvm can be opened and the guest architecture is the host's, TCG otherwise. TCG is full emulation, slow but functional, and vmlab warns loudly when it falls back, naming the machine and the architecture. Two cases hit it on purpose: a foreign-architecture guest such as an aarch64 or riscv64 template on an x86_64 host, which can only ever be emulated, and a host with no /dev/kvm. Give an emulated guest a couple of minutes to boot. Nested virtualisation inside a guest is separate: a KVM guest on x86 sees VMX or SVM only when its VM sets nested = true, and only when the host's own kvm_intel or kvm_amd module has nested virtualisation on. vmlab up warns with the fix when it is off.