System startup After achieving control over system power, the next step was taking control of which operating system starts there. I considered two options: A USB boot drive, emulated from an RPi Network boot The first option turned out to be problematic when combined with emulated USB input devices (see below), at least on some laptops. While a single USB device can have multiple interfaces (basically being sub-devices), many types of system firmware do not like to boot from such devices. When I exposed a USB device that has both a storage interface and a HID interface (keyboard/mouse), the system didn’t consider it a bootable device. One solution would be to use two separate devices, but that would require yet another RPi (or something similar), since most (all?) such boards support emulating only a single device. Another way around it could be emulating a USB hub and getting two virtual devices this way, but Linux does not support USB hub emulation. Since I had an alternative, I didn’t explore this option any further. On systems that are fine with a single multi-function USB device, I can use that. On others, I use network boot. The second option turned out not to be that straightforward either. First of all, not all systems support booting from the network to begin with. To solve this problem, I got a USB stick and put iPXE (https://ipxe.org/) on it. Then, I configured the system to boot from that USB stick. I couldn’t use Grub here to gain network boot, because Grub supports only network devices via the system firmware (BIOS/EFI) support, and this support is missing on systems not capable of network booting. iPXE, on the other hand, supports a wide range of network devices on its own, and also allows simple scripting, like booting different systems depending on various settings. Unfortunately, it cannot boot Xen via the multiboot2 protocol (required to boot with full EFI support), it can only do multiboot1. So, I did need Grub. Luckily, iPXE does register its drivers as appropriate EFI services, so when I load Grub from iPXE, it can talk to the network. I prepared a Grub configuration that can boot different systems on different laptops depending on a separate configuration file (loaded via the load_env Grub command) and a tool to conveniently switch between them. This got me a nice menu: $ testbed-control 2 help Selected target: 2 Available commands: - reset - hard reset the target - poweron - power on the target - poweroff - (hard) power off the target - wake - wake up the system (either wake-on-lan, or button press) - rescue - switch next boot to rescue system (doesn't load anything from the disk) - fallback - switch next boot to fallback system (loads /boot/efi/EFI/qubes/grub-fallback.cfg) - normal - switch next boot to normal system - custom - switch next boot to custom grub config (/srv/tftp/test2/grub.cfg) The first four commands are about power control (see above), and the rest are about choosing what to boot. The normal command simply starts the system installed on the local disk, while rescue allows booting an initramfs-only system to diagnose why the normal system doesn’t work. The custom option allows, in practice, starting an arbitrary kernel (not necessarily from the disk). That option is especially useful for debugging Linux and Xen issues, including doing automatic bisection, although it requires a bit more in terms of glue scripts (but that’s a topic for another article). Surprisingly, I had one case where booting the local system turned out to be tricky. When the bootloader is loaded from the network, that particular UEFI does not register services to access the local disk. As it turns out, Grub does not support NVMe drives directly; it supports them only via UEFI services. I could have switched to another disk, or to booting via USB, but neither of those options felt appealing. I wanted to run tests on NVMe drives too, and