Automated OS testing on physical laptops www.qubes-os.org/news/20…-laptops Our journey towards automating OS tests on physical laptops started a few years ago with the idea of using Intel AMT to drive tests on physical machines. To start, I got an initial implementation (github.com/os-auto…pull/983) working. In particular, VNC for input/output and power control worked. I tried to get a virtual CD working, but it turned out to be quite unstable. Worse — and more importantly — it was really just a CD, not a CD/DVD, which meant that the protocol couldn’t handle images larger than 2 GB. Some time later I abandoned this approach, for two related reasons: Many machines that we want Qubes OS to support intentionally do not have Intel AMT. The single AMT-enabled machine that I had been using to develop this feature broke. If anyone would like to resume this work, this page (web.archive.org/web/202…amt.html) includes a lot of useful info about Intel AMT on Linux. Recently, I came back to the project with a new approach: to capture video from HDMI output and use an emulated USB keyboard and mouse for input. Then, I added power control to the mix, combined everything on a Raspberry Pi, and got a working prototype (github.com/os-auto…ull/1741) of an openQA worker that runs the tests on a physical machine, instead of a virtual one. The whole setup includes several devices: One “central” Raspberry Pi that controls a power strip and serves boot files. One Raspberry Pi per laptop that runs an openQA worker for that laptop. It emulates a USB device for that laptop and captures HDMI output from it. All these elements are detailed below. Base system The goal was to run an openQA worker on a Raspberry Pi 4. Why a Raspberry Pi (RPi)? Their USB controllers can play the role of a device, not just that of USB host. They’re powerful enough to run the video processing required by openQA. They’re (mostly) readily available and relatively cheap. As a base system, I chose OpenSUSE, because that’s openQA’s native distribution. Getting OpenSUSE to work on an RPi was rather straightforward (https://en.opensuse.org/HCL:Raspberry_Pi4), but the choice did lead to a few issues discussed later in this article. Power control Power control was the first stage of this project. I thought it looked like the simplest part. To reliably run unattended tests, I needed a way to interrupt a test when it went into some unrecoverable state (kernel panic, hard hang, etc.). With AMT, I had a built-in API for that, but now I needed something else. I chose a power strip that was remotely controlled via USB. Then, I removed the batteries from the laptops connected to the setup. This gave me a very reliable way to interrupt whatever was running on the machines by simply powering them down. But it turned out that powering them back on may not be that simple. In the current setup, there are several laptops, each of them slightly different, and each (sic!) requiring a slightly different approach to power management. Here are some things I tried and that worked on some machines: Setting the BIOS to automatically power on the machine when a power supply was connected. This is the simplest method. Sadly, only one of the machines supported it. Sending a Wake-On-Lan packet. Here, reliability depends on the device. For some, it just works, while others require enabling it in the network card (with the ethtool -s eth0 wol g command), and some lose the setting either on system startup or on disconnecting the power… When all else fails, one can just press the physical power button. Of course it would be too much work to do it manually, so I attached a servo motor in the exact spot where the power button is. Then, I drove that servo motor from an RPi.