On the test system, something needs to actually enable HDMI output. For this
purpose, I passed a kickstart file to the Qubes OS installer that includes
commands to execute before installation (the %pre section). While at it, I
could use the same kickstart file for other test-related customizations, like
restoring the default boot order at the end or enabling SSH access for
collecting logs.
HID input
Recording video output is not everything. To run tests, one also needs to send
commands to the system under test (SUT). This can be done in several ways,
including via serial console and SSH connection. In order to have the most
realistic setup, I chose to emulate USB input devices. With this, we could
interact with the system in the same way a user would. To emulate USB input
device(s), I used the Linux USB Gadget subsystem. To emulate HID devices, I had
to prepare a HID descriptor — a description for the driver, listing what kind
of device it is and what events it can send.
I wanted the device(s) to meet the following requirements:
Have two interfaces (which in practice is two separate HID devices): keyboard
and pointer (mouse/tablet)
Be properly categorized by udev (so the input
proxy (github.com/QubesOS…ut-proxy) picks it up
properly)
Be properly categorized by Xorg
Support both absolute pointer position events (like “move mouse to a specific
point” instead of “move mouse a bit to the right”) and normal mouse buttons
I searched for a descriptor meeting the above requirements. The one for
keyboards is rather standard, but the one for mouse/tablet devices is not. So,
I took the Device Class Definition for HID 1.11 together with HID Usage Tables
1.22 (https://www.usb.org/hid) and crafted one myself. This was a bit of a
challenge, because both udev and Xorg have a set of heuristics to categorize
devices, and they differ in subtle ways.
Then, I wrote a
script (gist.github.com/marmare…8780c208)
that sets this all up and controls the device(s) according to what openQA
requests.
The last detail is about connecting an RPi to the target system. The RPi4 has a
single USB-C port used both for powering the RPi itself as well as for USB
device emulation. Generally, this would be fine, with the exception that the
target system is going to be disconnected from power from time to time. If the
RPi were powered this way, it would lose power too, and there would be nothing
capable of turning it back on. This is yet another case where the PiKVM project
provided an inspiration (https://github.com/pikvm/pikvm#setting-up-the-v2): a
Y-split cable that connects the VBUS pin to only one end and the data pins to
the other.
Serial console
Several openQA functions require some kind of console access. This includes
retrieving command outputs (and exit codes), waiting for various events, etc.
Unfortunately, a real serial console is very rare in modern laptops. I could
restructure the tests not to use those functions, but that would be rather
disappointing in terms of test result quality. As a solution, I added a small
qrexec service in dom0 that reads a pipe that pretends to be a serial console,
then I used qvm-connect-tcp in sys-net to redirect the TCP port to that
service. This isn’t as reliable as a real serial console (especially for things
like restarting sys-net), but it does work in the majority of cases. In the
future, I will restructure the tests not to rely on this functionality in order
to account for the few rare cases where it doesn’t work.
Bonus: remote-controlled test laptops for developers
Remote power and boot control is useful not only for automatic tests, but also
for ordinary developers. There are several cases where it is useful:
Additional machines to develop and test features on different versions of
Qubes
Access to specific hardware
I’ve prepared the whole setup to be usable not only with openQA, but also to