Simple, Painless Application Testing on Virtualized Hardware
One strategy for delivering software of the highest quality is practicing Continuous Integration and Delivery (CI/CD). While CI/CD is a relatively common quality assurance practice among software companies, it is much more difficult to use with low-level software such as kernel and embedded applications. These types of applications need to be tested on real, physical hardware – especially for acceptance tests. But it can still be beneficial to use virtual machines in the majority of the CI/CD pipeline.
There are many existing virtualization solutions, but few of them are well suited for use in CI. For this reason, we developed Transient, an open-source QEMU-wrapper focused on ease-of-use within a CI/CD pipeline. Ultimately, Transient makes testing applications on virtualized hardware a much more simple and painless endeavour.
Transient: A lightweight QEMU wrapper designed for CI/CD pipelines
Here’s a common test case that you would find in Star Lab’s CI/CD pipeline:
The test configures and deploys a virtual machine (VM).
The test provisions the VM with the system-under-test.
The test exercises the system-under-test.
The test destroys the VM and releases its resources.
Although this pattern seems simple enough, configuring, deploying, and provisioning VMs in an automated fashion is rather tricky. Existing tools don’t quite fit this use-case; a few issues include requiring manual intervention through a GUI, being incredibly complex and unstable, and having incredibly slow boot and shutdown times. But by interfacing directly with QEMU, we provide a command line utility that is easily scriptable, fast, and stable. With Transient, we can easily configure a VM with command line arguments or a simple TOML configuration file and deploy it in seconds:
We built Transient with the following features to help us with specific challenges we encountered in our CI/CD pipeline:
Written in Python3, Transient is maximally portable and works "out-of-the-box" on most Linux systems, including Ubuntu and CentOS.
Transient supports shared directories via SSHFS, eliminating issues with NFS.
Being a lightweight QEMU-wrapper, Transient boots VMs quickly.
Since Transient interfaces directly with QEMU, hardware virtualization is extremely flexible.
Files and directories can be added to the VM before booting and retrieved after shutdown, making it extremely easy to set-up and retrieve test data (even for VMs without network interfaces).
Like Vagrant, Transient makes it simple to pull an existing disk image from the Vagrant cloud and start a VM quickly.
Why not just use Docker, like every other CI/CD pipeline?
Docker is an excellent tool to use in production, development, and CI/CD environments. By isolating and declaring the dependencies of a software application, it facilitates a consistent environment to avoid the all too common problem of "well, it works on my machine". In fact, Star Lab uses Docker for building and testing several of our user space tools. But when it comes to testing applications that work in kernel space and embedded environments, Docker falls short. Despite being an incredibly useful application, Docker is designed for user space applications and its underlying containerization technology cannot work with kernel applications.
Why not just use Vagrant, libvirt, VirtualBox, or <Insert Hypervisor Tech Here\>?
Because Docker and other container-based technology weren't a viable solution, our next best choice was a standard virtualization solution. This allowed us to virtualize hardware and, consequently, enabled us to test low-level applications.
As mentioned earlier, we want a virtual machine to start up, run a suite of tests, and shut down. And we want to do it quickly because we have many hardware configurations to test. Additionally, the VM configuration should be easily configurable and scriptable so that it's simple to set up in a CI/CD pipeline. Eventually, we ended up settling with Vagrant and its vagrant-libvirt plugin.
But that came with issues.
Vagrant does not allow easy access to many aspects of a virtual machine that are important for testing low-level components. For example, there is no way to retrieve the console output for a VM without bypassing Vagrant and calling to libvirt functionality directly. Additionally, Vagrant is not generally safe to run concurrently. Some operations such as image downloading for the addition of NFS exports will fail if two Vagrant processes attempt to perform them at once. This can cause sporadic failures that are often difficult to debug – something we don’t want on a test server.
Onward to simple, scriptable virtual machine management
In the end, we decided to build our own tool. One that provides an easy-to-use interface for configuring, deploying, and provisioning virtual machines while avoiding the shortcomings of other tools.
The source code for Transient is hosted here on GitHub. There is also extensive documentation for how to install and use Transient available here. All development of Transient is open source under the terms of the MIT license, so contributions are welcome!