TL;DR - LAVA + openQA = automated, continuous kernel testing
Everyone wants the same thing: stable software for the long term, including the latest features. To achieve this, you need to have confidence in your software and in the software you depend on. This confidence has to come from testing.
But testing modern software platforms can be extremely complex and time consuming, and it’s easy to miss out on the latest fixes and features in Linux for fear of the time-consuming manual regression tests required.
Codethink always advocates staying as closely aligned to mainline as possible, limiting the overhead required to consume the latest upstream and benefit from the advances made there. But this is often not possible or avoided due to the burden of testing.
To solve this, we need continuous, automated and comprehensive testing. This blog post explains how we have developed a testing pipeline for the kernel, enabling automated boot tests in both QEMU and on hardware, as well as UI tests and the issuing of command line instructions to test responses.
Making use of Open Source Software
To build the testing pipeline we utilised openQA and LAVA. Both are focused on test automation and validation as part of a Continuous Integration development process. For an overview of each project, read on. If you already know the technology, feel free to skip to the next section: Kernel Testing Setup.
LAVA offers the ability to automate the deployment of operating systems onto both physical devices and virtual hardware. However, the support for actual hardware is more developed than for emulated devices. It focuses on providing a range of deployment and boot methods, rather than the tests themselves, so once an OS is deployed and logged into the device under test is achieved, the type & range of tests are only limited by what the tester chooses to execute.
The architecture of a LAVA instance comprises of a single server component which can control one or multiple worker components. Contained within the server is a web interface, which can be used to issue test jobs, monitor the status of the tests, and retrieve the results of completed tests. A test job request is initially queued and the LAVA server's scheduler periodically checks the server's database for jobs and the current availability of appropriate test devices, allocating jobs to available workers whenever resources match requests. Inside the LAVA worker component is a dispatcher, which manages all the operations on the test device that arise from the parameters of the job request and also the device parameters provided by the LAVA server.
While LAVA focuses on the deployment and booting of the OS onto emulated or actual hardware, openQA is an automated test tool that allows for the possibility of testing the whole installation process of an OS. These tests can be used to check the output of the console and can also use OpenCV for fuzzy image matching of the screen, using a concept called needles. This can be used to determine if rendering occurs in the expected order and correctness. OpenQA also offers the ability to send keystrokes that could be used, for example, to progress the installation process with an appropriate input, or for entering passwords when required by the system.
The architecture of openQA is similar to that of LAVA, in that it has a web interface that can be used to monitor the progress of submitted tests. It is also responsible for matching tests to available and appropriate machine resources, and assigning a job to a worker component, which interfaces with the openQA UI to collect data and input files that will be needed by the tests.
For each openQA worker there is an associated os-autoinst component which, in the case of an emulated machine, is responsible for the machine being created. It is also responsible for interacting with the machine under test, whether it is virtual or actual hardware, via a VNC connection, performing the actual tests and collecting the test results. These results include video of the machine's display as it is tested, along with screenshots of each test point and a detailed output log.
Setting up the Kernel Testing
In order to investigate the possibilities of LAVA and openQA for testing a Linux kernel in both QEMU and actual hardware, we created a small test setup using a Raspberry Pi (Rpi4) and a laptop. The main aim of this test setup is to determine if a change in the kernel has any impact on the deployment of a Linux OS and how it functions. For our testing purposes we used Ubuntu 20.04 for the OS, but the general concept of the setup should be flexible enough to test other Linux distributions.
The openQA testing under QEMU is the simpler of the two testing scenarios, as openQA can be directly instructed to use a particular QEMU image, and the installation of the kernel under test can form part of the openQA tests.
For the hardware testing scenario, the Rpi4 is connected to the laptop via it's serial port, which will allow a LAVA instance running on the laptop to communicate with it. The LAVA instance is created from two Docker containers, one for the LAVA server and one for the LAVA worker. The Rpi4 has two connected storage devices, an SD card and a USB drive, which are for a base image and test image respectively. This configuration allows us to recover from any issue that arises during the testing, as the Rpi4 defaults to booting from the base image, which is never used during testing.
A LAVA job, as part of a deploy action, first checks that all the files needed to create a ramdisk for the Rpi4 exist and then downloads them to the LAVA dispatcher assigned to that job. Once the dispatcher has created the ramdisk, it begins a boot action, or more specifically, a U-Boot action. This entails the creation of a bootloader overlay specific for the board under test and then triggering a reboot of the Rpi4. Using a bootloader interrupt method from LAVA, it uses the previously created overlay to configure U-Boot and download the ramdisk to the Rpi4's memory. Once complete, the final command of the bootloader overlay makes it boot in to the test image on the USB drive, using the kernel in the ramdisk.
Once booted, the LAVA dispatcher sends the commands to log in to the Rpi4, and then runs a configuration script (that is already part of the test image, along with a configuration to enable screen sharing). Once the two devices are connected to the same network, the LAVA dispatcher triggers the running of a containerised image of an openQA client. This client requests that the openQA instance, running in the laptop, assign a worker/os-autoinst pair to perform a specific set of tests on the Rpi4. The openQA instance on the laptop is constructed from three Docker containers; for the UI, worker and supporting database.
In order to run the openQA tests on actual hardware, a new backend for the os-autoinst component of openQA needed to be created, i.e. one that only provides a VNC connection to the board under test, as there is now no need to create a virtual machine. This was a relatively simple task, as openQA's architecture allows for the creation of new backends, providing a generic code template.
From this point, the openQA testing part of the investigation is almost identical in both the case of QEMU and on actual hardware. The only difference is in the hardware case, where the LAVA dispatcher has control of the serial console. So in order to run script-based, non-graphical tests, on the hardware, a new openQA test 'distribution' (or distri) needs to be defined for the test group. This allows you to define the additional consoles you intend to use, which in our case was a 'serial over SSH' console.
Linux Kernel Testing Pipeline
The combination of LAVA and openQA has proven successful in our implementation of a Linux kernel testing pipeline, with the results now available to view from our public facing openQA UI.
QEMU test of mainline kernel
Hardware test of LTS kernel
The pipeline means we’re now performing automated daily tests of the Linux mainline kernel in QEMU, and LTS kernels in both QEMU and hardware. The results of which form part of regular test reports provided to the Linux kernel's stable branch maintainers.
Whilst this is obviously not yet a comprehensive test suite in its own right, it is a start, and helps us to catch any major regressions as early as possible, stopping them ever being added to the kernel. We welcome any additions to the pipeline from other interested parties.
Next up, we plan to output the results of the test pipeline to the KernelCI project, and add more architectures to the setup, starting with RISC-V.
Related blog posts:
- OpenQA and LAVA for GNOME: Higher quality of FOSS: How we are helping GNOME to improve their test pipeline >>
- Aligning to the mainline Linux kernel: Why aligning with open source mainline is the way to go >>
- Using STPA with software-intensive systems
- Codethink achieves ISO 26262 ASIL D Tool Certification
- RISC-V: running GNOME OS on SiFive hardware for the first time
- Native compilation on Arm servers is so much faster now
- Higher quality of FOSS: How we are helping GNOME to improve their test pipeline
- RISC-V: A Small Hardware Project
- Why aligning with open source mainline is the way to go
- Build Meetup 2021: The BuildTeam Community Event
- A new approach to software safety
- Does the "Hypocrite Commits" incident prove that Linux is unsafe?
- ABI Stability in freedesktop-sdk
- Why your organisation needs to embrace working in the open-source ecosystem
- RISC-V User space access Oops
- Tracking Players at the Edge: An Overview
- What is Remote Asset API?
- Running a devroom at FOSDEM: Safety and Open Source
- Meet the codethings: Understanding BuildGrid and BuildBox with Beth White
- Streamlining Terraform configuration with Jsonnet
- Bloodlight: Designing a Heart Rate Sensor with STM32, LEDs and Photodiode
- Making the tech industry more inclusive for women
- Bloodlight Case Design: Lessons Learned
- Safety is a system property, not a software property
- RISC-V: Codethink's first research about the open instruction set
- Meet the Codethings: Safety-critical systems and the benefits of STPA with Shaun Mooney
- Why Project Managers are essential in an effective software consultancy
- FOSDEM 2021: Devroom for Safety and Open Source
- Meet the Codethings: Ben Dooks talks about Linux kernel and RISC-V
- Here we go 2021: 4 open source events for software engineers and project leaders
- Xmas Greetings from Codethink
- Call for Papers: FOSDEM 2021 Dev Room Safety and Open Source Software
- Building the abseil-hello Bazel project for a different architecture using a dynamically generated toolchain
- Advent of Code: programming puzzle challenges
- Improving performance on Interrogizer with the stm32
- Introducing Interrogizer: providing affordable troubleshooting
- Improving software security through input validation
- More time on top: My latest work improving Topplot
- Cycling around the world
- Orchestrating applications by (ab)using Ansible's Network XML Parser
- My experience of the MIT STAMP workshop 2020
- Red Hat announces new Flatpak Runtime for RHEL
- How to keep your staff healthy in lockdown
- Bloodlight: A Medical PPG Testbed
- Bringing Lorry into the 2020s
- How to use Tracecompass to analyse kernel traces from LTTng
- Fixing Rust's test suite on RISC-V
- The challenges behind electric vehicle infrastructure
- Investigating kernel user-space access
- Consuming BuildStream projects in Bazel: the bazelize plugin
- Improving RISC-V Linux support in Rust
- Creating a Build toolkit using the Remote Execution API
- Trusting software in a pandemic
- The Case For Open Source Software In The Medical Industry
- My experiences moving to remote working
- Impact of COVID-19 on the Medical Devices Industry
- COVID-19 (Coronavirus) and Codethink
- Codethink develops Open Source drivers for Microsoft Azure Sphere MediaTek MT3620
- Codethink partners with Wirepas
- Testing Bazel's Remote Execution API
- Passing the age of retirement: our work with Fortran and its compilers
- Sharing technical knowledge at Codethink
- Using the REAPI for Distributed Builds
- An Introduction to Remote Execution and Distributed Builds
- Gluing hardware and software: Board Support Packages (BSPs)
- Engineering's jack of all trades: an intro to FPGAs
- Bust out your pendrives: Debian 10 is out!
- Why you should attend local open source meet-ups
- Acceptance, strife, and progress in the LGBTIQ+ and open source communities
- Codethink helps York Instruments to deliver world-beating medical brain-scanner
- Codethink open sources part of staff onboarding - 'How To Git Going In FOSS'
- Getting into open source
- How to put GitOps to work for your software delivery
- Open Source Safety Requirements Analysis for Autonomous Vehicles based on STPA
- Codethink engineers develop custom debug solution for customer project
- Codethink contributes to CIP Super Long Term Kernel maintenance
- Codethink creates custom USB 3 switch to support customer's CI/CD pipeline requirements
- Codethink unlocks data analysis potential for British Cycling
- MIT Doctor delivers Manchester masterclass on innovative safety methodology
- Balance for Better: Women in Technology Codethink Interviews
- Introducing BuildGrid
- Full archive