There is a lot to test on a modern digital product, and testing takes time. Companies often miss out on the latest fixes in features in the open source components they use, for fear of the time-consuming manual regression tests they have to do after upgrading a component such as Linux or GStreamer.
Codethink always advises our partners and customers that closely following mainline gives a competitive advantage (and we're not the only ones). To do this, you need to be backed up by comprehensive automated testing.
We can save even more effort by helping upstreams to improve their own automated testing capabilities. We recently built a new test pipeline for the GNOME project, covering parts of the desktop environment that usually prove difficult to test.
How do you test a GUI?
Volunteers at the GNOME project have worked hard and brought significant improvements to the project's build, test and release process. GNOME's CI builds a testing-only OS image with bleeding-edge versions of the many components that make up a modern desktop Linux system. As well as powering developer workstations, you'll see these components in many commercial Linux-based products.
Everyone uses Gitlab CI to test components and libraries in isolation, but how do you test that an operating system installs correctly, and simulate the mouse clicks and keypresses of a new user logging in for the first time? At the moment that falls to volunteers in the #gnome-os channel who manually test the OS images for regressions. But we can do better.
OpenQA for GNOME
The Linux world has a solution for UI testing: OpenQA, originally developed and maintained by SuSE. OpenQA allows writing tests for the whole installation process of an OS, checking console output, and also using OpenCV for fuzzy image matching of the screen. The image matching tests are called needles, and can determine if rendering occurs in the expected order and correctness. OpenQA also offers the ability to send keystrokes and mouse clicks that could be used to progress the installation process with an appropriate input or for entering passwords.
Working with GNOME, we set up openqa.gnome.org and created an initial set of tests. We're now working to integrate the tests into the gnome-build-meta repo where GNOME's many components are integrated ready for release.
Here's a screenshot from openqa.gnome.org:
Here's the gnome-build-meta merge request.
Integrating OpenQA with Gitlab CI
OpenQA is often used with dedicated worker machines, but we wanted to avoid creating a "pet" system that would require special maintenance. So after the CI pipeline has a Gitlab runner successfully build the artifacts, it then has another runner create an openQA worker instance with a unique ID. This ID ensures it can only be used by the pipeline that created it. The runner then issues a command to the openQA UI to perform the tests on its created worker, using API key and secret values configured in the Gitlab project.
The tests themselves are located in a sub-directory of the Gitlab repository, utilising needles cloned from the master branch of a separate needle Gitlab repository. When the test is executed the runner proceeds to poll the status of the openQA test job, to determine if it has 'passed' or not.
The design and layout of GNOME's user interface can evolve and change over time, and the tests will need to adapt. We would like to maintain separate branches of the tests following GNOME's 6 month release cadence, which is a workflow that the OpenQA UI doesn't yet support. The current implementation of the OpenQA UI requires the needles to be already in a particular directory, so they are cloned from the needles repository when the openQA UI container is created. The common workaround for this limitation is as follows:
- Ensure openQA UI will automatically update the needles repository on a change
- When a test fails, use the openQA UI to create the new or modified needle in the needles repo.
- Have a cron job running in the openQA UI container to pull from the master branch of the needle repository every minute, to ensure synchronicity if other users of the needle repository simultaneously modify the needles.
Another future goal is to run the OpenQA tests for every merge request. For now the tests can be manually started for specific merge requests, and can also be scheduled to run at regular intervals on the 'master' branch. Before we can enabling OpenQA for all merge requests, we'll need to speed up the build process so we don't want to add a big delay to every new MR.
We are looking forward to seeing OpenQA in action testing the latest GNOME desktop images in QEMU virtual machines, and if you are building a complex project with a graphical interface, you might want to investigate OpenQA as well.
Related Blog Posts:
- Written by Neill Whillans: Why aligning with open source mainline is the way to go >>
- Codethink's approach to Long-term Maintainability: Codethink contributes to CIP Super Long Term Kernel maintenance >>
- Using STPA with software-intensive systems
- Codethink achieves ISO 26262 ASIL D Tool Certification
- RISC-V: running GNOME OS on SiFive hardware for the first time
- Automated Linux kernel testing
- Native compilation on Arm servers is so much faster now
- RISC-V: A Small Hardware Project
- Why aligning with open source mainline is the way to go
- Build Meetup 2021: The BuildTeam Community Event
- A new approach to software safety
- Does the "Hypocrite Commits" incident prove that Linux is unsafe?
- ABI Stability in freedesktop-sdk
- Why your organisation needs to embrace working in the open-source ecosystem
- RISC-V User space access Oops
- Tracking Players at the Edge: An Overview
- What is Remote Asset API?
- Running a devroom at FOSDEM: Safety and Open Source
- Meet the codethings: Understanding BuildGrid and BuildBox with Beth White
- Streamlining Terraform configuration with Jsonnet
- Bloodlight: Designing a Heart Rate Sensor with STM32, LEDs and Photodiode
- Making the tech industry more inclusive for women
- Bloodlight Case Design: Lessons Learned
- Safety is a system property, not a software property
- RISC-V: Codethink's first research about the open instruction set
- Meet the Codethings: Safety-critical systems and the benefits of STPA with Shaun Mooney
- Why Project Managers are essential in an effective software consultancy
- FOSDEM 2021: Devroom for Safety and Open Source
- Meet the Codethings: Ben Dooks talks about Linux kernel and RISC-V
- Here we go 2021: 4 open source events for software engineers and project leaders
- Xmas Greetings from Codethink
- Call for Papers: FOSDEM 2021 Dev Room Safety and Open Source Software
- Building the abseil-hello Bazel project for a different architecture using a dynamically generated toolchain
- Advent of Code: programming puzzle challenges
- Improving performance on Interrogizer with the stm32
- Introducing Interrogizer: providing affordable troubleshooting
- Improving software security through input validation
- More time on top: My latest work improving Topplot
- Cycling around the world
- Orchestrating applications by (ab)using Ansible's Network XML Parser
- My experience of the MIT STAMP workshop 2020
- Red Hat announces new Flatpak Runtime for RHEL
- How to keep your staff healthy in lockdown
- Bloodlight: A Medical PPG Testbed
- Bringing Lorry into the 2020s
- How to use Tracecompass to analyse kernel traces from LTTng
- Fixing Rust's test suite on RISC-V
- The challenges behind electric vehicle infrastructure
- Investigating kernel user-space access
- Consuming BuildStream projects in Bazel: the bazelize plugin
- Improving RISC-V Linux support in Rust
- Creating a Build toolkit using the Remote Execution API
- Trusting software in a pandemic
- The Case For Open Source Software In The Medical Industry
- My experiences moving to remote working
- Impact of COVID-19 on the Medical Devices Industry
- COVID-19 (Coronavirus) and Codethink
- Codethink develops Open Source drivers for Microsoft Azure Sphere MediaTek MT3620
- Codethink partners with Wirepas
- Testing Bazel's Remote Execution API
- Passing the age of retirement: our work with Fortran and its compilers
- Sharing technical knowledge at Codethink
- Using the REAPI for Distributed Builds
- An Introduction to Remote Execution and Distributed Builds
- Gluing hardware and software: Board Support Packages (BSPs)
- Engineering's jack of all trades: an intro to FPGAs
- Bust out your pendrives: Debian 10 is out!
- Why you should attend local open source meet-ups
- Acceptance, strife, and progress in the LGBTIQ+ and open source communities
- Codethink helps York Instruments to deliver world-beating medical brain-scanner
- Codethink open sources part of staff onboarding - 'How To Git Going In FOSS'
- Getting into open source
- How to put GitOps to work for your software delivery
- Open Source Safety Requirements Analysis for Autonomous Vehicles based on STPA
- Codethink engineers develop custom debug solution for customer project
- Codethink contributes to CIP Super Long Term Kernel maintenance
- Codethink creates custom USB 3 switch to support customer's CI/CD pipeline requirements
- Codethink unlocks data analysis potential for British Cycling
- MIT Doctor delivers Manchester masterclass on innovative safety methodology
- Balance for Better: Women in Technology Codethink Interviews
- Introducing BuildGrid
- Full archive