There is a lot to test on a modern digital product, and testing takes time. Companies often miss out on the latest fixes in features in the open source components they use, for fear of the time-consuming manual regression tests they have to do after upgrading a component such as Linux or GStreamer.
Codethink always advises our partners and customers that closely following mainline gives a competitive advantage (and we're not the only ones). To do this, you need to be backed up by comprehensive automated testing.
We can save even more effort by helping upstreams to improve their own automated testing capabilities. We recently built a new test pipeline for the GNOME project, covering parts of the desktop environment that usually prove difficult to test.
How do you test a GUI?
Volunteers at the GNOME project have worked hard and brought significant improvements to the project's build, test and release process. GNOME's CI builds a testing-only OS image with bleeding-edge versions of the many components that make up a modern desktop Linux system. As well as powering developer workstations, you'll see these components in many commercial Linux-based products.
Everyone uses Gitlab CI to test components and libraries in isolation, but how do you test that an operating system installs correctly, and simulate the mouse clicks and keypresses of a new user logging in for the first time? At the moment that falls to volunteers in the #gnome-os channel who manually test the OS images for regressions. But we can do better.
OpenQA for GNOME
The Linux world has a solution for UI testing: OpenQA, originally developed and maintained by SuSE. OpenQA allows writing tests for the whole installation process of an OS, checking console output, and also using OpenCV for fuzzy image matching of the screen. The image matching tests are called needles, and can determine if rendering occurs in the expected order and correctness. OpenQA also offers the ability to send keystrokes and mouse clicks that could be used to progress the installation process with an appropriate input or for entering passwords.
Working with GNOME, we set up openqa.gnome.org and created an initial set of tests. We're now working to integrate the tests into the gnome-build-meta repo where GNOME's many components are integrated ready for release.
Here's a screenshot from openqa.gnome.org:
Here's the gnome-build-meta merge request.
Integrating OpenQA with Gitlab CI
OpenQA is often used with dedicated worker machines, but we wanted to avoid creating a "pet" system that would require special maintenance. So after the CI pipeline has a Gitlab runner successfully build the artifacts, it then has another runner create an openQA worker instance with a unique ID. This ID ensures it can only be used by the pipeline that created it. The runner then issues a command to the openQA UI to perform the tests on its created worker, using API key and secret values configured in the Gitlab project.
The tests themselves are located in a sub-directory of the Gitlab repository, utilising needles cloned from the master branch of a separate needle Gitlab repository. When the test is executed the runner proceeds to poll the status of the openQA test job, to determine if it has 'passed' or not.
The design and layout of GNOME's user interface can evolve and change over time, and the tests will need to adapt. We would like to maintain separate branches of the tests following GNOME's 6 month release cadence, which is a workflow that the OpenQA UI doesn't yet support. The current implementation of the OpenQA UI requires the needles to be already in a particular directory, so they are cloned from the needles repository when the openQA UI container is created. The common workaround for this limitation is as follows:
- Ensure openQA UI will automatically update the needles repository on a change
- When a test fails, use the openQA UI to create the new or modified needle in the needles repo.
- Have a cron job running in the openQA UI container to pull from the master branch of the needle repository every minute, to ensure synchronicity if other users of the needle repository simultaneously modify the needles.
Another future goal is to run the OpenQA tests for every merge request. For now the tests can be manually started for specific merge requests, and can also be scheduled to run at regular intervals on the 'master' branch. Before we can enabling OpenQA for all merge requests, we'll need to speed up the build process so we don't want to add a big delay to every new MR.
We are looking forward to seeing OpenQA in action testing the latest GNOME desktop images in QEMU virtual machines, and if you are building a complex project with a graphical interface, you might want to investigate OpenQA as well.
Stay up to date on our Long Term Maintainability news
Receive our recent articles about Long Term Maintainability in your email.
Related Blog Posts:
- Written by Neill Whillans: Why aligning with open source mainline is the way to go >>
- Codethink's approach to Long-term Maintainability: Codethink contributes to CIP Super Long Term Kernel maintenance >>
- Using Git LFS and fast-import together
- Testing in a Box: Streamlining Embedded Systems Testing
- SDV Europe: What Codethink has planned
- How do Hardware Security Modules impact the automotive sector? The final blog in a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part two of a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part one of a three part discussion
- Automated Kernel Testing on RISC-V Hardware
- Automated end-to-end testing for Android Automotive on Hardware
- GUADEC 2023
- Embedded Open Source Summit 2023
- RISC-V: exploring a bug in stack unwinding
- Adding RISC-V Vector Cryptography Extension support to QEMU
- Introducing Our New Open-Source Tool: Quality Assurance Daemon
- Long Term Maintainability
- FOSDEM 2023
- Think before you Pip
- BuildStream 2.0 is here, just in time for the holidays!
- A Valuable & Comprehensive Firmware Code Review by Codethink
- GNOME OS & Atomic Upgrades on the PinePhone
- Flathub-Codethink Collaboration
- Codethink proudly sponsors GUADEC 2022
- Tracking Down an Obscure Reproducibility Bug in glibc
- Web app test automation with `cdt`
- FOSDEM Testing and Automation talk
- Protecting your project from dependency access problems
- Porting GNOME OS to Microchip's PolarFire Icicle Kit
- YAML Schemas: Validating Data without Writing Code
- Deterministic Construction Service
- Codethink becomes a Microchip Design Partner
- Hamsa: Using an NVIDIA Jetson Development Kit to create a fully open-source Robot Nano Hand
- Using STPA with software-intensive systems
- Codethink achieves ISO 26262 ASIL D Tool Certification
- RISC-V: running GNOME OS on SiFive hardware for the first time
- Automated Linux kernel testing
- Native compilation on Arm servers is so much faster now
- RISC-V: A Small Hardware Project
- Why aligning with open source mainline is the way to go
- Build Meetup 2021: The BuildTeam Community Event
- A new approach to software safety
- Does the "Hypocrite Commits" incident prove that Linux is unsafe?
- ABI Stability in freedesktop-sdk
- Why your organisation needs to embrace working in the open-source ecosystem
- RISC-V User space access Oops
- Tracking Players at the Edge: An Overview
- What is Remote Asset API?
- Running a devroom at FOSDEM: Safety and Open Source
- Meet the codethings: Understanding BuildGrid and BuildBox with Beth White
- Streamlining Terraform configuration with Jsonnet
- Bloodlight: Designing a Heart Rate Sensor with STM32, LEDs and Photodiode
- Making the tech industry more inclusive for women
- Bloodlight Case Design: Lessons Learned
- Safety is a system property, not a software property
- RISC-V: Codethink's first research about the open instruction set
- Meet the Codethings: Safety-critical systems and the benefits of STPA with Shaun Mooney
- Why Project Managers are essential in an effective software consultancy
- FOSDEM 2021: Devroom for Safety and Open Source
- Meet the Codethings: Ben Dooks talks about Linux kernel and RISC-V
- Here we go 2021: 4 open source events for software engineers and project leaders
- Xmas Greetings from Codethink
- Call for Papers: FOSDEM 2021 Dev Room Safety and Open Source Software
- Building the abseil-hello Bazel project for a different architecture using a dynamically generated toolchain
- Advent of Code: programming puzzle challenges
- Full archive