At Codethink we have an increasing interest in the RISC-V architecture, and have, in past years, written several articles related to it, on subjects ranging from fixing a RISC-V kernel space Oops to running GNOME OS on SiFive hardware. We are also strong proponents of using robust testing pipelines to aid in the goal of long-term maintainability (LTM) of complex software platforms, such as the Linux kernel, advocating to align as closely to mainline as possible. For those not familiar with the concept of LTM, as it relates to software, we have written an article on Codethink's approach to it.
In the past we have written about our automated kernel testing pipeline that performed tests in QEMU (x86_64) and on actual hardware, namely a Raspberry Pi 4 (arm64). The results of these tests are provided to the Linux kernel's stable branch maintainers, and while any additional automated testing is appreciated for these common architectures, more 'exotic' configurations are just as important to test. For this reason, it was always our intention to implement a similar kernel testing pipeline for RISC-V hardware, and it was nice to see this approach confirmed as useful by Linus himself. As mentioned in previous articles, we already have several SiFive Unmatched boards, so it was the obvious choice for testing.
RISC-V Kernel Testing Pipeline
The main elements of our testing pipeline are shown in the figure below, consisting of a GitLab CI that fetches Linux kernel sources and a small patchset for RISC-V support that hasn't yet been upstreamed, that it then uses to build a .fit kernel image for the RISC-V board (although LAVA also supports the loading of the kernel, ramdisk and dtb from different binaries). The final stage of the GitLab CI involves the triggering of a LAVA job that downloads the image from the CI job artifacts, stores it into the worker's TFTP server and performs the testing of the kernel. The test automates and supervises the booting of the SiFive Unmatched board and user login. When succesfully logged into the OS and LAVA has detected the expected prompt, it then triggers the running of a script prelocated in the board's OS (Ubuntu 21.04), that initiates a set of OpenQA-controlled tests of the OS.
We use LAVA for the deployment and booting of the OS in the SiFive Unmatched board, and OpenQA to ensure that the system has booted correctly into the OS with the new kernel and the graphical interface is working as expected. The architectures of LAVA and OpenQA, and how they function, have already been described in the previously mentioned automated kernel testing pipeline article, so we won't go into them further. In that article we described how everything related to OpenQA and LAVA was located on a single laptop directly connected to the RPi4 we were testing on. However, the SiFive Unmatched is slightly louder than an RPi4, and as this kernel testing pipeline was expected to be running on a daily basis for the foreseeable future, we decided to place the RISC-V board and the components directly controlling it (the RPi4-hosted LAVA worker and power control relay) in a rack mount so it could run undisturbed in a server room. The remaining components of the pipeline were then deployed in VMs within Codethink's internal infrastructure. This deployment architecture is shown in the figure below.
On one VM, our public-facing LAVA and OpenQA servers are proxied through Traefik, while on the second VM, OpenQA workers can be created as needed. In the rack mount, as mentioned previously, we have an RPi4 acting as a controller for SiFive Unmatched board. Using the board's serial connection allows it to communicate with the LAVA worker instance, which in turn can control a USB power relay to power on/off and reset the board.
In general, now that we have deployed LAVA and OpenQA servers into our infrastructure, and added the ability to quickly spin up LAVA and OpenQA workers capable of interacting with different types of hardware, our aim is to extend our kernel testing capabilities for more architectures. One example of this is that we now also perform LTS kernel testing on the MIPS Creator CI20 platform. The results of this testing are available, just as they are for our RPi4 and RISC-V Unmatched testing pipelines, from our public OpenQA server. Eventually these results could be provided to the KernelCI project.
- Using Git LFS and fast-import together
- Testing in a Box: Streamlining Embedded Systems Testing
- SDV Europe: What Codethink has planned
- How do Hardware Security Modules impact the automotive sector? The final blog in a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part two of a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part one of a three part discussion
- Automated end-to-end testing for Android Automotive on Hardware
- GUADEC 2023
- Embedded Open Source Summit 2023
- RISC-V: exploring a bug in stack unwinding
- Adding RISC-V Vector Cryptography Extension support to QEMU
- Introducing Our New Open-Source Tool: Quality Assurance Daemon
- Long Term Maintainability
- FOSDEM 2023
- Think before you Pip
- BuildStream 2.0 is here, just in time for the holidays!
- A Valuable & Comprehensive Firmware Code Review by Codethink
- GNOME OS & Atomic Upgrades on the PinePhone
- Flathub-Codethink Collaboration
- Codethink proudly sponsors GUADEC 2022
- Tracking Down an Obscure Reproducibility Bug in glibc
- Web app test automation with `cdt`
- FOSDEM Testing and Automation talk
- Protecting your project from dependency access problems
- Porting GNOME OS to Microchip's PolarFire Icicle Kit
- YAML Schemas: Validating Data without Writing Code
- Deterministic Construction Service
- Codethink becomes a Microchip Design Partner
- Hamsa: Using an NVIDIA Jetson Development Kit to create a fully open-source Robot Nano Hand
- Using STPA with software-intensive systems
- Codethink achieves ISO 26262 ASIL D Tool Certification
- RISC-V: running GNOME OS on SiFive hardware for the first time
- Automated Linux kernel testing
- Native compilation on Arm servers is so much faster now
- Higher quality of FOSS: How we are helping GNOME to improve their test pipeline
- RISC-V: A Small Hardware Project
- Why aligning with open source mainline is the way to go
- Build Meetup 2021: The BuildTeam Community Event
- A new approach to software safety
- Does the "Hypocrite Commits" incident prove that Linux is unsafe?
- ABI Stability in freedesktop-sdk
- Why your organisation needs to embrace working in the open-source ecosystem
- RISC-V User space access Oops
- Tracking Players at the Edge: An Overview
- What is Remote Asset API?
- Running a devroom at FOSDEM: Safety and Open Source
- Meet the codethings: Understanding BuildGrid and BuildBox with Beth White
- Streamlining Terraform configuration with Jsonnet
- Bloodlight: Designing a Heart Rate Sensor with STM32, LEDs and Photodiode
- Making the tech industry more inclusive for women
- Bloodlight Case Design: Lessons Learned
- Safety is a system property, not a software property
- RISC-V: Codethink's first research about the open instruction set
- Meet the Codethings: Safety-critical systems and the benefits of STPA with Shaun Mooney
- Why Project Managers are essential in an effective software consultancy
- FOSDEM 2021: Devroom for Safety and Open Source
- Meet the Codethings: Ben Dooks talks about Linux kernel and RISC-V
- Here we go 2021: 4 open source events for software engineers and project leaders
- Xmas Greetings from Codethink
- Call for Papers: FOSDEM 2021 Dev Room Safety and Open Source Software
- Building the abseil-hello Bazel project for a different architecture using a dynamically generated toolchain
- Advent of Code: programming puzzle challenges
- Full archive