Codethink has automated some functional testing of Android Automotive on our public openQA instance. These tests run on a Raspberry Pi 4 and we intend to automate the build and flash process, so that we have the possibility to add them to CI pipelines on any given project. See an example test run here.
Why does this testing matter?
At Codethink, over the past few years we have been focusing on how we can improve the state-of-the-art for testing. We’re big fans of continuous integration, where we want to constantly integrate and test the latest version of every component together at the same time, in order to catch issues or conflicts as early as possible, when they are less expensive to fix. But, whilst constant build/compile testing is incredibly useful on its own, when it comes to software testing, CI pipelines will all too often only run unit tests, and teams overlook the need for functional testing that verifies the whole system, where we replicate both the end-user environment and also how the user will actually interact with the software. We call this end-to-end testing.
Currently, to achieve the level of confidence required, many large projects are reliant on manual testing, with some having hundreds - even thousands - of testers. Whilst some manual testing will always be needed, we really want to be able to automate as much as possible, making more efficient use of the hardware resources (overnight, for example) and freeing up human beings to do more interesting and more valuable testing.
So we are not trying to replace unit testing: we are trying to replace manual end-to-end testing as much as we can, because we believe projects should be performing robust, valuable testing as early as possible in the development process.
To achieve this, we turned to openQA, an open source tool developed by SUSE, originally designed for end-to-end testing of desktop Linux distributions. We've built a number end-to-end test pipelines since then, some for our customers in the embedded and automotive world, and some public pipelines that provide extra testing for open source projects. We're continually testing mainline Linux on a set of development boards, using LAVA and openQA; we spoke about this at FOSDEM 2022. We developed QAD, a lightweight alternative to openQA's usual remote desktop backend, and an open-hardware USB Switch for automating tests that require physically connecting and disconnecting USB devices from a test rig, and we spoke at EOSS Prague 2023 about this work.
We’ve now added Android Automotive testing, using Android 13 from Snapp Automotive and a Raspberry Pi 4. This matters because a lot of our customers are turning to Android Automotive, and we want to be able to test that in CI from day one of the project, and easily add additional tests over time.
Next Steps
The tests are currently quite straightforward, and include booting to the home screen, navigating to the apps screen and then back to home. In the near future we want to add tests for CarPlay screen projection and USB Media devices.
We are also looking to automate the build and flashing process (currently manual), in order to allow us to frequently update to the newest version of Android Automotive. With this and the additional tests in place, we can usefully start to find and report bugs against the Snapp Automotive project.
Interested in learning more? Contact us at sales@codethink.co.uk.
Other Content
- Open Source Summit Europe (OSSEU) 2024
- Watch: Real-time Scheduling Fault Simulation
- Improving systemd’s integration testing infrastructure (part 2)
- Meet the Team: Laurence Urhegyi
- A new way to develop on Linux - Part II
- GUADEC 2024
- Developing a cryptographically secure bootloader for RISC-V in Rust
- Meet the Team: Philip Martin
- Improving systemd’s integration testing infrastructure (part 1)
- A new way to develop on Linux
- RISC-V Summit Europe 2024
- Safety Frontier: A Retrospective on ELISA
- Codethink sponsors Outreachy
- The Linux kernel is a CNA - so what?
- GNOME OS + systemd-sysupdate
- Codethink has achieved ISO 9001:2015 accreditation
- Outreachy internship: Improving end-to-end testing for GNOME
- Lessons learnt from building a distributed system in Rust
- FOSDEM 2024
- Introducing Web UI QAnvas and new features of Quality Assurance Daemon
- Outreachy: Supporting the open source community through mentorship programmes
- Using Git LFS and fast-import together
- Testing in a Box: Streamlining Embedded Systems Testing
- SDV Europe: What Codethink has planned
- How do Hardware Security Modules impact the automotive sector? The final blog in a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part two of a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part one of a three part discussion
- Automated Kernel Testing on RISC-V Hardware
- GUADEC 2023
- Embedded Open Source Summit 2023
- RISC-V: Exploring a Bug in Stack Unwinding
- Adding RISC-V Vector Cryptography Extension support to QEMU
- Introducing Our New Open-Source Tool: Quality Assurance Daemon
- Long Term Maintainability
- FOSDEM 2023
- Think before you Pip
- BuildStream 2.0 is here, just in time for the holidays!
- A Valuable & Comprehensive Firmware Code Review by Codethink
- GNOME OS & Atomic Upgrades on the PinePhone
- Flathub-Codethink Collaboration
- Codethink proudly sponsors GUADEC 2022
- Tracking Down an Obscure Reproducibility Bug in glibc
- Web app test automation with `cdt`
- FOSDEM Testing and Automation talk
- Protecting your project from dependency access problems
- Porting GNOME OS to Microchip's PolarFire Icicle Kit
- YAML Schemas: Validating Data without Writing Code
- Deterministic Construction Service
- Codethink becomes a Microchip Design Partner
- Hamsa: Using an NVIDIA Jetson Development Kit to create a fully open-source Robot Nano Hand
- Using STPA with software-intensive systems
- Codethink achieves ISO 26262 ASIL D Tool Certification
- Full archive