Codethink has automated some functional testing of Android Automotive on our public openQA instance. These tests run on a Raspberry Pi 4 and we intend to automate the build and flash process, so that we have the possibility to add them to CI pipelines on any given project. See an example test run here.
Why does this testing matter?
At Codethink, over the past few years we have been focusing on how we can improve the state-of-the-art for testing. We’re big fans of continuous integration, where we want to constantly integrate and test the latest version of every component together at the same time, in order to catch issues or conflicts as early as possible, when they are less expensive to fix. But, whilst constant build/compile testing is incredibly useful on its own, when it comes to software testing, CI pipelines will all too often only run unit tests, and teams overlook the need for functional testing that verifies the whole system, where we replicate both the end-user environment and also how the user will actually interact with the software. We call this end-to-end testing.
Currently, to achieve the level of confidence required, many large projects are reliant on manual testing, with some having hundreds - even thousands - of testers. Whilst some manual testing will always be needed, we really want to be able to automate as much as possible, making more efficient use of the hardware resources (overnight, for example) and freeing up human beings to do more interesting and more valuable testing.
So we are not trying to replace unit testing: we are trying to replace manual end-to-end testing as much as we can, because we believe projects should be performing robust, valuable testing as early as possible in the development process.
To achieve this, we turned to openQA, an open source tool developed by SUSE, originally designed for end-to-end testing of desktop Linux distributions. We've built a number end-to-end test pipelines since then, some for our customers in the embedded and automotive world, and some public pipelines that provide extra testing for open source projects. We're continually testing mainline Linux on a set of development boards, using LAVA and openQA; we spoke about this at FOSDEM 2022. We developed QAD, a lightweight alternative to openQA's usual remote desktop backend, and an open-hardware USB Switch for automating tests that require physically connecting and disconnecting USB devices from a test rig, and we spoke at EOSS Prague 2023 about this work.
We’ve now added Android Automotive testing, using Android 13 from Snapp Automotive and a Raspberry Pi 4. This matters because a lot of our customers are turning to Android Automotive, and we want to be able to test that in CI from day one of the project, and easily add additional tests over time.
Next Steps
The tests are currently quite straightforward, and include booting to the home screen, navigating to the apps screen and then back to home. In the near future we want to add tests for CarPlay screen projection and USB Media devices.
We are also looking to automate the build and flashing process (currently manual), in order to allow us to frequently update to the newest version of Android Automotive. With this and the additional tests in place, we can usefully start to find and report bugs against the Snapp Automotive project.
Interested in learning more? Contact us at sales@codethink.co.uk.
Other Content
- Using Git LFS and fast-import together
- Testing in a Box: Streamlining Embedded Systems Testing
- SDV Europe: What Codethink has planned
- How do Hardware Security Modules impact the automotive sector? The final blog in a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part two of a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part one of a three part discussion
- Automated Kernel Testing on RISC-V Hardware
- GUADEC 2023
- Embedded Open Source Summit 2023
- RISC-V: exploring a bug in stack unwinding
- Adding RISC-V Vector Cryptography Extension support to QEMU
- Introducing Our New Open-Source Tool: Quality Assurance Daemon
- Long Term Maintainability
- FOSDEM 2023
- Think before you Pip
- BuildStream 2.0 is here, just in time for the holidays!
- A Valuable & Comprehensive Firmware Code Review by Codethink
- GNOME OS & Atomic Upgrades on the PinePhone
- Flathub-Codethink Collaboration
- Codethink proudly sponsors GUADEC 2022
- Tracking Down an Obscure Reproducibility Bug in glibc
- Web app test automation with `cdt`
- FOSDEM Testing and Automation talk
- Protecting your project from dependency access problems
- Porting GNOME OS to Microchip's PolarFire Icicle Kit
- YAML Schemas: Validating Data without Writing Code
- Deterministic Construction Service
- Codethink becomes a Microchip Design Partner
- Hamsa: Using an NVIDIA Jetson Development Kit to create a fully open-source Robot Nano Hand
- Using STPA with software-intensive systems
- Codethink achieves ISO 26262 ASIL D Tool Certification
- RISC-V: running GNOME OS on SiFive hardware for the first time
- Automated Linux kernel testing
- Native compilation on Arm servers is so much faster now
- Higher quality of FOSS: How we are helping GNOME to improve their test pipeline
- RISC-V: A Small Hardware Project
- Why aligning with open source mainline is the way to go
- Build Meetup 2021: The BuildTeam Community Event
- A new approach to software safety
- Does the "Hypocrite Commits" incident prove that Linux is unsafe?
- ABI Stability in freedesktop-sdk
- Why your organisation needs to embrace working in the open-source ecosystem
- RISC-V User space access Oops
- Tracking Players at the Edge: An Overview
- What is Remote Asset API?
- Running a devroom at FOSDEM: Safety and Open Source
- Meet the codethings: Understanding BuildGrid and BuildBox with Beth White
- Streamlining Terraform configuration with Jsonnet
- Bloodlight: Designing a Heart Rate Sensor with STM32, LEDs and Photodiode
- Making the tech industry more inclusive for women
- Bloodlight Case Design: Lessons Learned
- Safety is a system property, not a software property
- RISC-V: Codethink's first research about the open instruction set
- Meet the Codethings: Safety-critical systems and the benefits of STPA with Shaun Mooney
- Why Project Managers are essential in an effective software consultancy
- FOSDEM 2021: Devroom for Safety and Open Source
- Meet the Codethings: Ben Dooks talks about Linux kernel and RISC-V
- Here we go 2021: 4 open source events for software engineers and project leaders
- Xmas Greetings from Codethink
- Call for Papers: FOSDEM 2021 Dev Room Safety and Open Source Software
- Building the abseil-hello Bazel project for a different architecture using a dynamically generated toolchain
- Advent of Code: programming puzzle challenges
- Full archive