Codethink has been developing a number of tools and processes in order to automate as much of the testing pipeline for embedded systems as possible. This includes everything from virtual hardware using QEMU, to testing on the hardware itself.
We have found that testing is often a neglected aspect of projects. It is often left to the late stages of development, leaving little time to fix problems that are found before a product goes to market. This in turn creates a stressful environment, causing “sticking plaster” fixes to be put in place rather than fixing issues at the root cause.
Testing is commonly a very manual process with teams of people scrolling through menus and performing actions on the device under test. This is both time consuming and expensive. It can also lead to the duplication of issues, unclear bug reporting and a lack of technical data to back the issue. By adding automated testing into the CI framework from the beginning of development, issues can be found and fixed early. The root cause is more easily identified, and fixed more efficiently, in a less stressful environment. This results in a more stable product with less technical debt.
We've used a number of different technologies on our journey including:
- Gitlab CI to run tests on daily builds or upon new merges
- openQA to run UI and graphics tests
- QAD to simulate keyboard, touch and gesture interactions
- CAN interfaces to simulate a vehicle on the road
- USB Switches to control multiple peripherals to the hardware under test
Typically, this has taken considerable time to set up. This can mean wading through the red tape of client's IT departments to gain access to client infrastructure, understand their build, CI and Test setup and integrate the tools required to provide useful data from testing.
Once set up and working, we've found that there are a significant number of dongles, adaptors, computers and cables hanging off the hardware under test. This increases the complexity and the chances of peripherals becoming accidentally disconnected, as well as looking untidy. The picture below can look familiar to some people reading this:
Whilst discussing a client project the conversation led to the question "Wouldn't it be great if we could have all of this in one box, easily configured to work with any hardware and without the need to access anything other than the hardware in question?"
So that is what we've done!
In order to tackle the issues of trailing cables and multiple peripherals, we have designed an IO board which facilitates serial I/O, I2c, SPI, GPIO, bluetooth/wifi, USB switching, a USB hub, Can bus emulation, HID emulation and even a host PC. As with the software, the box has been developed in a modular fashion, so you are able to only include the parts you need for a particular use case.
Also, we’ve our baseline software infrastructure defined in ansible scripts that we use/deploy in docker containers. This has been carried out with a modular approach, to give the flexibility to configure and deploy only what is required in a particular scenario.
To date, we’ve produced a first revision of the board, which had a number of issues which had to be rewired or otherwise resolved. There is a design for a Rev-B board which has been manufactured and tested. We are currently using this version in a variety of internal projects.
There are plans for a version 2 board which will provide the same functionality but with less components that is even cheaper to produce! In addition, we also have plans to add a power distribution module to the box to allow us to control the power to multiple devices under test in a board farm scenario.
Currently, the tools to configure the hardware run as application specific CLI tools. We have plans to write an abstraction layer to bring these tools into a single user interface, which, will allow the user to configure the hardware from a single window.
Get in touch!
The project is entirely open source and is available here.
Documentation for the project is available here.
If you’re interested in learning more about this project and how we can help you, please reach out to us via email@example.com.
- Lessons learnt from building a distributed system in Rust
- FOSDEM 2024
- Introducing Web UI QAnvas and new features of Quality Assurance Daemon
- Outreachy: Supporting the open source community through mentorship programmes
- Using Git LFS and fast-import together
- SDV Europe: What Codethink has planned
- How do Hardware Security Modules impact the automotive sector? The final blog in a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part two of a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part one of a three part discussion
- Automated Kernel Testing on RISC-V Hardware
- Automated end-to-end testing for Android Automotive on Hardware
- GUADEC 2023
- Embedded Open Source Summit 2023
- RISC-V: Exploring a Bug in Stack Unwinding
- Adding RISC-V Vector Cryptography Extension support to QEMU
- Introducing Our New Open-Source Tool: Quality Assurance Daemon
- Long Term Maintainability
- FOSDEM 2023
- Think before you Pip
- BuildStream 2.0 is here, just in time for the holidays!
- A Valuable & Comprehensive Firmware Code Review by Codethink
- GNOME OS & Atomic Upgrades on the PinePhone
- Flathub-Codethink Collaboration
- Codethink proudly sponsors GUADEC 2022
- Tracking Down an Obscure Reproducibility Bug in glibc
- Web app test automation with `cdt`
- FOSDEM Testing and Automation talk
- Protecting your project from dependency access problems
- Porting GNOME OS to Microchip's PolarFire Icicle Kit
- YAML Schemas: Validating Data without Writing Code
- Deterministic Construction Service
- Codethink becomes a Microchip Design Partner
- Hamsa: Using an NVIDIA Jetson Development Kit to create a fully open-source Robot Nano Hand
- Using STPA with software-intensive systems
- Codethink achieves ISO 26262 ASIL D Tool Certification
- RISC-V: running GNOME OS on SiFive hardware for the first time
- Automated Linux kernel testing
- Native compilation on Arm servers is so much faster now
- Higher quality of FOSS: How we are helping GNOME to improve their test pipeline
- RISC-V: A Small Hardware Project
- Why aligning with open source mainline is the way to go
- Build Meetup 2021: The BuildTeam Community Event
- A new approach to software safety
- Does the "Hypocrite Commits" incident prove that Linux is unsafe?
- ABI Stability in freedesktop-sdk
- Why your organisation needs to embrace working in the open-source ecosystem
- RISC-V User space access Oops
- Tracking Players at the Edge: An Overview
- What is Remote Asset API?
- Running a devroom at FOSDEM: Safety and Open Source
- Meet the codethings: Understanding BuildGrid and BuildBox with Beth White
- Streamlining Terraform configuration with Jsonnet
- Bloodlight: Designing a Heart Rate Sensor with STM32, LEDs and Photodiode
- Making the tech industry more inclusive for women
- Bloodlight Case Design: Lessons Learned
- Safety is a system property, not a software property
- Full archive