Raumfeld is a multi room audio system (as the name, translated from German, suggests: "Raum" = "room", "feld" = "field"). Raumfeld began as a startup in Berlin and is now owned by Lautsprecher Teufel, a leading manufacturer of speakers (also based in Berlin).
The Raumfeld family of speakers all communicate over standard Wi-Fi. They are controlled remotely from Android and iOS devices. They can stream directly from various popular music-streaming services (i.e. the speaker connects to the service, not the phone). One speaker can stream to another over Wi-Fi, with automatic compensation for the time delay (the stereo pairs can still be wired together to give optimal quality). All this means that there has to be complex firmware running in those devices.
Codethink delivered this project for Raumfeld to improve the firmware build system, leaving the client’s own development resource to concentrate on customer facing activity.
The Original Build System
Raumfeld have been shipping devices since 2009. Some of these are now obsolete and are no longer sold but all of them are still supported. There are 3 different machine architectures that the firmware has to target and multiple devices using each architecture. Did I mention that each device image actually consists of multiple Buildroot images? As well as a filesystem image for each device, there is an installer image for each device that deploys the real filesystem. This leads to a total of 15 different Buildroot builds to manage.
Developers were able to do local, incremental builds quite quickly to test their changes but to do a clean rebuild of the firmware for all platforms took a whole 8 hours. This meant that the test and release process was majorly inconvenienced.
Easy Wins
An obvious route to faster builds is to use a faster computer to build on. Teufel replaced the aging build server they were using with a new machine which reduced build times to just under 4 hours.
Another quick win when doing multiple Buildroot builds is to build the toolchain once as a separate step and then use the external toolchain backend in subsequent builds. The toolchain takes about 15 minutes to build and we went from building eleven toolchains to three: one for each architecture in use. This got build time for a full rebuild down to under 2 hours.
However, this also added 3 more Buildroot targets into the build process, leading to a total of 18. The overall build was driven by a couple of simple shell scripts which knew nothing about the dependencies between the tasks they were running. It was also easy to accidentally overwrite one build with another as they all took place in the same buildroot.git clone. The next step was to get a handle on this complexity.
Buildroot.cmake
The core functionality of Raumfeld’s devices is provided by a set of C and C++ modules developed internally. Codethink did the work of converting the build systems for these modules to CMake. Previously they used a mix of GNU Autotools and hand-written Makefiles. The move to CMake meant one consistent syntax for all of the build instructions, as well as much better IDE integration and the ability to do incremental builds across all of the modules.
Having used CMake there, we looked to CMake again to create a 'toplevel' build system that would manage all of the Buildroot builds. CMake's forte is generating 'low level' build systems that run compilers and linkers directly. But there is a precedent for using CMake as a “meta” build system, for example the ExternalProject module.
What we ended up with was Buildroot.cmake, a module that helps you drive Buildroot builds from CMake. Here’s a simple example of the CMakeLists.txt file building a toolchain with Buildroot, then using that toolchain to build a rootfs:
buildroot_toolchain(
buildroot-toolchain-arm
CONFIG arm-toolchain.config
)
buildroot_target(
buildroot-arm-device
CONFIG arm-device.config
OUTPUT images/rootfs.tar.gz
TOOLCHAIN buildroot-toolchain-arm
)
Using the Buildroot.cmake module, we could remove the shell scripts that hardcoded a specific build order and replace it with a CMakeLists.txt file that is a bit longer, but is explicit and exact about the inputs and outputs of the build process and the interdependencies. Each Buildroot build also runs as an out-of-tree build, so there’s no risk of mixing two targets together and having to start from scratch.
This clarity was vital for our further work on reducing the time spent doing clean rebuilds of the firmware. We had more or less reached the limit of optimising build times. The next step was to implement a reliable caching mechanism, so we could avoid building something altogether if it hadn’t changed. We will talk about this in our next article.
Other Content
- Using Git LFS and fast-import together
- Testing in a Box: Streamlining Embedded Systems Testing
- SDV Europe: What Codethink has planned
- How do Hardware Security Modules impact the automotive sector? The final blog in a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part two of a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part one of a three part discussion
- Automated Kernel Testing on RISC-V Hardware
- Automated end-to-end testing for Android Automotive on Hardware
- GUADEC 2023
- Embedded Open Source Summit 2023
- RISC-V: exploring a bug in stack unwinding
- Adding RISC-V Vector Cryptography Extension support to QEMU
- Introducing Our New Open-Source Tool: Quality Assurance Daemon
- Long Term Maintainability
- FOSDEM 2023
- Think before you Pip
- BuildStream 2.0 is here, just in time for the holidays!
- A Valuable & Comprehensive Firmware Code Review by Codethink
- GNOME OS & Atomic Upgrades on the PinePhone
- Flathub-Codethink Collaboration
- Codethink proudly sponsors GUADEC 2022
- Tracking Down an Obscure Reproducibility Bug in glibc
- Web app test automation with `cdt`
- FOSDEM Testing and Automation talk
- Protecting your project from dependency access problems
- Porting GNOME OS to Microchip's PolarFire Icicle Kit
- YAML Schemas: Validating Data without Writing Code
- Deterministic Construction Service
- Codethink becomes a Microchip Design Partner
- Hamsa: Using an NVIDIA Jetson Development Kit to create a fully open-source Robot Nano Hand
- Using STPA with software-intensive systems
- Codethink achieves ISO 26262 ASIL D Tool Certification
- RISC-V: running GNOME OS on SiFive hardware for the first time
- Automated Linux kernel testing
- Native compilation on Arm servers is so much faster now
- Higher quality of FOSS: How we are helping GNOME to improve their test pipeline
- RISC-V: A Small Hardware Project
- Why aligning with open source mainline is the way to go
- Build Meetup 2021: The BuildTeam Community Event
- A new approach to software safety
- Does the "Hypocrite Commits" incident prove that Linux is unsafe?
- ABI Stability in freedesktop-sdk
- Why your organisation needs to embrace working in the open-source ecosystem
- RISC-V User space access Oops
- Tracking Players at the Edge: An Overview
- What is Remote Asset API?
- Running a devroom at FOSDEM: Safety and Open Source
- Meet the codethings: Understanding BuildGrid and BuildBox with Beth White
- Streamlining Terraform configuration with Jsonnet
- Bloodlight: Designing a Heart Rate Sensor with STM32, LEDs and Photodiode
- Making the tech industry more inclusive for women
- Bloodlight Case Design: Lessons Learned
- Safety is a system property, not a software property
- RISC-V: Codethink's first research about the open instruction set
- Meet the Codethings: Safety-critical systems and the benefits of STPA with Shaun Mooney
- Why Project Managers are essential in an effective software consultancy
- FOSDEM 2021: Devroom for Safety and Open Source
- Meet the Codethings: Ben Dooks talks about Linux kernel and RISC-V
- Here we go 2021: 4 open source events for software engineers and project leaders
- Xmas Greetings from Codethink
- Call for Papers: FOSDEM 2021 Dev Room Safety and Open Source Software
- Building the abseil-hello Bazel project for a different architecture using a dynamically generated toolchain
- Advent of Code: programming puzzle challenges
- Full archive