Once upon a time, you could design and build a device, flash a well-engineered firmware onto it and support it for 20 years, with minimal changes to the core software. Those days are over. A new approach is needed to provide long-term support for devices while managing the complexity of today’s embedded software stacks.
Programming has become harder in the last 25 years. Back in 1999, in the embedded software world, device firmwares were coded from scratch by small teams who could understand every codepath. Even the top-selling video game of ‘99 (RollerCoaster Tycoon) was coded by a single software developer working in x86 assembly language.
Today that is unthinkable. For all but the smallest microcontroller platforms, it’s no longer possible for a small team to understand every codepath. There’s no one person alive who understands all 30 million lines of code that make up the Linux kernel, never mind the additional software used to build, compile and deploy it. The techniques we use to provide “long term support” need to adapt accordingly.
Today’s software is developed by larger teams. In open source projects like Linux, the “team” is relatively informal and works largely in public. And in nearly all cases they are working on the latest versions of the project’s code. Nobody is volunteering much time to look at 10 or 20 year old versions of Linux. Those that do, are highly paid specialists, and even they can only do so much in terms of backporting fixes for today’s highly complex security issues, to kernels from the last decade.
That means, if you’re shipping a device with an old version of Linux, there are many codepaths on your device that nobody really understands any more. How are you going to provide real long-term support for a software stack that nobody fully understands?
Make your project Long Term Maintainable
To provide long-term support for a modern device, your software and your development processes must be Long Term Maintainable. This means being able to regularly update the software to modern versions which are actively maintained and understood by the development communities working on them. For devices running Linux, this means regularly updating to a modern version of Linux. As of 2023, that’s how you ensure the device is as secure and reliable as possible.
At Codethink, a core part of our business is helping customers to get to a point where they can fearlessly build, test and upgrade device firmwares. We’ve been researching and recommending best practices since 2008, when we first started helping customers with embedded device issues.
Achieving Long Term Maintainability is a matter of improving your development, testing and release processes. Common things we aim to polish:
- The build process must be fast, reproducible, and flexible. Your build inputs will probably change significantly during the lifetime of the device. (The ideal is to support distributed build).
- The test pipelines must be fully automated and cover every aspect of device functionality, so component upgrades can be done fearlessly.
- The device upgrade mechanism must be fast, atomic and capable of rolling back to older versions if needed.
- The team should proactively monitor security issues reported in upstream projects.
- The team should work together with upstream projects to test new versions of open source dependencies as they’re developed, and catch issues that affect your project before they hit stable releases. (See: "Why aligning with open source mainline is the way to go")
- All third party source code should be mirrored locally, so you can be confident the inputs to your build process will not disappear over time.
All of this can be automated as part of a well designed continuous integration pipeline, so project engineers are not wasting time on repetitive tasks.
What tools should you use?
We recommend using the tools that best fit your project. There are many great open source tools that can help you today. For example, we've demonstrated atomic upgrades on a phone using OSTree, and kernel testing using OpenQA and Lava.
That said, we do have preferences. In a few cases, where we wanted to try a fundamentally different approach to existing tools, we’ve built our own open source tools. Apache BuildStream is a great example where we saw a need for an integration tool that took repeatability and caching to the next level. BuildGrid is another example.
We've written extensively about Long Term Maintainability - here's a list of all historical posts about LTM. In the coming months we have several articles and exciting announcements. Follow us on LinkedIn, Mastodon or Twitter so you don't miss anything.
Contact Codethink, and find out how we can help ensure that your project is Long Term Maintainable.
- Using Git LFS and fast-import together
- Testing in a Box: Streamlining Embedded Systems Testing
- SDV Europe: What Codethink has planned
- How do Hardware Security Modules impact the automotive sector? The final blog in a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part two of a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part one of a three part discussion
- Automated Kernel Testing on RISC-V Hardware
- Automated end-to-end testing for Android Automotive on Hardware
- GUADEC 2023
- Embedded Open Source Summit 2023
- RISC-V: exploring a bug in stack unwinding
- Adding RISC-V Vector Cryptography Extension support to QEMU
- Introducing Our New Open-Source Tool: Quality Assurance Daemon
- FOSDEM 2023
- Think before you Pip
- BuildStream 2.0 is here, just in time for the holidays!
- A Valuable & Comprehensive Firmware Code Review by Codethink
- GNOME OS & Atomic Upgrades on the PinePhone
- Flathub-Codethink Collaboration
- Codethink proudly sponsors GUADEC 2022
- Tracking Down an Obscure Reproducibility Bug in glibc
- Web app test automation with `cdt`
- FOSDEM Testing and Automation talk
- Protecting your project from dependency access problems
- Porting GNOME OS to Microchip's PolarFire Icicle Kit
- YAML Schemas: Validating Data without Writing Code
- Deterministic Construction Service
- Codethink becomes a Microchip Design Partner
- Hamsa: Using an NVIDIA Jetson Development Kit to create a fully open-source Robot Nano Hand
- Using STPA with software-intensive systems
- Codethink achieves ISO 26262 ASIL D Tool Certification
- RISC-V: running GNOME OS on SiFive hardware for the first time
- Automated Linux kernel testing
- Native compilation on Arm servers is so much faster now
- Higher quality of FOSS: How we are helping GNOME to improve their test pipeline
- RISC-V: A Small Hardware Project
- Why aligning with open source mainline is the way to go
- Build Meetup 2021: The BuildTeam Community Event
- A new approach to software safety
- Does the "Hypocrite Commits" incident prove that Linux is unsafe?
- ABI Stability in freedesktop-sdk
- Why your organisation needs to embrace working in the open-source ecosystem
- RISC-V User space access Oops
- Tracking Players at the Edge: An Overview
- What is Remote Asset API?
- Running a devroom at FOSDEM: Safety and Open Source
- Meet the codethings: Understanding BuildGrid and BuildBox with Beth White
- Streamlining Terraform configuration with Jsonnet
- Bloodlight: Designing a Heart Rate Sensor with STM32, LEDs and Photodiode
- Making the tech industry more inclusive for women
- Bloodlight Case Design: Lessons Learned
- Safety is a system property, not a software property
- RISC-V: Codethink's first research about the open instruction set
- Meet the Codethings: Safety-critical systems and the benefits of STPA with Shaun Mooney
- Why Project Managers are essential in an effective software consultancy
- FOSDEM 2021: Devroom for Safety and Open Source
- Meet the Codethings: Ben Dooks talks about Linux kernel and RISC-V
- Here we go 2021: 4 open source events for software engineers and project leaders
- Xmas Greetings from Codethink
- Call for Papers: FOSDEM 2021 Dev Room Safety and Open Source Software
- Building the abseil-hello Bazel project for a different architecture using a dynamically generated toolchain
- Advent of Code: programming puzzle challenges
- Full archive