In the mid-2010s, Codethink developed the Baserock software suite for specifying, developing, and building Linux-based systems. The work has been used by several long-standing customers and Codethink even uses it for some of our own infrastructure. This blog will take you through work done to modernise one of the components of Baserock - Lorry.
Some of the components of Baserock are useful in their own right, and one of those is Lorry: a program for maintaining mirrors of source code. When building systems that include many third-party source components, it is important to make sure you keep copies of all the source code within your organisation. Otherwise, if an upstream repository or file server becomes temporarily or permanently unavailable, you might be unable to rebuild or deploy your system. You might also be unable to comply with copyleft license requirements to provide source to your users.
While some Git hosting services can maintain a mirror of an upstream Git repository, Lorry can mirror to Git from many other version control systems and from file-based upstream releases. The Lorry Controller (LC) component implements the services needed to run Lorry regularly for each mirror, and pulls the mirror configuration from another Git repository. One of my projects during lockdown has been modernising Lorry: making these components compatible with current versions of their dependencies and with various Git hosting services, documenting how to install and configure them, and building container images.
The earlier Trovekube project began the work of getting Lorry and LC running on Python 3, and added GitLab CI configurations which have helped me a lot with understanding their dependencies. As I tested out all their features, I found and fixed a few more issues with Python 3 compatibility, mostly involving distinctions between binary and text I/O.
As part of a Baserock Trove appliance, LC was originally co-installed with the Gitano hosting service (downstream host), and had specific support for mirroring the whole of another Trove (upstream host). Later on, support for Gerrit as downstream host, and for GitLab as downstream or upstream host, was added, but I found that this support had bit-rotted. There were also no internal abstractions that would allow adding another type of host cleanly. I have refactored support for each host type into classes, updated the Gerrit and GitLab support to work with current versions of those programs, and added support for Gitea as a downstream host. It turned out to be possible to run a test instance of each of these hosting services in a VM or system container on my development laptop, though it wasn't always the most comfortable experience!
The documentation for Lorry and LC now lists all their direct dependencies and the versions I've tested. For LC I documented all the installation steps and how to integrate it with each of the supported downstream host types.
Finally I needed to build container images using BuildStream. This part was probably the most challenging, since I had no prior experience with either technology. BuildStream is also a fast-moving project that hasn't been kept up-to-date in Debian, my usual development environment. I used the experimental "oci" plugin that can build container images in the OCI or Docker v1.2 format.
I found BuildStream's tutorial and documentation generally excellent, and was impressed at how easy it was to write definitions for building a component. Still, defining how to build all of Lorry and LC's dependencies would have required a huge amount of work. However, the Freedesktop-SDK already has BuildStream definitions for many common open source components, even including Git, OpenSSH, and Perl (used in some of Lorry's file importers). My colleague, Valentin David, helped me to understand how to reuse those, so that I only had to write definitions for the more obscure dependencies.
While my initial testing of Lorry and LC was done on Debian 9 "stretch", the last version where I could easily test all the supported version control systems, Freedesktop-SDK has much more recent versions of some of the dependencies including Git. The current version of Git creates additional files under
.git that weren't expected by Lorry's test suite, but it was easy enough to fix the affected tests.
After I built the first container image, I still needed to learn how to use container managers and to add configuration for various components to work properly in the container. This was not a typical application container, as LC depends on systemd to run multiple services together. I used podman initially, since I understood that it had better support for containers with systemd. BuildStream's build environments are sandboxes with only a (virtual) root user, so I had to defer creation of the lorry service user and its files to first boot of the container using systemd-sysusers and systemd-tmpfiles. Container images should not contain any security credentials, so I also arranged to create SSH key pairs at first boot and to propagate access tokens from the container environment into a LC configuration file. The biggest stumbling block was in logging. Container managers normally run a single application process and expect it to log through its stdout or stderr, but this isn't possible in a multi-service container. However, both podman and Docker can optionally create a
/dev/console device node that will also feed into their logs, and systemd-journald can be configured to log through this instead of to files.
This project was an interesting break from my usual work on kernel development and maintenance. I had the opportunity to explore and learn about various unfamiliar but important technologies. My colleagues helped a lot in getting me started and in reviewing my work toward the end. All of this work is now public under the CodethinkLabs group on GitLab.
- Using Git LFS and fast-import together
- Testing in a Box: Streamlining Embedded Systems Testing
- SDV Europe: What Codethink has planned
- How do Hardware Security Modules impact the automotive sector? The final blog in a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part two of a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part one of a three part discussion
- Automated Kernel Testing on RISC-V Hardware
- Automated end-to-end testing for Android Automotive on Hardware
- GUADEC 2023
- Embedded Open Source Summit 2023
- RISC-V: exploring a bug in stack unwinding
- Adding RISC-V Vector Cryptography Extension support to QEMU
- Introducing Our New Open-Source Tool: Quality Assurance Daemon
- Long Term Maintainability
- FOSDEM 2023
- Think before you Pip
- BuildStream 2.0 is here, just in time for the holidays!
- A Valuable & Comprehensive Firmware Code Review by Codethink
- GNOME OS & Atomic Upgrades on the PinePhone
- Flathub-Codethink Collaboration
- Codethink proudly sponsors GUADEC 2022
- Tracking Down an Obscure Reproducibility Bug in glibc
- Web app test automation with `cdt`
- FOSDEM Testing and Automation talk
- Protecting your project from dependency access problems
- Porting GNOME OS to Microchip's PolarFire Icicle Kit
- YAML Schemas: Validating Data without Writing Code
- Deterministic Construction Service
- Codethink becomes a Microchip Design Partner
- Hamsa: Using an NVIDIA Jetson Development Kit to create a fully open-source Robot Nano Hand
- Using STPA with software-intensive systems
- Codethink achieves ISO 26262 ASIL D Tool Certification
- RISC-V: running GNOME OS on SiFive hardware for the first time
- Automated Linux kernel testing
- Native compilation on Arm servers is so much faster now
- Higher quality of FOSS: How we are helping GNOME to improve their test pipeline
- RISC-V: A Small Hardware Project
- Why aligning with open source mainline is the way to go
- Build Meetup 2021: The BuildTeam Community Event
- A new approach to software safety
- Does the "Hypocrite Commits" incident prove that Linux is unsafe?
- ABI Stability in freedesktop-sdk
- Why your organisation needs to embrace working in the open-source ecosystem
- RISC-V User space access Oops
- Tracking Players at the Edge: An Overview
- What is Remote Asset API?
- Running a devroom at FOSDEM: Safety and Open Source
- Meet the codethings: Understanding BuildGrid and BuildBox with Beth White
- Streamlining Terraform configuration with Jsonnet
- Bloodlight: Designing a Heart Rate Sensor with STM32, LEDs and Photodiode
- Making the tech industry more inclusive for women
- Bloodlight Case Design: Lessons Learned
- Safety is a system property, not a software property
- RISC-V: Codethink's first research about the open instruction set
- Meet the Codethings: Safety-critical systems and the benefits of STPA with Shaun Mooney
- Why Project Managers are essential in an effective software consultancy
- FOSDEM 2021: Devroom for Safety and Open Source
- Meet the Codethings: Ben Dooks talks about Linux kernel and RISC-V
- Here we go 2021: 4 open source events for software engineers and project leaders
- Xmas Greetings from Codethink
- Call for Papers: FOSDEM 2021 Dev Room Safety and Open Source Software
- Building the abseil-hello Bazel project for a different architecture using a dynamically generated toolchain
- Advent of Code: programming puzzle challenges
- Full archive