In the mid-2010s, Codethink developed the Baserock software suite for specifying, developing, and building Linux-based systems. The work has been used by several long-standing customers and Codethink even uses it for some of our own infrastructure. This blog will take you through work done to modernise one of the components of Baserock - Lorry.
Some of the components of Baserock are useful in their own right, and one of those is Lorry: a program for maintaining mirrors of source code. When building systems that include many third-party source components, it is important to make sure you keep copies of all the source code within your organisation. Otherwise, if an upstream repository or file server becomes temporarily or permanently unavailable, you might be unable to rebuild or deploy your system. You might also be unable to comply with copyleft license requirements to provide source to your users.
While some Git hosting services can maintain a mirror of an upstream Git repository, Lorry can mirror to Git from many other version control systems and from file-based upstream releases. The Lorry Controller (LC) component implements the services needed to run Lorry regularly for each mirror, and pulls the mirror configuration from another Git repository. One of my projects during lockdown has been modernising Lorry: making these components compatible with current versions of their dependencies and with various Git hosting services, documenting how to install and configure them, and building container images.
The earlier Trovekube project began the work of getting Lorry and LC running on Python 3, and added GitLab CI configurations which have helped me a lot with understanding their dependencies. As I tested out all their features, I found and fixed a few more issues with Python 3 compatibility, mostly involving distinctions between binary and text I/O.
As part of a Baserock Trove appliance, LC was originally co-installed with the Gitano hosting service (downstream host), and had specific support for mirroring the whole of another Trove (upstream host). Later on, support for Gerrit as downstream host, and for GitLab as downstream or upstream host, was added, but I found that this support had bit-rotted. There were also no internal abstractions that would allow adding another type of host cleanly. I have refactored support for each host type into classes, updated the Gerrit and GitLab support to work with current versions of those programs, and added support for Gitea as a downstream host. It turned out to be possible to run a test instance of each of these hosting services in a VM or system container on my development laptop, though it wasn't always the most comfortable experience!
The documentation for Lorry and LC now lists all their direct dependencies and the versions I've tested. For LC I documented all the installation steps and how to integrate it with each of the supported downstream host types.
Finally I needed to build container images using BuildStream. This part was probably the most challenging, since I had no prior experience with either technology. BuildStream is also a fast-moving project that hasn't been kept up-to-date in Debian, my usual development environment. I used the experimental "oci" plugin that can build container images in the OCI or Docker v1.2 format.
I found BuildStream's tutorial and documentation generally excellent, and was impressed at how easy it was to write definitions for building a component. Still, defining how to build all of Lorry and LC's dependencies would have required a huge amount of work. However, the Freedesktop-SDK already has BuildStream definitions for many common open source components, even including Git, OpenSSH, and Perl (used in some of Lorry's file importers). My colleague, Valentin David, helped me to understand how to reuse those, so that I only had to write definitions for the more obscure dependencies.
While my initial testing of Lorry and LC was done on Debian 9 "stretch", the last version where I could easily test all the supported version control systems, Freedesktop-SDK has much more recent versions of some of the dependencies including Git. The current version of Git creates additional files under .git
that weren't expected by Lorry's test suite, but it was easy enough to fix the affected tests.
After I built the first container image, I still needed to learn how to use container managers and to add configuration for various components to work properly in the container. This was not a typical application container, as LC depends on systemd to run multiple services together. I used podman initially, since I understood that it had better support for containers with systemd. BuildStream's build environments are sandboxes with only a (virtual) root user, so I had to defer creation of the lorry service user and its files to first boot of the container using systemd-sysusers and systemd-tmpfiles. Container images should not contain any security credentials, so I also arranged to create SSH key pairs at first boot and to propagate access tokens from the container environment into a LC configuration file. The biggest stumbling block was in logging. Container managers normally run a single application process and expect it to log through its stdout or stderr, but this isn't possible in a multi-service container. However, both podman and Docker can optionally create a /dev/console
device node that will also feed into their logs, and systemd-journald can be configured to log through this instead of to files.
This project was an interesting break from my usual work on kernel development and maintenance. I had the opportunity to explore and learn about various unfamiliar but important technologies. My colleagues helped a lot in getting me started and in reviewing my work toward the end. All of this work is now public under the CodethinkLabs group on GitLab.
Other Content
- Open Source Summit Europe (OSSEU) 2024
- Watch: Real-time Scheduling Fault Simulation
- Improving systemd’s integration testing infrastructure (part 2)
- Meet the Team: Laurence Urhegyi
- A new way to develop on Linux - Part II
- GUADEC 2024
- Developing a cryptographically secure bootloader for RISC-V in Rust
- Meet the Team: Philip Martin
- Improving systemd’s integration testing infrastructure (part 1)
- A new way to develop on Linux
- RISC-V Summit Europe 2024
- Safety Frontier: A Retrospective on ELISA
- Codethink sponsors Outreachy
- The Linux kernel is a CNA - so what?
- GNOME OS + systemd-sysupdate
- Codethink has achieved ISO 9001:2015 accreditation
- Outreachy internship: Improving end-to-end testing for GNOME
- Lessons learnt from building a distributed system in Rust
- FOSDEM 2024
- Introducing Web UI QAnvas and new features of Quality Assurance Daemon
- Outreachy: Supporting the open source community through mentorship programmes
- Using Git LFS and fast-import together
- Testing in a Box: Streamlining Embedded Systems Testing
- SDV Europe: What Codethink has planned
- How do Hardware Security Modules impact the automotive sector? The final blog in a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part two of a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part one of a three part discussion
- Automated Kernel Testing on RISC-V Hardware
- Automated end-to-end testing for Android Automotive on Hardware
- GUADEC 2023
- Embedded Open Source Summit 2023
- RISC-V: Exploring a Bug in Stack Unwinding
- Adding RISC-V Vector Cryptography Extension support to QEMU
- Introducing Our New Open-Source Tool: Quality Assurance Daemon
- Long Term Maintainability
- FOSDEM 2023
- Think before you Pip
- BuildStream 2.0 is here, just in time for the holidays!
- A Valuable & Comprehensive Firmware Code Review by Codethink
- GNOME OS & Atomic Upgrades on the PinePhone
- Flathub-Codethink Collaboration
- Codethink proudly sponsors GUADEC 2022
- Tracking Down an Obscure Reproducibility Bug in glibc
- Web app test automation with `cdt`
- FOSDEM Testing and Automation talk
- Protecting your project from dependency access problems
- Porting GNOME OS to Microchip's PolarFire Icicle Kit
- YAML Schemas: Validating Data without Writing Code
- Deterministic Construction Service
- Codethink becomes a Microchip Design Partner
- Hamsa: Using an NVIDIA Jetson Development Kit to create a fully open-source Robot Nano Hand
- Using STPA with software-intensive systems
- Codethink achieves ISO 26262 ASIL D Tool Certification
- Full archive