In the mid-2010s, Codethink developed the Baserock software suite for specifying, developing, and building Linux-based systems. The work has been used by several long-standing customers and Codethink even uses it for some of our own infrastructure. This blog will take you through work done to modernise one of the components of Baserock - Lorry.
Some of the components of Baserock are useful in their own right, and one of those is Lorry: a program for maintaining mirrors of source code. When building systems that include many third-party source components, it is important to make sure you keep copies of all the source code within your organisation. Otherwise, if an upstream repository or file server becomes temporarily or permanently unavailable, you might be unable to rebuild or deploy your system. You might also be unable to comply with copyleft license requirements to provide source to your users.
While some Git hosting services can maintain a mirror of an upstream Git repository, Lorry can mirror to Git from many other version control systems and from file-based upstream releases. The Lorry Controller (LC) component implements the services needed to run Lorry regularly for each mirror, and pulls the mirror configuration from another Git repository. One of my projects during lockdown has been modernising Lorry: making these components compatible with current versions of their dependencies and with various Git hosting services, documenting how to install and configure them, and building container images.
The earlier Trovekube project began the work of getting Lorry and LC running on Python 3, and added GitLab CI configurations which have helped me a lot with understanding their dependencies. As I tested out all their features, I found and fixed a few more issues with Python 3 compatibility, mostly involving distinctions between binary and text I/O.
As part of a Baserock Trove appliance, LC was originally co-installed with the Gitano hosting service (downstream host), and had specific support for mirroring the whole of another Trove (upstream host). Later on, support for Gerrit as downstream host, and for GitLab as downstream or upstream host, was added, but I found that this support had bit-rotted. There were also no internal abstractions that would allow adding another type of host cleanly. I have refactored support for each host type into classes, updated the Gerrit and GitLab support to work with current versions of those programs, and added support for Gitea as a downstream host. It turned out to be possible to run a test instance of each of these hosting services in a VM or system container on my development laptop, though it wasn't always the most comfortable experience!
The documentation for Lorry and LC now lists all their direct dependencies and the versions I've tested. For LC I documented all the installation steps and how to integrate it with each of the supported downstream host types.
Finally I needed to build container images using BuildStream. This part was probably the most challenging, since I had no prior experience with either technology. BuildStream is also a fast-moving project that hasn't been kept up-to-date in Debian, my usual development environment. I used the experimental "oci" plugin that can build container images in the OCI or Docker v1.2 format.
I found BuildStream's tutorial and documentation generally excellent, and was impressed at how easy it was to write definitions for building a component. Still, defining how to build all of Lorry and LC's dependencies would have required a huge amount of work. However, the Freedesktop-SDK already has BuildStream definitions for many common open source components, even including Git, OpenSSH, and Perl (used in some of Lorry's file importers). My colleague, Valentin David, helped me to understand how to reuse those, so that I only had to write definitions for the more obscure dependencies.
While my initial testing of Lorry and LC was done on Debian 9 "stretch", the last version where I could easily test all the supported version control systems, Freedesktop-SDK has much more recent versions of some of the dependencies including Git. The current version of Git creates additional files under
.git that weren't expected by Lorry's test suite, but it was easy enough to fix the affected tests.
After I built the first container image, I still needed to learn how to use container managers and to add configuration for various components to work properly in the container. This was not a typical application container, as LC depends on systemd to run multiple services together. I used podman initially, since I understood that it had better support for containers with systemd. BuildStream's build environments are sandboxes with only a (virtual) root user, so I had to defer creation of the lorry service user and its files to first boot of the container using systemd-sysusers and systemd-tmpfiles. Container images should not contain any security credentials, so I also arranged to create SSH key pairs at first boot and to propagate access tokens from the container environment into a LC configuration file. The biggest stumbling block was in logging. Container managers normally run a single application process and expect it to log through its stdout or stderr, but this isn't possible in a multi-service container. However, both podman and Docker can optionally create a
/dev/console device node that will also feed into their logs, and systemd-journald can be configured to log through this instead of to files.
This project was an interesting break from my usual work on kernel development and maintenance. I had the opportunity to explore and learn about various unfamiliar but important technologies. My colleagues helped a lot in getting me started and in reviewing my work toward the end. All of this work is now public under the CodethinkLabs group on GitLab.
- Meet the Codethings: Ben Dooks talks about Linux kernel and RISC-V
- Here we go 2021: 4 open source events for software engineers and project leaders
- Xmas Greetings from Codethink
- Call for Papers: FOSDEM 2021 Dev Room Safety and Open Source Software
- Building the abseil-hello Bazel project for a different architecture using a dynamically generated toolchain
- Advent of Code: programming puzzle challenges
- Improving performance on Interrogizer with the stm32
- Introducing Interrogizer: providing affordable troubleshooting
- Improving software security through input validation
- More time on top: My latest work improving Topplot
- Cycling around the world
- Orchestrating applications by (ab)using Ansible's Network XML Parser
- My experience of the MIT STAMP workshop 2020
- Red Hat announces new Flatpak Runtime for RHEL
- How to keep your staff healthy in lockdown
- Bloodlight: A Medical PPG Testbed
- How to use Tracecompass to analyse kernel traces from LTTng
- Fixing Rust's test suite on RISC-V
- The challenges behind electric vehicle infrastructure
- Investigating kernel user-space access
- Consuming BuildStream projects in Bazel: the bazelize plugin
- Improving RISC-V Linux support in Rust
- Creating a Build toolkit using the Remote Execution API
- Trusting software in a pandemic
- The Case For Open Source Software In The Medical Industry
- My experiences moving to remote working
- Impact of COVID-19 on the Medical Devices Industry
- COVID-19 (Coronavirus) and Codethink
- Codethink develops Open Source drivers for Microsoft Azure Sphere MediaTek MT3620
- Codethink partners with Wirepas
- Testing Bazel's Remote Execution API
- Passing the age of retirement: our work with Fortran and its compilers
- Sharing technical knowledge at Codethink
- Using the REAPI for Distributed Builds
- An Introduction to Remote Execution and Distributed Builds
- Gluing hardware and software: Board Support Packages (BSPs)
- Engineering's jack of all trades: an intro to FPGAs
- Bust out your pendrives: Debian 10 is out!
- Why you should attend local open source meet-ups
- Acceptance, strife, and progress in the LGBTIQ+ and open source communities
- Codethink helps York Instruments to deliver world-beating medical brain-scanner
- Codethink open sources part of staff onboarding - 'How To Git Going In FOSS'
- Getting into open source
- How to put GitOps to work for your software delivery
- Open Source Safety Requirements Analysis for Autonomous Vehicles based on STPA
- Codethink engineers develop custom debug solution for customer project
- Codethink contributes to CIP Super Long Term Kernel maintenance
- Codethink creates custom USB 3 switch to support customer's CI/CD pipeline requirements
- Codethink unlocks data analysis potential for British Cycling
- MIT Doctor delivers Manchester masterclass on innovative safety methodology
- Balance for Better: Women in Technology Codethink Interviews
- Introducing BuildGrid
- Configuring Linux to stabilise latency
- GUADEC 2018 Talks
- Hypervisor Not Required
- Full archive