The dependency problem
Almost all software projects large or small rely on dependencies — software that isn’t part of the project itself. The use of dependencies is widespread because it saves repeating development work someone else has already performed and generously shared.
The problem with introducing these dependencies, is that part of your project now exists outside of your control. If the dependencies disappear and you don’t have a backup, then you may not be able to build your project at all. This actually happened in a big way in 2016 when “Leftpad” was removed from npm and caused a large number of services that relied on this small piece of code, often inadvertently through dependencies within dependencies, to stop working.
There are a variety of other reasons you may need access to the source code that makes up your dependencies, for example:
- You need to debug any misbehaviour coming from one of the dependencies.
- You need to comply with licensing terms: a common open-source licensing term is that source code is available upon request.
- You need to provide a complete manifest of everything that goes into a system.
You might be wondering, if your dependencies are voluntarily shared, why would they disappear? Well, even large and established platforms are not immune to downtime — in October 2018, GitHub was down for 12 hours. Alternatively, software could be relocated to a new server or maintainers of a particular piece of software could choose to stop supporting it and take it down permanently. In the last case, while it's prudent to avoid using unsupported software, it may be necessary to keep the build going for a short amount of time.
The mirroring solution
A common solution to the problem of disappearing dependencies is to use a mirror. This is a second source of the required software that can be accessed if the primary one is unavailable. Many build systems such as BuildStream allow you to add mirrors into your build scripts so you can supply multiple sources for a single resource.
Maintenance of these mirrors can be a daunting task, especially for large software projects with hundreds of dependencies. However, choosing to not mirror your dependencies is a dangerous thing. While a lot of build servers will cache resources that have been downloaded, it will not be possible to run a clean build or build on a new machine that doesn't have access to that cache. This can be a major barrier to testing and deployment of your software project. The more dependencies you have, the bigger the risk since it's more likely one won't be available.
For small projects, it may be tempting to manage your mirrors manually. Manual management of mirrors can easily become a burden as there are tasks that will have to be done regularly. For example, it's important to check that the location of the original source code stays the same and that versions are the same.
There are some existing solutions available for mirror management but none of these are ideal — either being proprietary and costly, or unable to handle multiple version control systems as well as commonly used individual files such as tarballs.
Engineers at Codethink discuss existing solutions and explore a free and open-source software solution in the following white paper.
Receive more information on Long Term Maintainability
Fill in the form below to receive a downloadable white paper.
- Using Git LFS and fast-import together
- Testing in a Box: Streamlining Embedded Systems Testing
- SDV Europe: What Codethink has planned
- How do Hardware Security Modules impact the automotive sector? The final blog in a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part two of a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part one of a three part discussion
- Automated Kernel Testing on RISC-V Hardware
- Automated end-to-end testing for Android Automotive on Hardware
- GUADEC 2023
- Embedded Open Source Summit 2023
- RISC-V: exploring a bug in stack unwinding
- Adding RISC-V Vector Cryptography Extension support to QEMU
- Introducing Our New Open-Source Tool: Quality Assurance Daemon
- Long Term Maintainability
- FOSDEM 2023
- Think before you Pip
- BuildStream 2.0 is here, just in time for the holidays!
- A Valuable & Comprehensive Firmware Code Review by Codethink
- GNOME OS & Atomic Upgrades on the PinePhone
- Flathub-Codethink Collaboration
- Codethink proudly sponsors GUADEC 2022
- Tracking Down an Obscure Reproducibility Bug in glibc
- Web app test automation with `cdt`
- FOSDEM Testing and Automation talk
- Porting GNOME OS to Microchip's PolarFire Icicle Kit
- YAML Schemas: Validating Data without Writing Code
- Deterministic Construction Service
- Codethink becomes a Microchip Design Partner
- Hamsa: Using an NVIDIA Jetson Development Kit to create a fully open-source Robot Nano Hand
- Using STPA with software-intensive systems
- Codethink achieves ISO 26262 ASIL D Tool Certification
- RISC-V: running GNOME OS on SiFive hardware for the first time
- Automated Linux kernel testing
- Native compilation on Arm servers is so much faster now
- Higher quality of FOSS: How we are helping GNOME to improve their test pipeline
- RISC-V: A Small Hardware Project
- Why aligning with open source mainline is the way to go
- Build Meetup 2021: The BuildTeam Community Event
- A new approach to software safety
- Does the "Hypocrite Commits" incident prove that Linux is unsafe?
- ABI Stability in freedesktop-sdk
- Why your organisation needs to embrace working in the open-source ecosystem
- RISC-V User space access Oops
- Tracking Players at the Edge: An Overview
- What is Remote Asset API?
- Running a devroom at FOSDEM: Safety and Open Source
- Meet the codethings: Understanding BuildGrid and BuildBox with Beth White
- Streamlining Terraform configuration with Jsonnet
- Bloodlight: Designing a Heart Rate Sensor with STM32, LEDs and Photodiode
- Making the tech industry more inclusive for women
- Bloodlight Case Design: Lessons Learned
- Safety is a system property, not a software property
- RISC-V: Codethink's first research about the open instruction set
- Meet the Codethings: Safety-critical systems and the benefits of STPA with Shaun Mooney
- Why Project Managers are essential in an effective software consultancy
- FOSDEM 2021: Devroom for Safety and Open Source
- Meet the Codethings: Ben Dooks talks about Linux kernel and RISC-V
- Here we go 2021: 4 open source events for software engineers and project leaders
- Xmas Greetings from Codethink
- Call for Papers: FOSDEM 2021 Dev Room Safety and Open Source Software
- Building the abseil-hello Bazel project for a different architecture using a dynamically generated toolchain
- Advent of Code: programming puzzle challenges
- Full archive