Over the last few years Codethink has been helping a series of customers understand and manage the scale and complexity of their software integration pipelines. We've seen and tried various approaches to visualise what's happening, but many projects end up falling back to some combination of:
- architecture diagrams
- data models
- workflow descriptions
- organisation charts
A key problem is that most of these approaches tend to describe how we think (or hope) things are, rather than how they actually work. It's hard for new people to grok what's happening, and even harder for executives to pinpoint the root causes of business-level problems. Even worse, as the real world projects evolve, the pictures don't keep up.
At the Cloud Foundry Summit in 2015, Codethink CEO Paul Sherwood was inspired by Alex Suraci's talk about Concourse, a new tool for CI which can help to demystify the process of building, integrating, validating and releasing complex software.
Given that Codethink works on some truly gigantic whole-stack projects for enterprise, devices and automotive customers, Paul immediately wondered if Concourse really is 'the CI that scales with your project'. Could it handle visualisation and integration of complete systems; from initial toolchain through BSP, kernel, OS, middleware and applications?
The answer is (more or less) YES!
Here's a minimal Linux stack integration pipeline, showing the flow of a build from toolchain on the left, through to the output system on the right. Click the image to see the live CI pipeline at concourse.baserock.org
And the Baserock GENIVI Demo Platform for x86_64:
And now a full integration of OpenStack software:
Important things to note about these pipelines:
- they represent actual software, being integrated for real
- when a new integration fails, its blindingly obvious where the failure happens, which is a huge win for interaction with a large ecosystem contributing to a common platform
- the pipeline structure can refresh automatically as the underlying software architecture changes
Chris Brown, who leads the Concourse team at Pivotal says "It's fantastic to see these impressive pipelines opened to the public. I was always hoping that Concourse would be used for this scale of project."
There's still lots of work to do, as Codethink improves and optimises this approach for commercial use-cases and multi-architecture projects, but early adopter customers are as excited as we are.
If you want to know more about how this new level of visualisation can help your team, please get in touch!
Other Articles
- Web app test automation with `cdt`
- FOSDEM Testing and Automation talk
- Protecting your project from dependency access problems
- Porting GNOME OS to Microchip's PolarFire Icicle Kit
- YAML Schemas: Validating Data without Writing Code
- Deterministic Construction Service
- Codethink becomes a Microchip Design Partner
- Hamsa: Using an NVIDIA Jetson Development Kit to create a fully open-source Robot Nano Hand
- Using STPA with software-intensive systems
- Codethink achieves ISO 26262 ASIL D Tool Certification
- RISC-V: running GNOME OS on SiFive hardware for the first time
- Automated Linux kernel testing
- Native compilation on Arm servers is so much faster now
- Higher quality of FOSS: How we are helping GNOME to improve their test pipeline
- RISC-V: A Small Hardware Project
- Why aligning with open source mainline is the way to go
- Build Meetup 2021: The BuildTeam Community Event
- A new approach to software safety
- Does the "Hypocrite Commits" incident prove that Linux is unsafe?
- ABI Stability in freedesktop-sdk
- Why your organisation needs to embrace working in the open-source ecosystem
- RISC-V User space access Oops
- Tracking Players at the Edge: An Overview
- What is Remote Asset API?
- Running a devroom at FOSDEM: Safety and Open Source
- Meet the codethings: Understanding BuildGrid and BuildBox with Beth White
- Streamlining Terraform configuration with Jsonnet
- Bloodlight: Designing a Heart Rate Sensor with STM32, LEDs and Photodiode
- Making the tech industry more inclusive for women
- Bloodlight Case Design: Lessons Learned
- Safety is a system property, not a software property
- RISC-V: Codethink's first research about the open instruction set
- Meet the Codethings: Safety-critical systems and the benefits of STPA with Shaun Mooney
- Why Project Managers are essential in an effective software consultancy
- FOSDEM 2021: Devroom for Safety and Open Source
- Meet the Codethings: Ben Dooks talks about Linux kernel and RISC-V
- Here we go 2021: 4 open source events for software engineers and project leaders
- Xmas Greetings from Codethink
- Call for Papers: FOSDEM 2021 Dev Room Safety and Open Source Software
- Building the abseil-hello Bazel project for a different architecture using a dynamically generated toolchain
- Advent of Code: programming puzzle challenges
- Improving performance on Interrogizer with the stm32
- Introducing Interrogizer: providing affordable troubleshooting
- Improving software security through input validation
- More time on top: My latest work improving Topplot
- Orchestrating applications by (ab)using Ansible's Network XML Parser
- My experience of the MIT STAMP workshop 2020
- Red Hat announces new Flatpak Runtime for RHEL
- How to keep your staff healthy in lockdown
- Bloodlight: A Medical PPG Testbed
- Bringing Lorry into the 2020s
- How to use Tracecompass to analyse kernel traces from LTTng
- Fixing Rust's test suite on RISC-V
- The challenges behind electric vehicle infrastructure
- Investigating kernel user-space access
- Consuming BuildStream projects in Bazel: the bazelize plugin
- Improving RISC-V Linux support in Rust
- Creating a Build toolkit using the Remote Execution API
- Trusting software in a pandemic
- The Case For Open Source Software In The Medical Industry
- My experiences moving to remote working
- Impact of COVID-19 on the Medical Devices Industry
- COVID-19 (Coronavirus) and Codethink
- Codethink develops Open Source drivers for Microsoft Azure Sphere MediaTek MT3620
- Codethink partners with Wirepas
- Testing Bazel's Remote Execution API
- Passing the age of retirement: our work with Fortran and its compilers
- Sharing technical knowledge at Codethink
- Using the REAPI for Distributed Builds
- An Introduction to Remote Execution and Distributed Builds
- Gluing hardware and software: Board Support Packages (BSPs)
- Engineering's jack of all trades: an intro to FPGAs
- Bust out your pendrives: Debian 10 is out!
- Why you should attend local open source meet-ups
- Acceptance, strife, and progress in the LGBTIQ+ and open source communities
- Codethink helps York Instruments to deliver world-beating medical brain-scanner
- Full archive