on Bazel's related documentation page, you may have wondered what BuildGrid is and why it is advertised there as a remote execution implementation. BuildGrid] is a compile farm system that implements a remote build execution service using the Remote Execution API (REAPI) and that distributes build requests over a farm of worker bots using the Remote Worker API (RWAPI).
If you've ever looked into remote build caching and execution and read about it on Bazel's related documentation page, you may have wondered what BuildGrid is and why it is advertised there as a remote execution implementation. BuildGrid is a compile farm system that implements a remote build execution service using the Remote Execution API (REAPI) and that distributes build requests over a farm of worker bots using the Remote Worker API (RWAPI).
Remote build system
BuildGrid is a complete remote build execution system providing:
- The central Execution service, the main end-point for remote execution clients and build jobs orchestrator.
- A ContentAddressableStorage (CAS) service with multiple backends including in-memory or on-disk storage as well as S3 or any remote REAPI CAS compatible storage service.
- An ActionCache (AC) service for fast build action result serving.
- Multiple worker bots supporting host-tools and sandboxed builds.
BuildGrid is open source: its source code and documentation are published under the Apache License 2.0. Its design is modular and makes it easy to either set up an all-in-one server implementing all the services or to split that setup and run each service separately on different machines over a network for better scalability.
On a more technical note, BuildGrid is written in Python and use Google's open
protobuf packages as a base for its REAPI and RWAPI
implementation. There is a minimal requirement on Python 3.5, primarily because
of its internal usage of
asyncio, and it uses
pyyaml for configuration.
Build tools support
BuildGrid aims to be a generic build service. As such, any remote-execution client using the REAPI should be compatible. Our testing and supporting efforts are mainly focused on three of them:
Bazel: BuildGrid currently only supports executing Bazel build requests using host-tools on worker side. Sandboxed builds, using Docker base images, is something we plan to support though. Bazel's documentation contains details on how to adapt your Bazel workspace for remote execution and BuildGrid's documentation has a complete usage guide on how to build a simple Bazel workspace remotely.
BuildStream: a tool for integrating software stacks. BuildStream is BuildGrid's sister project and its need for speeding-up large builds is one of the main reasons why BuildGrid was created in the first place. BuildStream enforces any build to be sandboxed for reproducibility. This is achieved, on BuildGrid's worker side, by delegating build command execution to BuildBox, a tool that, with a bit of FUSE magic combined with the lightness of bubblewrap unprivileged sandboxing, lets BuildStream perform efficient isolated builds. BuildGrid's documentation also has a usage guide on how to build a BuildStream project remotely.
RECC: the Remote Execution Caching Compiler, operates like distcc, by wrapping compiler command calls, and forwards them to a build service using the REAPI. The tools thus allow one to remotely build any existing project as long as their build system allows the adjustment of the underlying compiler calls. BuildGrid's documentation of course has a guide on how to build a project remotely using RECC.
You should expect these clients to just work with BuildGrid out of the box. If you are interested in testing BuildGrid with your favourite remote-execution client, should it be one of these three or any other, we would love to have your feedback and feel free to open an issue on GitLab if you face any problems.
Progress and future plans
BuildGrid is still in the early stages of development. When the project was created, little more than four months ago, Bazel Buildfarm was the only existing REAPI implementation freely available but it was using a custom interface on the worker side. Implementing a service that would use the RWAPI for managing the bot farm was, at the time, another main reason why we decided to write BuildGrid.
We are currently focusing our development effort on completing the REAPI and RWAPI server implementations. Most of the core functionality is now supported but we are aiming to be fully compliant with both APIs. Two of the main features that we plan to work on soon are support for job cancellation and server instrumentation for performance analysis. Our development activity is driven and tracked publicly using GitLab project management features.
No stable version has been released yet but installing a development
version and starting to use the BuildGrid system should be relatively
easy. As design is still improving and code is moving fast, we strongly
encourage you to join the
#buildgrid channel of the BuildTeam Slack
group and discuss with us your remote execution plans or expectations,
as well as any issues or questions you could have about BuildGrid.
Finally, BuildGrid was presented at BazelCon 2018 in a talk now available on YouTube. The first half of the presentation is a nice introduction to remote-execution and how BuildStream, BuildGrid and BuildBox play together. The second half dives deeper into BuildGrid internals and demonstrate actual builds using Bazel, RECC and BuildStream. Highly recommended.
- Automated Linux kernel testing
- Native compilation on Arm servers is so much faster now
- Higher quality of FOSS: How we are helping GNOME to improve their test pipeline
- RISC-V: A Small Hardware Project
- Why aligning with open source mainline is the way to go
- Build Meetup 2021: The BuildTeam Community Event
- A new approach to software safety
- Does the "Hypocrite Commits" incident prove that Linux is unsafe?
- ABI Stability in freedesktop-sdk
- Why your organisation needs to embrace working in the open-source ecosystem
- RISC-V User space access Oops
- Tracking Players at the Edge: An Overview
- What is Remote Asset API?
- Running a devroom: FOSDEM 2021 Safety and Open Source
- Meet the codethings: Understanding BuildGrid and BuildBox with Beth White
- Streamlining Terraform configuration with Jsonnet
- Bloodlight: Designing a Heart Rate Sensor with STM32, LEDs and Photodiode
- Making the tech industry more inclusive for women
- Bloodlight Case Design: Lessons Learned
- Safety is a system property, not a software property
- RISC-V: Codethink's first research about the open instruction set
- Meet the Codethings: Safety-critical systems and the benefits of STPA with Shaun Mooney
- Why Project Managers are essential in an effective software consultancy
- FOSDEM 2021: Devroom for Safety and Open Source
- Meet the Codethings: Ben Dooks talks about Linux kernel and RISC-V
- Here we go 2021: 4 open source events for software engineers and project leaders
- Xmas Greetings from Codethink
- Call for Papers: FOSDEM 2021 Dev Room Safety and Open Source Software
- Building the abseil-hello Bazel project for a different architecture using a dynamically generated toolchain
- Advent of Code: programming puzzle challenges
- Improving performance on Interrogizer with the stm32
- Introducing Interrogizer: providing affordable troubleshooting
- Improving software security through input validation
- More time on top: My latest work improving Topplot
- Cycling around the world
- Orchestrating applications by (ab)using Ansible's Network XML Parser
- My experience of the MIT STAMP workshop 2020
- Red Hat announces new Flatpak Runtime for RHEL
- How to keep your staff healthy in lockdown
- Bloodlight: A Medical PPG Testbed
- Bringing Lorry into the 2020s
- How to use Tracecompass to analyse kernel traces from LTTng
- Fixing Rust's test suite on RISC-V
- The challenges behind electric vehicle infrastructure
- Investigating kernel user-space access
- Consuming BuildStream projects in Bazel: the bazelize plugin
- Improving RISC-V Linux support in Rust
- Creating a Build toolkit using the Remote Execution API
- Trusting software in a pandemic
- The Case For Open Source Software In The Medical Industry
- My experiences moving to remote working
- Impact of COVID-19 on the Medical Devices Industry
- COVID-19 (Coronavirus) and Codethink
- Codethink develops Open Source drivers for Microsoft Azure Sphere MediaTek MT3620
- Codethink partners with Wirepas
- Testing Bazel's Remote Execution API
- Passing the age of retirement: our work with Fortran and its compilers
- Sharing technical knowledge at Codethink
- Using the REAPI for Distributed Builds
- An Introduction to Remote Execution and Distributed Builds
- Gluing hardware and software: Board Support Packages (BSPs)
- Engineering's jack of all trades: an intro to FPGAs
- Bust out your pendrives: Debian 10 is out!
- Why you should attend local open source meet-ups
- Acceptance, strife, and progress in the LGBTIQ+ and open source communities
- Codethink helps York Instruments to deliver world-beating medical brain-scanner
- Codethink open sources part of staff onboarding - 'How To Git Going In FOSS'
- Getting into open source
- How to put GitOps to work for your software delivery
- Open Source Safety Requirements Analysis for Autonomous Vehicles based on STPA
- Codethink engineers develop custom debug solution for customer project
- Codethink contributes to CIP Super Long Term Kernel maintenance
- Codethink creates custom USB 3 switch to support customer's CI/CD pipeline requirements
- Codethink unlocks data analysis potential for British Cycling
- MIT Doctor delivers Manchester masterclass on innovative safety methodology
- Balance for Better: Women in Technology Codethink Interviews
- Configuring Linux to stabilise latency
- Full archive