on Bazel's related documentation page, you may have wondered what BuildGrid is and why it is advertised there as a remote execution implementation. BuildGrid] is a compile farm system that implements a remote build execution service using the Remote Execution API (REAPI) and that distributes build requests over a farm of worker bots using the Remote Worker API (RWAPI).
If you've ever looked into remote build caching and execution and read about it on Bazel's related documentation page, you may have wondered what BuildGrid is and why it is advertised there as a remote execution implementation. BuildGrid is a compile farm system that implements a remote build execution service using the Remote Execution API (REAPI) and that distributes build requests over a farm of worker bots using the Remote Worker API (RWAPI).
Remote build system
BuildGrid is a complete remote build execution system providing:
- The central Execution service, the main end-point for remote execution clients and build jobs orchestrator.
- A ContentAddressableStorage (CAS) service with multiple backends including in-memory or on-disk storage as well as S3 or any remote REAPI CAS compatible storage service.
- An ActionCache (AC) service for fast build action result serving.
- Multiple worker bots supporting host-tools and sandboxed builds.
BuildGrid is open source: its source code and documentation are published under the Apache License 2.0. Its design is modular and makes it easy to either set up an all-in-one server implementing all the services or to split that setup and run each service separately on different machines over a network for better scalability.
On a more technical note, BuildGrid is written in Python and use Google's open
protobuf packages as a base for its REAPI and RWAPI
implementation. There is a minimal requirement on Python 3.5, primarily because
of its internal usage of
asyncio, and it uses
pyyaml for configuration.
Build tools support
BuildGrid aims to be a generic build service. As such, any remote-execution client using the REAPI should be compatible. Our testing and supporting efforts are mainly focused on three of them:
Bazel: BuildGrid currently only supports executing Bazel build requests using host-tools on worker side. Sandboxed builds, using Docker base images, is something we plan to support though. Bazel's documentation contains details on how to adapt your Bazel workspace for remote execution and BuildGrid's documentation has a complete usage guide on how to build a simple Bazel workspace remotely.
BuildStream: a tool for integrating software stacks. BuildStream is BuildGrid's sister project and its need for speeding-up large builds is one of the main reasons why BuildGrid was created in the first place. BuildStream enforces any build to be sandboxed for reproducibility. This is achieved, on BuildGrid's worker side, by delegating build command execution to BuildBox, a tool that, with a bit of FUSE magic combined with the lightness of bubblewrap unprivileged sandboxing, lets BuildStream perform efficient isolated builds. BuildGrid's documentation also has a usage guide on how to build a BuildStream project remotely.
RECC: the Remote Execution Caching Compiler, operates like distcc, by wrapping compiler command calls, and forwards them to a build service using the REAPI. The tools thus allow one to remotely build any existing project as long as their build system allows the adjustment of the underlying compiler calls. BuildGrid's documentation of course has a guide on how to build a project remotely using RECC.
You should expect these clients to just work with BuildGrid out of the box. If you are interested in testing BuildGrid with your favourite remote-execution client, should it be one of these three or any other, we would love to have your feedback and feel free to open an issue on GitLab if you face any problems.
Progress and future plans
BuildGrid is still in the early stages of development. When the project was created, little more than four months ago, Bazel Buildfarm was the only existing REAPI implementation freely available but it was using a custom interface on the worker side. Implementing a service that would use the RWAPI for managing the bot farm was, at the time, another main reason why we decided to write BuildGrid.
We are currently focusing our development effort on completing the REAPI and RWAPI server implementations. Most of the core functionality is now supported but we are aiming to be fully compliant with both APIs. Two of the main features that we plan to work on soon are support for job cancellation and server instrumentation for performance analysis. Our development activity is driven and tracked publicly using GitLab project management features.
No stable version has been released yet but installing a development
version and starting to use the BuildGrid system should be relatively
easy. As design is still improving and code is moving fast, we strongly
encourage you to join the
#buildgrid channel of the BuildTeam Slack
group and discuss with us your remote execution plans or expectations,
as well as any issues or questions you could have about BuildGrid.
Finally, BuildGrid was presented at BazelCon 2018 in a talk now available on YouTube. The first half of the presentation is a nice introduction to remote-execution and how BuildStream, BuildGrid and BuildBox play together. The second half dives deeper into BuildGrid internals and demonstrate actual builds using Bazel, RECC and BuildStream. Highly recommended.
- Using Git LFS and fast-import together
- Testing in a Box: Streamlining Embedded Systems Testing
- SDV Europe: What Codethink has planned
- How do Hardware Security Modules impact the automotive sector? The final blog in a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part two of a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part one of a three part discussion
- Automated Kernel Testing on RISC-V Hardware
- Automated end-to-end testing for Android Automotive on Hardware
- GUADEC 2023
- Embedded Open Source Summit 2023
- RISC-V: exploring a bug in stack unwinding
- Adding RISC-V Vector Cryptography Extension support to QEMU
- Introducing Our New Open-Source Tool: Quality Assurance Daemon
- Long Term Maintainability
- FOSDEM 2023
- Think before you Pip
- BuildStream 2.0 is here, just in time for the holidays!
- A Valuable & Comprehensive Firmware Code Review by Codethink
- GNOME OS & Atomic Upgrades on the PinePhone
- Flathub-Codethink Collaboration
- Codethink proudly sponsors GUADEC 2022
- Tracking Down an Obscure Reproducibility Bug in glibc
- Web app test automation with `cdt`
- FOSDEM Testing and Automation talk
- Protecting your project from dependency access problems
- Porting GNOME OS to Microchip's PolarFire Icicle Kit
- YAML Schemas: Validating Data without Writing Code
- Deterministic Construction Service
- Codethink becomes a Microchip Design Partner
- Hamsa: Using an NVIDIA Jetson Development Kit to create a fully open-source Robot Nano Hand
- Using STPA with software-intensive systems
- Codethink achieves ISO 26262 ASIL D Tool Certification
- RISC-V: running GNOME OS on SiFive hardware for the first time
- Automated Linux kernel testing
- Native compilation on Arm servers is so much faster now
- Higher quality of FOSS: How we are helping GNOME to improve their test pipeline
- RISC-V: A Small Hardware Project
- Why aligning with open source mainline is the way to go
- Build Meetup 2021: The BuildTeam Community Event
- A new approach to software safety
- Does the "Hypocrite Commits" incident prove that Linux is unsafe?
- ABI Stability in freedesktop-sdk
- Why your organisation needs to embrace working in the open-source ecosystem
- RISC-V User space access Oops
- Tracking Players at the Edge: An Overview
- What is Remote Asset API?
- Running a devroom at FOSDEM: Safety and Open Source
- Meet the codethings: Understanding BuildGrid and BuildBox with Beth White
- Streamlining Terraform configuration with Jsonnet
- Bloodlight: Designing a Heart Rate Sensor with STM32, LEDs and Photodiode
- Making the tech industry more inclusive for women
- Bloodlight Case Design: Lessons Learned
- Safety is a system property, not a software property
- RISC-V: Codethink's first research about the open instruction set
- Meet the Codethings: Safety-critical systems and the benefits of STPA with Shaun Mooney
- Why Project Managers are essential in an effective software consultancy
- FOSDEM 2021: Devroom for Safety and Open Source
- Meet the Codethings: Ben Dooks talks about Linux kernel and RISC-V
- Here we go 2021: 4 open source events for software engineers and project leaders
- Xmas Greetings from Codethink
- Call for Papers: FOSDEM 2021 Dev Room Safety and Open Source Software
- Building the abseil-hello Bazel project for a different architecture using a dynamically generated toolchain
- Advent of Code: programming puzzle challenges
- Full archive