Tue 23 October 2018

Introducing BuildGrid

by Martin Blanchard , 2018 , Tags remote-execution remove-caching bazel builstream buildgrid buildbox

If you've ever looked into remote build caching and execution and read about it on Bazel's related documentation page, you may have wondered what BuildGrid is and why it is advertised there as a remote execution implementation. BuildGrid is a compile farm system that implements a remote build execution service using the Remote Execution API (REAPI) and that distributes build requests over a farm of worker bots using the Remote Worker API (RWAPI).

Remote build system

BuildGrid is a complete remote build execution system providing:

  • The central Execution service, the main end-point for remote execution clients and build jobs orchestrator.
  • A ContentAddressableStorage (CAS) service with multiple backends including in-memory or on-disk storage as well as S3 or any remote REAPI CAS compatible storage service.
  • An ActionCache (AC) service for fast build action result serving.
  • Multiple worker bots supporting host-tools and sandboxed builds.

BuildGrid is open source: its source code and documentation are published under the Apache License 2.0. Its design is modular and makes it easy to either set up an all-in-one server implementing all the services or to split that setup and run each service separately on different machines over a network for better scalability.

On a more technical note, BuildGrid is written in Python and use Google's open source grpcio and protobuf packages as a base for its REAPI and RWAPI implementation. There is a minimal requirement on Python 3.5, primarily because of its internal usage of asyncio, and it uses pyyaml for configuration.

Build tools support

BuildGrid aims to be a generic build service. As such, any remote-execution client using the REAPI should be compatible. Our testing and supporting efforts are mainly focused on three of them:

  • Bazel: BuildGrid currently only supports executing Bazel build requests using host-tools on worker side. Sandboxed builds, using Docker base images, is something we plan to support though. Bazel's documentation contains details on how to adapt your Bazel workspace for remote execution and BuildGrid's documentation has a complete usage guide on how to build a simple Bazel workspace remotely.

  • BuildStream: a tool for integrating software stacks. BuildStream is BuildGrid's sister project and its need for speeding-up large builds is one of the main reasons why BuildGrid was created in the first place. BuildStream enforces any build to be sandboxed for reproducibility. This is achieved, on BuildGrid's worker side, by delegating build command execution to BuildBox, a tool that, with a bit of FUSE magic combined with the lightness of bubblewrap unprivileged sandboxing, lets BuildStream perform efficient isolated builds. BuildGrid's documentation also has a usage guide on how to build a BuildStream project remotely.

  • RECC: the Remote Execution Caching Compiler, operates like distcc, by wrapping compiler command calls, and forwards them to a build service using the REAPI. The tools thus allow one to remotely build any existing project as long as their build system allows the adjustment of the underlying compiler calls. BuildGrid's documentation of course has a guide on how to build a project remotely using RECC.

You should expect these clients to just work with BuildGrid out of the box. If you are interested in testing BuildGrid with your favourite remote-execution client, should it be one of these three or any other, we would love to have your feedback and feel free to open an issue on GitLab if you face any problems.

Progress and future plans

BuildGrid is still in the early stages of development. When the project was created, little more than four months ago, Bazel Buildfarm was the only existing REAPI implementation freely available but it was using a custom interface on the worker side. Implementing a service that would use the RWAPI for managing the bot farm was, at the time, another main reason why we decided to write BuildGrid.

We are currently focusing our development effort on completing the REAPI and RWAPI server implementations. Most of the core functionality is now supported but we are aiming to be fully compliant with both APIs. Two of the main features that we plan to work on soon are support for job cancellation and server instrumentation for performance analysis. Our development activity is driven and tracked publicly using GitLab project management features.

No stable version has been released yet but installing a development version and starting to use the BuildGrid system should be relatively easy. As design is still improving and code is moving fast, we strongly encourage you to join the #buildgrid channel of the BuildTeam Slack group and discuss with us your remote execution plans or expectations, as well as any issues or questions you could have about BuildGrid.

Finally, BuildGrid was presented at BazelCon 2018 in a talk now available on YouTube. The first half of the presentation is a nice introduction to remote-execution and how BuildStream, BuildGrid and BuildBox play together. The second half dives deeper into BuildGrid internals and demonstrate actual builds using Bazel, RECC and BuildStream. Highly recommended.