Instantaneous CI: Reducing Development Lag Through Cicada with Local-first Design and Caching

Understanding Continuous Integration

Continuous integration and continuous deployment (CI/CD) enable the automation of building, testing, and releasing software. This approach streamlines collaboration between development and operations teams. CI/CD reduces large points of failure by providing immediate feedback on errors or bugs in an update cycle that can be made more efficient and reliable through being smaller and more frequent. Since the 1990s, CI has become a fundamental practice for devops, with CD following suit in the 2010s. At Codethink, we have a storied history of integrating CI/CD into our software development cycle, being early and prominent advocates of its use in the automotive industry. Our latest approach uses a self-hosted GitLab namespace, maintaining our open source ethos and enabling scalability.

Available Tooling for CI/CD

Tooling including Zuul, Jenkins, and GitLab CI offer framework support in the CI/CD process but each have benefits and drawbacks, which must be carefully considered when used for developing software. There is no perfect CI tool out there that will solve the problem for every use case. Often, modifications are required in the tooling to tailor and adapt the process for your project. This can result in unmanageable maintenance costs in the development cycle, which creates frustration, delays, and unwieldy dependency management.

GitLab CI makes local debugging difficult and does not offer a way to test jobs pre-merging. GitLab attempts to be an all-in-one tool and the successive feature creep increasingly makes it harder to manage.

Jenkins CI runs on a system of plugins, which too often are abandonware, increasing security risk and requiring a lot of overhead and dependency mitigation if an upgrade is required. The maintenance and hosting cost can become unreasonable when scaling a project.

Zuul CI still has a smaller uptake. While it is powerful and open source, which provides great project management flexibility, there are limited resources and documentations for its integration into a development workflow at this stage.

REAPI and CI

Remote Execution Application Programming Interface (REAPI) is an open and standardised gRPC-based protocol that enables build tools to offload actions onto remote servers for more efficient execution. REAPI operates on a client-server model where tasks are serialised by the build tool before being executed remotely, returning the output. Caching is added to this process in order to avoid delays from the duplication of task runs.

It has been used as a component part of CI/CD to make reproducible builds faster at scale with tools like Buildstream and Bazel. However, these benefits could be applied further to CI by incorporating REAPI in the CI system itself, rather than just within individual jobs.

By combining REAPI with integration tooling, remote caching could be shared and would produce a near instantaneous result on a job output after it has been run once if the input is exactly the same.

This would improve the efficiency of the integration process.

A common pain point with many existing CI systems is the inability to run jobs outside of the CI workflow. If the CI system is an REAPI client, we can develop a way to submit the same jobs to the REAPI server from outside of the CI itself. This local-first design improves the feedback loop when developing new CI jobs, as well as giving full confidence that a CI job will pass after pushing.

Using REAPI caching within a CI job allows reuse of old results of the same job for a given set of inputs. However, by using REAPI caching as part of the CI system itself we can also cache the job's environment construction, saving time spent fetching the git repository and dependencies.

This combination provides the opportunity for instantaneous CI.

The performance of our CI system's cache could be enhanced with speculative execution of CI jobs across multiple MRs, essentially pre-testing the potential merges to provide instantaneous CI when the target branch of an MR is updated.

Developing Cicada

Advancing from the theoretical application and implications of this variety of CI tooling, Cicada was built as a prototype.

Cicada jobs and pipelines are declared in YAML, then modeled as REAPI Actions to provide caching and offload job scheduling to REAPI server tooling. By using remote execution, the cache can be safely shared between locally invoked runs of Cicada jobs and jobs invoked during a CI run, making the actual CI run provide an almost instantaneous result.

You can read more about the development of Cicada on this blog post. There are real-world applications of this tool, notes on how this could work with Rust, and a list of ideas for future development.