Open Source Insider

Fugue offers Zim, an open source monorepo caching build system 

Adrian Bridgwater
Adrian Bridgwater

Cloud security company Fugue is an interesting technology proposition.

The company’s core tech offering bids to prove compliance by building security into cloud development to eliminate misconfiguration for cloud infrastructure (on AWS, Azure, Google Cloud) and so put software engineers in control of that element of cloudy unknowns.

Fugue is actually pronounced “few-g” and the name comes from a polyphoneic musical style that was popular in the baroque era.

But although Fugue’s stack stems from security-centric software, the company has now released a new ‘non-security’ (not strictly a term, but you get the point) open source project called Zim.

Zim is a caching build system for teams using monorepos.

What is a monorepo?

A monorepo is a software development strategy where code for many projects is stored in the same repository. As Atlassian notes, a monorepo’s projects are most likely unrelated, loosely connected or can be connected by other means (e.g via dependency management tools). A monorepo is good for single-source of truth visibility (all code and assets are in one place) and it aids collaboration (same reason) and it also brings speed as it allows developers to make ‘atomic changes’ i.e. where one code change action in then brought into affect across several projects.

So what does Zim do? As stated, it is a caching build system designed for developers using monorepos that contain many components and dependencies. It provides for ‘fast incremental parallel builds’ across a team and is entirely language agnostic with built-in support for cross-platform builds via Docker. 

Zim is available as an open source project hosted on GitHub. 

“In developing Zim, we drew inspiration from the core concepts of GNU Make and the caching strategy employed by Buck, Bazel and Please. Like Make, Zim has a lightweight way to express new rules that define inputs, outputs… and the commands needed to create the outputs. Zim computes Rule Keys which are used to determine whether the output of a rule is already available in the cache, based on the combined hashes of the rule’s inputs and configuration,” notes Fugue co-founder and VP of comms Drew Wright, on its product launch notes.

When Fugue began building its cloud security and compliance SaaS on Amazon Web Services (AWS), it favored a serverless approach via AWS Lambda and AWS Fargate as a way to avoid managing virtual servers (e.g., EC2 instances), gain security and isolation for jobs via an AWS mechanism, and allow us to scale in a very responsive and dynamic way. 

The team, therefore, opted to store all our code in a monorepo and scripted all build and deploy tasks using Makefiles. 

However, as the system grew to hundreds of components, the Makefile approach grew unwieldy and they knew we needed improved tooling for builds and artifact management. 

“With Zim and its shared cache, we reduced build times for a number of common tasks by an order of magnitude. Reusing shared Zim build templates also significantly reduced ongoing maintenance of Makefile based build scripts. Our library of templates supports efficiently building Lambdas and binaries for Fargate that are written in Python, Go, and Node JS,” explained Wright.

You can read the full product blog for more… and the key product highlights are bulleted below:

  • Fast, parallel builds: Rules run only if inputs have changed and outputs are pulled from a shared cache if someone else built it already.
  • Trivial build step definitions: Define how to build new component types in a few lines of YAML.
  • Isolated build environments and cross-platform compilation: via the built-in Docker support. Just specify the Docker image to be used when building a component.
  • Flexible input and output resource types: Zim is able to work with both files and Docker images as natively supported resources.
  • Easy setup for a shared cache in S3 via an AWS CloudFormation stack.
  • Lightweight & easy to install. Zim is written in Go and consists of a single binary when built. It is compatible with virtually any CI/CD pipeline.

Image source: Fugue