Build Cache - Overview

This feature is only available in Incredibuild's Enterprise and Business Plans and is currently in limited availability. For more details, speak to your Customer Success Representative.

Build Cache saves time and resources by reusing portions of previous builds. As you run builds, we store the outputs in a file cache. When future builds are run, we check the cache to avoid regenerating portions of the build that have not changed.

This can have a major impact, as you can reuse previous build data stored in Build Cache to drastically reduce build times without impacting your bandwidth.

In order to achieve this, Build Cache has specific System Requirements and Supported Tools.

Benefits and Use Cases

  • Boost developer productivity from anywhere
    Stop running builds from scratch every time you switch branches, even when they've already been built elsewhere. Build Cache automatically stores previous build data and drastically reduces local build times.

  • Build from home without impacting your speed
    Working from home affects build speeds due to limited upstream bandwidth. Build Cache lets you rely more on downstream bandwidth, giving you greater speed and better performance when starting new builds.

  • Use cached data between CI builds
    Use Build Cache to share build artifacts generated by one CI server with another to save time and resources. Bring the incremental build experience to your CI.

  • Reuse CI results for faster development builds
    Instead of having to incorporate every single change made when restarting a build from version control, Build Cache lets developers reuse outputs generated on later CI builds and workflows, even when working on prior versions.

License Model

Build cache uses special licenses called Cache Cores. One Cache Core is consumed whenever using the Build Cache saves you one hour of a helper core's execution. Cache Cores are reusable and are refreshed every hour.

If you run out of Cache Cores, data is still written to the cache, but cannot be read until your Cache Cores are refreshed. You can view information about your license usage at the top of the Agent List.

If you are using Incredibuild Cloud in addition to an on-prem license, see Using Build cache with Incredibuild Cloud to learn about how these licenses interact.

How Does Build Cache Work? 

Build Cache examines each task (compilation or other) and decides whether it can use existing cache contents, or the task needs to run. It assumes that if the inputs for the task (source files, header files, pre-compiled headers, etc.) did not change, then the resulting artifact (object file, PDB, etc.) will not have changed in a significant way either. There is an assumption that most tasks are deterministic, and that the entire build is deterministic. Whatever part of the build is not deterministic will need to be re-run for every build and will not be improved by Build Cache.

A deterministic build is a process of building the same source code with the same build environment and build instructions producing the same binary in two builds, even if they are made on different machines, build directories and with different names. They are also sometimes called reproducible or hermetic builds if it is guaranteed to produce the same binaries even compiling from different folders.

Creating and Managing the Cache

Builds are often processed in tree-like structures based on the dependencies in your code. The files in Layer 1 are used to create the files in Layer 2, Layer 2 is used to create the files in Layer 3, and so on until the final output is produced.

As builds are processed, Incredibuild saves details of the process for future use. Each part of the process that is generated from other files is saved as a pair of input files and output files in the cache. If you run a build after changing some of the files, Build Cache can reuse the portions that are identical to prevent parts of the process from running.

The build cache is optimized in various ways such as removing duplicate files and automatic cleanups.

Changing Source Code

When your source files change, obviously the compilation must be re-run. Build Cache identifies that the contents of the source file do not match anything in the cache and declare this a "cache miss". At this point, the task must be rerun as part of your build and the new results are stored in the Build Cache for future use.

The same thing happens if a header file which the source file includes is changed, or a header included by a header, and so on. Build Cache knows exactly which files were read in each task and can validate they did not change. The more your code changes, the less cache-hits Build Cache will find.

Build Cache is aware of the project folder structure. However, the root folder for the project does not need to be identical on all Initiators: one Initiator may place the project at C:\Project and anther Initiator may use D:\Project. They can use a shared cache efficiently.

Build Cache Size

The cache must be large enough to hold 2-3 builds results (objects, precompiled headers, libraries, and executables), from any builds that will contribute to this file cache. These requirements may vary depending on your environment.

You can find out how much space the build results of a project require by looking at the size of the build output folder(s).

Make sure that the filesystem holding the cache has enough free space to allow it to grow to its full size. Build Cache will stop filling the cache if the filesystem has less than 3% free space.

Once Build Cache is operational, you can view how much of your cache has been used by each build in the output tab of the Build Monitor, as well as the output you see in the command line.

File Cache Cleaning

The file cache is optimized in various ways such as removing duplicate files and automatic cleanups. If the file cache fills up to 95% of it's capacity, it is automatically cleaned up by removing some of the content until it reaches 85% capacity.

You can manually clear the cache as well in the Agent Settings of the Build Cache Endpoint.

You can view how much of your cache has been used in the output tab of the Build Monitor, as well as the output you see in the command line.

How is Build Cache Different from the Helper Cache

When Helpers are given a task, the input files (e.g. source and header files) they need to process are stored in the Helper Cache. If the same exact file is sent, the Helpers still process it again, but the files do not need to be re-transferred.

In Build Cache, the Initiator recognizes tasks that have already been processed, and they are skipped entirely. Output files (e.g. object files) do not need to be re-transferred and tasks do not need to be rerun.

Supported Tools

  • Build Systems: All build systems supported by Incredibuild.

  • Compilers: 

    • cl.exe (Visual Studio 2017 or higher)

    • clang-cl.exe (beta)

    • prospero-clang.exe (PlayStation PS5 SDK)

    • orbis-clang.exe (Playstation PS4 SDK)

    • clang ++ (Nintendo Switch SDK)

  • Linker: link.exe (Visual Studio 2017 or higher). Link.exe support in Build Cache is disabled by default. Enable it if your project has many DLL's or executables that rarely change.

  • /Z7 is fully supported. /Zi is automatically converted to /Z7 when Build Cache is used