Build Cache

Overview

Build Cache saves time and resources by reusing portions of previous builds. As you run builds, we store the outputs in a file cache. When future builds are run, we check the cache to avoid regenerating portions of the build that have not changed.

This can have a major impact, as you can reuse previous build data stored in Build Cache to drastically reduce build times without impacting your bandwidth.

You can use Build Cache locally by creating and reusing content from a single machine, create a shared cache for more than one Initiator (beta feature), or use both local and shared.

To use Build Cache when running projects, add the parameters below to your build commands. This checks to see if there is anything useful in the cache, and stores the build you are running in the cache for future use. If you are using shared cache, you also need to install the Build Cache Service on a dedicated machine.

Using local build cache requires additional storage space on the Initiator machine. We recommend allocating at least twice the size of the project output folder.

Local vs. Shared Build Cache

Note: Shared Cache is currently a beta feature. Contact your Customer Success Manager to join the beta program.

You can use both a local and a shared Build Cache in the same build command. To use a local cache, simply use the --build-cache-local parameters when running a build (details below).

To use a shared cache, you install the Build Cache Service on a dedicated machine, and then use the parameters below. An additional flag is required to point to the Build Cache Service.

When both local and shared are used, Incredibuild uses the following flow when building each object:

Check the local cache, if there is a cache hit retrieve it and avoid building the object.
If there is no local cache hit, check the shared cache. If there is a shared cache hit, retrieve it and avoid building the object. Afterwards, update the local cache.
If there is no local or shared cache hit, build the object and update both the local and shared caches.

Installing the Build Cache Service

If you are using shared cache, you need to install the Build Cache Service on a dedicated machine. Use the following format:

sudo bash ./incredibuild_<version>.ubin install -Y -O <Primary_Coordinator_Machine_IP_or_Hostname> -A <Path_to_Incredibuild_Installation_Directory> -Y -Q <size of cache>

For more information, see Installing Components.

Build Cache Parameters

The following parameters can be added to your build command to use Build Cache. Even if Build Cache is enabled, the parameters must be used on every build that you want to use Build Cache.

Parameter	Description	Notes
--build-cache-local-user --build-cache-local-shared --build-cache-local-group="groupname"	Add one of these three commands to use local Build Cache. You can specify whether the cache is accessible and shared across a single user on this machine (-user), everyone on this initiator (-shared), or a specific group of users that includes the current user (-group) This can be used on it's own, or in addition to a Shared Cache. Note: These parameters were changed in version 3.19. They used to be --avoid-local-user\|group\|shared. The original parameters will still work but may be deprecated in future releases.
--build-cache-service= <URL>	Use a Shared Cache running on a Build Cache Service defined by the protocol, an IP or hostname, and the port number of the host machine. For exampe: http://build-cache-service:8080. The protocol can be either http or https, but we recommend using the http protocol to improve performance. You can define a default Build Cache Service in the Agent Settings or using the CLI. If you have defined a default, you can use this parameter with no URL. If you define a default and use a different URL in the parameter, the value specified in the parameter will be used.
--build-cache-basedir=<path>	Incredibuild views files originating from different paths as cache misses, even if the files are identical. This parameter allows you to define which part of the path is NOT shared by other user running similar builds. Incredibuild will then ignore this portion of the path when determining if two files are identical. For example, if you include usernames as part of your path, (e.g. /home/bill_gates/windows_vista/libs/src/pm-functions.cpp), you might define the basedir as /home/bill_gates/windows_vista/. This parameter is required. Note: This parameter was changed in version 3.19. It used to be --avoid-basedir. The original parameter will still work but may be deprecated in future releases.

Parameter

Description

Notes

--build-cache-local-user

--build-cache-local-shared

--build-cache-local-group="groupname"

Add one of these three commands to use local Build Cache. You can specify whether the cache is accessible and shared across a single user on this machine (-user), everyone on this initiator (-shared), or a specific group of users that includes the current user (-group)

This can be used on it's own, or in addition to a Shared Cache.

Note: These parameters were changed in version 3.19. They used to be --avoid-local-user|group|shared. The original parameters will still work but may be deprecated in future releases.

--build-cache-service= <URL>

Use a Shared Cache running on a Build Cache Service defined by the protocol, an IP or hostname, and the port number of the host machine. For exampe: http://build-cache-service:8080.

The protocol can be either http or https, but we recommend using the http protocol to improve performance.

You can define a default Build Cache Service in the Agent Settings or using the CLI. If you have defined a default, you can use this parameter with no URL. If you define a default and use a different URL in the parameter, the value specified in the parameter will be used.

--build-cache-basedir=<path>

Incredibuild views files originating from different paths as cache misses, even if the files are identical. This parameter allows you to define which part of the path is NOT shared by other user running similar builds. Incredibuild will then ignore this portion of the path when determining if two files are identical.

For example, if you include usernames as part of your path, (e.g. /home/bill_gates/windows_vista/libs/src/pm-functions.cpp), you might define the basedir as /home/bill_gates/windows_vista/.

This parameter is required.

Note: This parameter was changed in version 3.19. It used to be --avoid-basedir. The original parameter will still work but may be deprecated in future releases.

Build Examples

Run local cache only for all users on an initiator

ib_console --build-cache-local-shared --build-cache-basedir=/home/myuser/opencv make -j 40

Run local and shared cache

ib_console --build-cache-local-shared --build-cache-service=http://192.54.11.9:8080 --build-cache-basedir=/home/myuser/opencv make -j 40

Local Cache Settings

You can edit the local Build Cache size in the Agent Settings > General > Local Build Cache Size setting. This can also be configured when installing an Initiator.

You can also run the following commands on an Initiator using local Build Cache to configure other local cache settings:

Return information about the location and size of your local build cache:

/opt/incredibuild/management/build_avoid_cache.sh <user|shared|group='groupname'> info
Set the local build cache size. By default, the size is 100GB, but you can modify it:

/opt/incredibuild/management/build_avoid_cache.sh <user|shared|group='groupname'> set_size <size in GB>
Clear the local build cache:

/opt/incredibuild/management/build_avoid_cache.sh <user|shared|group='groupname'> clear

Shared Cache Settings

You can define a default Build Cache Service in the Agent Settings > General > Build Cache Service URL setting. If you use the --build-cache-service parameter with no value, the value defined here will be used. If you specify a different value, it will override the value in the Agent Settings.

You can also modify this setting using the command line interface.

You can also run the following commands on the Build Cache Service machine to configure other settings:

Get the shared build cache size. By default, the size is 100GB:

sudo /opt/incredibuild/management/build_cache_service.py get-size-limit
Set the shared build cache size. By default, the size is 100GB.

sudo /opt/incredibuild/management/build_cache_service.py set-size-limit <size>
Return the location of the shared build cache on the Build Cache Service machine:

sudo /opt/incredibuild/management/build_cache_service.py get-path
Clear the shared build cache:

sudo /opt/incredibuild/management/build_cache_service.py clear
Return the currently used size of the Build Cache on the Build Cache Service machine:

sudo /opt/incredibuild/management/build_cache_service.py get-used-size