Kubernetes

Incredibuild lets you accelerate CI processes by distributing them to available Incredibuild Helper machines. You install an Incredibuild Initiator in your build server container image, and connect it to an Incredibuild Coordinator that connects you Incredibuild Helper machines outside of Kubernetes (K8S).

This article will describe a typical CI environment using K8S, and explain how to integrate Incredibuild in the build process. We assume you already have a working CI environment (without Incredibuild).

Note:  The audience for this document are DevOps engineers, with sufficient experience in managing K8S clusters, pods and containers. We also assume the reader is familiar with Incredibuild and has used it outside of K8S.

Limitations

  • Only the Incredibuild Coordinator and Initiator Agents can be run in K8S . The Incredibuild Coordinator can be either in or outside of K8S. Incredibuild Helper Agents cannot be in K8S.

  • Build Cache is not supported for Initiators running in K8S.

  • Initiators spun up using K8S will automatically be associated with the Default Build Group. Using other Build Groups for K8S machines can be done using the CLI.

  • Typically, build servers running in CI are torn down (i.e. destroyed) when the build job is finished, which means:

    • The build history database and the build logs are lost, unless you collect these and store them in some log collection repository.

    • The build monitor is only available during the build, and the agent will disappear from the coordinator agent list once the container is terminated.

Setup the Incredibuild Environment

  1. Obtain SUVM licenses.

    1. Since build servers running in CI are typically torn down (i.e. destroyed) when the build job is finished, you will need SUVM licenses for enough cores to support the maximum concurrent build jobs. For details, see Single-Use Virtual Machine (SUVM) Licenses.

  2. Deploy Incredibuild in your environment:

    1. Deploy an Incredibuild Coordinator (can be in or outside of K8S).

    2. Deploy a set of Incredibuild Helper machines, AKA the helper grid outside the K8S cluster (can be bare metal, VM, or cloud).

    3. The Incredibuild initiator agents (inside K8S) need network access to the Coordinator and all Helper agents. Make sure that the relevant TCP/IP ports are open, see Hardware and Network Requirements

Note: Avoid over-provisioning the build server pods: starvation of CPU and other resources may damage your build performance even if Incredibuild acceleration is working.

Modify or Create a Kubernetes Image to Deploy an Incredibuild Initiator

You probably already have a build-server golden image, with a Linux distribution, compilers, and make system. The following Dockerfile represents a typical build-server golden-image (without Incredibuild):

Copy 
#######################################################
# build-image.Dockerfile
#######################################################
FROM ubuntu:20.04
ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=Etc/GMT 
RUN apt-get update
RUN apt-get install -y gcc make git bzip2

#######################################################
# The compilation commands executed during each CI job
#######################################################
CMD bash -c " \
  git clone https://github.com/gilnadel/Sample.git; \
  cd Sample; \
  make -j 10 ALL; "

The Dockerfile above begins with the commands that create the golden image, and ends with the compilation commands (getting the source code and compiling it).

  1. Add the following lines to the Dockerfile to install Incredibuild before the compilation command, but change IB_VERSION and IB_COORDINATOR to match your Incredibuild Coordinator:

    Copy 
    #######################################################
    # Install IB4Linux
    #######################################################
    ENV IB_VERSION=3.18.0
    ENV IB_COORDINATOR=192.168.10.50
    EXPOSE 8080
    EXPOSE 8081
    WORKDIR /root
    ADD https://www.incredibuild.com/downloads/incredibuild_$IB_VERSION.ubin ./incredibuild.ubin
    RUN chmod a+x ./incredibuild.ubin
    RUN ./incredibuild.ubin install -I -O $IB_COORDINATOR -A /etc/incredibuild -R SUVM -P

  2. If you want to use the user interfaces of the Initiator Agents on containers (build monitor, build history, etc.), make sure the HTTP and/or HTTPS ports are open on the container for external access. For more information, see Modifying Ports.

  3. Add the following line to the compilation command (this will start the Incredibuild agent when the pod is run during the CI). These commands make sure that the Incredibuild agent you start consume a fixed number of CPU cores from your SUVM license limit. Change the value 8 in the 1st command to match the number of cores you want your containerized build server to really use for building (typically this should match the number of CPUs you allocate to the container).

    Copy 
    # Change the parameters below to fit the size of your pod
    /opt/incredibuild/management/set_agent_params.py max-initiator-cores 8;
    /opt/incredibuild/etc/init.d/incredibuild start; 
    sleep 10;

  4. Wrap the make command with the following ib_console command, and increase the build parallelism according to the potential of your project and Helper cores availability:

    Copy 
    ib_console -f make -j 100 ALL;

  5. In order to allow Incredibuild to diagnose build operation, add the following after the build execution. Upload the resulting file /etc/incredibuild/tmp/incredibuild_logs.tgz to a location outside of Kubernetes (e.g. your log collection repository). Without exporting and saving the logs, Incredibuild will not be able to provide support for failed or under-performing builds.

    Copy 
    tar -C /etc/incredibuild -I '/opt/incredibuild/bin/pigz' -cf /etc/incredibuild/tmp/incredibuild_logs.tgz db log;

Sample Complete Dockerfile with Incredibuild

Every environment is different so this example will probably need to be adjusted for your environment.

Copy 
#######################################################
# build-image.Dockerfile
#######################################################
FROM ubuntu:20.04
ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=Etc/GMT 
RUN apt-get update
RUN apt-get install -y gcc make git

#######################################################
# Install IB4Linux
#######################################################
ENV IB_VERSION=3.18.0             
ENV IB_COORDINATOR=192.168.10.50  
EXPOSE 8080
EXPOSE 8081
WORKDIR /root
ADD https://www.incredibuild.com/downloads/incredibuild_$IB_VERSION.ubin ./incredibuild.ubin
RUN chmod a+x ./incredibuild.ubin
RUN ./incredibuild.ubin install -I -O $IB_COORDINATOR -A /etc/incredibuild -R SUVM -P

#######################################################
# Compilation step, including IB agent startup
#######################################################
CMD bash -c -x " \
  /opt/incredibuild/management/set_agent_params.py max-initiator-cores 8; \
  /opt/incredibuild/etc/init.d/incredibuild start; sleep 10; \
  git clone https://github.com/gilnadel/Sample.git; \
  cd Sample; \
  ib_console -f -p gcc.xml --no-cgroups make -j 100 ALL; \
  tar -C /etc/incredibuild -I '/opt/incredibuild/bin/pigz' -cf /etc/incredibuild/tmp/incredibuild_logs.tgz db log; "

Building the Golden Image

Build the container golden image and push it to the image repository. See the following sample Docker command:

Copy 
docker build -f build-image.Dockerfile -t build-image:latest --ulimit nofile=100000:100000 .

Note the necessary ulimit build option.

Upgrading Incredibuild

  1. Upgrade your Incredibuild Coordinator. At the end of this process, any Helper Agents connected to the Coordinator will be automatically upgraded.

  2. Update your Kubernetes golden image so that future Helper Agents will be installed using the latest version of Incredibuild.
    If this is not done, they will upgraded as soon as they are connected to the Coordinator. However, this will greatly slow down their deployment.