Scaling Helpers in Azure

Overview

This topic explains the mechanics involved in scaling cloud Helpers automatically using Virtual Machines Scale Sets (VMSSs), for Incredibuild for Windows and Incredibuild for Linux. The process involves these steps:

  1. Create a Virtual Machine Scale Set (one or more).

  2. For each of the Virtual Machine Scale Sets above, attach a startup script to install Incredibuild.

  3. For each of the Virtual Machine Scale Sets above, create autoscale rules.

Because of the way autoscaling works, you will need to have a single Helper machine alive in the VPC at all times. The Autoscaling Group spins up machines based on the load on the entire group, so one machine needs to be available as a bootstrap.

Requirements

To use autoscaling you must have:

  • An Azure account, with enough permissions.

  • Access between your Coordinator and Initiators to the Virtual Network (VNET) where the helpers will run, over all relevant TCP ports (see System Requirements).

  • It is required to allow the Cloud Helpers access to the Coordinator via TCP port 32103 (or 31104 if you choose to leave the communication unencrypted).

It is assumed that:

  • You have a working Incredibuild environment, either on-prem or in your VNET.

  • You have a network connection between your Coordinator and Initiators to/from your VNET.

  • You are experienced with Incredibuild technology.

  • You are well versed in managing Azure Virtual Machines.

Create a Virtual Machine Scale Set

The Virtual Machine Scale Set (VMSS) you create will include the details about the image (Windows or Linux), the network requirements for the Helper instances (so they can communicate with the Coordinator and Initiators) and the Virtual Machine size.

Incredibuild for Windows

Create the VMSS with the Win2022Datacenter image, and default login credentials. Select the appropriate machine size and authentication method.

Copy 
az vmss create \ 
    --name vmss3 --resource-group VMSS-tests \ 
    --instance-count 0 --vm-sku Standard_D4lds_v5 \ 
    --image Win2022Datacenter --os-disk-size-gb 127 \ 
    --admin-username Incredibuild --admin-password ChangePass123  

Incredibuild for Linux

Use the following example to create a startup script for Linux. Make sure you download the Incredibuild release that matches your Coordinator.

Copy 
COORD=IN GCP:"192.168.1.1"
  
URL="https://incredibuild.com/downloads/incredibuild_4.13.0.run" 

SETTINGS='{"commandToExecute" : "set -x > /tmp/boot.log 2>&1; curl -L '$URL' -o ./ibsetup.run;  chmod a+x ./ibsetup.run; sudo apt-get install –y bzip2; sudo ./ibsetup.run --action install --helper enabled --license-type SUVM --data-dir /etc --coordinator-machine '$COORD' --skip-coordinator-test true; sudo /opt/incredibuild/etc/init.d/incredibuild start "}' 
 
az vmss extension set \ 
    --vmss-name vmss4 --resource-group VMSS-tests \ 
    --publisher Microsoft.Azure.Extensions \ 
    --name CustomScript \ 
    --settings "$SETTINGS" 

Create Autoscale Rules

The autoscale commands allow you to specify when the VMSS will spin up/down virtual machines. The following sample CLI commands instruct the VMSS to spin up a new VM when the load on an existing VM exceeds 50% for 2 minutes, and spin a VM down if the load is below 30% for 2 minutes.

Copy 
az monitor autoscale create \ 
    --resource-group VMSS-tests --resource vmss3 \ 
    --resource-type Microsoft.Compute/virtualMachineScaleSets \ 
    --name vmss3autoscale \ 
    --min-count 1 --max-count 4 --count 1 

az monitor autoscale rule create \ 
    --resource-group VMSS-tests --autoscale-name vmss3autoscale \ 
    --condition "Percentage CPU > 50 avg 2m" --scale out 1 

az monitor autoscale rule create \ 
    --resource-group VMSS-tests --autoscale-name vmss3autoscale \ 
    --condition "Percentage CPU < 30 avg 5m" --scale in 1 

Set Helper Assignment Priority

To achieve the highest efficiency and save on cloud costs, you need to give on-prem Helpers a lower assignment priority than Helpers that launch in the cloud VPC. A lower priority value means the Coordinator would choose to use the on-prem Helpers over cloud Helpers.

The simplest way to do that is to use the Coordinator Manager and set the Helper Assignment Priority of the on-prem Helpers to 1 (highest priority).

Alternatively, the Incredibuild installer includes the option to set the priority during the installation. Instead of the above, change the installation command above (in the Launch Template definition) to set their priority to 5.