Scaling Helpers in AWS EC2 - IncrediBuild Documentation

This topic explains how to scale Incredibuild Helper Agents automatically with AWS Auto Scaling Groups, using the AWS CLI or the AWS Management Console.

Prerequisites

Create an Incredibuild grid that consists of the following:

Initiator
Coordinator
Helper (on Windows, SCL or Linux) which will be used to create the Launch Template.

Ensure that you know the credentials or have the PKI key of the Helper machine.
Ensure that hostname of the Helper machine is NOT static.
Ensure that the Incredibuild grid (especially the Helper) is working prior to creating the Auto Scaling group. Do this by testing the build capability and ensuring the Helper is used during the build.

Create an Auto Scaling Group using AWS CLI

Copy the following values from the Incredibuild grid:

Helper instance: <instanceID>
Subnet: <SubnetID>
Security group or list of security groups: <SecurityGroup>

Select a VM size for machines to be created by the Auto Scaling group:

Instance type: <instanceType>, for example, c5a.2xlarge

Stop the Helper instance:

aws ec2 stop-instances \
--instance-ids <instanceID>

Create the AMI (image) from the Helper instance:

aws ec2 create-image \
--instance-id <instanceID> \
--name "Helper-ImageName" \
--tag-specifications "ResourceType=image,Tags=[{Key=Name,Value=Helper-ImageName}]"

Copy the AMI ID <amiID>, for example:

"ImageId": "ami-0df8cbefb46402855"

Check the AMI creation state:

aws ec2 describe-images --image-id <amiID> --query 'Images[0].State'

When the returned State is available, create the Launch Template:

aws ec2 create-launch-template --launch-template-name Helper_Template --version-description Helper_Template_v1 \
--launch-template-data '{"NetworkInterfaces":[{"DeviceIndex":0,"SubnetId":"<SubnetID>","Groups": ["<SecurityGroup>"]}],"ImageId":"<amiID>","InstanceType":"<instanceType>"}'

Note the launch template ID <LaunchTemplateId>, for example:

"LaunchTemplateId": "lt-02d7a3f031e94d432"

Create the Auto Scaling group, changing the max-size to the required integer value:

aws autoscaling create-auto-scaling-group \
--auto-scaling-group-name Helper_ASG --launch-template LaunchTemplateId=<LaunchTemplateId> \
--min-size 1 --max-size 5 --desired-capacity 1 --default-cooldown 30 --default-instance-warmup 60 --vpc-zone-identifier "<SubnetID>"

Add a scaling policy to the Auto Scaling group based on metrics of CPU utilization (50%):

aws autoscaling put-scaling-policy --auto-scaling-group-name Helper_ASG \
--policy-name cpu50-target-tracking-scaling-policy \
--policy-type TargetTrackingScaling \
--target-tracking-configuration '{"TargetValue": 50.0, "PredefinedMetricSpecification": {"PredefinedMetricType": "ASGAverageCPUUtilization"}}'

Full example:

aws ec2 stop-instances \
--instance-ids i-03642a8f7975ac99f
# Wait until the instance stops by running the following command:
aws ec2 wait instance-stopped --instance-ids i-03642a8f7975ac99f
aws ec2 create-image \
--instance-id i-03642a8f7975ac99f \
--name "Helper-ImageName" \
--tag-specifications "ResourceType=image,Tags=[{Key=Name,Value=Helper-ImageName}]"
# Copy the <amiID>, for example:
"ImageId": "ami-0df8cbefb46402855"
# Wait until the AMI creation finishes by running the following command:
aws ec2 wait image-available --image-ids ami-0df8cbefb46402855
aws ec2 create-launch-template --launch-template-name Helper_Template --version-description Helper_Template_v1 \
--launch-template-data '{"NetworkInterfaces":[{"DeviceIndex":0,"SubnetId":"subnet-0b184d86f43ece9ce","Groups": ["sg-0629670940885903c"]}],"ImageId":"ami-0df8cbefb46402855","InstanceType":"c5a.2xlarge"}'
# Copy the <LaunchTemplateId>, for example:
"LaunchTemplateId": "lt-02d7a3f031e94d432"
aws autoscaling create-auto-scaling-group \
--auto-scaling-group-name Helper_ASG --launch-template LaunchTemplateId=lt-02d7a3f031e94d432 \
--min-size 1 --max-size 5 --desired-capacity 1 --default-cooldown 30 --default-instance-warmup 60 --vpc-zone-identifier "subnet-0b184d86f43ece9ce"
aws autoscaling put-scaling-policy --auto-scaling-group-name Helper_ASG \
--policy-name cpu50-target-tracking-scaling-policy \
--policy-type TargetTrackingScaling \
--target-tracking-configuration '{"TargetValue": 50.0, "PredefinedMetricSpecification": {"PredefinedMetricType": "ASGAverageCPUUtilization"}}'

Create an Auto Scaling Group using AWS Console

Gather Incredibuild Initiator Information

Sign in to the AWS Management Console.
Navigate to the EC2 > Instances.
Select the Initiator instance and copy the following info:

VPC ID
Subnet ID

Select the Security tab and copy the required Security Group.

Create an AMI from Helper Machine

Sign in to the AWS Management Console.
Navigate to EC2 > Instances.
Select the Helper instance and choose Instance state > Stop instance. When prompted for confirmation, choose Stop. It can take a few minutes for the instance to stop.
To create the AMI, select the Helper instance, select Actions > Image and templates > Create image.
Enter the Image name.
Select Tag image and snapshots together.

Click Create image.
Copy the Image name for future use.

Create Template from AMI

In EC2 > Instances, select Launch Templates > Create launch template.
Enter the Launch template name.
Copy the Launch template name for future use.

Under Application and OS Images (Amazon Machine Image), paste the AMI Image name.

Press Enter. The Choose an Amazon Machine Image (AMI) page appears.
Select your AMI image and then click Select.

Scroll down and configure the Network settings:

From the Subnet dropdown, select the SubnetID.
From the Common security groups dropdown, select the Security Group ID of the grid.

Click Create Launch template.

Once created, a Success message appears.

Create Auto Scaling Group from Template

In EC2, select Auto Scaling > Auto Scaling Groups.
Click Create Auto Scaling group. A wizard appears.
In Step 1 - Choose launch template or configuration, configure the following:

Enter the Auto Scaling Group name.
From the Launch template dropdown, select the template to be used for machine creation in the Auto Scaling group (created in Create Template from AMI).

Click Next.
In Step 2- Choose instance launch options configure the following:

In the Instance type requirements area, select the base model for the Helper machine to be used for scaling. Either select a specific AWS instance type or configure it manually.

(Optional) In the Instance purchase options area, select On-Demand or Spot instances.

In the Network area, select the VPC ID and Subnet ID copied from the Initiator configuration.

Click Next.
In Step 3(optional) - Configure advanced options, click Next.
In Step 4 (optional) - Configure group size and scaling, provision the scaling of the Helpers group:

In the Group size area, enter the Desired capacity of the group.
In the Scaling area, enter the Min desired capacity and Max desired capacity (minimum and maximum number of Helpers in the grid).
Select the Target tracking scaling policy.
From the Metric type dropdown, select Average CPU utilization. This metric is used for sizing.
Enter a Target Value, for example, 50, meaning 50% CPU usage.
In Instance warmup, enter the number of seconds that it takes for a newly launched instance to warm up. Until its specified warmup time has expired, an instance is not counted toward the aggregated EC2 instance metrics. In this example, 60 seconds.

Click Next until you get to the end of the wizard.
Click Skip to review at the bottom of the page.
In the Review page, scroll down and click Create Auto Scaling group. The Auto Scaling Groups page appears.

Note that the first Helper instance is started (Updating capacity).
Click on group Name.
On the Details tab, scroll down to Advanced configurations and click Edit.
Set the Default cooldown to 0 second.
Click Update. This means that instances are terminated immediately when there is no need for scale capacity.

On the Activity tab, view the log of the latest actions. Each time an additional instance is created or terminated, it appears in the log.

On the Instance management tab, view the details of the active instances. For example, when a new Helper is created. The new Helper instance also appears in the Coordinator Management page a short while later.

Now start the build with Incredibuild. When the CPU utilization on the Helper machine exceeds 50%, new instances are created.
Navigate to your group in EC2 > the Auto Scaling groups, and view the Activity history in the Activity tab.

Create an Auto Scaling Group with Mixed Instance Types

Create an Auto Scaling Group with Mixed Instance Types using AWS Console

To create an Auto Scaling group with mixed instance types, in Step 2- Choose instance launch options click Override launch template.

Configure the specific Instance types requirements and the On-Demand and Spot instances.

Create an Auto Scaling Group with Mixed Instance Types using AWS CLI

To create an Auto Scaling group with mixed instance type in AWS CLI:

ASG_NAME="Helper_SCL_ASG"
LAUNCH_TEMPLATE_ID="lt-0b66a45ee243120f9"
MIN_SIZE=1
MAX_SIZE=5
DESIRED_CAPACITY=1
COOLDOWN=30
WARMUP=60
SUBNETS="subnet-0b184d86f43ece9ce"
INSTANCE_TYPE_1="c5a.2xlarge"
INSTANCE_TYPE_2="c6a.2xlarge"
aws autoscaling create-auto-scaling-group \
--auto-scaling-group-name "$ASG_NAME" \
--mixed-instances-policy '{"LaunchTemplate":{"LaunchTemplateSpecification":{"LaunchTemplateId":"'"$LAUNCH_TEMPLATE_ID"'"},"Overrides":[{"InstanceType":"'"$INSTANCE_TYPE_1"'","WeightedCapacity":"1"},{"InstanceType":"'"$INSTANCE_TYPE_2"'","WeightedCapacity":"1"}]},"InstancesDistribution":{"OnDemandBaseCapacity":0,"OnDemandPercentageAboveBaseCapacity":100,"SpotAllocationStrategy":"lowest-price"}}' \
--min-size $MIN_SIZE \
--max-size $MAX_SIZE \
--desired-capacity $DESIRED_CAPACITY \
--default-cooldown $COOLDOWN \
--default-instance-warmup $WARMUP \
--vpc-zone-identifier "$SUBNETS"

Create an SCL Helper for the Auto Scaling Group

Create an SCL Helper

A VPC, subnet and security groups are required to connect the Auto Scaling group instance to an existing Incredibuild grid (Coordinator/Initiator machines) and to your machine for AMI updates.

In the AWS Management Console, navigate to EC2 > AMI Catalog.
Filter, locate, and select the required AMI.

In the upper-right corner, click Launch instance from AMI. The Launch an instance page appears.
Complete the following:
- Name: Helper-SCL
- Instance type: c5a.2xlarge
- Key pair (login): If you already have a key pair that you use to connect to your instances, select it. Otherwise, choose Create new key pair. A <key_pair_name>.pem file will be downloaded and can be used to connect to the instance.
- Network settings: Select the appropriate VPC, subnet, and security group(s).
- Configure storage: At least 20 GB.
Click Launch instance.

Copy the Instance ID (for example, i-0b0edfce604652eff) and open the instance details page.
Once the instance reaches the Running state, note its Public IPv4 address. Use SSH or SCP to connect to the instance. On Windows, it is recommended to use a Bash-based SSH client, such as Git Bash or MobaXterm.
Before proceeding, obtain the Incredibuild console installer (ibsetup_<version>_console.exe). Ensure that the installer version matches the Incredibuild version deployed in your grid.
Create a folder named SCL and copy the installer into it.
Within the SCL folder, create a file named Dockerfile with the following contents:

FROM ubuntu:22.04
# Declare build-time arguments
ARG HELPER_CORES
ARG SERVICE_PORT
ARG DEFAULT_SERVICE_PORT=31105
ARG HELPER_PORT
ARG DEFAULT_HELPER_PORT=31106
ARG COORD_IP
# Make them available as environment variables
ENV HELPER_CORES=$HELPER_CORES \
SERVICE_PORT=$SERVICE_PORT \
DEFAULT_SERVICE_PORT=$DEFAULT_SERVICE_PORT \
HELPER_PORT=$HELPER_PORT \
DEFAULT_HELPER_PORT=$DEFAULT_HELPER_PORT \
COORD_IP=$COORD_IP
# Specify a workdir to better organize files
RUN mkdir -p /ib-agent
WORKDIR /ib-agent
# Update package lists and install necessary packages
RUN apt-get update && \
apt-get install -y --no-install-recommends wget software-properties-common gnupg2 winbind xvfb curl vim
# Add WineHQ GPG key and store it in /usr/share/keyrings/
RUN wget -nc https://dl.winehq.org/wine-builds/winehq.key && \
gpg --no-default-keyring --keyring /usr/share/keyrings/winehq-archive-keyring.gpg --import winehq.key
# Add the WineHQ repository with the signed-by option
RUN echo "deb [signed-by=/usr/share/keyrings/winehq-archive-keyring.gpg] https://dl.winehq.org/wine-builds/ubuntu/ jammy main" > /etc/apt/sources.list.d/winehq.list
# Update package lists and install Wine
RUN dpkg --add-architecture i386 && \
echo 'quiet = on' >> ~/.wgetrc && \
apt-get update && \
apt-get install -y --no-install-recommends wine-stable=8.0.2~jammy-1 wine-stable-amd64=8.0.2~jammy-1 wine-stable-i386=8.0.2~jammy-1
# Install winetricks and configure Wine
#RUN apt-get install -y winetricks && \
# winetricks msxml6 wininet
RUN apt-get install -y winetricks && \
wget -O /tmp/msxml6-KB973686-enu-amd64.exe "https://s3.us-east-1.amazonaws.com/downloads.incredibuild.com/public+s3+uploads/winetricks/MSXML6-KB973686-enu-amd64.exe" && \
mkdir -p ~/.cache/winetricks/msxml6 && \
cp /tmp/msxml6-KB973686-enu-amd64.exe ~/.cache/winetricks/msxml6/ && \
winetricks msxml6 wininet
# Cleanup unnecessary files
RUN apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
# Create a symbolic link for the wine command to point to wine64 binary
RUN ln -s /opt/wine-stable/bin/wine64 /usr/local/bin/wine
# Create x11.reg file with the specified content
RUN echo 'REGEDIT4\n\n[HKEY_CURRENT_USER\\Software\\Wine\\X11 Driver]\n"UseXRandR"="N"\n"UseXVidMode"="N"' > ./x11.reg
RUN /opt/wine-stable/bin/wine64 regedit ./x11.reg
# Copy install required packages
COPY ./ibsetup_*_console.exe ./ib_setup_console.exe
COPY ./winemodal.sh ./winemodal.sh
RUN chmod +x ./winemodal.sh
RUN ["sh", "-c", "xvfb-run -a ./winemodal.sh wine ib_setup_console.exe /Install /Components=Agent /Agent:AgentRole=Helper /Agent:HelperType=Floating /Agent:SingleUse=True /Agent:InstallAddins=OFF /Agent:OpenFirewall=OFF /Agent:AutoSelectPorts=OFF /Agent:HelperCores=$HELPER_CORES /Agent:ServicePort=${SERVICE_PORT:-$DEFAULT_SERVICE_PORT} /Agent:HelperPort=${HELPER_PORT:-$DEFAULT_HELPER_PORT} /Coordinator=$COORD_IP"]
RUN ["sh","-c","cat >./ib-group.reg <<'EOF'\nWindows Registry Editor Version 5.00\n\n[HKEY_LOCAL_MACHINE\\Software\\Xoreax\\IncrediBuild\\BuildService]\n\"Group\"=\"Universal Helpers\"\n\"GroupId\"=\"__UNIVERSAL__\"\nEOF"]
# Set the CMD to start incredibuild_agent
#CMD ["sh", "-c", "wine sc start incredibuild_agent; sleep 10; wine sc query incredibuild_agent | grep -q RUNNING || exit 1; sleep infinity"]
CMD ["sh", "-c", "wine regedit.exe /S ./ib-group.reg; xvfb-run -a ./winemodal.sh wine sc start incredibuild_agent; sleep 10; xvfb-run -a ./winemodal.sh wine sc query incredibuild_agent | grep -q RUNNING || exit 1; sleep infinity"]

Note: The Helper is created in the Universal Helpers build group, and it will be able to allocate Helper cores to all available Initiators in the grid.

Create a file named winemodal.sh with the following contents:

#!/bin/bash -u
export WINEDEBUG=-all
setupLogFile="./winemodal.log"
echo "$(date +'%Y-%m-%d %T'): executing $@" >> $setupLogFile
$@

Tar (or 7zip on Windows) the SCL folder with the three files:

winemodal.sh
Dockerfile
ibsetup_<version>_console.exe

In Bash, for example, Git Bash:

# Copy the relevant files to the new instance created in the previous step
# Provide the location of the pem file and SCL.tar files
scp -i "Desktop/WINDOWS/AWS_keys/yuri_temp.pem" Desktop/CLOUD_HELPERS/SCL.tar ubuntu@10.133.9.86:.

Once the file is copied to the instance, connect to the instance with SSH:

ssh -i "Desktop/WINDOWS/AWS_keys/yuri_temp.pem" ubuntu@10.133.9.86

Once logged in, run the following commands:

sudo apt update && sudo apt install -y docker.io
sudo usermod -aG docker $USER
exit

Connect again with SSH:

ssh -i "Desktop/WINDOWS/AWS_keys/yuri_temp.pem" ubuntu@10.133.9.86

Once logged in, run the following commands:

tar -xvf SCL.tar
cd SCL
docker build \
--build-arg HELPER_CORES="$(nproc)" \
--build-arg COORD_IP="10.133.27.8" \
-t scl .
# You may use the following commands for testing purposes
# docker run -d --network host --name scl scl
# docker rm -f scl
sudo tee /etc/systemd/system/scl.service <<'EOF'
[Unit]
Description=SCL Container
After=docker.service
Requires=docker.service
[Service]
Restart=always
ExecStartPre=-/usr/bin/docker container prune -f
ExecStart=/usr/bin/docker run --rm \
--name scl \
--network host \
scl
ExecStop=/usr/bin/docker stop scl
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable scl.service
sudo poweroff

Create an AMI from SCL Helper using AWS Console

Sign in to the AWS Management Console.
Navigate to EC2 > Instances.
Select the Helper instance and choose Instance state > Stop instance. When prompted for confirmation, choose Stop. It can take a few minutes for the instance to stop.
To create the AMI, select the Helper instance, select Actions > Image and templates > Create image.
Enter the Image name (Helper-SCL).
Add a new tag with Key=Name and Value=Helper-SCL.
Click Create image.

Create an AMI from SCL Helper using AWS CLI

To create an AMI from SCL Helper in AWS CLI:

aws ec2 stop-instances \
--instance-ids i-0b0edfce604652eff
# Wait until instance stops:
aws ec2 wait instance-stopped --instance-ids i-0b0edfce604652eff
aws ec2 create-image \
--instance-id i-0b0edfce604652eff \
--name "Helper-SCL" \
--tag-specifications "ResourceType=image,Tags=[{Key=Name,Value=Helper-SCL}]"
# Note <amiID>, for example:
"ImageId": "ami-06c5b4e97014be3d9"
# Wait for AMI creation to finish by running the following command:
aws ec2 wait image-available --image-ids ami-0e6d0cc30a40e4ad4

Create a Windows Helper for the Auto Scaling Group

Create a Windows Helper

A VPC, subnet and security groups are required to connect the Auto Scaling group instance to an existing Incredibuild grid (Coordinator/Initiator machines) and to your machine for AMI updates.

In the AWS Management Console, navigate to EC2 > AMI Catalog.
Filter, locate, and select the required AMI.
In the upper-right corner, click Launch instance from AMI. The Launch an instance page appears.
Complete the following:
- Name: Helper-WIN
- Instance type: c5a.2xlarge
- Key pair (login): If you already have a key pair that you use to connect to your instances, select it. Otherwise, choose Create new key pair. A <key_pair_name>.pem file will be downloaded and can be used to connect to the instance.
- Network settings: Select the appropriate VPC, subnet, and security group(s).
- Configure storage: At least 20 GB.
Click Launch instance.
Copy the Instance ID (for example, i-0b0edfce604652eff) and open the instance details page.
Once the instance reaches the Running state, connect to it using RDP.
Download the Remote Desktop File.
Click Get Windows password, provide the .pem key file, and decrypt the administrator password.
Use the downloaded RDP file and the decrypted password to connect to the Windows instance using an RDP client.
Before starting, prepare the Incredibuild installation file for either GUI or console installation. Use the same version that is installed in your Incredibuild grid (ibsetup_<version>.exe or ibsetup_<version>_console.exe).
Copy the installation file to the Windows instance and run the Incredibuild installation using either the GUI or the command prompt.
If using the command prompt:

set "HELPER_CORES=%NUMBER_OF_PROCESSORS%"
set "COORD_IP=10.133.27.8"
set "HELPER_PORT=31105"
set "SERVICE_PORT=31106"
ibsetup_10.32.0.11685_console.exe /Install /Components=Agent /Agent:AgentRole=Helper /Agent:HelperType=Floating /Agent:SingleUse=True /Agent:InstallAddins=OFF /Agent:OpenFirewall=OFF /Agent:AutoSelectPorts=OFF /Agent:HelperCores=%HELPER_CORES% /Agent:ServicePort=%SERVICE_PORT% /Agent:HelperPort=%HELPER_PORT% /Coordinator=%COORD_IP% /Agent:Group="Universal Helpers"

Note: The Helper is created in the Universal Helpers build group, and it will be able to allocate Helper cores to all available Initiators in the grid.

Once the Incredibuild installation is completed, clear the machine state. In CMD run:

"C:\Program Files\Amazon\EC2Launch\ec2launch.exe" sysprep

After the RDP session disconnects, create the AMI from the instance.

Create a Linux Helper for the Auto Scaling Group

Create a Linux Helper

A VPC, subnet, and security groups are required to connect the Auto Scaling Group instance to an existing Incredibuild grid (Coordinator/Initiator machines) and to your machine for AMI updates.

In the AWS Management Console, navigate to EC2 > AMI Catalog.
Filter, locate, and select the required AMI.

In the upper-right corner, click Launch instance from AMI. The Launch an instance page appears.
Complete the following:
- Name: Helper-LIN
- Instance type: c5a.2xlarge
- Key pair (login): If you already have a key pair that you use to connect to your instances, select it. Otherwise, choose Create new key pair. A <key_pair_name>.pem file will be downloaded and can be used to connect to the instance.
- Network settings: Select the appropriate VPC, subnet, and security group(s).
- Configure storage: At least 30 GB.
Click Launch instance.
Note the Instance ID (for example, i-0b0edfce604652eff) and open the instance details page.
Once the instance reaches the Running state, note its public IP address and connect to it using SSH.
Prepare the Linux Incredibuild installation package. Use the same version that is installed in your existing Incredibuild grid (ibsetup_linux_<version>.run).
Copy the installation package to the instance using SCP.

scp -i "<path_to_key>.pem" ibsetup_linux_<version>.run ubuntu@<instance_ip>:.

Connect to the instance using SSH.

ssh -i "<path_to_key>.pem" ubuntu@<instance_ip>

Make the installer executable and install the Helper.

sudo chmod +x ibsetup_linux_<version>.run
sudo ./ibsetup_linux_<version>.run \
--action install \
--helper enabled \
--license-type SUVM \
--coordinator-machine <Coordinator_IP> \
--data-dir /etc/

After the installation completes, prepare the instance for AMI creation by clearing machine-specific information.

sudo systemctl stop cloud-init
sudo cloud-init clean --logs --seed
sudo truncate -s 0 /etc/machine-id
sudo rm -f /var/lib/dbus/machine-id
sudo shutdown -h now

Create an AMI from Linux Helper using AWS Console

Sign in to the AWS Management Console.
Navigate to EC2 > Instances.
Select the Helper instance and choose Instance state > Stop instance. When prompted for confirmation, choose Stop. It can take a few minutes for the instance to stop.
To create the AMI, select the Helper instance, select Actions > Image and templates > Create image.
Enter the Image name (Helper-LIN).
Add a new tag with Key=Name and Value=Helper-LIN.
Click Create image.

Create an AMI from Linux Helper using AWS CLI

To create an AMI from Linux Helper in AWS CLI:

aws ec2 stop-instances \
--instance-ids i-0b0edfce604652eff
# Wait until instance stops:
aws ec2 wait instance-stopped --instance-ids i-0b0edfce604652eff
aws ec2 create-image \
--instance-id i-0b0edfce604652eff \
--name "Helper-LIN" \
--tag-specifications "ResourceType=image,Tags=[{Key=Name,Value=Helper-LIN}]"
# Note <amiID>, for example:
"ImageId": "ami-0e6d0cc30a40e4ad4"
# Wait for AMI creation to finish by running the following command:
aws ec2 wait image-available --image-ids ami-0e6d0cc30a40e4ad4

Optimize the AWS Auto Scaling Group with SCL or Windows

To optimize the AWS Auto Scaling group, run ibc_onprem, a python script, as a service on the Coordinator machine. This builds on the existing Auto Scaling group configuration by scaling Helpers based on actual Coordinator demand, enabling faster and more responsive provisioning than standard Auto Scaling behaviour.

Create an AMI from the Incredibuild Helper (SCL or Windows).
Copy the AMI ID, for example, ami-017821fb4a6cdf386.
Create the Auto Scaling group with the following AWS CLI script, updating the AMI_ID, SUBNETS, and SECURITY_GROUPS values based on your existing grid config, as well as any optional values as described below:

#!/bin/bash
set -e
ASG_NAME="Helper_WIN_ASG"
AMI_ID="ami-017821fb4a6cdf386"
SUBNETS="subnet-0b184d86f43ece9ce"
SECURITY_GROUPS="\"sg-06c37393593df9fea\", \"sg-0a65bb9651c5525ba\""
INSTANCE_TYPE_1="c5a.2xlarge"
INSTANCE_CPUS_1=8
INSTANCE_TYPE_2="c6a.2xlarge"
INSTANCE_CPUS_2=8
# % of spot instances (100% of spot instances - choose ON_DEMAND_PERCENT=0)
ON_DEMAND_PERCENT=0
MIN_SIZE=8
MAX_SIZE=40
DESIRED_CAPACITY=8
# NOT TO EDIT
LT_NAME="${ASG_NAME}_Template"
LT_VER_DESC="${ASG_NAME}_Template_v1"
COOLDOWN=30
WARMUP=60
echo "=== Creating Launch Template ==="
LAUNCH_TEMPLATE_ID=$(
aws ec2 create-launch-template \
--launch-template-name "$LT_NAME" \
--version-description "$LT_VER_DESC" \
--launch-template-data "{
\"NetworkInterfaces\": [
{
\"DeviceIndex\": 0,
\"SubnetId\": \"$SUBNETS\",
\"Groups\": [$SECURITY_GROUPS]
}
],
\"ImageId\": \"$AMI_ID\",
\"InstanceType\": \"$INSTANCE_TYPE_1\"
}" \
--query 'LaunchTemplate.LaunchTemplateId' \
--output text
)
echo "Launch Template ID: $LAUNCH_TEMPLATE_ID"
sleep 5
echo "=== Creating AutoScaling Group ==="
aws autoscaling create-auto-scaling-group \
--auto-scaling-group-name "$ASG_NAME" \
--mixed-instances-policy "{
\"LaunchTemplate\": {
\"LaunchTemplateSpecification\": {
\"LaunchTemplateId\": \"$LAUNCH_TEMPLATE_ID\"
},
\"Overrides\": [
{\"InstanceType\": \"$INSTANCE_TYPE_1\", \"WeightedCapacity\": \"$INSTANCE_CPUS_1\"},
{\"InstanceType\": \"$INSTANCE_TYPE_2\", \"WeightedCapacity\": \"$INSTANCE_CPUS_2\"}
]
},
\"InstancesDistribution\": {
\"OnDemandBaseCapacity\": 0,
\"OnDemandPercentageAboveBaseCapacity\": $ON_DEMAND_PERCENT,
\"SpotAllocationStrategy\": \"lowest-price\"
}
}" \
--min-size $MIN_SIZE \
--max-size $MAX_SIZE \
--desired-capacity $DESIRED_CAPACITY \
--default-cooldown $COOLDOWN \
--default-instance-warmup $WARMUP \
--vpc-zone-identifier "$SUBNETS"

Optional parameters:

INSTANCE_TYPE_1 and INSTANCE_TYPE_2: The instance types, and if you specify different instance types, also update the INSTANCE_CPUS_1 and INSTANCE_CPUS_2 values.
ON_DEMAND_PERCENT: Thepercentage for On-Demand instances. The remaining instances will be created as Spot instances. For 100% Spot instances, leave the parameter unchanged as 0.
MIN_SIZE/MAX_SIZE: The minimum and maximum size for the Auto Scaling group.The maximum is a multiple of the minimum and must correspond to the CPUs specified for INSTANCE_CPUS_1 and INSTANCE_CPUS_2. For example, if there are 16 CPU cores in both machines, the minimum is 16 and the maximum is a multiple of 16 (for example, 96).
DESIRED_CAPACITY: The initial capacity when the Auto Scaling group is created without any load (this usually corresponds to the number of CPUs in a single Helper machine).

# EC2 Auto-scaling
autoscaling:SetDesiredCapacity
autoscaling:DescribeAutoScalingGroups

# Use ASG name you used when creating the ASG (in the example it is Helper_WIN_ASG)
# --desired-capacity should be set to the same value you used during ASG creation
aws autoscaling set-desired-capacity --auto-scaling-group-name Helper_WIN_ASG --desired-capacity 8
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names Helper_WIN_ASG --query "AutoScalingGroups[0]"

Verify that the results do not include AccessDenied errors with identity-based policy allows requirements.
Copy ibc_onprem (https://github.com/Incredibuild-RND/ibc_onprem) to the Coordinator machine.
Update the config.ini, editing the [AWS] section, providing your AWS Region name (region_name) and Auto Scaling group name (asg_name), as well as any optional values as described below:

[AWS]
region_name=us-east-1
asg_name=Helper_WIN_ASG
[SCALE]
period_sec=10
evaluation_periods=3
scale_in_protection_sec=120
[APP]
log_file_path=ibc_onprem.log
debug=False

Optional parameters:

period_sec: How frequently (in seconds) the program polls the Coordinator for demand for Helper cores.
evaluation_periods: How many evaluation periods(default: 3) before making a decision to scale-out/scale-in or remain on same number of Helper cores.
scale_in_protection_sec: To prevent “thrashing” (rapid scale in/out cycles) of instances, set this to 120 seconds or more.

To apply the configuration changes, restart the ibc_onprem utility. Install the pip requirements and run ibc_onprem.py as a service, for example, using nssm.exe:

nssm.exe install ibc_onprem <python_path>\python.exe <ibc_onprem_path>\ibc_onprem.py
nssm.exe set ibc_onprem Start SERVICE_DEMAND_START
nssm.exe start ibc_onprem

Delete Auto Scaling Group and Launch Template

To delete an Auto Scaling group and Launch Template in the AWS CLI:

aws autoscaling delete-auto-scaling-group --auto-scaling-group-name Helper_WIN_ASG --force-delete
aws ec2 delete-launch-template --launch-template-name "Helper_WIN_ASG_Template"

How Required Helper Cores are Calculated

Helper core requirements are calculated separately for each Build Group. Only Build Groups with active builds are included in the calculation. For each active Build Group, the service retrieves the incredibuild_needed_helper_cores and incredibuild_available_helper_cores metrics. The total required Helper cores are calculated as the difference between the sum of the required helper cores and the sum of the available helper cores across all active Build Groups. Build Groups allow resources to be allocated independently, while the Universal Helpers Build Group can assist builds from any Build Group. To prevent over-provisioning, the service accounts for available on-premises Universal Helpers. It calculates the number of available Universal Helper cores as the reported available helper cores minus the Helper cores already provisioned in the AWS Auto Scaling Group. This value is then subtracted from the total required Helper cores to determine the additional cloud capacity that must be provisioned.

Using ibc_onprem in Parallel with ASG Dynamic Target Tracking

Using the ibc_onprem in parallel with Auto Scaling Group dynamic target tracking is possible when most of the cores are allocated by the Auto Scaling group, and core usage is not “spiky” over the day.

In this case, add an Auto Scaling Group dynamic target tracking:

aws autoscaling put-scaling-policy --auto-scaling-group-name "$ASG_NAME" \
--policy-name cpu50-target-tracking-scaling-policy \
--policy-type TargetTrackingScaling \
--target-tracking-configuration '{"TargetValue": 50.0, "PredefinedMetricSpecification": {"PredefinedMetricType": "ASGAverageCPUUtilization"}}'

In many cases ibc_onprem and dynamic target tracking may compete to set desired capacity. Example The Auto Scaling Group is configured with a capacity of 40 Helper cores.

Optimize the AWS Auto Scaling Group with Linux

Create an AMI from the Incredibuild Helper (Linux).
Copy the AMI ID, for example, ami-017821fb4a6cdf386.
Create the Auto Scaling group with the following AWS CLI script, updating the AMI_ID, SUBNETS, and SECURITY_GROUPS values based on your existing grid config, as well as any optional values as described below:

#!/bin/bash
set -e
ASG_NAME="Helper_LIN_ASG"
AMI_ID="ami-0b96ef5ea8e2b2252" # LIN AMI
SUBNETS="subnet-0b184d86f43ece9ce"
SECURITY_GROUPS="\"sg-02d2bdc6681c5b1c7\", \"sg-0a65bb9651c5525ba\""
INSTANCE_TYPE_1="c5a.2xlarge"
INSTANCE_CPUS_1=8
INSTANCE_TYPE_2="c6a.2xlarge"
INSTANCE_CPUS_2=8
# % of spot instances (100% of spot instances - choose ON_DEMAND_PERCENT=0)
ON_DEMAND_PERCENT=0
MIN_SIZE=8
MAX_SIZE=40
DESIRED_CAPACITY=8
# NOT TO EDIT
LT_NAME="${ASG_NAME}_Template"
LT_VER_DESC="${ASG_NAME}_Template_v1"
COOLDOWN=30
WARMUP=60
echo "=== Creating Launch Template ==="
LAUNCH_TEMPLATE_ID=$(
aws ec2 create-launch-template \
--launch-template-name "$LT_NAME" \
--version-description "$LT_VER_DESC" \
--launch-template-data "{
\"NetworkInterfaces\": [
{
\"DeviceIndex\": 0,
\"SubnetId\": \"$SUBNETS\",
\"Groups\": [$SECURITY_GROUPS]
}
],
\"ImageId\": \"$AMI_ID\",
\"InstanceType\": \"$INSTANCE_TYPE_1\"
}" \
--query 'LaunchTemplate.LaunchTemplateId' \
--output text
)
echo "Launch Template ID: $LAUNCH_TEMPLATE_ID"
sleep 5
echo "=== Creating AutoScaling Group ==="
aws autoscaling create-auto-scaling-group \
--auto-scaling-group-name "$ASG_NAME" \
--mixed-instances-policy "{
\"LaunchTemplate\": {
\"LaunchTemplateSpecification\": {
\"LaunchTemplateId\": \"$LAUNCH_TEMPLATE_ID\"
},
\"Overrides\": [
{\"InstanceType\": \"$INSTANCE_TYPE_1\", \"WeightedCapacity\": \"$INSTANCE_CPUS_1\"},
{\"InstanceType\": \"$INSTANCE_TYPE_2\", \"WeightedCapacity\": \"$INSTANCE_CPUS_2\"}
]
},
\"InstancesDistribution\": {
\"OnDemandBaseCapacity\": 0,
\"OnDemandPercentageAboveBaseCapacity\": $ON_DEMAND_PERCENT,
\"SpotAllocationStrategy\": \"lowest-price\"
}
}" \
--min-size $MIN_SIZE \
--max-size $MAX_SIZE \
--desired-capacity $DESIRED_CAPACITY \
--default-cooldown $COOLDOWN \
--default-instance-warmup $WARMUP \
--vpc-zone-identifier "$SUBNETS"

Optional parameters:

INSTANCE_TYPE_1 and INSTANCE_TYPE_2: The instance types, and if you specify different instance types, also update the INSTANCE_CPUS_1 and INSTANCE_CPUS_2 values.
ON_DEMAND_PERCENT: Thepercentage for On-Demand instances. The remaining instances will be created as Spot instances. For 100% Spot instances, leave the parameter unchanged as 0.
MIN_SIZE/MAX_SIZE: The minimum and maximum size for the Auto Scaling group.The maximum is a multiple of the minimum and must correspond to the CPUs specified for INSTANCE_CPUS_1 and INSTANCE_CPUS_2. For example, if there are 16 CPU cores in both machines, the minimum is 16 and the maximum is a multiple of 16 (for example, 96).
DESIRED_CAPACITY: The initial capacity when the Auto Scaling group is created without any load (this usually corresponds to the number of CPUs in a single Helper machine).

sudo /opt/incredibuild/management/set_debug_level.sh coordinator allocation
sudo /opt/incredibuild/etc/init.d/incredibuild restart
# EC2 Auto-scaling
autoscaling:SetDesiredCapacity
autoscaling:DescribeAutoScalingGroups

# Use ASG name you used when creating the ASG (in the example it is Helper_LIN_ASG)
# --desired-capacity should be set to the same value you used during ASG creation
aws autoscaling set-desired-capacity --auto-scaling-group-name Helper_LIN_ASG --desired-capacity 8
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names Helper_LIN_ASG --query "AutoScalingGroups[0]"

Verify that the results do not include AccessDenied errors with identity-based policy allows requirements.
Copy ibc_onprem (https://github.com/Incredibuild-RND/ibc_onprem) to the Coordinator machine.
Update the config.ini, editing the [AWS] section, providing your AWS Region name (region_name) and Auto Scaling group name (asg_name), as well as any optional values as described below:

[AWS]
region_name=us-east-1
asg_name=Helper_LIN_ASG
[SCALE]
period_sec=10
evaluation_periods=3
scale_in_protection_sec=120
[APP]
log_file_path=ibc_onprem.log
debug=False

Optional parameters:

period_sec: How frequently (in seconds) the program polls the Coordinator for demand for Helper cores.
evaluation_periods: How many evaluation periods(default: 3) before making a decision to scale-out/scale-in or remain on same number of Helper cores.
scale_in_protection_sec: To prevent “thrashing” (rapid scale in/out cycles) of instances, set this to 120 seconds or more.

To apply the configuration changes, restart the ibc_onprem utility. Install the pip requirements and run ibc_onprem.py as a service, for example, using nssm.exe:

nssm.exe install ibc_onprem <python_path>\python.exe <ibc_onprem_path>\ibc_onprem.py
nssm.exe set ibc_onprem Start SERVICE_DEMAND_START
nssm.exe start ibc_onprem

Delete Auto Scaling Group and Launch Template

To delete an Auto Scaling group and Launch Template in the AWS CLI:

aws autoscaling delete-auto-scaling-group --auto-scaling-group-name Helper_LIN_ASG --force-delete
aws ec2 delete-launch-template --launch-template-name "Helper_LIN_ASG_Template"

Using ibc_onprem in Parallel with ASG Dynamic Target Tracking

In this case, add an Auto Scaling Group dynamic target tracking:

aws autoscaling put-scaling-policy --auto-scaling-group-name "$ASG_NAME" \
--policy-name cpu50-target-tracking-scaling-policy \
--policy-type TargetTrackingScaling \
--target-tracking-configuration '{"TargetValue": 50.0, "PredefinedMetricSpecification": {"PredefinedMetricType": "ASGAverageCPUUtilization"}}'

In many cases ibc_onprem and dynamic target tracking may compete to set desired capacity. Example The Auto Scaling Group is configured with a capacity of 40 Helper cores.

​Prerequisites

​Create an Auto Scaling Group using AWS CLI

​Create an Auto Scaling Group using AWS Console

​Gather Incredibuild Initiator Information

​Create an AMI from Helper Machine

​Create Template from AMI

​Create Auto Scaling Group from Template

​Create an Auto Scaling Group with Mixed Instance Types

​Create an Auto Scaling Group with Mixed Instance Types using AWS Console

​Create an Auto Scaling Group with Mixed Instance Types using AWS CLI

​Create an SCL Helper for the Auto Scaling Group

​Create an SCL Helper

​Create an AMI from SCL Helper using AWS Console

​Create an AMI from SCL Helper using AWS CLI

​Create a Windows Helper for the Auto Scaling Group

​Create a Windows Helper

​Create a Linux Helper for the Auto Scaling Group

​Create a Linux Helper

​Create an AMI from Linux Helper using AWS Console

​Create an AMI from Linux Helper using AWS CLI

​Optimize the AWS Auto Scaling Group with SCL or Windows

​Delete Auto Scaling Group and Launch Template

​How Required Helper Cores are Calculated

​Using ibc_onprem in Parallel with ASG Dynamic Target Tracking

​Optimize the AWS Auto Scaling Group with Linux

​Delete Auto Scaling Group and Launch Template

​Using ibc_onprem in Parallel with ASG Dynamic Target Tracking

Prerequisites

Create an Auto Scaling Group using AWS CLI

Create an Auto Scaling Group using AWS Console

Gather Incredibuild Initiator Information

Create an AMI from Helper Machine

Create Template from AMI

Create Auto Scaling Group from Template

Create an Auto Scaling Group with Mixed Instance Types

Create an Auto Scaling Group with Mixed Instance Types using AWS Console

Create an Auto Scaling Group with Mixed Instance Types using AWS CLI

Create an SCL Helper for the Auto Scaling Group

Create an SCL Helper

Create an AMI from SCL Helper using AWS Console

Create an AMI from SCL Helper using AWS CLI

Create a Windows Helper for the Auto Scaling Group

Create a Windows Helper

Create a Linux Helper for the Auto Scaling Group

Create a Linux Helper

Create an AMI from Linux Helper using AWS Console

Create an AMI from Linux Helper using AWS CLI

Optimize the AWS Auto Scaling Group with SCL or Windows

Delete Auto Scaling Group and Launch Template

How Required Helper Cores are Calculated

Using ibc_onprem in Parallel with ASG Dynamic Target Tracking

Optimize the AWS Auto Scaling Group with Linux

Delete Auto Scaling Group and Launch Template

Using ibc_onprem in Parallel with ASG Dynamic Target Tracking