Accelerating Custom Build Steps, Events, and Tools

You can also distribute custom build steps, events, and tools, for execution on remote machines, for example, code generation, rendering, simulations, or calculations that are part of the code building.

You can also instruct Incredibuild to execute these in parallel with other tasks that would otherwise be waiting for the end of the execution of the custom steps, events, and tools. For example, Project A is a static library project with a post-link step that deletes some files and writes text to a log file. Project A is defined as a dependency (sub-project) of project B, because project B links with its output. Visual Studio (and thus Incredibuild's) default behavior is to wait for the completion of all project-level steps in a project's sub-projects before starting that project's compilation. As a result, project B would not start to compile until project A's post-link step has completed, slowing down the build. In this case, you can change the default behavior and instruct Incredibuild to execute start compiling dependent projects as soon as possible. This directive can also be used to speed up the execution of a single project, as Visual Studio runs tasks for custom build steps, events, tools, and rules, within a project sequentially, and in many cases this is irrelevant.

You can also further accelerate the build by allowing for the distribution and parallelization of the contents of the build steps at the process level.

Accelerating Build Steps in Visual Studio

A custom build step, event, or tool, can be further broken down into smaller tasks, and parallelized to achieve further acceleration. This allows for the distribution and parallelization at the process level. This feature is only available in scenarios where:

1. The Incredibuild predicted feature (on by default) is not turned off.

2. The Incredibuild UseMSBuild directive is not used.

3. The build is not executed with "build regardless of errors".

Example 1: If within a custom step, a user executes a complete Make command that builds a complete project, you can instruct Incredibuild to highly accelerate your Make command execution by using the Incredibuild Make and Build Tools package. Instead of treating the Make command as a single task, Incredibuild “breaks” this command into sub-tasks and executes the compilation in a parallel and distributed manner.

Example 2: If you have a custom step, with a tool that runs many rendering commands, you can use a profile XML file to distribute the rendering, and to intercept the tool that spawns the rendering processes. 

Distributing Tasks

By default, custom steps, exec tasks, and special build tools are not distributed to remote agents. In order to allow Incredibuild to distribute and manage these tasks, use a profile XML file.

• In the Incredibuild BuildConsole command, use the /profile directive to specify a profile for Incredibuild to use: /profile=[profile_file_name].xml

  -or-

• If you are building from within Visual Studio, or would like your solution to have a default profile to use (per solution), create a [solution_folder]\[solution_name].ib_profile.xml profile XML file. This profile is automatically loaded by Incredibuild if no profile file is specified with the /profile=[profile_file_name].xml switch.

  The following XML file instructs Incredibuild to distribute every gawk task executed by the build (there is no need to specify the file extension for gawk.exe):

  Copy

  <?xml version="1.0" encoding="UTF-8" standalone="no" ?>
   <Profile FormatVersion="1">
    <Tools>
     <Tool Filename="gawk" AllowRemote="true" />
    </Tools>
   </Profile>

Batching Tasks

Use the MSBuild engine, which executes all custom build tools and rules within the context of a single project in a sequential  manner, on a single CPU core. For example, a custom build tool running moc.exe (as in the GUI SDK QT) is executed in a sequential manner in project A for each file that needs to be generated.

Specifying a tool name, for example, moc.exe, in your Incredibuild profile XML with the AllowPredictedBatch option, instructs Incredibuild to execute this tool in parallel, resulting in a significant performance boost. As the tool now runs in a parallel manner, it can also be configured to be distributed to remote machines. Parallelization is applied until a different executable is invoked (for example, CL or LINK).

The tools that can be safely accelerated with this feature must have the following characteristics:

1. Two instances of the executable do not attempt to write to the same file (e.g. output file, log file).

2. None of the instances of the executable should be dependent on anything that the other instances do. For example, one instance does not depend on the output file of another instance, and none of the instances deletes any files or folders thgat the other instances use).

>To batch tasks in Visual Studio:

• Add the AllowPredictedBatch="true" tool tag to your solution’s .ib_profile.xml.

Usage Example

If your solution name file is my_solution.sln, and you would like to parallelize and distribute moc.exe execution:

• Add the file my_solution.ib_profile.xml to the folder where my_solution.sln resides. This applies the profile to any build of this solution run from within Visual Studio.

• Alternatively, if building from the command line, use the /profile switch to specify the profile file: Buildconsole my_solution.sln /rebuild /cfg=”debug|win32” /profile=”my_solution.ib_profile.xml”

Example contents of my_solution.ib_profile.xml:

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<Profile FormatVersion="1">
  <Tools>
    <Tool Filename="moc" AllowRemote="true" AllowPredictedBatch="true"/>
  </Tools>
</Profile>