Choosing the Ideal Interface 

Guidelines for Choosing the Ideal Interface

The Incredibuild interfaces are the means to tell Incredibuild how to integrate with your software or script file. Incredibuild allows you to utilize all idle cores existing in your network on machines that have Incredibuild Agents installed on them to execute multiple tasks in parallel, and as a result accelerates your software's performance.

Incredibuild interfaces are intended for the following:

• (Mandatory) Integration with custom software, i.e. for Incredibuild for DevTools solution. In this case, you have to use one or more of the interfaces offered by Incredibuild.

• (Optional) Further extend and accelerate the Visual Studio and Make and Build Tools solutions. (See the sections on Accelerating Custom Build Steps, Custom Build Events and Custom Build Tools in Visual Studio.Example: If as part of a build, you have an execution in Make and Build Tools that does rendering for a large number of files, you can also accelerate the rendering process that is part of that build.

Before you decide which interface is ideal for your usage, you need to be aware of the following aspects of the task executions that you want Incredibuild to accelerate:

• Is it serial, parallel, or a bit of both?

• Does it have the ability to use multiple local cores?

• Does it have the ability to execute multiple processes?

• Does it contain a command line interface?

Once you are familiar with the three interface options and understand how they apply to your application, you can make changes only to the scripts and tools that you wrote, so that the tasks run in parallel.

For a third-party tool, you can use Incredibuild to try forcing many of its tasks to run in parallel on all the cores existing in the network, instead of running only on local cores, thus significantly improving performance. This is also relevant if the third-party tool has a command line enabling you to run from it many commands in parallel. 

For third party tools, instead of only running on cores on your local machine, you can use Incredibuild to force as many tasks as possible to run in parallel on all existing cores on your network.

Note: Incredibuild does not have the ability to distribute threads. The smallest unit that can be distributed by Incredibuild is a Task (the OS-level process). Usually, multithreaded applications can be easily converted to have multiprocess functionality, enabling them to benefit from Incredibuild's distribution technology. This is relevant only for your own applications, and not for third-party applications.

The following sections describing the interfaces in detail will guide you in making such decisions and implementing them in an optimal manner.

Automatic Interception Interface

Today, more and more software applications support a mode of operation which allows execution of several processes in parallel, taking advantage of multiple cores/processors on a local machine. Using Automatic Interception Interfaces, execution of these parallel processes can be accelerated without requiring any changes to the source code. With Automatic Interception Interfaces, Incredibuild receives a list of process types (names) and a description on how to execute each type of process.

Example: An application, “MyApp.exe”, knows to execute in parallel processes by the name of “DoSomething.exe”, according to the number of cores on your computer. This means that if there are four cores on your computer,  Myapp runs four instances of DoSomething.exe for every execution. You can greatly accelerate the execution of this application in Incredibuild, by telling MyApp that you have a larger number of cores on your computer, e.g. 200, thus causing 200 tasks of DoSomething.exe to be sent to the Incredibuild queue. In this case, instead of using only local cores, Interception Interface does automatic distribution of all the 196 remaining DoSomething.exe tasks across cores on the network.

The following steps should be performed for Incredibuild to highly accelerate the total execution time:

• Write a small XML file that describes which process types names should run remotely, and which process types are invokers that are responsible for executing sub-processes. (In this example, MyApp.exe is the invoking/spawning process, and DoSomething.exe is the process Incredibuild should run on remote cores.)

• Indicate to MyApp.exe to run in parallel a large number of DoSomething.exe tasks, and not only an amount equivalent to the number of local cores. This number should be at least as large as the maximum number of tasks that can be run in parallel on the cores on your network.It is also possible to specify a number larger (even much larger) than the number of parallel tasks that can run, for example, 1000 tasks, even if there are not that many cores in the network. In this case, Incredibuild knows how to take all the 1000 requests and manage the queue itself according to the number of available cores.

• Execute MyApp.exe using the Incredibuild IBConsole command, which should have been defined with a large number of allowed parallel processes in its command line. This IBConsole command activates the entire job. The greater the number of tasks MyApp.exe executes in parallel, the greater the acceleration results of Incredibuild.

Incredibuild references the XML file to see how to execute each Task in the Job. When MyApp (that is defined as the invoker) runs in Incredibuild, Incredibuild listens to all the calls that MyApp makes to the operating system. Every time it sees that MyApp invokes a process, it checks whether the process name appears in the XML file (created in step 1 above) as a process that should run remotely or as an invoker process that is responsible for process spawning. If the process is specified to run remotely, Incredibuild runs it on a remote computer instead of on a local computer. Note that if local cores are available, Incredibuild can run Tasks that have been defined in the XML file to run remotely, on the local machine.

The AllowIntercept function marks the invoker process that spawns sub-processes. For example, if a single process “a” executes a single process “b”, which executes process “c”,  both “a” and “b” are declared as “AllowIntercept”, while “c” is declared as “AllowRemote”.

Advantages and Disadvantages:

The benefit of this interface are:

• If the application that you want to accelerate (whether it is an application that you wrote or a third-party application) can already execute many processes in parallel, no changes to  the application source code is needed in order to accelerate the application using Incredibuild.

• The application executes as many processes in parallel as possible and lets Incredibuild manage the execution queue, while distributing the processes according to the available cores in the network and the local machine.

The drawback is if a third-party application does not support multiprocessing, the processes run sequentially, and Incredibuild does not know how to run these processes in parallel. If it is not a third-party application and its source code can be modified, you can change the source code so that it knows to run processes in parallel.

Note: If a third-party application only knows to run tasks sequentially and has source code that cannot be modified, you can check if it has a command line. If so, instead of running at once all the application's tasks sequentially, you can write many command lines that will run in parallel. In this case, it is better to work with the Submission Interface.

Use this Interface if:

1. You are running a third-party tool or application that is executing multiple processes in parallel.

2. You are running your own custom tool, which supports parallel process execution (or can be modified to support it), and you would like to accelerate the process through distribution over the network.

Submission Interface

The Submission Interface allows direct submission of Tasks for distributed execution by Incredibuild. This Interface allows easy distribution from batch files, scripts, and custom tools; it can also be used at a higher level, i.e., to execute every command line or process in parallel on remote machines.With this interface, calls to original command lines are replaced with calls to xgSubmit.exe while accepting the original command line as input. Each call to xgSubmit.exe queues the specified command line for distributed and parallel execution. xgWait.exe can be called from the script to pause execution until a specified group of tasks has completed (allowing dependencies between groups of parallel tasks). Finally, the script/tool itself is called using IBConsole.exe. As with all Incredibuild interfaces, all remotely performed tasks fully emulate the initiating machine's environment and file system, eliminating the need to copy files or install applications on remote machines.

Advantages and Disadvantages

This approach is particularly suitable for running tasks from within scripts, such as batch files. Use of a Submission Interface does not entail significant changes to existing code and is most suitable when you want full control of executing the same type of process, sometimes locally and sometimes remotely, depending on certain scenarios. However, minor changes do need to take place and if an application or script has thousands of tasks, their execution can be tedious. If you are not concerned with which type of processes will be executed locally or remotely by Incredibuild, the “Automatic Interception” interface would be the best fit for your needs.

Advantages include:

• Simple integration with any script or tool.

• Process execution can be executed locally or remotely, as determined by the user.

• Command files allow fast and efficient submission of large sets of commands.

Use this interface if you want to:

1. Speed up build runs by Make tools, which do not support local parallel task execution (such as Microsoft NMAKE).

2. Distribute a process running through a batch file or script.

3. Distribute a process running through a custom application, and prefer the direct Submission interface over creating an XML file describing the tasks to be distributed or over customizing your tool to support local parallel task execution.

XML Interface

The XML Interface is a general-purpose distribution interface that uses an input XML file describing a set of command-line tasks to be executed through the Incredibuild engine, along with their input files, output files, and dependencies.

This interface is used for complex processes defined in the XML file. With it you can construct the master process of the software to be accelerated. The XML file acts as script that manages the execution of the processes, instructing Incredibuild on how to run the processes.

IBConsole initiates the process, using the XML file as input on how to distribute the tasks. As with all Incredibuild interfaces, all remotely performed tasks fully emulate the initiating machine's environment and file system, eliminating the need to copy files or install applications on remote machines.

The XML syntax for each Incredibuild Project describes the Project’s Tasks and the dependencies between them.

As in the other interfaces, the execution of scripts is managed via the IBConsole command. example:

xgConsole myjob.xml  

The beauty of this interface is that Incredibuild manages the job execution, while giving you complete control of the execution flow (tasks and dependencies).

Benefits:

• Completely generic and autonomous

• Supports complex task and file dependencies

• Supports full rebuild, incremental build, and clean operations

• Supports file searches and response file loading

• Supports Environment Variables read-write access

Use this Interface if:

1. You wish to replace your current execution script/build tool with Incredibuild's XML-based build system in order to distribute the process.

2. You would like to distribute the execution of a sequentially running process run from a custom application, and are able to describe the set of commands to be distributed in XML format.

3. You would like to distribute execution of an identical command line run on a group of files described through a wildcard expression (e.g. run command line "analyze.exe /out:./TargetDir " on all files matching the expression "MyFolder\*.cpp").

Note: When using Incredibuild for DevTools, you are required to use one of the three integration interfaces or an interface combination. Before doing so, you must use the IBConsole command to execute the Incredibuild engine itself and activate any interface that you choose. For details regarding usage, see the IB Console Command Reference chapter