Process Virtualization Flow

This section describes in detail how the Incredibuild architecture is used for Process Virtualization, when Visual Studio performs compilation of C++ projects.

• This is a simplistic and general explanation. It omits many details that occur during the process.

• In this step by step illustration, Visual Studio is used to perform the necessary compilation of a C++ project. However, the steps are basically identical for any other compiler you decide to use.

1. In Visual Studio builds, Incredibuild executes a VS command (such as DevEnv or MSBuild command), and requests VS to build a solution. When the VS command runs and the VS process is ready to build the solution that Incredibuild passed to it, Incredibuild injects DevEnv or MSBuild into the VS process, and acts as a mediator between the calls that the VS process makes and the OS. It monitors all the calls to the OS and decides how to process them.

   

2. As part of the build command, the build runs compilation processes (cl.exe). When VS runs a cl.exe process (a compilation process), VS runs a Create Process command, which is intercepted by Incredibuild before it reaches the OS. Incredibuild detects cl.exe as a process predefined to run on a remote machine, and does not allow this process to reach the OS; instead, it tells VS that a dummy process has been created.

   

3. Instead of letting the cl.exe process run on the OS, Incredibuild asks the Coordinator component for a Helper Agent (on a single core), on which it can run the compilation process.

   

4. The Coordinator, which has many Helper Agents connected to it, brings an available Helper (core) that is also currently the strongest in the network.

   

5. The Coordinator connects an Helper Agent to the Initiating machine.

   

6. The Initiating machine and the Helper machine communicate directly with each other, without the help of the Coordinator. Note that the Incredibuild Agent is installed on both the Helper machine and the Initiating machine.

   

7. The compiler is transferred from the Initiating machine to the Helper machine.

   

8. The Incredibuild Agent on the Initiating machine sends a request to the Incredibuild Agent on the Helper machine to run cl.exe.

   

9. When the Incredibuild Agent on the Helper machine runs cl.exe, the Incredibuild Virtualization layer is once again injected into the process that monitors all the calls that cl.exe sends to the OS.

   

10. The processing now takes place on the Helper machine. cl.exe sends many calls and requests to the OS. However, each file-system-related request is sent to Incredibuild Virtualization, as it cannot be performed directly by OS, since the files that cl.exe requests are not present on the remote machine. For example, when cl.exe sends an Open File request, the request goes to Incredibuild Virtualization, which knows that the file that needs to be opened is not on the remote machine, but is on the Initiating machine that initiated this process execution.

    

    

11. The Incredibuild Agent on the Helper machine requests from the Incredibuild Agent on the Initiating machine to bring the file.

    

12. The requested file is copied from the file system of the Initiating machine to a cache of Incredibuild in a dedicated remote directory.

13. Incredibuild Virtualization performs redirection to the parameter that contains the path to the file that cl.exe asked to open.

    

    

14. Instead of giving the OS the original path address of the file that cl.exe requested to open, Incredibuild Virtualization gives the OS the path to the file in the cache.The same virtualization techniques are applied to other file-system events such as Registry calls, DLL loading, and the running of executables. The virtualization environment handles StdOut, StdErr, and StdIn for the process in a similar manner.

    

15. Incredibuild Virtualization makes use of many Helpers to significantly accelerate processes and to supply the Initiating machine with hundreds of cores, which it can use to accelerate its processes instead of only using the cores on its local machine.