VM Tuning

As you set up a dedicated CI system for your iOS builds, you will be faced with a few high level design considerations that are unique to macOS VMs. Chief among them will be your means of integrating macOS build nodes into your CI pipeline, and infrastructure considerations such as the number of cores per VM, the number of VMs per host, and machine type.

Current best practices dictate that a virtualization layer is the preferred means of creating scalable build agents in a CI system. Anka and VMware are the two leading providers of virtualization layers for teams that depend on MacStadium’s infrastructure, but meaningful comparisons of the two have generally been made by DevOps teams in-house, so the data has been scattered at best.

Recently though, one of MacStadium’s customers set their DevOps team to the task of finding the optimum iOS build solution for their workload with a rigorous set of tests, and they were generous enough to share their findings.

Interestingly, the findings shared below show little variance in build times between the two virtualization technologies, so that choice will likely be guided by other factors. You can learn more about the trade offs between virtualization technologies here. Conversely, specific CI design choices, such as the number of cores in a VM and the number of VMs on a given host, were shown to have a significant impact on build times.

Overview of Test

The customer performed a number of tests to determine the speed of a CI pull request (summed up as “build time” in the graphs below). Over the course of the tests, their DevOps team explored several key design variations in their pursuit of a best-fit configuration from a price/performance perspective.

Specifically, they looked at performance as determined by number of cores in a given VM, whether or not hyperthreading is enabled, and finally MacStadium hardware considerations for VM nodes. Their average build time was ~200 seconds for 250MB iOS App. Build times varied throughout the tests from 130s to 350s depending on the configurations.

Build Times vs. Number of Cores per VM

Among the most influential factors in determining CI build times is the number of cores allotted for a given VM. Perhaps not so surprisingly, more cores translate into faster build times. However, the decrease in time per job does not maintain a linear relationship with the number of cores being used, which illustrates the likelihood of something of a “sweet spot” for a given development team -- depending on their unique wants and needs.

Specifically, this test measured build speed in seconds. The machine type -- MacPros -- remained constant, while configurations of between two to eight cores per VM for both virtualization layer providers -- Anka and VMware -- were explored.

Build Time vs. Number of VM Instances on a Given Host

From a design standpoint, the next most impactful choice a DevOps team will be faced with is the number of VM instances to spin up on a given MacStadium host machine. The incentive here is cost-savings, as the cost incurred by a team is determined by the number of hosts they require, not the number of VMs.

The following illustrates the increase in build times (as measured in seconds) that accompanied incremental steps from one to eight VM instances running on a given host.

Notice that both hyperthreaded Anka and standard VMware systems were tested. Again, the performance similarities between Anka and VMware are apparent. That said, this test illustrates inherent performance breakpoints at which the time per build increases notably should additional VMs be added to a given host. It is also worth noting that Mac Pros only have 12 physical cores and 24 vCPUs when hyper threaded. It is not recommended to allocate more cores than exist on the physical machine. This explains the dramatic build time increases as the number of concurrent instances climbs above 4.

MacStadium Hardware Performance Considerations

MacStadium offers both Mac Minis and Mac Pros as host machines. If your team decides that an Anka-driven virtualization layer will best serve your needs, you will then need to settle on your preferred machine type, as Anka supports both. Conversely, VMware is only compatible with Mac Pros, so if your team settles on VMware to provide your virtualization layer, your hardware decision will default to Mac Pros.

The graph below illustrates the performance impact of running between one and five VM instances on an Anka-managed Mac Mini. Notice that both hyperthreaded and non-hyperthreaded instances were tested.

Summary of Findings

Variations in build times as determined by virtualization layer provider were marginal at best. However, this choice of provider will influence design considerations because they affect the choice of Mac host, so teams will do well to consider their needs in this arena early.

Furthermore, while there is a decrease in build times as additional processing cores are engaged by a given VM, this decrease isn’t linear. Past a certain point, there are diminishing returns in terms of processing speed. This means that teams will likely find the greatest value by striking a balance between the number of cores per VM and their total build times.

Finally, there will again be a balance struck between the cost savings of running multiple VMs in parallel on a single host and the decrease in build speeds that will accompany them. Since every build will respond slightly differently to the underlying infrastructure, a team’s choice should be based on running their own experiments. Further, each team will balance price and performance differently. Hopefully the experiments above give good context on what to expect as you test to find your ideal configuration.