MacTech Labs: Virtualization Benchmarks, January 2013

Launch and CPU Tests

There are three situations in which users commonly launch a virtual machine:

Launch the virtual machine from "off" mode, including a full Windows boot and ending with launching of an application. For testing purposes, we chose the NotePad application. There are two types of these: from the Finder and from within the virtualization software. The reason for this is that both products have revised how they launch to meet Apple’s standards.

Launch the virtual machine from a suspended state, and resume from suspend (Adam).

Launch the virtual machine from a suspended state, and resume from suspend (Successive).

For the first test, we started at the Finder and launched the virtualization applications, which were set up to immediately launch the virtual machine. The visual feedback is fairly different between Parallels Desktop and VMware Fusion when Windows first starts up. Windows actually continues “starting up” for quite some time after you see the desktop. In some cases, it can take quite some time for Windows to complete its boot process. Most users don’t care if things continue so long as they aren’t held up.

As a result, we focused on timing to the point of actually accomplishing something. In this case, we configured NotePad to automatically launch. The test ended when the window started to render. This gave us a real world scenario of being able to actually do something as opposed to Windows just looking like it was booted.

The primary difference between the two types of VM launch test is that the computer is fully rebooted (both the virtual machine as well as Mac OS X) in between the "Adam" tests. The successive tests are launching the virtual machines and restoring them without restarting the Mac in between.

Successive tests benefit from both Mac OS X and possibly virtual machine caching and are significantly faster. However, you may only see these types of situations if you are constantly switching in and out of your virtual machine.

As with all of our tests, we performed these tests multiple times to handle the variability that can occur. Of those results, we took the best results for each product.

Figure 4: Virtual Machine Performance: Boot, Shut Down, and Compress

Clearly, virtual machines with more vm memory take longer to restore, so “more” is not necessarily better here. Use the smallest amount that does what you need to do well. In our case, we focused on 1 GB virtual machines in Windows 7 and 8 for these tests.

This year, we saw substantive difference in compression that is one of the best ways to compare CPU processing capabilities in different environments.

Figure 5: Virtual Machine Performance: Suspend and Resume

Most benchmarking suites measure CPU performance using file compression as at least one part of their analysis. We did the same. As a matter of interest, we used compression instead of decompression, because with today’s fast computers, decompression is actually much closer to a file copy than it is to CPU work. Compression requires the system to perform analysis to do the compression, and is therefore a better measurement of CPU.