VirtualBox performance issues on Multi-core CPU Tweaking ...

VirtualBox performance issues solution for Multi-core CPUs

This article explains in detail how you can vastly improve performance of VirtualBox on latest generation Multi Core processors, by tuning VirtualBox and system settings to ensure correct operation and overcoming compatibility issues.

VirtualBox sluggish performance on hyper-threading multi-core Intel Haswell, IvyBridge, SandyBridge

Low performance issue

Running a development environment inside a VirtualBox Debian, for serving Magento with NGiNX revealed a tremenduous slow performance when VirtualBox was configured to use more than one core, as any enthusiast with a multi-core recent generation CPU would intend to set it.

Test bench machine configuration

The test machine was equipped with an i7-4500U procesoor, which has 2 physical cores, each with 2 threads (pseudo cores), totalling 4 execution threads.

Host OS – Windows 7

Guest OS – Debian 7/NGiNX/Magento/+

Test bench results

Requesting a simple phpinfo() page responds and loads almost instantly, but requesting a Magento backend page was taking at least 5 seconds and sometimes going until max execution time of 30 seconds was exceeding. Performance and consistency was so bad that it could not be used.

Down to the “core” of the VirtualBox Performance Issues

As the subtitle says, the core of the problem are really the cores allocation of the physical CPU to VirtualBox. The pseudocores / threads enabled by Intel HyperThreading are not advertised as not being real CPU cores and this is confusing VirtualBox when allocating CPU cores.

The Intel Core i7-4500U for example, has 2 physical cores and 4 threads total, meaning each physical core can run 2 threads with the aid of HyperThreading technology. The problem is that a thread is like an extra hand of the core, but is not having the same physical resource as a real core.

More worse, VirtualBox leaves the first two cores for the host OS, which in Windows 7/8 are the real CPU cores, and allocates the next cores to the virtual machine, which, guess what, are the virtual threads.

Virtual Threads offered by HyperThreading are very light on implementation, not duplicating real CPU resources (Cache, ALU, etc.), so the performance degradation cause is of no wonder.

Test this yourself scenarioRun VirtualBox under Windows 7/8, on a “2 core / 4 threads” processor, and allocate 2 cores to the virtual machine in Settings/System/Processor.Start the Guest OS virtual machine with some continuosly running task, prefferably something demanding to see the impact of the better settings laterGo to Windows Task Manager/Processes in host OS, look for the VirtualBox.exe process that seems to be the virtual machine (you’ll find two of them, one for the Manager and one for the Machine), Right-click and select Set Affinity..Disable cores 3rd and 4th and you’ll see that the guest machine locks upDisable cores one by one and check how functionality/performance is impactedNOW Shutdown the guest machine and go to Settings/System/Processor, and allocate only one core to the virtual machineCheck for vastly improved performanceRepeat steps 3-6 and check performance

VirtualBox CPU Core Performance Tuning

Solution and Tips for Improving performance of VirtualBox

Main Fix

Disable Hyper-Threading in machine BIOS if allowedHint 1: Some laptops, for example ASUS brand, have custom BIOS versions that limit user access to this setting (usually found in the Advanced menu) – HyperThreading can only be disabled by BIOS as it’s a hardware feature)VirtualBox – go to Settings/System/Processor and set equal number of cores for host and guest, as now you’re seeing only the real cores

Workaround if HyperThreading setting is not accessible

VirtualBox – go to Settings/System/Processor and allocate only 1 core to Guest machine. Hopefully, VirtualBox will choose the next core that Windows host OS offers, which will be a real core.

Workaround 2

Check/Set Processor Affinity each time for the VirtualBox.exe process of the virtual machine, to ensure it is running on a physical core

Tips for maximising performance of the whole system

Other VirtualBox fine-tuning settings for the machine, which worked for me:

System/Motherboard/Chipset: ICH9System/Processor/Execution Cap: 100%System/Acceleration: Enable VT-x/AMD-v, Enable Nested PagingStorage/Controller:SATA/Solid State Drive checked

Don’t forget to also max out the performance of your hardware machine, if it is a portable computer:

Select High Performance profile in Power OptionsCustomise High Performance profile in Power Options, setting Minimum processor state to 100%Test bench results after addressing mentioned VirtualBox performance issues

Performance is consistent and predictable, with page loads of uncached Magento backend around 4-5 seconds.

More exact explanations

vCPUs are tied directly to physical cores, but no specifically to 1 core. If you create a VM with 1 vCPU, then you are allocating 1 physical core to that VM whenever it needs to process something. In sticking with our example, this means that you can in theory create 8 single vCPU VMs and never have any resource contention between the VMs. Once you start treking into multi-vCPU VMs, this is where you start running into problems. VMs with multiple vCPUs require that all allocated cores be free before processing can begin. This means, if you have a 2 vCPU machine, 2 physical cores must be available, and a 4 vCPU requires 4 physical cores, etc.

adding more cores would have the potential to make a VM slower. The reason being, virtualbox’s CPU scheduler has to schedule access to a core for the VM and has to allocate all cores assigned to the vm at the same time.

So for example lets use the 4 cores (not going into the physical vs. logical core issue). Let’s say you have 4 cores. First off you’re running Ubuntu and Virtualbox which is of course using various processes. If between the linux OS and virtualbox say 2 cores are always in use with a third used occasionally. Now you have a VM (winxp win7 doesn’t matter) and it has 2 cores assigned. The VM must wait until 2 cores are available before it can process anything and I do mean ANYTHING. Launching a program, responding to keyboard/mouse input, anything. So in this setup anytime your actual system (Ubuntu and VirtualBox) is using over 2 cores your VM must wait, and wait, and wait. This waiting time is called “CPU Ready” time. Lots of info online about CPU Ready times.

… the best practice is to make the VM as lean as possible. Start at 1 cpu/core and increase only if necessary. If your VM boot time is faster with 1 core, then your system doesn’t have the available resources to schedule more in a timely manner, since all of the assigned cores must be available at the same time. Sometimes you can get away with over-provisioning other resources to a vm, such as RAM, and not suffer such a performance hit, but not vCPUs.

Sources:

CPU Ready is:

The amount of time a virtual machine waits in the queue in a ready-to-run state before it can be scheduled on a CPU.

This means that a VM is ready to process something, however, it has to wait because the CPU resources it requires are not available on the physical host.

Source: http://virtualblocks.wordpress.com/2010/06/22/cpu-ready-over-built-vm-or-over-utilized-host/

Further testing

To determine if issues are caused by cpu ready times, I’ve made the following new test, on a new machine:

Test System

Intel i7-4710MQ (4 cores, 8 total threads)Windows 7 host systemVirtualBOX 4.3.10Debian7 with NGINX

4 CPUs allocated

Test results show loading time of a Magento page:

First refreshSecond refreshThird refresh, while repeatedly opening and closing start menu on host (Windows) OS

It can be easily seen, that keeping the host OS doing an extra task, just as simple as opening and closing the start menu many times, while the guest OS was serving the page, caused an extra wait time of more than 1 second. This confirm that the issue is related to CPU Ready times.

1 CPU allocated

Single core allocated to the VM shows vastly improved page load performance, stable between refreshes, not impacted by light host cpu load (like opening start menu).

There was also no performance impact between I/O APIC and Chipset settings.

I/O APIC and Chipset settings of VirtualBox

I/O APIC

From VirtualBox documentation:

Enable I/O APIC

Advanced Programmable Interrupt Controllers (APICs) are a newer x86 hardware feature that have replaced old-style Programmable Interrupt Controllers (PICs) in recent years. With an I/O APIC, operating systems can use more than 16 interrupt requests (IRQs) and therefore avoid IRQ sharing for improved reliability.

Note: Enabling the I/O APIC is required for 64-bit guest operating systems, especially Windows Vista; it is also required if you want to use more than one virtual CPU in a virtual machine.

However, software support for I/O APICs has been unreliable with some operating systems other than Windows. Also, the use of an I/O APIC slightly increases the overhead of virtualization and therefore slows down the guest OS a little.

Warning: All Windows operating systems starting with Windows 2000 install different kernels depending on whether an I/O APIC is available. As with ACPI, the I/O APIC therefore must not be turned off after installation of a Windows guest OS. Turning it on after installation will have no effect however.

In addition, you can turn off the Advanced Configuration and Power Interface (ACPI) which VirtualBox presents to the guest operating system by default. ACPI is the current industry standard to allow operating systems to recognize hardware, configure motherboards and other devices and manage power. As all modern PCs contain this feature and Windows and Linux have been supporting it for years, it is also enabled by default in VirtualBox. It can be turned off on the command line; e see the section called “VBoxManage modifyvm”.