Now it’s the time to
go into CPU Virtualization. I have to mention here that ESXi 5.x was used as
the testing environment for the concepts mentioned in this and the next posts.
Since ESXi is NUMA
aware OS, the way how CPU virtualization works differ based on the underlying
system (NUMA or non-NUMA).
For non-NUMA SMP
In this type, ESXi
CPU Scheduler will split the load from all VMs across all cores or HT logical
processors in round robin manner (which similar to other Operating Systems such as Windows in the way of distributing threads across core as we mentioned in the previous posts).
Keeping in mind that each core or HT logical processing is serving one
VM at a time. What does this mean?
Assuming that we
have an SMP system with 4 cores running on top of it 2 VMs each having 1 vCPU,
in this case ESXi CPU scheduler will split the load of both VMs across the 4
cores in round robin manner (Thread01 (VM1 + VM2) on cores 0 & 1, Thread02
(VM1 + VM2) on cores 2&3, Thread03 (VM1 + VM2) on cores 1&3, etc).
Below is a
test done by loading the vCPU of MS-AD VM. From ESXTOP we can see
that the %PCPU is hunting in a round robin manner.
In case you have
8 VMs each with 1 vCPU running on top of
this system, the scheduler will split the load of those VMs across the 4 cores.
Since each core will serve one VM at a time, you will find %RDY counter increased on all VMs. %RDY counter describes the amount of time a VM
is waiting for a free core to handle it based on CPU scheduler. During this
waiting time, the VM will be freeze.
Let's consider another example of vSMP VM over
non-NUMA SMP System
Assume that we have
2 VMs with the 1vCPU and 2vCPU configurations running on top 4 cores SMP
system. We said previously that the load from all VMs will be spread across all
cores. Does this mean that VM1 will use core-01
while VM2 will use core-02?
In fact No! ESXi CPU
scheduler will split vSMP VM across multiple cores based on the number of
vCPUs. In our example VM01 will use core-01 while VM02 will use cores 02 &
03. Then round robin hunting will start (VM01 to core-02 + VM02 to cores-03 &
04, etc). This is to provide more cycles to the VM as expected by adding extra
vCPUs. On the other hand,
this has a major drawback that you need to have two free cores (or HT logical processors) simultaneously to
execute VM02.Till then the VM will be freeze, i.e. higher %RDY count. Therefore, adding extra vCPUs isn't
always an advantage and needs to be addressed properly.
Note: Also, keep in mind that extra vCPUs needs to
be consider from DRS and HA point of view to make sure that other hosts in the
cluster are having the same CPU capacity.
Below is a
test done by loading a VM named VM-01 which has
2vCPUs. We can see that two %PCPU are loaded at
a time with round robin hunting. Note that the total amount of %USED is almost 200% which the sum
of usage of 2vCPUs (you may refer to ESXTOP Bible to understand %USED more and
see how its exactly calculated).
Last in this
section, I thought to add this example. Two VMs named MS-AD (1vCPU) and VM-01
(4vCPU) are loaded at the same time on top of 4 cores system. Let's look at
ESXTOP output.
If you think that
VM-01 (%USED) is 400% and MS-AD (%USED) is 100%, then you are wrong!!
The total number of
vCPUs is 5 while the total number of physical cores is 4. This means that CPU
scheduler will give one core to MS-AD (during this time VM-01 freeze. Look at
its %RDY). Next CPU scheduler will give 4 cores to VM-01 (during this time MS-AD
freeze. Look at its %RDY). We just said up the drawback of extra vCPUs is that
you need to have the same number of physical cores free simultaneously to run
the VM.
Note: From the guest OS, you will
find that CPU utilization is 100% for both VMs since the guest is utilizating
the max given CPU cycles.
