Single Root I/O Virtualization (SR-IOV) - Concept

I thought to divide this into two parts. One explaining the theory of SR-IOV feature introduced in vSphere 5.1 while the other will show detailed configuration example.

Introduction

As we saw in Direct Path IO, each PCIe device can be used by one VM ONLY. Also, a max of 6 PCIe devices can be used for DirectPath IO per host. To overcome this limitation, SR-IOV feature was introduced in version 5.1.

SR-IOV is a standard that can present single PCIe device (this is called Physical Function) as multiple independent PCIe devices (each one is called Virtual Function) to OS or hypervisor. This PCIe device can be NIC, HBA, USB, etc.

PFs are full-featured PCIe functions; they are discovered, managed, and manipulated like any other PCIe device. PFs have full configuration resources, meaning that it’s possible to configure or control the PCIe device via the PF, and (of course) the PF has full ability to move data in and out of the device. VFs are similar to PFs but lack configuration resources; basically, they only have the ability to move data in and out. VFs can’t be configured, because that would change the underlying PF and thus all other VFs; configuration can only be done against the PF.

Note: In the rest of the sections, we will be considering the PCIe device as Network Adapter.
Prerequisites

• vSphere 5.1

• The physical server must support Input/Output Memory Management Unit (IOMMU) and SR-IOV.
• The BOIS of the server must have IOMMU and SR-IOV enabled.
• NIC firmware that supports SR-IOV, with SR-IOV enabled.
• NIC driver that support SR-IOV.
• Depending on the guest OS, you may need to update your virtual functions (VF) driver

Restrictions

VMs configured for SR-IOV will lose the following features:

 

• vMotion
• Storage vMotion
• vShield

• Netflow

• Virtual Wire

• High availability

• Fault tolerance
• DRS
• DPM
• Suspend and resume

• Snapshots

• MAC-based VLAN is not supported for passthrough virtual functions
• Hot adding and removing of virtual devices, memory, and vCPU

• Should not be used in a cluster environment

Supported NICs

1. Intel 82576 (Kawela)
2. Intel 82599 (Niantic)

3. Intel X540 Family (Twinville)

4. Emulex OneConnect (BE3)

5. Solarflare

6. Broadcom 57712

Note: If you upgrade from vSphere 5.0 or earlier to vSphere 5.1 or later, SR-IOV support is not available until you update affected NIC drivers.

Old PCIe devices won't support SR-IOV due to missing ACS capability (faced this with Quad-Port Intel 82576). VMkernal will check the PCIe switches between the Endpoint and the Root Complex. If any PCIe switch don't support ACS, VMkernal won't enable SR-IOV. VMware introduced this check by VMkernal in ESX/ESXi 4.1. More details about this restriction can be found in KB:1036811

To understand more about interoperability between PCIe and SR-IOV, refer to the link http://www.intel.com/content/dam/doc/application-note/pci-sig-sr-iov-primer-sr-iov-technology-paper.pdf.

Supported Guest OS

• Red Hat Enterprise Linux 6.1
• Microsoft Windows Server 2008 R2 Service Pack 1 (SP1)