Saturday, 27 October 2012

Single Root I/O Virtualization (SR-IOV) - Concept


I thought to divide this into two parts. One explaining the theory of SR-IOV feature introduced in vSphere 5.1 while the other will show detailed configuration example.

Introduction
As we saw in Direct Path IO, each PCIe device can be used by one VM ONLY. Also, a max of 6 PCIe devices can be used for DirectPath IO per host. To overcome this limitation, SR-IOV feature was introduced in version 5.1.

SR-IOV is a standard that can present single PCIe device (this is called Physical Function) as multiple independent PCIe devices (each one is called Virtual Function) to OS or hypervisor. This PCIe device can be NIC, HBA, USB, etc.

PFs are full-featured PCIe functions; they are discovered, managed, and manipulated like any other PCIe device. PFs have full configuration resources, meaning that it’s possible to configure or control the PCIe device via the PF, and (of course) the PF has full ability to move data in and out of the device. VFs are similar to PFs but lack configuration resources; basically, they only have the ability to move data in and out. VFs can’t be configured, because that would change the underlying PF and thus all other VFs; configuration can only be done against the PF.
Note: In the rest of the sections, we will be considering the PCIe device as Network Adapter.
Prerequisites
  • vSphere 5.1
  • The physical server must support Input/Output Memory Management Unit (IOMMU) and SR-IOV.
  • The BOIS of the server must have IOMMU and SR-IOV enabled.
  • NIC firmware that supports SR-IOV, with SR-IOV enabled.
  • NIC driver that support SR-IOV.
  • Depending on the guest OS, you may need to update your virtual functions (VF) driver
Restrictions

VMs configured for SR-IOV will lose the following features:
 
  • vMotion
  • Storage vMotion
  • vShield
  • Netflow
  • Virtual Wire
  • High availability
  • Fault tolerance
  • DRS
  • DPM
  • Suspend and resume
  • Snapshots
  • MAC-based VLAN is not supported for passthrough virtual functions
  • Hot adding and removing of virtual devices, memory, and vCPU
  • Should not be used in a cluster environment

Supported NICs
  1. Intel 82576 (Kawela)
  2. Intel 82599 (Niantic)
  1. Intel X540 Family (Twinville)
  1. Emulex OneConnect (BE3)
  1. Solarflare
  1. Broadcom 57712

Note: If you upgrade from vSphere 5.0 or earlier to vSphere 5.1 or later, SR-IOV support is not available until you update affected NIC drivers.

Old PCIe devices won't support SR-IOV due to missing ACS capability (faced this with Quad-Port Intel 82576). VMkernal will check the PCIe switches between the Endpoint and the Root Complex. If any PCIe switch don't support ACS, VMkernal won't enable SR-IOV. VMware introduced this check by VMkernal in ESX/ESXi 4.1. More details about this restriction can be found in KB:1036811
To understand more about interoperability between PCIe and SR-IOV, refer to the link http://www.intel.com/content/dam/doc/application-note/pci-sig-sr-iov-primer-sr-iov-technology-paper.pdf.

Supported Guest OS

  • Red Hat Enterprise Linux 6.1
  • Microsoft Windows Server 2008 R2 Service Pack 1 (SP1)

3 comments:

  1. VMware updated the supported NIC list, see page 9 in the Networking configuration guide.

    http://pubs.vmware.com/vsphere-51/topic/com.vmware.vsphere.networking.doc/GUID-E8E8D7B2-FE67-4B4F-921F-C3D6D7223869.html

    Supported NICs
    The following NICs are supported for virtual machines configured with SR-IOV. All NICs must have drivers and firmware that support SR-IOV.

    Products based on the Intel® 82599ES 10 Gigabit Ethernet Controller Family (Niantic)

    Products based on the Intel® Ethernet Controller X540 Family (Twinville)

    Emulex OneConnect (BE3)

    As a side note, the original Quad-port based on the Intel 82576 used a bridge that did not support ACS but we replaced it with the Intel® Gigabit ET2 Quad Server Adapter that does have a bridge that supports ACS.
    Also, it is important that any network adapter that will be enabled for SR-IOV needs to be in a PCIe slot that is either provided by the Intel Integrated I/O directly off of the Intel Xeon processor E5-2600 or off the Northbridge on older systems. PCIe slots off of a Southbridge typically do not support ACS or ARI so SR-IOV will not be supported.

    ReplyDelete
    Replies
    1. Thanks for sharing the updated list from VMware. I have tested Intel 82576 on vSphere 5.1 and got it working.

      Check this link from intel regarding SR-IOV support.

      http://www.intel.com/support/network/adapter/pro100/sb/CS-031492.htm

      Probably VMware is considering the TAC support part of it instead of functionality.

      Back to ACS support point. This is absolutely correct. In my post "http://vmwarehints.blogspot.com/2012/10/pci-passthrough-direct-io-or-sr-iov.html", I showed how to bypass ACS check. ALTHOUGH I DON'T RECOMMEND THIS. I did it for testing only.

      Delete
  2. Thanks for providing this informative information you may also refer.
    http://www.s4techno.com/vmware-training-pune-online/

    ReplyDelete