Single Root I/O Virtualization (SR-IOV) - Configuration

Here I am detailing the configuration steps and the caveats faced during implementation. 

1. Upgrade your Server BIOS and NIC Firmware to a version supporting SR-IOV. For Dell PowerEdge R710, minimum BIOS is 6.2.3 and 82576 Intel NIC Firmware is 13.5. Refer to my previous blog "Upgrade Dell PowerEdge Components" to understand how to upgrade server components. 

2. From the BIOS, enable Intel VT-d and SR-IOV Global.

 

3. Identify the driver for the NIC that will be used for SR-IOV (in VMware terms its called kernel module). This can be identified from vSphere Client or using CLI (I prefer this method since it provides more details).

From vSphere Client,

a. Navigate to Home > Inventory > Hosts and Clusters > #Select the host# > Configuration Tab > Networking.

b. In case of standard switch, select Properties > Network Adapters Tab. In case of distributed switch, select Manage Physical Adapters.

c. Select the adapter to be used for SR-IOV. On the right pane you will get all the details for the NIC.

From CLI,

a. SSH to the server

b. List all the NICs available on the server

c. Identify the NIC of interest and get more details about it. Remember this NIC should be one of the listed models which support SR-IOV. Refer to the previous blog to see the supported NICs.

~ # ethtool -i vmnic5

driver: igb

version: 2.1.11.1

firmware-version: 1.5-1

bus-info: 0000:06:00.1

This NIC supports SR-IOV as listed in Intel and VMware (Intel Corporation 82576 Gigabit Network Connection).

4. The driver version for vmnic5 is 2.1.11.1. This version doesn't support SR-IOV which is listed in VMware support list as well as Intel support list. You need to upgrade the driver of your NIC. For Intel 82576, I have upgrade to version 3.4.7.3. I will brief how to upgrade the driver.

a. Download the driver from VMware download center based on the version of ESXi (in my case it was VMware ESXi 5.0 Driver for Intel 82580 and I350 Gigabit Ethernet Controllers)

b. Unzip the driver file and identify a file named as #driver_name#-#version#-offline_bundle-#build_number# (in my case it was igb-3.4.7.3-offline_bundle-804663).
c. From vSphere Client, navigate to Home > Solutions and Applications > Update Manager > Patch Repository.

d. Select Import Patches hyperlink and select the file from your machine (igb-3.4.7.3-offline_bundle-804663).

e. Create a new Baseline with type as Extension and include your driver in this baseline.

f. Navigate to Home > Inventory > Hosts and Clusters > #Select the desired Host# > Update Manager Tab.

g. Attach the baseline to your host and Stage/Remediate.

You can verify the update by logging into CLI.

~ # ethtool -i vmnic5

driver: igb

version: 3.4.7.3

firmware-version: 1.5-1

bus-info: 0000:06:00.1

Also, you can update the driver using CLI instead of VUM.

5. Create a Host Profile for the host which has the NIC to be used for SR-IOV. To create the profile Right-Click the Host > Host Profile > Manage Profile. 
6. Navigate to Home > Management > Host Profiles. Select the interested profile and Edit Settings. 
7. From the Settings Navigate to Kernel Module Configuration > Kernel Module > #Driver_Name# > Kernel Module Parameters > max_vfs > Module Parameter Settings.

Note: In case your driver doesn't support SR-IOV, max_vfs kernel parameter won't be listed under the driver. This is another method to check your driver support to SR-IOV.

Enter the max number of Virtual Functions (VFs) per Physical Function (PF). In case of dual-port 82576, you can enter the value as max_vfs=x,x. The max number of VFs per PF depends on the NIC. For Intel 82576 it is 16.

Note: A value of '0' means that SR-IOV is disable for the specified PF.

8. Apply the profile to the host after the modifications and reboot the host. 
9. Once the host is up, Select the host and navigate to Configuration > Network Adapters. The NICs enabled for SR-IOV won't be listed. Navigate to Advanced Settings under Hardware section, you will see that VFs are listed as Passthrough devices. At this stage, you have completed SR-IOV configuration.  
10. Shutdown you VMs > Edit Settings > Add PCI Devices. You VM Guest OS should be supporting SR-IOV and having VFs drivers.