KVM Libvirt Network Hostdev - KVM Libvirt Network Hostdev

Network Hostdev exposes VFs directly into guest VMs allowing the data path to fully bypass the host OS, and therefore provides maximum acceleration for network traffic.

Figure 1. KVM Libvirt Network Hostdev

The hostdev configuration delivers the highest throughput and lowest latency performance. Because the guest is directly linked to the virtual function it is therefore directly connected to the underlying hardware.
Migration is not supported in this configuration because the VM has knowledge of the network adapter hardware (VF) present in the server.
The VF is visible in the guest. This allows applications using the VF interface to be accelerated using Onload or to use other AMD Solarflare applications such as SolarCapture.
The AMD Solarflare net driver (sfc.ko) must be installed in the guest.

Configuring KVM Libvirt Network Hostdev

To configure a KVM Libvirt Network Hostdev setup:

Ensure SR-IOV and the IOMMU are enabled on the host server kernel command line.
Refer to Setting Up the Server for SR-IOV.
Create the virtual machine (VM).
You can create VMs from the standard Linux virt-manager GUI interface, or from the equivalent virsh command line tool.
Note: To start the GUI interface, run the virt-manager command as root from a terminal .

You can also create VMs from an existing VM XML file.
Install the AMD Solarflare adapter driver (sfc.ko) on the host and guest.
Enable VFs on the adapter by loading a configuration file onto it. These example parameters select the full_featured firmware variant (which is required), and configure four VFs per PF:
```
[global]
firmware_variant=full_featured
vf_count=4
```
Refer to Configuring the Adapter for SR-IOV.
Give the server a cold reboot to perform the adapter reconfiguration.
Create the VFs, up to the maximum number of VFs that you enabled in step 4. For example using sysfs to create four VFs on the eth8 PF:
```
echo 4 > /sys/class/net/eth8/device/sriov_numvfs
```
Confirm the setting you have made:
```
cat /sys/class/net/eth8/device/sriov_totalvfs
```
Refer to Configuring SR-IOV.
Confirm that the PFs and VFs are visible using the ifconfig command:
```
# ifconfig
```
Refer to Examining PFs and VFs.

Use the lspci command to get the PCI® addresses for the VFs:

# lspci -D -d1924:
0000:03:00.0 Ethernet controller: Solarflare Communications SFC9120 (rev 01)
0000:03:00.1 Ethernet controller: Solarflare Communications SFC9120 (rev 01)
0000:03:00.2 Ethernet controller: Solarflare Communications Device 1903 (rev 01)
0000:03:00.3 Ethernet controller: Solarflare Communications Device 1903 (rev 01)
0000:03:00.4 Ethernet controller: Solarflare Communications Device 1903 (rev 01)
0000:03:00.5 Ethernet controller: Solarflare Communications Device 1903 (rev 01)

The VFs are presented as Communications Device 1903. Note their PCI® addresses for the steps that follow.

Using the PCIe address, unbind the VFs from the host sfc driver . For example:
```
# echo 0000:03:00.5 > /sys/bus/pci/devices/0000\:03\:00.5/driver/unbind
```
Confirm that the required VF interface is no longer visible in the host using ifconfig:
```
# ifconfig
```
On the host, edit the VM XML file:
```
# virsh edit <vmname>
```

For each VF that is to be passed to the guest, use its PCI® address to add the following <interface type> section to the VM XML file :

<interface type='hostdev' managed='yes'>
  <source>
   <address type='pci' domain='0x0000' bus='0x03' slot='0x00'  function='0x5'/>
  </source>
</interface>

Restart the virtual machine in the host:
```
# virsh start <vmname>
```
The VF interfaces are now visible in the guest.

XML Description for KVM Libvirt Network Hostdev

The following extract is from the VM XML file after a VF has been passed through to the guest using the configuration procedure above. Line numbers have been added for ease of description:

   1. <interface type='hostdev' managed='yes'>
   2.  <mac address='52:54:00:d1:ec:85'/>
        <source>
   3.    <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x5'/>
        </source>
   4.  <alias name='hostdev0'/>
   5.  <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>

The interface type. A user-added definition of how the VF interface is managed:
- When managed=yes, the VF is detached from the host before being passed to the guest, and the VF is automatically reattached to the host after the guest exits.
- When managed=no, you must call virNodeDeviceDetach (or use the command virsh nodedev-detach) before starting the guest or hot-plugging the device, and call virNodeDeviceReAttach (or use the command virsh nodedev-reattach) after hot-unplug or after stopping the guest.
The MAC address. If this is not specified a random MAC address is assigned automatically by libvirt.
The VF PCIe address (as known by the host). This should be entered by the user when editing the XML file.
The alias name. If this is not specified a name is assigned automatically by libvirt.
The VF PCIe address (as known by the guest). This is added automatically by libvirt.

For further information about the hostdev configuration and XML formats, refer to the following link:

http://libvirt.org/formatdomain.html#elementsNICSHostdev

KVM Libvirt Network Hostdev - KVM Libvirt Network Hostdev - UG1739

AMD Solarflare X4 Series Ethernet Adapter User Guide (UG1739)

Configuring KVM Libvirt Network Hostdev

XML Description for KVM Libvirt Network Hostdev