KVM Libvirt Direct Passthrough - KVM Libvirt Direct Passthrough

In this configuration, VFs are used in the host OS to provide network acceleration for guest VMs. The guest continues to use a paravirtualized driver and is unaware this is backed with a VF from the network adapter.

Figure 1. KVM Libvirt Direct Passthrough

The AMD Solarflare net driver is bound over the top of each VF.
Each macvtap interface is implicitly created by libvirt over a single VF network interface and is not visible to the host OS.
Each macvtap instance builds over a different network interface, so there is no implicit macvtap bridge.
Macvtap does not currently forward multicast joins from the guests to the underlying network driver, so all multicast traffic received by the physical port is forwarded to all guests. Due to this limitation this configuration is not recommended for deployments that use a non-trivial amount of multicast traffic.
Guest migration is fully supported as there is no physical hardware state in the VM guests. A guest can be reconfigured to a host using a different VF or a host without an SR-IOV capable adapter.
The MAC address from the VF is passed through to the para-virtualized driver.
Because there is no VF present in a VM, Onload and other AMD Solarflare applications such as SolarCapture cannot be used in the VM.

Configuring KVM Libvirt Direct Passthrough

To configure a KVM Libvirt Direct Passthrough setup:

Ensure SR-IOV and the IOMMU are enabled on the host server kernel command line.
Refer to Setting Up the Server for SR-IOV.
Ensure the AMD Solarflare adapter driver (sfc.ko) is installed on the host.
Enable VFs on the adapter by loading a configuration file onto it. These example parameters select the full_featured firmware variant (which is required), and configure four VFs per PF:
```
[global]
firmware_variant=full_featured
vf_count=4
```
Refer to Configuring the Adapter for SR-IOV.
Give the server a cold reboot to perform the adapter reconfiguration.
Create the VFs, up to the maximum number of VFs that you enabled in step 3. For example using sysfs to create two VFs on the eth8 PF:
```
echo 2 > /sys/class/net/eth8/device/sriov_numvfs
```
Confirm the setting you have made:
```
cat /sys/class/net/eth8/device/sriov_totalvfs
```
Refer to Configuring SR-IOV.
Confirm that the PFs and VFs are visible using the lspci command:
```
# lspci -D -d1924:
```
Refer to Examining PFs and VFs.
Confirm that the PFs and VFs are visible using the ifconfig command:
```
# ifconfig
```
Refer to Examining PFs and VFs.
Create the virtual machine (VM).
You can create VMs from the standard Linux virt-manager GUI interface, or from the equivalent virsh command line tool.
Note: To start the GUI interface, run the virt-manager command as root from a terminal .

You can also create VMs from an existing VM XML file.
For each VF to be passed through to a VM, create a configuration file in the /etc/sysconfig/network-scripts directory (for example ifcfg-eth6):
```
DEVICE=eth6
TYPE=Ethernet
HWADDR=AE:82:AB:C9:67:49
BOOTPROTO=none
ONBOOT=yes
```
The above example is the file ifcfg-eth6, and identifies the MAC address assigned to the VF. One file is required for each VF.
On the host, edit the VM XML file:
```
# virsh edit <vmname>
```
Add the interface component to the VM XML file. For example:
```
<interface type='direct'>
  <source dev='eth3' mode='passthrough'/>
  <model type='virtio'/>
</interface>
```
One interface type component is required for each VF.
Restart the VM after editing the XML file.
```
# virsh start <vmname>
```
The passed through VF interface is visible when viewed from the GUI Virtual Machine Manager:

Figure 2. Virtual Machine Manager, Showing the Passthrough Interface

XML Description for KVM Libvirt Direct Passthrough

The following extract is from the VM XML file after a VF has been passed through to the guest using the configuration procedure above. Line numbers have been added for ease of description:

   1. <interface type='direct'>
    2.  <mac address='52:54:00:96:40:28'/>
    3.  <source dev='eth1' mode='passthrough'/>
    4.  <model type='virtio'/>
    5.  <address type='pci' domain='0x0000' bus='0x00' slot='0x07'  function='0x0'/>
    6. </interface>

The interface type. A user-added definition of how the VF interface is managed.
The MAC address. If this is not specified a random MAC address is assigned automatically by the guest OS.
The source dev. This is the VF interface identifier added in configuration step 9 above. The mode must also be specified as ‘passthrough’.
The model type. If this is not specified it is assigned automatically by libvirt when the guest is started. Use 'virtio' for best performance.
The VF PCIe address (as known by the guest). This is added automatically by libvirt.

For further information about the direct passthrough configuration and XML formats, refer to the following link:

http://libvirt.org/formatdomain.html#elementsNICSDirect

KVM Libvirt Direct Passthrough - KVM Libvirt Direct Passthrough - UG1739

AMD Solarflare X4 Series Ethernet Adapter User Guide (UG1739)

Configuring KVM Libvirt Direct Passthrough

XML Description for KVM Libvirt Direct Passthrough