Interrupt Affinity

Interrupt affinity describes the set of host CPUs that can service a particular interrupt.

This affinity therefore dictates the CPU context where received packets are processed and where transmit packets are freed after they are sent. If your application is running in the same CPU context by being affinitized to the relevant CPU, this can improve latency and CPU utilization. This improvement is achieved because well tuned affinities reduce inter-CPU communication.

Tuning interrupt affinity is most relevant when MSI-X interrupts and RSS are being used. The irqbalance service, which typically runs by default in most Linux distributions, is a service that automatically changes interrupt affinities based on CPU workload.

In many cases the irqbalance service hinders rather than enhances network performance. It is therefore necessary to disable it and then set interrupt affinities.

To disable irqbalance permanently, run:
```
systemctl disable irqbalance
```
To see whether irqbalance is currently running, run:
```
systemctl status irqbalance
```
To disable irqbalance temporarily, run:
```
systemctl stop irqbalance
```

Once you have stopped the irqbalance service, you can manually configure the Interrupt affinities.

Note: The AMD Solarflare driver evenly distributes interrupts across the available host CPUs (based on the rss_cpus module parameter). To use the AMD Solarflare driver default affinities (recommended), the irqbalance service must be disabled before the AMD Solarflare driver is loaded (otherwise it immediately overwrites the affinity configuration values set by the AMD Solarflare driver).

Using the Same CPU

How affinities are best manually set depends on the application. For a single streamed application such as Netperf, one recommendation would be to affinitize all the RX queues and the application on the same CPU. You can achieve this with the following steps:

Determine which interrupt line numbers the network interface uses. Assuming the interface is eth0, this can be done with:

# cat /proc/interrupts | grep eth0-
123:   13302       0       0       0  PCI-MSI-X eth0-0
131:       0      24       0       0  PCI-MSI-X eth0-1
139:       0       0      32       0  PCI-MSI-X eth0-2
147:       0       0       0      21  PCI-MSI-X eth0-3

This output shows that there are four channels (rows) set up between four CPUs (columns).

Determine the CPUs to which these interrupts are assigned:
```
# cat /proc/irq/123/smp_affinity
00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
# cat /proc/irq/131/smp_affinity
00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000002
# cat /proc/irq/139/smp_affinity
00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000004
# cat /proc/irq/147/smp_affinity
00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000008
```
This shows that RXQ[0] is affinitized to CPU[0], RXQ[1] is affinitized to CPU[1], and so on. With this configuration, the latency and CPU utilization for a particular TCP flow depends on the RSS hash for the flow, and the CPU onto which the hash resolves.
Note: Interrupt line numbers and their initial CPU affinity are not guaranteed to be the same across reboots and driver reloads. Typically, it is therefore necessary to write a script to query these values and apply the affinity accordingly.
Set all network interface interrupts to a single CPU (in this case CPU[0]):
```
# echo 1 > /proc/irq/123/smp_affinity
# echo 1 > /proc/irq/131/smp_affinity
# echo 1 > /proc/irq/139/smp_affinity
# echo 1 > /proc/irq/147/smp_affinity
```
Note: The read-back of /proc/irq/N/smp_affinity returns the old value until a new interrupt arrives.
Set the application to run on the same CPU as the network interface interrupts (in this case CPU[0]):
```
# taskset 1 netperf
# taskset 1 netperf -H <host>
```
Note: Using taskset is typically only suitable for affinity tuning single threaded, single traffic flow applications. Using taskset is not suitable for a multi threaded application, where each thread might process a subset of receive traffic. In such applications you must instead use RSS and Interrupt affinity to spread receive traffic over more than one CPU, and then have each receive thread bind to each of the respective CPUs. You can set hread affinities inside the application with the shed_setaffinity() function (see Linux man pages). Use of this call and how to tune a particular application is beyond the scope of this guide.

If the settings are correctly applied, all interrupts from eth0 are now being handled on CPU[0]. You can check this:

# cat /proc/interrupts | grep eth0-
123:   13456       0       0       0  PCI-MSI-X eth0-0
131:      27      24       0       0  PCI-MSI-X eth0-1
139:      54       0      32       0  PCI-MSI-X eth0-2
147:      37       0       0      21  PCI-MSI-X eth0-3

Using the Same Package

This example demonstrates affinitizing each interface to a CPU on the same package.

Identify which interrupt lines are servicing which CPU and IO device:

# cat /proc/interrupts | grep eth0-
123:   13302       0 1278131       0  PCI-MSI-X eth0-0
# cat /proc/interrupts | grep eth1-
131:       0      24       0       0  PCI-MSI-X eth1-0

Use the sys files to find CPUs on the same package (their physical_package_id is the same):

# more /sys/devices/system/cpu/cpu*/topology/physical_package_id
::::::::::::::
/sys/devices/system/cpu/cpu0/topology/physical_package_id
::::::::::::::
1
::::::::::::::
/sys/devices/system/cpu/cpu10/topology/physical_package_id
::::::::::::::
1
::::::::::::::
/sys/devices/system/cpu/cpu11/topology/physical_package_id
::::::::::::::
0
…

This output shows that cpu0 and cpu10 are both on package 1.

Assign the MSI-X interrupt for each interface to its own CPU on the same package. This example uses package 1:

# echo 1 > /proc/irq/123/smp_affinity    # 1hex is bit 0 = CPU0
# echo 400 > /proc/irq/131/smp_affinity  # 400hex is bit 10 = CPU10

Interrupt Affinity - UG1739

AMD Solarflare X4 Series Ethernet Adapter User Guide (UG1739)

Using the Same CPU

Using the Same Package