Affinitizing RSS Channels to CPUs

Onload User Guide (UG1586)

Document ID

UG1586

Release Date

2023-07-31

Revision

1.2 English

As described in the previous section, the default behavior of the Solarflare network driver is to create one RSS channel per CPU core. At load time the driver affinitizes the interrupt associated with each RSS channel to a separate CPU core so the interrupt load is evenly distributed over the available CPU cores.

Note: These initial interrupt affinities will be disrupted and changed if the Linux IRQ balancer daemon is running. To stop the IRQ balancer use the following command: # service irqbalance stop

In the following example, we have a server with two dual-port network adapters (total of four network interfaces), installed in a server with two CPU sockets with eight cores per socket (hyperthreading is disabled).

If we set rss_cpus=4, each interface will create four RSS channels. The driver takes care to spread the affinitized interrupts evenly over the CPU topology, that is evenly between the two CPU sockets and evenly over shared L2/L3 caches.

The driver also attempts to spread the interrupt load of the multiple network interfaces by using different CPU cores for different interfaces:

Table 1. Example RSS Channel Mapping
Interface	Number of RX Queues	Map to Cores
1	4	0,1,2,3
2	4	4,5,6,7
3	4	8,9,10,11
4	4	12,13,14,15

With four receive queues created per interface this results, on this machine, to the first network interface mapping to the four lowest number CPU cores, that is two cores from each CPU socket as illustrated below. The next network interface uses the next four CPUs until each CPU core is loaded with a single RSS channel – as illustrated in the following figure.

Figure 1. Mapping RSS Channels to CPU cores.

To identify the mapping of receive queues to CPU cores, use the following command:

# cat /proc/interrupts | grep eth4
106: 19  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  IR-PCI-MSI-edge eth4-0
107:  0 11  0  0  0  0  0  0  0  0  0  0  0  0  0  0  IR-PCI-MSI-edge eth4-1
108:  0  0 10  0  0  0  0  0  0  0  0  0  0  0  0  0  IR-PCI-MSI-edge eth4-2
109:  0  0  0  2  0  0  0  0  0  0  0  0  0  0  0  0  IR-PCI-MSI-edge eth4-3

Observe that each receive queue has an assigned IRQ. Receive queue eth4-0 is served by IRQ 106, eth4-1 by IRQ 107 etc.

sfcaffinity_config

The OpenOnload distribution also includes the sfcaffinity_config script which can also be used to affinitize RSS channel interrupts. sfcaffinity_config has a number of command line options but a common way of running it is with the auto command:

# sfcaffinity_config auto

Auto instructs sfcaffinity_config to set interrupts affinities to evenly spread the RSS channels over the available CPU cores. Using the above scenario as an example, where rss_cpus has been set to 4, the command will affinitize the interrupt associated with each receive queue evenly over the CPU topology – in this case the first four CPU cores.

sfcaffinity_config: INFO: eth4: Spreading 4 interrupts evenly over 2 shared caches
sfcaffinity_config: INFO: eth4: bind rxq 0 (irq 106) to core 1
sfcaffinity_config: INFO: eth4: bind rxq 1 (irq 107) to core 0
sfcaffinity_config: INFO: eth4: bind rxq 2 (irq 108) to core 3
sfcaffinity_config: INFO: eth4: bind rxq 3 (irq 109) to core 2
sfcaffinity_config: INFO: eth4: configure sfc_affinity n_rxqs=4 cpu_to_rxq=1,0,3,2,1,0,3,2,1,0,3,2,1,0,3,2

Figure 2. Mapping with sfcaffinity_config auto

In this example, after running the sfcaffinity_config auto command, interrupts for the four receive queues from the four interfaces are now all directed to the same four cores 0,1,2,3 as illustrated by the preceding figure.

Note: Running the sfcaffinity_config auto command also disables the kernel irqbalance service to prevent interrupts being redirected by the kernel to other cores.

Using the irqbalance Service

If you want to keep using the irqbalance service, do not use the sfcaffinity_config auto command. Configure the irqbalance service using the /etc/sysconfig/irqbalance file:

To prevent the network adapter interrupts from being redirected by irqbalance, append instances of the --banirq option. to the IRQBALANCE_ARGS environment variable. For example, to exclude interrupts 106-109 inclusive:
```
IRQBALANCE_ARGS="--banirq=106 --banirq=107 --banirq=108 --banirq=109"
```
To exclude irqbalance from redirecting any interrupts to specific CPUs, include them in the IRQBALANCE_BANNED_CPUS bitmask. For example, to exclude CPUs 1 and 2, set it to 3 (bits 1 and 2 are set):
```
IRQBALANCE_BANNED_CPUS=3
```
Note: If this bitmask is not set, recent versions of irqblance do not use CPUs that are listed in the isolcpus kernel configuration parameter.

You can then manually configure the affinity of any excluded interrupts.