Example XPS Configuration - UG1739

This section gives an example of XPS configuration.

System Configuration

This example uses the following system configuration:

• Single X4 series adapter.
• Only 1 interface on the adapter is configured.
• 2 × 8 core processors with hyperthreading enabled to give a total of 32 cores.
• rss_cpus=8.
• The irqbalance service is disabled.

Identifying Interrupts and CPUs for the Configured Interface

First identify the interrupts for the configured interface. Assuming the interface is named eth3:

# cat /proc/interrupts | grep 'eth3\|CPU'

Output from the above command lists IRQs 132-139 inclusive. Now find which CPUs each of these interrupts can be routed to:

> cat /proc/irq/132/smp_affinity
00000000,00000000,00000000,00000001
> cat /proc/irq/133/smp_affinity
00000000,00000000,00000000,00000100
> cat /proc/irq/134/smp_affinity
00000000,00000000,00000000,00000002
[...snip...]
> cat /proc/irq/139/smp_affinity
00000000,00000000,00000000,00000800

The bitmasks that are output identify that IRQ 132 is routed to CPU0, IRQ 133 is routed to CPU8, IRQ 134 to CPU2, and so on.

Mapping TX queues to CPUs

Hyperthreaded cores are included with the associated physical core:

> echo 110011 > /sys/class/net/eth3/queues/tx-0/xps_cpus
> echo 11001100 > /sys/class/net/eth3/queues/tx-1/xps_cpus
> echo 220022 > /sys/class/net/eth3/queues/tx-2/xps_cpus
> echo 22002200 > /sys/class/net/eth3/queues/tx-3/xps_cpus
> echo 440044 > /sys/class/net/eth3/queues/tx-4/xps_cpus
> echo 44004400 > /sys/class/net/eth3/queues/tx-5/xps_cpus
> echo 880088 > /sys/class/net/eth3/queues/tx-6/xps_cpus
> echo 88008800 > /sys/class/net/eth3/queues/tx-7/xps_cpus

Configuring Global and Per Queue Tables

• The flow count (number of active connections at any one time) = 32768
• The number of queues = 8 (rss_cpus)
• So the flow count for each queue is 32768/8 = 4096:

> echo 32768 > /proc/sys/net/core/rps_sock_flow_entries
> echo 4096 > /sys/class/net/eth3/queues/rx-0/rps_flow_cnt
> echo 4096 > /sys/class/net/eth3/queues/rx-1/rps_flow_cnt
> echo 4096 > /sys/class/net/eth3/queues/rx-2/rps_flow_cnt
> echo 4096 > /sys/class/net/eth3/queues/rx-3/rps_flow_cnt
> echo 4096 > /sys/class/net/eth3/queues/rx-4/rps_flow_cnt
> echo 4096 > /sys/class/net/eth3/queues/rx-5/rps_flow_cnt
> echo 4096 > /sys/class/net/eth3/queues/rx-6/rps_flow_cnt
> echo 4096 > /sys/class/net/eth3/queues/rx-7/rps_flow_cnt