Determine and Set Interrupt Affinity for Queues

Alveo X3522 User Guide (UG1523)

Document ID
UG1523
Release Date
2023-08-25
Revision
1.4 English

As well as the layout of your processor, you must determine the layout of interrupts for the receive and transmit queues.

To determine and set interrupt affinity for the receive and transmit queues, use the following commands:

  • To view the interrupts assigned to a particular interface:
    $ ls /sys/class/net/<interface>/device/msi_irqs/

    For example to view the interrupts assigned to the enp1s0f0np0 interface:

    $ ls /sys/class/net/enp1s0f0np0/device/msi_irqs/
    46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  61  62  63  64  65  66  67  68  69  70  71  72  73  74  75  76  77  78  79  80  81  82  83  84  85  86  87  88  89  90  91  92  93
  • To view the interrupts that are currently in use, and determine the queues for which they are being used:
    $ cat /proc/interrupts
    Note: The example outputs for this command show only the interrupts for a single X3522 interface, and omit all output related to other devices.

    Initially, you see the interrupts for the receive queues (<interface>-rx-<queue_num>) and for the default transmit queue (<interface>-tx-0):

    $ cat /proc/interrupts
             CPU0    CPU1    CPU2    CPU3    CPU4    CPU5     CPU6    CPU7
    ...
     114:      44       0       0       0       1       0   125010       0   PCI-MSI 49283072-edge   enp1s0f0np0-rx-0
     116:  125010       0       0       0       1       0       44       0   PCI-MSI 49283073-edge   enp1s0f0np0-rx-1
     118:   58288      44       1       0       0       0    66723       0   PCI-MSI 49283074-edge   enp1s0f0np0-rx-2
     120:  125010       0       0       0       1       0        0      44   PCI-MSI 49283075-edge   enp1s0f0np0-rx-3
     122:       0       0       0       0       0       0        0       0   PCI-MSI 49283076-edge   enp1s0f0np0-rx-4
     124:       0       0       0       0       0       0        0       0   PCI-MSI 49283077-edge   enp1s0f0np0-rx-5
     126:       0       0       0       0       0       0        0       0   PCI-MSI 49283078-edge   enp1s0f0np0-rx-6
     128:       0       0       0       0       0       0        0       0   PCI-MSI 49283079-edge   enp1s0f0np0-rx-7
     130:      18       0       1       0       0       0       31       0   PCI-MSI 49283080-edge   enp1s0f0np0-tx-0
    ...

    As applications start, the interrupts for their transmit queues are added to the output (onld-<n> below):

    $ cat /proc/interrupts
             CPU0    CPU1    CPU2    CPU3    CPU4    CPU5     CPU6    CPU7
    ...
     114:      44       0       0       0       1       0   125010       0   PCI-MSI 49283072-edge   enp1s0f0np0-rx-0
     116:  125010       0       0       0       1       0       44       0   PCI-MSI 49283073-edge   enp1s0f0np0-rx-1
     118:   58288      44       1       0       0       0    66723       0   PCI-MSI 49283074-edge   enp1s0f0np0-rx-2
     120:  125010       0       0       0       1       0        0      44   PCI-MSI 49283075-edge   enp1s0f0np0-rx-3
     122:       0       0       0       0       0       0        0       0   PCI-MSI 49283076-edge   enp1s0f0np0-rx-4
     124:       0       0       0       0       0       0        0       0   PCI-MSI 49283077-edge   enp1s0f0np0-rx-5
     126:       0       0       0       0       0       0        0       0   PCI-MSI 49283078-edge   enp1s0f0np0-rx-6
     128:       0       0       0       0       0       0        0       0   PCI-MSI 49283079-edge   enp1s0f0np0-rx-7
     130:      18       0       1       0       0       0       31       0   PCI-MSI 49283080-edge   enp1s0f0np0-tx-00
     132:       0       0       0       0       0       0        0       0   PCI-MSI 49283081-edge   onld-0
     133:       0       0       0       0       0       0        0       0   PCI-MSI 49283113-edge   onld-1
     180:       0       0       0       0       0       0        0       0   PCI-MSI 49285129-edge   onld-2
     181:       0       0       0       0       0       0        0       0   PCI-MSI 49285161-edge   onld-3
    ...
  • To determine the current affinity of the interrupt <n> that is being used for a particular queue:
    $ cat /proc/irq/<n>/smp_affinity

    For example in the previous output, interrupt 114 is being used for receive queue 0 on the enp1s0f0np0 interface. To determine its current affinity:

    $ cat /proc/irq/114/smp_affinity
    8

    The output is a hexadecimal bitmask. In this example the output is 8 hex (i.e. bit 3 is set), and so interrupt 114 is affinitized to CPU core 3.

  • To set the current affinity of the interrupt <n>:
    $ sudo echo <bitmask> > /proc/irq/<n>/smp_affinity

    For example to affinitize interrupt 114 to CPU core 2, use a bitmask with value 4 (i.e. bit 2 is set):

    $ sudo echo 4 > /proc/irq/114/smp_affinity

The above techniques enable you to determine the associations between queues and CPU cores, and to change them if necessary.