Set the CPU and Receive Queue for your Application

Alveo X3522 User Guide (UG1523)

Document ID
UG1523
Release Date
2023-08-25
Revision
1.4 English

Now that you have determined the layout of your processor and receive queues, decide which CPU and receive queue you want to use for your application. If necessary, revisit the previous topics in this section.

Set the CPU for your application, and add ethtool rules to ensure that your application uses your desired receive queue:

  1. As root, install an ethtool rule to direct the traffic for your application to your desired receive queue:
    # ethtool -N <interface> flow-type tcp4|udp4|tcp6|udp6 dst-ip <ip_addr> dst-port <port> queue <n>

    For example if your application is using the enp1s0f0np0 interface, and will be receiving UDP v4 traffic sent to port 1234 at address 192.168.10.200, then to use receive queue 3:

    # ethtool -N enp1s0f0np0 flow-type udp4 dst-ip 192.168.10.200 dst-port 1234 queue 3
    Added rule with ID 0
    Note: If this would exceed the maximum number of filters, an error message is output, and the filter is not inserted. For example:
    rmgr: Cannot find appropriate slot to insert rule
    Cannot insert classification rule
  2. Optionally list the rules for the interface and confirm that the new rule is present:
    # ethtool -n <interface>

    For example:

    # ethtool -n enp1s0f0np0
    8 RX rings available
    Total 1 rules
    
    Filter: 0
            Rule Type: UDP over IPv4
            Src IP addr: 0.0.0.0 mask: 255.255.255.255
            Dest IP addr: 192.168.10.200 mask: 0.0.0.0
            TOS: 0x0 mask: 0xff
            Src port: 0 mask: 0xffff
            Dest port: 1234 mask: 0x0
            Action: Direct to queue 3
  3. Use the taskset command to start your application on its desired CPU:
    taskset -c <cpu> <command line>
  4. Have your application create a filter with the same parameters as the ethtool rule.

    The pre-installed ethtool rule is recognized as identical and is re-used. The application therefore uses the receive queue that was specified for the ethtool rule.

  5. When your application quits, delete the ethtool rule:
    ethtool -N <interface> delete <rule number>

    For example, to delete rule 0 that was installed in step 1 above:

    ethtool -N enp1s0f0np0 delete 0
  6. List the rules for the interface and confirm that the rule is deleted:
    # ethtool -n <interface>

    For example:

    # ethtool -n enp1s0f0np0
    8 RX rings available
    Total 0 rules
    
Note: The use of taskset is typically only suitable for affinity tuning single threaded, single traffic flow applications. For a multi-threaded application, whose threads for example process a subset of receive traffic, taskset is not suitable. In such applications, you might use multiple ethtool rules to spread receive traffic over multiple queues and then have each application thread bind to an adjacent CPU. Thread affinities can be set inside the application with the shed_setaffinity() function (see Linux man pages). Use of this call and how a particular application can be tuned is beyond the scope of this guide.