Tuning Settings - UG1523

Alveo X3522 User Guide (UG1523)

Document ID
UG1523
Release Date
2024-10-18
Revision
1.5 English

This section describes Linux tuning settings that can affect adapter performance.

Interrupt Moderation (Interrupt Coalescing)

Interrupt moderation reduces the number of interrupts generated by the adapter by coalescing multiple received packet events and/or transmit completion events together into a single interrupt.

The interrupt moderation interval sets the minimum time (in microseconds) between two consecutive interrupts. Coalescing occurs only during this interval:

  • When the driver generates an interrupt, it starts timing the moderation interval.
  • Any events that occur before the moderation interval expires are coalesced together into a single interrupt, that is raised only when the interval expires.

    A new moderation interval then starts, during which no interrupt is raised.

  • An event that occurs after the moderation interval has expired gets its own dedicated interrupt, that is raised immediately.

    A new moderation interval then starts, during which no interrupt is raised.

X3522 adapters, by default, use an adaptive algorithm where the interrupt moderation delay is automatically adjusted between zero (no interrupt moderation) and 60 microseconds. The adaptive algorithm detects latency sensitive traffic patterns and adjusts the interrupt moderation interval accordingly.

Interrupt moderation settings are critical for tuning adapter latency:

  • Disabling the adaptive algorithm will:
    • reduce jitter
    • allow setting the moderation interval as required to suit conditions.
  • Increasing the interrupt moderation interval will:
    • generate less interrupts
    • reduce CPU utilization (because there are less interrupts to process)
    • increase latency
    • improve peak throughput.
  • Decreasing the interrupt moderation interval will:
    • generate more interrupts
    • increase CPU utilization (because there are more interrupts to process)
    • decrease latency
    • reduce peak throughput.
  • Turning off interrupt moderation will:
    • generate the most interrupts
    • give the highest CPU utilization
    • give the lowest latency
    • give the biggest reduction in peak throughput.

For many transaction request-response type network applications, the benefit of reduced latency to overall application performance can be considerable. Such benefits typically outweigh the cost of increased CPU utilization. It is recommended that:

  • Interrupt moderation is disabled for applications that require best latency and jitter performance, such as market data handling.
  • Interrupt moderation is enabled for high throughput single (or few) connection TCP streaming applications, such as iSCSI.

Interrupt moderation can be changed using ethtool, where <interface> is the interface name. Before adjusting the interrupt moderation interval, it is recommended to disable adaptive moderation:

ethtool -C <interface> adaptive-rx off

To set the RX interrupt moderation interval in microseconds (μs):

ethtool –C <interface> rx-usecs <interval>

To turn off interrupt moderation, set an interval of zero (0):

ethtool –C <interface> rx-usecs 0

To set the TX interrupt moderation interval:

ethtool –C <interface> tx-usecs <interval>

The above examples set both the receive and the transmit interrupt moderation interval. Packet RX and TX completions share interrupts, so RX and TX interrupt moderation intervals must be equal, and the adapter driver automatically adjusts tx-usecs to match rx-usecs.

Interrupt moderation settings can be checked using ethtool –c .

TCP/IP Checksum Offload

Checksum offload moves calculation and verification of IP Header, TCP, and UDP packet checksums to the adapter. The driver has all checksum offload features enabled by default. Therefore, there is no opportunity to improve performance from the default.

Receive checksum offload is controlled using ethtool:

# /sbin/ethtool –K <interface> rx <on|off>

As it is fully cut-through, the X3522 does not support TCP/IP checksum offload on transmit.

Verification of the checksum settings can be performed by running ethtool with the –k option.

Note: It is not recommended to disable the checksum offload.