Tuning Settings - UG1739

This section describes adapter settings for tuning.

Port Mode

Ensure the selected port mode corresponds to the speed and number of connectors in use, after using any splitter cables. If a restricted set of connectors is configured, the driver can then transfer resources from the unused connectors to those configured, potentially improving performance.

Adapter MTU (Maximum Transmission Unit)

The default MTU of 1500 bytes ensures that the adapter is compatible with all other Ethernet endpoints. However if a larger MTU is used, adapter throughput and CPU utilization can be improved. CPU utilization is improved, because it takes fewer packets to send and receive the same amount of data. X4 series adapters support an MTU of up to 9100 bytes on the Enterprise datapath (this does not include the Ethernet preamble or frame-CRC).

You can increase the MTU for an interface only if it always uses the Enterprise datapath. If an interface is passing traffic for accelerated applications it might use the Express datapath, and so you should not change the MTU from its default of 1500 bytes.

The MTU is ideally matched across all endpoints in the same LAN (VLAN), and the LAN switch infrastructure must be able to forward such packets, therefore the decision to deploy a larger than default MTU requires careful consideration. It is recommended that experimentation with MTU be done in a controlled test environment.

To change the MTU dynamically use ip, where <interface> is the interface name and <size> is the MTU size in bytes:

# ip link set dev <interface> mtu <size>

You can verify the MTU setting by running ip link with no options and checking the MTU value associated with the interface. To make a change in MTU size persist across reboots, edit the file /etc/sysconfig/network-scripts/ifcfg-<interface> and add MTU=<mtu> on a new line.

Interrupt Moderation (Interrupt Coalescing)

Interrupt moderation reduces the number of interrupts generated by the adapter by coalescing multiple received packet events and/or transmit completion events together into a single interrupt.

The interrupt moderation interval sets the minimum time (in microseconds) between two consecutive interrupts. Coalescing occurs only during this interval:

• When the driver generates an interrupt, it starts timing the moderation interval.
• Any events that occur before the moderation interval expires are coalesced together into a single interrupt, that is raised only when the interval expires.

  A new moderation interval then starts, during which no interrupt is raised.

• An event that occurs after the moderation interval has expired gets its own dedicated interrupt, that is raised immediately.

  A new moderation interval then starts, during which no interrupt is raised.

X4 series adapters, by default, use an adaptive algorithm where the interrupt moderation delay is automatically adjusted between zero (no interrupt moderation) and 60 microseconds. The adaptive algorithm detects latency sensitive traffic patterns and adjusts the interrupt moderation interval accordingly.

Interrupt moderation settings are critical for tuning adapter latency:

• Disabling the adaptive algorithm:
  • Reduces jitter
  • Allows setting the moderation interval as required to suit conditions.
• Increasing the interrupt moderation interval:
  • Generates fewer interrupts
  • Reduces CPU utilization (because there are less interrupts to process)
  • Increases latency
  • Improves peak throughput.
• Decreasing the interrupt moderation interval:
  • Generates more interrupts
  • Increases CPU utilization (because there are more interrupts to process)
  • Decreases latency
  • Reduces peak throughput.
• Turning off interrupt moderation:
  • Generates the most interrupts.
  • Gives the highest CPU utilization.
  • Gives the lowest latency.
  • Gives the biggest reduction in peak throughput.

For many transaction request-response type network applications, the benefit of reduced latency to overall application performance can be considerable. Such benefits typically outweigh the cost of increased CPU utilization. It is recommended that:

• Interrupt moderation is disabled for applications that require best latency and jitter performance, such as market data handling.
• Interrupt moderation is enabled for high throughput single (or few) connection TCP streaming applications, such as iSCSI.

To change interrupt moderation use ethtool, where <interface> is the interface name. Before adjusting the interrupt moderation interval, it is recommended to disable adaptive moderation:

ethtool -C <interface> adaptive-rx off

To set the RX interrupt moderation <interval> in microseconds (μs):

ethtool –C <interface> rx-usecs <interval>

To turn off interrupt moderation, set an interval of zero (0):

ethtool –C <interface> rx-usecs 0

The above example also sets the transmit interrupt moderation interval. (Packet RX and TX completions share interrupts, so RX and TX interrupt moderation intervals must be equal, and the adapter driver automatically adjusts tx-usecs to match rx-usecs.) Refer to Module Parameters.

You can check interrupt moderation settings using ethtool -c.

TCP/IP Checksum Offload

Checksum offload moves calculation and verification of IP Header, TCP and UDP packet checksums to the adapter. The driver has all checksum offload features enabled by default. Therefore, there is no opportunity to improve performance from the default.

To control checksum offload use ethtool:

• Receive checksum offload:

  # ethtool –K <interface> rx <on|off>

• Transmit checksum: offload:

  # ethtool –K <interface> tx <on|off>

To verify the checksum offload settings run ethtool with the –k option.

TCP Segmentation Offload (TSO)

TCP Segmentation Offload (TSO) offloads the splitting of outgoing TCP data into packets to the adapter. TSO benefits applications using TCP. Applications using protocols other than TCP are not affected by TSO.

Enabling TSO reduces CPU usage on the transmit side of a TCP connection and improves peak throughput, if the CPU is fully used. TSO has no effect on latency, therefore it can be enabled at all times. The driver has TSO enabled by default. Therefore, there is no opportunity to improve performance from the default.

To control TSO use ethtool:

# /sbin/ethtool –K <interface> tso <on|off>

To verify the TSO settings run ethtool with the –k option.

TCP and IP checksum offloads must be enabled for TSO to work.

TCP Large Receive Offload (LRO)

TCP Large Receive Offload (LRO) is a feature whereby the adapter coalesces multiple packets received on a TCP connection into a single larger packet before passing this onto the network stack for receive processing. This reduces CPU usage and improves peak throughput when the CPU is fully used. The effectiveness of LRO is bounded by the interrupt moderation delay, and is limited if interrupt moderation is disabled (see Interrupt Moderation (Interrupt Coalescing)). Enabling LRO does not itself negatively impact latency.

To control LRO on all adapters, use the lro module parameter. To disable LRO add the following line either to the /etc/modprobe.conf file, or to a file with a .conf extension under the /etc/modprobe.d directory:

options sfc lro=0

You must reload the driver to enable the option change:

rmmod sfc
modprobe sfc

You must then rebuild the initramfs to ensure the option change persists after a reboot:

dracut -f

To find the current value of this parameter:

cat /sys/module/sfc/parameters/lro

to control LRO on a per-adapter basis, write to this file in sysfs:

/sys/class/net/<interface>/device/lro

where <interface> is the name of the adapter interface.

• To disable LRO:

  echo 0 > /sys/class/net/<interface>/device/lro

• To enable LRO:

  echo 1 > /sys/class/net/<interface>/device/lro

• To show the current value of the per-adapter LRO state:

  cat /sys/class/net/<interface>/device/lro

Modifying this file instantly enables or disables LRO. No reboot or driver reload is required. This setting takes precedence over the lro module parameter.

To verify the current LRO settings run ethtool with the –k option.

ethtool -k <interface>

TCP and IP checksum offloads must be enabled for LRO to work.

The performance_profile Module Option

You can set a performance profile to optimize adapter performance either for low latency or for high throughput.

Set the performance_profile module parameter to latency or throughput as required. For example to optimize for throughput add the following line either to the /etc/modprobe.conf file, or to a file with a .conf extension under the /etc/modprobe.d directory:

options sfc performance_profile=throughput

You must reload the driver to enable the option change:

rmmod sfc
modprobe sfc

You must then rebuild the initramfs to ensure the option change persists after a reboot:

dracut -f

To find the current value of this parameter run the following:

cat /sys/module/sfc/parameters/performance_profile

For further information, see Module Parameters.