Other Considerations

This section describes some other considerations.

Physical Medium

Latency is affected by the type of physical medium used: twinaxial (direct-attach) or fiber. This is because the physical media interface chip (PHY) used on the adapter can introduce additional latency.

Modules Fitted

Latency can also be affected by the type of SFP/DSFP/QSFP module fitted.

PCI Express Lane Configurations

PCI® Express (PCIe) interfaces can function at different speeds and widths. This is independent of their physical slot size. The possible widths are multiples ×1, ×2, ×4, ×8, and ×16 lanes of (2.5 Gb/s for PCIe Gen 1, 5.0 Gb/s for PCIe Gen 2, 8.0 Gb/s for PCIe Gen 3 and 16.0 Gb/s for PCIe Gen 4) in each direction.

On some server motherboards, choice of PCIe slot is important. This is because some slots (including those that are physically ×8 or ×16 lanes) might only electrically support ×4 lanes. X3522 adapters are designed for ×8 or ×16 lane operation. The X3522 driver warns if it detects that the adapter is plugged into a PCIe slot which electrically has fewer than ×8 lanes.

X3522 adapters require a PCIe Gen 4 ×8 or Gen 3 ×8 slot for optimal performance. The X3522 driver warns if it detects that the adapter is placed in a sub-optimal slot.

Warning messages can be viewed in dmesg from /var/log/messages.

The lspci command can be used to discover the currently negotiated PCIe lane width and speed:

$ sudo lspci -d 10ee: -vv
01:00.0 Ethernet controller: Xilinx Corporation Device 5084
                ...
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM not supported
                ...
                LnkSta: Speed 8GT/s (ok), Width x16 (ok)

In addition, the latency of communications between the host CPUs, system memory and the X3522 PCIe adapter might be PCIe slot dependent. Some slots might be “closer” to the CPU, and therefore have lower latency and higher throughput. If possible, install the adapter in a slot which is local to the desired NUMA node.

Consult your server user guide for more information.

Memory Bandwidth

Many chipsets use multiple channels to access main system memory. Maximum memory performance is only achieved when the chipset can make use of all channels simultaneously. This should be taken into account when selecting the number of memory modules (DIMMs) to populate in the server. For optimal memory bandwidth in the system, it is likely that:

• All DIMM slots should be populated.
• All NUMA nodes should have memory installed.

Consult the motherboard documentation for details.

Server Motherboard, Server BIOS, Chipset Drivers

Tuning or enabling other system capabilities might further enhance adapter performance. Readers should consult their server user guide. Possible opportunities include tuning the PCIe memory controller via the PCIe Latency Timer setting available in some BIOS versions.

Out of Scope Considerations

In addition, you might need to consider other issues influencing performance, such as application settings, server motherboard chipset, CPU speed, cache size, RAM size, additional software installed on the system such as a firewall, and the specification and configuration of the LAN. Consideration of such issues is not within the scope of this guide.