Transmit

Alveo X3522 User Guide (UG1523)

Document ID
UG1523
Release Date
2023-08-25
Revision
1.4 English

The only method for transmitting packets on the X3522 is Cut-Through Programmed I/O (CTPIO). This has been heavily updated compared to its predecessors to reduce latency and make it more robust:

  • The X2522 has a single CTPIO transmit aperture per network port. The aperture is shared between all senders, and so there is a risk of clashes between senders. If CTPIO fails, the sender must fallback to DMA.
  • The X3522 adapter supports up to 16 CTPIO transmit apertures per network port (so 16×4 in total). The intention is that every sender can have a dedicated CTPIO aperture and so avoid clashes. If CTPIO fails, fallback automatically occurs to store-and-forward, which will always succeed.
Figure 1. CTPIO Apertures on the X2522 and X3522

One of the CTPIO apertures on the X3522 is used by the Linux kernel, and so 15 are available for other senders. Onload uses one of these apertures per stack.

In situations where there are more than 16 senders, either some will not be able to perform accelerated sends, or they will need to share access to an aperture. If processes are sharing transmit apertures they must lock any access themselves entirely within software, and this does not give optimal performance. It would make sense for such processes to have similar locality to the shared receive queues, so apertures are shared between processes running on the same CPU cluster.

Each transmit aperture has an associated event queue. The common use case is for each transmit aperture to have a separate event queue that is consumed by the user-level sender. This means that a sending thread can process transmit events without having to handle receive events or other traffic. Event queues can also in theory be shared between transmit apertures.

The X3522 includes 32 KB of buffering per transmit aperture to handle bursts of packets and reordering of any out-of-order PCIe transfers.

The X3522 allows warming the code path, with a flag not to send data.

As it is fully cut-through, the X3522 does not support TCP/IP checksum offload on transmit.

Like its predecessors, the X3522 sends a poisoned frame on failure, but its improved design means this is significantly less likely to occur.