On the receive side, because there are a reduced number of queues they need to be shared in some situations. The intended model is that a receive queue is associated with a CPU cluster to benefit from cache locality, and all the applications running on that cluster share that queue. Because the receive queues are shared they must be managed by the driver rather than directly by the applications.
On predecessors to the X3522 such as the X2522, packets are received into a 4 KB standard page:
- Applications own and manage the packet buffers, and post them to the NIC.
- The driver uses events to notify applications of new packets.
See the following figure:
On the X3522, packets are received into much larger 1 MB super buffers, and the NIC delivers multiple packets at a fixed stride (2 KB) to each super buffer.
- The super buffers are allocated using explicit huge pages. All super buffers for an RX queue are held in a single pool.
- The driver owns the packet buffers, posts them to the NIC, and publishes them to the applications that are using that queue.
- The driver handles all the receive events and buffer management leaving receivers free to detect presence of new packets by watching the packet buffer memory for changes instead of consuming events.
See the following figure:
Because the receive queues can be shared by more than one client, there must be a trust relationship between processes on the system. The X3522 should not be used in scenarios where strict POSIX isolation is necessary.