When Onload detects that a stack is close to allocating all available packet buffers it will take action to try and avoid packet buffer exhaustion. Onload will automatically start dropping packets on receive and, where possible, will reduce the receive descriptor ring fill level in an attempt to alleviate the situation. A ‘memory pressure’ condition can be identified using the onload_stackdump lots command where the pkt_bufs field will display the CRITICAL indicator. See Identifying Memory Pressure below.
Complete packet buffer exhaustion can result in deadlock. In an Onload stack, if all available packet buffers are allocated (for example currently queued in socket receive and send buffers) the stack is prevented from transmitting further data as there are no packet buffers available for the task.
If all available packet buffers are allocated then Onload will also fail to keep its adapters receive queues replenished. If the queues fall empty further data received by the adapter is instantly dropped. On a TCP connection packet buffers are used to hold unacknowledged data in the retransmit queue, and dropping received packets containing ACKs delays the freeing of these packet buffers back to Onload. Setting the value of EF_MIN_FREE_PACKETS=0 can result in a stack having no free packet buffers and this, in turn, can prevent the stack from shutting down cleanly.
Identifying Memory Pressure
The following extracts from the onload_stackdump command identify an Onload stack under memory pressure.
The EF_MAX_PACKETS value identifies the maximum number of packet buffers that can be used by the stack. EF_MAX_RX_PACKETS is the maximum number of packet buffers that can be used to hold packets received. EF_MAX_TX_PACKETS is the maximum number of packet buffers that can be used to hold packets to send. These two values are always less than EF_MAX_PACKETS to ensure that neither the transmit or receive paths can starve the other of packet buffers. Refer to Parameter Reference for detailed descriptions of these per stack variables.
The example Onload stack has the following default environment variable values:
EF_MAX_PACKETS: 32768
EF_MAX_RX_PACKETS: 24576
EF_MAX_TX_PACKETS: 24576The onload_stackdump lots command identifies packet buffer allocation and the onset of a memory pressure state:
pkt_bufs: size=2048 max=32768 alloc=24576 free=32 async=0 CRITICAL
pkt_bufs: rx=24544 rx_ring=9 rx_queued=24535There are potentially 32768 packet buffers available and the stack has allocated (used) 24576 packet buffers.
In the socket receive buffers there are 24544 packets buffers waiting to be
processed by the application. This is approaching the EF_MAX_RX_PACKETS limit and is
the reason the CRITICAL flag is present. The
Onload stack is under memory pressure. Only
nine packet buffers are available to the receive descriptor ring.
Onload will aim to keep the RX descriptor ring full at all times. If there are not enough available packet buffers to refill the RX descriptor ring this is indicated by the LOW memory pressure flag.
The onload_stackdump lots command will also identify the number of memory pressure events and number of packets dropped when Onload fails to allocate a packet buffer on the receive path.
memory_pressure_enter: 1
memory_pressure_drops: 22096The memory_pressure enter/exit counters count the number of times Onload enter/exits a state when it is trying to refill the receive queue (rxq) when the adapter runs out of packet buffers.
The Onload module option, max_packets_per_stack, places an upper limit on
EF_MAX_PACKETS. When max_packets_per_stack is not
set in the /etc/modprobe.d/<file>, its default value is
512K.