Onload User Guide (UG1586)

Document ID
Release Date
1.2 English

Applications that send data infrequently may see increased send latency compared to an application that is making frequent sends. This is due to the send path and associated data structures not being cache and TLB resident (which can occur even if the CPU has been otherwise idle since the previous send call).

Onload allows applications to repeatedly call send() to keep the TCP fast send path ‘warm’ in the cache without actually sending data. This is particularly useful for applications that only send infrequently and helps maintain low latency performance for those TCP connections that do not send often. These “fake” sends are performed by setting the ONLOAD_MSG_WARM flag when calling the TCP send calls. The message warm feature does not transmit any packets.

char buf[10];
send(fd, buf, 10, ONLOAD_MSG_WARM);

Onload stackdump supports counters to indicate the level of message warm use:

  • warm_aborted is a count of the number of times a message warm send function was called, but the sendpath was not exercised due to Onload locking constraints.
  • warm is a count of the number of times a message warm send function was called when the send path was exercised.
  • A send() ONLOAD_MSG_WARM can return 0 (length of data sent) if the send() was unsuccessful. This will be due to normal stack and TCP networking conditions such as cannot get stack lock, insufficient send window available, other packets in the send queue or retransmit queue etc.
Onload applications should not invoke send(MSG_WARM) and send(normal) from different threads on the same socket. This is not a supported feature.
The ONLOAD_MSG_WARM flag is an Onload feature. It can be applied to sockets created by Onload. However if sockets are subsequently handed off to the kernel - so they are not accelerated by Onload, it can cause the message warm packets to be actually sent. This is due to a limitation in some Linux distributions which appear to ignore this flag. The Onload extensions API can be used to check whether a socket supports the MSG_WARM feature via the onload_fd_check_feature() API (onload_fd_check_feature).
Note: When using the MSG_WARM feature, Onload does not attempt to split large packets into multiple segments and for this reason, the size of data passed to Onload when using the MSG_WARM feature must not exceed the MSS value.
Note: Onload versions earlier than 201310 do not support the ONLOAD_MSG_WARM socket flag, therefore setting the flag will cause message warm packets to be sent.