Zero-Copy UDP Receive

Onload User Guide (UG1586)
Document ID
UG1586
Release Date
2023-07-31
Revision
1.2 English

The onload_zc_recv() function specifies a callback to invoke for each received UDP datagram. The callback is invoked in the context of the call to onload_zc_recv() (it blocks/spins waiting for data).

Before calling, the application must set the following in the struct onload_zc_recv_args:

Table 1. onload_zc_recv_args Fields
Field Value 
cb
 Set to the callback function pointer. 
user_ptr
 Set to point to application state, this is not touched by Onload 
msg.msghdr.msg_control
msg_controllen
msg_name
msg_namelen
 The user application should set these to appropriate buffers and lengths (if required) as you would for recvmsg (or NULL and 0 if not used). 
flags
 Set to indicate behavior (for example ONLOAD_MSG_DONTWAIT).
Figure 1. Zero-Copy recv_args 
typedef enum onload_zc_callback_rc
(*onload_zc_recv_callback)(struct onload_zc_recv_args *args, int flags);
struct onload_zc_recv_args
{
  struct onload_zc_msg msg;
  onload_zc_recv_callback cb;
  void* user_ptr;
  int flags;
};
int onload_zc_recv(int fd, struct onload_zc_recv_args *args);

The callback gets to examine the data, and can control what happens next:

  • whether or not the buffer(s) are kept by the callback or are immediately freed by Onload
  • whether or not onload_zc_recv() will internally loop and invoke the callback with the next datagram, or immediately return to the application.

The next action is determined by setting flags in the return code as follows:

Table 2. Flags for Setting in the Return Code
Flag Description 
ONLOAD_ZC_KEEP
 The callback function can elect to retain ownership of received buffer(s) by returning ONLOAD_ZC_KEEP. Following this, the correct way to release retained buffers is to call onload_zc_release_buffers() to explicitly release the first buffer from each received datagram. Subsequent buffers pertaining to the same datagram will then be automatically released. 
ONLOAD_ZC_CONTINUE
 To suggest that Onload should loop and process more datagrams 
ONLOAD_ZC_TERMINATE
 To insist that Onload immediately return from the onload_zc_recv()

Flags can also be set by Onload:

Table 3. Flags That Can Be Set By Onload
Flag Description 
ONLOAD_ZC_END_OF_BURST
 Onload sets this flag to indicate that this is the last packet 
ONLOAD_ZC_MSG_SHARED
 Packet buffers are read only

If there is unaccelerated data on the socket from the kernel’s receive path this cannot be handled without copying. The application has two choices as follows:

Table 4. Flags for Handling Unaccelerated Data
Flag Description 
ONLOAD_MSG_RECV_OS_INLINE
 Set this flag when calling onload_zc_recv(). Onload will deal with the kernel data internally and pass it to the callback
check return code Check the return code from onload_zc_recv(). If it returns ENOTEMPTY then the application must call onload_recvmsg_kernel() to retrieve the kernel data.