Zero-Copy TCP Send

Onload User Guide (UG1586)

Document ID
UG1586
Release Date
2023-07-31
Revision
1.2 English

The zero-copy send API supports the sending of multiple messages to different sockets in a single call. Data buffers must be allocated in advance and for best efficiency these should be allocated in blocks and off the critical path. The user should avoid simply moving the copy from Onload into the application, but where this is unavoidable, it should also be done off the critical path.

Figure 1. Zero-Copy Send
int onload_zc_send(struct onload_zc_mmsg* msgs, int mlen, int flags);
Figure 2. Zero-Copy Allocate Buffers
int onload_zc_alloc_buffers(int fd,
                            struct onload_zc_iovec* iovecs,
                            int iovecs_len,
                            onload_zc_buffer_type_flags flags);
int onload_zc_release_buffers(int fd,
                              onload_zc_handle* bufs,
                              int bufs_len);

The onload_zc_send() function return value identifies how many of the onload_zc_mmsg array’s rc fields are set. Each onload_zc_mmsg.rc returns how many bytes (or error) were sent in for that message. Refer to the table below.

Table 1. onload_zc_send Return Values
rc = onload_zc_send()
rc < 0 application error calling onload_zc_send(). rc is set to the error code
rc == 0 should not happen
0 < rc <= n_msgs

rc is set to the number of messages whose status has been sent in mmsgs[i].rc.

rc == n_msgs is the normal case

rc = mmsg[i].rc
rc < 0 error sending this message. rc is set to the error code
rc >= 0 rc is set to the number of bytes that have been sent in this message. Compare to the message length to establish which buffers sent

Sent buffers are owned by Onload. Unsent buffers are owned by the application and must be freed or reused to avoid leaking.

Note: Buffers sent with the ONLOAD_MSG_WARM feature enabled are not actually sent buffers, ownership remains with the user who is responsible for freeing these buffers.