What’s New
New Features in OpenOnload-8.1.0
Linux Distribution Support
Drivers for X3522
Controlling Access to RX Queues on X3522
Hardware Filter IDs on X3522
IPv6 on X3522
Transmit Path Warming on X3522
Deprecation of RHEL 7
New Features in OpenOnload-8.0.2
Library Versioning and Backward Compatibility
Checksum Validation and Non-TCP/UDP with ef_vi on X3522
New Features in OpenOnload-8.0.1
New Features in OpenOnload-8.0.0
Linux Distribution Support
Packaging, Source and Licensing Changes
Use with X3-series Adapters
Installing for X3-series Adapters
Python 3 Required
Deprecation
New Features in OpenOnload-7.1.3
Linux Distribution Support
Deprecation
New Features in OpenOnload-7.1.2
Linux Distribution Support
XDP/eBPF Filtering
New ef_vi Event EF_EVENT_TYPE_RESET
Deprecation
New Features in OpenOnload-7.1.1
setuid/setgid
EF_TCP_COMBINE_SENDS_MODE
Linux Distribution Support
New Features in OpenOnload-7.1.0
Feature Activation
TCPDirect
System-level Interface Denylisting
Onload Remote Monitor Enhancement
eBPF/XDP
Modified Configuration Options
Linux Distribution Support
Deprecation
New Features in OpenOnload-7.0.0
RHEL 8.x Dependencies
Activation Files and Build Profiles
eXpress Data Path (XDP)
Equal Cost Multipath Routing
IPv6 Acceleration
Extensions API Timestamps
IPVLAN
Change History
X3 Low Latency Quickstart
Software Installation
Onload
TCPDirect
Netperf
Sfnettest
Test Setup
BIOS Settings
Pre-Test Configuration
Reference System Specification
Latency Tests
Layer 2 ef_vi Latency
TCPDirect Latency
Onload Latency with netperf
Onload Latency with sfnt-pingpong
Latency Test Results
Latency against Payload
Latency for UDP Payloads at 10 Gb
Latency for TCP Payloads at 10 Gb
Further Information
Overview
Contrasting with Conventional Networking
How Onload Increases Performance
Further Information
Installation
Onload Distributions
Cloud Build Profile
Onload LICENSE Files
Kubernetes Onload
Hardware and Software Supported Platforms
Supported Network Adapters
Supported Processors
Supported Operating Systems
Onload and Network Adapter Drivers
X3 Series Drivers
XtremeScale Drivers
Identifying and Removing Previously Installed Drivers
Removing an Existing Installation
Removing RPMs
Pre-install Notes
Building and Installing from a Tarball
Download and Untar Onload
Build and Install Onload
Cloud Build Profile
Load the Onload Drivers
Confirm Onload Installation
Building a Source RPM from a Tarball
Building and Installing from a DKMS Package
Install on RHEL
Install on Ubuntu
Building and Installing from a Source RPM
Build the RPMs
Install the Built RPMs
Load the Onload Drivers
Building and Installing from a Source DEB
Onload Kernel Modules
Configuring the Network Interfaces
Installing Netperf and sfnettest
Running Onload
Testing the Onload Installation
Applying an Onload Patch
Patching a Tarball Installation
Patching a Source RPM Installation
Kernel and OS Upgrades
Tuning Onload
System Tuning
Sysjitter
Timer (TSC) Stability
CPU Power Saving Mode
Spinning, Polling and Interrupts
Spinning (busy-wait)
Enabling Spinning
When to Use Spinning
Polling vs. Interrupts
Onload Deployment on NUMA Systems
Useful Commands
Driver Loading - NUMA Node
Memory Policy
Application Processing
Workqueues
Interrupts
Verification
Interrupt Handling for the sfc Driver
Default Behavior
Affinitizing RSS Channels to CPUs
Restrict RSS to Local NUMA Node
Restrict RSS Receive Queues
Interrupt Handling - Using Onload
Performance Jitter
Using Onload Tuning Profiles
Benchmark Testing
Application-Specific Tuning
Monitoring Using onload_stackdump
Worked Examples
Reducing Jitter from Page Faults
Processing at User-Level
As Few Interrupts as Possible
Eliminating Drops
Minimizing Lock Contention
Stack Contention - Deferred Work
Onload Functionality
Onload Transparency
Onload Stacks
Virtual Network Interface (VNIC)
Functional Overview
Onload with Mixed Network Adapters
Maximum Number of Network Interfaces
Allowlist and Denylist for Interfaces
Onload Accelerated Process IDs
File Descriptors, Stacks, and Sockets
System Calls Intercepted by Onload
Linux Sysctls
tcp_slow_start_after_idle
tcp_congestion_control
tcp_adv_win_scale
tcp_rmem
tcp_wmem
tcp_dsack
tcp_fack
tcp_sack
tcp_max_syn_backlog
tcp_synack_retries
Namespaces
User-space Control Plane Server
Onload Options for the Control Plane Server
Parameters for onload_cp_server
Changing Onload Control Plane Table Sizes
Changing Table Sizes for Onload-201710 and Later
Changing Table Sizes before Onload-201710
SO_BINDTODEVICE
Multiplexed I/O
Poll, ppoll
Select, pselect
Epoll
Wire Order Delivery
Example API for Wire Order Delivery
Stack Sharing
Application Clustering
eXpress Data Path (XDP)
OS Requirements
Advantages of XDP
Including eBPF with Onload
Onload Tools
eBPF Return Codes
Programmatic Access
BPF Statistics
Zero-Copy API
Debug and Logging
Timestamps
Introduction
Software Timestamps
TCP Streams
Interrupt Driven Applications
Spinning Applications
Software Timestamp Values
Software Timestamp Formats
Hardware Timestamps
Requirements
Hardware Timestamp Format
Received Packets
Transmitted Packets
Zeroed Timestamps
Synchronizing Time
Example Timestamping Applications
Building the Examples
Running the Examples
Setting the Adapter Clock Time
Order of Timestamps in the Example Applications
rx_timestamping Example
tx_timestamping Example
Example UDP Commands
Zeroed Timestamps
Onload and TCP
TCP Operation
TCP Handshake, SYN and SYNACK
TCP SYN Cookies
TCP Socket Options
TCP Level Options
TCP File Descriptor Control
TCP Congestion Control
Small Receive Window Size
TCP SACK
TCP QUICKACK
TCP Delayed ACK
TCP Dynamic ACK
Limit Duplicate ACK Rate
Limit Challenge ACK Rate
TCP Loopback Acceleration
TCP Striping
TCP Connection Reset on RTO
ONLOAD_MSG_WARM
Listen/Accept Sockets
Socket Caching
TCP Passive Socket Caching
TCP Active Socket Caching
Caching for Web Proxies
Caching Stackdump
Caching Requirements
Shared Local Ports
Scalable Filters
Scalable Filter Restrictions
Configuring Scalable Filters
Partitioning the NIC
Scalable Filters and Bonding
Transparent Reverse Proxy Modes
Restrictions
Example Configuration Settings
Transparent Reverse Proxy on Multiple CPUs
Performance in Lossy Network Environments
Tail-drop Probe
Early Retransmit (RFC 5827) Algorithm
SACK Improvements
Initial Sequence Number Caching
Urgent Data Processing
TIMEWAIT Assassination
Onload and UDP
UDP Operation
Socket Options
Source Specific Socket Options
Onload Sockets vs. Kernel Sockets
Send and Receive Paths for UDP Sockets
Fragmented UDP
User Level recvmmsg for UDP
User-Level sendmmsg for UDP
UDP sendfile
Multicast Replication
Multicast Operation and Stack Sharing
Multicast Transmit Using Different Onload Stacks
Multicast Transmit Sharing an Onload Stack
Multicast Receive to Onload or Kernel Stack
Multicast Receive and Multiple Sockets
Multicast Loopback
Hardware Multicast Loopback
IP_MULTICAST_ALL
Packet Buffers
Network Adapter Buffer Table Mode
Large Buffer Table Support
Huge Pages
Allocating Huge Pages
How Onload Uses Packet Buffers
Identifying Packet Buffer Requirements
Running Out of Packet Buffers
Controlling Onload Packet Buffer Use
Physical Addressing Mode
Programmed I/O
CTPIO
Capabilities
Requirements for CTPIO
CTPIO Modes
CTPIO Frame Length
Cost of CTPIO
Using CTPIO with Onload
Using CTPIO with TCPDirect
Using CTPIO with ef_vi
Latency Tests
CTPIO Timestamps
CTPIO Statistics
Interfaces
Bonding, Link Aggregation and Failover
Polling the Bonding Configuration
Teaming
VLANS
MACVLAN
IPVLAN
Accelerated pipe()
Onload and Virtualization
Overview
Onload and Linux KVM
Onload and NIC Partitioning
Onload in a Docker Container
Pre-Installation
Installation
MACVLAN Support
Create Onload Docker Image
Migration
Onload Docker Images
Copying Files Between Host and Container
Limitations
Introduction
Resources
Devices
Changes to Behavior
Multithreaded Applications Termination
Thread Cancellation
Packet Capture
Firewalls
Socket Visibility to System Tools
Signals
Onload and IP_MULTICAST_TTL
Source/Policy Based Routing
Routing Table Metrics
Multipath Routes
Reverse Path Filtering
SO_REUSEPORT
Thread Safe
Control of Duplicated Sockets
UDP Sockets shutdown()
SOF_TIMESTAMPING_OPT_ID
Limits to Acceleration
IP Fragmentation
Broadcast Traffic
IPv6 Traffic
IPv6 Kernel Support
Raw Sockets
Socketpair and UNIX Domain Sockets
UDP sendfile()
Statically Linked Applications
Local Port Address
Bonding, Link Aggregation
VLANs
Ethernet Bridge Configuration
TCP RTO During Overload Conditions
Packet Loss on the Transmission Path
TCP Packets with Unsupported Routing
Application Clustering
Duplicate IP or MAC Addresses
Known Issues with Epoll
Nested Epoll Sets
Timing Issues and Spinning
Configuration Issues
Mixed Adapters Sharing a Broadcast Domain
IGMP Operation and Multicast Process Priority
Dynamic Loading
Huge Pages with IPC Namespace
Huge Pages with Shared Stacks
Huge Page Size
Huge Pages and shmmni
Use of vfork() in Java 7 Applications
PIO Not Supported in KVM/ESXi
IP_MTU_DISCOVER Socket Option
Onload Change History
Mapping Onload Versions
Features
Environment Variables
Module Options
Adapter Net Drivers
Parameter Reference
Parameter List
EF_ACCEPTQ_MIN_BACKLOG
EF_ACCEPT_INHERIT_NONBLOCK
EF_AF_XDP_ZEROCOPY
EF_AUTO_FLOWLABELS
EF_BINDTODEVICE_HANDOVER
EF_BURST_CONTROL_LIMIT
EF_BUZZ_USEC
EF_CHALLENGE_ACK_LIMIT
EF_CLUSTER_HOT_RESTART
EF_CLUSTER_IGNORE
EF_CLUSTER_NAME
EF_CLUSTER_RESTART
EF_CLUSTER_SIZE
EF_COMPOUND_PAGES_MODE
EF_CONG_AVOID_SCALE_BACK
EF_CTPIO
EF_CTPIO_CT_THRESH
EF_CTPIO_MAX_FRAME_LEN
EF_CTPIO_MODE
EF_CTPIO_SWITCH_BYPASS
EF_DEFER_ARP_MAX
EF_DEFER_ARP_TIMEOUT
EF_DEFER_WORK_LIMIT
EF_DELACK_THRESH
EF_DONT_ACCELERATE
EF_DYNAMIC_ACK_THRESH
EF_ENDPOINT_PACKET_RESERVE
EF_EPOLL_CTL_FAST
EF_EPOLL_CTL_HANDOFF
EF_EPOLL_MT_SAFE
EF_EPOLL_SPIN
EF_EVS_PER_POLL
EF_FDS_MT_SAFE
EF_FDTABLE_SIZE
EF_FDTABLE_STRICT
EF_FORCE_SEND_MULTICAST
EF_FORCE_TCP_NODELAY
EF_FORK_NETIF
EF_FREE_PACKETS_LOW_WATERMARK
EF_HELPER_PRIME_USEC
EF_HELPER_USEC
EF_HIGH_THROUGHPUT_MODE
EF_ICMP_PKTS
EF_INTERFACE_BLACKLIST
EF_INTERFACE_WHITELIST
EF_INT_DRIVEN
EF_INT_REPRIME
EF_INVALID_ACK_RATELIMIT
EF_IRQ_CHANNEL
EF_IRQ_CORE
EF_KEEPALIVE_INTVL
EF_KEEPALIVE_PROBES
EF_KEEPALIVE_TIME
EF_KERNEL_PACKETS_BATCH_SIZE
EF_KERNEL_PACKETS_TIMER_USEC
EF_LOAD_ENV
EF_LOG
EF_LOG_FILE
EF_LOG_TIMESTAMPS
EF_LOG_VIA_IOCTL
EF_MAX_ENDPOINTS
EF_MAX_PACKETS
EF_MAX_RX_PACKETS
EF_MAX_TX_PACKETS
EF_MCAST_JOIN_BINDTODEVICE
EF_MCAST_JOIN_HANDOVER
EF_MCAST_RECV
EF_MCAST_RECV_HW_LOOP
EF_MCAST_SEND
EF_MIN_FREE_PACKETS
EF_MULTICAST_LOOP_OFF
EF_NAME
EF_NETIF_DTOR
EF_NONAGLE_INFLIGHT_MAX
EF_NO_FAIL
EF_ONLOAD_FD_BASE
EF_PACKET_BUFFER_MODE
EF_PERIODIC_TIMER_CPU
EF_PER_SOCKET_CACHE_MAX
EF_PIO
EF_PIO_THRESHOLD
EF_PIPE
EF_PIPE_RECV_SPIN
EF_PIPE_SEND_SPIN
EF_PIPE_SIZE
EF_PKT_WAIT_SPIN
EF_POLL_FAST
EF_POLL_FAST_USEC
EF_POLL_IN_KERNEL
EF_POLL_NONBLOCK_FAST_USEC
EF_POLL_ON_DEMAND
EF_POLL_SPIN
EF_POLL_USEC
EF_PREALLOC_PACKETS
EF_PREFAULT_PACKETS
EF_PROBE
EF_RETRANSMIT_THRESHOLD
EF_RETRANSMIT_THRESHOLD_ORPHAN
EF_RETRANSMIT_THRESHOLD_SYN
EF_RETRANSMIT_THRESHOLD_SYNACK
EF_RFC_RTO_INITIAL
EF_RFC_RTO_MAX
EF_RFC_RTO_MIN
EF_RXQ_LIMIT
EF_RXQ_MIN
EF_RXQ_SIZE
EF_RX_TIMESTAMPING
EF_RX_TIMESTAMPING_ORDERING
EF_SA_ONSTACK_INTERCEPT
EF_SCALABLE_ACTIVE_WILDS_NEED_FILTER
EF_SCALABLE_FILTERS
EF_SCALABLE_FILTERS_ENABLE
EF_SCALABLE_FILTERS_IFINDEX_ACTIVE
EF_SCALABLE_FILTERS_IFINDEX_PASSIVE
EF_SCALABLE_FILTERS_MODE
EF_SCALABLE_LISTEN_MODE
EF_SELECT_FAST
EF_SELECT_FAST_USEC
EF_SELECT_NONBLOCK_FAST_USEC
EF_SELECT_SPIN
EF_SEND_POLL_MAX_EVS
EF_SEND_POLL_THRESH
EF_SHARE_WITH
EF_SIGNALS_NOPOSTPONE
EF_SLEEP_SPIN_USEC
EF_SOCKET_CACHE_MAX
EF_SOCKET_CACHE_PORTS
EF_SOCK_LOCK_BUZZ
EF_SO_BUSY_POLL_SPIN
EF_SPIN_USEC
EF_STACK_LOCK_BUZZ
EF_STACK_PER_THREAD
EF_SYNC_CPLANE_AT_CREATE
EF_TAIL_DROP_PROBE
EF_TCP
EF_TCP_ACCEPT_SPIN
EF_TCP_ADV_WIN_SCALE_MAX
EF_TCP_BACKLOG_MAX
EF_TCP_CLIENT_LOOPBACK
EF_TCP_COMBINE_SENDS_MODE
EF_TCP_CONNECT_HANDOVER
EF_TCP_CONNECT_SPIN
EF_TCP_EARLY_RETRANSMIT
EF_TCP_FASTSTART_IDLE
EF_TCP_FASTSTART_INIT
EF_TCP_FASTSTART_LOSS
EF_TCP_FIN_TIMEOUT
EF_TCP_FORCE_REUSEPORT
EF_TCP_INITIAL_CWND
EF_TCP_ISN_2MSL
EF_TCP_ISN_CACHE_SIZE
EF_TCP_ISN_INCLUDE_PASSIVE
EF_TCP_ISN_MODE
EF_TCP_ISN_OFFSET
EF_TCP_LISTEN_HANDOVER
EF_TCP_LOSS_MIN_CWND
EF_TCP_MIN_CWND
EF_TCP_RCVBUF
EF_TCP_RCVBUF_ESTABLISHED_DEFAULT
EF_TCP_RCVBUF_MODE
EF_TCP_RCVBUF_STRICT
EF_TCP_RECV_SPIN
EF_TCP_RST_DELAYED_CONN
EF_TCP_RX_CHECKS
EF_TCP_RX_LOG_FLAGS
EF_TCP_SEND_NONBLOCK_NO_PACKETS_MODE
EF_TCP_SEND_SPIN
EF_TCP_SERVER_LOOPBACK
EF_TCP_SHARED_LOCAL_PORTS
EF_TCP_SHARED_LOCAL_PORTS_MAX
EF_TCP_SHARED_LOCAL_PORTS_NO_FALLBACK
EF_TCP_SHARED_LOCAL_PORTS_PER_IP
EF_TCP_SHARED_LOCAL_PORTS_PER_IP_MAX
EF_TCP_SHARED_LOCAL_PORTS_REUSE_FAST
EF_TCP_SHARED_LOCAL_PORTS_STEP
EF_TCP_SNDBUF
EF_TCP_SNDBUF_ESTABLISHED_DEFAULT
EF_TCP_SNDBUF_MODE
EF_TCP_SOCKBUF_MAX_FRACTION
EF_TCP_SYNCOOKIES
EF_TCP_SYNRECV_MAX
EF_TCP_SYN_OPTS
EF_TCP_TCONST_MSL
EF_TCP_TIME_WAIT_ASSASSINATION
EF_TCP_TSOPT_MODE
EF_TCP_URG_MODE
EF_TIMESTAMPING_REPORTING
EF_TXQ_SIZE
EF_TX_MIN_IPG_CNTL
EF_TX_PUSH
EF_TX_PUSH_THRESHOLD
EF_TX_QOS_CLASS
EF_TX_TIMESTAMPING
EF_UDP
EF_UDP_CONNECT_HANDOVER
EF_UDP_FORCE_REUSEPORT
EF_UDP_PORT_HANDOVER2_MAX
EF_UDP_PORT_HANDOVER2_MIN
EF_UDP_PORT_HANDOVER3_MAX
EF_UDP_PORT_HANDOVER3_MIN
EF_UDP_PORT_HANDOVER_MAX
EF_UDP_PORT_HANDOVER_MIN
EF_UDP_RCVBUF
EF_UDP_RECV_SPIN
EF_UDP_SEND_NONBLOCK_NO_PACKETS_MODE
EF_UDP_SEND_SPIN
EF_UDP_SEND_UNLOCKED
EF_UDP_SEND_UNLOCK_THRESH
EF_UDP_SNDBUF
EF_UL_EPOLL
EF_UL_POLL
EF_UL_SELECT
EF_UNCONFINE_SYN
EF_UNIX_LOG
EF_URG_RFC
EF_USE_DSACK
EF_USE_HUGE_PAGES
EF_VALIDATE_ENV
EF_VFORK_MODE
EF_WODA_SINGLE_INTERFACE
Meta Options
Environment Variables
EF_POLL_USEC
EF_BUZZ_USEC
Build Dependencies
General
Building Kernel Modules
onload
onload_tcpdump
solar_clusterd
onload_bpftools
IPv6 Support
Red Hat Enterprise Linux 8.x
Onload Extensions API
Common Components
onload_is_present
onload_fd_stat
onload_fd_check_feature
onload_thread_set_spin
onload_thread_get_spin
onload_socket_nonaccel
onload_socket_unicast_nonaccel
Stacks API
onload_set_stackname
onload_move_fd
onload_stackname_save
onload_stackname_restore
onload_stack_opt_set_int
onload_stack_opt_reset
onload_ordered_epoll_wait
onload_timestamping_request
Stacks API Examples
Zero-Copy API
Zero-Copy Data Buffers
Zero-Copy UDP Receive Overview
Zero-Copy UDP Receive
Zero-Copy Receive Example #1
Zero-Copy Receive Example #2
Zero-Copy TCP Send Overview
Zero-Copy TCP Send
Zero-Copy Send with Single Message and Buffer
Zero-Copy Send with Multiple Messages and Buffers
Zero-Copy Send Full Example
Receive Filtering API
Receive Filtering API
Receive Filtering Example
Templated Sends API
Description
MSG Template
MSG Update
MSG Allocation
MSG Template Update
MSG Template Abort
Delegated Sends API
Description
Performance
Standard Send vs. Delegated Send
Example Code
Run Client/Server
struct onload_delegated_send
onload_delegated_send_rc
onload_delegated_send_prepare
onload_delegated_send_tcp_update
onload_delegated_send_tcp_advance
onload_delegated_send_complete
onload_delegated_send_cancel
onload_stackdump
General Use
List Onloaded Processes
List Onloaded Threads, Priority and Affinity
List Onload Environment Variables
TX PIO Counters
Send RST on a TCP Socket
Removing Zombie and Orphan Stacks
Snapshot vs. Dynamic Views
Monitoring Receive and Transmit Packet Buffers
Packet Sets
TCP Application Statistics
The onload_stackdump LOTS Command.
TCP Stacks
TCP ESTABLISHED Connection Sockets
TCP LISTEN Sockets
UDP Sockets
Statistics
Environment Variables
Onload Stackdump Filters
Remote Monitoring
orm_webserver
orm_json
orm_json_lib
sfnettest
sfnt-pingpong
sfnt-stream
Running Without Spinning
onload_tcpdump
Building onload_tcpdump
Using onload_tcpdump
Examples
VLAN Examples
Dependencies
Limitations
Known Issues
SolarCapture
ef_vi
Components
Compiling and Linking
Documentation
onload_iptables
How it Works
Files
Features
Rules
Preview Firewall Rules
To Convert Linux iptables to Onload Firewall Rules
To View Rules for a Specific Interface
To Add a Rule for a Selected Interface
To Delete a Rule from a Selected Interface
Error Checking
Interface and Port
Virtual/Bonded Interface
Error Messages
eflatency
eflatency
To Run eflatency
Management Information Base
Host
Container
Namespaces
List Available Options
Tables
Usage
Version
hwport
llap
ipif
fwd
stats
internal
all
X2 Low Latency Quickstart
Software Installation
Firmware Variant
Onload
Netperf
Sfnettest
Test Setup
BIOS Settings
Pre-Test Configuration
Reference System Specification
Latency Tests
Layer 2 ef_vi Latency
TCPDirect Latency
Onload Latency with netperf
Onload Latency with sfnt-pingpong
Latency without CTPIO
Kernel Latency
Latency Test Results
Latency against Payload
Latency for UDP Payloads at 25 Gb
Latency for TCP Payloads at 25 Gb
Latency for UDP Payloads at 10 Gb
Latency for TCP Payloads at 10 Gb
Notes on Latency and Payload Graphs
Further Information
X2 Throughput Quickstart
Software Installation
Onload
Netperf
Wrk
Nginx
Test Setup
Reference System Specification
Throughput
Pre-test Configuration
Throughput with Onload
Throughput without Onload
HTTP connections
Pre-test Configuration
Connections with Onload
Connections without Onload
Further Information
Additional Resources and Legal Notices
Finding Additional Documentation
Support Resources
References
Revision History
Please Read: Important Legal Notices
Using multiple onload_tcpdump instances to capture from the same onload stack is not a supported configuration.
Currently onload_tcpdump
captures only packets from Onload stacks and not from kernel stacks.
onload_tcpdump
delivers timestamps with microsecond resolution. onload_tcpdump
does not support nanosecond precision.
The onload_tcpdump application monitors stack creation events and will attach to newly created stacks however, there is a short period (normally only a few milliseconds) between stack creation and the attachment during which packets sent/received will not be captured.