Pre-allocation of memory is advised to prevent latency jitter caused by the overheads of allocation and memory-mapping.
You can set the Onload EF_PREALLOC_PACKETS and EF_MAX_PACKETS environment variables to allocate packet buffers during stack creation. For example:
EF_PREALLOC_PACKETS=1 EF_MAX_PACKETS=64000 onload <myapplication>...Alternatively you can set the Onload EF_PREFAULT_PACKETS environment variable. This causes the user process to ‘touch’ the specified number of packet buffers when an Onload stack is created, so that memory for these packet buffers is pre-allocated and memory-mapped into the user-process address space.
When deciding how many packets to prefault, the user should look at the alloc value when the onload_stackdump packets command is run. The alloc value is a high water mark identifying the maximum the number of packets being used by the stack at any singular point. Setting EF_PREFAULT_PACKETS to at least this value is recommended.
onload_stackdump packets$ onload_stackdump packets
ci_netif_pkt_dump_all: id=0
pkt_sets: pkt_size=2048 set_size=1024 max=32 alloc=2
pkt_set[0]: free=544
pkt_set[1]: free=446 current
pkt_bufs: max=32768 alloc=2048 free=990 async=0
pkt_bufs: rx=1058 rx_ring=992 rx_queued=2 pressure_pool=64
pkt_bufs: tx=0 tx_ring=0 tx_oflow=0
pkt_bufs: in_loopback=0 in_sock=0
994: 0x200 Rx
n_zero_refs=1054 n_freepkts=1 estimated_free_nonb=1053
free_nonb=0 nonb_pkt_pool=ffffffffffffffffWhen deciding how many packets to prefault the user should consider that Onload must allocate from the EF_MAX_PACKETS pool, a number of packet buffers per receive ring per interface. Once these have been allocated, any remainder can be prefaulted.
Users who require to prefault the maximum possible number of available packets can set EF_PREFAULT_PACKETS and EF_MAX_PACKETS to the same value:
EF_PREFAULT_PACKETS=64000 EF_MAX_PACKETS=64000 onload <myapplication>...Refer to Parameter Reference for details of these variables.