You must provide some 2 MB huge pages for the X3522 network driver to use for buffering before you can use any networking other than the kernel stack. In particular, huge pages are required to run applications using the following network acceleration technologies:
- Onload
- TCPDirect
- ef_vi.
Determine the Minimum Number of Huge Pages
To determine the minimum number of huge pages that are required:
- Determine the number of receive queues that are being used by finding the
interrupts associated with them. The interrupt names matches the pattern
<interface_name>-rx-<n>
. For example (omitting other interrupts from the output):
In this example there are four receive queues per interface, for a total of 16 receive queues.$ cat /proc/interrupts 46: 51 346 0 1854 PCI-MSI 524288-edge enp1s0f0np0-rx-0 47: 71 200 2 1961 PCI-MSI 524320-edge enp1s0f0np2-rx-0 48: 0 2230 2 0 PCI-MSI 524289-edge enp1s0f0np0-rx-1 49: 280 128 1822 2 PCI-MSI 524321-edge enp1s0f0np2-rx-1 50: 0 0 2231 1 PCI-MSI 524290-edge enp1s0f0np0-rx-2 51: 2183 2 47 0 PCI-MSI 524322-edge enp1s0f0np2-rx-2 52: 2184 0 0 48 PCI-MSI 524291-edge enp1s0f0np0-rx-3 53: 2183 0 2 47 PCI-MSI 524323-edge enp1s0f0np2-rx-3 94: 1113 0 40 1081 PCI-MSI 526336-edge enp1s0f1np1-rx-0 95: 1233 561 0 440 PCI-MSI 526368-edge enp1s0f1np3-rx-0 96: 342 1209 681 0 PCI-MSI 526337-edge enp1s0f1np1-rx-1 97: 560 367 883 422 PCI-MSI 526369-edge enp1s0f1np3-rx-1 98: 0 0 1151 1081 PCI-MSI 526338-edge enp1s0f1np1-rx-2 99: 342 0 408 1482 PCI-MSI 526370-edge enp1s0f1np3-rx-2 100: 921 1000 142 169 PCI-MSI 526339-edge enp1s0f1np1-rx-3 101: 343 400 0 1489 PCI-MSI 526371-edge enp1s0f1np3-rx-3
- Identify the current receive ring buffer size for each interface (in packet
buffers):
For example:$ ethtool -g <interface>
In this example, the receive ring buffer size is 2048 packet buffers.$ ethtool -g enp1s0f0np0 Ring parameters for enp1s0f0np0: Pre-set maximums: RX: 6144 RX Mini: 0 RX Jumbo: 0 TX: 512 Current hardware settings: RX: 2048 RX Mini: 0 RX Jumbo: 0 TX: 512
- Calculate the minimum number of huge pages that are required:
- Each receive queue requires the following minimum number of huge
pages:
ROUNDUP (rx_ring_buffer_size ÷ 1024) + 1
- Assuming the receive ring buffer size is the same for all interfaces,
you can use the following formula:
<num_rx_queues> × (ROUNDUP (rx_ring_buffer_size ÷ 1024) + 1)
So for this example, you would require a minimum of:
16 × (ROUNDUP (2048 ÷ 1024) + 1)
which is 48 huge pages.
You might later increase this value when tuning your application for best performance.
- Each receive queue requires the following minimum number of huge
pages:
Check the Current Huge Page Size
Check the current default huge page size by inspection of /proc/meminfo:
# cat /proc/meminfo | grep Hugepagesize
Hugepagesize: 2048 kB
The method you use to allocate huge pages differs depending on the huge page size.
If the Huge Page Size is 2048 KB
If the huge page size is 2048 KB, allocate huge pages as follows:
- Check the current huge page allocation by inspection of
/proc/meminfo:
# cat /proc/meminfo | grep Huge AnonHugePages: 2048 kB ShmemHugePages: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB
In the output:
-
HugePages_Total
is the total number of huge pages available on the system. -
HugePages_Free
is the number of huge pages that are free on the system.
-
- If too few huge pages are free, allocate some extra huge pages. For example,
to configure a total of 1024 huge
pages:
# sysctl -w vm.nr_hugepages=1024
- To make this change persistent, update /etc/sysctl.conf:
# echo "vm.nr_hugepages = 1024" >> /etc/sysctl.conf
If the Huge Page Size is not 2048 KB
If the huge page size is not 2048 KB, instead allocate huge pages as follows:
- Check the current 2048 KB huge page allocation by inspection of the files in
/sys/kernel/mm/hugepages/hugepages-2048kB:
# sh -c 'cd /sys/kernel/mm/hugepages/hugepages-2048kB; grep "" *' free_hugepages:0 nr_hugepages:0 nr_hugepages_mempolicy:0 nr_overcommit_hugepages:0 resv_hugepages:0 surplus_hugepages:0
In the output:
-
nr_hugepages
is the total number of 2048 KB huge pages available on the system. -
free_hugepages
is the number of 2048 KB huge pages that are free on the system.
-
- If too few 2048 KB huge pages are free, allocate some extra huge pages. For
example, to configure a total of 1024 huge
pages:
# echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
- To make this change persistent, add the following to the kernel
parameters for your
system:
hugepagesz=2M hugepages=1024