Huge Pages - UG1523

Alveo X3522 User Guide (UG1523)

Document ID
UG1523
Release Date
2024-10-18
Revision
1.5 English

You must provide some 2 MB huge pages for the X3522 network driver to use for buffering before you can use any networking other than the kernel stack. In particular, huge pages are required to run applications using the following network acceleration technologies:

  • Onload
  • TCPDirect
  • ef_vi.

Determine the Minimum Number of Huge Pages

To determine the minimum number of huge pages that are required:

  1. Determine the number of receive queues that are being used by finding the interrupts associated with them. The interrupt names matches the pattern <interface_name>-rx-<n>. For example (omitting other interrupts from the output):
    $ cat /proc/interrupts
     46:         51        346          0       1854   PCI-MSI 524288-edge      enp1s0f0np0-rx-0
     47:         71        200          2       1961   PCI-MSI 524320-edge      enp1s0f0np2-rx-0
     48:          0       2230          2          0   PCI-MSI 524289-edge      enp1s0f0np0-rx-1
     49:        280        128       1822          2   PCI-MSI 524321-edge      enp1s0f0np2-rx-1
     50:          0          0       2231          1   PCI-MSI 524290-edge      enp1s0f0np0-rx-2
     51:       2183          2         47          0   PCI-MSI 524322-edge      enp1s0f0np2-rx-2
     52:       2184          0          0         48   PCI-MSI 524291-edge      enp1s0f0np0-rx-3
     53:       2183          0          2         47   PCI-MSI 524323-edge      enp1s0f0np2-rx-3
     94:       1113          0         40       1081   PCI-MSI 526336-edge      enp1s0f1np1-rx-0
     95:       1233        561          0        440   PCI-MSI 526368-edge      enp1s0f1np3-rx-0
     96:        342       1209        681          0   PCI-MSI 526337-edge      enp1s0f1np1-rx-1
     97:        560        367        883        422   PCI-MSI 526369-edge      enp1s0f1np3-rx-1
     98:          0          0       1151       1081   PCI-MSI 526338-edge      enp1s0f1np1-rx-2
     99:        342          0        408       1482   PCI-MSI 526370-edge      enp1s0f1np3-rx-2
    100:        921       1000        142        169   PCI-MSI 526339-edge      enp1s0f1np1-rx-3
    101:        343        400          0       1489   PCI-MSI 526371-edge      enp1s0f1np3-rx-3
    In this example there are four receive queues per interface, for a total of 16 receive queues.
  2. Identify the current receive ring buffer size for each interface (in packet buffers):
    $ ethtool -g <interface>
    For example:
    $ ethtool -g enp1s0f0np0
    Ring parameters for enp1s0f0np0:
    Pre-set maximums:
    RX:             6144
    RX Mini:        0
    RX Jumbo:       0
    TX:             512
    Current hardware settings:
    RX:             2048
    RX Mini:        0
    RX Jumbo:       0
    TX:             512
    In this example, the receive ring buffer size is 2048 packet buffers.
  3. Calculate the minimum number of huge pages that are required:
    • Each receive queue requires the following minimum number of huge pages:

      ROUNDUP (rx_ring_buffer_size ÷ 1024) + 1

    • Assuming the receive ring buffer size is the same for all interfaces, you can use the following formula:

      <num_rx_queues> × (ROUNDUP (rx_ring_buffer_size ÷ 1024) + 1)

    So for this example, you would require a minimum of:

    16 × (ROUNDUP (2048 ÷ 1024) + 1)

    which is 48 huge pages.

    You might later increase this value when tuning your application for best performance.

Check the Current Huge Page Size

Check the current default huge page size by inspection of /proc/meminfo:

# cat /proc/meminfo | grep Hugepagesize
Hugepagesize:       2048 kB

The method you use to allocate huge pages differs depending on the huge page size.

If the Huge Page Size is 2048 KB

If the huge page size is 2048 KB, allocate huge pages as follows:

  1. Check the current huge page allocation by inspection of /proc/meminfo:
    # cat /proc/meminfo | grep Huge
    AnonHugePages:    2048 kB
    ShmemHugePages:        0 kB
    HugePages_Total:       0
    HugePages_Free:        0
    HugePages_Rsvd:        0
    HugePages_Surp:        0
    Hugepagesize:       2048 kB
    Hugetlb:               0 kB

    In the output:

    • HugePages_Total is the total number of huge pages available on the system.
    • HugePages_Free is the number of huge pages that are free on the system.
  2. If too few huge pages are free, allocate some extra huge pages. For example, to configure a total of 1024 huge pages:
    # sysctl -w vm.nr_hugepages=1024
  3. To make this change persistent, update /etc/sysctl.conf:
    # echo "vm.nr_hugepages = 1024" >> /etc/sysctl.conf

If the Huge Page Size is not 2048 KB

If the huge page size is not 2048 KB, instead allocate huge pages as follows:

  1. Check the current 2048 KB huge page allocation by inspection of the files in /sys/kernel/mm/hugepages/hugepages-2048kB:
    # sh -c 'cd /sys/kernel/mm/hugepages/hugepages-2048kB; grep "" *'
    free_hugepages:0
    nr_hugepages:0
    nr_hugepages_mempolicy:0
    nr_overcommit_hugepages:0
    resv_hugepages:0
    surplus_hugepages:0

    In the output:

    • nr_hugepages is the total number of 2048 KB huge pages available on the system.
    • free_hugepages is the number of 2048 KB huge pages that are free on the system.
  2. If too few 2048 KB huge pages are free, allocate some extra huge pages. For example, to configure a total of 1024 huge pages:
    # echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
  3. To make this change persistent, add the following to the kernel parameters for your system:
    hugepagesz=2M hugepages=1024