The Express datapath of the X4-series adapter does not support multicast replication between queues. However, replication is possible between a single Express datapath queue and multiple Enterprise datapath queues.
By default, the Express datapath of the X4-series adapter uses a dedicated queue for each application. This means that:
- The first application to attach to a given multicast stream on the Express
datapath is the only one that can do so.
The Express datapath delivers the multicast stream to the dedicated queue for that application.
- All other applications must attach to the multicast stream on the Enterprise
datapath.
The Enterprise datapath delivers the multicast stream to these applications using hardware replication. See Multicast Replication.
If you instead want multiple applications to receive the same multicast stream on the Express datapath, you must configure Onload to use shared readable user buffers (shrub). The multicast stream is then routed to a single shared buffer, from where applications can read it. The traffic is immediately available to all applications.
Buffer management is performed by a daemon known as a shrub controller.
To enable shrub, set the EF_SHRUB_CONFIGURE environment variable when starting Onload:
- The value of the variable gives the numerical id of the shrub controller to use.
- Onload ensures the shrub controller is running.
- Onload stacks that are using the same shrub controller all receive the same multicast traffic via the shared buffer.
- For best performance, ensure that Onload stacks that are using the same shrub controller are on the same Core Complex Die (CCD), and so have a shared cache.
You might want to know the following about how the shrub controller appears on your system:
- Onload starts the shrub controller as a daemon during stack creation (if needed), and stops it when no remaining stacks are using it.
- The shrub controller is named shrub_controller in your process list.
- The shrub controller creates some Unix domain sockets in
/run/onload/controller-<id>/:
- The shrub_config socket is created for configuration.
- A shrub-<n> socket is created per NIC for its associated buffer.