Batch Processing Support - Batch Processing Support - 6.3 English - XD056

The vvas_xinfer plug-in supports batch processing mode exclusively with the VART backend runtime to improve throughput by processing multiple frames in a single inference call. This feature is not available with ONNX Runtime backend.

Key Features:

• Automatic Detection: The plug-in automatically detects the batch mode configuration from the compiled VART model during initialization. No explicit configuration is required from you.

• Split Batch Mode: The plug-in operates in split batch mode, where individual pre-processed tensors are gathered and sent for inference once the batch size (N) is reached. There is no need to create a single concatenated tensor for the entire batch.

• Partial Batch Processing: The plug-in intelligently handles partial batches in the following scenarios:

  • End-of-Stream (EOS): When EOS is received and the number of accumulated frames is less than the batch size, the plug-in processes the partial batch to ensure all frames are inferenced.

  • Timeout: If frames do not accumulate to the full batch size within a specified timeout period, the plug-in processes the available frames as a partial batch to maintain pipeline responsiveness. The timeout duration can be configured via the batch-timeout property (in milliseconds).

How It Works:

1. The plug-in accumulates pre-processed input tensors until the batch size is reached.

2. Once N tensors are gathered, they are sent together for a single inference operation.

3. Inference results are then distributed back to their corresponding input frames.

4. In case of EOS or timeout, any remaining frames (less than batch size) are processed immediately.

Example Pipeline with Batch Processing:

gst-launch-1.0 filesrc location=<INPUT>.nv12 ! \
 rawvideoparse width=<INPUT_WIDTH> height=<INPUT_HEIGHT> format=nv12 ! \
 vvas_xinfer config-file=/etc/vvas/configs/infer/resnet50_vart.json ! \
 fakesink -v

In this example, if the resnet50 model is compiled with batch N, the plug-in will automatically accumulate N frames, perform inference on all of them together, and then output the results. Users need to ensure that the model is compiled with the desired batch size using Vitis AI compiler tools, as the plug-in relies on the batch size specified in the compiled model.