Temporal inference refers to the process of dynamically swapping out models running on NPU and is required when ML compute requirements (combined for all tenants) are greater than available NPU resources. This limitation highlights the intricacies involved in managing model execution within the constraints of the NPU, emphasizing the need for efficient resource allocation and prioritization during the inference process.
Note
Temporal sharing can be controlled from user space when using vart runtime for model execution.
To enable temporal sharing, set the vart-config property aie-columns-sharing to true and start-column to same value for both models.
The following Gstreamer pipelines demonstrate the temporal sharing of two ResNet50 models. Each command is executed in the background for 1000 frames.
{
"vart-config" : {
...
"aie-columns-sharing": true,
"start-column": 0,
...
}
}
Note
The following JSON examples are only samples. Ensure model-path is set to point to your available model before running.
gst-launch-1.0 \
multifilesrc location=<INPUT>.nv12 loop=true ! \
rawvideoparse format=nv12 width=<INPUT_WIDTH> height=<INPUT_HEIGHT> ! queue ! \
vvas_xinfer config-file=/etc/vvas/configs/infer/resnet50_vart.json ! queue ! \
vvas_xmetaconvert config-location=/etc/vvas/configs/metaconvert/metaconvert_config.json ! vvas_xoverlay ! queue ! identity eos-after=1001 ! \
fakesink -v &
gst-launch-1.0 \
multifilesrc location=<INPUT>.nv12 loop=true ! \
rawvideoparse format=nv12 width=<INPUT_WIDTH> height=<INPUT_HEIGHT> ! queue ! \
vvas_xinfer config-file=/etc/vvas/configs/infer/resnet50_vart.json ! queue ! \
vvas_xmetaconvert config-location=/etc/vvas/configs/metaconvert/metaconvert_config.json ! vvas_xoverlay ! queue ! identity eos-after=1001 ! \
fakesink -v &
To cross-verify the temporal sharing on the NPU, ensure the GStreamer commands are active and run:
xrt-smi examine -d 0 -r aie-partitions
xrt-smi examine -d 0 -r all # includes all the reported parameters
If needed, you can increase the eos-after value to keep them running for longer time.
The following is the sample output from the xrt-smi examine -d 0 -r aie-partitions command and we can see that same partition index is used for both instances.