Verify that json parameters are accurate and correct as per compiled model. For example, verify values in cache-dir, cache-key.
If model is partially offloaded by VAIML compiler then rest of the ops are offloaded to CPU.
Check if any issue with model compilation.
The following warnings are emitted when cache-dir or cache-key issues are detected at pipeline startup:
a) cache-dir does not exist:
WARNING vvas_xinfer gstvvas_xinfer_helper.cpp:read_onnxrt_config:<vvas_xinfer0>
cache-dir <cache-dir> does not exist, inference may run on CPU instead of hardware accelerator
Possible fix - cache-dir in vitisai-ep-config must be a directory that exists on disk. Create it or correct the path:
"vitisai-ep-config": {
"cache-dir": "/etc/vai/models"
}
b) cache-key: no compiled model found at expected path:
WARNING vvas_xinfer gstvvas_xinfer_helper.cpp:read_onnxrt_config:<vvas_xinfer0>
cache-key <cache-key> may be incorrect: expected compiled model not found at <cache-dir>/<cache-key>/<cache-key>.rai,
verify cache-key in vitisai-ep-config, inference may run on CPU instead of hardware accelerator
Possible fix - cache-key must match the filename (without extension) of the compiled model. The plugin checks for <cache-dir>/<cache-key>/<cache-key>.rai:
"vitisai-ep-config": {
"cache-dir": "/etc/vai/models",
"cache-key": "ResNet50_INT8"
}