Understand Inference Backend Runtimes - Understand Inference Backend Runtimes - 6.3 English - XD056

vvas_xinfer supports two inference runtime families. The inference-backend-runtime field selects the family; for ONNX Runtime, the onnxrt-ep field selects whether inference runs on the NPU (Vitis AI EP) or on the CPU (CPU EP). If inference-backend-runtime is not set, vvas_xinfer defaults to onnxrt. If onnxrt-ep is not set, the plugin defaults to the CPU EP.

For best throughput, use vart when a VART model is available. onnxrt with Vitis AI EP is also hardware accelerated and is appropriate when the deployment artifact is an ONNX model. The main performance risk is accidentally using the CPU EP (for example, omitting onnxrt-ep or vitisai-ep-config), not choosing onnxrt itself.

Example vart configuration:

{
  "infer-config": {
    "inference-backend-runtime": "vart",
    "vart-config": {
      "vart-model-path": "/etc/vai/models/ResNet50/ResNet50.rai"
    }
  }
}

Example onnxrt configuration with Vitis AI EP:

{
  "infer-config": {
    "inference-backend-runtime": "onnxrt",
    "onnxrt-config": {
      "onnx-model-path": "/etc/vai/models/ResNet50_INT8/ResNet50_INT8.onnx",
      "input-tensor-layout": "NCHW",
      "onnxrt-ep": "vitisai",
      "vitisai-ep-config": {
        "config-file-path": "/etc/vai/models/ResNet50_INT8/vitisai_config.json",
        "cache-dir": "/etc/vai/models",
        "cache-key": "ResNet50_INT8"
      }
    }
  }
}

Check the backend field in the profiler JSON to verify the selected runtime. For ONNX Runtime, the profiler reports onnxruntime:vitisai-ep when the Vitis AI EP is active and onnxruntime:cpu-ep when the CPU EP is active.