Installing and Deploying - 3.5 English

Vitis AI Library User Guide (UG1354)
Document ID
UG1354
Release Date
2023-06-29
Version
3.5 English
More than 10 deployment examples that are based on the ONNX Runtime are provided in Vitis AI 3.5. You can find the examples in the samples_onnx folder. To deploy the ONNX model using Vitis AI, follow these steps:
  1. Git clone the corresponding Vitis AI Library from https://github.com/Xilinx/Vitis-AI.
  2. Install the cross-compilation system on the host side. Refer to Installation for instructions.
  3. Prepare the quantized model in ONNX format. Use the Vitis AI Quantizer to quantize the model and output the quantized model in the ONNX format.
  4. Download the ONNX runtime package vitis_ai_2023.1-r3.5.0.tar.gz and install it on the target board. 
    tar -xzvf vitis_ai_2023.1-r3.5.0.tar.gz -C /

    Then, download the voe-0.1.0-py3-none-any.whl and onnxruntime_vitisai-1.16.0-py3-none-any.whl. Make sure the device is online and install them online.

    pip3 install voe*.whl
pip3 install onnxruntime_vitisai*.whl
  5. In Vitis AI 3.5, both ONNX Runtime C++ API and Python API are supported. For the details of ONNX Runtime API, refer to ONNX Runtime API docs. The following shows the ONNX model deployment code snippet based on the C++ API.
    // ...
#include <experimental_onnxruntime_cxx_api.h>
// include user header files
// ...

auto onnx_model_path = "resnet50_pt.onnx"
Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "resnet50_pt");
auto session_options = Ort::SessionOptions();

auto options = std::unorderd_map<std::string,std::string>({});
options["config_file"] = "/etc/vaip_config.json";
// optional, eg: cache path : /tmp/my_cache/abcdefg // Replace abcdefg with your model name, eg. onnx_model_md5
options["cacheDir"] = "/tmp/my_cache";
options["cacheKey"] = "abcdefg"; // Replace abcdefg with your model name, eg. onnx_model_md5

// Create an inference session using the Vitis AI execution provider
session_options.AppendExecutionProvider("VitisAI", options);

auto session = Ort::Experimental::Session(env, model_name, session_options);

auto input_shapes = session.GetInputShapes();
// preprocess input data
// ...

// Create input tensors and populate input data
std::vector<Ort::Value> input_tensors;
input_tensors.push_back(Ort::Experimental::Value::CreateTensor<float>(
input_data.data(), input_data.size(), input_shapes[0]));

auto output_tensors = session.Run(session.GetInputNames(), input_tensors,
session.GetOutputNames());
// postprocess output data
// ...
    To leverage the Python APIs, use the following example as a reference:
    import onnxruntime

# Add other imports
# ...

# Load inputs and do preprocessing
# ...

# Create an inference session using the Vitis-AI execution provider

session = onnxruntime.InferenceSession(
'[model_file].onnx',
providers=["VitisAIExecutionProvider"],
provider_options=[{"config_file":"/etc/vaip_config.json"}])

input_shape = session.get_inputs()[0].shape
input_name = session.get_inputs()[0].name

# Load inputs and do preprocessing by input_shape
input_data = [...]
result = session.run([], {input_name: input_data})
  6. Create a build.sh file as shown below, or copy one from the Vitis AI Library ONNX examples and modify it.
    result=0 && pkg-config --list-all | grep opencv4 && result=1
if [ $result -eq 1 ]; then
	OPENCV_FLAGS=$(pkg-config --cflags --libs-only-L opencv4)
else
	OPENCV_FLAGS=$(pkg-config --cflags --libs-only-L opencv)
fi

lib_x=" -lglog -lunilog -lvitis_ai_library-xnnpp -lvitis_ai_library-model_config -lprotobuf -lxrt_core -lvart-xrt-device-handle -lvaip-core -lxcompiler-core -labsl_city -labsl_low_level_hash -lvart-dpu-controller -lxir -lvart-util -ltarget-factory -ljson-c" 
lib_onnx=" -lonnxruntime" 
lib_opencv=" -lopencv_videoio -lopencv_imgcodecs -lopencv_highgui -lopencv_imgproc -lopencv_core " 
inc_x=" -I=/usr/include/onnxruntime -I=/install/Release/include/onnxruntime -I=/install/Release/include -I=/usr/include/xrt " 
link_x=" -L=/install/Release/lib" 

name=$(basename $PWD)

CXX=${CXX:-g++}
$CXX -O2 -fno-inline -I. \
 ${inc_x} \
 ${link_x} \
 -o ${name}_onnx -std=c++17 \
 $PWD/${name}_onnx.cpp \
 ${OPENCV_FLAGS} \
 ${lib_opencv} \
 ${lib_x} \
 ${lib_onnx}
  7. Cross-compile the program.
    sh -x build.sh
  8. Copy the executable program and the quantized ONNX model to the target board using the scp command.
  9. Execute the program on the target board. Before running the program, ensure that the target board has the Vitis AI Library installed, and prepare the images you want to test.
    ./resnet50_onnx <Onnx model> <image>
    Note: For the ONNX model deployment, the input model is the quantized ONNX model. If the environmental variable WITH_XCOMPILER is on, it will do the model compiling online first when you run the program. It can take some time to compile the model.