Deployment - 3.5 English

Vitis AI User Guide (UG1414)

Document ID
Release Date
3.5 English

This section describes deploying the PyTorch model with custom op in graph_runner APIs. graph_runner APIs support both C++ and Python. You can refer to the graph_runner samples in Vitis-AI/examples/vai_library/samples/graph_runner:

  1. Create a new directory for your test code.
  2. Create a source file and implement the following functions for this sample:
    1. Parameter parse and initialize
    2. Preprocess
    3. Model_run
    4. Postprocess
    The basic flow of this sample is shown in the following image.
    Figure 1. The basic flow of this sample
  3. Build the program.
    1. If your project is relatively simpler, consisting of a single or few .cpp source files, you can easily set up the build process by duplicating any existing script from Vitis-AI/examples/vai_library/samples/graph_runner and adapting it to your specific needs. After you have customized the script, execute the following command to build your program:
      cd <your sample folder>
      The following figure shows the of the resnet50_graph_runner sample.
    2. If your project is more complex, such as this sample, using CMakeLists.txt for easy compiling is better. For more details about CMakeLists.txt, refer to Vitis-AI/examples/custom_operator/pytorch_example/deployment/cpp/pointpillars_graph_runner/CMakeLists.txt
      Then, run the following command to build the program:
      cd <your sample folder>
      mkdir build
      cd build
      cmake ..

    After a successful compilation, the executable program sample_pointpillars_graph_runner will be generated under <your sample folder>

  4. Test the program.
    Before you test the program, copy the XMODEL, test image, custom op library and the sample_pointpillars_graph_runner executable program to the board. Place the custom op library in /usr/lib. Then, run the following command to do the test:
    ./sample_pointpillars_graph_runner ./pointpillars_full_customer_op.xmodel sample_pointpillars.bin

    The following is a running result snapshot of this sample:

    root@xilinx-zcu102-2021_2:~/pointpillars_graph_runner# ./sample_pointpillars_graph_runner pointpillars_op.xmodel sample_pointpillars.bin
    WARNING: Logging before InitGoogleLogging() is written to STDERR
    W1202 05:59:20.517452 1307 tool_function.cpp:177] [UNILOG][WARNING] The operator named VoxelNet__VoxelNet_input_4, type: PPScatterV2, is not defined in XIR. XIR creates the definition of this operator automatically. You should specify the shape and the data_type of the output tensor of this operation by set_attr("shape", std::vector) and set_attr("data_type", std::string)
    result: 0
    0 18.541065 3.999999 -1.732742 1.703191 4.419279 1.465484 1.679375 0.880797
    0 34.522400 1.505865 -1.515198 1.503061 3.550991 1.420396 1.710625 0.851953
    0 10.917599 4.705865 -1.622433 1.650789 4.350764 1.634866 1.632500 0.851953
    1 21.338514 -2.400001 -1.681677 0.600000 1.963422 1.784916 4.742843 0.777300
    0 57.891731 -4.188268 -1.536627 1.575194 3.780010 1.512004 2.007500 0.679179
If you want to profile the sample of custom op, use the DEEPHI_PROFILING=1 environment variable:
env DEEPHI_PROFILING=1 ./sample_pointpillars_graph_runner ./pointpillars_full_customer_op.xmodel sample_pointpillars.bin
The profiling result is as follows:
I1130 01:29:53.038476 15571 cpu_task.cpp:163] CPU_UPDATE_INPUT : 5us
I1130 01:29:53.038684 15571 cpu_task.cpp:166] CPU_UPDATE_OUTPUT : 55us
I1130 01:29:53.038872 15571 cpu_task.cpp:169] CPU_SYNC_FOR_READ : 46us
I1130 01:29:53.039050 15571 cpu_task.cpp:181] CPU_OP_EXEC : 32us
I1130 01:29:53.039232 15571 cpu_task.cpp:181] CPU_OP_EXEC : 36us
I1130 01:29:53.039597 15571 cpu_task.cpp:181] CPU_OP_EXEC : 232us
I1130 01:29:53.066352 15571 cpu_task.cpp:181] CPU_OP_EXEC : 26575us
I1130 01:29:53.066745 15571 cpu_task.cpp:195] CPU_SYNC_FOR_WRITE : 1us