Deployment - Deployment - 3.0 English - UG1414

Vitis AI User Guide (UG1414)

Document_ID
UG1414
Release_Date
2023-02-24
Version
3.0 English

This section will introduce how to deploy the tensorflow2 model with custom op in graph_runner APIs. For graph_runner APIs, it supports both C++ and Python. For C++ example, refer to Vitis-AI/examples/custom_operator/tensorflow2_example/deployment/cpp . For Python example, refer to Vitis-AI/examples/custom_operator/tensorflow2_example/deployment/python.

Tips: You can create a new folder for your application, then all your code and build files can be placed under this folder.

Code

  1. Create cpp source file, such as tf2_custom_op_graph_runner.cpp
  2. Include the header: <vitis/ai/graph_runner.hpp>
  3. Implement your model's pre-process, post-process and result-display-process in the following functions.
    static void preprocess_tf2_custom_op(…)
    static void postprocess_tf2_custom_op(…)
    static void print_result(…)

    Tips: You can copy the cpp source file from tf2_custom_layer_graph_runner.cpp and focus on the step 3 to implement the above 3 functions.

    The following shows the flow of tf2_custom_op_graph_runner.cpp

    Figure 1. The work flow of tf2_custom_op_graph_runner
    Part of the code is shown below.
      // create graph runner
      auto graph = xir::Graph::deserialize(xmodel_file);
      auto attrs = xir::Attrs::create();
      auto runner =
          vitis::ai::GraphRunner::create_graph_runner(graph.get(), attrs.get());
      CHECK(runner != nullptr);
    
      // get input/output tensor buffers
      auto input_tensor_buffers = runner->get_inputs();
      auto output_tensor_buffers = runner->get_outputs();
    
      // preprocess
      preprocess_tf2_custom_op(image_file, input_tensor_buffers);
    
      // sync input tensor buffers
      for (auto& input : input_tensor_buffers) {
        input->sync_for_write(0, input->get_tensor()->get_data_size() /
                                     input->get_tensor()->get_shape()[0]);
      }
    
      // run graph runner
      auto v = runner->execute_async(input_tensor_buffers, output_tensor_buffers);
      auto status = runner->wait((int)v.first, -1);
      CHECK_EQ(status, 0) << "failed to run the graph";
    
      // sync output tensor buffers
      for (auto output : output_tensor_buffers) {
        output->sync_for_read(0, output->get_tensor()->get_data_size() /
                                     output->get_tensor()->get_shape()[0]);
      }
    
      // postprocess
      postprocess_tf2_custom_op(output_tensor_buffers);

Build

  1. Create build.sh to build the code, as shown below. You can also refer to Vitis-AI/examples/custom_operator/tensorflow2_example/deployment/cpp/build.sh , as shown below.
    result=0 && pkg-config --list-all | grep opencv4 && result=1
    if [ $result -eq 1 ]; then
        OPENCV_FLAGS=$(pkg-config --cflags --libs-only-L opencv4)
    else
        OPENCV_FLAGS=$(pkg-config --cflags --libs-only-L opencv)
    fi
    
    CXX=${CXX:-g++}
    $CXX -std=c++17 -O2 -I. \
        -o tf2_custom_op_graph_runner \
        tf2_custom_op_graph_runner.cpp \
        -lglog \
        -lxir \
        -lvart-runner \
        -lvitis_ai_library-graph_runner \
        ${OPENCV_FLAGS} \
        -lopencv_core \
        -lopencv_imgcodecs \
        -lopencv_imgproc
  2. Execute bash build.sh to build the program on the target and the executable program tf2_custom_op_graph_runner will be generated.

Run

Before you run the demo, make sure the environment of the board has been set up correctly. Also, make sure the following files are generated or ready. Then copy them to the target.
  • compiled model, such as tf2_custom_op.xmodel
  • custom op library, such as libvart_op_imp_Mylayer.so
  • test image from here
  • executable program, such as tf2_custom_op_graph_runner
Copy the custom op library to /usr/lib on the target. Then, run the following command to test the model on the target.
./tf2_custom_op_graph_runner tf2_custom_op.xmodel sample.jpg

The following figure shows the result of running tf2_custom_op_graph_runner.

Figure 2. tf2_custom_op_graph_runner example