This section will introduce how to deploy the tensorflow2 model with custom op in graph_runner APIs. For graph_runner APIs, it supports both C++ and Python. For C++ example, refer to Vitis-AI/examples/custom_operator/tensorflow2_example/deployment/cpp . For Python example, refer to Vitis-AI/examples/custom_operator/tensorflow2_example/deployment/python.
Tips: You can create a new folder for your application, then all your code and build files can be placed under this folder.
Code
- Create cpp source file, such as tf2_custom_op_graph_runner.cpp
- Include the header: <vitis/ai/graph_runner.hpp>
- Implement your model's pre-process, post-process and result-display-process
in the following functions.
static void preprocess_tf2_custom_op(…) static void postprocess_tf2_custom_op(…) static void print_result(…)Tips: You can copy the cpp source file from tf2_custom_layer_graph_runner.cpp and focus on the step 3 to implement the above 3 functions.
The following shows the flow of tf2_custom_op_graph_runner.cpp
Figure 1. The work flow of tf2_custom_op_graph_runnerPart of the code is shown below.// create graph runner auto graph = xir::Graph::deserialize(xmodel_file); auto attrs = xir::Attrs::create(); auto runner = vitis::ai::GraphRunner::create_graph_runner(graph.get(), attrs.get()); CHECK(runner != nullptr); // get input/output tensor buffers auto input_tensor_buffers = runner->get_inputs(); auto output_tensor_buffers = runner->get_outputs(); // preprocess preprocess_tf2_custom_op(image_file, input_tensor_buffers); // sync input tensor buffers for (auto& input : input_tensor_buffers) { input->sync_for_write(0, input->get_tensor()->get_data_size() / input->get_tensor()->get_shape()[0]); } // run graph runner auto v = runner->execute_async(input_tensor_buffers, output_tensor_buffers); auto status = runner->wait((int)v.first, -1); CHECK_EQ(status, 0) << "failed to run the graph"; // sync output tensor buffers for (auto output : output_tensor_buffers) { output->sync_for_read(0, output->get_tensor()->get_data_size() / output->get_tensor()->get_shape()[0]); } // postprocess postprocess_tf2_custom_op(output_tensor_buffers);
Build
- Create build.sh to build the code, as
shown below. You can also refer to Vitis-AI/examples/custom_operator/tensorflow2_example/deployment/cpp/build.sh
, as shown
below.
result=0 && pkg-config --list-all | grep opencv4 && result=1 if [ $result -eq 1 ]; then OPENCV_FLAGS=$(pkg-config --cflags --libs-only-L opencv4) else OPENCV_FLAGS=$(pkg-config --cflags --libs-only-L opencv) fi CXX=${CXX:-g++} $CXX -std=c++17 -O2 -I. \ -o tf2_custom_op_graph_runner \ tf2_custom_op_graph_runner.cpp \ -lglog \ -lxir \ -lvart-runner \ -lvitis_ai_library-graph_runner \ ${OPENCV_FLAGS} \ -lopencv_core \ -lopencv_imgcodecs \ -lopencv_imgproc - Execute bash build.sh to build the program on the target and the executable program tf2_custom_op_graph_runner will be generated.
Run
- compiled model, such as tf2_custom_op.xmodel
- custom op library, such as libvart_op_imp_Mylayer.so
- test image from here
- executable program, such as tf2_custom_op_graph_runner
./tf2_custom_op_graph_runner tf2_custom_op.xmodel sample.jpg
The following figure shows the result of running tf2_custom_op_graph_runner.