Custom OP Registration - Custom OP Registration - 3.0 English - UG1414

Vitis AI User Guide (UG1414)

Document_ID
UG1414
Release_Date
2023-02-24
Version
3.0 English

This section will illustrate how to register a custom op step-by-step.

First of all, you can use the Netron to check the custom op's properties, such as inputs and outputs.
Figure 1. Custom op's properties
You can also use xdputil to check the OP's detailed information. Run the following command to check the custom_layer OP.
xdputil xmodel tf2_custom_op.xmodel --op custom_layer

Then, you can create a directory for the new op and all your coding and building can be done under this new folder. Since Mylayer op has been realized in VAI2.0, you may use ~/Vitis-AI/examples/custom_operator/tensorflow2_example/op_registration/cpp/op_Mylayer for reference.

Code

Custom OP registration supports both C++ and Python. The following shows how to implement the OP in C++. For the OP implementation in Python, refer to Vitis-AI/examples/custom_operator/tensorflow2_example/op_registration/python/
  1. Create cpp file for the new op. It's my_Mylayer_op.cpp in this example.
  2. Include the header <vart/op_imp.h>.
  3. Define a class with a constructor and a function named calculate. In this example, the class name is MylayerOp.
  4. Implement your algorithm in the calculate function. For this example, PReLU is implemented.
  5. Register your class by DEF_XIR_OP_IMP(className), for Mylayer, className is MylayerOp.

    Tips: You can copy the cpp file from my_Mylayer_op.cpp and modify the class name as desired, and then you can focus on step 4 and 5. The details of cpp file for Mylayer op are shown below.

Build

  1. Create a Makefile to build the op library.

    Refer to ~/Vitis-AI/examples/custom_operator/tensorflow2_example/op_registration/cpp/op_Mylayer/Makefile.

  2. Set the output directory and add the dependency between target (output .so), object (.o) and source (.cpp) files.

    If you use the reference Makefile in step 1, you only need to replace the file names with yours, including libvart_op_imp_Mylayer.so, my_Mylayer_op.o and my_Mylayer_op.cpp.

  3. Execute “make” to complete the build of the library.

    After building, libvart_op_imp_Mylayer.so library will be generated.

The details of Makefile for Mylayer op are shown below.

OUTPUT_DIR = $(PWD)

all: $(OUTPUT_DIR) $(OUTPUT_DIR)/libvart_op_imp_Mylayer.so

$(OUTPUT_DIR):
mkdir -p $@

$(OUTPUT_DIR)/my_Mylayer_op.o: my_Mylayer_op.cpp
$(CXX) -std=c++17 -fPIC -c -o $@ -I. -I=/install/Debug/include -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 $<

$(OUTPUT_DIR)/libvart_op_imp_Mylayer.so: $(OUTPUT_DIR)/my_Mylayer_op.o
$(CXX) -Wl,--no-undefined -shared -o $@ $+ -L=/install/Debug/lib -lglog -lvitis_ai_library-runner_helper -lvart-runner -lxir
Note: The name of the output library must keep the same format as libvart_op_imp_Mylayer.so and in this format, “Mylayer”, which is the type of Mylayer op as shown in the figure of custom op's properties, must be changed to the type of your custom op, otherwise, the library can’t be linked when debugging the op or running the graph.

Debug

This section will introduce how to debug the custom op after you generate the custom op library. Before you debug the custom op, please copy the libvart_op_imp_Mylayer.so to /usr/lib on the target. Then, use run_op command in xdputil to test the op on the target. The usage of run_op is shown below.
xdputil run_op <model_file> <op_name> [-r ref] [-d dump]
Note: For edge, run xdputil run_op on the board.

Execute xdputil run_op -h to view the valid arguments for run_op.

In the ref directory, you should provide all the input bin files with the same names as the input tensors of the custom op. For Mylayer, it has two inputs and the input tensor names are quant_max_pooling2d_1_fix_custom_layer and custom_op_wrapper/p_re_lu_10/alpha:0_const, as shown in the following figure. Thus, the input files stored in the ref directory should be quant_max_pooling2d_1_fix_custom_layer.bin and custom_op_wrapper_p_re_lu_10_alpha:0_const.bin.
Note: You can dump the golden files when you do the model quantization.
Note: The slash ("/") marks in the name should be replace by underscore ("_").

If you still don’t know what the names should be, you can just put your input files in ref, then try to execute run_op, then you can also find the file names expected in the resulting error message as shown in the following figure.

After the successfully execution of run_op, as shown in the following figure, you can find the output bin file with the same name as the output tensor in the dump directory. Then, you can compare it with the golden output and debug your code until they are the same. For Mylayer, the output tensor name is custom_layer, so the output bin file name should be custom_layer.bin as shown in the following figure.

Finally, you can compare the custom_layer.bin with the golden output file of the custom op. If they are same, that means the op library you implement is correct.
Note: Since the floating point numbers on different platforms may be different, this will cause the results of your dump and the golden results to not exactly match. Thus, the following tool for floating-point number comparison is supplied. It the difference is within a certain threshold, then it can be considered to be consistent.
xdputil comp_float <golden_file> <dump_file> [-t threshold] [--verbose]

-t: threshold, the default value is 0.5, in %