Quantizing Using the vai_q_onnx API - 3.5 English

The static quantization method first runs the model using a set of inputs called calibration data. During these runs, the quantization parameters are computed for each activation. These quantization parameters are written as constants to the quantized model and used for all inputs.

Vai_q_onnx quantization tool has expanded calibration methods to power-of-2 scale/float scale quantization methods. Float scale quantization methods include MinMax, Entropy, and Percentile. Power-of-2 scale quantization methods include MinMax and MinMSE:

import vai_q_onnx

vai_q_onnx.quantize_static(
    model_input,
    model_output,
    calibration_data_reader,
    quant_format=vai_q_onnx.VitisQuantFormat.FixNeuron,
    calibrate_method=vai_q_onnx.PowerOfTwoMethod.MinMSE,
    input_nodes=[],
    output_nodes=[],
    extra_options=None,)

model_input

File path of the model to quantize.

model_output

File path of the quantized model.

calibration_data_reader

A calibration data reader. It enumerates calibration data and generates inputs for the original model.

quant_format

• QOperator: quantizes the model with quantized operators directly.
• QDQ: quantizes the model by inserting QuantizeLinear/DeQuantizeLinear on the tensor. It only supports 8-bit quantization.
• VitisQuantFormat.QDQ: quantizes the model by inserting VAIQuantizeLinear/VAIDeQuantizeLinear on the tensor. Supports more bit-width and configurations.
• VitisQuantFormat.FixNeuron: quantizes the model by inserting FixNeuron (composition of QuantizeLinear and DeQuantizeLinear) on the tensor.

calibrate_method

For DPU devices, set calibrate_method to vai_q_onnx.PowerOfTwoMethod.NonOverflow or vai_q_onnx.PowerOfTwoMethod.MinMSE to apply power-of-2 scale quantization. The PowerOfTwoMethod has two supported methods currently: MinMSE and NonOverflow. The default method is MinMSE.

input_nodes

A list(string) object. Names of the start nodes to be quantized. The nodes before these start nodes in the model are not optimized or quantized. For example, this argument can be used to skip some pre-processing nodes or stop quantizing the first node. The default value is [].

output_nodes

A list(string) object. Names of the end nodes to be quantized. The nodes after these nodes in the model are not optimized or quantized. For example, this argument can be used to skip some post-processing nodes or stop quantizing the last node. The default value is [].