Quantization and Dequantization - Quantization and Dequantization - 6.3 English - XD056

Quantization scaling adjusts the pre-processed input to the fixed-point range expected by the model. Dequantization scaling converts the raw integer output tensors back to floating-point values before the post-processing library interprets them.

Quantization (preprocess -> model input)

The preprocess-config.quantization.scale-factor is a multiplicative factor applied to the normalized input tensor. It converts floating-point pixel values into the INT8 range the compiled model expects.

Dequantization (model output -> postprocess)

The postprocess-config.dequantization.scale-factor is a multiplicative factor applied to the raw output tensor. It converts INT8 model outputs back to float values before the post-processing library (softmax, NMS, etc.) interprets them.

VART backend - auto-detection

When using the VART backend, vvas_xinfer reads the per-tensor quantization parameters embedded in the compiled .rai model at start-up. The following precedence applies:

1. If scale-factor is explicitly set in the config JSON, that value is used.

2. Otherwise, the model’s embedded scale is used automatically.

3. If neither is available, a default factor of 1.0 is applied.

For most pre-compiled .rai models the quantization and dequantization blocks can be omitted - the correct values are read from the model.

ONNX Runtime backend

Auto-detection is not available with the ONNX Runtime backend. scale-factor must be specified explicitly in both quantization and dequantization blocks when quantization scaling is required.