Quantization - Quantization - 5.3 English - 57404

AOCL-DLP supports quantized GEMM operations for efficient inference workloads. This section covers symmetric quantization, mixed-precision workflows, and how to configure scale factors and zero-points.

Quantization Concepts:

Symmetric quantization centers the quantized range around zero:

q = round(x * scale)
x = q / scale

Asymmetric quantization uses a zero-point offset:

q = round(x * scale) + zero_point
x = (q - zero_point) / scale

Integer GEMM with Dequantization Post-Ops:

For workloads where both inputs are already quantized, use integer GEMM variants and apply SCALE and BIAS post-ops for dequantization:

// Scale factors for dequantization (one per output channel)
float scale_vals[N] = { /* calibrated scales */ };
dlp_sf_t sf = {
    .scale_factor      = scale_vals,
    .scale_factor_len  = n,
    .scale_factor_type = DLP_F32
};
dlp_scale_t scale_op = { .sf = &sf, .zp = NULL };

// Bias (applied after scaling)
float bias_vals[N] = { /* bias per channel */ };
dlp_post_op_bias bias_op = {
    .bias = bias_vals, .stor_type = DLP_F32,
    .sf = NULL, .zp = NULL
};

// Chain: SCALE then BIAS
DLP_POST_OP_TYPE seq[] = { SCALE, BIAS };

dlp_metadata_t meta = {0};
meta.seq_length = 2;
meta.seq_vector = seq;
meta.scale      = &scale_op;
meta.bias       = &bias_op;

float output_f32[M * N];
aocl_gemm_u8s8s32of32(
    'R', 'N', 'N', m, n, k,
    1, activations, lda, 'N',
    weights, ldb, 'N',
    0, output_f32, ldc, &meta);

Symmetric Quantization GEMM:

AOCL-DLP provides specialized symmetric quantization variants that handle grouped quantization natively:

• aocl_gemm_s8s8s32of32_sym_quant

• aocl_gemm_s8s8s32obf16_sym_quant

The reorder functions (aocl_get_reorder_buf_size_s8s8s32os32_sym_quant and aocl_reorder_s8s8s32os32_sym_quant) accept a DLP_SYMM_STAT_QUANT* parameter to pack quantization group metadata alongside the reordered matrix. The GEMM call itself uses the standard signature with dlp_metadata_t* as the last parameter.

// Symmetric quantization config
DLP_SYMM_STAT_QUANT symq = {
    .group_size = 128
};

// Reorder weights with symmetric quantization metadata
msz_t buf_size = aocl_get_reorder_buf_size_s8s8s32os32_sym_quant(
    'R', 'N', 'B', k, n, &symq, NULL);

int8_t *b_reordered = (int8_t *)malloc(buf_size);
aocl_reorder_s8s8s32os32_sym_quant(
    'R', 'N', 'B', weights, b_reordered, k, n, ldb, &symq, NULL);

// Compute with symmetric quantization
float output_f32[M * N];
aocl_gemm_s8s8s32of32_sym_quant(
    'R', 'N', 'N', m, n, k,
    1, activations_s8, lda, 'N',
    b_reordered, ldb, 'R',
    0, output_f32, ldc, NULL);

Mixed-Precision Quantized GEMM:

For workloads where activations are in higher precision and weights are quantized:

Tips:

• Calibrate scales carefully – scale factors significantly impact accuracy.

• Validate against float baselines to verify acceptable accuracy loss.

• Use per-channel quantization for better accuracy at minimal performance cost.

• Reorder quantized weights for repeated inference calls.

• Choose output type wisely – writing quantized output (os8, ou8) avoids a separate requantization pass.

For more details, see the Quantization Guide Wiki.