GEMM Operations - GEMM Operations - 5.3 English - 57404

AOCL User Guide (57404)

Document_ID
57404
Release_Date
2026-05-13
Version
5.3 English

AOCL-DLP provides multiple GEMM variants optimized for different precision requirements and use cases. Every GEMM call computes:

C = post_ops( alpha * op(A) * op(B) + beta * C )

Where op(X) is either X (no transpose) or X^T (transpose), and optional fused post-operations can be applied to the result via dlp_metadata_t.

Naming Convention

Function names follow a consistent pattern: aocl_gemm_<A_type><B_type><accumulator_type>o<output_type>

For example, aocl_gemm_bf16bf16f32of32 means bfloat16 inputs, float32 accumulation, float32 output.

Supported Data Type Combinations:

Float Precision:

Input A

Input B

Accumulator

Supported Outputs

Min ISA

f32

f32

f32

f32

AVX2

f16

f16

f16

f16

AVX512_FP16

BFloat16 Precision:

Input A

Input B

Accumulator

Supported Outputs

Min ISA

bf16

bf16

f32

f32, bf16

AVX2 (*)

bf16

s4

f32

f32, bf16

AVX512

bf16

u4

f32

f32, bf16

AVX512

bf16

s8

s32

s32, f32, bf16, s8, u8

AVX512_VNNI

Float-Quantized Mixed Precision:

Input A

Input B

Accumulator

Supported Outputs

Min ISA

f32

s8

s32

s32, f32, bf16, s8, u8

AVX512_VNNI

Integer Quantized:

Input A

Input B

Accumulator

Supported Outputs

Min ISA

u8

s8

s32

s32, s8, u8, f32, bf16

AVX512_VNNI

s8

s8

s32

s32, s8, u8, f32, bf16

AVX512_VNNI

Symmetric Quantization:

Input A

Input B

Accumulator

Supported Outputs

Min ISA

s8

s8 (sym_quant)

s32

f32, bf16

AVX512_VNNI

(*) BFloat16 operations on hardware without native AVX512_BF16 automatically fall back to float32 kernels with transparent conversion. See BFloat16 Fallback Behavior.

Choosing a GEMM Variant:

Need

Recommended Variant

Why

Maximum accuracy

f32f32f32of32

Full 32-bit precision throughout

Good accuracy, less memory

bf16bf16f32of32

BF16 inputs save memory, f32 accumulation preserves range

Quantized inference

u8s8s32os32 or s8s8s32os32

Integer math is fastest on VNNI hardware

Weight-quantized inference

bf16s4f32of32

BF16 activations with 4-bit weights

Half-precision pipeline

f16f16f16of16

Native FP16 end-to-end (requires AVX512_FP16)

For a complete list of GEMM variants, see the GEMM API Reference and the GEMM Guide Wiki.

Basic GEMM Call Pattern:

The basic pattern for calling AOCL-DLP GEMM functions follows this structure:

#include "aocl_dlp.h"

// Basic f32 GEMM call: C = alpha * A * B + beta * C
aocl_gemm_f32f32f32of32(
    'R',        // Storage format (R=row-major, C=column-major)
    'N',        // TransA (N=no transpose, T=transpose)
    'N',        // TransB
    m, n, k,    // Matrix dimensions
    1.0f,       // alpha scalar
    a, lda, 'N', // Matrix A, leading dimension, memory format
    b, ldb, 'N', // Matrix B, leading dimension, memory format
    0.0f,       // beta scalar
    c, ldc,     // Matrix C, leading dimension
    NULL        // Post-operations metadata (NULL = no post-ops)
);

Matrix Reordering for Performance:

For matrices that will be reused multiple times, reordering can significantly improve performance:

// Get buffer size needed for reordering matrix B
msz_t buffer_size = aocl_get_reorder_buf_size_f32f32f32of32(
    'R',     // Storage order (row-major)
    'N',     // TransB
    'B',     // Matrix to reorder ('A' or 'B')
    k, n,    // Dimensions (rows and cols of B)
    NULL     // Post-operations metadata
);

// Allocate buffer and reorder matrix B
float* reordered_b = (float*)malloc(buffer_size);
aocl_reorder_f32f32f32of32(
    'R',           // Storage order
    'N',           // TransB
    'B',           // Matrix to reorder
    b,             // Source matrix
    reordered_b,   // Destination buffer
    k, n,          // Dimensions
    ldb,           // Leading dimension of source
    NULL           // Post-operations metadata
);

// Use reordered matrix in GEMM calls
aocl_gemm_f32f32f32of32(
    'R', 'N', 'N', m, n, k,
    1.0f, a, lda, 'N',
    reordered_b, ldb, 'R',  // 'R' indicates reordered format
    0.0f, c, ldc, NULL
);

// Clean up
free(reordered_b);

For detailed buffer size and reorder APIs, see the API Lifecycle documentation.

Batch GEMM Operations:

AOCL-DLP supports batch GEMM operations for processing multiple matrix multiplications efficiently:

// Batch f32 GEMM example
aocl_batch_gemm_f32f32f32of32(
    order_array,                          // Array of storage layouts
    transa_array, transb_array,           // Arrays of transpose flags
    m_array, n_array, k_array,            // Arrays of dimensions
    alpha_array,                          // Array of alpha scalars
    (const float**)a_array, lda_array,    // Arrays of A matrices
    (const float**)b_array, ldb_array,    // Arrays of B matrices
    beta_array,                           // Array of beta scalars
    c_array, ldc_array,                   // Arrays of C matrices
    group_count, group_sizes,             // Grouping parameters
    mem_format_a_array, mem_format_b_array, // Memory format arrays
    post_ops_array                        // Array of post-operations metadata
);

For more information, see aocl_batch_gemm_f32f32f32of32.

Function Parameters:

All GEMM functions share a common parameter pattern:

Parameter

Values

Description

order

'R', 'C'

Row-major or column-major layout

transa, transb

'N', 'T'

No transpose or transpose

m, n, k

integers

Matrix dimensions: A is m×k, B is k×n, C is m×n

alpha, beta

scalars

C = alpha × A×B + beta × C

lda, ldb, ldc

integers

Leading dimensions (stride between rows/columns)

mem_format_a/b

'N', 'P', 'R'

Normal, packed, or reordered format

metadata

dlp_metadata_t*

Post-operations (NULL for none)

Error Handling:

GEMM functions validate parameters and report errors via dlp_metadata_t.error_hndl when a metadata pointer is provided:

float alpha=1.0f, beta=0.0f;
dlp_metadata_t meta = {0};
aocl_gemm_f32f32f32of32('R', 'N', 'N', m, n, k,
    alpha, a, lda, 'N', b, ldb, 'N', beta, c, ldc, &meta);

if (meta.error_hndl.error_code != DLP_CLSC_SUCCESS) {
    fprintf(stderr, "Error code: %d\n", meta.error_hndl.error_code);
}

Common error codes (from dlp_clsc_err_t):

Code

Meaning

DLP_CLSC_SUCCESS

Operation completed successfully

DLP_CLSC_NULL_POINTER

NULL pointer passed as argument

DLP_CLSC_INVALID_MATRIX_DIMENSION

Invalid m, n, or k

DLP_CLSC_INVALID_LEADING_DIMENSION

Leading dimension too small

DLP_CLSC_INVALID_ORDER

Invalid memory layout character

DLP_CLSC_INVALID_TRANSPOSE

Invalid transpose character

DLP_CLSC_INVALID_MEMORY_TAG

Invalid mem_format character