LLM Compressed Models - LLM Compressed Models

LLM Compressed Models - LLM Compressed Models - 57300

ZenDNN User Guide (57300)

Document ID

57300

Release Date

2026-07-10

Revision

6.0.0 English

Hugging Face models can be quantized using llm-compressor with the W8A8 and W4A16 scheme.

Refer to the model cards for step-by-step quantization scripts.

Table 1. Config Model Card Quantization Recipe Compatibility Matrix
Scheme	Model Card	Quantization Recipe
W8A8 - Dynamic quantization	amd/gpt-oss-20b-BF16-w8a8-llmcompressor-v0.10.0.2	Link to recipe
W4A16 - Asymmetric per-group quantization	amd/Llama-3.1-8B-Instruct-w4a16-llmcompressor-v0.10.0.2	Link to recipe

Note:

Ensure you have the required dependencies installed: pip install transformers>=5.0.0 torchao==0.17.0
llm-compressor 0.10.0.2 is required to quantize the models.
zentorch v2.11.0.2 is compatible with TorchAO. AMD Quark is no longer required for quantization.
Use MappingType.SYMMETRIC for optimal performance with zentorch.
Use scale_dtype=torch.bfloat16 for compatibility with AMD EPYC™ CPU optimizations.
For per-group quantization, we recommend a group_size of 128, as this configuration has been validated by zentorch across a broad set of mainstream models.