vLLM In-Tree Platform - vLLM In-Tree Platform - 57300

Starting with vLLM 0.18.0, AMD Zen CPU support is available directly inside vLLM through the in-tree ZenCpuPlatform. This path is separate from the out-of-tree zentorch vLLM plugin described earlier. The in-tree platform lets vLLM detect supported AMD Zen CPUs at startup and route selected CPU execution paths through zentorch-optimized kernels when the required zentorch operators are available.

Installation

export VLLM_VERSION=0.23.0

pip install "vllm[zen]==${VLLM_VERSION}" \
    --extra-index-url https://wheels.vllm.ai/${VLLM_VERSION}/cpu \
    --extra-index-url https://download.pytorch.org/whl/cpu

vLLM selects the in-tree Zen CPU platform when all the following conditions are true:

• vLLM is running in CPU mode.
• The host CPU is an AMD Zen CPU with AVX-512 support.
• The zentorch Python package is installed and importable.

If these conditions are met, vLLM uses: vllm.platforms.zen_cpu.ZenCpuPlatform

If the conditions are not met, vLLM falls back to the standard CPU platform.

The ZenCpuPlatform extends vLLM’s standard CPU platform and marks the platform as Zen-capable through is_zen_cpu(). This allows vLLM’s CPU execution code to choose zentorch-backed kernels where applicable. For example, vLLM can route eligible linear layers to: torch.ops.zentorch.zentorch_linear_unary

When VLLM_ZENTORCH_WEIGHT_PREPACK=1 (default), vLLM can also prepack linear weights at model load time using: torch.ops.zentorch.zentorch_weight_prepack_for_linear

This avoids repeated layout conversion during inference.

The in-tree platform also participates in vLLM’s quantized linear kernel selection. When the required zentorch operators are registered, vLLM can select zentorch-backed kernels for supported quantized paths, including:

If a zentorch operator is not available, or if a layer does not satisfy the kernel’s layout or quantization requirements, vLLM falls back to the next available CPU implementation.

Precision Support

On the in-tree ZenCpuPlatform, vLLM supports:

float16 compute is not supported on AMD Zen CPU through this path. Users should explicitly use bfloat16 for LLM inference unless float32 is required.

Example

vllm serve <model-name> --dtype bfloat16

How is it Different From the Out-of-Tree Plugin

The in-tree ZenCpuPlatform is a part of vLLM itself. The out-of-tree zentorch vLLM plugin is shipped by the zentorch package and can apply additional version-gated patches and integration logic.

If both are present, the out-of-tree plugin takes precedence depending on how zentorch was built and installed. To use the in-tree platform path without the out-of-tree vLLM plugin, install zentorch in a configuration that does not register the out-of-tree vLLM plugin.

Use the in-tree platform when you want the native vLLM integration. Use the out-of-tree plugin when you need the zentorch-shipped compatibility layer for a supported vLLM release.