AI Engine-ML Data Types - 2024.1 English

AI Engine-ML Kernel and Graph Programming Guide (UG1603)

Document ID
Release Date
2024.1 English

The AI Engine-ML scalar unit supports signed and unsigned integers in 8, 16, and 32-bit widths, along with some single-precision floating-point for specific operations.

The two main vector data types offered by the AI Engine API are vectors (aie::vector) and accumulators (aie::accum).

A vector represents a collection of elements of the same type which is transparently mapped to the corresponding vector registers supported on AI Engine-ML architectures. Vectors are parameterized by the element type and the number of elements, and any combination that defines a 128b/256b/512b/1024b vector is supported.
Note: The integer in bold is the default vector size for the data type supported in the AI Engine API. For example, aie::broadcast((bfloat16){1}) is equivalent to aie::broadcast<bfloat16,32>((bfloat16){1}) where specifying the size as <bfloat16,32> is optional because it is the default vector size for the bfloat16 data type.
Table 1. Supported Vector Data Types and Sizes
Vector Data Type Supported Sizes (Default Highlighted )
int8 16/32/64/128
uint8 16/32/64/128
int16 8/16/32/64
uint16 8/16/32/64
int32 4/8/16/32
uint32 4/8/16/32
float 4/8/16/32
cint16 4/8/16/32
cint32 2/4/8/16
bfloat16 8/16/32/64
cfloat 4/8/16

For example, aie::vector<int32,16> is a 16-element vector of integers with 32 bits. Each element of the vector is referred to as a lane. Using the smallest bit width necessary can improve performance by making good use of registers.

Figure 1. aie::vector<int32,16>

An accumulator represents a collection of elements of the same class, typically obtained as a result of a multiplication operation, which is transparently mapped to the corresponding accumulator registers supported on each architecture. Accumulators commonly provide a larger number of bits, allowing you to perform long chains of operations whose intermediate results might exceed the range of regular vector types. Accumulators are parameterized by the element type, and the number of elements.

Table 2. Supported Accumulator Types and Sizes
  acc32 accfloat caccfloat acc40 acc48 acc56 acc64 cacc32 cacc40 cacc48 cacc56 cacc64
Native accumulation bits 32 64

The native accumulation bits define the minimum number of bits. The AI Engine API maps different accumulator types to the nearest native accumulator type that supports the requirement. For example, acc48 maps to acc64 for the AI Engine-ML architecture. The accumulator data types listed can be used in the kernel code and is supported by AI Engine APIs.

There is a limited subset of the accumulator datatypes support for interfaces in the graph code. For details, see Stream and Cascade Data Types.

aie::vector and aie::accum have member functions to do type casting, data extraction and insertion, and indexing. These operations are covered in the following sections.

For more information on streams, see Streaming Data API.