Introduction to Scalar and Vector Programming - Introduction to Scalar and Vector Programming - 2023.1 English - UG1079

AI Engine Kernel and Graph Programming Guide (UG1079)

Document_ID
UG1079
Release_Date
2023-06-23
Version
2023.1 English

This section provides an overview of the key elements of kernel programming for scalar and vector processing elements. The details of each element and optimization skills will be seen in following sections.

The following example demonstrates a for loop iterating through 512 int32 elements. Each loop iteration performs a single multiply of int32 a and int32 b storing the result in c and writing it to an output buffer. The scalar_mul kernel operates on two input blocks (buffer) of data input_buffer<int32> and produces an output buffer of data output_buffer<int32>.

The iterators are used to read and write to the buffers outside the kernel. The code sample below does not use any vector registers and will be implemented on the scalar engine.

#include <aie_api/aie.hpp>
#include <aie_api/aie_adf.hpp>
#include "aie_api/utils.hpp"
using namespace adf;
void scalar_mul(input_buffer<int32>& __restrict data1, input_buffer<int32>& __restrict data2, output_buffer<int32>& __restrict out){
  auto inIter1=aie::begin(data1);
  auto inIter2=aie::begin(data2);
  auto outIter=aie::begin(out);
  for(int i=0;i<512;i++) {
    int32 a=*inIter1++;
    int32 b=*inIter2++;
    int32 c=a*b;
    *outIter++=c;
  }
}

The following example is a vectorized version for the same kernel and will be implemented on the vector processor.

#include <aie_api/aie.hpp>
#include <aie_api/aie_adf.hpp>
#include "aie_api/utils.hpp"
using namespace adf;
void vect_mul(input_buffer<int32>& __restrict data1, input_buffer<int32>& __restrict data2, output_buffer<int32>& __restrict out){
  auto inIter1=aie::begin_vector<8>(data1);//iterator for vector of 8 elements
  auto inIter2=aie::begin_vector<8>(data2);
  auto outIter=aie::begin_vector<8>(out);
  for(int i=0;i<512/8;i++) chess_prepare_for_pipelining {
    auto va=*inIter1++;//vector of 8 elements
    auto vb=*inIter2++;
    auto vt=aie::mul(va,vb); //element-by-element multiplication
    *outIter++=vt.to_vector<int32>(0);
  }
}

The iterators returns a vector of 8 int32s and stores them in variables named va and vb. These two variables are vector type variables and they are passed to the API function aie::mul which outputs vt, which is an aie::accum<acc80,8> data type. The aie::accum<acc80,8> type is reduced by a shift round saturate function to_vector that allows an aie::vector<int32,8> type, to be returned and then written to the output buffer. Additional details on the data types supported by the AI Engine are covered in the following sections.

The __restrict keyword used on the input and output parameters of the functions, allows for more aggressive compiler optimization by explicitly stating independence between data.

chess_prepare_for_pipelining is a compiler pragma that explicitly directs kernel compiler to achieve optimized pipeline for the loop.

The scalar version of this example function needs 1045 cycles, while the vectorized and optimized version needs only 88 cycles. That means that there is more than ten times speedup for the vectorized version of the kernel. Vector processing itself would give 8x the throughput for int32 multiplication. However, with the loop optimizations done, it can get more than 10x.

To calculate the maximum performance for a given datapath, it is necessary to multiply the number of MACs per instruction with the clock frequency of the AI Engine kernel. For example, with 32-bit input vectors X and Z, the vector processor can achieve 8 MACs per instruction. Using the clock frequency for the slowest speed grade device results in:

8 MACs * 1 GHz clock frequency = 8 Giga MAC operations/second

The sections that follow describe in detail the various data types that can be used, registers available, and also the kinds of optimizations that can be achieved on the AI Engine using concepts like software pipelining in loops and keywords like __restrict.