Latency and Throughput Estimates - 2024.2 English - UG1076

AI Engine Tools and Flows User Guide (UG1076)

Document ID
UG1076
Release Date
2024-11-28
Version
2024.2 English

During an AI Engine simulation, average throughput is automatically calculated and displayed at the end of the simulation. If you generate a VCD file during the simulation, it can also provide estimates for continuous latency and throughput.

Continuous latency refers to how the latency between specified input and output ports changes over time, whereas continuous throughput refers to how the throughput at a specified input or output port changes.

Input and output ports can be either PLIO or GMIO. You can generate a VCD file with the command:

aiesimulator --pkg-dir=./Work --dump-vcd foo --options-file=aiesim-options.txt

To minimize the size of the VCD file, only generate SHIM and STREAM_SWITCH trace data. In the aiesim-options.txt file, set AIE_DUMP_VCD_SHIM and AIE_DUMP_VCD_STREAM_SWITCH to true while setting other options to false as follows:

AIE_DUMP_VCD_IO=false
AIE_DUMP_VCD_CORE=false
AIE_DUMP_VCD_SHIM=true
AIE_DUMP_VCD_MEM=false
AIE_DUMP_VCD_STREAM_SWITCH=true
AIE_DUMP_VCD_CLK=false
Figure 1. Continuous Throughput

Latency and Continuous Latency

The Vitis IDE has a feature that allows you to analyze latency and throughput of the graph. If you open the default.run_summary file using Vitis Analyzer, you can view the latency measurements for the AI Engine Array in the Trace section of the Latency tab. You can access this feature by running:

vitis_analyzer aie/aiesimulator_output/default.aierun_summary

The Latency tab displays three columns with latency data:

First Latency

The time it takes for the first input data to reach the first output data.

Last Latency

The time it takes for the last input data to reach the last output data.

Average Latency
The difference between the average output sample time and average input sample time.

In addition, all corresponding inputs are shown for each output.

Figure 2. Latency Table

For more information about the latency of a simulation:

  1. Right-click on an input port line, and select one of three options Export Continuous Latency, Plot Continuous Latency, or Export Table.
    Note: Export Table option is only available for an output row.
    Figure 3. Options in the Context Menu for Input Ports
  2. When you choose to export or plot continuous latency, provide the filename and window width in the dialog box. The continuous latency and throughput are calculated over consecutive windows, and you can select either the number of intervals or active cycles.
    • Number of intervals: The simulation duration is divided into N intervals on which the average latency (or throughput) is computed.
    • Active Cycles: The interval length specified in terms of number of cycles.
    Figure 4. Export Continuous Latency Dialog Box

A CSV file is then generated.

Start Time (ps), End Time (ps), Latency (ps) between 'Inputp_0' and 'Outputp_0' for '64' intervals
3362400, 4181599, 1876000
4252800, 5071999, 1944800
5143200, 5962399, 1964000
6033600, 6852799, 1964800
...
58593600, 59412799, 1991200
59484000, 60303199, 1991200

After you make this selection, a CSV file is generated containing the start and end times of each interval, as well as the computed latency between input and output ports. The CSV file can be used by post processing scripts for further analysis. To view the latency graph, select the Plot Continuous Latency option. The same dialog box appears, but the latency plot is displayed in Vitis Analyzer. Several plots can be displayed on the same graph, as shown in the following image.

Figure 5. Continuous Latency

Throughput and Continuous Throughput

Throughput estimates are available also in the Trace section but in the I/O tab. The THROUGHPUT (MBYTES/S) column gives you the average throughput on each port, computed on the total duration of the simulation.

Figure 6. Throughput Table

For more information about the throughput of a simulation:

  1. Right-click on an input port line, and choose to export continuous throughput, plot continuous throughput, or export table.
    Figure 7. Options in the Context Menu for all Ports
  2. When you choose to export or plot continuous throughput, provide the filename and window width in the dialog box. Window width is specified as follows:
    • Graph Iterations: In the case of Graph Iterations, the throughput is calculated based on equally sized sections of input or output data. By default, the iteration number used during the simulation is used for this calculation, which provides the most comprehensible results regarding the evolution of throughput over time.
    • Number of intervals: The simulation duration is divided into N number of intervals to compute the average throughput for each interval.
    • Active cycles per interval: Specify the length of the interval in terms of the number of active cycles per interval.
Figure 8. Export of Plot Continuous Throughput Dialog Box
If Export Continuous Throughput has been selected, a CSV file is generated.
Start Time (ps), End Time (ps), Throughput (Mbytes/s) for 'Inputp_1 for '64' graph iterations
1486400, 2305600, 5000.0
2305600, 3128000, 4980.545
3128000, 4000000, 4697.2476
4000000, 4890400, 4600.1797
4890400, 5781600, 4596.0503
...
55673600, 56564800, 4596.0503
56564800, 57456000, 4596.0503
57456000, 58347200, 4596.0503

The CSV file contains the start and end times of each interval, as well as the computed throughput for the ports.To view the throughput graph, select the Plot Continuous Throughput option.