System Level Trace - 2024.2 English - UG1076

AI Engine Tools and Flows User Guide (UG1076)
Document ID
UG1076
Release Date
2024-11-28
Version
2024.2 English

In order to simplify system level debug when running applications on the board, the System Level Trace provides a single view for all the events from AI Engine, Programmable Logic, and Processing System. Vitis Analyzer provides such a view with all the tools available to navigate across all events.

Event trace must be enabled for AI Engine and PL so that all events can be captured and displayed on the same time line. PS trace is also enabled at when running the application on hardware.

AI Engine

As in the previous sections, event trace must be set to runtime during compilation. This can be done using a configuration file enabled in the v++ command line:

v++ -c --mode aie --include "./src" --include "./src/kernels" --aie.workdir ./Work ${GRAPH} --platform $(PLATFORM) --config aiecompiler_trace_options.cfg
The file aiecompiler_trace_options.cfg contains the options that will allow event trace through PLIOs or GMIOs (default):
[aie]
event-trace=runtime
broadcast-enable-core=true

# For PLIO runtime Trace
event-trace-port=plio
num-trace-streams=16
trace-plio-width=128

xlopt=0
[aie]
event-trace=runtime
broadcast-enable-core=true

# For GMIO runtime Trace
event-trace-port=gmio

xlopt=0

broadcast-enable-core is true by default. Options can be set to start tracing at some event or at a specific iteration.

Programmable Logic

PL trace is enable during system linking. This is done in the command line adding the--profile.data option:

v++ -g -l --platform ${PLATFORM} ${XOS} ${LIBADF} -t hw --save-temps --verbose --config ${VPP_SPEC} --profile.data all:all:all -o XCLBIN_File

The --profile option can be used to profile many different activities. For more information, see --profile Options .

--profile.data all:all:all will monitor data on all kernels and compute units.

Runtime Trace

When the application is run on hardware, using XRT on Linux, the trace is captured using the following xrt.ini file specification,. 
# Debug group for the aie, ps and pl
[Debug]
aie_trace = true
device_trace=fine
host_trace=true

# PL Trace buffer
trace_buffer_size = 32M
trace_buffer_offload_interval_ms = 5

[AIE_trace_settings]
# PLIO
tile_based_aie_tile_metrics = all:all:functions_all_stalls

# Global system timeline
enable_system_timeline = true

[Runtime]
verbosity = 10

enable_system_timeline is true by default.

More details on the xrt.ini file for PL and PS Trace, see xrt.ini File .

Vitis Unified IDE

After the application is run on the board, a number of files are generated containing trace information for the different domains of the system: AI Engine, PS and PL. Here is an example of files generated by on-board trace:
device_trace_0.csv
2-device_trace_0.csv
3-device_trace_0.csv
aie_event_runtime_config.json
aie_event_timestamps.bin
aie_trace_0_0.txt
aie_trace_0_1.txt
aie_trace_0_2.txt
...
aie_trace_0_13.txt
aie_trace_0_14.txt
aie_trace_0_15.txt
native_trace.csv
summary.csv
xrt.run_summary

Vitis Analyzer must be run with xrt.run_summary as a parameter to visualize the entire system in the Timeline view:

Figure 1. Left Menu in Vitis IDE

On the left side of the Timeline view there is the list of all traces that have been captured during application execution in hardware:

Figure 2. System Trace

To view the details in this timeline view zooming is required:

Figure 3. System Timeline Overview

Zooming in the XRT API rows will allow you to view the PL kernel launch and AI Engine RTP update:

Figure 4. Programmable Logic Kernel Launch
Figure 5. AI Engine RTP Update

The Graph Run and End can be seen on the graph, as well as, how the API correlates with effective AI Engine graph execution:

Figure 6. Graph Run and Graph End API