Troubleshooting Event Trace in Hardware - 2022.1 English

Versal ACAP AI Engine Programming Environment User Guide (UG1076)
Document ID
UG1076
Release Date
2022-05-25
Version
2022.1 English
Table 1. Troubleshooting Event Trace in Hardware
Issue Resolution
Trace packets are being dropped on certain streams. (This can be determined by observing black bars on event trace streams in the Vitis Analyzer GUI.) Option 1: Increase the number of trace streams coming out of AI Engine using the option below in build time.
aiecompiler --num-trace-streams=<N>
Option 2: Enable periodic event trace offload feature during run time in XRT flow. An example of xrt.ini file where,
  • aie_trace_periodic_offload = true enables AI Engine event trace periodic offload feature.
  • aie_trace_buffer_offload_interval_ms = 10 sets time interval, in millisecond, that the PL data mover uses to offload data from AI Engine to DDR.
  • aie_trace_file_dump_interval_s = 3 sets the time interval, in seconds, that the PL data mover uses to offload from DDR to SD card.
[Debug]
aie_trace = true
aie_trace_buffer_size = 10M
aie_trace_metrics = functions_all_stalls

aie_trace_periodic_offload = true
aie_trace_buffer_offload_interval_ms = 10
aie_trace_file_dump_interval_s = 3
Note: The periodic event trace offload feature is supported in XRT flow only and event trace port has to be PLIO in build.
Kernel function names are not visible from trace. This could be because the kernel is inlined either by the compiler or via an attribute. Apply --xlopt=0 to compile the design or specify kernel functions with __attribute__((noinline)) attribute to disable kernel in-lining.
Kernels start time are off by more than one hundred cycles. Apply --broadcast-enable-core=true option to the compiler to ensure the design starts all kernels in a few clock cycles range.
Tool issues a warning message indicating trace buffer is full. Increase event trace buffer size.
For the XSDB flow, specify larger value with the -depth option in the AI Engine trace start command.
%xsdb aietrace start -graphs dut -config-level functions_all_stalls -work-dir ./Work -link-summary ./bf_hw.xsa.xclbin.link_summary -base-address 0x900000000 -depth 0x8000000
For the XRT flow, update xrt.ini file and aie_trace_buffer_size line.
[Debug]
aie_trace=true
aie_trace_buffer_size=100M
aie_trace_metrics = functions_all_stalls
Note: The following warning message is issued by both XSDB and XRT when the trace buffer is full.

Warning message:

AI Engine Trace Buffer size is full, Device trace could be incomplete.
Need to know if the PL is sending or receiving data to/from the AI Engine as expected.

Add monitors on PL kernels and their memory-mapped AXI4 masters.

v++ -l --profile_kernel <data:[kernel_name|all]:[compute_unit_name
|all]:[interface_name|all](:[counters|all])><[stall|exec]:[kernel_name|all]:[compute_u
nit_name|all](:[counters|all])>

For example, to monitor every kernel master, add the following:

v++ -l --profile_kernel data:all:all:all

Running on hardware, add the following lines in the xrt.ini file.

[Debug]
profile=true
native_xrt_trace=true
data_transfer_trace=coarse

After running the application on hardware, device_trace_0.csv, native_trace.csv, summary.csv, and xrt.run_summary files are generated on sd_card. Copy those files to where your project is and at same level as the project's Work directory. Issue the command vitis_analyzer xrt.run_summary and select the timeline Trace view to inspect PL trace.