Troubleshooting Event Trace in Hardware - 2023.2 English

AI Engine Tools and Flows User Guide (UG1076)
Document ID
UG1076
Release Date
2023-12-04
Version
2023.2 English
Table 1. Troubleshooting Event Trace in Hardware
Issue Resolution
Trace packets are being dropped on certain streams. (This can be determined by observing black bars on event trace streams in the Vitis IDE.) Option 1: Increase the number of trace streams coming out of AI Engine using the option below in build time.
aiecompiler --num-trace-streams=<N>

Option 2: If using PLIO event trace streams, use theaiecompiler flag --trace-plio-width=128 which uses the128-bit PLIO stream width. This ensures the PLIO stream width matches the AI Engine data rates.

Option 3: Reduce the number of tiles connected per trace stream. You can configure the AI Engine event trace metric to be applied on specific graphs or tiles. More details on these options can be found in Table 1 and Table 1.

Option 4: aiecompiler allocates trace streams for every tile in a horizontal fashion. To ensure optimal trace nets routing it is recommended to use location constraints to ensure AI Engines are placed vertically wherever possible so that all active AI Engines equally distributed to trace streams.
Kernel function names are not visible from trace. This could be because the kernel is inlined either by the compiler or via an attribute. Apply --xlopt=0 to compile the design or specify kernel functions with __attribute__((noinline)) attribute to disable kernel in-lining.
Kernels start time are off by more than one hundred cycles. Apply --broadcast-enable-core=true option to the compiler to ensure the design starts all kernels in a few clock cycles range.
Tool issues a warning message indicating trace buffer is full. Increase event trace buffer size.
For the XSDB flow, specify larger value with the -depth option in the AI Engine trace start command.
%xsdb aietrace start -graph-based-aie-tile-metrics dut:all:functions_all_stalls -work-dir ./Work -link-summary ./bf_hw.xsa.xclbin.link_summary -base-address 0x900000000 -depth 0x8000000
For the XRT flow, update xrt.ini file and aie_trace_buffer_size line.
[Debug]
aie_trace=true

[AIE_trace_settings]
buffer_size=100M
graph_based_aie_tile_metrics = dut:all:functions_all_stalls
Note: The following warning message is issued by both XSDB and XRT when the trace buffer is full.

Warning message:

AI Engine Trace Buffer size is full, Device trace could be incomplete.
Alternatively, you can also delay the event trace start based on time, iteration or user-defined event using the trace option -start-type under Table 1 and Table 1 topics.
Need to know if the PL is sending or receiving data to/from the AI Engine as expected. Add monitors on PL kernels and their memory-mapped AXI4 masters.

For more information on adding Acceleration Monitors and AXI Performance Monitors, see Generating and Opening the Timeline Trace in the Vitis Unified Software Platform Documentation: Application Acceleration Development (UG1393).