AI Engine Event Trace and Analysis - 2024.1 English - XD100

Build the Design

To run the event trace on hardware, it is required to compile the AI Engine graph with --event-trace and other appropriate flags. The flags are categorized based on the way the trace data needs to be captured.

• Using the runtime as an argument, you can compile the AI Engine graph to be set up for event trace, and specify the type of profile data to capture at runtime.

• The other way is to specify one of the functions, functions_partial_stalls, or functions_all_stalls as a type of profile data during compile time, and recompile the design to capture a different type of data during runtime.

For more information on different event trace options for AI Engine compilation, refer to Event Trace Options in AI Engine Tools and Flows User Guide (UG1076).

This tutorial uses the --event-trace=runtime, --event-trace-port=plio, --num-trace-streams=8, and --xlopt=0 options.

• --event-trace=runtime option enables runtime event trace configuration.

• --event-trace-port=plio option sets the AI Engine event tracing port to be plio. Default is gmio.

• --num-trace-streams=8 option sets the number of trace streams to be 8 to collect the generated event trace data.

• --xlopt=0 option disables the aiecompiler optimization for debug purposes.

Design with the --event-trace=runtime option in the build that enables runtime events during compile time. This only needs to build the design once and allows different event trace levels to be generated during runtime via the XSDB or XRT flow.

1. Navigate to the cmd_src/ directory, and open the Makefile.

2. Search for AIE_INCLUDE_FLAGS, and add the --event-trace=runtime --event-trace-port=plio --num-trace-streams=8 --xlopt=0 options at the end. This flag is passed to the aiecompiler command during compilation

3. Do make all TARGET=hw.

4. Make sure the package step is completed by checking the sd_card.img inside the sw/ directory.

Prepare for the Hardware Run

After the design is built, you are ready to run on the hardware board.

• Flash the SD card with the built sd_card.img.

• Plug the flashed SD card into the SD card slot of the VCK190 board.

• Connect the USB type C cable to the board and computer that supports the serial port connection.

• Set the serial port configuration with Speed=115200, Data=8 bit, Parity=none, Stop bits=1 bit, and flow control=none.

• Power up the VCK190 board to see boot messages from the serial connection.

XRT Flow

1. In the hardware Linux console, create the xrt.ini file on the SD card using the following lines:

   #xrt.ini
   [Debug]
      aie_trace = true
   
   [AIE_trace_settings]
      reuse_buffer = true
      periodic_offload = true
      buffer_offload_interval_us = 100
      buffer_size = 16M
      graph_based_aie_tile_metrics = all:all:all_stalls
   

   More details about these settings are explained in XRT Trace Options in the AI Engine Tools and Flows User Guide (UG1076).

2. Run the application.

   cd /run/media/mmcblk0p1
   ./ps_app.exe a.xclbin
   

3. After a successful run, the files created on the SD card are:

   • aie_trace_N.txt

   • aie_event_runtime_config.json

   • xrt.run_summary

4. Copy these files back to the work space at same level as the design’s Work/ directory.

XSDB Flow

1. Program the device using the sd_card image, and remove any xrt.ini files in the sd_card to avoid misbehavior with the XSDB commands.

2. Target connection setup: Run the hardware server from the computer that connects to the target board. To do so, launch the hardware server from the computer that has a JTAG connection to the VCK190 board.

3. Go to the directory where the AI Engine compile Work/directory is present, and launch XSDB.

4. From the XSDB terminal, issue the following commands from the XSDB prompt:

   xsdb
   %xsdb connect -url TCP:${COMPUTER NAME/IP}:3121
   %xsdb ta
   %xsdb ta 1
   %xsdb source $::env(XILINX_VITIS)/scripts/vitis/util/aie_trace.tcl
   %xsdb aietrace start -graphs mygraph -link-summary ./tutorial.xsa.link_summary -base-address 0x900000000 -depth 0x800000 -graph-based-aie-tile-metrics "all:all:all_stalls"
   

   • -base-address 0x900000000 is the address that needs to avoid collision with your design.

   • -depth 0x8000000 is the size of the event trace file. Adjust accordingly with your design size and amount of the event trace data.

5. After the above aietrace start command is run, switch to the hardware Linux console, and run the application.

   cd /run/media/mmcblk0p1
   ./host.exe a.xclbin
   

6. After the design run completes on the hardware, stop the trace using aietrace stop. The generated events and run_summary files need to be collected and are ready to be examined.

7. Inspect the generated events and run_summary files.(aie_trace_N.txt and aie_trace_profile.run_summary) in your local workspace where XSDB is launched.

8. Open the aie_trace_profile.run_summary file in the Vitis Analyzer, and observe the results as explained in the XRT flow.

Event Trace Considerations

Event Trace Analysis Using HSDP

In the traditional hardware event trace, the trace information is stored in DDR memory available in the Versal device initially, and offloaded to SD card after the application run completes. This imposes limitations on the amount of trace information that can be stored and analyzed. AI Engine trace offload via HSDP(High Speed Debug Port) has more DDR memory in the SmartLynq+ module and supports analyzing large quantities of trace information for complex designs.

1. To run the event trace on hardware, it is required to compile the AI Engine graph with  `-event-trace-port=plio` option. This sets the event tracing port to PLIO. 
   Note: If the event tracing port is set to GMIO, the AI Engine trace cannot be offloaded via HSDP.

  Add `--aie.event-trace-port=plio` to AIE_FLAGS in Makefile 

2. After the AI Engine graph and the C/C++ kernels are compiled, and any RTL kernels are packaged, the Vitis v++ --link command links them with the target platform to build the platform file (XSA). For offloading the AI Engine trace via HSDP, it is required to add the `–profile.aie_trace_offload=HSDP` option to the v++ -link command. Add below lines in system.cfg file
```
[profile]
aie_trace_offload=HSDP
```
With this, a new HSDP IP gets instantiated for AI Engine trace offload and all the PLIO event trace streams are connected to the HSDP IP.  You can see System_DPA getting instantiated in VitisRegion.

3. Only the host program with XRT APIs controlling the AI Engine graph should be used for offloading AI Engine trace via HSDP.

4. Package the HSDP enabled XSA file generated during linking step and libadf.a to generate a sd_card image.

• You must download and install the latest microSD card image and SmartLynq+ application package to set up the SmartLynq+ module correctly. For more information on this installation process, see Downloading and Installing SW Tools. Once the setup of SmartLynq+ is done, you need to connect the Versal device evaluation board.

• The SmartLynq+ module can be accessed by a host system using the USB 3.0 connection. The driver setup information for both Windows and Linux environments is provided in the USB 3.0 Host Connection.

• For more information on connecting the Versal device evaluation board, see Connection to Versal Evaluation Boards SmartLynq+ Module User Guide (UG1514).
Note: Embedded Design Tutorial walks you through a system design example for High-Speed Debug Port with SmartLynq+ module.

• Additional information on overview of the SmartLynq+ module for the Versal adaptive SoCs is available in SmartLynq+ Module User Guide (UG1514).

1. Power on the Versal Evaluation board and SmartLynq+ module.

2. Observe the Linux boot messages in the minicom application (SmartLynq+ module can be used as a serial terminal to remotely view the UART output from the  Versal board using the pre-installed minicom application).

3. Launch XSDB from your local directory, where the AI Engine design Work/ directory is located.

``` xsdb
	%xsdb connect -url TCP:${COMPUTER NAME/IP}:3121
	%xsdb ta
	%xsdb ta 1
	%xsdb source $::env(XILINX_VITIS)/scripts/vitis/util/aie_trace.tcl
	%xsdb  `aietrace start -graphs mygraph -work-dir Work/ -link-summary tutorial.xsa.link_summary -graph-based-aie-tile-metrics "all:all:all_stalls"`
```
Note: You should be able to use all the options specified in XSDB Flow. However, the options -baseaddress and -depth are not applicable for HSDP offload. XSDB issues an error if you try to use these options.

6. Run the application on the hardware. Make sure xrt.ini file does not contain any AI Engine trace configurations which conflicts with the configuration given by XSDB.
7. Stop AI Engine trace using the command `aietrace stop`. You should see the file hsdp_event_trace0.txt corresponding to the AI Engine trace and a JSON file that contains event configuration.
8. Now, the dumped trace should be processed using the hw_analyze command to generate the .wdb file which can be used in Vitis IDE.
`hwanalyze --pkg-dir=<WORK_DIR> --trace=hsdp_event_trace_aie.txt --trace_config=<PATH_TO_JSON> -wdb`
9. Launch the Vitis IDE and open the .wdb file to analyze the event trace.

Limitations of the Source Code Debug on Hardware

• There are maximum of four breakpoints available for each tile. One of these four breakpoints is assigned to first line of main() function by default and can be cleared and then assigned to other lines during debug.

• Due to the compiler, the -O0 option is not supported; non-sequential execution when stepping through the source code is expected.

• If an individual kernel is highlighted, select the resume button to continue execution until the next breakpoint or blocked to wait for I/O. If the PeakDetect design is highlighted, select the resume button to resume all kernels execution until meet each kernel’s breakpoint or blocked waiting for each kernel’s I/O operation. Due to compiler optimization, some variables’ values are stored in registers. “N/A” is shown in the variables view for those optimized variables’ values.

• printf() is not supported in hardware debug.