AI Engine Deadlock Analysis - 2024.1 English

Common Deadlock Scenarios

A deadlock is usually caused by insufficient FIFOs, or the access rate not matching between multiple FIFOs (or different branches of the same net in stream multicast). The following figure shows some deadlock scenarios:

Scenario 1: This scenario occurs when K1 tries to write to FIFO0, but the FIFO is full. K2 is still waiting for data coming from FIFO1 before consuming data from FIFO0.

Scenario 2: This scenario occurs when NET0 multicasts to multiple destinations, and the destinations are connected by stream or cascade stream (FIFO1). The NET0 branch 1 is full because K2 is waiting for data from FIFO1, but K1 is still hungry for data from NET0 branch 0 to produce data for FIFO1.

Scenario 3: This scenario occurs when K1 and K2 are connected by buffers (including RTP buffers) and streams. When K1 is trying to write data to K2 using FIFO0, K2 is still trying to acquire lock for the ping or pong buffer. K1 will not release the lock of the buffer until it finishes its current iteration.

AI Engine Deadlock Example and Analysis in AI Engine Simulator

The example is similar to the one used in AI Engine Execution and Measurement, except that it does not have a FIFO for the stream connection:

When the design stalls, graph::wait() and graph::end() hang. It needs to interrupt graph execution by:

• Using graph::wait(CYCLE_NUMBER): Specifying the number of cycles to wait for the API to return (if the graph does not return after CYCLE_NUMBER cycles, this API still returns immediately).

• Using graph::end(CYCLE_NUMBER): Specifying the number of cycles to wait for the graph to be ended (if the graph does not return after CYCLE_NUMBER cycles, this API still ends the graph immediately).

• Using the --simulation-cycle-timeout CYCLE_NUMBER option for aiesimulator.

The CYCLE_NUMBER should be large enough for AI Engine simulator to record all the stall events, or for hardware to run into hang status.

1. In this example, examine aie/graph.cpp. We wait for 10000 cycles:

   gr.init();
   gr.run(4);
   gr.wait(10000);
   

2. Run AI Engine simulator using the following command:

   make aiesim
   

3. Open Trace view in Vitis Analyzer by using the following command:

   vitis_analyzer aiesimulator_output/default.aierun_summary
   

   

   The hang occurs after the following activities:

   1: Kernel aie_dest1 acquires the lock of read buffer (buf0) and write buffer (buf1).

   2: Kernel aie_dest1 starts.

   3: Kernel hangs in stream stall.

   4: S2mm is waiting for kernel aie_dest1 to release buffer buf0.

aiesimulator --pkg-dir=./Work --online -wdb -ctf