MPI Full Tracing Using CLI - MPI Full Tracing Using CLI - 57368

uProf User Guide
Document ID
57368
Release Date
2025-06-09
Revision
5.1 English

Full tracing mode traces more APIs than LWT tracing, see the List of Supported MPI APIs for Full Tracing table in MPI Full Tracing Using CLI for a complete list of APIs. This mode is helpful for in-depth analysis of an MPI application activity.

The report file for the full tracing includes multiple tables to represent various details.

  • MPI Communicator Summary consists of the following columns :
    • Ranks: Member rank IDs
    • Communicator Size: Number of the member ranks
    • Elapsed Time: Time spent by the MPI APIs in the communicator

    • See the Figure 1.

  • Rank Summary Table (All Ranks) consists of the following columns:
    • Rank: Rank ID.
    • PID: Process ID.
    • MPI Time (seconds): Total time spent on the MPI APIs.
    • MPI Time (%): Percentage of MPI Time with respect to the total MPI time of all the ranks.
    • Wait Time (seconds): Time spent by the rank waiting.
    • Wait Time (%): Percentage of the rank wait time with respect to the application runtime.
    • Call Count: Number of times MPI APIs are called.
    • Volume (bytes): Volume of data in bytes sent or received.
    • Volume (%): Percentage of volume with respect to the total volume sent or received by all the ranks.
    • Elapsed Time (seconds): Application runtime.
    • Time (%): Percentage of elapsed time with respect to the total elapsed time.

    • See the Figure 2.

  • MPI Function Summary Table consists of the following columns.
    • Function: MPI API name.
    • Min Time (seconds): Minimum time spent in this API across all the ranks.
    • Max Time (seconds): Maximum time spent in this API across all the ranks.
    • Average Time (seconds): Average time spent on the API per rank.
    • MPI Time (%): Percentage of the time spent on this API with respect to the total time spent on all the MPI APIs.
    • Volume (Bytes): Total volume sent or received by this MPI API.
    • Calls: Number of times this MPI API is called.
    • Total Time (seconds): Total time spent in the API in all the ranks.

    See Figure 3. Also see, non-collective communication APIs: Table 1, Table 2, and Table 3

  • Communication Matrix consists of the following columns:
    • Rank: Directed arrow from Sender rank ID to Receiver rank ID.
    • MPI Time (seconds): Total time spent on the APIs sending data from the sender rank to the receiver rank.
    • MPI Time (%): Percentage of MPI time with respect to the total MPI Time spent on all the APIs.
    • Volume (Bytes): Total volume of data sent from the sender rank to the receiver rank.
    • Volume (%): Percentage of volume with respect to the total volume transferred between all the ranks.
    • Transfers: Number of transfers from the sender rank to the receiver rank.

      See Figure 4.

  • Collective Events Summary consists of the following columns:
    • Function: API name.
    • Min Time (seconds): Minimum time spent on this API across all ranks.
    • Max Time (seconds): Maximum time spent on this API across all ranks.
    • Average time (seconds): Average time spent on this API per rank.
    • MPI Time (%): Percentage of time spent on this API with respect to the total time spent on all the MPI calls.
    • Input Volume (Bytes): Total data in bytes received by all the ranks involved in this API call.
    • Output Volume (Bytes): Total data sent by all the ranks involved in this API call.
    • Calls: Number of times this API is called.
    • Total Time (seconds): Total time spent in the API in all the ranks.

      See Figure 5. Also see: Table 1 for a list of APIs covered for this section.

The list of supported MPI APIs is as follows:

Table 1. List of Supported MPI APIs for Full Tracing
Sr. No. API Sr. No. API Sr. No. API
1 MPI_Pcontrol 30 MPI_Ssend_init 59 MPI_Iscatterv
2 MPI_Mrecv 31 MPI_Neighbor_alltoallv 60 MPI_Intercomm_create
3 MPI_Reduce 32 MPI_Ibarrier 61 MPI_Waitsome
4 MPI_Iallreduce 33 MPI_Test 62 MPI_Scatterv
5 MPI_Cancel 34 MPI_Rsend_init 63 MPI_Igather
6 MPI_Imrecv 35 MPI_Bcast 64 MPI_Intercomm_merge
7 MPI_Allreduce 36 MPI_Ibcast 65 MPI_Barrier
8 MPI_Ialltoall 37 MPI_Testall 66 MPI_Gather
9 MPI_Probe 38 MPI_Send_init 67 MPI_Igatherv
10 MPI_Send 39 MPI_Scan 68 MPI_Cart_create
11 MPI_Alltoall 40 MPI_Comm_create 69 MPI_Recv
12 MPI_Ialltoallv 41 MPI_Testany 70 MPI_Gatherv
13 MPI_Iprobe 42 MPI_Ibsend 71 MPI_Iallgather
14 MPI_Bsend 43 MPI_Reduce_scatter 72 MPI_Cart_sub
15 MPI_Alltoallv 44 MPI_Comm_dup 73 MPI_Irecv
16 MPI_Ialltoallw 45 MPI_Testsome 74 MPI_Allgather
17 MPI_Mprobe 46 MPI_Issend 75 MPI_Iallgatherv
18 MPI_Ssend 47 MPI_Ireduce_scatter 76 MPI_Graph_create
19 MPI_Alltoallw 48 MPI_Comm_dup_with_info 77 MPI_Sendrecv
20 MPI_Ineighbor_alltoall 49 MPI_Wait 78 MPI_Allgatherv
21 MPI_Improbe 50 MPI_Irsend 79 MPI_Ineighbor_allgather
22 MPI_Rsend 51 MPI_Iscan 80 MPI_Dist_graph_create
23 MPI_Neighbor_alltoall 52 MPI_Comm_split 81 MPI_Sendrecv_replace
24 MPI_Ineighbor_alltoallw 53 MPI_Waitall 82 MPI_Neighbor_allgather
25 MPI_Start 54 MPI_Isend 83 MPI_Ineighbor_allgatherv
26 MPI_Bsend_init 55 MPI_Iscatter 84 MPI_Dist_graph_create_adjacent
27 MPI_Neighbor_alltoallw 56 MPI_Comm_split_type 85 MPI_Recv_init
28 MPI_Ineighbor_alltoallv 57 MPI_Waitany 86 MPI_Neighbor_allgatherv
29 MPI_Startall 58 MPI_Scatter 87 MPI_Ireduce

Collect Profile Data

Example of a command to FULL trace an MPI application using AMD uProf CLI:

$ mpirun -np <number of processes> ./AMDuProfCLI collect --trace mpi --mpi-impl mpich --mpi-scope full -o <output_directory> <application>

After completing the tracing, the path to the session directory is displayed on the terminal.

MPI implementation MPICH or Open MPI should be passed in the command; MPICH is the default. Following are the sample commands:
$ mpirun -np <number of processes> ./AMDuProfCLI collect --trace mpi --mpi-impl openmpi --mpi-scope full -o <output_directory> <application>
$ mpirun -np <number of processes> ./AMDuProfCLI collect --trace mpi --mpi-impl mpich --mpi-scope full -o <output_directory> <application>

Ensure that the correct option (mpich or openmpi) is passed depending on the MPI implementation used for compiling the MPI application. Passing an incorrect option might cause undefined behavior.

Generate Profile Report

Example of a command to generate the report in .csv format. Pass the session directory path with the-i option:
$ ./AMDuProfCLI report -i <output_directory>/<SESSION_DIR>

After completing the report generation, the report.csv file path is displayed on the terminal.

Tables in the Report file

The following screenshots show example sections of a full tracing report file:

Figure 1. MPI Communicator Summary Table
Figure 2. MPI Rank Summary Table
Figure 3. MPI Function Summary Table
Figure 4. MPI Communication Matrix
Figure 5. MPI Collective API Summary Table