MPI Full Tracing Report Visualization Using GUI - MPI Full Tracing Report Visualization Using GUI - 57368

uProf User Guide
Document ID
57368
Release Date
2025-06-09
Revision
5.1 English

Collecting using CLI and Importing to GUI

Use CLI to trace a target MPI application and generate the report using CLI. For the steps, see MPI Full Tracing Using CLI. Import the report to GUI as shown in the following figure to analyze the trace data:

Figure 1. Import Profile Session

Analyzing MPI Communication Matrix

After the import is complete, use MPI Communication Matrix view to analyze the MPI trace data in the GUI. Navigate to HPC > MPI Communication Matrix to view the MPI communication matrix visualizer. This view displays rank-to-rank communication summary in matrix format. The x and y- axis in the matrix are receiver and sender ranks, respectively.

Following figure shows the MPI communication matrix:
Figure 2. MPI Communication Matrix

By default, the communication matrix appears in a zoomed-out view, displaying interactions between sender and receiver ranks. You can use the mouse wheel to zoom in and out and the scroll bar to navigate horizontally and vertically. When zoomed in, the matrix also reveals the volume of data transferred between ranks in bytes.

Legend:

  1. Ranks ordered in row-wise and column-wise.
  2. Each cell displays the total data volume transferred from one rank to another rank.
  3. Tool-tip shows additional details when the mouse is hovered over a cell.
  4. Color-coding legend based on data volume.
  5. Sum of all the data transfers for the rank.
  6. Mean of all the data transfers for the rank.

Analyzing MPI Rank Timeline

Navigate to HPC > MPI Rank Timeline to view to MPI Ranks timeline. This view shows the MPI activities in the timeline graph as follows:
Figure 3. MPI Rank Timeline

Legend:

  1. Rank ID
  2. Graph of one of the following depending on the selected data source:
    • MPI API Activity (running or waiting)
    • MPI data transfer activity (receiving or sending)
    • MPI APIs called
  3. Tool-tip shows more information about the MPI activity.
  4. Displays the time range.
  5. To select the data source MPI Activity. For more information, see MPI Data Source in the section MPI Full Tracing Report Visualization Using GUI.
  6. To load more rank details.
  7. To filter the ranks from the view.
  8. Trace Overlay Cutoff can be used to specify duration in nanoseconds, which acts as a cutoff to load the trace data, that is, any traced data source which takes less than the specified nanoseconds will not be displayed.
  9. Color coding legends for data source and trace overlay.

Analyzing MPI P2P API Summary

Navigate to HPC > MPI P2P API Summary. This view summarizes the P2P APIs called by the application as follows:
Figure 4. MPI P2P API Summary

Analyzing MPI Collective API Summary

Navigate to HPC > MPI Collective API Summary. This view summarizes the collective APIs called by the application as follows:
Figure 5. MPI Collective API Summary

MPI Data Source

Supported list of MPI data source is as follows:

  • An MPI Activity that classifies MPI APIs into either "waiting" APIs (MPI_Barrier, MPI_Wait, MPI_Waitall, MPI_Waitany, or MPI_Waitsome) or "active" APIs (all the other MPI functions).
  • MPI APIs can be classified as shown in the following three tables:
Table 1. List of P2P and Collective Communication APIs
P2PSend P2PReceive Collective Communication

MPI_BSEND

MPI_BSEND_INIT

MPI_IBSEND

MPI_IRSEND

MPI_ISEND

MPI_ISSEND

MPI_RSEND

MPI_RSEND_INIT

MPI_SEND

MPI_SEND_INIT

MPI_SENDRECV

MPI_SENDRECV_REPLACE

MPI_SSEND

MPI_SSEND_INIT

MPI_IMRECV

MPI_IRECV

MPI_MRECV

MPI_RECV

MPI_RECV_INIT

MPI_ALLGATHER

MPI_ALLGATHERV

MPI_ALLREDUCE

MPI_ALLTOALL

MPI_ALLTOALLV

MPI_ALLTOALLW

MPI_BARRIER

MPI_BCAST

MPI_GATHER

MPI_GATHERV

MPI_IALLGATHER

MPI_IALLGATHERV

MPI_IALLREDUCE

MPI_IALLTOALL

MPI_IALLTOALLV

MPI_IALLTOALLW

MPI_IBARRIER

MPI_IBCAST

MPI_IGATHER

MPI_IGATHERV

MPI_IREDUCE

MPI_IREDUCE_SCATTER

MPI_ISCAN

MPI_ISCATTER

MPI_ISCATTERV

MPI_REDUCE

MPI_REDUCE_SCATTER

MPI_SCAN

MPI_SCATTER

MPI_SCATTERV
Table 2. List of Control, Request and Communication APIs
Control API RequestAPI Communication API
MPI_PCONTROL

MPI_CANCEL

MPI_START

MPI_STARTALL

MPI_TEST

MPI_TESTALL

MPI_TESTANY

MPI_TESTSOME

MPI_WAIT

MPI_WAITALL

MPI_WAITANY

MPI_WAITSOME

MPI_IMPROBE

MPI_IPROBE

MPI_MPROBE

MPI_PROBE

MPI_COMM_CREATE

MPI_COMM_DUP

MPI_COMM_DUP_WITH_INFO

MPI_COMM_SPLIT

MPI_COMM_SPLIT_TYPE

MPI_COMM_SET_NAME

MPI_INTERCOMM_CREATE

MPI_INTERCOMM_MERGE

MPI_CART_CREATE

MPI_CART_SUB

MPI_GRAPH_CREATE

MPI_DIST_GRAPH_CREATE

MPI_DIST_GRAPH_CREATE_ADJA CENT
Table 3. List of Topology and Environment APIs
Topology API Environment API

MPI_NEIGHBOR_ALLGATHER

MPI_NEIGHBOR_ALLGATHERV

MPI_NEIGHBOR_ALLTOALL

MPI_NEIGHBOR_ALLTOALLV

MPI_NEIGHBOR_ALLTOALLW

MPI_INEIGHBOR_ALLGATHER

MPI_INEIGHBOR_ALLTOALL

MPI_INEIGHBOR_ALLGATHERV

MPI_INEIGHBOR_ALLTOALLV

MPI_INEIGHBOR_ALLTOALLW

MPI_ABORT

MPI_FINALIZE

MPI_INIT

MPI_INIT_THREAD

Limitations

  • The MPI environment parameters such as Total number of ranks and Number of ranks running on each node are currently supported only for OpenMPI. MPI tracing with system-wide profiling scope is not supported.
  • MPI profiling and MPI tracing are not supported with ProfileControlAPIs.
  • MPI Tracing is not supported with Hotspots, Threading, and Overview profile configs.