Collecting Data Using CLI - Collecting Data Using CLI - 57368

The MPI jobs are launched using MPI launchers such as mpirun and mpiexec. Use AMDuProfCLI to collect the CPU profile data for an MPI application.

The MPI job launch through mpirun uses the following syntax:

$ mpirun [options] <program> [<args>]

AMDuProfCLI is launched using <program> and the application is launched using the AMDuProfCLI's arguments. So, use the following syntax to profile an MPI application using AMDuProfCLI:

$ mpirun [options] AMDuProfCLI [options] <program> [<args>]

The specific AMDuProfCLI flags for profiling MPI applications:

• --mpi option must be specified for multi-rank job launched via mpirun/mpiexec to collect the profiling data (e.g. CPU profiling data). This option is to notify uProf that the MPI launcher is used. Only specifying this option does not enable CPU profiling data collection. Refer Application Analysis - Getting Started for more details on how to enable profiling data collection.
• --trace mpi option can be specified to collect the MPI trace data. When this option is specified, --mpi option can be omitted. Refer Parallelism - MPI Trace Analysis section for more details.
• --output-dir <output dir> specifies the path to a directory in which the profile files are saved. A session directory will be created within the <output dir> containing all the data collected from all the ranks.

A typical command uses the following syntax:

$ mpirun -np <np> /tmp/AMDuProf/bin/AMDuProfCLI collect –config <config-type> --trace mpi --output-dir <output_dir> [mpi_app] [<mpi_app_options>]

If an MPI application is launched on multiple nodes, AMDuProfCLI will profile all the MPI rank processes running on all the nodes. You can analyze the data for processes run on one/many/all node(s).

Method 1 - Profile All the Ranks On Single/Multiple Node(s)

To collect profile data for all the ranks running on a single node, execute the following commands:

$ mpirun -np 16 /tmp/AMDuProf/bin/AMDuProfCLI collect --config tbp --trace mpi --output-dir /tmp/myapp-perf myapp.exe

To collect profile data for all the ranks in multiple nodes, use the options -H / --host mpirun or specify -hostfile <hostfile>:

$ mpirun -np 16 -H host1,host2 /tmp/AMDuProf/bin/AMDuProfCLI collect --config tbp --trace mpi --output-dir /tmp/myapp-perf myapp.exe
$ mpirun -np 16 -H host1,host2 /tmp/AMDuProf/bin/AMDuProfCLI collect
--config tbp --mpi --output-dir /tmp/myapp-perf myapp.exe

Method 2 - Profiling Specific Rank(s)

To profile only a single rank running on host2, execute the following commands:

$ export AMDUPROFCLI_CMD=/tmp/AMDuProf/bin/AMDuProfCLI collect --config tbp --trace mpi --output-dir /tmp/myapp-perf 
$ mpirun -np 4 -host host1 myapp.exe : -host host2 -np 1 $AMDUPROFCLI_CMD myapp.exe

To profile only a single rank in setup where 256 ranks running on 2 hosts (128 ranks per host):

$ mpirun -host host1:128 -np 1 $AMDUPROFCLI_CMD myapp.exe : -host host2:128,host1:128 -np 255
--map-by core myapp.exe

Method 3 – Using MPI Config File

The mpirun also takes config file as an input and the AMDuProfCLI can be used with the config file to profile the MPI application

Config file (myapp_config):

#MPI - myapp config file 
-host host1 -n 4 myapp.exe 
-host host2 -n 2 /tmp/AMDuProf/bin/AMDuProfCLI collect --config tbp --trace mpi \ 
--output-dir /tmp/myapp-perf myapp.exe

To run this config to collect data only for the MPI processes running on host2, execute the following command:

$ mpirun --app myapp_config