OSU Micro-Benchmarks (OMB) 7.5.2 commands: the MPI and NCCL/RCCL benchmark programs, their run-time options and the configure flags for CUDA, ROCm, SYCL, OpenACC, NCCL and RCCL builds, taken from the README of the 7.5.2 release. Point-to-point tests take two buffer arguments, one per rank (D or H), while collectives use -d cuda. Persistent and neighborhood collectives and the OpenSHMEM, UPC, UPC++, Java and Python benchmarks are left out.
Point configure at ImageMagickPDF graphs also need the convert tool; pass its directory when convert is not on the PATH.
--enable-papi
Build with PAPI supportEnables the -P option for PAPI event output; pair it with --with-papi pointing at the PAPI build.
--with-papi=/path/to/papi/build
Path to the PAPI build
Accelerator build options 12
Command
What it does
--enable-cuda
Build CUDA extensionsLets the tests place buffers in NVIDIA GPU memory; the MPI library itself must handle GPU buffers.
--with-cuda-include=/path/to/cuda/include
CUDA header directory
--with-cuda-libpath=/path/to/cuda/lib
CUDA library directory
--enable-rocm
Build ROCm extensionsFor AMD GPUs; at run time the tests then need the -d rocm option to use device buffers.
--with-rocm=/path/to/rocm/install
ROCm install directory
--enable-openacc
Build OpenACC extensionsWhen both CUDA and OpenACC are built in, the -d option chooses between them at run time.
--enable-sycl
Build SYCL extensionsThe tests use SYCL device buffers only when they are run with -d sycl.
--with-sycl=/path/to/sycl/install
SYCL install directory
--enable-ncclomb
Build the NCCL testsAdd it to the CUDA options; NCCL and RCCL tests share one XCCL code base selected at configure time.
--with-nccl=/path/to/nccl/build
NCCL build directory
--enable-rcclomb
Build the RCCL testsPairs with --enable-rocm, so the XCCL tests run on RCCL rather than NCCL.
--with-rccl=/path/to/rocm/build
RCCL build directory
Point-to-point tests 13
osu_mbw_mr expects block rank placement: all processes on the same machine hold consecutive ranks. Check the launcher's mapping before reading its results.
Command
What it does
osu_latency
Latency testPing-pong with blocking MPI_Send and MPI_Recv; prints the average one-way latency for each message size.
osu_latency_mt
Multi-threaded latencyOne sending process against a receiver running several threads, two by default; change the counts with -t.
osu_latency_mp
Multi-process latencyForks child processes on both sides to check fork safety; with MVAPICH2 set MV2_SUPPORT_FORK_SAFETY to 1.
osu_bw
Bandwidth testSends a window of back-to-back non-blocking messages before each reply to find the peak sustained rate.
osu_bibw
Bidirectional bandwidthBoth sides send their window of messages at once, giving the aggregate bandwidth of the two nodes.
osu_mbw_mr
Multi-pair bandwidth and message rateAggregate one-way bandwidth and message rate of several sender-receiver pairs; ranks must be assigned in block order.
osu_multi_lat
Multi-pair latencySeveral pairs run the latency test at the same moment; across two nodes list the hostnames in block order.
osu_latency_p
Persistent latencySame measurement as osu_latency, but with persistent point-to-point requests.
osu_bw_p
Persistent bandwidthPersistent-request version of the osu_bw bandwidth test.
osu_bibw_p
Persistent bidirectional bandwidthCounterpart of osu_bibw that runs on persistent point-to-point requests.
osu_partitioned_latency
Partitioned latencyVariant of osu_latency that uses MPI partitioned communication.
osu_bw_fan_out
Fan-out congestion bandwidthProcesses on one node each send a window of messages to partners spread over several other nodes.
osu_bw_fan_in
Fan-in congestion bandwidthSenders spread over several nodes all target receiving processes on a single node, measuring receive bandwidth.
Blocking collective tests 14
Command
What it does
osu_allgather
Allgather latencyAccepts -l to run with MPI_IN_PLACE and -D to send derived datatypes.
osu_allgatherv
Allgatherv latency
osu_allreduce
Allreduce latencyAmong the tests that can check the correctness of their results as well as time them.
osu_alltoall
Alltoall latencyBelongs to the data-validation set, alongside the allreduce and reduce tests.
osu_alltoallv
Alltoallv latency
osu_alltoallw
Alltoallw latency
osu_barrier
Barrier latency
osu_bcast
Broadcast latencyReports the average latency across all processes per message size; add -f for minimum and maximum too.
osu_gather
Gather latencyRooted collective: pick the root with -r or rotate it on every iteration.
osu_gatherv
Gatherv latency
osu_reduce
Reduce latencyTakes -r to choose the root rank and can also validate its results.
osu_reduce_scatter
Reduce-scatter latency
osu_scatter
Scatter latencyLike the gather test, it takes -r to fix or rotate the root rank.
osu_scatterv
Scatterv latency
Non-blocking collective tests 14
Command
What it does
osu_iallgather
Non-blocking allgather latency
osu_iallgatherv
Non-blocking allgatherv latency
osu_iallreduce
Non-blocking allreduce latencyReports overlap as well: how much computation proceeds while the collective runs in the background.
osu_ialltoall
Non-blocking alltoall latencyUse -t to set how many MPI_Test calls happen during the dummy computation.
osu_ialltoallv
Non-blocking alltoallv latency
osu_ialltoallw
Non-blocking alltoallw latency
osu_ibarrier
Non-blocking barrier latency
osu_ibcast
Non-blocking broadcast latency
osu_igather
Non-blocking gather latency
osu_igatherv
Non-blocking gatherv latency
osu_ireduce
Non-blocking reduce latency
osu_ireduce_scatter
Non-blocking reduce-scatter latency
osu_iscatter
Non-blocking scatter latency
osu_iscatterv
Non-blocking scatterv latency
One-sided tests 9
Command
What it does
osu_put_latency
Put latencyDefaults to an MPI_Win_allocate window with MPI_Win_flush; -w and -s choose other variants.
osu_get_latency
Get latencyThe time reported includes the synchronization calls, not only the MPI_Get itself.
osu_put_bw
Put bandwidth
osu_get_bw
Get bandwidth
osu_put_bibw
Put bidirectional bandwidthActive synchronization only: it offers -s pscw, its default, and -s fence.
osu_acc_latency
Accumulate latency
osu_cas_latency
Compare-and-swap latency
osu_fop_latency
Fetch-and-op latency
osu_get_acc_latency
Get-accumulate latency
Startup tests 3
Command
What it does
osu_init
MPI_Init timingReports minimum, maximum and average MPI_Init time per process; with -I it initializes through an MPI session.
osu_hello
Init and finalize timingA hello-world program; wrap the launch in time to measure MPI_Init plus MPI_Finalize for all processes.
time mpirun_rsh -np 2 -hostfile hostfile osu_hello
Time a two-process startupLaunches two processes from a hostfile with the MVAPICH2 launcher mpirun_rsh and times the whole run.
NCCL and RCCL collective tests 6
Command
What it does
osu_xccl_allgather
NCCL or RCCL allgather
osu_xccl_allreduce
NCCL or RCCL allreduceCan validate its results, like its MPI counterpart osu_allreduce.
osu_xccl_alltoall
NCCL or RCCL alltoall
osu_xccl_bcast
NCCL or RCCL broadcastGives the minimum, maximum and average broadcast latency over all processes for each message length.
osu_xccl_reduce
NCCL or RCCL reduce
osu_xccl_reduce_scatter
NCCL or RCCL reduce-scatter
Message size and iterations 6
Command
What it does
-m 128
Set the maximum message sizeKeeps the default minimum and stops at 128; without -m the collectives go up to 1MB.
-m 2:128
Set message size rangeLimits the run to message lengths from 2 up to 128.
-m 2:
Set the minimum message sizeStarts at 2 and keeps the default upper limit.
-x ITERATIONS
Warm-up iterationsSets how many warm-up iterations per message size are excluded from the results.
-i ITERATIONS
Timed iterationsSets how many measured iterations run at each message size.
-h
Show helpPrints the usage line, the buffer arguments and the options of that benchmark.
Output options 6
Command
What it does
-f
Full statisticsAdds minimum and maximum latency and the iteration count to the default average.
-z
Tail percentilesPrints P99, P90 and P50; write -z followed by a comma-separated list for other percentiles.
-G tty
ASCII graph in the terminalPlots the results as text; needs gnuplot on the PATH or given at configure time.
-G png
Graph as PNG
-G pdf
Graph as PDFNeeds the ImageMagick convert tool as well as gnuplot.
Record PAPI eventsComma-separated PAPI event names, then a colon and the output file; needs a build with --enable-papi.
Device buffers 8
Point-to-point tests take the buffer locations as two trailing arguments, rank 0 first; collectives switch to device buffers through -d alone.
Command
What it does
-d cuda
CUDA device buffersCollectives place their buffers on the device whenever -d is given; point-to-point tests choose per rank with D or H.
-d openacc
OpenACC device buffersSwitches to OpenACC when OMB was built with both CUDA and OpenACC.
-d rocm
ROCm device buffersNeeded to use device buffers in a ROCm build; point-to-point tests still pick each rank's buffer with D or H.
-d sycl
SYCL device buffers
-d managed
Managed memory for collectivesCollectives then allocate their buffers as CUDA Managed Memory through cudaMallocManaged.
D
Buffer on the devicePositional argument of the point-to-point tests: the first value is for rank 0, the second for rank 1.
H
Buffer in host memoryThe default for both ranks when no buffer arguments are given.
M
Managed buffer for a rankPoint-to-point counterpart of -d managed, for a send or receive buffer.
Example GPU runs 7
The README examples use the MVAPICH2 launcher mpirun_rsh. With other launchers and MPI libraries, start the benchmark through get_local_rank so LOCAL_RANK is set.
Command
What it does
mpirun_rsh -np 2 -hostfile hostfile MV2_USE_CUDA=1 osu_latency D D
GPU-to-GPU latencyBoth ranks keep their buffers in GPU device memory, so the result is device-to-device latency.
mpirun_rsh -np 2 -hostfile hostfile MV2_USE_ROCM=1 osu_latency D D
GPU-to-GPU latency on ROCmAMD GPU form of the same run, with MV2_USE_ROCM=1 in place of the CUDA variable.
mpirun_rsh -np 2 -hostfile hostfile MV2_USE_CUDA=1 osu_bw D H
Device-to-host bandwidthRank 0 uses a GPU buffer and rank 1 a host buffer.
mpirun_rsh -np 2 -hostfile hostfile MV2_USE_ROCM=1 osu_bw D H
Device-to-host bandwidth on ROCm
mpirun_rsh -np 2 -hostfile hostfile MV2_USE_CUDA=1 get_local_rank ./osu_latency D D
Run through get_local_rankThe wrapper exports LOCAL_RANK so each process can pick its GPU before MPI_Init.
mpirun_rsh -np 2 -hostfile hostfile MV2_USE_ROCM=1 get_local_rank ./osu_latency D D
get_local_rank on ROCm
export LOCAL_RANK=$MV2_COMM_WORLD_LOCAL_RANK
Set the local rankGPU affinity comes from LOCAL_RANK; with mpirun_rsh and MVAPICH2 it is found automatically since OMB 5.4.4.
One-sided window options 9
Command
What it does
-w create
Window from MPI_Win_create
-w allocate
Window from MPI_Win_allocateThe default window type for every one-sided test.
-w dynamic
Dynamic window
-s lock
Lock and unlock syncPassive synchronization; the put bidirectional bandwidth test does not offer it.
-s flush
Flush synchronizationThe default synchronization of every one-sided test except put bidirectional bandwidth.
-s flush_local
Local flush synchronization
-s lock_all
Lock-all synchronization
-s pscw
Post-start-complete-wait syncActive synchronization, and the default for the put bidirectional bandwidth test.
-s fence
Fence synchronization
Thread, root and session options 7
Command
What it does
-t 4
Set the receiver countFor osu_latency_mt and osu_latency_mp: four receiving threads or processes against one sender.
-t 4:6
Set sender and receiver countsFour senders and six receivers; in osu_latency_mp the counts include the parent processes.
-t CALLS
MPI_Test calls during computeNon-blocking collectives only: 100, 1000 or any positive number of MPI_Test calls in the dummy computation.
-r rotate
Rotate the root rankChanges the root on every iteration of a rooted collective such as gather, reduce or scatter.
-r
Set the root rankGive fixed: followed by a rank number to keep one root for the whole run.
-l
Use MPI_IN_PLACERuns collectives such as allreduce, gather and alltoall with in-place buffers.
-I
Initialize with an MPI sessionNeeds an MPI-4 library; supported by osu_init and the one-sided tests, among others.
Data type options 5
Command
What it does
-T mpi_int
Use the MPI_INT typeAvailable in osu_latency, osu_bw, osu_bibw and the neighborhood collectives, among others.
-T all
Run every supported MPI type
-D cont
Contiguous derived datatype-D turns on derived datatype support; cont selects a contiguous type.
-D vect:8:4
Vector derived datatypeThe two numbers are the stride and the block length.
-D indx:ddt_sample.txt
Indexed datatype from a fileTakes the path of a file that describes the indexed layout.
No row on this page matches that filter.
Notes
Point to point
osu_latency between two ranks on different nodes gives the round-trip half-time per message size; on a healthy InfiniBand fabric the small-message number is a few microseconds and anything higher points at a configuration problem. osu_bw and osu_bibw measure one-way and bidirectional bandwidth, which should approach the link rate for large messages. osu_multi_lat runs several latency pairs at once and osu_mbw_mr reports aggregate bandwidth and message rate across pairs, which is closer to what a real job does; both expect ranks placed in block order. -m 2:128 restricts the message size range of a test that accepts it (-h lists a binary's options), useful for focusing on the small-message regime.
Collectives
osu_allreduce is the one that predicts deep-learning performance, since gradient synchronisation is an allreduce; run it at the node count and message sizes your job uses. osu_allgather, osu_alltoall, osu_bcast and osu_reduce cover the other patterns, with alltoall the most sensitive to network topology. Compare results at 2, 4, 8 nodes to see how the fabric scales.
GPUs
Built with --enable-cuda (or --enable-rocm), the point-to-point tests take two buffer-location arguments, one per rank: D for device memory, H for host, M for CUDA managed memory. mpirun_rsh -np 2 -hostfile hostfile MV2_USE_CUDA=1 osu_latency D D measures latency between GPU buffers on both ranks, the number to check before trusting NCCL results; collectives move to device buffers with -d cuda. Launch with the same placement and binding flags as a real job, since binding changes the answer, and outside the MVAPICH2 launcher mpirun_rsh start the benchmark through get_local_rank so each process finds its GPU.
Questions
How do I build the OSU Micro-Benchmarks with CUDA support?
Run ./configure CC=/path/to/mpicc CXX=/path/to/mpicxx with --enable-cuda, --with-cuda-include=/path/to/cuda/include and --with-cuda-libpath=/path/to/cuda/lib added, then make and make install. The MPI library must itself be able to communicate from GPU memory, and the NCCL tests also need --enable-ncclomb and --with-nccl=/path/to/nccl/build.
How do I measure latency between two GPUs with osu_latency?
Give D for both ranks: mpirun_rsh -np 2 -hostfile hostfile MV2_USE_CUDA=1 osu_latency D D. Without buffer arguments both buffers stay in host memory. With launchers other than mpirun_rsh, start the test through get_local_rank, which exports LOCAL_RANK so each process selects its GPU before MPI_Init.
Which option limits the message sizes an OSU benchmark runs?
-m: -m 128 sets only the maximum, -m 2: only the minimum and -m 2:128 both. Without it the collective tests stop at 1MB. -i ITERATIONS and -x ITERATIONS set the timed and the warm-up iterations for each size.
Why does osu_mbw_mr need block rank placement?
The test is written for block assignment, where all processes on the same machine hold consecutive ranks; cyclic assignment alternates ranks between hosts instead. With mpirun_rsh, ranks follow the order in which hosts appear in the hostfile or on the command line, so list each host's entries together.