Intel MPI Benchmarks (IMB) commands

Command lines, benchmark names and options of the Intel MPI Benchmarks, release 2021.11, checked against Intel's 2021.8 user guide and the IMB-v2021.11 sources. Each component is a separate binary started under mpirun, and benchmark names such as PingPong are plain arguments after it; without names the default set runs. Since IMB 2019 the lists after -include and -exclude take commas. IMB-IO and IMB-EXT appear only as build targets and example runs, and the NBC and RMA lists are a selection.

141 commands in 13 sections. Checked row by row against Intel MPI Benchmarks User Guide 2021.8: Command-line Control on . Edited by Jay.

Building the benchmarks 10

CommandWhat it does
make IMB-MPI1Build IMB-MPI1Set the CC variable to the MPI compiler wrapper before this or any other target.
make IMB-NBCBuild IMB-NBC
make IMB-RMABuild IMB-RMA
make IMB-EXTBuild IMB-EXT
make IMB-IOBuild IMB-IO
make IMB-MTBuild IMB-MTBuilds IMB-MPI1 first, because the target lists it as a prerequisite.
make IMB-P2PBuild IMB-P2PIts Makefile target compiles the C sources in src_c/P2P, not the C++ tree in src_cpp.
make IMB-MPI4Build IMB-MPI4Not part of make all, so this target has to be named on its own.
make allBuild every standard componentThe all target makes IMB-MPI1, NBC, RMA, EXT, IO, MT and P2P; the GPU and MPI4 binaries are left out.
make cleanRemove build output

Running benchmarks 14

CommandWhat it does
mpirun -np 10 IMB-MPI1 PingPong AllreduceRun chosen benchmarks onlyNames are case-insensitive; with none given, every default benchmark of the component runs.
mpirun -np 2 IMB-MPI1 -msglog 3:7 PingPongPingPong from 8 to 128 bytesMeasures 0 bytes and then every power of two from 8 up to 128 bytes.
mpirun -np 2 IMB-MPI1 -msglog 3 PingPongPingPong up to 8 bytesWith only the upper exponent given, the sizes are 0, 1, 2, 4 and 8 bytes.
mpirun -np 4 IMB-MPI1 -msglog 16 PingPong PingPing PingPongAnySource PingPingAnySourcePingPong and PingPing to 64 KBSizes run from 0 and 1 byte up to 65536 bytes; the AnySource variants are named because they are not run by default.
mpirun -np 2 IMB-MPI1 PingPong -off_cache -1PingPong without cache reuseBuffers advance through memory between repetitions, using the cache sizes set in IMB_mem_info.h.
mpirun -np 512 IMB-MPI1 -npmin 512 alltoallv -iter 20 -time 1.5 -mem 2Large Alltoallv runOnly at the full 512 ranks, with at most 20 iterations, 1.5 seconds per size and 2 GB of buffers per process.
mpirun -np 14 IMB-IO P_Read_shared -npmin 7Shared-file read from 7 ranksOf the 14 processes started, the I/O test runs on 7 and then on all 14.
mpirun -np 11 IMB-EXT -npmin 5One-sided tests from 5 ranksWith 11 processes started, the benchmarks run on 5, 10 and 11 of them.
mpirun -np 16 IMB-MPI1 -multi 0 bcastBcast in multiple modeRuns 8, 4 and 2 simultaneous groups for 2, 4 and 8 processes, then one normal 16-process run.
mpirun -np 16 IMB-MPI1 -multi 0 bcast -npmin 12Multiple mode that cannot splitStarting at 12 of 16 ranks leaves room for one group only, so the run is an ordinary one.
mpirun -np 8 IMB-MPI1 -map 4x2 PingPongPingPong between two nodesWith four ranks per node the map pairs rank 0 with rank 4, while the other six ranks wait in a barrier.
mpirun -np 8 IMB-MPI1 -map 2x4 BibandBiband on a 2x4 rank mapRuns on 2, 4 and 8 processes, each time with the first ranks of the reordered communicator.
IMB-EXT -input IMB_SELECT_EXTRun benchmarks listed in a fileIn the sample file only Unidir_Get and Accumulate are left uncommented, so only those two run.
IMB-MPI1 -helpShow command-line helpEvery component binary takes -help and lists its own options.

Point-to-point benchmarks 10

Single-transfer tests such as PingPong and PingPing use two active ranks; any further ranks wait until that pair has finished.

CommandWhat it does
PingPongPingPong testTwo active ranks bounce one message; the time shown is half the round trip.
PingPongSpecificSourcePingPong with a named sourceNot in the default set, so it runs only when named or added with -include.
PingPongAnySourcePingPong with any sourceReceives with MPI_ANY_SOURCE instead of a fixed rank and is also left out by default.
PingPingPingPing testBoth ranks issue MPI_Isend at the same time, so each message meets one coming the other way.
PingPingSpecificSourcePingPing with a named sourceLike PingPongSpecificSource, it is skipped unless requested by name.
PingPingAnySourcePingPing with any sourceThe MPI_ANY_SOURCE variant of PingPing, likewise absent from the default list.
SendrecvSendrecv testEach rank sends to its right neighbour and receives from its left, counting two messages per sample.
ExchangeExchange testEvery rank swaps data with both chain neighbours, four messages per sample, as in boundary exchanges.
UnibandUnidirectional bandwidthThe first half of the ranks sends windows of messages to the second half; message rate is reported too.
BibandBidirectional bandwidthBoth halves send windows to each other, so the message rate counts two windows per pair.

Collective benchmarks 15

CommandWhat it does
BcastBroadcast testOne root sends the message to every other rank; IMB prints only bare timings for it.
AllgatherAllgather testEach rank contributes X bytes and ends up holding the data of all ranks.
AllgathervAllgather with vector countsMoves the same data as Allgather, so the difference shows any overhead of the vector call.
ScatterScatter testThe root holds a separate X-byte block for each rank and sends each rank its own.
ScattervScatter with vector counts
GatherGather test
GathervGather with vector countsEvery rank sends X bytes and the root receives X times the number of ranks.
AlltoallAlltoall testEvery rank sends a separate X-byte block to each rank and receives one from each.
AlltoallvAlltoall with vector counts
ReduceReduce testSums float elements with MPI_SUM onto a single root rank.
Reduce_scatterReduce-scatter testSums float elements, then splits the result as evenly as possible across the ranks.
Reduce_scatter_blockReduce-scatter with equal blocksSame float sum as Reduce_scatter, timed through MPI_Reduce_scatter_block.
AllreduceAllreduce testSums float elements with MPI_SUM and leaves the result on every rank.
BarrierBarrier testTimes MPI_Barrier by itself; the output has bare times and no throughput column.
Reduce_localLocal reduction test

Choosing benchmarks and processes 7

Without -npmin, collective and parallel-transfer benchmarks run at 2, 4, 8 and further powers of two below the launch size, then at the launch size itself, for example 2, 4, 8 and 11 on 11 processes.

CommandWhat it does
-include BENCHMARK1[,BENCHMARK2[,...]]Include benchmarkAdds names to the default set, comma-separated since IMB 2019; needed for benchmarks excluded by default.
-exclude BENCHMARK1[,BENCHMARK2[,...]]Exclude benchmarksRemoves the named benchmarks from the run; the list takes commas just like the one for -include.
-input FILERead benchmark names from a fileOne name per line in a text file; lines starting with # are skipped as comments.
-npmin P_MINMinimum process countRuns at P_MIN, doubles while below the launch size, then uses all ranks; a larger value means all ranks.
-multi 0Multiple mode, slowest group shownSplits the ranks into groups that run at once and reports only the worst timings across them.
-multi 1Multiple mode, all groups shownRuns exactly like -multi 0 but prints every group separately, which makes long reports.
-map PxQReorder ranks as a gridWith P ranks per node, partner ranks land on different nodes; P times Q must not be below the rank count.

Message sizes and iterations 14

CommandWhat it does
-msglog MIN:MAXSet message size rangeSizes are 0 and then 2^MIN up to 2^MAX bytes; the default 0:22 stops at 4 MB.
-msglen LENGTHS_FILEMessage sizes from a fileOne non-negative length per line replaces the default sizes; for IMB-IO they are I/O portions.
-msgsize LISTMessage sizes as a listSizes in bytes, separated by commas, given on the command line instead of in a file.
-zero_size disableSkip the 0-byte sizeSize 0 is measured by default; disable starts the size list at the first power of two instead.
-iter MSGSPERSAMPLE[,OVERALL_VOL[,MSGS_NONAGGR]]Iteration limitsOne to three integers that replace the compiled-in MSGSPERSAMPLE, OVERALL_VOL and MSGS_NONAGGR values.
-iter 2000Set messages per sampleA single number changes only MSGSPERSAMPLE and leaves the other two limits at their defaults.
-iter 2000,100Set messages and overall volume
-iter 1000,40,150Set all three iteration limitsMSGS_NONAGGR comes third, so the first two values must be written out even when unchanged.
-iter_policy dynamicDynamic iteration policyCuts repetitions when a size would overrun the -time limit: quicker, at some cost in accuracy.
-iter_policy multiple_npFewer repetitions for large sizesKeeps results accurate as messages grow but can run longer; bound it with -time.
-iter_policy autoPick the policy per benchmarkUses multiple_np for rooted collectives such as Bcast and dynamic for all other benchmarks.
-iter_policy offFixed number of iterations
-time SECONDSTime budget per message sizeA float; IMB picks the largest repetition count that fits both this and the -iter limits.
-time 0.150Cap each size at 150 msOnly shortens a size whose planned repetitions would take longer than that.

Buffers and data types 10

CommandWhat it does
-mem GBBuffer memory per processIf the run needs more, IMB warns how much would be required and carries on.
-mem 0.2Limit buffers to 200 MB
-off_cache CACHE_SIZE[,CACHE_LINE_SIZE]Avoid cache reuseSizes are the last-level cache in MB and its line size; meant for IMB-MPI1 and IMB-EXT, not IMB-IO.
-off_cache -1Cache sizes from IMB_mem_info.hUses the values compiled into that header, which must be filled in before building.
-off_cache 2.5Assume a 2.5 MB cacheOnly the cache size is given, so the default line size applies.
-off_cache 16,12816 MB cache, 128-byte lines
-window_size SIZEUniband and Biband window sizeMessages posted per window in Uniband and Biband; the default went from 64 to 256 in IMB 2021.2.
-data_type TYPEMessage data typebyte by default; char, int, float, double, float16 and bfloat16 also work, in IMB-MPI1 only.
-red_data_type TYPEData type for reductionsfloat unless changed; char, int, double, float16 and bfloat16 are accepted too.
-contig_type TYPEDatatype layoutbase is a plain MPI type; base_vec and resize_vec wrap it in a vector, resize doubles its extent.

Synchronization and threading 5

CommandWhat it does
-sync offNo barrier between iterationsCollective benchmarks otherwise run MPI_Barrier after each iteration step, which is the default.
-imb_barrier onUse the internal IMB barrierSwaps MPI_Barrier for an MPI-independent barrier so collective timings compare better across MPI libraries.
-root_shift onRotate the root rankAffects Bcast, Reduce, Gather and other rooted collectives; without it rank 0 is always the root.
-msg_pause onPause between measurementsTakes on or off, off by default; on adds a pause between benchmarks or message sizes.
-thread_level LEVELMPI thread support levelsingle, funneled, serialized or multiple, passed on to MPI_Init_thread.

GPU benchmarks 8

IMB-MPI1-GPU leaves the SpecificSource and AnySource variants, Uniband and Biband out of its default run.

CommandWhat it does
make IMB-MPI1-GPUBuild IMB-MPI1-GPUNeeds CUDA 11.3 or later or Level Zero 1.0, with CUDA_INCLUDE_DIR or ZE_INCLUDE_DIR set.
make IMB-RMA-GPUBuild IMB-RMA-GPUBuilds the RMA suite with GPU_ENABLE=1 and saves the binary as IMB-RMA-GPU.
I_MPI_OFFLOAD=1 mpirun -np 2 IMB-MPI1-GPU -msglog 3:7 PingPongGPU PingPong in device memoryMessage buffers sit in GPU device memory, which is what IMB-MPI1-GPU uses by default.
I_MPI_OFFLOAD=1 mpirun -np 2 IMB-MPI1-GPU -msglog 3:7 -mem_alloc_type cpu PingPongGPU PingPong with CPU buffersThe same run with the buffers in ordinary host memory instead of on the GPU.
-mem_alloc_type deviceBuffers in device memoryThe default; the option exists only in IMB-MPI1 builds with GPU support.
-mem_alloc_type hostGPU-registered host buffers
-mem_alloc_type sharedShared memory buffers
-mem_alloc_type cpuPlain host memory buffers

Multithreaded options 12

Defaults in the 2021.11 source: stride 0, 100 warm-up and 1000 timed cycles, counts 1,2,4,8, 64-byte alignment, serial allocation, int data, barrier on, checking off.

CommandWhat it does
-thread_level multipleTurn on multithreaded modeAny other level makes IMB-MT behave much like IMB-MPI1; OMP_NUM_THREADS sets the threads per rank.
-stride STRIDEDistance between paired ranks0, the default, means half the ranks; matching the ranks per node tests inter-node traffic only.
-warmup NUntimed warm-up cycles
-repeat NTimed cycles
-barrier offTime without a barrier
-barrier specialSpecial barrier implementation
-count LENGTHSMessage lengths in elementsComma-separated list counted in datatype elements rather than bytes.
-malloc_align BYTESBuffer alignmentDefaults to 64 bytes, which suits most cases.
-malloc_algo ALGORITHMAllocation strategy for threadsChoose serial (the default), continuous or parallel buffer allocation.
-check onCheck transferred dataFills buffers with rank-based integer sequences and verifies them after each transfer; needs -datatype int.
-datatype charSend char elementsIMB-MT sends int elements unless told otherwise, and correctness checks need int.
-noheaderOmit benchmark headers

Multithreaded benchmarks 10

CommandWhat it does
PingPongMTThreaded PingPong
PingPingMTThreaded PingPing
SendrecvMTThreaded Sendrecv
ExchangeMTThreaded Exchange
UnibandMTThreaded Uniband
BibandMTThreaded Biband
BcastMTThreaded Bcast
ReduceMTThreaded Reduce
AllreduceMTThreaded Allreduce
BarrierMTThreaded Barrier

Shared-memory transport benchmarks 13

CommandWhat it does
UnirandomRandom-pair one-way transfersPartners change randomly for every message, so ranks do unequal work; reports latency, bandwidth and message rate.
BirandomRandom-pair two-way transfersBidirectional counterpart of Unirandom, again with partners picked at random for each message.
CorandomCoherent random transfersRandom partners in a balanced pattern where every rank does the same amount of work.
Stencil2D2D stencil exchange
Stencil3D3D stencil exchange
-iter MAX,NUMERATORP2P iteration ruleIterations are NUMERATOR MiB divided by the message size, kept between 1 and MAX; default 100000,800.
-msglog NMessage lengths 0 to 2^NCovers 0 and 1 byte and then each power of two up to 2^N.
-msglog K:NMessage lengths 2^K to 2^NThis range has no 0-byte size; it starts directly at 2^K.
-msgsz SIZESingle message sizeBytes, or a number with a K, M or G suffix such as 5k or 2m.
-msglen FILENAMEP2P sizes from a fileAn ASCII file with one length per line; K, M and G suffixes are allowed.
-msgwr offSkip writing the send bufferBy default IMB-P2P touches the send buffer cache lines before each send, as a real payload would.
-msgrd offSkip reading the receive bufferBy default each message is read back from the receive buffer after it arrives.
-pause MICROSECONDSPause between runsDefaults to 100000 microseconds, a tenth of a second.

Nonblocking and one-sided benchmarks 13

CommandWhat it does
IallreduceNonblocking Allreduce with overlapEnabled by default; measures how far the operation overlaps with computation.
Iallreduce_pureNonblocking Allreduce, pure timeTimes communication alone and is disabled by default, so request it by name.
IbcastNonblocking broadcast
IalltoallNonblocking Alltoall
IbarrierNonblocking barrier
Unidir_putUnidirectional putA single transfer between two active processes, and one of the RMA tests that support aggregate mode.
Bidir_putBidirectional put
One_put_allPut from one rank to allIts throughput is multiplied by the number of target ranks.
All_put_allEvery rank puts to every rankParallel transfer test that reports bare minimum, maximum and average times.
Compare_and_swapAtomic compare-and-swap
Truly_passive_putTest truly passive putsShows whether a put completes while the target process is busy computing outside MPI.
-aggregate_mode aggregateAggregate launches onlyFor IMB-IO, IMB-EXT and IMB-RMA; the default, multi, runs aggregate and non-aggregate launches.
-aggregate_mode non_aggregateNon-aggregate launches only

Notes

Running a first test

Each IMB component is its own program started under mpirun, and the benchmark names after it decide what runs: mpirun -np 10 IMB-MPI1 PingPong Allreduce runs only those two. Without names the default set runs, and collective and parallel-transfer benchmarks repeat at 2, 4, 8 … processes up to the launch size, which is how one job yields a scaling curve; -npmin P_MIN skips the small counts. -msglog MIN:MAX narrows the message sizes, which otherwise run from 0 bytes to 4 MB.

Point to point and collectives

PingPong bounces one message between two ranks and reports half the round-trip time, so its small-message result is the latency of the path between them; PingPing sends in both directions at once. Put the two ranks on different nodes to measure the network, or reorder ranks with -map PxQ and P set to the ranks per node, as in mpirun -np 8 IMB-MPI1 -map 4x2 PingPong. Collectives such as Allreduce print minimum, maximum and average times per message size; a wide spread between minimum and maximum means some ranks finish well after others.

Keeping runs short and realistic

IMB gives each message size up to 10 seconds by default (-time SECONDS), and the default -iter_policy dynamic cuts repetitions to stay inside that budget. -iter_policy off keeps a fixed count, and -iter_policy multiple_np favours accuracy at large sizes over run time. Without -off_cache CACHE_SIZE[,CACHE_LINE_SIZE] the same buffer serves every repetition, so throughput can come from cache. Some examples in the 2021.8 guide predate IMB 2019, which moved the policy out of -iter and made the -include and -exclude lists comma-separated.

Questions

Why does PingPong use only two ranks when IMB-MPI1 is started on more?

PingPong and the other single-transfer tests always use two active processes, and the rest wait until that pair is done. Add -multi 0 to split the ranks into pairs that run at the same time; the slowest pair is reported, and -multi 1 lists every pair.

How do I run the AnySource and SpecificSource variants that IMB-MPI1 skips?

Name them after the binary, or add them to the default set with -include BENCHMARK1[,BENCHMARK2[,...]], for example -include PingPongAnySource,PingPingAnySource. Since IMB 2019 the names after -include and -exclude are separated by commas; some examples in the 2021.8 user guide still show the older space-separated form.

How can IMB measure between nodes instead of inside one node?

Reorder the ranks with -map PxQ, with P the ranks per node and Q the number of nodes. On 2 nodes with 4 ranks each, mpirun -np 8 IMB-MPI1 -map 4x2 PingPong pairs rank 0 with rank 4, so the measured pair spans both nodes; with -multi 0 added, all four cross-node pairs run at once.

Which binary benchmarks MPI transfers from GPU buffers?

IMB-MPI1-GPU, built with make IMB-MPI1-GPU against CUDA 11.3 or later or Level Zero. It allocates message buffers in device memory by default; -mem_alloc_type cpu puts them in host memory instead, as in I_MPI_OFFLOAD=1 mpirun -np 2 IMB-MPI1-GPU -msglog 3:7 -mem_alloc_type cpu PingPong.

Why can IMB throughput look unrealistically high?

By default one buffer is reused for all repetitions of a message size, so the data can come from cache. -off_cache -1 avoids that with the cache sizes set in IMB_mem_info.h, and -off_cache 16,128 states a 16 MB last-level cache with 128-byte lines. The option works for IMB-MPI1 and IMB-EXT, not IMB-IO.

Sources and checks

33 more source pages

Every row on this page was matched to a line in the documentation above on . A binding that could not be found there is not listed.

Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.