Building the benchmarks 10
| Command | What it does |
|---|---|
make IMB-MPI1 | Build IMB-MPI1Set the CC variable to the MPI compiler wrapper before this or any other target. |
make IMB-NBC | Build IMB-NBC |
make IMB-RMA | Build IMB-RMA |
make IMB-EXT | Build IMB-EXT |
make IMB-IO | Build IMB-IO |
make IMB-MT | Build IMB-MTBuilds IMB-MPI1 first, because the target lists it as a prerequisite. |
make IMB-P2P | Build IMB-P2PIts Makefile target compiles the C sources in src_c/P2P, not the C++ tree in src_cpp. |
make IMB-MPI4 | Build IMB-MPI4Not part of make all, so this target has to be named on its own. |
make all | Build every standard componentThe all target makes IMB-MPI1, NBC, RMA, EXT, IO, MT and P2P; the GPU and MPI4 binaries are left out. |
make clean | Remove build output |
Running benchmarks 14
| Command | What it does |
|---|---|
mpirun -np 10 IMB-MPI1 PingPong Allreduce | Run chosen benchmarks onlyNames are case-insensitive; with none given, every default benchmark of the component runs. |
mpirun -np 2 IMB-MPI1 -msglog 3:7 PingPong | PingPong from 8 to 128 bytesMeasures 0 bytes and then every power of two from 8 up to 128 bytes. |
mpirun -np 2 IMB-MPI1 -msglog 3 PingPong | PingPong up to 8 bytesWith only the upper exponent given, the sizes are 0, 1, 2, 4 and 8 bytes. |
mpirun -np 4 IMB-MPI1 -msglog 16 PingPong PingPing PingPongAnySource PingPingAnySource | PingPong and PingPing to 64 KBSizes run from 0 and 1 byte up to 65536 bytes; the AnySource variants are named because they are not run by default. |
mpirun -np 2 IMB-MPI1 PingPong -off_cache -1 | PingPong without cache reuseBuffers advance through memory between repetitions, using the cache sizes set in IMB_mem_info.h. |
mpirun -np 512 IMB-MPI1 -npmin 512 alltoallv -iter 20 -time 1.5 -mem 2 | Large Alltoallv runOnly at the full 512 ranks, with at most 20 iterations, 1.5 seconds per size and 2 GB of buffers per process. |
mpirun -np 14 IMB-IO P_Read_shared -npmin 7 | Shared-file read from 7 ranksOf the 14 processes started, the I/O test runs on 7 and then on all 14. |
mpirun -np 11 IMB-EXT -npmin 5 | One-sided tests from 5 ranksWith 11 processes started, the benchmarks run on 5, 10 and 11 of them. |
mpirun -np 16 IMB-MPI1 -multi 0 bcast | Bcast in multiple modeRuns 8, 4 and 2 simultaneous groups for 2, 4 and 8 processes, then one normal 16-process run. |
mpirun -np 16 IMB-MPI1 -multi 0 bcast -npmin 12 | Multiple mode that cannot splitStarting at 12 of 16 ranks leaves room for one group only, so the run is an ordinary one. |
mpirun -np 8 IMB-MPI1 -map 4x2 PingPong | PingPong between two nodesWith four ranks per node the map pairs rank 0 with rank 4, while the other six ranks wait in a barrier. |
mpirun -np 8 IMB-MPI1 -map 2x4 Biband | Biband on a 2x4 rank mapRuns on 2, 4 and 8 processes, each time with the first ranks of the reordered communicator. |
IMB-EXT -input IMB_SELECT_EXT | Run benchmarks listed in a fileIn the sample file only Unidir_Get and Accumulate are left uncommented, so only those two run. |
IMB-MPI1 -help | Show command-line helpEvery component binary takes -help and lists its own options. |
Point-to-point benchmarks 10
Single-transfer tests such as PingPong and PingPing use two active ranks; any further ranks wait until that pair has finished.
| Command | What it does |
|---|---|
PingPong | PingPong testTwo active ranks bounce one message; the time shown is half the round trip. |
PingPongSpecificSource | PingPong with a named sourceNot in the default set, so it runs only when named or added with -include. |
PingPongAnySource | PingPong with any sourceReceives with MPI_ANY_SOURCE instead of a fixed rank and is also left out by default. |
PingPing | PingPing testBoth ranks issue MPI_Isend at the same time, so each message meets one coming the other way. |
PingPingSpecificSource | PingPing with a named sourceLike PingPongSpecificSource, it is skipped unless requested by name. |
PingPingAnySource | PingPing with any sourceThe MPI_ANY_SOURCE variant of PingPing, likewise absent from the default list. |
Sendrecv | Sendrecv testEach rank sends to its right neighbour and receives from its left, counting two messages per sample. |
Exchange | Exchange testEvery rank swaps data with both chain neighbours, four messages per sample, as in boundary exchanges. |
Uniband | Unidirectional bandwidthThe first half of the ranks sends windows of messages to the second half; message rate is reported too. |
Biband | Bidirectional bandwidthBoth halves send windows to each other, so the message rate counts two windows per pair. |
Collective benchmarks 15
| Command | What it does |
|---|---|
Bcast | Broadcast testOne root sends the message to every other rank; IMB prints only bare timings for it. |
Allgather | Allgather testEach rank contributes X bytes and ends up holding the data of all ranks. |
Allgatherv | Allgather with vector countsMoves the same data as Allgather, so the difference shows any overhead of the vector call. |
Scatter | Scatter testThe root holds a separate X-byte block for each rank and sends each rank its own. |
Scatterv | Scatter with vector counts |
Gather | Gather test |
Gatherv | Gather with vector countsEvery rank sends X bytes and the root receives X times the number of ranks. |
Alltoall | Alltoall testEvery rank sends a separate X-byte block to each rank and receives one from each. |
Alltoallv | Alltoall with vector counts |
Reduce | Reduce testSums float elements with MPI_SUM onto a single root rank. |
Reduce_scatter | Reduce-scatter testSums float elements, then splits the result as evenly as possible across the ranks. |
Reduce_scatter_block | Reduce-scatter with equal blocksSame float sum as Reduce_scatter, timed through MPI_Reduce_scatter_block. |
Allreduce | Allreduce testSums float elements with MPI_SUM and leaves the result on every rank. |
Barrier | Barrier testTimes MPI_Barrier by itself; the output has bare times and no throughput column. |
Reduce_local | Local reduction test |
Choosing benchmarks and processes 7
Without -npmin, collective and parallel-transfer benchmarks run at 2, 4, 8 and further powers of two below the launch size, then at the launch size itself, for example 2, 4, 8 and 11 on 11 processes.
| Command | What it does |
|---|---|
-include BENCHMARK1[,BENCHMARK2[,...]] | Include benchmarkAdds names to the default set, comma-separated since IMB 2019; needed for benchmarks excluded by default. |
-exclude BENCHMARK1[,BENCHMARK2[,...]] | Exclude benchmarksRemoves the named benchmarks from the run; the list takes commas just like the one for -include. |
-input FILE | Read benchmark names from a fileOne name per line in a text file; lines starting with # are skipped as comments. |
-npmin P_MIN | Minimum process countRuns at P_MIN, doubles while below the launch size, then uses all ranks; a larger value means all ranks. |
-multi 0 | Multiple mode, slowest group shownSplits the ranks into groups that run at once and reports only the worst timings across them. |
-multi 1 | Multiple mode, all groups shownRuns exactly like -multi 0 but prints every group separately, which makes long reports. |
-map PxQ | Reorder ranks as a gridWith P ranks per node, partner ranks land on different nodes; P times Q must not be below the rank count. |
Message sizes and iterations 14
| Command | What it does |
|---|---|
-msglog MIN:MAX | Set message size rangeSizes are 0 and then 2^MIN up to 2^MAX bytes; the default 0:22 stops at 4 MB. |
-msglen LENGTHS_FILE | Message sizes from a fileOne non-negative length per line replaces the default sizes; for IMB-IO they are I/O portions. |
-msgsize LIST | Message sizes as a listSizes in bytes, separated by commas, given on the command line instead of in a file. |
-zero_size disable | Skip the 0-byte sizeSize 0 is measured by default; disable starts the size list at the first power of two instead. |
-iter MSGSPERSAMPLE[,OVERALL_VOL[,MSGS_NONAGGR]] | Iteration limitsOne to three integers that replace the compiled-in MSGSPERSAMPLE, OVERALL_VOL and MSGS_NONAGGR values. |
-iter 2000 | Set messages per sampleA single number changes only MSGSPERSAMPLE and leaves the other two limits at their defaults. |
-iter 2000,100 | Set messages and overall volume |
-iter 1000,40,150 | Set all three iteration limitsMSGS_NONAGGR comes third, so the first two values must be written out even when unchanged. |
-iter_policy dynamic | Dynamic iteration policyCuts repetitions when a size would overrun the -time limit: quicker, at some cost in accuracy. |
-iter_policy multiple_np | Fewer repetitions for large sizesKeeps results accurate as messages grow but can run longer; bound it with -time. |
-iter_policy auto | Pick the policy per benchmarkUses multiple_np for rooted collectives such as Bcast and dynamic for all other benchmarks. |
-iter_policy off | Fixed number of iterations |
-time SECONDS | Time budget per message sizeA float; IMB picks the largest repetition count that fits both this and the -iter limits. |
-time 0.150 | Cap each size at 150 msOnly shortens a size whose planned repetitions would take longer than that. |
Buffers and data types 10
| Command | What it does |
|---|---|
-mem GB | Buffer memory per processIf the run needs more, IMB warns how much would be required and carries on. |
-mem 0.2 | Limit buffers to 200 MB |
-off_cache CACHE_SIZE[,CACHE_LINE_SIZE] | Avoid cache reuseSizes are the last-level cache in MB and its line size; meant for IMB-MPI1 and IMB-EXT, not IMB-IO. |
-off_cache -1 | Cache sizes from IMB_mem_info.hUses the values compiled into that header, which must be filled in before building. |
-off_cache 2.5 | Assume a 2.5 MB cacheOnly the cache size is given, so the default line size applies. |
-off_cache 16,128 | 16 MB cache, 128-byte lines |
-window_size SIZE | Uniband and Biband window sizeMessages posted per window in Uniband and Biband; the default went from 64 to 256 in IMB 2021.2. |
-data_type TYPE | Message data typebyte by default; char, int, float, double, float16 and bfloat16 also work, in IMB-MPI1 only. |
-red_data_type TYPE | Data type for reductionsfloat unless changed; char, int, double, float16 and bfloat16 are accepted too. |
-contig_type TYPE | Datatype layoutbase is a plain MPI type; base_vec and resize_vec wrap it in a vector, resize doubles its extent. |
Synchronization and threading 5
| Command | What it does |
|---|---|
-sync off | No barrier between iterationsCollective benchmarks otherwise run MPI_Barrier after each iteration step, which is the default. |
-imb_barrier on | Use the internal IMB barrierSwaps MPI_Barrier for an MPI-independent barrier so collective timings compare better across MPI libraries. |
-root_shift on | Rotate the root rankAffects Bcast, Reduce, Gather and other rooted collectives; without it rank 0 is always the root. |
-msg_pause on | Pause between measurementsTakes on or off, off by default; on adds a pause between benchmarks or message sizes. |
-thread_level LEVEL | MPI thread support levelsingle, funneled, serialized or multiple, passed on to MPI_Init_thread. |
GPU benchmarks 8
IMB-MPI1-GPU leaves the SpecificSource and AnySource variants, Uniband and Biband out of its default run.
| Command | What it does |
|---|---|
make IMB-MPI1-GPU | Build IMB-MPI1-GPUNeeds CUDA 11.3 or later or Level Zero 1.0, with CUDA_INCLUDE_DIR or ZE_INCLUDE_DIR set. |
make IMB-RMA-GPU | Build IMB-RMA-GPUBuilds the RMA suite with GPU_ENABLE=1 and saves the binary as IMB-RMA-GPU. |
I_MPI_OFFLOAD=1 mpirun -np 2 IMB-MPI1-GPU -msglog 3:7 PingPong | GPU PingPong in device memoryMessage buffers sit in GPU device memory, which is what IMB-MPI1-GPU uses by default. |
I_MPI_OFFLOAD=1 mpirun -np 2 IMB-MPI1-GPU -msglog 3:7 -mem_alloc_type cpu PingPong | GPU PingPong with CPU buffersThe same run with the buffers in ordinary host memory instead of on the GPU. |
-mem_alloc_type device | Buffers in device memoryThe default; the option exists only in IMB-MPI1 builds with GPU support. |
-mem_alloc_type host | GPU-registered host buffers |
-mem_alloc_type shared | Shared memory buffers |
-mem_alloc_type cpu | Plain host memory buffers |
Multithreaded options 12
Defaults in the 2021.11 source: stride 0, 100 warm-up and 1000 timed cycles, counts 1,2,4,8, 64-byte alignment, serial allocation, int data, barrier on, checking off.
| Command | What it does |
|---|---|
-thread_level multiple | Turn on multithreaded modeAny other level makes IMB-MT behave much like IMB-MPI1; OMP_NUM_THREADS sets the threads per rank. |
-stride STRIDE | Distance between paired ranks0, the default, means half the ranks; matching the ranks per node tests inter-node traffic only. |
-warmup N | Untimed warm-up cycles |
-repeat N | Timed cycles |
-barrier off | Time without a barrier |
-barrier special | Special barrier implementation |
-count LENGTHS | Message lengths in elementsComma-separated list counted in datatype elements rather than bytes. |
-malloc_align BYTES | Buffer alignmentDefaults to 64 bytes, which suits most cases. |
-malloc_algo ALGORITHM | Allocation strategy for threadsChoose serial (the default), continuous or parallel buffer allocation. |
-check on | Check transferred dataFills buffers with rank-based integer sequences and verifies them after each transfer; needs -datatype int. |
-datatype char | Send char elementsIMB-MT sends int elements unless told otherwise, and correctness checks need int. |
-noheader | Omit benchmark headers |
Multithreaded benchmarks 10
| Command | What it does |
|---|---|
PingPongMT | Threaded PingPong |
PingPingMT | Threaded PingPing |
SendrecvMT | Threaded Sendrecv |
ExchangeMT | Threaded Exchange |
UnibandMT | Threaded Uniband |
BibandMT | Threaded Biband |
BcastMT | Threaded Bcast |
ReduceMT | Threaded Reduce |
AllreduceMT | Threaded Allreduce |
BarrierMT | Threaded Barrier |
Nonblocking and one-sided benchmarks 13
| Command | What it does |
|---|---|
Iallreduce | Nonblocking Allreduce with overlapEnabled by default; measures how far the operation overlaps with computation. |
Iallreduce_pure | Nonblocking Allreduce, pure timeTimes communication alone and is disabled by default, so request it by name. |
Ibcast | Nonblocking broadcast |
Ialltoall | Nonblocking Alltoall |
Ibarrier | Nonblocking barrier |
Unidir_put | Unidirectional putA single transfer between two active processes, and one of the RMA tests that support aggregate mode. |
Bidir_put | Bidirectional put |
One_put_all | Put from one rank to allIts throughput is multiplied by the number of target ranks. |
All_put_all | Every rank puts to every rankParallel transfer test that reports bare minimum, maximum and average times. |
Compare_and_swap | Atomic compare-and-swap |
Truly_passive_put | Test truly passive putsShows whether a put completes while the target process is busy computing outside MPI. |
-aggregate_mode aggregate | Aggregate launches onlyFor IMB-IO, IMB-EXT and IMB-RMA; the default, multi, runs aggregate and non-aggregate launches. |
-aggregate_mode non_aggregate | Non-aggregate launches only |
No row on this page matches that filter.
Notes
Running a first test
Each IMB component is its own program started under mpirun, and the benchmark names after it decide what runs: mpirun -np 10 IMB-MPI1 PingPong Allreduce runs only those two. Without names the default set runs, and collective and parallel-transfer benchmarks repeat at 2, 4, 8 … processes up to the launch size, which is how one job yields a scaling curve; -npmin P_MIN skips the small counts. -msglog MIN:MAX narrows the message sizes, which otherwise run from 0 bytes to 4 MB.
Point to point and collectives
PingPong bounces one message between two ranks and reports half the round-trip time, so its small-message result is the latency of the path between them; PingPing sends in both directions at once. Put the two ranks on different nodes to measure the network, or reorder ranks with -map PxQ and P set to the ranks per node, as in mpirun -np 8 IMB-MPI1 -map 4x2 PingPong. Collectives such as Allreduce print minimum, maximum and average times per message size; a wide spread between minimum and maximum means some ranks finish well after others.
Keeping runs short and realistic
IMB gives each message size up to 10 seconds by default (-time SECONDS), and the default -iter_policy dynamic cuts repetitions to stay inside that budget. -iter_policy off keeps a fixed count, and -iter_policy multiple_np favours accuracy at large sizes over run time. Without -off_cache CACHE_SIZE[,CACHE_LINE_SIZE] the same buffer serves every repetition, so throughput can come from cache. Some examples in the 2021.8 guide predate IMB 2019, which moved the policy out of -iter and made the -include and -exclude lists comma-separated.
Questions
Why does PingPong use only two ranks when IMB-MPI1 is started on more?
PingPong and the other single-transfer tests always use two active processes, and the rest wait until that pair is done. Add -multi 0 to split the ranks into pairs that run at the same time; the slowest pair is reported, and -multi 1 lists every pair.
How do I run the AnySource and SpecificSource variants that IMB-MPI1 skips?
Name them after the binary, or add them to the default set with -include BENCHMARK1[,BENCHMARK2[,...]], for example -include PingPongAnySource,PingPingAnySource. Since IMB 2019 the names after -include and -exclude are separated by commas; some examples in the 2021.8 user guide still show the older space-separated form.
How can IMB measure between nodes instead of inside one node?
Reorder the ranks with -map PxQ, with P the ranks per node and Q the number of nodes. On 2 nodes with 4 ranks each, mpirun -np 8 IMB-MPI1 -map 4x2 PingPong pairs rank 0 with rank 4, so the measured pair spans both nodes; with -multi 0 added, all four cross-node pairs run at once.
Which binary benchmarks MPI transfers from GPU buffers?
IMB-MPI1-GPU, built with make IMB-MPI1-GPU against CUDA 11.3 or later or Level Zero. It allocates message buffers in device memory by default; -mem_alloc_type cpu puts them in host memory instead, as in I_MPI_OFFLOAD=1 mpirun -np 2 IMB-MPI1-GPU -msglog 3:7 -mem_alloc_type cpu PingPong.
Why can IMB throughput look unrealistically high?
By default one buffer is reused for all repetitions of a message size, so the data can come from cache. -off_cache -1 avoids that with the cache sizes set in IMB_mem_info.h, and -off_cache 16,128 states a 16 MB last-level cache with 128-byte lines. The option works for IMB-MPI1 and IMB-EXT, not IMB-IO.
Sources and checks
- Intel MPI Benchmarks User Guide 2021.8: Command-line Control Options of IMB-MPI1 and the other components, with the example runs
- Intel MPI Benchmarks User Guide 2021.8: Building Intel MPI Benchmarks Make targets and GPU build prerequisites
- Intel MPI Benchmarks User Guide 2021.8: MPI-1 Benchmarks IMB-MPI1 benchmark list and the ones excluded by default
- Intel MPI Benchmarks User Guide 2021.8: Classification of MPI-1 Benchmarks Single, parallel and collective benchmark classes
- Intel MPI Benchmarks User Guide 2021.8: Running Intel MPI Benchmarks Default process counts
- Intel MPI Benchmarks User Guide 2021.8: PingPong, PingPongSpecificSource, PingPongAnySource PingPong definition
33 more source pages
- Intel MPI Benchmarks User Guide 2021.8: PingPing, PingPingSpecificSource, PingPingAnySource PingPing definition
- Intel MPI Benchmarks User Guide 2021.8: Sendrecv Sendrecv definition
- Intel MPI Benchmarks User Guide 2021.8: Exchange Exchange definition
- Intel MPI Benchmarks User Guide 2021.8: Uniband Uniband definition
- Intel MPI Benchmarks User Guide 2021.8: Biband Biband definition
- Intel MPI Benchmarks User Guide 2021.8: Bcast Bcast definition
- Intel MPI Benchmarks User Guide 2021.8: Allgather Allgather definition
- Intel MPI Benchmarks User Guide 2021.8: Allgatherv Allgatherv definition
- Intel MPI Benchmarks User Guide 2021.8: Scatter Scatter definition
- Intel MPI Benchmarks User Guide 2021.8: Scatterv Scatterv definition
- Intel MPI Benchmarks User Guide 2021.8: Gather Gather definition
- Intel MPI Benchmarks User Guide 2021.8: Gatherv Gatherv definition
- Intel MPI Benchmarks User Guide 2021.8: Alltoall Alltoall definition
- Intel MPI Benchmarks User Guide 2021.8: Reduce Reduce definition
- Intel MPI Benchmarks User Guide 2021.8: Reduce_scatter Reduce_scatter definition
- Intel MPI Benchmarks User Guide 2021.8: Reduce_scatter_block Reduce_scatter_block definition
- Intel MPI Benchmarks User Guide 2021.8: Allreduce Allreduce definition
- Intel MPI Benchmarks User Guide 2021.8: Barrier Barrier definition
- Intel MPI Benchmarks User Guide 2021.8: MPI-1 Benchmarks with GPU support IMB-MPI1-GPU runs and buffer types
- Intel MPI Benchmarks User Guide 2021.8: Command-Line Control for IMB-MT Benchmarks IMB-MT options
- Intel MPI Benchmarks User Guide 2021.8: Multithreaded MPI-1 Benchmarks IMB-MT benchmark list
- Intel MPI Benchmarks User Guide 2021.8: Command-Line Control for IMB-P2P Benchmarks IMB-P2P options
- Intel MPI Benchmarks User Guide 2021.8: Unirandom Unirandom definition
- Intel MPI Benchmarks User Guide 2021.8: Birandom Birandom definition
- Intel MPI Benchmarks User Guide 2021.8: Corandom Corandom definition
- Intel MPI Benchmarks User Guide 2021.8: IMB-NBC Benchmarks Nonblocking collective benchmark list
- Intel MPI Benchmarks User Guide 2021.8: Classification of IMB-RMA Benchmarks RMA benchmark list, classes and aggregate support
- Intel MPI Benchmarks User Guide 2021.8: Truly_passive_put Truly_passive_put definition
- Intel MPI Benchmarks README, release tag IMB-v2021.11 Help option and the options and benchmarks added in later releases
- Intel MPI Benchmarks Makefile, release tag IMB-v2021.11 Build targets, including those the guide does not list
- Intel MPI Benchmarks src_cpp/MPI1/MPI1_suite.cpp, release tag IMB-v2021.11 Built-in help text of the IMB-MPI1 options
- Intel MPI Benchmarks src_cpp/RMA/RMA_suite.cpp, release tag IMB-v2021.11 The -aggregate_mode option
- Intel MPI Benchmarks src_cpp/MT/MT_suite.cpp, release tag IMB-v2021.11 Default values of the IMB-MT options
Every row on this page was matched to a line in the documentation above on . A binding that could not be found there is not listed.
Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.