Building 6
| Command | What it does |
|---|---|
make | Build the test binariesExpects CUDA in /usr/local/cuda and NCCL in /usr; the binaries are written to ./build. |
make -j | Build with parallel jobs |
make CUDA_HOME=/path/to/cuda NCCL_HOME=/path/to/nccl | Build against custom CUDA and NCCLNeeded when CUDA is not under /usr/local/cuda or NCCL is not under /usr. |
make MPI=1 MPI_HOME=/path/to/mpi CUDA_HOME=/path/to/cuda NCCL_HOME=/path/to/nccl | Build with MPI supportRequired before any multi-process or multi-node run; MPI_HOME is the MPI install prefix. |
make MPI=1 MPI_HOME=/path/to/mpi MPI_INCLUDE=/path/to/mpi/headers CUDA_HOME=/path/to/cuda NCCL_HOME=/path/to/nccl | Set the MPI header pathFor distributions that keep the MPI headers outside the include directory of MPI_HOME. |
make MPI=1 NAME_SUFFIX=_mpi MPI_HOME=/path/to/mpi CUDA_HOME=/path/to/cuda NCCL_HOME=/path/to/nccl | Add a suffix to binary namesWith _mpi as the suffix, the all-reduce test is built as all_reduce_perf_mpi. |
Running a test 5
| Command | What it does |
|---|---|
./build/all_reduce_perf -b 8 -e 128M -f 2 -g 8 | All-reduce on one nodeEight GPUs in a single process, message sizes from 8 bytes to 128 MiB, doubling at every step. |
alltoallv_perf -b 1M -e 32M -f 2 | Imbalanced all-to-allDefault generated pattern: ranks farther away in rank ID receive more of each rank's traffic. |
NCCL_TESTS_ALLTOALLV_SPREAD=0.0 alltoallv_perf -b 1M -e 32M -f 2 | All-to-all with uniform spreadSpread runs from 0.0, equal traffic to every peer, to the default 1.0, fully weighted by distance. |
NCCL_TESTS_ALLTOALLV_MATRIX_FILE=traffic.txt alltoallv_perf -b 32M -e 32M -f 2 | All-to-all from a traffic matrixThe file holds a square, whitespace-separated matrix: cell [i][j] is the byte count rank i sends to rank j. |
NCCL_TESTS_ALLTOALLV_PRINT_SUMMARY=1 | Per-rank alltoallv summaryAdds send bytes, receive bytes and both bandwidths for every rank to the alltoallv_perf output. |
Multi-node runs with MPI 11
With NCCL_TESTS_SPLIT the bandwidth printed is per group; multiply by the number of groups for the total.
| Command | What it does |
|---|---|
mpirun -np 64 -N 8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1 | All-reduce across nodes64 processes, 8 per node and one GPU each, so 64 GPUs on 8 nodes; the binary must come from an MPI=1 build. |
NCCL_TESTS_SPLIT="MOD 8" | Test the inter-node network onlyOn 8-GPU nodes this gives 8 parallel operations with one GPU per node in each. |
NCCL_TESTS_SPLIT="AND 0x7" | Inter-node groups by bit maskSame grouping as MOD 8, written as a mask; values may be decimal, 0x hex or 0b binary. |
NCCL_TESTS_SPLIT="DIV 8" | Keep each group inside a nodeEvery 8-GPU node forms its own group, so nothing crosses the network. |
NCCL_TESTS_SPLIT="OR 0x7" | Per-node groups by mask |
NCCL_TESTS_SPLIT="MOD 2" | Two interleaved groupsEven and odd ranks each run the operation as a separate group. |
NCCL_TESTS_SPLIT="AND 0x1" | Two groups by maskMask form of the MOD 2 split, giving the same two operations. |
NCCL_TESTS_SPLIT_MASK | Split ranks with a plain maskShorthand for an AND split: the value is used as the mask without writing the operation. |
mpirun -np 8 ./build/comm_ops_perf init | Time communicator initReports latency in ms; this binary exists only in MPI=1 builds and ignores the collective arguments. |
mpirun -np 8 ./build/comm_ops_perf split --sweep-comms fixed | Time communicator splitKeeps the communicator size fixed instead of sweeping through sizes. |
mpirun -np 8 ./build/comm_ops_perf grow --resize factor-1.5 | Time communicator growThe -r resize option also applies to the split and shrink modes. |
Test binaries 12
busbw is the algorithm bandwidth times a per-collective correction factor, so it can be compared with the hardware peak whatever the rank count.
| Command | What it does |
|---|---|
all_reduce_perf | All-reduce benchmarkIts bus bandwidth is the algorithm bandwidth times 2(n-1)/n for n ranks. |
all_gather_perf | All-gather benchmarkCopies each rank's data to every rank; the bus bandwidth factor is (n-1)/n. |
broadcast_perf | Broadcast benchmarkAll data leaves the root rank, so bus bandwidth equals algorithm bandwidth. |
reduce_perf | Reduce benchmarkEvery rank sends to the root; the bus bandwidth correction factor is 1. |
reduce_scatter_perf | Reduce-scatter benchmarkPerforms only the addition half of an all-reduce; bus bandwidth factor (n-1)/n. |
alltoall_perf | All-to-all benchmarkPERFORMANCE.md gives its bus bandwidth factor as (n-1)/n. |
alltoallv_perf | Irregular all-to-all benchmarkEach rank sends different amounts to different peers; it accepts the standard arguments. |
gather_perf | Gather benchmark |
scatter_perf | Scatter benchmark |
sendrecv_perf | Send/receive benchmark |
hypercube_perf | Hypercube benchmark |
comm_ops_perf | Communicator operations benchmarkTimes init, split, shrink and grow of NCCL communicators rather than a collective. |
Size and iteration flags 10
Total ranks = MPI processes × threads per process × GPUs per thread; the process count comes from MPI, never from a test argument.
| Command | What it does |
|---|---|
-t THREADS | Threads per processLong form --nthreads; the default is 1. |
-g GPUS | GPUs per threadLong form --ngpus, default 1; ranks equal processes times threads times this value. |
-b SIZE | Min message sizeWhere the scan starts, in bytes or with a suffix as in 128M; long form --minbytes, default 32M. |
-e SIZE | Max message sizeWhere the scan stops; long form --maxbytes, default 32M. |
-i SIZE | Fixed size incrementAdds a constant amount at every step; long form --stepbytes, default 1M, and an alternative to -f. |
-f FACTOR | Step factorMultiplies the size at every step, so 2 doubles it; long form --stepfactor, off by default. |
-n COUNT | Timed iterationsLong form --iters; the default is 20. |
-w COUNT | Warmup iterationsRuns that are executed but left out of the timing; long form --warmup_iters, default 1. |
-m COUNT | Aggregated operationsGroups that many operations into every iteration; long form --agg_iters, default 1. |
-N COUNT | Run cyclesRepeats the whole scan and prints every pass; 0 loops forever, long form --run_cycles. |
Operation and datatype flags 11
The built-in help of v2.21.1 also lists mulsum as an operation when the tests are built against NCCL 2.11 or newer.
| Command | What it does |
|---|---|
-o sum/prod/min/max/avg/all | Reduction operationMatters only for all-reduce, reduce and reduce-scatter; long form --op, default sum. |
-d NCCLTYPE/all | Element datatypeLong form --datatype; the default is float and all runs every type. |
-r ROOT/all | Root rankUsed by rooted operations such as broadcast and reduce; long form --root, default 0. |
-z 0/1/2/3 | Blocking mode0 never blocks; 1 and 2 wait once per inner (-m) iteration, 3 once per outer (-n) iteration; long form --blocking. |
-G LAUNCHES | CUDA graph replayRecords the timed loop into one CUDA graph, then launches that graph the given number of times; long form --cudagraph. |
-p 0/1 | Parallel NCCL init1 initialises NCCL from several threads at once; long form --parallel_init, default 0. |
-R 0/1/2 | Buffer registration1 registers the send and receive buffers locally, 2 symmetrically; long form --local_register, default 0. |
-D NUMBER | Device API implementationValues above 0 select a custom device API path (all_reduce and alltoall only) and need -R 2. |
-H CONTEXTS | Host one-sided RMA pathSwaps the built-in collective for host RMA calls; needs -R 2 and excludes -D. |
-V NUMBER | Device API CTA countA positive value below 128; long form --device_cta_count, default 16. |
-u INDEX | Misalign buffersShifts where the send and receive data start inside their buffers; long form --unalign, default 0. |
Checking and output flags 11
| Command | What it does |
|---|---|
-c COUNT | Check correctnessVerifies the results on that many iterations; slow with many GPUs, long form --check, default 1. |
-a 0/1/2/3 | Cross-rank reporting mode0 reports rank 0, 1 the average, 2 the minimum, 3 the maximum; MPI builds only, default 1. |
-I 0/1 | Per-iteration timingRecords a CUDA event timing for every iteration and adds summary columns; cannot be combined with -G. |
-K COUNT | Skip leading samplesLeaves the first samples out of the -I summary while the JSON keeps all of them. |
-C 0/1 | Report CPU timeReplaces the latency figure with CPU time; long form --report_cputime, default 0. |
-S 0/1 | Timestamp report linesPuts a date and time on every result line; long form --report_timestamps, default 0. |
-J FILE | JSON outputSaves the results to the given path; the format follows the suffix and only json is supported. |
-T SECONDS | Per-test timeoutEnds a test that runs longer than this; long form --timeout, disabled by default. |
-M 0/1 | Memory usage reportLong form --memory; the default is 0. |
-U 0/1 | Tuning reportPrints NCCL tuning information; needs NCCL 2.28 or newer, with full detail from 2.31. |
-h | Show usagePrints the argument list and exits; long form --help. |
Analysing JSON results 8
| Command | What it does |
|---|---|
python3 analyze_perf_json.py results.json | Analyse per-iteration JSONReads a file produced with -I 1 and -J; the script is in the tools directory of the repository. |
python3 analyze_perf_json.py results.json --sizes 8,1048576,34359738368 | Limit to chosen sizesTakes a comma-separated list of message sizes; without it the script picks sizes itself. |
python3 analyze_perf_json.py results.json --straggler | Find straggler processes |
python3 analyze_perf_json.py results.json --spikes --threshold 1.5 | Detect latency spikesThe threshold is a multiple of the average; 1.5 is also what the script uses when it is omitted. |
python3 analyze_perf_json.py results.json --in-place --spikes | Analyse in-place resultsLooks at the in-place measurements, not the out-of-place ones. |
python3 analyze_perf_json.py results.json --iter0 | Measure iteration 0 impactQuantifies the warmup effect on the first timed iteration. |
python3 analyze_perf_json.py results.json --nodes --procs-per-node 4 | Analyse by nodeGroups processes into nodes of four, which is also the default group size. |
python3 analyze_perf_json.py results.json --all | Run every analysis |
Transport selection variables 30
| Command | What it does |
|---|---|
NCCL_SOCKET_IFNAME | Choose IP interfacesComma-separated name prefixes; start the list with ^ to exclude and with = to match an exact name such as =eth0. |
NCCL_IB_HCA | Choose RDMA adaptersComma-separated HCA[:PORT] entries; add the = prefix for exact names, since mlx5_1 alone also matches mlx5_10 to mlx5_19. |
NCCL_NET | Force a network moduleInternal names on Linux are IB and Socket, plugins define their own; the match ignores case. |
NCCL_IB_DISABLE | Disable InfiniBand and RoCESet to 1 and NCCL falls back to another transport such as IP sockets. |
NCCL_CROSS_NIC | Cross-rail NIC use0 keeps a ring or tree on one NIC, 1 lets it mix NICs, 2 (default) prefers one NIC but may mix. |
NCCL_IB_MERGE_NICS | Merge dual-port NICsOn by default (1); set 0 to stop dual-port IB NICs being treated as one logical device. |
NCCL_NET_GDR_LEVEL | GPUDirect RDMA distance limitLOC, PIX, PXB, PHB or SYS: the farthest NIC-to-GPU path on which GPU Direct RDMA is still used. |
NCCL_NET_GDR_READ | GPUDirect RDMA on send1 sends straight from GPU memory to the NIC; the default is 1 only on NVLink-based platforms. |
NCCL_P2P_DISABLE | Disable GPU peer-to-peerSet to 1 to turn off direct GPU-to-GPU access over NVLink or PCI. |
NCCL_P2P_LEVEL | Peer-to-peer distance limitLOC, NVL, PIX, PXB, PHB or SYS: the largest GPU-to-GPU distance that still uses P2P. |
NCCL_SHM_DISABLE | Disable shared memory transportSet to 1; devices that cannot use peer-to-peer then go through the network, not host memory. |
NCCL_NVLS_ENABLE | NVLink SHARP offload0 turns NVLS off; 1 and 2 (default) use it and fail only when its resources cannot be allocated. |
NCCL_MNNVL_ENABLE | Multi-node NVLink0 disables it, 1 requires it, 2 detects it automatically; an operational IMEX domain across the nodes is needed. |
NCCL_PXN_DISABLE | Disable PXN routingSet to 1 to stop inter-node traffic reaching a remote NIC by way of NVLink and another GPU. |
NCCL_ALGO | Restrict collective algorithmsA list among Ring, Tree, CollnetChain, CollnetDirect, NVLS, NVLSTree and PAT; a leading ^ excludes the listed ones. |
NCCL_ALGO=allreduce:^tree | Exclude tree for all-reduceOther functions keep every available algorithm; this per-function syntax needs NCCL 2.24 or later. |
NCCL_ALGO="ring,collnetdirect;allreduce:tree,collnetdirect;broadcast:ring" | Set algorithms per functionEntries are separated by semicolons; the first one, without a function name, is the default for all functions. |
NCCL_PROTO | Restrict protocolsA list among LL, LL128 and Simple; NVIDIA discourages setting it except to rule out one protocol. |
NCCL_IB_TIMEOUT | InfiniBand verbs timeoutThe timeout is 4.096 µs times 2 to the power of the value; raising it can help on very large networks. |
NCCL_IB_RETRY_CNT | InfiniBand retry countValid values are 0 to 7 and the default is 7; the total wait is this count times the timeout. |
NCCL_IB_GID_INDEX | RoCE GID indexThe default AUTO lets NCCL find the GID itself; show_gids lists the indexes a host offers. |
NCCL_IB_SL | InfiniBand service levelThe default AUTO uses 0, or the communicator's traffic class when one is configured. |
NCCL_IB_TC | InfiniBand traffic classThe default AUTO means class 0 unless the communicator configures a traffic class of its own. |
NCCL_IB_PKEY | Partition key indexIndex into the port's PKey table, default 0; NCCL_IB_PKEY_VALUE selects by value when indexes differ between hosts. |
NCCL_IB_QPS_PER_CONNECTION | Queue pairs per connection1 to 128, default 1; more queue pairs give better routing entropy on multi-level fabrics. |
NCCL_IB_ADAPTIVE_ROUTING | Adaptive routing transfersOn by default for InfiniBand networks and off for RoCE; set 1 to force it. |
NCCL_SOCKET_NTHREADS | Socket helper threads1 to 16 CPU threads per connection; 4 can be set by hand on generic 100G networks. |
NCCL_NSOCKS_PERTHREAD | Sockets per helper threadIts product with NCCL_SOCKET_NTHREADS cannot exceed 64. |
NCCL_IGNORE_CPU_AFFINITY | Ignore launcher CPU affinityWith the default 0 NCCL intersects the inherited affinity with the GPU's; 1 disregards the job's affinity. |
NCCL_BUFFSIZE | Inter-GPU buffer sizeIn bytes, default 4194304 (4 MiB); powers of 2 are recommended. |
Debugging variables 12
| Command | What it does |
|---|---|
NCCL_DEBUG | Set the debug output levelAccepts VERSION, WARN, ATTN (since 2.32), INFO or TRACE. |
NCCL_DEBUG=WARN | Log version and errorsPrints the NCCL version at start and a message whenever an NCCL call returns an error code. |
NCCL_DEBUG=INFO | Print debug informationThe level whose output NCCL_DEBUG_SUBSYS narrows down; TRACE goes further and logs every call. |
NCCL_DEBUG=WARN NCCL_DEBUG_LEVELS=ATTN | Add attention notices to WARNSafe on mixed-version clusters: releases before 2.32 ignore the second variable and keep the WARN baseline. |
NCCL_DEBUG_LEVELS | Add individual log levelsA comma-separated list of VERSION, WARN, ATTN, INFO, ABORT, TRACE or ALL, merged with the NCCL_DEBUG choice. |
NCCL_DEBUG_SUBSYS | Filter the log by subsystemNames such as INIT, NET, GRAPH, TUNING or ALL, comma-separated; ^ before a name removes it, default INIT,BOOTSTRAP,ENV. |
NCCL_DEBUG_FILE | Write the log to a fileA name like filename.%h.%p expands to hostname and PID so processes do not overwrite each other; ~ is not accepted. |
NCCL_DEBUG_TIMESTAMP_FORMAT | Log timestamp formatA strftime format, default [%F %T]; an empty value removes the timestamp. |
NCCL_DEBUG_TIMESTAMP_LEVELS | Levels that get timestampsComma-separated level names or ALL; a leading ^ inverts the list. |
NCCL_TOPO_DUMP_FILE | Dump the detected topologyWrites the topology as NCCL sees it, in XML, creating or overwriting the file. |
NCCL_TOPO_FILE | Load a topology fileRead before detection; by default /var/run/nvidia-topologyd/virtualTopology.xml is loaded when present. |
NCCL_CONF_FILE | Static configuration fileSince 2.23; takes the place of .nccl.conf in the home directory, next to the system-wide /etc/nccl.conf. |
No row on this page matches that filter.
Notes
A first run
./build/all_reduce_perf -b 8 -e 128M -f 2 -g 8 runs all-reduce on eight GPUs in one process, from 8 bytes to 128 MiB, doubling at each step. The tests report the average operation time in ms and two bandwidths in GB/s. Time is the figure to read at small sizes, where it shows latency; at large sizes read busbw, the algorithm bandwidth corrected per collective so that it reflects the bottleneck link (NVLink, PCI, QPI or network) whatever the number of ranks. The other test binaries take the same arguments.
The flags
-b SIZE and -e SIZE set the message range (both default to 32M, so a bare run tests one size); -f FACTOR multiplies the size between steps and -i SIZE adds a fixed increment instead. -g GPUS is GPUs per thread and -t THREADS threads per process. The number of processes is not a test argument: build with MPI=1 and let the launcher set it, as in mpirun -np 64 -N 8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1, which puts 64 single-GPU processes on 8 nodes. -c COUNT checks results for correctness (one iteration by default, and slow on large numbers of GPUs), -w COUNT sets untimed warm-up iterations, and -J FILE writes the results as JSON.
Reading a bad result
When bus bandwidth is far below the link speed, look at what NCCL selected before changing anything: NCCL_DEBUG=INFO prints debug information and NCCL_DEBUG_SUBSYS narrows it, with NET for the network transport, GRAPH for topology detection and TUNING for algorithm and protocol choices. Across nodes, check that NCCL_IB_HCA names the intended adapters and what NCCL_NET_GDR_LEVEL allows for GPU Direct RDMA; within a node, NVIDIA's troubleshooting guide uses nvidia-smi topo -m for the topology and nvidia-smi topo -p2p n to confirm that GPUs reach each other over NVLink. The variables in the guide's Debugging section, such as NCCL_P2P_DISABLE or NCCL_ALGO, are for experiments and should not stay in production scripts.
Questions
Which nccl-tests build do I need for a multi-node run?
One compiled with MPI support: make MPI=1 MPI_HOME=/path/to/mpi CUDA_HOME=/path/to/cuda NCCL_HOME=/path/to/nccl. The tests rely on MPI to run on several processes, so the process count comes from mpirun and each process adds threads times GPUs per thread ranks.
What does busbw mean in the NCCL Tests output?
Bus bandwidth is the algorithm bandwidth (size divided by time) multiplied by a factor that depends on the collective: 2(n-1)/n for all-reduce, (n-1)/n for reduce-scatter, all-gather and all-to-all, and 1 for broadcast and reduce. It is meant to be compared with the hardware peak bandwidth independently of the number of ranks.
How can NCCL Tests exercise only the network between nodes?
Set NCCL_TESTS_SPLIT="MOD 8" on systems with 8 GPUs per node: the ranks are split into 8 groups with one GPU per node each, so every operation runs purely over the inter-node network. The bandwidth printed is per group. NCCL_TESTS_SPLIT="DIV 8" does the opposite and keeps every operation inside one node.
How do I get per-iteration timings out of an NCCL test?
Run with -I 0/1 set to 1 and -J FILE with a .json name. The summary columns are added to the normal output, the JSON keeps the raw per-iteration data, and python3 analyze_perf_json.py results.json from the repository's tools directory analyses it. Per-iteration timing cannot be combined with CUDA graph capture.
Why should NCCL_ALGO or NCCL_P2P_DISABLE not stay set after a test?
The NCCL User Guide lists them under Debugging: forcing them prevents NCCL from selecting the best setting automatically, and newer NCCL versions may behave differently. They are fine for experiments or for isolating a problem, not for production code.
Sources and checks
- NVIDIA nccl-tests README, release tag v2.21.1 Build lines, example runs, the argument list, NCCL_TESTS_SPLIT and comm_ops_perf
- nccl-tests v2.21.1 source file src/common.cu (built-in usage text) The -h help option and the operation names the help prints
- nccl-tests v2.21.1 src/Makefile The list of test programs the build produces
- nccl-tests v2.21.1 source file src/alltoallv.cu (header comment) alltoallv_perf example runs and its environment variables
- nccl-tests v2.21.1 tools/analyze_perf_json.py (usage text) Commands of the JSON analysis script
- NVIDIA NCCL User Guide: Environment Variables (edition covering NCCL up to 2.32) NCCL transport and debugging variables
- nccl-tests v2.21.1 doc/PERFORMANCE.md Meaning of the time, algbw and busbw columns
Every row on this page was matched to a line in the documentation above on . A binding that could not be found there is not listed.
Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.