How to Benchmark a GPU Cluster: NCCL, OSU, HPL, and MLPerf

Published: July 9, 2026 · 9 min read

A GPU cluster that boots successfully isn't the same as a GPU cluster that performs. Drivers can load, Slurm can schedule, and storage can mount while the interconnect quietly runs at half the bandwidth it should — and the first place that shows up is a mysteriously slow training run, days after everyone assumed the hardware was fine. Benchmarking is the step between "it's up" and "it's ready," and it happens in a fixed order: network and compute microbenchmarks first, a full-system stress test second, and application-level benchmarks like MLPerf last, once you already trust the layers underneath.

Why benchmark in this order?

Each layer assumes the one below it is healthy, so testing them out of order wastes time chasing the wrong problem. A slow MLPerf run could mean a slow GPU, a slow interconnect, a slow filesystem, or an inefficient model implementation — four different fixes. Start narrow and cheap (does this one link move data at the rate it should?) and only move up to expensive, multi-hour application benchmarks once the fundamentals check out:

1. Interconnect microbenchmarks — NCCL Tests, in minutes, per pair or per node.
2. General MPI health — OSU / IMB, confirms the fabric works for non-NCCL workloads too.
3. Full-system stress — HPL, hours, proves sustained load across every node at once.
4. Application-level — MLPerf, the slowest and most realistic, run only once the above pass.

How do you test GPU interconnect bandwidth?

NCCL Tests is the standard first check on any NVIDIA cluster — it isolates exactly the collective operations (all-reduce, all-gather, broadcast) that distributed training depends on, separate from anything MPI or the filesystem might be doing wrong.

How do you check general MPI communication health?

NCCL Tests only exercises NVIDIA's own collective library. OSU Micro-Benchmarks and Intel MPI Benchmarks (IMB) test the underlying MPI implementation itself — useful because plenty of HPC and simulation workloads on the same cluster never touch NCCL at all.

Run both suites once after any driver, MPI library, or NIC firmware update — regressions here are the most common silent cause of "the cluster got slower" tickets.

How do you stress-test the whole cluster at once?

HPL (Linpack) is the benchmark behind the TOP500 supercomputer ranking. It solves a large dense linear system across every node simultaneously, which makes it the closest thing to a full-system burn-in test: it will surface a bad DIMM, a throttling node, or a marginal power supply that microbenchmarks on individual links won't catch.

What is MLPerf, and how is it different?

MLPerf, maintained by MLCommons, is the industry-standard suite for measuring training and inference speed on real models rather than raw hardware limits — Training covers time-to-accuracy on models like ResNet and BERT-class LLMs, while Inference covers throughput and latency under fixed accuracy targets. It answers a different question than the tools above: not "is my hardware healthy" but "how fast can this specific model actually run here."

The practical difference shows up immediately in the CLI. MLPerf's automation tooling (mlcr, part of MLCommons' MLCFlow project) doesn't have a short, memorizable command set the way mpirun or nccl-tests do — every invocation is parameterized by model, framework, scenario, and category, and it changes with each release:

Each combination of model, scenario (Offline / Server / SingleStream), and framework is effectively its own long command copied from the docs rather than typed from memory — which is exactly why we cover MLPerf here as a workflow rather than as a shortcut table.

A practical starting sequence: install the CLI (pip install mlc-scripts), pick one reference model close to your actual workload (ResNet50 or BERT are the fastest to validate the pipeline), run it once in _find-performance mode to estimate a target, then run the real _performance-only and _accuracy-only passes. Treat the first full run as a pipeline test, not a result you'd report — official submissions have strict rules about divisions (Closed vs Open) and categories (Available vs Preview vs RDI) that don't matter for internal validation.

How do you read and compare the results?

• NCCL Tests / OSU / IMB: compare busbw or bandwidth (GB/s) against the link's theoretical peak — NVLink, PCIe, or your InfiniBand line rate. Consistently below ~80% of peak on large messages usually points at topology or NIC configuration, not a fundamentally broken link.
• HPL: compare achieved GFLOPS against the theoretical peak FLOPS of your CPUs (or GPUs, for GPU-accelerated HPL builds like NVIDIA's HPC-Benchmarks container). 70–90% of peak is typical for a well-tuned run.
• MLPerf: compare against MLCommons' published results for similar hardware on the same benchmark and scenario — the public results dashboard is the fairest baseline, since raw throughput numbers alone don't mean much without a reference point.

Keep a baseline from day one. The first clean benchmark run after a cluster is commissioned is the number every future regression gets compared against — without it, "did that firmware update slow things down?" becomes a guess instead of a diff.

This post is part of our GPU cluster series — see also GPU cluster management: the full command guide for the hardware, storage, and orchestration layers this benchmarking sequence assumes are already in place.

Frequently asked questions

What order should I run GPU cluster benchmarks in?

Start narrow and cheap, then move to expensive and realistic: NCCL Tests for interconnect bandwidth, OSU or IMB for general MPI health, HPL for a full-system stress test, and MLPerf last for application-level training or inference throughput. Each layer assumes the one below it is already healthy.

What is the difference between NCCL Tests and MLPerf?

NCCL Tests measures raw GPU-to-GPU communication bandwidth using NVIDIA's collective library, isolated from any specific model. MLPerf measures end-to-end training or inference speed on real reference models, which depends on the GPU, interconnect, storage, and software stack all working together.

What does the busbw column in NCCL Tests mean?

Bus bandwidth is algorithm bandwidth adjusted by a formula specific to each collective operation so that it can be compared directly against hardware peak bandwidth, independent of how many ranks were used. It is the number to compare against NVLink or InfiniBand theoretical peak, not algbw.

Do I need MLPerf if I already ran HPL?

HPL and MLPerf test different things. HPL proves the cluster can sustain heavy floating-point load across every node, which is a good full-system health check, but it does not reflect the memory access patterns, communication frequency, or precision (fp16/bf16) of actual deep learning training. MLPerf is the closer proxy if the cluster's job is AI training or inference specifically.