Building and test run 5
| Command | What it does |
|---|---|
gunzip hpl-2.3.tar.gz; tar -xvf hpl-2.3.tar | Unpack the releaseCreates the hpl-2.3 top-level directory that the make commands below are run from. |
make arch=<ARCH> | Build xhplNeeds a Make.ARCH file in the top-level directory, usually adapted from an example in setup; the binary lands in bin/ARCH. |
mpirun -np 4 xhpl | Run HPL benchmarkThe install guide runs this from bin/ARCH, next to the HPL.dat that xhpl reads. |
make clean arch=<ARCH> | Clean the buildRun it after changing HPL_OPTS in Make.ARCH so the next build recompiles every object. |
make build arch=<ARCH> | Rebuild xhplFollows make clean to produce an executable with the new compile options. |
Compile-time options 4
| Command | What it does |
|---|---|
-DHPL_COPY_L | Copy panel L before broadcastAdded to HPL_OPTS in Make.ARCH; off by default, and an MPI user-defined datatype may avoid the copy anyway. |
-DHPL_CALL_CBLAS | Use the C BLAS interfaceWithout it HPL calls the Fortran 77 BLAS; choose only one of the BLAS interfaces or VSIPL. |
-DHPL_CALL_VSIPL | Use the VSIPL libraryTakes the computational kernels from the Vector Signal Image Processing Library instead of a BLAS. |
-DHPL_DETAILED_TIMING | Turn on detailed timersDetailed timing information is not printed unless the build includes this option. |
Input lines for sizes and grids 16
The FAQ's rule of thumb for N is the largest matrix that fits in memory, about 80 percent of the total, counted per process when several share a node.
| Command | What it does |
|---|---|
HPL Linpack benchmark input file | Line 1: titleFree text that HPL skips, handy for noting what the file is meant to test. |
Innovative Computing Laboratory, University of Tennessee | Line 2: second commentAlso ignored, so any text can go on this line. |
HPL.out output file name (if any) | Line 3: output file nameOnly the first word is read, and only when line 4 sends the output to a file. |
6 device out (6=stdout,7=stderr,file) | Line 4: output device6 prints to standard output, 7 to standard error, any other integer writes to the file named on line 3. |
3 # of problems sizes (N) | Line 5: number of problem sizesAt most 20; HPL reads the first integer and ignores the rest of the line. |
3000 6000 10000 Ns | Line 6: problem sizes NMatrix orders to solve, as many as line 5 announces; further numbers on the line are ignored. |
5 # of NBs | Line 7: number of block sizesUp to 20 block sizes, listed on the following line. |
80 100 120 140 160 NBs | Line 8: block sizes NBEach block size is combined with every problem size and grid in the file. |
0 PMAP process mapping (0=Row-,1=Column-major) | Line 9: process mappingOnly matters when nodes have several processors; the tuning guide recommends row-major, value 0. |
2 # of process grids (P x Q) | Line 10: number of gridsUp to 20 grids, which xhpl runs one after another. |
1 2 Ps | Line 11: process rows PRead in pairs with line 12; here the grids are 1 by 6 and then 2 by 8. |
6 8 Qs | Line 12: process columns QStart at least as many MPI processes as the largest P times Q; any extra processes stay idle. |
4 2 Ps | Rows for a 52-process gridWith the Qs line below, the first grid is 4 by 13, so all 52 started processes take part. |
13 8 Qs | Columns for a 52-process gridPairs 13 with the 4 on the Ps line; the second pair gives a 2 by 8 grid. |
16.0 threshold | Line 13: residual thresholdRuns whose residual check exceeds it are flagged as failed; a threshold of 0.0 fails every test. |
-16.0 threshold | Skip the residual checksAny negative value bypasses the checks, which saves time during long tuning sweeps. |
Input lines for algorithm variants 22
xhpl runs every combination of the listed sizes, block sizes, grids and variants, so each extra value multiplies the number of runs.
| Command | What it does |
|---|---|
1 # of panel fact | Line 14: number of PFACTsHow many panel factorization variants the next line lists; the tuning guide starts with one. |
1 PFACTs (0=left, 1=Crout, 2=Right) | Line 15: panel factorizationCrout is the guide's starting point; left-looking, Crout and right-looking give the same numerical result. |
0 1 2 PFACTs (0=left, 1=Crout, 2=Right) | Try all three PFACTsNeeds 3 on line 14, and each variant then gets its own run. |
2 # of recursive stopping criterium | Line 16: number of NBMINs |
4 8 NBMINs (>= 1) | Line 17: recursion stop widthRecursion ends once a panel is NBMIN columns wide or narrower; the guide starts with 4 and 8. |
1 # of panels in recursion | Line 18: number of NDIVs |
2 NDIVs | Line 19: subpanels per stepEach recursion step splits the panel into this many parts; classic recursion uses 2. |
1 # of recursive panel fact. | Line 20: number of RFACTs |
2 RFACTs (0=left, 1=Crout, 2=Right) | Line 21: recursive factorizationVariant for the recursive part of the panel factorization; right-looking is the guide's first pick. |
2 # of broadcast | Line 22: number of broadcasts |
1 3 BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM) | Line 23: panel broadcastThe guide's first guess, two modified ring topologies; 4 or 5 can win when nodes are fast relative to the network. |
1 BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM) | Use one broadcast topologyIncreasing ring modified, the choice most runs want; set line 22 to 1. |
2 # of lookahead depth | Line 24: number of depths |
0 1 DEPTHs (>=0) | Line 25: look-ahead depthTimes no look-ahead and depth 1; depth 1 usually wins but may need a large N to show it. |
1 DEPTHs (>=0) | Fix look-ahead at depth 1The next panel is factorized as soon as it is updated; set line 24 to 1. |
2 SWAP (0=bin-exch,1=long,2=mix) | Line 26: swapping algorithmMix uses binary exchange for narrow row panels and spread-roll once they exceed the line 27 threshold. |
1 SWAP (0=bin-exch,1=long,2=mix) | Spread-roll swapping onlyLikely the more efficient algorithm on large problems; the threshold on line 27 is then ignored. |
60 swapping threshold | Line 27: swap thresholdColumn count where mix switches algorithms; the guide suggests a value close to the block size. |
0 L1 in (0=transposed,1=no-transposed) form | Line 28: L1 storageStorage form of the upper triangle of the column panel; the guide says it hardly matters. |
0 U in (0=transposed,1=no-transposed) form | Line 29: U storageTransposed storage of the row panel is the clear recommendation unless the GEMM kernel handles it badly. |
1 Equilibration (0=no,1=yes) | Line 30: equilibrationHas an effect only with swap setting 1 or 2; its overhead is small, so the guide keeps it on. |
8 memory alignment in double (> 0) | Line 31: memory alignmentAlignment of HPL allocations, counted in doubles; 4 is usually enough and 8 is the safe pick. |
GPU runs with hpl.sh 6
NVIDIA HPL reads the standard HPL.dat but ignores its broadcast, panel factorization, look-ahead, L1, U, equilibration and alignment lines.
| Command | What it does |
|---|---|
srun -N 16 --ntasks-per-node=4 --cpu-bind=none --mpi=pmix ./hpl.sh --dat ./hpl-linux-x86_64/sample-dat/HPL-64GPUs.dat | Run via SlurmNVIDIA HPL on 16 nodes with 4 GPUs each, one rank per GPU, using the bundled 64-GPU sample input. |
srun -N 1 --ntasks-per-node=4 --cpu-bind=none --mpi=pmix ./hpl.sh --dat ./hpl-linux-x86_64/sample-dat/HPL-4GPUs.dat --cpu-affinity 0-71:72-143:144-215:216-287 --mem-affinity 0:1:2:3 | One Grace Hopper x4 nodeBinds each of the four ranks to its own 72-core range and NUMA node. |
CONT='nvcr.io#nvidia/hpc-benchmarks:26.02.01' | Name the container imageImage reference that the Pyxis srun examples pass to --container-image. |
srun -N 1 --cpu-bind=none --mpi=pmix --container-image="${CONT}" --container-mounts="${MOUNT}" --pty bash | Shell inside the containerWith MOUNT set to "$PWD:/home_pwd", copy the sample Slurm scripts and input files out through /home_pwd. |
srun -N 8 --ntasks-per-node=8 --cpu-bind=none --mpi=pmix --container-image="${CONT}" ./hpl.sh --dat /workspace/hpl-linux-x86_64/sample-dat/HPL-64GPUs.dat | NVIDIA container runEight nodes with 8 GPUs each through Pyxis and Enroot; pulling NGC images needs an enroot .credentials file. |
srun -N 2 --ntasks-per-node=2 --cpu-bind=none --mpi=pmix ./hpl-aarch64.sh --dat /my-dat-files/HPL.dat --cpu-affinity 0-71:72-143 --mem-affinity 0:1 | Run on Grace CPU nodesUses the CPU-only build through hpl-aarch64.sh, two ranks per node, each tied to one NUMA node. |
Options for hpl.sh 11
NVIDIA HPL and HPL-MxP expect one GPU per MPI process: start as many ranks as there are GPUs.
| Command | What it does |
|---|---|
--dat PATH | Input file pathThe only required option; the file uses the same format as Netlib HPL.dat. |
--cuda-compat | Enable CUDA forward compatibility |
--no-multinode | Single node without a network |
--gpu-affinity LIST | GPU per rankColon-separated GPU indices such as 0:1:2:3, one GPU for each MPI process. |
--cpu-affinity LIST | CPU cores per rankColon-separated core ranges on the local node, for example 0-71:72-143. |
--mem-affinity LIST | NUMA node per rankColon-separated NUMA node indices on the local node, for example 0:1:2:3. |
--ucx-affinity LIST | UCX device per rank |
--ucx-tls TRANSPORT | Choose the UCX transport |
--exec-name FILE | Choose a different HPL executable |
--mem-affinity 0:0:0:0:1:1:1:1 --cpu-affinity 0-13:14-27:28-41:42-55:56-69:70-83:84-97:98-111 | DGX H100 and B200 affinityEight ranks with 14 cores each, the first four on NUMA node 0 and the rest on node 1. |
--mem-affinity 2:3:0:1:6:7:4:5 --cpu-affinity 32-47:48-63:0-15:16-31:96-111:112-127:64-79:80-95 | DGX A100 affinityEight ranks with 16 cores each; on this system the NUMA list does not run in ascending order. |
GPU tuning variables 15
| Command | What it does |
|---|---|
HPL_P2P_AS_BCAST | Final solve communicationLibrary for the last solve step: NCCL broadcast or send/recv (default 1), CUDA-aware MPI, host MPI or NVSHMEM, numbered 0 to 4. |
HPL_USE_NVSHMEM | Toggle NVSHMEMNVSHMEM support is on by default; set 0 to run without it. |
HPL_NVSHMEM_INIT | NVSHMEM start-up method0, the default, initializes NVSHMEM through MPI; 1 uses a unique ID. |
HPL_FCT_COMM_POLICY | Factorization communicationLibrary for panel factorization traffic: 0 NVSHMEM (default) or 1 host MPI. |
HPL_NVSHMEM_SWAP | Row swaps over NVSHMEMOff by default, when row swaps go through NCCL; set 1 to switch them to NVSHMEM. |
HPL_CHUNK_SIZE_NBS | Blocks grouped per computationHow many NB-sized matrix blocks are processed together; the default is 16. |
HPL_DIST_TRSM_FLAG | Distributed triangular solve1, the default, spreads the TRSM solve step across ranks instead of only the owners of that part. |
HPL_CTA_PER_FCT | Thread blocks for factorizationNumber of CTAs the factorization uses; the default is 16. |
HPL_ALLOC_HUGEPAGES | Hugepages for host memoryUses 2 MB pages through madvise, which needs transparent_hugepage set to madvise. |
WARMUP_END_PROG | Warm-up stop pointA warm-up pass of the main loop stops at this percentage before the measured run; default 5. |
TEST_LOOPS | Repeat the main loopNumber of times the main loop runs; the default is a single pass. |
HPL_CUSOLVER_MP_TESTS | Component self-testsOn by default: tests individual pieces such as GEMMs and communication; 0 turns them off. |
HPL_CUSOLVER_MP_TESTS_GEMM_ITERS | GEMM repeats in self-testsHow many GEMM calls each component test repeats; the default is 128. |
HPL_WARMUP_MPI | Warm up MPI firstOff by default; 1 makes every MPI call once before the main loop so MPI can set up its internals. |
HPL_FCT_CHUNK_SIZE | Factorization chunk sizeAccepts 32, 64 or 128; the default is 32. |
Out-of-core and FP64 emulation variables 9
| Command | What it does |
|---|---|
HPL_OOC_MODE=1 | Turn on out-of-core modeMatrix data beyond GPU memory is kept in host memory, so raise N in HPL.dat to use it. |
HPL_OOC_MAX_GPU_MEM | Cap GPU memory for OOCLimit in GiB on the GPU memory out-of-core mode may use; default -1. |
HPL_OOC_TILE_M | OOC row blockingRow blocking factor in out-of-core mode; the default is 4096. |
HPL_OOC_TILE_N | OOC column blockingColumn blocking factor in out-of-core mode, also 4096 by default. |
HPL_OOC_NUM_STREAMS | OOC stream countStreams used for out-of-core operations; the default is 3. |
HPL_OOC_SAFE_SIZE | GPU memory kept freeGiB left for the driver, default 3.0; raise it when out-of-core runs hit GPU out-of-memory errors. |
HPL_EMULATE_DOUBLE_PRECISION | FP64 emulation modeBlackwell GPUs: 1 emulates DGEMM with Ozaki-I, 2 with Ozaki-II, 0 keeps native FP64. |
HPL_DOUBLE_PRECISION_EMULATION_MANTISSA_BIT_COUNT | Mantissa bits for Ozaki-IDefault 53, counting the implicit bit; ignored unless the emulation mode is 1. |
HPL_DOUBLE_PRECISION_EMULATION_NUM_MODULI | Moduli for Ozaki-IIDefault 15, at most 20; read only in mode 2 and marked experimental. |
Mixed-precision runs 15
| Command | What it does |
|---|---|
srun -N 4 --ntasks-per-node=8 --cpu-bind=none --mpi=pmix ./hpl-mxp.sh --n 280000 --nb 2048 --nprow 8 --npcol 4 --nporder row --gpu-affinity 0:1:2:3:4:5:6:7 | HPL-MxP on 32 GPUsFour nodes with eight GPUs each, an 8 by 4 row-major grid and a matrix order of 280000. |
srun -N 4 --ntasks-per-node=1 ./hpl-mxp.sh --n 380000 --nb 2048 --nprow 2 --npcol 2 --nporder row --cpu-affinity 0-71 --mem-affinity 0 --gpu-affinity 0 | HPL-MxP on Grace Hopper nodesOne rank per node, pinned to cores 0-71, NUMA node 0 and GPU 0, on a 2 by 2 grid. |
srun -N 4 --ntasks-per-node=4 --container-image="${CONT}" ./hpl-mxp.sh --n 280000 --nb 2048 --nprow 4 --npcol 4 --nporder row --gpu-affinity 0:1:2:3 --cpu-affinity 0-55:56-111:112-167:168-223 | HPL-MxP from the containerSixteen GPUs on four nodes; NVIDIA warns that CPU core binding strongly affects the score. |
./hpl-mxp-aarch64.sh --n 16384 --nb 1023 --nprow 1 --npcol 1 --cpu-affinity 0-71:72-143 --mem-affinity 0:1 | HPL-MxP on Grace CPUTies each rank to one NUMA node, the binding NVIDIA recommends on Grace CPU. |
--n SIZE | Matrix order NRequired; HPL-MxP takes its problem description from flags like this one. |
--nb SIZE | Panel width NBRequired blocking constant; the NVIDIA examples use 2048. |
--nprow ROWS | Process grid rows |
--npcol COLUMNS | Process grid columns |
--nporder row | Row-major grid orderRequired; pass column instead for a column-major layout. |
--tolerance FLOAT | Harness toleranceTolerance applied by the HPL harness; the default is 1e-12. |
--test-loop LOOPS | Repeat the test |
--sloppy-type FP8 | Use FP8 as low precisionAlso written 1; FP16, or 2, is the default and FP4 is the other choice. |
--monitor-gpu 1 | Watch GPU health while runningChecks every second by default; pair it with the clock, power, temperature and PCIe warning thresholds. |
--use-host-mpi | Avoid CUDA-aware MPI |
--help | List every HPL-MxP optionPrints the full option set, including the tuning flags left out here, then exits. |
No row on this page matches that filter.
Notes
Launching
The reference build produces one executable, xhpl, which reads HPL.dat from the directory it runs in; mpirun -np 4 xhpl, started in bin/ARCH, is the quick check the install guide gives. Start at least as many MPI processes as the largest P × Q in the file: extra processes stay idle rather than joining the grid. On NVIDIA GPU systems the hpl.sh wrapper starts xhpl with one GPU per rank and takes the placement as lists, for example --gpu-affinity LIST, --cpu-affinity LIST and --mem-affinity LIST; NVIDIA's srun examples switch Slurm's own binding off with --cpu-bind=none.
Tuning HPL.dat
Aim N at the largest matrix that fits in memory; the FAQ's rule of thumb is about 80 percent of total memory, and a size that is too large makes the node swap and the score drop. Good block sizes almost always lie between 32 and 256, ideally a small multiple of the blocking factor of the matrix-multiply library. Keep the grid close to square with Q slightly larger than P (2 × 4, 4 × 4, 4 × 8); on a plain Ethernet network, flat grids such as 1 × 8 tend to do better. For the algorithm lines, the tuning guide's starting set is Crout panel factorization, 4 and 8 for NBMIN, 2 NDIVs, right-looking RFACT, broadcasts 1 and 3, look-ahead depths 0 and 1 and mixed swapping with a threshold near NB.
Reading the result
xhpl prints every parameter it read, then one result per combination with N, NB, P, Q, the time and the Gflops rate, followed by the residual check against the threshold on line 13. A residual a little above 16.0 is flagged as failed although the answer can still be taken as correct; a run is really wrong only when the residual is several orders of magnitude above 1. -16.0 threshold skips the check during tuning sweeps; leave it positive for the run you report.
Questions
Where does xhpl look for its input file?
In the directory it is started from: HPL.dat sits next to the executable in bin/ARCH, and the install guide runs mpirun -np 4 xhpl from there. NVIDIA's hpl.sh takes the path instead, through --dat PATH.
How large should N be in HPL.dat?
The largest problem that fits in memory, roughly 80 percent of the total. Memory use is essentially the N × N matrix of 8-byte numbers, and when several processes share a node, what counts is the memory available to each one.
Why does xhpl report FAILED for a run that looks fine?
The residual is compared with the threshold on line 13 (16.0 threshold). Values slightly above it, such as 35.6, are flagged but can be considered correct; a threshold of 0.0 fails every run, and a negative one skips the check.
Which HPL.dat lines does NVIDIA HPL ignore?
The broadcast, panel factorization, look-ahead, L1 and U layout, equilibration and memory alignment lines. N, NB and the process grid still come from the file, and the GPU version adds its own environment variables such as HPL_OOC_MODE=1 for out-of-core runs.
Sources and checks
- HPL 2.3 release (netlib.org, December 2, 2018): TUNING file, "Performance Tuning and setting up the input data file HPL.dat" Every HPL.dat line, its meaning and the tuning guide lines
- HPL 2.3 release (netlib.org): INSTALL file Build, quick check run and rebuild commands
- HPL 2.3 release (netlib.org): default input file testing/ptest/HPL.dat The process mapping line (line 9), which the tuning text describes but does not print
- Netlib: HPL Software (download, installation and compile time options) Unpack command and the compile time options table
- Netlib: HPL Frequently Asked Questions Choosing N, NB and the P x Q grid
- NVIDIA HPC Benchmarks documentation: NVIDIA HPL Benchmark (package 26.02.01) hpl.sh and hpl-aarch64.sh options, example runs and environment variables
- NVIDIA HPC Benchmarks documentation: NVIDIA HPL-MxP Benchmark (package 26.02.01) hpl-mxp.sh options and example runs
- NVIDIA NGC catalog: HPC-Benchmarks container overview, image hpc-benchmarks:26.02.01 Pyxis/Enroot container runs
Every row on this page was matched to a line in the documentation above on . A binding that could not be found there is not listed.
8 rows from the earlier version of this page were removed in that check: duplicates, and bindings the documentation does not list or lists differently.
Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.