GPU Cluster Management: Essential Commands for Operators
Published: July 8, 2026 · 9 min read
Day-to-day GPU cluster operations come down to five toolboxes: nvidia-smi for device state, NVSM and DCGM for health checks on DGX-class systems, ipmitool for out-of-band power and sensors, the InfiniBand utilities for fabric diagnostics, and a vendor BMC CLI such as Dell's racadm. This page collects the commands an on-call operator actually types, in the order a typical incident uses them.
Table of Contents
What nvidia-smi commands should every operator know?
| Command | What it does |
|---|---|
| nvidia-smi | One-shot summary: utilization, memory, temperature, processes |
| nvidia-smi -l 1 | Refresh the summary every second |
| nvidia-smi -L | List GPUs with UUIDs |
| nvidia-smi -q | Full detailed query for one-off inspection |
| nvidia-smi topo -m | GPU/NIC topology matrix — NVLink vs PCIe paths |
| nvidia-smi nvlink -s | NVLink status per link |
| nvidia-smi -pm 1 | Enable persistence mode (do this on every node) |
| nvidia-smi -i 0 -pl 250 | Set a 250 W power limit on GPU 0 |
| nvidia-smi --query-gpu=utilization.gpu,temperature.gpu --format=csv -l 5 | Machine-readable logging every 5 s |
| nvidia-smi --query-compute-apps=pid,used_memory --format=csv | Which process is holding memory |
--query-gpu ... --format=csv family is the automation workhorse. Pipe it into your monitoring agent instead of scraping the human-readable table — the columns are stable across driver versions.How do you check node health on NVIDIA DGX systems?
DGX systems add two layers above the driver. NVSM answers “is this box healthy?” in one command, and DCGM runs active diagnostics when you suspect a GPU:
| Command | What it does |
|---|---|
| nvsm show health | Aggregate system health verdict |
| nvsm show gpu / nvsm show nvswitch | Per-GPU and NVSwitch status |
| nvsm dump | Full support bundle for vendor escalation |
| dcgmi diag -r 3 | Level-3 GPU diagnostic (takes several minutes, run before RMA) |
| dcgmi health -c -g 0 | DCGM background health check on GPU group 0 |
How do you manage servers out-of-band with ipmitool?
When a node stops answering SSH, the BMC still answers. These work locally or remotely with -I lanplus -H <bmc-ip> -U <user>:
| Command | What it does |
|---|---|
| ipmitool power status | Is the box actually on? |
| ipmitool power on / off | Hard power control |
| ipmitool power cycle | Full power cycle — the last resort that usually works |
| ipmitool sensor list | Temperatures, fans, voltages |
| ipmitool sel list | System event log — check this before blaming software |
Which InfiniBand commands diagnose the fabric?
Multi-node training jobs live and die by the fabric. The diagnostic ladder, from cheapest to most invasive:
| Command | What it does |
|---|---|
| ibstat / ibstatus | Local port state and rate — is the link even up? |
| ibv_devinfo | HCA device and firmware details |
| iblinkinfo | Every link in the fabric with speed and state |
| ibhosts / ibswitches | Inventory of hosts and switches the subnet manager sees |
| perfquery | Port error and traffic counters — rising symbol errors mean a bad cable |
| ib_write_bw / ib_read_bw | Point-to-point bandwidth test between two nodes |
| ib_send_lat | Latency test |
| ibping [LID] | Reachability check at the IB layer |
What racadm commands matter on Dell nodes?
On Dell servers, iDRAC's CLI covers what ipmitool can't reach — RAID state and structured inventory:
| Command | What it does |
|---|---|
| racadm getsysinfo | Service tag, firmware, and system summary |
| racadm serveraction poweron / poweroff / hardreset | Power control via iDRAC |
| racadm getsel | iDRAC event log |
| racadm raid status | Controller and virtual disk health |
How do these fit into an on-call triage flow?
A sequence that resolves most “node is sick” pages: start with nvidia-smi — if a GPU is missing from the list or shows ERR!, go straight to dcgmi diag -r 3 and nvsm show health. If the job is slow rather than dead, check the fabric with ibstat and perfquery for climbing error counters. If the node is unreachable entirely, pivot to the BMC: ipmitool power status, read sel list for hardware events, and only then power cycle. Log a nvsm dump before any RMA conversation. And when the recurring fix turns out to be “we need more GPUs,” that's a capacity question, not an ops one — our GPU sizing calculator at gpusize.com estimates cluster requirements for training and inference workloads.
Two habits keep the pager quiet in the first place. Enable persistence mode (nvidia-smi -pm 1) at boot on every node — without it, driver reinitialization adds seconds of latency to the first CUDA call and can surface as mysterious job-start timeouts. And wire dcgmi health into your scheduler's node health checks or job epilog, so a degrading GPU is drained from the pool automatically instead of failing the next user's training run at hour nine.
Each tool above has a full reference page with descriptions and printable cheat sheets: nvidia-smi, NVIDIA DGX, NVIDIA DCGM (dcgmi), ipmitool, Mellanox / InfiniBand, and Dell iDRAC racadm.
Once the hardware is healthy, the next layer is scheduling and launching the actual workload: Slurm for job queuing, and torchrun or OpenMPI (mpirun) for launching the distributed training job itself.
None of that matters if the GPUs are starved for data. The parallel filesystem underneath a training cluster is just as much a performance lever as the network fabric: WEKA and DDN EXAScaler (Lustre) are the two most common choices purpose-built for AI throughput, while NetApp ONTAP, VAST Data, and Pure Storage FlashBlade cover the broader enterprise-storage side of the same job.
Once every layer is in place, the last step is proving it actually performs before real training jobs go anywhere near it. NCCL Tests is the standard way to validate GPU-to-GPU bandwidth over NVLink and InfiniBand, while OSU Micro-Benchmarks and Intel MPI Benchmarks check general MPI latency and bandwidth, and HPL (Linpack) is the classic full-system floating-point benchmark behind the TOP500 list. For the AI-specific angle — training and inference throughput on real models rather than raw hardware limits — see our guide to benchmarking a GPU cluster with MLPerf.
Frequently asked questions
How do I monitor GPU usage in real time?
Run nvidia-smi -l 1 to refresh the summary every second, or watch -n 1 nvidia-smi. For logging and dashboards, use nvidia-smi --query-gpu=utilization.gpu,temperature.gpu --format=csv -l 5, which emits stable machine-readable output.
How do I check NVLink status on a GPU server?
nvidia-smi nvlink -s shows the state of each NVLink link, and nvidia-smi topo -m prints the topology matrix showing which GPU pairs communicate over NVLink versus PCIe.
What is the difference between ipmitool and racadm?
ipmitool speaks the vendor-neutral IPMI protocol and works against almost any BMC for power control, sensors, and event logs. racadm is Dell's iDRAC-specific CLI with deeper access to Dell features such as RAID status and structured system inventory.
How do I run a full GPU diagnostic on a DGX system?
Use dcgmi diag -r 3 for a level-3 DCGM diagnostic, which takes several minutes and is the standard evidence to collect before an RMA. Pair it with nvsm show health for the system-level verdict and nvsm dump to produce a support bundle.