Status at a glance 13
| Command | What it does |
|---|---|
nvidia-smi | GPU status summaryWith no options it prints a single summary table of all GPUs in the system. |
nvidia-smi -col | Multi-column summaryLong form --columns; prints the same summary in a multi-column layout. |
nvidia-smi -i 0 | Summary for one GPULimits output to GPU 0; -i also takes a UUID, a PCI bus ID or a board serial number. |
nvidia-smi -L | List GPUsLong form --list-gpus; prints one line per GPU with its UUID, and MIG devices beneath their parent. |
nvidia-smi -B | List excluded GPUsLong form --list-excluded-gpus; prints the GPUs marked as excluded, each with its UUID. |
nvidia-smi -q | Detailed infoPrints every attribute of all GPUs once as plain text; values a device cannot report show N/A. |
nvidia-smi -q -x | Full report as XMLLong form --xml-format; adding --dtd embeds the document type definition in the output. |
nvidia-smi -q -d UTILIZATION | Utilization detailsShows only the utilization section, where NVJPG and NVOFA figures sit next to GPU and memory. |
nvidia-smi -q -d MEMORY | Memory detailsShows only the memory section of the full query; several types can be joined with commas. |
nvidia-smi -q -d TEMPERATURE | Temperature detailsLimits the full query to the temperature section. |
nvidia-smi -f FILE | Write output to a fileLong form --filename=FILE; an existing file of that name is overwritten, and nothing goes to stdout. |
nvidia-smi --version | Show version information |
nvidia-smi -h | Show built-in helpLists all options and subcommands; each subcommand prints its own page with -h, as in nvidia-smi mig -h. |
Querying fields to CSV 7
| Command | What it does |
|---|---|
nvidia-smi --format=csv,noheader --query-gpu=uuid,persistence_mode | CSV queryPrints each GPU's UUID and its persistence mode as comma-separated lines without a header row. |
nvidia-smi -i 0 --query-gpu=pci.bus_id,mig.mode.current --format=csv | Query one GPU as CSVRestricts the query to GPU 0; the first output line names the two columns. |
--query-gpu= | Query GPU propertiesTakes a comma-separated property list, for example pci.bus_id,persistence_mode, and must be paired with --format=csv. |
--format=csv | Output as CSVMandatory with every --query switch; noheader and nounits can follow csv after commas. |
--format=csv,noheader | CSV without the header rowLeaves out the first line, which normally carries the column names. |
nounits | Omit units from valuesAdded to the --format list after csv, it leaves the numbers bare, without unit suffixes. |
nvidia-smi --help-query-gpu | List GPU query fieldsPrints every property name that --query-gpu accepts, each with a short description. |
Query fields for identity and state 16
| Command | What it does |
|---|---|
index | GPU indexZero-based position of the GPU, which may differ after a reboot; prefer uuid or pci.bus_id for stable identification. |
uuid | GPU UUIDAlso accepted as gpu_uuid; a globally unique, immutable identifier that is not printed on the board. |
name | GPU product nameAlso accepted as gpu_name; returns the model name NVIDIA gives the product. |
serial | Board serial numberAlso accepted as gpu_serial; matches the number physically printed on the board. |
pci.bus_id | PCI bus IDAlso accepted as gpu_bus_id; given in hex as domain:bus:device.function. |
count | Number of GPUsReturns how many NVIDIA GPUs are present on the machine. |
timestamp | Time of the queryRecords the date and time of the query with millisecond precision. |
kmd_version | Kernel driver versionReplaces driver_version, which this release marks as deprecated. |
vbios_version | VBIOS version |
persistence_mode | Persistence mode stateEnabled or Disabled; when enabled the driver stays loaded with no clients attached. |
compute_mode | Compute modeDefault allows many contexts per device, Exclusive_Process one, and Prohibited none. |
pstate | Performance stateP0 is the fastest state and P12 the slowest. |
mig.mode.current | Current MIG modeShows whether MIG is enabled now; mig.mode.pending shows the mode that applies after a reset. |
gpu_recovery_action | Recommended recovery actionNames the step that clears an earlier fault: None, Reset, Reboot, Drain P2P, or Drain and Reset. |
pcie.link.gen.gpucurrent | Current PCIe link generationMay read lower than the maximum while the GPU is idle. |
pcie.link.width.current | Current PCIe link widthLike the link generation, the width can drop while the GPU is idle. |
Query fields for load, memory and power 16
| Command | What it does |
|---|---|
utilization.gpu | GPU utilizationPercent of the sample period in which at least one kernel was running; the period is 1/6 to 1 second. |
utilization.memory | Memory read/write activityPercent of the sample period spent reading or writing device memory, not the share of memory allocated. |
memory.total | Installed GPU memory |
memory.used | Memory in useCounts memory allocated by active contexts; driver and firmware overhead is in memory.reserved. |
memory.free | Free GPU memory |
memory.reserved | Driver-reserved memoryCovers what the NVIDIA driver and firmware hold back for themselves. |
temperature.gpu | GPU core temperatureGiven in degrees C; temperature.memory is the separate reading for HBM memory. |
fan.speed | Intended fan speedA percentage of the fan's rated maximum that the fan is asked to run at, not a measured speed. |
power.draw | Board power drawIn watts, accurate to about 5 W; Ampere and newer GPUs report a one-second average. |
power.draw.instant | Instant power drawAlso accepted as gpu.power.draw.instant; the latest instantaneous reading for the whole board, in watts. |
power.limit | Requested power limitThe value -pl changes, in watts; gpu.power.requested_limit is a second name for the same field. |
enforced.power.limit | Enforced power limitThe ceiling the power management algorithm actually holds the board under; the lowest of several limiters. |
power.max_limit | Highest settable power limitUpper bound in watts for a power limit; power.min_limit gives the lower bound. |
clocks.current.graphics | Graphics clock frequencyAlso accepted as clocks.gr; the current shader clock. |
clocks.current.memory | Memory clock frequencyAlso accepted under the shorter name clocks.mem. |
clocks_event_reasons.active | Active clock event reasonsA bitmask explained in nvml.h; the older name clocks_throttle_reasons.active is still accepted. |
Query fields for memory errors 8
| Command | What it does |
|---|---|
ecc.mode.current | Current ECC modeecc.mode.pending shows the mode that takes over after the next reboot. |
ecc.errors.corrected.volatile.total | Corrected ECC errors since driver loadSingle-bit errors the hardware fixed, counted across the whole chip since the driver was last loaded. |
ecc.errors.uncorrected.volatile.total | Uncorrected ECC errors since driver loadDouble-bit errors, which are detected but not fixed; the counter restarts at each driver load. |
ecc.errors.uncorrected.aggregate.total | Lifetime uncorrected ECC errorsAggregate counts persist indefinitely, so this is the total over the life of the board. |
retired_pages.pending | Pages pending retirementReports whether pages await retirement at the next reboot; such pages can still be allocated meanwhile. |
retired_pages.dbe | Pages retired for double-bit errorsShort for retired_pages.double_bit.count; retired_pages.sbe is the single-bit counterpart. |
remapped_rows.uncorrectable | Rows remapped for double-bit errorsAlso accepted as remapped_rows.dbe; remapped_rows.correctable counts rows remapped after repeated single-bit errors. |
remapped_rows.pending | Row remapping pendingReports whether a row remap is waiting; remapped_rows.failure reports a remap that failed. |
Monitoring over time 22
| Command | What it does |
|---|---|
nvidia-smi -l SEC | Repeat at an intervalLong form --loop=SEC; reruns until Ctrl+C, every 5 seconds when no value is given. |
nvidia-smi -lms MS | Repeat every few millisecondsLong form --loop-ms=MS; the same loop as -l with a finer interval unit. |
nvidia-smi -q -d ECC,POWER -i 0 -l 10 -f out.log | Log chosen sections to a fileWrites the ECC and power sections for GPU 0 to out.log every 10 seconds until interrupted. |
nvidia-smi dmon | Device monitorPrints one line of default metrics per GPU every second, for up to 16 GPUs, until Ctrl+C. |
nvidia-smi dmon -i DEVICE1,DEVICE2 | Monitor chosen GPUsTakes a comma-separated device list in place of the first 16 GPUs. |
nvidia-smi dmon -s METRIC_GROUP | Choose metric groupsCombine letters: p power and temperature, u utilization, c clocks, v violations, m memory, e ECC, t PCIe throughput. |
nvidia-smi dmon -c SAMPLES | Stop after N samplesExits by itself once the given number of lines has been printed. |
nvidia-smi dmon -d SECS | Set the sampling intervalChanges the default one-second cycle to the given number of seconds. |
nvidia-smi dmon -o D | Prefix lines with the dateAdds the date in YYYYMMDD form at the start of every line. |
nvidia-smi dmon -o T | Prefix lines with the timeAdds an HH:MM:SS clock time at the start of every line. |
nvidia-smi dmon -f FILE | Log dmon output to a fileLong form --filename; lines go to the file and no longer to stdout. |
nvidia-smi dmon --format csv | CSV output from dmonnounit and noheader, added after commas, drop the unit line and the header line. |
nvidia-smi dmon --gpm-metrics GPMMETRIC1,GPMMETRIC2 | Watch GPM metricsHopper or newer GPUs; the metric IDs are listed by dmon -h, for example 2 for SM activity. |
nvidia-smi daemon | Start background loggingExperimental; samples every 10 seconds and writes date-stamped compressed logs under /var/log/nvstats. |
nvidia-smi daemon -d SECS | Set the logging intervalReplaces the 10-second default with your own number of seconds. |
nvidia-smi daemon -p PATH | Choose the log directoryLog files are created in that directory in place of /var/log/nvstats. |
nvidia-smi daemon -j STRING | Tag the log file nameAppends your string to the name of the log file the daemon creates. |
nvidia-smi daemon -t | Stop the daemonUses the PID stored in /var/run/nvsmi.pid; termination is best effort, so check it has exited. |
nvidia-smi replay -f FILE | Replay a daemon logReads a compressed log and prints one timestamped line of default metrics per stored sample. |
nvidia-smi replay -f FILE -s METRIC_GROUP | Replay chosen metricsUses the same letter groups as dmon -s to pick which stored metrics are shown. |
nvidia-smi replay -f FILE -b HH:MM:SS -e HH:MM:SS | Replay a time windowLimits output to samples between the two times; 0 stands for the start or the end of the file. |
nvidia-smi replay -f FILE -r OUTPUT | Extract a log to textWrites the default metrics from the log into the named output file. |
Per-process information 21
| Command | What it does |
|---|---|
nvidia-smi pmon | Process monitorPrints one line per process on each GPU every second, covering compute and graphics processes. |
nvidia-smi pmon -i DEVICE1,DEVICE2 | Watch processes on chosen GPUsTakes a comma-separated device list and reports only the processes on those GPUs. |
nvidia-smi pmon -s METRIC_GROUP | Choose process metricsu gives average utilization since the last cycle, m gives current memory use; both letters can be combined. |
nvidia-smi pmon -c SAMPLES | Stop pmon after N samplesPrints that many sampling rounds and then exits. |
nvidia-smi pmon -d SECS | Set the pmon intervalAccepts 1 to 10 seconds between samples; the default is 1. |
nvidia-smi pmon -o T | Timestamp each pmon lineAdds HH:MM:SS to each line; -o D adds the date as YYYYMMDD. |
nvidia-smi pmon -f FILE | Log pmon output to a fileLong form --filename; the process lines are written to the file, not to stdout. |
nvidia-smi --show-all-processes | Show every GPU clientAdds clients without a context, such as monitoring tools, to the process list. |
nvidia-smi -q -d PIDS | Process detailsPrints only the process list from the full query, GPU by GPU. |
--query-compute-apps= | Query compute processesTakes fields such as pid, process_name and used_gpu_memory, and needs --format=csv. |
nvidia-smi --help-query-compute-apps | List process query fieldsPrints the field names --query-compute-apps accepts, with a line about each. |
pid | Process IDA --query-compute-apps field that identifies the application holding the context. |
process_name | Process nameA --query-compute-apps field, also accepted under the shorter name, name. |
used_gpu_memory | GPU memory used by a processAlso accepted as used_memory; not available on Windows in WDDM mode. |
nvidia-smi -am 1 | Enable accounting modeKeeps usage statistics for each process over its lifetime; needs administrator rights, and 0 turns it off. |
--query-accounted-apps= | Query accounted processesReads the statistics accounting mode has collected; not supported on a vGPU host. |
nvidia-smi --help-query-accounted-apps | List accounting query fieldsFields include gpu_utilization, mem_utilization, max_memory_usage and time, the active time in milliseconds. |
max_memory_usage | Peak memory of a processA --query-accounted-apps field: the most device memory the context ever held. |
nvidia-smi -caa | Clear accounted processesLong form --clear-accounted-apps; empties the buffer of process IDs accounted so far. |
nvidia-smi -q -d ACCOUNTING | Accounting detailsLimits the full query to the accounting section. |
nvidia-smi encodersessions | List encoder sessionsShows the encoder sessions on each GPU; not supported on a vGPU host. |
Power and clocks 26
A power limit set with -pl is not permanent: it returns to the default power limit when the driver is unloaded.
| Command | What it does |
|---|---|
nvidia-smi -q -d POWER | Power detailsShows only the power section, including the base power settings added in v610. |
nvidia-smi -pl POWER_LIMIT | Set power limitTakes watts, whole or decimal, between the minimum and maximum limits nvidia-smi reports; needs root. |
nvidia-smi -q -d SUPPORTED_CLOCKS | List supported clocksPrints the supported clock frequencies of every GPU. |
nvidia-smi -q -d PERFORMANCE | Clock event reasonsShows why clocks were reduced, with the event reason counters and the sparse operation mode. |
nvidia-smi -q -d CLOCK | Clock detailsLimits the full query to clock frequencies and includes sampled maximum, minimum and average values. |
nvidia-smi -lgc 1500,1500 | Lock GPU clocksSets the minimum and maximum GPU clock in MHz; a single value also works. Volta or newer, root. |
nvidia-smi -rgc | Reset clocksLong form --reset-gpu-clocks; returns locked GPU clocks to their defaults. Needs root, Volta or newer. |
nvidia-smi -lmc 5100,5100 | Lock memory clocksSets the memory clock range in MHz; not for Hopper GPUs, which need the deferred lock, -lmcd. |
nvidia-smi -rmc | Reset memory clocksLong form --reset-memory-clocks; puts memory clocks back to normal and needs root. |
nvidia-smi -lmcd MEMCLOCK | Lock memory clock at next initThe value applies when the GPU is next initialised, for example after reloading the kernel module. |
nvidia-smi -rmcd | Clear the deferred memory lockLong form --reset-memory-clocks-deferred; takes effect once the driver has been unloaded and loaded again. |
nvidia-smi -i 0 --applications-clocks 2500,745 | Set application clocksDeprecated in favour of -lmc and -lgc; here GPU 0 gets a 2500 MHz memory clock and 745 MHz for graphics. |
nvidia-smi -rac | Reset application clocksLong form --reset-applications-clocks; deprecated together with the application clock option it undoes. |
nvidia-smi -cc 1 | Override CUDA clocksLets clocks run higher while CUDA applications run; 0 restores the default. Volta or newer, administrator rights. |
nvidia-smi -gtt TEMP | Set GPU target temperatureTakes degrees Celsius within the limits listed by the SUPPORTED_GPU_TARGET_TEMP query type; needs root. |
nvidia-smi clocks --sync-boost-list | List sync boost groupsPrints the group IDs that --sync-boost-remove expects. |
nvidia-smi clocks -som 1 | Enable sparse operation modeLong form --sparse-operation-mode; 0 disables it, and the current state appears under -q -d PERFORMANCE. |
nvidia-smi power-profiles -l | List workload power profilesPrints every workload power profile that the device supports. |
nvidia-smi power-profiles -ld | List profiles with metadataAdds each profile's ID, its priority, where a lower number ranks higher, and the profiles it conflicts with. |
nvidia-smi power-profiles -gr | Show requested profilesLists what has been requested; conflicting profiles may not all be enforced. |
nvidia-smi power-profiles -ge | Show enforced profilesCan differ from the requested list when the selected profiles conflict. |
nvidia-smi power-profiles -sr PROFILE | Request a power profileAdds one or more profile IDs, comma-separated without spaces, to the requested list; needs root. |
nvidia-smi power-profiles -cr PROFILE | Remove a requested profileTakes the same comma-separated ID list and drops those profiles from the requested set; needs root. |
nvidia-smi power-smoothing -ppd | Show power smoothing presetsPrints the full definition of every preset profile. |
nvidia-smi power-smoothing -spp PROFILE | Activate a smoothing presetMakes the given preset profile ID the active one; needs root. |
nvidia-smi power-smoothing -s STATE | Turn power smoothing on or offSTATE is 1 or ENABLED to switch the feature on, 0 or DISABLED to switch it off; needs root. |
Compute and persistence modes 8
Persistence mode and compute mode change at once but are lost at reboot; an ECC change is the reverse: it waits for the next reboot and then persists.
| Command | What it does |
|---|---|
nvidia-smi -pm 1 | Enable persistence modeKeeps the driver loaded when no client is connected; needs root, takes effect at once and is lost at reboot. |
nvidia-smi -pm 0 | Disable persistence modeLets the driver unload again once the last client exits; this is the state after every reboot. |
nvidia-smi -c 0 | Default compute modeAllows multiple contexts per device; every GPU returns to this mode after a reboot. |
nvidia-smi -c 3 | Exclusive process modeAllows one context per device, usable from several threads; EXCLUSIVE_PROCESS is the same value by name. |
nvidia-smi -c 2 | Prohibited compute modeAllows no compute contexts on the device at all. |
nvidia-smi -i 0 -c EXCLUSIVE_PROCESS | Exclusive mode on one GPUThe step NVIDIA's MPS guide runs as root on GPU 0 before starting the MPS control daemon. |
nvidia-smi compute-policy -l | List compute policiesThe same subcommand sets a GPU's timeslice to default, short, medium or long. |
--gom=MODE | Set GPU operation mode0/ALL_ON, 1/COMPUTE or 2/LOW_DP on Kepler-era Tesla and Titan cards; needs administrator rights and a reboot. |
ECC and resets 14
NVIDIA does not guarantee that a GPU reset works in every case: verify the health of each reset GPU, and power cycle the node if one is not healthy.
| Command | What it does |
|---|---|
nvidia-smi -q -d ECC | ECC error detailsLimits the full query to ECC mode and error counts, with the repair-pending flags added in recent releases. |
nvidia-smi -e 1 | Enable ECCNeeds root; the new mode starts only after the next reboot and then persists. |
nvidia-smi -e 0 | Disable ECCLike enabling, the change waits for a reboot; ecc.mode.pending shows it in the meantime. |
nvidia-smi -p 0 | Reset volatile ECC countersLong form --reset-ecc-errors=0; takes effect immediately and needs root. |
nvidia-smi -p 1 | Reset aggregate ECC countersClears the lifetime counts; the manual says this is not supported on Ampere and newer GPUs. |
nvidia-smi -i GPU -q -d PAGE_RETIREMENT | Retired page summaryShows retired page counts for one GPU and whether any page is still pending. |
--query-retired-pages= | Query retired pagesFields include retired_pages.address, retired_pages.cause and retired_pages.timestamp; needs --format=csv. |
nvidia-smi --help-query-retired-pages | List retired page fieldsPrints the field names with descriptions, including the two possible retirement causes. |
nvidia-smi -q -d ROW_REMAPPER | Row remapping detailsLimits the full query to the row remapper section. |
--query-remapped-rows= | Query remapped rowsFields include remapped_rows.correctable, remapped_rows.uncorrectable, remapped_rows.pending and remapped_rows.failure. |
nvidia-smi --help-query-remapped-rows | List remapped row fieldsPrints the field names for --query-remapped-rows, among them the histogram buckets from max to none. |
nvidia-smi -r | Reset GPULong form --gpu-reset; resets every GPU unless -i names some, needs root and no process on the device. |
nvidia-smi -i GPU -r | Reset one GPUTargets a single device; NVLink peers may have to be reset in the same command. |
nvidia-smi -r bus | Bus resetRequests a bus reset; plain -r does a function level reset, the only kind on some platforms. |
MIG partitioning 15
Enabling MIG mode is not enough to run CUDA workloads: GPU instances and compute instances must exist, and they have to be created again after every reboot.
| Command | What it does |
|---|---|
nvidia-smi -i 0 -mig 1 | Enable MIG modeRun as root; -i accepts an index, a PCI bus ID or a UUID, and without it every GPU is switched. |
nvidia-smi -mig 0 | Disable MIG modeLong form --multi-instance-gpu=0; needs root. On Ampere GPUs the mode otherwise survives reboots. |
nvidia-smi mig -lgip | List GPU instance profilesShows the instance sizes the GPU offers, with the profile IDs and names that -cgi accepts. |
nvidia-smi mig -lgipp | List possible placementsPrints each profile's placements as {start indexes}:size, the slots where an instance can sit. |
nvidia-smi mig -cgi 9,3g.20gb -C | Create GPU and compute instancesCreates two 3g.20gb GPU instances, one by profile ID and one by name; -C adds a default compute instance to each. |
nvidia-smi mig -cgi 19,14,5 | Create mixed-size instancesBuilds a 4-2-1 layout from three profile IDs; the driver picks where each instance goes. |
nvidia-smi mig -cgi 19:2 | Create at a placement indexA colon after the profile ID gives the placement start index, here 2. |
nvidia-smi mig -lgi | List GPU instancesShows each created instance with its profile, instance ID and placement. |
nvidia-smi mig -lcip -gi 1 | List compute instance profilesShows the compute instance profiles that GPU instance 1 can be divided into. |
nvidia-smi mig -cci 0,0,0 -gi 1 | Create compute instancesCreates three compute instances of profile ID 0 inside GPU instance 1. |
nvidia-smi mig -lci -gi 1 | List compute instancesLists the compute instances that exist inside GPU instance 1. |
nvidia-smi mig -lcipp | List compute instance placementsUses the same {Start}:Size notation as -lgipp; -i and -gi narrow it to one GPU or GPU instance. |
nvidia-smi mig -dci -ci 0,1,2 -gi 1 | Destroy chosen compute instancesRemoves compute instances 0, 1 and 2 from GPU instance 1 only. |
nvidia-smi mig -dci | Destroy all compute instancesRun it before -dgi: compute instances are removed first, then the GPU instances that held them. |
nvidia-smi mig -dgi | Destroy all GPU instancesRemoves every GPU instance once its compute instances are gone; -i and -gi limit the scope. |
Topology, NVLink and PCIe 26
| Command | What it does |
|---|---|
nvidia-smi topo -m | Topology matrixShows how every GPU and NIC pair is connected, plus the CPU and memory affinity of each GPU. |
nvidia-smi topo -mp | PCI-only topology matrixSame matrix with NVLink connections left out, so only the PCI paths between GPUs show. |
nvidia-smi topo -gpu | GPU-to-GPU matrixPrints just the connections between GPUs, in fixed-width columns. |
nvidia-smi topo -nic | GPU-to-NIC matrixAdds a NIC legend with InfiniBand name, network device name, PCI address and slot number. |
nvidia-smi topo -nvme | GPU-to-NVMe matrixShows the PCI connections between each GPU and each NVMe device. |
nvidia-smi topo -cpu | CPU and NUMA affinityOne line per GPU with three columns: memory NUMA affinity, the GPU's NUMA ID, and CPU affinity. |
nvidia-smi topo -all | Complete topology matrixCombines GPUs, NICs and NVMe devices in one matrix with affinity and NUMA information. |
nvidia-smi topo -c CPU | GPUs near a CPUTakes a CPU number and lists every GPU that has an affinity to it. |
nvidia-smi topo -n TRAVERSAL_PATH -i DEVICEID | Nearest GPUs by path typeTRAVERSAL_PATH runs from 0, GPUs on one dual-GPU board, to 5, GPUs in different NUMA nodes. |
nvidia-smi topo -p -i IDS | Path between two GPUsIDS is two device IDs separated by a comma; the answer is the most direct PCIe route. |
nvidia-smi topo -p2p CAPABILITY | Peer-to-peer statusCAPABILITY is r for read, w for write, n for NVLink, a for atomics or p for PCIe. |
nvidia-smi topo -C -i DEVICEID | NUMA node of nearest CPU-M gives the node of the nearest memory and -gnid the GPU's own NUMA ID. |
nvidia-smi nvlink -s | NVLink statusShows each link as active with its bandwidth, inactive, or asleep. |
nvidia-smi nvlink -c | NVLink capabilitiesLong form --capabilities; prints what each of the GPU's links supports. |
nvidia-smi nvlink -p | Remote PCI bus IDShows the PCI bus ID of the node at the far end of each link. |
nvidia-smi nvlink -R | Remote link informationGives the PCI bus ID of the remote device and the NVLink ID on that side. |
nvidia-smi nvlink -e | NVLink error countersOn NVLink 4 these are replay, recovery and CRC errors; NVLink 5 reports packet, FEC and BER counters. |
nvidia-smi nvlink -ec | Per-lane CRC error countersCovers CRC and ECC errors per lane up to NVLink 4; deprecated from NVLink 5 onwards. |
nvidia-smi nvlink -re | Reset NVLink error countersSets all the error counters to zero; not supported on NVLink 5. |
nvidia-smi nvlink -gt d | NVLink data throughputReports transmitted and received data payload in KiB; r adds raw payload with protocol overhead. |
nvidia-smi nvlink -dr | NVLink data rateShows the per-link data rate with FEC overhead taken into account. |
nvidia-smi nvlink -info | NVLink device information |
nvidia-smi nvlink -gLowPwrInfo | NVLink low power informationAlso shows the range and units that the low power threshold option, -sLowPwrThres, accepts. |
nvidia-smi c2c -s | C2C link statusShows the state of each C2C link and, for active links, the bandwidth. |
nvidia-smi pci -i GPU -gErrCnt | PCI error countersReads the PCI error counters of one GPU; -cErrCnt clears them. |
nvidia-smi pci -i GPU -gCnt | PCI traffic countersReads the PCI receive and transmit counters of the chosen GPU. |
Drain and maintenance 8
| Command | What it does |
|---|---|
nvidia-smi drain -p PCIID -m 1 | Put a GPU into drain stateThe GPU stops accepting new clients; persistence mode must be off, and PCIID has the form domain:bus:device.function. |
nvidia-smi drain -p PCIID -m 0 | Take a GPU out of drain stateMarks the GPU as not draining, so it accepts new clients again. |
nvidia-smi drain -p PCIID -q | Query drain stateReports whether the GPU at that PCI address is currently draining. |
nvidia-smi drain -p PCIID -r | Remove a drained GPUWorks only after -m has put the GPU in drain state and with persistence mode disabled for it. |
nvidia-smi drain -d | Rediscover removed GPUsRediscovers every GPU that was removed earlier with the remove option. |
--debug=FILE | Write an encrypted debug logProduces a log file meant to be attached to a bug report sent to NVIDIA. |
nvidia-smi event-log | Show driver event logDisplays events recorded since the driver was loaded; new in the v610 change log. |
nvidia-smi cper | Read CPER recordsPrints CPER events recorded since the driver loaded, encoded as base64. |
vGPU and confidential computing 11
| Command | What it does |
|---|---|
nvidia-smi vgpu | Summarise active vGPUsReports the vGPUs currently running on each physical GPU of a hypervisor host. |
nvidia-smi vgpu -q | Detailed vGPU reportAdds driver version, utilization and other details for every active vGPU. |
nvidia-smi vgpu -s | List supported vGPU typesShows the vGPU types each device supports; -v prints the details of each type. |
nvidia-smi vgpu -c | List creatable vGPU typesThe list changes with the vGPUs already running on the device. |
nvidia-smi vgpu -u | vGPU utilization monitorPrints average utilization per active vGPU every second until stopped with Ctrl+C. |
nvidia-smi vgpu -p | Per-process vGPU usageShows GPU engine usage of the processes running inside the vGPU virtual machines. |
nvidia-smi conf-compute -f | Confidential computing statusShows the state of the CC feature on the system. |
nvidia-smi conf-compute -gm | Protected memory sizesShows how GPU memory is split between protected and unprotected sizes. |
nvidia-smi conf-compute -grs | Show GPUs ready state |
nvidia-smi conf-compute -srs VALUE | Set GPUs ready statePass 1 to mark the GPUs ready or 0 to clear the flag; needs root. |
nvidia-smi conf-compute -q | Detailed confidential computing report |
No row on this page matches that filter.
Notes
→ Related guide: GPU cluster management: the full command guide
Watching
nvidia-smi with no options prints a summary of the GPUs connected to the system. nvidia-smi -l SEC repeats it at an interval until Ctrl+C, every five seconds when no value is given. nvidia-smi dmon streams one compact line of metrics per GPU each second, and nvidia-smi pmon does the same per process — better than the default screen for spotting a job that holds memory but does no work. nvidia-smi -q dumps every attribute, and a display type narrows it: nvidia-smi -q -d ECC for error counts, nvidia-smi -q -d POWER for power. nvidia-smi -L lists GPUs with UUIDs, including MIG devices. nvidia-smi topo -m shows how GPUs and NICs are connected and nvidia-smi nvlink -s the state of each NVLink.
Changing settings
nvidia-smi -pm 1 enables persistence mode so the driver stays loaded when no client is connected, which minimizes the driver load latency before each job; it needs root and is lost at reboot. nvidia-smi -pl POWER_LIMIT sets a power limit in watts, the usual lever for a hot rack; add -i to target one GPU. Clocks are pinned with nvidia-smi -lgc 1500,1500 and released with nvidia-smi -rgc; the older nvidia-smi -i 0 --applications-clocks 2500,745 form is deprecated. nvidia-smi -e 1 enables ECC (reboot required) and nvidia-smi -r resets GPUs whose state would otherwise need a machine reboot, provided no application is using them. Before powering a GPU down, nvidia-smi drain -p PCIID -m 1 stops it accepting new clients.
Querying for scripts
--query-gpu= takes a comma-separated field list and must come with --format=csv; nvidia-smi --format=csv,noheader --query-gpu=uuid,persistence_mode is the manual's own example. Adding nounits after noheader strips the units, which makes the output parseable, and the -l SEC loop option on the same line turns a query into a log for a run's duration. --query-compute-apps= lists processes with their memory, which is how a job's footprint is recorded. nvidia-smi --help-query-gpu lists every available field; in v610 kmd_version replaces the deprecated driver_version.
Questions
How do I check GPU usage on Linux?
Run nvidia-smi for a summary of all GPUs connected to the system. For scripts, query the fields you need, such as utilization.gpu, memory.used and temperature.gpu, with --query-gpu= and --format=csv.
How do I monitor GPU in real time?
Use nvidia-smi -l SEC to repeat the summary at an interval in seconds, or nvidia-smi dmon for one scrolling line of metrics per GPU every second. nvidia-smi pmon does the same per process.
How do I check GPU memory usage per process?
Pass --query-compute-apps= a field list that includes pid and used_gpu_memory, together with --format=csv. nvidia-smi --help-query-compute-apps prints the field names, and nvidia-smi pmon -s METRIC_GROUP with m shows per-process memory as a live monitor.
How do I enable persistence mode?
Run nvidia-smi -pm 1 as root to keep the NVIDIA driver loaded when no client is connected, which reduces driver load latency. The setting takes effect immediately but does not persist across reboots; nvidia-smi -pm 0 turns it off.
How do I check GPU topology and NVLink?
Use nvidia-smi topo -m for the connection matrix between GPUs and NICs with CPU and memory affinities, and nvidia-smi nvlink -s for the status of each NVLink. In v610 nvidia-smi topo -gpu, nvidia-smi topo -nic and nvidia-smi topo -cpu print the parts of that matrix separately.
Sources and checks
- NVIDIA: nvidia-smi(1) manual page, "NVIDIA System Management Interface program", edition for nvidia-smi v610 (docs.nvidia.com)
- NVIDIA driver 610.57.04 for Linux: built-in help of nvidia-smi ("nvidia-smi -h", "nvidia-smi --help-query-gpu" and "nvidia-smi SUBCOMMAND -h"), captured from the binary in NVIDIA's libnvidia-compute package
- NVIDIA Multi-Instance GPU User Guide: Getting Started with MIG (docs.nvidia.com, last updated Sep 11, 2026) MIG command lines: enabling MIG mode, listing profiles and placements, creating and destroying GPU and compute instances
- NVIDIA Multi-Process Service documentation: Common Tasks (docs.nvidia.com, last updated Sep 09, 2026) Setting exclusive process compute mode on one GPU
- NVIDIA Dynamic Page Retirement documentation: Blacklisting and ECC Error Recovery (docs.nvidia.com, last updated Sep 09, 2026) Checking pending page retirement and resetting one GPU or all GPUs
Every row on this page was matched to a line in the documentation above on . A binding that could not be found there is not listed.
2 rows from the earlier version of this page were removed in that check: duplicates, and bindings the documentation does not list or lists differently.
Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.