nvidia-smi (GPU) commands

nvidia-smi command lines and options for GPU servers, taken from NVIDIA's nvidia-smi manual page for v610, the built-in help of driver 610.57.04 and the MIG User Guide, grouped by task from a first status check to MIG, NVLink and drain states. Rows that start with nvidia-smi are whole commands; bare entries such as --query-gpu= or utilization.gpu are switches and field names to combine. Words in capitals stand for your own value, and most commands that change a setting need root.

211 commands in 14 sections. Checked row by row against NVIDIA: nvidia-smi(1) manual page, "NVIDIA System Management Interface program", edition for nvidia-smi v610 (docs.nvidia.com) on . Edited by Jay.

Status at a glance 13

CommandWhat it does
nvidia-smiGPU status summaryWith no options it prints a single summary table of all GPUs in the system.
nvidia-smi -colMulti-column summaryLong form --columns; prints the same summary in a multi-column layout.
nvidia-smi -i 0Summary for one GPULimits output to GPU 0; -i also takes a UUID, a PCI bus ID or a board serial number.
nvidia-smi -LList GPUsLong form --list-gpus; prints one line per GPU with its UUID, and MIG devices beneath their parent.
nvidia-smi -BList excluded GPUsLong form --list-excluded-gpus; prints the GPUs marked as excluded, each with its UUID.
nvidia-smi -qDetailed infoPrints every attribute of all GPUs once as plain text; values a device cannot report show N/A.
nvidia-smi -q -xFull report as XMLLong form --xml-format; adding --dtd embeds the document type definition in the output.
nvidia-smi -q -d UTILIZATIONUtilization detailsShows only the utilization section, where NVJPG and NVOFA figures sit next to GPU and memory.
nvidia-smi -q -d MEMORYMemory detailsShows only the memory section of the full query; several types can be joined with commas.
nvidia-smi -q -d TEMPERATURETemperature detailsLimits the full query to the temperature section.
nvidia-smi -f FILEWrite output to a fileLong form --filename=FILE; an existing file of that name is overwritten, and nothing goes to stdout.
nvidia-smi --versionShow version information
nvidia-smi -hShow built-in helpLists all options and subcommands; each subcommand prints its own page with -h, as in nvidia-smi mig -h.

Querying fields to CSV 7

CommandWhat it does
nvidia-smi --format=csv,noheader --query-gpu=uuid,persistence_modeCSV queryPrints each GPU's UUID and its persistence mode as comma-separated lines without a header row.
nvidia-smi -i 0 --query-gpu=pci.bus_id,mig.mode.current --format=csvQuery one GPU as CSVRestricts the query to GPU 0; the first output line names the two columns.
--query-gpu=Query GPU propertiesTakes a comma-separated property list, for example pci.bus_id,persistence_mode, and must be paired with --format=csv.
--format=csvOutput as CSVMandatory with every --query switch; noheader and nounits can follow csv after commas.
--format=csv,noheaderCSV without the header rowLeaves out the first line, which normally carries the column names.
nounitsOmit units from valuesAdded to the --format list after csv, it leaves the numbers bare, without unit suffixes.
nvidia-smi --help-query-gpuList GPU query fieldsPrints every property name that --query-gpu accepts, each with a short description.

Query fields for identity and state 16

CommandWhat it does
indexGPU indexZero-based position of the GPU, which may differ after a reboot; prefer uuid or pci.bus_id for stable identification.
uuidGPU UUIDAlso accepted as gpu_uuid; a globally unique, immutable identifier that is not printed on the board.
nameGPU product nameAlso accepted as gpu_name; returns the model name NVIDIA gives the product.
serialBoard serial numberAlso accepted as gpu_serial; matches the number physically printed on the board.
pci.bus_idPCI bus IDAlso accepted as gpu_bus_id; given in hex as domain:bus:device.function.
countNumber of GPUsReturns how many NVIDIA GPUs are present on the machine.
timestampTime of the queryRecords the date and time of the query with millisecond precision.
kmd_versionKernel driver versionReplaces driver_version, which this release marks as deprecated.
vbios_versionVBIOS version
persistence_modePersistence mode stateEnabled or Disabled; when enabled the driver stays loaded with no clients attached.
compute_modeCompute modeDefault allows many contexts per device, Exclusive_Process one, and Prohibited none.
pstatePerformance stateP0 is the fastest state and P12 the slowest.
mig.mode.currentCurrent MIG modeShows whether MIG is enabled now; mig.mode.pending shows the mode that applies after a reset.
gpu_recovery_actionRecommended recovery actionNames the step that clears an earlier fault: None, Reset, Reboot, Drain P2P, or Drain and Reset.
pcie.link.gen.gpucurrentCurrent PCIe link generationMay read lower than the maximum while the GPU is idle.
pcie.link.width.currentCurrent PCIe link widthLike the link generation, the width can drop while the GPU is idle.

Query fields for load, memory and power 16

CommandWhat it does
utilization.gpuGPU utilizationPercent of the sample period in which at least one kernel was running; the period is 1/6 to 1 second.
utilization.memoryMemory read/write activityPercent of the sample period spent reading or writing device memory, not the share of memory allocated.
memory.totalInstalled GPU memory
memory.usedMemory in useCounts memory allocated by active contexts; driver and firmware overhead is in memory.reserved.
memory.freeFree GPU memory
memory.reservedDriver-reserved memoryCovers what the NVIDIA driver and firmware hold back for themselves.
temperature.gpuGPU core temperatureGiven in degrees C; temperature.memory is the separate reading for HBM memory.
fan.speedIntended fan speedA percentage of the fan's rated maximum that the fan is asked to run at, not a measured speed.
power.drawBoard power drawIn watts, accurate to about 5 W; Ampere and newer GPUs report a one-second average.
power.draw.instantInstant power drawAlso accepted as gpu.power.draw.instant; the latest instantaneous reading for the whole board, in watts.
power.limitRequested power limitThe value -pl changes, in watts; gpu.power.requested_limit is a second name for the same field.
enforced.power.limitEnforced power limitThe ceiling the power management algorithm actually holds the board under; the lowest of several limiters.
power.max_limitHighest settable power limitUpper bound in watts for a power limit; power.min_limit gives the lower bound.
clocks.current.graphicsGraphics clock frequencyAlso accepted as clocks.gr; the current shader clock.
clocks.current.memoryMemory clock frequencyAlso accepted under the shorter name clocks.mem.
clocks_event_reasons.activeActive clock event reasonsA bitmask explained in nvml.h; the older name clocks_throttle_reasons.active is still accepted.

Query fields for memory errors 8

CommandWhat it does
ecc.mode.currentCurrent ECC modeecc.mode.pending shows the mode that takes over after the next reboot.
ecc.errors.corrected.volatile.totalCorrected ECC errors since driver loadSingle-bit errors the hardware fixed, counted across the whole chip since the driver was last loaded.
ecc.errors.uncorrected.volatile.totalUncorrected ECC errors since driver loadDouble-bit errors, which are detected but not fixed; the counter restarts at each driver load.
ecc.errors.uncorrected.aggregate.totalLifetime uncorrected ECC errorsAggregate counts persist indefinitely, so this is the total over the life of the board.
retired_pages.pendingPages pending retirementReports whether pages await retirement at the next reboot; such pages can still be allocated meanwhile.
retired_pages.dbePages retired for double-bit errorsShort for retired_pages.double_bit.count; retired_pages.sbe is the single-bit counterpart.
remapped_rows.uncorrectableRows remapped for double-bit errorsAlso accepted as remapped_rows.dbe; remapped_rows.correctable counts rows remapped after repeated single-bit errors.
remapped_rows.pendingRow remapping pendingReports whether a row remap is waiting; remapped_rows.failure reports a remap that failed.

Monitoring over time 22

CommandWhat it does
nvidia-smi -l SECRepeat at an intervalLong form --loop=SEC; reruns until Ctrl+C, every 5 seconds when no value is given.
nvidia-smi -lms MSRepeat every few millisecondsLong form --loop-ms=MS; the same loop as -l with a finer interval unit.
nvidia-smi -q -d ECC,POWER -i 0 -l 10 -f out.logLog chosen sections to a fileWrites the ECC and power sections for GPU 0 to out.log every 10 seconds until interrupted.
nvidia-smi dmonDevice monitorPrints one line of default metrics per GPU every second, for up to 16 GPUs, until Ctrl+C.
nvidia-smi dmon -i DEVICE1,DEVICE2Monitor chosen GPUsTakes a comma-separated device list in place of the first 16 GPUs.
nvidia-smi dmon -s METRIC_GROUPChoose metric groupsCombine letters: p power and temperature, u utilization, c clocks, v violations, m memory, e ECC, t PCIe throughput.
nvidia-smi dmon -c SAMPLESStop after N samplesExits by itself once the given number of lines has been printed.
nvidia-smi dmon -d SECSSet the sampling intervalChanges the default one-second cycle to the given number of seconds.
nvidia-smi dmon -o DPrefix lines with the dateAdds the date in YYYYMMDD form at the start of every line.
nvidia-smi dmon -o TPrefix lines with the timeAdds an HH:MM:SS clock time at the start of every line.
nvidia-smi dmon -f FILELog dmon output to a fileLong form --filename; lines go to the file and no longer to stdout.
nvidia-smi dmon --format csvCSV output from dmonnounit and noheader, added after commas, drop the unit line and the header line.
nvidia-smi dmon --gpm-metrics GPMMETRIC1,GPMMETRIC2Watch GPM metricsHopper or newer GPUs; the metric IDs are listed by dmon -h, for example 2 for SM activity.
nvidia-smi daemonStart background loggingExperimental; samples every 10 seconds and writes date-stamped compressed logs under /var/log/nvstats.
nvidia-smi daemon -d SECSSet the logging intervalReplaces the 10-second default with your own number of seconds.
nvidia-smi daemon -p PATHChoose the log directoryLog files are created in that directory in place of /var/log/nvstats.
nvidia-smi daemon -j STRINGTag the log file nameAppends your string to the name of the log file the daemon creates.
nvidia-smi daemon -tStop the daemonUses the PID stored in /var/run/nvsmi.pid; termination is best effort, so check it has exited.
nvidia-smi replay -f FILEReplay a daemon logReads a compressed log and prints one timestamped line of default metrics per stored sample.
nvidia-smi replay -f FILE -s METRIC_GROUPReplay chosen metricsUses the same letter groups as dmon -s to pick which stored metrics are shown.
nvidia-smi replay -f FILE -b HH:MM:SS -e HH:MM:SSReplay a time windowLimits output to samples between the two times; 0 stands for the start or the end of the file.
nvidia-smi replay -f FILE -r OUTPUTExtract a log to textWrites the default metrics from the log into the named output file.

Per-process information 21

CommandWhat it does
nvidia-smi pmonProcess monitorPrints one line per process on each GPU every second, covering compute and graphics processes.
nvidia-smi pmon -i DEVICE1,DEVICE2Watch processes on chosen GPUsTakes a comma-separated device list and reports only the processes on those GPUs.
nvidia-smi pmon -s METRIC_GROUPChoose process metricsu gives average utilization since the last cycle, m gives current memory use; both letters can be combined.
nvidia-smi pmon -c SAMPLESStop pmon after N samplesPrints that many sampling rounds and then exits.
nvidia-smi pmon -d SECSSet the pmon intervalAccepts 1 to 10 seconds between samples; the default is 1.
nvidia-smi pmon -o TTimestamp each pmon lineAdds HH:MM:SS to each line; -o D adds the date as YYYYMMDD.
nvidia-smi pmon -f FILELog pmon output to a fileLong form --filename; the process lines are written to the file, not to stdout.
nvidia-smi --show-all-processesShow every GPU clientAdds clients without a context, such as monitoring tools, to the process list.
nvidia-smi -q -d PIDSProcess detailsPrints only the process list from the full query, GPU by GPU.
--query-compute-apps=Query compute processesTakes fields such as pid, process_name and used_gpu_memory, and needs --format=csv.
nvidia-smi --help-query-compute-appsList process query fieldsPrints the field names --query-compute-apps accepts, with a line about each.
pidProcess IDA --query-compute-apps field that identifies the application holding the context.
process_nameProcess nameA --query-compute-apps field, also accepted under the shorter name, name.
used_gpu_memoryGPU memory used by a processAlso accepted as used_memory; not available on Windows in WDDM mode.
nvidia-smi -am 1Enable accounting modeKeeps usage statistics for each process over its lifetime; needs administrator rights, and 0 turns it off.
--query-accounted-apps=Query accounted processesReads the statistics accounting mode has collected; not supported on a vGPU host.
nvidia-smi --help-query-accounted-appsList accounting query fieldsFields include gpu_utilization, mem_utilization, max_memory_usage and time, the active time in milliseconds.
max_memory_usagePeak memory of a processA --query-accounted-apps field: the most device memory the context ever held.
nvidia-smi -caaClear accounted processesLong form --clear-accounted-apps; empties the buffer of process IDs accounted so far.
nvidia-smi -q -d ACCOUNTINGAccounting detailsLimits the full query to the accounting section.
nvidia-smi encodersessionsList encoder sessionsShows the encoder sessions on each GPU; not supported on a vGPU host.

Power and clocks 26

A power limit set with -pl is not permanent: it returns to the default power limit when the driver is unloaded.

CommandWhat it does
nvidia-smi -q -d POWERPower detailsShows only the power section, including the base power settings added in v610.
nvidia-smi -pl POWER_LIMITSet power limitTakes watts, whole or decimal, between the minimum and maximum limits nvidia-smi reports; needs root.
nvidia-smi -q -d SUPPORTED_CLOCKSList supported clocksPrints the supported clock frequencies of every GPU.
nvidia-smi -q -d PERFORMANCEClock event reasonsShows why clocks were reduced, with the event reason counters and the sparse operation mode.
nvidia-smi -q -d CLOCKClock detailsLimits the full query to clock frequencies and includes sampled maximum, minimum and average values.
nvidia-smi -lgc 1500,1500Lock GPU clocksSets the minimum and maximum GPU clock in MHz; a single value also works. Volta or newer, root.
nvidia-smi -rgcReset clocksLong form --reset-gpu-clocks; returns locked GPU clocks to their defaults. Needs root, Volta or newer.
nvidia-smi -lmc 5100,5100Lock memory clocksSets the memory clock range in MHz; not for Hopper GPUs, which need the deferred lock, -lmcd.
nvidia-smi -rmcReset memory clocksLong form --reset-memory-clocks; puts memory clocks back to normal and needs root.
nvidia-smi -lmcd MEMCLOCKLock memory clock at next initThe value applies when the GPU is next initialised, for example after reloading the kernel module.
nvidia-smi -rmcdClear the deferred memory lockLong form --reset-memory-clocks-deferred; takes effect once the driver has been unloaded and loaded again.
nvidia-smi -i 0 --applications-clocks 2500,745Set application clocksDeprecated in favour of -lmc and -lgc; here GPU 0 gets a 2500 MHz memory clock and 745 MHz for graphics.
nvidia-smi -racReset application clocksLong form --reset-applications-clocks; deprecated together with the application clock option it undoes.
nvidia-smi -cc 1Override CUDA clocksLets clocks run higher while CUDA applications run; 0 restores the default. Volta or newer, administrator rights.
nvidia-smi -gtt TEMPSet GPU target temperatureTakes degrees Celsius within the limits listed by the SUPPORTED_GPU_TARGET_TEMP query type; needs root.
nvidia-smi clocks --sync-boost-listList sync boost groupsPrints the group IDs that --sync-boost-remove expects.
nvidia-smi clocks -som 1Enable sparse operation modeLong form --sparse-operation-mode; 0 disables it, and the current state appears under -q -d PERFORMANCE.
nvidia-smi power-profiles -lList workload power profilesPrints every workload power profile that the device supports.
nvidia-smi power-profiles -ldList profiles with metadataAdds each profile's ID, its priority, where a lower number ranks higher, and the profiles it conflicts with.
nvidia-smi power-profiles -grShow requested profilesLists what has been requested; conflicting profiles may not all be enforced.
nvidia-smi power-profiles -geShow enforced profilesCan differ from the requested list when the selected profiles conflict.
nvidia-smi power-profiles -sr PROFILERequest a power profileAdds one or more profile IDs, comma-separated without spaces, to the requested list; needs root.
nvidia-smi power-profiles -cr PROFILERemove a requested profileTakes the same comma-separated ID list and drops those profiles from the requested set; needs root.
nvidia-smi power-smoothing -ppdShow power smoothing presetsPrints the full definition of every preset profile.
nvidia-smi power-smoothing -spp PROFILEActivate a smoothing presetMakes the given preset profile ID the active one; needs root.
nvidia-smi power-smoothing -s STATETurn power smoothing on or offSTATE is 1 or ENABLED to switch the feature on, 0 or DISABLED to switch it off; needs root.

Compute and persistence modes 8

Persistence mode and compute mode change at once but are lost at reboot; an ECC change is the reverse: it waits for the next reboot and then persists.

CommandWhat it does
nvidia-smi -pm 1Enable persistence modeKeeps the driver loaded when no client is connected; needs root, takes effect at once and is lost at reboot.
nvidia-smi -pm 0Disable persistence modeLets the driver unload again once the last client exits; this is the state after every reboot.
nvidia-smi -c 0Default compute modeAllows multiple contexts per device; every GPU returns to this mode after a reboot.
nvidia-smi -c 3Exclusive process modeAllows one context per device, usable from several threads; EXCLUSIVE_PROCESS is the same value by name.
nvidia-smi -c 2Prohibited compute modeAllows no compute contexts on the device at all.
nvidia-smi -i 0 -c EXCLUSIVE_PROCESSExclusive mode on one GPUThe step NVIDIA's MPS guide runs as root on GPU 0 before starting the MPS control daemon.
nvidia-smi compute-policy -lList compute policiesThe same subcommand sets a GPU's timeslice to default, short, medium or long.
--gom=MODESet GPU operation mode0/ALL_ON, 1/COMPUTE or 2/LOW_DP on Kepler-era Tesla and Titan cards; needs administrator rights and a reboot.

ECC and resets 14

NVIDIA does not guarantee that a GPU reset works in every case: verify the health of each reset GPU, and power cycle the node if one is not healthy.

CommandWhat it does
nvidia-smi -q -d ECCECC error detailsLimits the full query to ECC mode and error counts, with the repair-pending flags added in recent releases.
nvidia-smi -e 1Enable ECCNeeds root; the new mode starts only after the next reboot and then persists.
nvidia-smi -e 0Disable ECCLike enabling, the change waits for a reboot; ecc.mode.pending shows it in the meantime.
nvidia-smi -p 0Reset volatile ECC countersLong form --reset-ecc-errors=0; takes effect immediately and needs root.
nvidia-smi -p 1Reset aggregate ECC countersClears the lifetime counts; the manual says this is not supported on Ampere and newer GPUs.
nvidia-smi -i GPU -q -d PAGE_RETIREMENTRetired page summaryShows retired page counts for one GPU and whether any page is still pending.
--query-retired-pages=Query retired pagesFields include retired_pages.address, retired_pages.cause and retired_pages.timestamp; needs --format=csv.
nvidia-smi --help-query-retired-pagesList retired page fieldsPrints the field names with descriptions, including the two possible retirement causes.
nvidia-smi -q -d ROW_REMAPPERRow remapping detailsLimits the full query to the row remapper section.
--query-remapped-rows=Query remapped rowsFields include remapped_rows.correctable, remapped_rows.uncorrectable, remapped_rows.pending and remapped_rows.failure.
nvidia-smi --help-query-remapped-rowsList remapped row fieldsPrints the field names for --query-remapped-rows, among them the histogram buckets from max to none.
nvidia-smi -rReset GPULong form --gpu-reset; resets every GPU unless -i names some, needs root and no process on the device.
nvidia-smi -i GPU -rReset one GPUTargets a single device; NVLink peers may have to be reset in the same command.
nvidia-smi -r busBus resetRequests a bus reset; plain -r does a function level reset, the only kind on some platforms.

MIG partitioning 15

Enabling MIG mode is not enough to run CUDA workloads: GPU instances and compute instances must exist, and they have to be created again after every reboot.

CommandWhat it does
nvidia-smi -i 0 -mig 1Enable MIG modeRun as root; -i accepts an index, a PCI bus ID or a UUID, and without it every GPU is switched.
nvidia-smi -mig 0Disable MIG modeLong form --multi-instance-gpu=0; needs root. On Ampere GPUs the mode otherwise survives reboots.
nvidia-smi mig -lgipList GPU instance profilesShows the instance sizes the GPU offers, with the profile IDs and names that -cgi accepts.
nvidia-smi mig -lgippList possible placementsPrints each profile's placements as {start indexes}:size, the slots where an instance can sit.
nvidia-smi mig -cgi 9,3g.20gb -CCreate GPU and compute instancesCreates two 3g.20gb GPU instances, one by profile ID and one by name; -C adds a default compute instance to each.
nvidia-smi mig -cgi 19,14,5Create mixed-size instancesBuilds a 4-2-1 layout from three profile IDs; the driver picks where each instance goes.
nvidia-smi mig -cgi 19:2Create at a placement indexA colon after the profile ID gives the placement start index, here 2.
nvidia-smi mig -lgiList GPU instancesShows each created instance with its profile, instance ID and placement.
nvidia-smi mig -lcip -gi 1List compute instance profilesShows the compute instance profiles that GPU instance 1 can be divided into.
nvidia-smi mig -cci 0,0,0 -gi 1Create compute instancesCreates three compute instances of profile ID 0 inside GPU instance 1.
nvidia-smi mig -lci -gi 1List compute instancesLists the compute instances that exist inside GPU instance 1.
nvidia-smi mig -lcippList compute instance placementsUses the same {Start}:Size notation as -lgipp; -i and -gi narrow it to one GPU or GPU instance.
nvidia-smi mig -dci -ci 0,1,2 -gi 1Destroy chosen compute instancesRemoves compute instances 0, 1 and 2 from GPU instance 1 only.
nvidia-smi mig -dciDestroy all compute instancesRun it before -dgi: compute instances are removed first, then the GPU instances that held them.
nvidia-smi mig -dgiDestroy all GPU instancesRemoves every GPU instance once its compute instances are gone; -i and -gi limit the scope.

Drain and maintenance 8

CommandWhat it does
nvidia-smi drain -p PCIID -m 1Put a GPU into drain stateThe GPU stops accepting new clients; persistence mode must be off, and PCIID has the form domain:bus:device.function.
nvidia-smi drain -p PCIID -m 0Take a GPU out of drain stateMarks the GPU as not draining, so it accepts new clients again.
nvidia-smi drain -p PCIID -qQuery drain stateReports whether the GPU at that PCI address is currently draining.
nvidia-smi drain -p PCIID -rRemove a drained GPUWorks only after -m has put the GPU in drain state and with persistence mode disabled for it.
nvidia-smi drain -dRediscover removed GPUsRediscovers every GPU that was removed earlier with the remove option.
--debug=FILEWrite an encrypted debug logProduces a log file meant to be attached to a bug report sent to NVIDIA.
nvidia-smi event-logShow driver event logDisplays events recorded since the driver was loaded; new in the v610 change log.
nvidia-smi cperRead CPER recordsPrints CPER events recorded since the driver loaded, encoded as base64.

vGPU and confidential computing 11

CommandWhat it does
nvidia-smi vgpuSummarise active vGPUsReports the vGPUs currently running on each physical GPU of a hypervisor host.
nvidia-smi vgpu -qDetailed vGPU reportAdds driver version, utilization and other details for every active vGPU.
nvidia-smi vgpu -sList supported vGPU typesShows the vGPU types each device supports; -v prints the details of each type.
nvidia-smi vgpu -cList creatable vGPU typesThe list changes with the vGPUs already running on the device.
nvidia-smi vgpu -uvGPU utilization monitorPrints average utilization per active vGPU every second until stopped with Ctrl+C.
nvidia-smi vgpu -pPer-process vGPU usageShows GPU engine usage of the processes running inside the vGPU virtual machines.
nvidia-smi conf-compute -fConfidential computing statusShows the state of the CC feature on the system.
nvidia-smi conf-compute -gmProtected memory sizesShows how GPU memory is split between protected and unprotected sizes.
nvidia-smi conf-compute -grsShow GPUs ready state
nvidia-smi conf-compute -srs VALUESet GPUs ready statePass 1 to mark the GPUs ready or 0 to clear the flag; needs root.
nvidia-smi conf-compute -qDetailed confidential computing report

Notes

→ Related guide: GPU cluster management: the full command guide

Watching

nvidia-smi with no options prints a summary of the GPUs connected to the system. nvidia-smi -l SEC repeats it at an interval until Ctrl+C, every five seconds when no value is given. nvidia-smi dmon streams one compact line of metrics per GPU each second, and nvidia-smi pmon does the same per process — better than the default screen for spotting a job that holds memory but does no work. nvidia-smi -q dumps every attribute, and a display type narrows it: nvidia-smi -q -d ECC for error counts, nvidia-smi -q -d POWER for power. nvidia-smi -L lists GPUs with UUIDs, including MIG devices. nvidia-smi topo -m shows how GPUs and NICs are connected and nvidia-smi nvlink -s the state of each NVLink.

Changing settings

nvidia-smi -pm 1 enables persistence mode so the driver stays loaded when no client is connected, which minimizes the driver load latency before each job; it needs root and is lost at reboot. nvidia-smi -pl POWER_LIMIT sets a power limit in watts, the usual lever for a hot rack; add -i to target one GPU. Clocks are pinned with nvidia-smi -lgc 1500,1500 and released with nvidia-smi -rgc; the older nvidia-smi -i 0 --applications-clocks 2500,745 form is deprecated. nvidia-smi -e 1 enables ECC (reboot required) and nvidia-smi -r resets GPUs whose state would otherwise need a machine reboot, provided no application is using them. Before powering a GPU down, nvidia-smi drain -p PCIID -m 1 stops it accepting new clients.

Querying for scripts

--query-gpu= takes a comma-separated field list and must come with --format=csv; nvidia-smi --format=csv,noheader --query-gpu=uuid,persistence_mode is the manual's own example. Adding nounits after noheader strips the units, which makes the output parseable, and the -l SEC loop option on the same line turns a query into a log for a run's duration. --query-compute-apps= lists processes with their memory, which is how a job's footprint is recorded. nvidia-smi --help-query-gpu lists every available field; in v610 kmd_version replaces the deprecated driver_version.

Questions

How do I check GPU usage on Linux?

Run nvidia-smi for a summary of all GPUs connected to the system. For scripts, query the fields you need, such as utilization.gpu, memory.used and temperature.gpu, with --query-gpu= and --format=csv.

How do I monitor GPU in real time?

Use nvidia-smi -l SEC to repeat the summary at an interval in seconds, or nvidia-smi dmon for one scrolling line of metrics per GPU every second. nvidia-smi pmon does the same per process.

How do I check GPU memory usage per process?

Pass --query-compute-apps= a field list that includes pid and used_gpu_memory, together with --format=csv. nvidia-smi --help-query-compute-apps prints the field names, and nvidia-smi pmon -s METRIC_GROUP with m shows per-process memory as a live monitor.

How do I enable persistence mode?

Run nvidia-smi -pm 1 as root to keep the NVIDIA driver loaded when no client is connected, which reduces driver load latency. The setting takes effect immediately but does not persist across reboots; nvidia-smi -pm 0 turns it off.

How do I check GPU topology and NVLink?

Use nvidia-smi topo -m for the connection matrix between GPUs and NICs with CPU and memory affinities, and nvidia-smi nvlink -s for the status of each NVLink. In v610 nvidia-smi topo -gpu, nvidia-smi topo -nic and nvidia-smi topo -cpu print the parts of that matrix separately.

Sources and checks

Every row on this page was matched to a line in the documentation above on . A binding that could not be found there is not listed.

2 rows from the earlier version of this page were removed in that check: duplicates, and bindings the documentation does not list or lists differently.

Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.