NVIDIA DCGM commands

Command lines for NVIDIA Data Center GPU Manager on Linux GPU servers: starting nv-hostengine, then dcgmi for discovery, groups, health watches, diagnostics, policies, job statistics, live monitoring, NVLink and configuration, as NVIDIA's reference and built-in help print them for DCGM 4.7.0. Rows that start with dashes are options of their section's subcommand, bare numbers are field IDs for dmon, and capitals mark values to replace. dcgm-exporter and the API bindings are left out.

271 commands in 20 sections. Checked row by row against NVIDIA DCGM documentation (4.7.0 release, pages last updated 2026-09-10): nv-hostengine command reference on . Edited by Jay.

Starting the host engine 20

Groups, field groups, watches, job records and configuration targets exist only in the memory of the running host engine; a restart loses them, while a value already written to a GPU can remain.

CommandWhat it does
nv-hostengineStart host engineWith no options it daemonizes and listens on TCP 127.0.0.1:5555, the endpoint dcgmi uses by default.
sudo systemctl enable --now nvidia-dcgmEnable and start the serviceStarts the packaged systemd unit immediately and on every boot; that unit runs the host engine as root.
systemctl is-active nvidia-dcgmCheck the service is activePrints active when the unit is running; follow it with a real dcgmi request to prove readiness.
systemctl cat nvidia-dcgm.serviceShow unit file and drop-insPrints all the pieces systemd will merge into the effective unit, so overrides are visible before a restart.
sudo systemctl edit nvidia-dcgm.serviceOpen a drop-in overrideAdds options or environment variables without touching the packaged unit; systemd reloads itself but does not restart the service.
sudo systemctl daemon-reloadReload unit filesNeeded after editing unit or drop-in files by hand; the running host engine keeps its old settings until restarted.
sudo systemctl restart nvidia-dcgm.serviceRestart the host engine serviceStarts a new process from the effective unit; groups, watches, jobs and stored targets of the old one are lost.
systemctl status nvidia-dcgm.service --no-pagerShow service statusPart of the verification sequence after a change, before listing GPUs with dcgmi again.
journalctl --unit=nvidia-dcgm.service --boot --no-pagerRead the service journalShows what the packaged host engine printed since boot, so no second server has to be started to see it.
nv-hostengine -n --service-account nvidia-dcgmRun as the packaged unit doesThe -n flag keeps it in the foreground for systemd; diagnostic child processes run under the nvidia-dcgm account.
nv-hostengine --no-daemon --service-account nvidia-dcgm --home-dir /var/log/nvidia-dcgmForeground on loopback TCPLong-option form of the packaged command; --home-dir is where diagnostic logs and statistics files go by default.
nv-hostengine --no-daemon --domain-socket /run/nvidia-dcgm/nv-hostengine.sock --service-account nvidia-dcgmListen on a Unix socketOpens no TCP port; clients must pass --host unix:// followed by the same absolute socket path.
nv-hostengine --port PORTChoose the listening portSets the TCP or VSOCK port, 5555 by default; the short form is -p.
nv-hostengine --bind-interface ADDRESSBind to one IP addressDefault is 127.0.0.1; ALL listens on every interface, and the listener has no TLS or client authentication.
nv-hostengine --vsock-cid CIDListen on VSOCKWithout a value it binds any local context ID; --port then selects the VSOCK port.
nv-hostengine --pid /run/nvhostengine.pidDaemonize with a PID fileSelf-daemonized mode creates the file after initialization and will not start while it names a live process.
nv-hostengine --term --pid /run/nvhostengine.pidStop a daemonized instanceSends SIGTERM to the recorded PID and waits up to 30 seconds; it cannot stop the systemd-managed service.
sudo nv-hostengine -f /tmp/nv-hostengine.log --log-level DEBUGStart with debug loggingWrites debug logs to the named file for the life of the process; the default level is ERROR.
nv-hostengine --no-daemon --denylist-modules 2,3,8Block modules at startupIDs 2, 3 and 8 are VGPU, Introspection and Profiling; the denylist is fixed until the host engine restarts.
nv-hostengine --versionHost engine build versionPrints version, build ID, date, commit, architecture and platform without initializing DCGM.

Connecting and version 6

CommandWhat it does
dcgmi -vVersionShows build information for the local dcgmi and the host engine it connects to; add --host for another node.
dcgmi --versionClient version onlyReports the dcgmi release installed on this machine and exits without contacting a host engine.
dcgmi discovery --host node01.example.com --listQuery a remote host engine--host takes an IP address, a domain name or host:port; without it dcgmi talks to localhost.
dcgmi modules --host unix:///run/nvidia-dcgm/nv-hostengine.sock --listUse a Unix socket addressUse the absolute path of the socket the host engine was started on with --domain-socket.
--host vsock://3:5555Connect over VSOCKAddress form vsock://CID:PORT, for a host engine started with --vsock-cid; the port defaults to 5555.
dcgmi diag -hSubcommand helpEvery dcgmi subsystem prints its usage forms and full option list with -h or --help.

Discovering GPUs and entities 10

CommandWhat it does
dcgmi discovery -lList GPUsLists active GPUs, NVSwitches, ConnectX devices and NVIDIA CPUs with their DCGM IDs; long form --list.
dcgmi discovery --list --allInclude inactive GPUsAdds detached or failed GPUs with a Status line; their cached name and bus ID are marked last known.
dcgmi discovery --info a --gpuid 0GPU identityReports name, PCI bus ID, UUID, serial number, InfoROM image version and VBIOS version for GPU 0.
dcgmi discovery --info apt --gpuid 0Identity, power and thermal limitsAttribute letters combine: a device info, p power limits, t thermal limits; this trio is also the default.
dcgmi discovery --info aptc --gpuid 0Add supported clock pairsThe c letter lists every supported memory and SM clock pair in MHz, the values --appclocks accepts.
dcgmi discovery --info w --gpuid 0Workload power profilesShows which profile indexes the GPU supports, which are requested and which the driver enforces.
dcgmi discovery --info p --gpuid IDPower limit rangeLists minimum and maximum watts plus current, default and enforced limits; check it before setting --powerlimit.
dcgmi discovery --info a --cpuid 0NVIDIA CPU identityFor a CPU the a attribute reports the core ranges it owns and its serial number.
dcgmi discovery -i ATTRIBUTES --group ID --verboseAttributes for a groupSummarises values the GPUs share and prints **** where they differ; --verbose lists each GPU separately.
dcgmi discovery --compute-hierarchyMIG hierarchyLists GPU instances and compute instances under each parent GPU; the short form is -c.

Entity selectors 15

CommandWhat it does
gpu:0GPU by IDShort name g; a bare number such as 0 means the same thing in an entity list.
instance:1MIG GPU instanceShort name i; the IDs are shown by the compute hierarchy listing.
compute_instance:2MIG compute instanceShort name c; listed under its parent by discovery --compute-hierarchy.
vgpu:4Virtual GPUShort name v; DCGM 4.6 has no dcgmi command that enumerates vGPU entity IDs.
nvswitch:0NVSwitchShort name n; switch IDs appear in the discovery list next to the GPUs.
cpu:0NVIDIA CPUNo short name; only NVIDIA CPUs that DCGM discovers can be selected.
core:12CPU coreNo short name; the cores owned by each CPU are listed by discovery.
cx:0ConnectX deviceNo short name; ConnectX IDs are part of the discovery list.
link:259Link by packed IDAlso accepted in hex as link:0x103; short name l. The IDs come from nvlink --link-status --show-entity-ids.
gpu_link:0:5GPU NVLink portReads as GPU 0, port 5, which is clearer than the packed link ID.
switch_link:0:5NVSwitch portReads as NVSwitch 0, link index 5, like the GPU form.
gpu:{0-3}Inclusive rangeBraces expand to every ID in the range; quote the argument so the shell leaves them alone.
cpu:*All CPUsWildcard form; a lone asterisk selects all GPUs in the same way.
*/*All MIG GPU instancesWildcard over GPU and instance; 0/* limits it to the instances on GPU 0.
*/*/*All MIG compute instancesThree levels: GPU, GPU instance, compute instance; accepted only by commands that support that entity type.

Entity groups 13

CommandWhat it does
dcgmi group -c NAMECreate groupCreates an empty group and prints its numeric ID; add --default to fill it with every GPU.
dcgmi group -g GROUPID -a ENTITYIDAdd GPUs to groupBare numbers such as 0,1 are GPU IDs; other entity types need a prefix like cpu:3 or instance:1.
dcgmi group --listList groupsShows every group ID with its members, including the built-in system groups; IDs do not survive a host engine restart.
dcgmi group --create training-gpus --add gpu:0,gpu:1Create and fill a groupOne command makes the group and puts two GPUs in it; the new numeric ID is printed.
dcgmi group --create all-available --defaultGroup of every GPU--default populates the new group with all GPUs present; --defaultnvswitches does the same with NVSwitches.
dcgmi group --create platform-entities --add gpu:0,nvswitch:0,cx:0Mixed-entity groupA group may hold GPUs, NVSwitches, ConnectX devices, CPUs, MIG instances and links; each command decides what it accepts.
dcgmi group --create job-4815 --add 'cpu:0,core:{0-3},gpu:0'Group a CPU, cores and a GPUBraces give an inclusive range, so core:{0-3} adds cores 0 through 3; quote it for the shell.
dcgmi group -c mig-ex1 -a 0,i:0,i:1Group a GPU with MIG instancesi: is the short prefix for a GPU instance, so this holds GPU 0 and instances 0 and 1.
dcgmi group --group 3 --infoShow group membersPrints the name and member entities of group 3; the short form is -g 3 -i.
dcgmi group --group 3 --add 'cpu:{0-3}'Add entities to a groupAppends four CPUs to an existing group, which may hold up to 1,024 entities.
dcgmi group --group 3 --remove gpu:1Remove an entityTakes a comma-separated entity list just as --add does; the short form is -r.
dcgmi group --delete GROUP-IDDelete a groupFrees one of the 62 slots left for user groups; configuration already written to its GPUs stays.
dcgmi group --list --jsonGroups as JSONJSON output has no stable schema across releases, so pin and test the DCGM version you parse.

Field groups 5

CommandWhat it does
dcgmi fieldgroup --listList field groupsShows each field group with the numeric ID to pass to dmon; a host engine holds at most 64.
dcgmi fieldgroup --create gpu-environment --fieldids 150,155Create a field groupNames a reusable set of 1 to 128 field IDs, here two environmental readings per GPU.
dcgmi fieldgroup --info --fieldgroup 5Show a field groupPrints the name and the field IDs stored in field group 5.
dcgmi fieldgroup --delete --fieldgroup FIELD-GROUP-IDDelete a field groupRemoves it from the running host engine; field groups also disappear when that process exits.
dcgmi fieldgroup --list --jsonField groups as JSONPrints a plain-text count and then one JSON document per field group, not a single document.

Background health watches 16

PCIe, memory, NVLink, thermal and power, and ConnectX watches can need about 60 seconds of samples before the first useful check.

CommandWhat it does
dcgmi health -s aStart health watchReplaces the group's watch mask with every system: PCIe, memory, InfoROM, thermal and power, NVLink, driver, ConnectX, IMEX and NVSwitch.
dcgmi health -cCheck healthEvaluates the samples retained by the enabled watches for all GPUs and reports Healthy, Warning or Failure.
dcgmi health --set pmi --group 1Set selected watchesLetters combine, except that a must stand alone; each --set overwrites the previous selection for the group.
dcgmi health --set pmitnd --group GROUP-ID --update-interval 30 --max-keep-age 600Watch with explicit samplingSamples the driver every 30 seconds and keeps 600 seconds of data, which are also the defaults.
dcgmi health -s pWatch PCIe healthFlags an excessive replay rate and bus or fabric related XIDs; needs about a minute of samples first.
dcgmi health -s mWatch GPU memory healthCovers double-bit ECC errors, faulty memory, page retirement and row-remap failures.
dcgmi health -s iWatch InfoROM healthRaises an incident when the InfoROM is reported as corrupt.
dcgmi health -s dWatch driver healthReports recovery states in which the driver needs a traffic drain, a GPU reset or a reboot.
dcgmi health -s tWatch thermal and power healthLooks for clock slowdowns caused by heat or power, threshold violations and unreadable power readings.
dcgmi health -s nWatch NVLink healthChecks link errors and their rates, bit-error-rate thresholds and the state of the fabric.
dcgmi health -s eWatch IMEX healthAdded in 4.7.0: follows the IMEX daemon and domain state independently of the NVLink selector.
dcgmi health -s xWatch ConnectX healthReports an unhealthy ConnectX device and its fatal or nonfatal PCIe AER errors.
dcgmi health --fetch --group 1Show enabled watchesLists which systems are watched for the group; the SM row it shows cannot be selected with --set.
dcgmi health --check --group g --jsonCheck all GPUs as JSONg is the built-in alias for all GPUs; s, i, c and a mean NVSwitches, GPU instances, compute instances and everything.
dcgmi health -g 1 -cCheck one groupPrints the overall result and each incident, such as excessive PCIe replays; it reads retained data and runs no active test.
dcgmi health --clear --group 1Disable health watchesTurns off every watch for the group; otherwise watches stay active in the host engine after dcgmi exits.

Diagnostic levels and named tests 25

NVIDIA states that DCGM diagnostics do not replace its offline field diagnostics and do not determine RMA eligibility; a failure can also originate in the host, libraries, cooling or power.

CommandWhat it does
dcgmi diag -r 1Quick diagnosticLevel 1 runs the built-in software deployment checks and normally finishes in seconds; use it as a readiness check.
dcgmi diag -r 2Medium diagnosticLevel 2 adds the memory and pcie tests to level 1 and normally takes about two minutes.
dcgmi diag -r 3Long diagnosticLevel 3 adds diagnostic, memory_bandwidth, targeted_stress and targeted_power; expect roughly a quarter of an hour.
dcgmi diag -r 4Extended diagnosticLevel 4 adds memtest and pulse_test to the level 3 set and runs longest.
dcgmi diag -r shortLevel 1 by namequick and short are aliases for level 1; medium, long and xlong select levels 2 to 4.
dcgmi diag --run 1 --entity-id gpu:0Quick suite on one GPUWithout a target the run covers all GPUs and all NVIDIA CPUs; --entity-id narrows it.
dcgmi diag --run 1 --entity-id gpu:0 --verboseShow which subcheck failedAdds informational and warning messages for each test to the report.
dcgmi diag --run 2 --entity-id gpu:0 --jsonMedium suite as JSONJSON keeps per-entity status, warnings and error codes for scripts, but its schema is release-specific.
dcgmi diag --run pcie,diagnostic --entity-id gpu:0,gpu:1Run two named testsCommas select several tests; a named run still performs the software deployment checks first.
dcgmi diag --run pcie --entity-id gpu:0 --iterations 3Repeat a named testRuns the PCIe test three times in a row on GPU 0 instead of a whole suite.
dcgmi diag --run context_create --entity-id gpu:0,gpu:1CUDA context testVerifies a CUDA context can be created on each GPU; no numbered suite includes it.
dcgmi diag --run pcie --entity-id gpu:0,gpu:1PCIe test on two GPUsWith a pair of targets the plugin can also measure GPU-to-GPU transfers, not only transfers to and from the host.
dcgmi diag -r memory -p "memory.is_allowed=True"GPU memory testRuns the framebuffer memory test by itself; -p passes a test.parameter=value override.
dcgmi diag --run memory --entity-id gpu:0 --configfile custom-diag-tests.yamlRun with a config fileReads test parameters from a local YAML file; a --parameters value on the command line still wins.
dcgmi diag -r 3 -p diagnostic.test_duration=300.0Level 3 with a duration overrideSets the diagnostic plugin to 300 seconds inside the level 3 suite.
dcgmi diag -r memory_bandwidth -p "memory_bandwidth.is_allowed=True"Memory bandwidth testMeasures local GPU memory throughput and fails when it is below the device threshold.
dcgmi diag -r targeted_stress -p targeted_stress.test_duration=120.0Targeted stress testHolds a requested compute load for 120 seconds and checks the GPU sustains it.
dcgmi diag -r targeted_power -p targeted_power.target_power=200.0Targeted power testDrives each GPU toward a 200-watt draw and reports errors seen under that load.
dcgmi diag --run targeted_power --parameters targeted_power.test_duration=300 --timeout 600Set an overall timeoutThe timeout in seconds must exceed the sum of the explicit test durations; zero means unlimited.
dcgmi diag -r 4 -p targeted_power.is_allowed=falseSkip one test in a suiteSetting is_allowed to false removes that plugin from the suite for this run.
dcgmi diag -r nvbandwidthInterconnect bandwidth testValidates GPU copy paths over NVLink or PCIe using the nvbandwidth executable, which must be installed.
dcgmi diag -r nvbandwidth -p nvbandwidth.testcases=1,2,3Choose nvbandwidth test casesLimits the run to the listed nvbandwidth case numbers.
dcgmi diag --run memtestMemory pattern testBy default two patterns run for ten minutes; others are switched on with memtest.testN=true parameters.
dcgmi diag -r pulse_testPower pulse testSwings GPU load and power draw quickly to expose board or power-delivery instability.
dcgmi diag -r short --debugLogFile /tmp/nvvs.log -d DEBUGSave a diagnostic debug logWrites the encrypted server-side log to the given path at DEBUG level, for a problem report.

Diagnostic run options 17

CommandWhat it does
--entity-id ENTITIESTarget entitiesAdd to dcgmi diag: a list such as gpu:0,gpu:1, not combinable with --group; short form -i.
--group GROUP-IDTarget a groupRuns the diagnostic on a persistent group instead of an entity list; short form -g.
--parameters TEST.PARAMETER=VALUEOverride a test parameterRepeatable; one argument may hold several assignments separated by semicolons. Short form -p.
--configfile PATHLoad a YAML configurationThe file is read on the client and its contents travel with the request; short form -c.
--iterations COUNTRepeat the runExecutes the selected diagnostics back to back that many times; the default is 1.
--timeout SECONDSOverall time limitZero, the default, means unlimited; short form -t.
--fail-earlyCheck for failures mid-testFour tests support it (diagnostic, SM stress and the two targeted ones); without it they are judged only once they end.
--check-interval SECONDSEarly-failure check intervalNeeds --fail-early; 5 seconds unless changed, allowed range 1 to 300.
--clocksevent-mask REASONS-OR-MASKIgnore clock-event reasonsTakes names such as HW_SLOWDOWN,SW_THERMAL or a decimal mask; replaces the deprecated --throttle-mask.
--ignoreErrorCodes SPECIFICATIONIgnore chosen error codesA list like 28,140 covers all entities; gpu0:28;gpu1:140 targets single GPUs.
--expectedNumEntities gpu:NRequire a GPU countAborts the run up front when the count of active GPUs differs from N; short form -n.
--statsonfailStats files only on failureKeeps the plugin statistics files only for runs in which some test failed.
--statspath DIRECTORYStatistics output directoryA server-side directory under 128 characters, used in place of the diagnostic working directory.
--debugLogFile PATHDebug log locationServer-side encrypted log; nvvs.log is used when only a debug level is given.
--debugLevel LEVELDiagnostic log levelOne of NONE, FATAL, ERROR, WARN, INFO, DEBUG or VERB; DEBUG is the default. Short form -d.
--watch-frequency MICROSECONDSField sampling intervalDefault 5,000,000 microseconds; accepted range 100,000 to 60,000,000.
--enable-heartbeatCancel if the client diesThe host engine cancels the diagnostic when its client disconnects or stops sending heartbeats; DCGM 4.4.2 and later.

Field-based policies 15

CommandWhat it does
dcgmi policy --create --field 150 --name "GPU temperature warning" --operator '>' --threshold 85 --entities allCreate a temperature policyRecords a violation when field 150 rises above 85 on any GPU; the host engine persists the policy.
dcgmi policy --listList field policiesLists every field-based policy; the deprecated group policy is read with --get and --group instead.
dcgmi policy --get --field 150 --policy-id POLICY-IDShow one policyWithout --policy-id it prints every policy defined for that field.
dcgmi policy --watch --field 150Watch for violationsPolls every second and prints new violations until interrupted.
dcgmi policy --watch --field 150 --once --jsonFetch violations once--once retrieves a single set and exits; with --json the result is an array.
dcgmi policy --modify --field FIELD-ID --policy-id IDChange a policyFollow with --name, --threshold, --operator, --entities, --channels or --rate-limit; the monitored field cannot change.
dcgmi policy --delete --field FIELD-ID --policy-id IDDelete one policyRemoves a single field-based policy identified by its field and policy ID.
dcgmi policy --delete --allDelete all field policiesClears the whole field-policy registry in one step.
dcgmi policy --disable --field FIELD-ID --policy-id IDDisable a policyStops the policy evaluating samples and recording violations until --enable is used.
dcgmi policy --enable --field FIELD-ID --policy-id IDEnable a policyResumes evaluation for a policy created with --disabled or switched off later.
dcgmi policy --import PATHImport policies from YAMLThe path is opened by the host engine, not by dcgmi, and a successful import replaces all existing field policies.
--operator OPERATORComparison operatorOne of >, >=, <, <=, ==, != or changed; text aliases such as gt and le also work.
--channels CHANNELSNotification channelsComma-separated mix of console, file and callback; console is used when the option is omitted.
--rate-limit SECONDSNotification rate limitSmallest gap allowed between two notifications for the same entity: 60 seconds by default, 0 to switch it off.
--disabledCreate in disabled stateAdd to --create so the policy exists but does not evaluate until enabled.

Legacy group policies 13

CommandWhat it does
dcgmi policy --get --group 2Show a group policyDeprecated interface: prints the violation conditions stored for group 2.
dcgmi policy --set 0,0 --group 2 --maxtemp 85Notify on temperatureThe pair 0,0 means no action and no validation; the condition fires once a GPU is at 85 or hotter.
dcgmi policy --set 1,1 --group 2 --eccerrors --pcierrorsNotify on ECC or PCIe errorsThe 1,1 pair is stored and shown by --get, but DCGM never resets a GPU or runs a validation because of it.
dcgmi policy --set 0,0 -pWatch PCIe replay errorsShort form of --pcierrors for the default all-GPU group; pair it with a --reg listener.
dcgmi policy --set 0,0 -eWatch double-bit ECC errorsShort form of --eccerrors; each --set replaces the previous policy of the group.
dcgmi policy --set 0,0 -T 50Set a 50-degree thresholdShort form of --maxtemp; the error-injection walkthrough runs it in a listening terminal to provoke a thermal violation.
dcgmi policy --reg --group 2Wait for notificationsBlocks and prints violations as they arrive; it subscribes to all seven conditions and starts the needed watches.
dcgmi policy --regListen on all GPUsWith no group the listener covers the built-in all-GPU group and waits for violations until interrupted.
dcgmi policy --clear --group GROUP-IDClear a group policyEmpties the group's legacy policy: conditions, action and validation.
--maxpages COUNTRetired-page thresholdFires once the number of retired memory pages is at or above the given figure; short form -M.
--maxpower WATTSPower thresholdFires when a GPU draws at least the given number of watts; short form -P.
--nvlinkerrorsNVLink error conditionFires on a fixed internal criterion for the NVLink error counters; short form -n.
--xiderrorsXID error conditionFires whenever the driver reports an XID error; short form -x.

Process and job statistics 9

In dcgmi stats the letter -j is the short form of --job; it does not switch the output to JSON as it does in other subcommands.

CommandWhat it does
dcgmi stats -eEnable job statsStarts the process-accounting watches for the default all-GPU group; enable it before the workload starts.
dcgmi stats --enable --group 1 --update-interval 5000 --max-keep-age 7200Enable with custom samplingSamples every 5,000 ms and keeps 7,200 seconds of data; defaults are 30,000 ms and 3,600 seconds.
dcgmi stats --pid 1234 --group 1 --verboseStatistics for a processReports energy, peak memory, utilization, ECC and XID events for the PID; --verbose breaks it down per GPU.
dcgmi stats --jstart batch-4815 --group 1Start recording a jobThe job ID is any unused string of up to 64 characters; call it just before launching the workload.
dcgmi stats --jstop batch-4815Stop recording a jobNo group is needed because the job ID was tied to its group at start.
dcgmi stats --job batch-4815 --verboseShow job statisticsShort form -j, which for stats means job rather than JSON; --verbose adds per-GPU details.
dcgmi stats --jremove batch-4815Remove a job recordDeletes the retained statistics of one job once the result has been exported.
dcgmi stats --jremoveallRemove all job recordsClears the retained statistics of every job in the host engine.
dcgmi stats --disableDisable watches (no effect)In DCGM 4.6 this only reports that process-statistics watches cannot be disabled; nothing is stopped.

Live monitoring with dmon 9

CommandWhat it does
dcgmi dmon --field-id 150,155 --count 5Device monitorPrints temperature and power draw for every GPU five times, one second apart; a field selection is mandatory.
dcgmi dmon --listList field IDsShows long name, short column name and numeric ID of every field the local dcgmi knows.
dcgmi dmon --field-id 100,101 --count 10Sample a fixed number of times--count ends the stream after that many samples; without it dmon runs until interrupted.
dcgmi dmon --field-id 150,155 --entity-id gpu:0,gpu:1 --count 10Monitor selected GPUs--entity-id limits rows to the listed entities and cannot be combined with --group-id.
dcgmi dmon --field-group-id 5Monitor a field groupStreams every field stored in field group 5, one row per GPU.
dcgmi dmon --field-group-id 5 --group-id 2 --delay 5000Field group on an entity group--delay is the gap between samples in milliseconds: 1,000 by default, 1 at minimum.
--group-id all_nvswitchesMonitor all NVSwitchesdmon accepts the names all_gpus and all_nvswitches in place of a numeric group ID.
dcgmi dmon --entity-id cpu:0 --field-id 1110,1130,1131 --count 10Monitor an NVIDIA CPUReads three CPU-level fields for CPU 0 and stops after the tenth sample.
dcgmi dmon --entity-id 'core:{0-3}' --field-id 1100,1101,1103,1120 --delay 1000 --count 10Monitor CPU coresFollows utilization and clock speed for four cores, sampling every 1,000 ms.

Field IDs for dmon and field groups 46

CommandWhat it does
150GPU temperatureDCGM_FI_DEV_GPU_TEMP_CELSIUS: the current GPU temperature reading in degrees Celsius.
140Memory temperatureDCGM_FI_DEV_MEMORY_TEMP_CELSIUS: the temperature reported for the GPU memory.
158Slowdown temperatureDCGM_FI_DEV_GPU_TEMP_SLOWDOWN_CELSIUS: the threshold at which the GPU begins to slow down.
155Power usageDCGM_FI_DEV_BOARD_POWER_WATTS: board power draw in watts.
157Instantaneous power usageDCGM_FI_DEV_BOARD_POWER_RAW_WATTS: the instantaneous reading in watts, next to the regular value in field 155.
160Power limitDCGM_FI_DEV_BOARD_POWER_LIMIT_REQUESTED_WATTS: the power limit currently set for the GPU, in watts.
156Energy since driver loadDCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION in mJ; deprecated in favour of field 1611, which counts whole joules.
203GPU utilizationDCGM_FI_DEV_GPU_UTIL: integer percent; field 1613 reports the same measurement as a 0.0 to 1.0 ratio.
1613GPU utilization ratioDCGM_FI_DEV_GPU_UTIL_RATIO: a double between 0.0 and 1.0, one of the base-unit fields added in 4.7.0.
204Memory copy utilizationDCGM_FI_DEV_MEM_COPY_UTIL: an integer percentage; the ratio form is field 1614.
206Encoder utilizationDCGM_FI_DEV_ENC_UTIL: an integer percentage; field 207 is the decoder counterpart.
250Framebuffer totalDCGM_FI_DEV_FB_TOTAL in MiB; field 1625 gives the capacity in bytes.
251Framebuffer freeDCGM_FI_DEV_FB_FREE in MiB; field 1626 gives the same value in bytes.
252Framebuffer usedDCGM_FI_DEV_FB_USED in MiB; field 1627 gives the same value in bytes.
254Framebuffer used ratioDCGM_FI_DEV_FB_USED_RATIO: used memory divided by total minus reserved, from 0.0 to 1.0.
100SM clockDCGM_FI_DEV_SM_CLOCK in MHz; field 1603 reports hertz.
101Memory clockDCGM_FI_DEV_MEM_CLOCK in MHz; field 1604 reports hertz.
190Performance stateDCGM_FI_DEV_GPU_PSTATE: P-state number, where 0 is the highest of 16 states.
191Fan speedDCGM_FI_DEV_FAN_SPEED in percent, which can exceed 100; field 1612 is the ratio form.
112Clock event reasonsDCGM_FI_DEV_CLOCKS_EVENT_REASONS: bitmask of the clock event reasons active now, such as HW_SLOWDOWN or SW_THERMAL.
202PCIe replay counterDCGM_FI_DEV_PCIE_REPLAY_TOTAL; the error-injection walkthrough writes 99999 into it to trip a policy.
230XID errorDCGM_FI_DEV_XID_ERROR: the reading is the number of the XID that occurred.
310Single-bit ECC errors, volatileDCGM_FI_DEV_ECC_SBE_VOL_TOTAL: the overall volatile count; fields 314, 316 and 318 split it into L1 cache, L2 cache and device memory.
311Double-bit ECC errors, volatileDCGM_FI_DEV_ECC_DBE_VOL_TOTAL: the overall volatile count, one of the things the memory health watch evaluates.
312Single-bit ECC errors, aggregateDCGM_FI_DEV_ECC_SBE_AGG_TOTAL: persistent counter that only ever increases.
313Double-bit ECC errors, aggregateDCGM_FI_DEV_ECC_DBE_AGG_TOTAL: persistent counter of double-bit errors that is monotonically increasing.
390Pages retired for single-bit errorsDCGM_FI_DEV_PAGE_RETIRED_SBE_TOTAL: a count that only ever increases.
391Pages retired for double-bit errorsDCGM_FI_DEV_PAGE_RETIRED_DBE_TOTAL: monotonically increasing, like field 390 for single-bit errors.
392Pages pending retirementDCGM_FI_DEV_PAGE_RETIRED_PENDING: 1 when any page waits to be retired, otherwise 0.
393Rows remapped, uncorrectableDCGM_FI_DEV_ROW_REMAP_UNCORRECTABLE_TOTAL: remapped rows caused by errors that could not be corrected.
394Rows remapped, correctableDCGM_FI_DEV_ROW_REMAP_CORRECTABLE_TOTAL: remapped rows caused by errors that were corrected.
395Row remap failureDCGM_FI_DEV_ROW_REMAP_FAILED: tells if a row remapping attempt did not succeed.
1001Graphics engine activityDCGM_FI_PROF_GR_ENGINE_UTIL_RATIO: share of time a graphics or compute context is bound and busy; dmon column GRACT.
1002SM activityDCGM_FI_PROF_SM_UTIL_RATIO: share of cycles in which a streaming multiprocessor has work assigned.
1003SM occupancyDCGM_FI_PROF_SM_OCCUPANCY_RATIO: resident warps relative to the theoretical maximum per SM.
1004Tensor Core activityDCGM_FI_PROF_TENSOR_UTIL_RATIO: share of cycles a tensor pipe is busy; dmon column TENSO.
1005Memory interface activityDCGM_FI_PROF_DRAM_UTIL_RATIO: share of cycles spent moving data to or from device memory; column DRAMA.
1009PCIe transmit bytesDCGM_FI_PROF_PCIE_TX_BYTES: header plus payload sent by the GPU, so device-to-host copies appear here.
1010PCIe receive bytesDCGM_FI_PROF_PCIE_RX_BYTES: header plus payload arriving at the GPU, so host-to-device copies appear here.
1011NVLink transmit bytesDCGM_FI_PROF_NVLINK_TX_BYTES: aggregate over all links for a GPU entity, or one link for a link entity.
1012NVLink receive bytesDCGM_FI_PROF_NVLINK_RX_BYTES: per GPU it sums every link; select a gpu_link entity for a single port.
1200NVLink transmit packetsDCGM_FI_DEV_NVLINK_TX_PACKET_TOTAL: NVLink5 and later only; unsupported on NVLink4 and older.
1202NVLink receive packetsDCGM_FI_DEV_NVLINK_RX_PACKET_TOTAL: cumulative counter that needs NVLink5 or newer hardware.
1100CPU utilizationDCGM_FI_DEV_CPU_UTIL_RATIO: total utilization of an NVIDIA CPU core as a ratio.
1110CPU temperatureDCGM_FI_DEV_CPU_TEMP_CELSIUS: degrees Celsius for an NVIDIA CPU entity.
1130CPU power usageDCGM_FI_DEV_CPU_POWER_WATTS: present draw of an NVIDIA CPU in watts.

GPU configuration 13

CommandWhat it does
dcgmi config --getShow configShows current and target values for the built-in all-GPU group: clocks, ECC, power limit, compute mode, power profiles.
dcgmi config --get --group 5 --verbosePer-GPU configuration--verbose prints each GPU separately instead of a group summary.
dcgmi config --set --group 1 --compmode 2 --powerlimit 300Set compute mode and power limitDCGM keeps both values as the target and reapplies them after a GPU reset while the host engine runs.
dcgmi config --set --group GROUP-ID --compmode 0Restore unrestricted compute modeDeleting a group does not undo its settings, so a job epilogue should request mode 0 first.
--compmode MODECompute mode valuesWith config --set: 0 is unrestricted, 1 prohibited and 2 exclusive process; short form -c.
--eccmode 1Enable ECC modeWith config --set, 1 turns ECC on and 0 turns it off; short form -e.
--syncboost 1Enable synchronized boostWith config --set, switches sync boost on for the group; every GPU in it must be compatible.
--appclocks MEMORY,SMSet application clocksAn exact supported pair in MHz, memory clock first; list valid pairs with discovery --info c.
--powerlimit WATTSSet the power limitApplies to each GPU and has to fall inside the range that GPU supports; short form -P.
dcgmi config --set --group 5 --workloadpowerprofile 2 --workloadpowerprofileaction aAdd a workload power profileAction a keeps the profiles already requested and adds this index; it is also the default action.
dcgmi config --set --group 5 --workloadpowerprofile 3 --workloadpowerprofileaction oOverwrite power profilesAction o makes profile 3 the only requested profile of the group; c would clear just that one.
dcgmi config --set --workloadpowerprofile -1Clear all power profilesIndex -1 removes every requested profile; the action option only matters for indexes of 0 and above.
dcgmi config --enforce --group 1Reapply target configurationPushes the stored target back onto every GPU of the group when the current state has drifted.

Modules, logging and driver attach 6

CommandWhat it does
dcgmi modules --listList modulesShows ID, name and state of each module; Not loaded is normal for a module nothing has used yet.
dcgmi modules --denylist ProfilingDenylist a moduleWorks only while the module is Not loaded and lasts for the running host engine; accepts a name or numeric ID.
dcgmi introspect --show --hostengineHost engine resource useReports memory in KB and CPU utilization of the host engine process; the first query can take longer.
dcgmi set --logging-severity INFO --target-logger BASEChange log level at runtimeAffects one logger of the running host engine until it restarts; the loggers are BASE and SYSLOG.
dcgmi set --detach-driverDetach the NVIDIA driverStops diagnostics, marks every GPU detached and drops its watches and group membership, so the driver can be updated.
dcgmi set --attach-driverReattach the driverRediscovers available GPUs but does not rebuild groups or reapply configuration removed by the detach.

Profiling metrics and load generation 10

CommandWhat it does
dcgmi profile --list --entity-id gpu:0List profiling metricsShows the metric groups, field IDs and tags this GPU offers; the set differs between GPU generations.
dcgmi profile --list --group-id 2 --jsonProfiling metrics for a groupReports only the first GPU of the group, as JSON.
dcgmi profile --pausePause profilingFrees the profiling hardware for tools such as Nsight for the whole host engine; fields read blank meanwhile.
dcgmi profile --resumeResume profilingTurns collection back on after a pause; do it as soon as the external profiler has finished.
dcgmi dmon -e 1001,1004,1005Watch profiling metricsStreams graphics engine, Tensor Core and memory interface activity in the columns GRACT, TENSO and DRAMA.
dcgmi dmon -e 1001,1004 -g 8Profiling fields for a MIG groupGroup 8 holds a GPU and two of its instances, so dmon prints a row for each MIG instance too.
dcgmi dmon --entity-id gpu:0 --field-id 1002,1005 --delay 1000Profile one GPUOne-second stream of SM activity and memory interface activity for GPU 0.
dcgmproftester13 -i 0 -t 1004,1005 -d 20 --no-dcgm-validationGenerate a test loadCreates CUDA load for the listed profiling fields on GPU 0 for 20 seconds, so a running host engine can measure it.
dcgmproftester12 -i 2 -t 1009,1010 -d 15 --no-dcgm-validationGenerate PCIe trafficThe numeric suffix must match the CUDA major version the driver reports: 11, 12 or 13.
sudo dcgmproftester13 -i 0 -t 1004 -d 30 --mode generateload,report,validate,nofast --percent-tolerance 5Validate a profiling fieldChecks measured against generated load and allows a 5 percent gap; it must be the only profiling consumer.

Multi-node diagnostics 6

CommandWhat it does
dcgmi mndiag --hostList 'node01;node02;node03'Run across several hostsRuns mnubergemm, the default test, on every active GPU; quote the list so the shell keeps the semicolons.
dcgmi mndiag --hostList 'node01:5000;node02:unix:///run/dcgm/hostengine.sock'Custom port or socket per hostA host without a port uses TCP 5555; host:unix:// plus a path selects a remote Unix socket.
dcgmi mndiag --hostfile hosts.txt --run mnnvbandwidthHosts from a fileOne host entry per line; --hostfile and --hostList are mutually exclusive, and one command takes at most 576 entries.
dcgmi mndiag --hostList 'node01;node02' --run nvloom --parameters 'nvloom.time_to_run=1200;nvloom.suite=fabric-stress'NVLink fabric testnvloom validates the multi-node NVLink fabric; parameters use the test name as prefix.
dcgmi mndiag --hostList 'node01;node02' --parameters mnubergemm.time_to_run=600 --jsonTimed run as JSONExit status 0 only means the command completed; read the rendered result for Pass, Fail, Warn or Skip.
--hostEngineAddress ADDRESSChoose the head nodeThe host engine that coordinates the run; defaults to localhost and is separate from the test-node list.

Error injection for testing 6

CommandWhat it does
dcgmi test --inject --gpuid 0 -f 202 -v 99999Inject a PCIe replay countWrites a synthetic value into field 202 in the cache; use only on an isolated test host engine.
dcgmi test --inject --gpuid 0 -f 319 -v 4Inject double-bit ECC errorsField 319 counts device-memory double-bit volatile ECC errors; the hardware counter itself is untouched.
dcgmi test --inject --gpuid 0 --field 150 --value 10 --offset 1Inject a temperature value--offset timestamps the sample that many seconds in the future; 1 second is the default.
dcgmi test --introspect --gpuid 0 --field 150Inspect cached field dataShows sample count, timestamps, monitor interval, retention age and watchers for the field.
dcgmi test --pausePause all metric collectionEvery watcher of the host engine receives unavailable values until --resume or a restart.
dcgmi test --resumeResume metric collectionEnds a collection pause that would otherwise persist for all clients of the host engine.

Notes

Diagnostics

dcgmi diag -r 1 runs only the software deployment checks and normally finishes in seconds, and dcgmi diag -r 2 adds the memory and PCIe tests for a run of about two minutes. dcgmi diag -r 3 adds the diagnostic, memory bandwidth, targeted stress and targeted power tests and normally lasts about fifteen minutes; dcgmi diag -r 4 puts memtest and the pulse test on top. Each level includes the ones below it. The longer suites load GPU compute, memory, power and interconnect, so they belong on drained nodes, and a single test can be run by name, as in dcgmi diag -r nvbandwidth. NVIDIA says the diagnostics do not replace its offline field diagnostics and do not determine RMA eligibility. They run through nv-hostengine, and dcgmi -v prints the build information NVIDIA asks for in a problem report. In a shell, exit status 226 means the diagnostic ran and reported an error.

Health and monitoring

dcgmi health -s a replaces the group's watch mask with every subsystem (PCIe, memory, InfoROM, thermal and power, NVLink, driver, ConnectX, IMEX, NVSwitch) and dcgmi health -c evaluates what those watches have retained. It is passive: no test load is applied, and several watches need about a minute of samples before the first useful check. dcgmi dmon --field-id 150,155 --count 5 prints temperature and power per GPU; dmon always needs a field list or a field group, and dcgmi dmon --list shows the IDs with their short column names.

Groups, stats and topology

dcgmi discovery -l lists GPUs with their DCGM IDs. Many commands act on a group: dcgmi group -c NAME creates one and dcgmi group -g GROUPID -a ENTITYID adds members, after which health, configuration, diagnostics and statistics can target that subset. dcgmi stats -e starts the process-accounting watches, which have to be running before the workload; a job is then bracketed with dcgmi stats --jstart batch-4815 --group 1 and dcgmi stats --jstop batch-4815. dcgmi topo -g GROUPID reports CPU affinity, NUMA optimality and the slowest interconnect path of a group. dcgmi config --get shows current and target settings such as compute mode, power limit and ECC. Groups, watches, job records and configuration targets live in the memory of the host engine and are gone after it restarts.

Questions

What is the difference between dcgmi diag -r 1, 2, 3 and 4?

dcgmi diag -r 1 runs the software deployment checks in seconds. dcgmi diag -r 2 adds the memory and PCIe tests and takes about two minutes. dcgmi diag -r 3 adds the diagnostic, memory bandwidth, targeted stress and targeted power tests and takes about 15 minutes. dcgmi diag -r 4 adds memtest and the pulse test and runs longest. Each level includes the lower ones.

Does dcgmi need nv-hostengine to be running?

Yes. dcgmi sends its requests to a DCGM host engine, by default on localhost. On a packaged install start it with sudo systemctl enable --now nvidia-dcgm and confirm with dcgmi discovery -l. Only dcgmi --version and the help output work without a connection.

Which dmon field IDs show GPU temperature, power, utilization and memory?

Field 150 is GPU temperature in degrees C, 155 is power usage in watts, 203 is GPU utilization in percent and 252 is used framebuffer in MiB. Pass them as a comma-separated list, as in dcgmi dmon --field-id 150,155 --count 5; dcgmi dmon --list prints every ID.

Is dcgmi health the same as dcgmi diag?

No. Health watches are passive: dcgmi health -c evaluates samples that the enabled watches collected during normal operation and applies no load. A diagnostic such as dcgmi diag -r 3 actively exercises the hardware and should run on GPUs taken out of service. A node can be healthy and still fail a diagnostic.

How do I record GPU statistics for one scheduler job?

Enable the watches once with dcgmi stats -e, call dcgmi stats --jstart batch-4815 --group 1 with your own job ID just before the workload, dcgmi stats --jstop batch-4815 after it, and read the result with dcgmi stats --job batch-4815 --verbose.

Do DCGM groups and watches survive a restart of the host engine?

No. Groups, field groups, watches, job records, health settings and configuration targets are in-memory objects of the running host engine and are lost when it restarts, so sudo systemctl restart nvidia-dcgm.service means recreating them. Field-based policies are the exception: the host engine persists them.

Sources and checks

43 more source pages

Every row on this page was matched to a line in the documentation above on . A binding that could not be found there is not listed.

Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.