NVIDIA DGX commands

Command lines for administering an NVIDIA DGX system, taken from the DGX OS 7 User Guide, the NVSM User Guide (release 25.03) and the DGX B200 user and firmware update guides. NVSM needs root, so its rows carry sudo as the guides print them, while short rows such as show GPU6 are typed inside an interactive sudo nvsm session. Words in capitals, some in angle brackets, stand for your own values. Redfish curl calls and installation procedures are left out.

191 commands in 14 sections. Checked row by row against NVIDIA System Management (NVSM) User Guide, release 25.03 on . Edited by Jay.

System health with NVSM 13

On DGX Station only show health and dump health are supported.

The nvsm CLI answers only while all NVSM services are up and running.

CommandWhat it does
sudo nvsm show healthShow system healthRuns the NVSM checks across subsystems and ends with an overall status line; the place to start after any alert.
sudo nvsm show health -swhCheck software healthVerifies DGX OS and container-stack packages, services and settings; the long flag is --software_health.
sudo nvsm show alertsList active alertsOne list of the warning and critical alerts raised by every subsystem.
sudo nvsm show versionsShow component versionsLists on-board hardware with its firmware plus the installed NVSM, DCGM and OS versions; expect a run of about three minutes.
sudo nvsm statusShow NVSM service statusTabulates the mqtt, core, api-gateway and notifier services, then gives an overall health and status.
sudo systemctl status nvsm-coreCheck the NVSM core serviceNVSM runs as several systemd units; the nvsm CLI works only when all of them are up.
sudo systemctl status -all nvsm*List every NVSM serviceShows the state of each NVSM unit in one systemctl call.
man nvsmOpen the NVSM man pageInstalled together with NVSM; each verb also has its own page, such as nvsm_show.
sudo nvsm --helpPrint NVSM helpPrints much the same overview as the man page: verbs, global options and examples.
sudo nvsm show -hHelp for the show verbThe same -h flag works after the other verbs, for example after set, start or stress-test.
sudo nvsm stress-test [--usage] [--force] [--no-prompt] [TEST...] [DURATION]Stress-test syntaxLoads GPUs, PCIe, DIMMs, drives, CPUs and network cards together and closes with a pass or fail summary per component.
sudo nvsm stress-test --usageShow stress-test help
sudo nvsm stress-test --forceRun the pre-flight stress testAbout 20 minutes over GPUs, processors, memory and storage; NVIDIA recommends it before production use and after servicing.

Interactive NVSM session 17

CommandWhat it does
sudo nvsmStart an interactive sessionOpens the nvsm-> prompt, where verbs are typed without the sudo nvsm prefix and cd moves between targets.
exitLeave the interactive sessionTyped at the nvsm-> prompt; returns to the system shell.
show fansRun a command at the promptAny NVSM command works here without the sudo nvsm prefix; the shell and prompt forms are interchangeable.
cd /systems/localhost/storage/alertsGo to past storage alertsEach earlier alert is a target such as alert0; a bare show lists them.
show alert1Show one alertPrints the message details, component ID, event time and recommended action of that alert.
cd /systems/localhost/processors/alertsGo to past processor alerts
cd /systems/localhost/memory/alertsGo to past memory alerts
cd /chassis/localhost/thermal/alertsGo to past thermal alerts
cd /chassis/localhost/power/alertsGo to past power alerts
cd /systems/localhost/storage/drivesGo to the drive targetsEvery drive is a target named after its device, such as nvme2n1.
show nvme2n1Show one drive
cd /systems/localhost/memoryGo to the memory targetsDIMMs appear by slot name, next to the alerts and policy targets.
show CPU2_DIMM_B1Show one DIMM
cd /chassis/localhost/thermal/fansGo to the fan targetsChassis items live under /chassis/localhost, system items under /systems/localhost.
show PDB_FAN2Show one fan
cd /chassis/localhost/powerGo to the power supply targets
show PSU4Show one power supplyPresence, health, last power output in watts, model and serial number.

Health reports and support bundles 13

CommandWhat it does
sudo nvsm dump healthCreate a health reportWrites a compressed tar archive under /tmp, named nvsm-health with host name and timestamp, for a support case.
nvsm dump health -qkdCreate a quick health reportShort for --quick_dump: finishes sooner but uses more CPU and memory; the guide shows it at a root prompt.
sudo tar xlf nvsm-health-dgx-1-20180907085048.tar.xzUnpack a health reportThe archive holds sos reports plus outputs such as dmidecode, lspci, lsmod and nvsysinfo_log.txt.
sudo nvsm set /callhome trigger=trueSend a Call Home report nowOn-demand mode opens a support case with a captured health dump; it needs internet access and a registered system.
sudo nvsm set /callhome trigger=true diagtest=trueTest Call Home readinessChecks that the system can reach NVIDIA Enterprise Services without opening a ticket.
sudo nvsm show /callhomeShow the last Call Home resultAfter a diagnostic run it confirms whether the setup is ready for Call Home.
sudo nvsm set /callhome trigger=falseCancel a Call Home in progress
sudo nvsm set /policy callhome_enable=trueEnable automatic Call HomeAutomatic mode sends alerts in batches at set intervals to the NVIDIA Enterprise Support portal.
sudo nvsm set /policy callhome_enable=falseDisable automatic Call HomeUse it before maintenance such as reseating parts, so the resulting critical alerts do not open support cases.
sudo nvsm set /policy offline_callhome_enable=trueEnable offline Call HomeFor air-gapped sites: alert and system data stay on the local system for manual hand-over to NVIDIA.
nvdebug -i BMCIP -u BMCUSER -p BMCPASS -t PLATFORMCollect debug logs through the BMCOut-of-band collection into a zip under /tmp; -t names the platform type, which is DGX for a DGX system.
nvdebug -l -t DGXList DGX log collectorsPrints each collector ID, name and log location, grouped by method such as Redfish.
nvdebug -i $BMC_IP -u $BMC_USER -p $BMC_PASS -t DGX -g RedfishCollect only Redfish logsThe -g option limits the run to one log group.

GPUs 13

CommandWhat it does
sudo nvsm show gpusList GPUs and their healthInventory and health for each GPU; NVSM takes GPU health from continuous DCGM monitoring.
cd /systems/localhost/gpusGo to the GPU targetInteractive session only: later show commands then work relative to the GPU target.
show -display propertiesShow the GPU health rollupIn the gpus target, Status_HealthRollup sums up all GPUs; Critical means at least one has a health incident.
show -display properties=*health /systems/localhost/gpus/*List the health of every GPUWildcards narrow the output to the health properties of each GPU target, which singles out the failing one.
show -level all GPU14/healthShow one GPU's health incidentsDescends into the incident targets under that GPU and prints each message, for example an NVLink link that is down.
show GPU6Show one GPURun inside the gpus target; prints the properties of that single GPU.
nvidia-smiCheck the GPU driverIts header carries the NVIDIA driver and CUDA versions; the guide uses it to confirm an upgrade.
nvidia-smi -LList GPUs with UUIDsOne line per GPU or MIG device with its UUID, the identifier that Docker accepts in device=.
nvidia-smi topo -mShow the GPU topology matrixMatrix of GPU and NIC connections with CPU affinity; NV# marks a bonded set of NVLinks.
nvidia-smi nvlink -sShow NVLink statusActive links show their bandwidth; a link that is present but down is reported as Inactive.
dcgmi diag -r 3Run the long DCGM diagnosticLevel 3 takes about 15 minutes and adds memory, PCIe/NVLink and targeted stress and power tests.
dcgmi health -g 1 -cCheck DCGM group healthGives an overall health score for group 1 and the problems found on each affected GPU.
dcgmi health -g 1 -s mpiSet DCGM health watchesTurns on the memory, PCIe and InfoROM watches for group 1; the letter a selects every watch.

Processors, memory, cooling and power 11

CommandWhat it does
sudo nvsm show processorsList CPUsHealth and properties for each installed CPU; show cpus is an alias.
sudo nvsm show memoryList memory modulesOne entry per installed DIMM with its health; show dimms is an alias.
sudo nvsm show fansList fansSpeed in RPM, thresholds and health for each fan.
sudo nvsm show temperaturesList temperature sensorsReadings from every sensor NVSM knows about; show temps is the short alias.
sudo nvsm show powerList power suppliesHealth and last output wattage for each power supply unit.
sudo nvsm show psusList power supplies (alias)Same output as show power under a PSU-named alias.
sudo nvsm show powermodeShow the power modeReports the current system power mode, MaxQ or MaxP on a DGX-2.
sudo nvsm set powermode=maxqSwitch to MaxQ modeDGX-2: efficiency mode that lets two systems share an 18 kW rack; the setting survives reboots.
sudo nvsm set powermode=maxpReturn to MaxP modeDGX-2 default: GPUs run unconstrained up to their thermal design power.
sudo nvsm show nvswitchesList NVSwitch interconnects
ipmitool sdr elist | grep -I fanList fan sensors from the BMCA reading of exactly 1848 RPM usually means the fan module is missing or unseated.

Storage and RAID 18

Changing the RAID level erases the data stored on the data array.

CommandWhat it does
sudo nvsm show storageShow all storage informationCovers the storage target, which groups drives, volumes, alerts and policy.
sudo nvsm show drivesList storage drives
sudo nvsm show volumesList storage volumesIncludes the RAID volumes and the EFI system partition volume with their health.
sudo nvsm show controllersList storage controllers
cd /systems/localhost/storageGo to the storage targetInteractive session: the starting point for viewing and rebuilding volumes.
show volumes/md0Show the OS RAID volumeA degraded mirror shows one drive and Status_Health = Critical.
start volumes/rebuildRebuild a RAID or ESP volumeAsks for raid-0, raid-1 or esp and, for RAID 1, the replaced drive such as nvme3; it cannot be stopped once started.
mdadm --detail /dev/md0Show RAID array detailsPrints level, state, member devices and the UUID that the NVSM volume mute policies take.
sudo configure_raid_array.py -m raid5Change the data array to RAID 5Erases the data array and rebuilds it with redundancy and less capacity; no unmount is needed.
sudo configure_raid_array.py -m raid0Change the data array to RAID 0Factory level: maximum capacity and no redundancy; changing level erases what the array holds.
sudo /usr/bin/configure_raid_array.py -c -fRecreate the RAID 0 data arrayRebuilds the /raid array, for example after nv-disk-encrypt erase removed it.
lsblkConfirm the RAID levelThe TYPE column of each data drive names the level of the array.
sudo systemctl status nvidia-raid-configCheck first-boot RAID setupWait for the Finished message, typically about 10 minutes after first boot, before rebooting.
sudo nvme listList NVMe drivesNode, serial number, model and firmware revision of every NVMe device.
sudo systemctl stop cachefilesdStop NFS cachingcachefilesd caches NFS mounts on the data drives; the next row keeps it from starting again.
sudo systemctl disable cachefilesdDisable NFS caching permanently
cat /proc/fs/nfsfs/volumesVerify NFS cachingFSC=yes in the output means the mount is cached on the data drives.
nvme format -s1 DEVICE-PATHSecure-erase an NVMe driveB200 data-deletion step, run from the installer rescue shell on a node such as /dev/nvme0n1; it destroys the data.

Drive encryption 11

CommandWhat it does
sudo apt install -y nv-disk-encryptInstall the SED management toolManages self-encrypting data drives only, never OS drives; reboot after installing.
sudo nv-disk-encrypt infoShow which drives can be encryptedPer drive: SED capable, boot disk, locked, lock enabled; only non-boot SEDs can be managed.
sudo nv-disk-encrypt info -jShow drive status as JSONEligible drives read sed_capable true and used_for_boot false.
sudo nv-disk-encrypt init -gInitialise, typing each passwordPrompts for a vault password and then one per eligible drive; -g adds random salt values.
sudo nv-disk-encrypt init -k YOUR-VAULT-PASSWORD -g -rInitialise with random passwordsNothing is prompted: -k passes the vault password and -r generates a password for each drive.
sudo nv-disk-encrypt lockEnable drive lockingRun after init: drives lock when they lose power and unlock by themselves once the system boots again.
sudo nv-disk-encrypt disableDisable drive lockingTurns locking off on all drives; the first step before changing passwords or adding or replacing disks.
sudo nv-disk-encrypt export -k yourvaultpasswordExport the drive keysWrites the vault contents to /tmp/secrets.out; back the keys up, because lost keys force a data-destroying factory reset.
sudo umount /raidUnmount the data arrayPart of stopping the RAID before an erase, after cachefilesd has been stopped.
sudo mdadm --stop /dev/md1Stop the data RAID arrayLast step before the erase; md1 is the data array in the guide.
sudo nv-disk-encrypt eraseErase the encrypted drivesDestroys all data: unlocks the drives, disables locking and removes the RAID 0 array.

Network adapters and InfiniBand 18

CommandWhat it does
sudo nvsm show networkadaptersList physical network adaptersThe adapter IDs it prints, such as PCI0000_0c_00, are what the port and policy targets use.
sudo nvsm show networkinterfacesList logical network interfaces
sudo nvsm show networkportsList network ports
sudo nvsm show networkdevicefunctionsList adapter PCIe functions
sudo nvsm show /chassis/localhost/NetworkAdaptersList adapter IDsPrints the adapter targets by PCI address, the IDs used when muting notifications.
sudo nvsm muteNICShow link-down alert mutingWith no arguments it lists each NIC, its PCI interface and whether its NV-NET-01 alert is muted.
sudo nvsm muteNIC ibp220s0=enable,ibp41s0f1=enableMute link-down alertsFor unused adapters: comma-separated NIC=enable pairs stop their NV-NET-01 alerts.
sudo nvsm muteNIC ibp220s0=disable,ibp41s0f1=disableUnmute link-down alertsRestores alerting; the alert database must also be cleared before the alerts show again.
sudo nvsm_database_rotate.sh -d alertClear the NVSM alert databaseStops NVSM, empties the stored alerts and starts it again.
sudo nvsm show /chassis/localhost/NetworkAdapters/policyShow muted adaptersThe mute_monitoring and mute_notification properties read NOT_SET when nothing is muted.
sudo nvsm set chassis/localhost/NetworkAdapters/policy mute_notification=PCI0000_0c_00,PCI0000_12_00Mute adapter notificationsTakes a comma-separated list of adapter IDs.
sudo mlxconfig -e -d aa:00.0 q | grep -e 'Device\|Configurations\|LINK_TYPE'Query a port's link typeIB(1) is InfiniBand and ETH(2) Ethernet, shown for the default, current and next-boot configuration.
sudo mlxconfig -y -d aa:00.0 set LINK_TYPE_P1=2Set a ConnectX port to EthernetThe value 2 selects Ethernet; power the system off and on, and reconfigure the switch side as well.
mlxconfig -d DEVICE set NUM_OF_PLANES_P1=0Set ConnectX-8 planes to zeroA step in splitting one 800 Gbps InfiniBand port into two 400 Gb Ethernet ports.
mlxconfig -d DEVICE set NUM_OF_PF=2Expose two physical functionsFinal split step: the OS then sees two 400 GbE interfaces after a power cycle.
mlxconfig -d DEVICE resetReset a ConnectX-8 deviceReturns the port to a single 800 Gbps InfiniBand link and reverts every other setting on the device.
sudo flint -d 05:00.2 q full | grep "FW Version"Query ConnectX firmware versionDGX B200 example; 28.47.1026 or later resolves the ACCESS_REG errors in the kernel log.
sudo netplan applyApply the network configurationAfter editing /etc/netplan/01-netcfg.yaml for a static address; an SSH session drops here, so use the BMC console.

Alerts and monitoring policies 19

CommandWhat it does
sudo nvsm show policyShow alert policiesPolicy settings for the subsystems, such as poll intervals and muted devices.
sudo nvsm show /policyShow the global policyThe /policy target holds the email settings used for alert notification.
sudo nvsm set /policy [email protected]Set the alert sender addressIt has to be a valid email address, or no alert mail goes out.
sudo nvsm set /policy email_smtp_server_name=smtpserver.nvidia.comSet the SMTP relay server
sudo nvsm set /policy [email protected],[email protected]Set the alert recipientsSeveral addresses are separated by commas; each new alert is mailed to all of them.
sudo nvsm set /policy email_smtp_server_port=465Set the SMTP port
sudo nvsm create testalertCreate a test alertRaises a dummy Warning alert on component Test0, which exercises the email or Call Home notification path.
sudo nvsm create testalert Email1 CriticalCreate a custom test alertThe two arguments are a component ID and a severity.
sudo nvsm clear testalertClear test alertsDismisses every test alert on component Test0; append a component ID to clear a different one.
sudo nvsm show testalertsList test alerts
sudo nvsm show /systems/localhost/gpus/policyShow the GPU policyIts mute_notification and mute_monitoring lists suppress alerts or monitoring for chosen GPUs.
sudo nvsm show /systems/localhost/memory/policyShow the memory policyCovers monitoring of DIMM correctable and uncorrectable errors.
sudo nvsm show /systems/localhost/processors/policyShow the processor policyCovers monitoring of CPU machine-check exceptions.
sudo nvsm show /systems/localhost/storage/policyShow the storage policyPoll intervals and mute lists for controllers, drives and volumes.
sudo nvsm show /systems/localhost/pcie/policyShow the PCIe policy
sudo nvsm show /chassis/localhost/thermal/policyShow the thermal policyHolds separate mute lists for fans and for PDB sensors.
sudo nvsm show /chassis/localhost/power/policyShow the power policyIts mute lists exclude chosen power supplies from monitoring or from notification.
nvsm set /systems/localhost/storage/policy volume_mute_monitoring=<UUID>Stop monitoring a volumePut the array UUID that mdadm --detail reports in place of <UUID>; the guide shows the command at a root prompt.
nvsm set /systems/localhost/storage/policy volume_mute_notification=<UUID>Silence alerts for a volumeSuppresses alert notification for the array with that UUID, which mdadm --detail reports.

BMC and remote console 13

CommandWhat it does
sudo nvsm set -bmccredStore BMC credentials for NVSMDGX H100: prompts for a BMC user and password and keeps them encrypted on the host for Redfish queries.
sudo ipmitool lan print 1Show BMC network settingsRun it first to see the current settings before switching to a static address.
sudo ipmitool lan set 1 ipsrc staticUse a static BMC addressFirst step; the address, netmask and gateway follow.
sudo ipmitool lan set 1 ipaddr MY-IP-ADDRESSSet the BMC IP addressThe BIOS setup calls this value the Station IP address.
sudo ipmitool lan set 1 netmask MY-NETMASK-ADDRESSSet the BMC subnet mask
sudo ipmitool lan set 1 defgw ipaddr MY-DEFAULT-GATEWAY-IP-ADDRESSSet the BMC default gatewayThe BIOS setup calls this value the Router IP address.
ipmitool -I lanplus -H BMC-IP -U USERNAME -P PASSWORD sol activateOpen the serial-over-LAN consoleKeeps console output visible during a vmcore crash dump, when the BMC screen goes blank.
sudo ipmitool mc reset coldReboot the BMCActivates a BMC firmware update; allow a couple of minutes for the BMC to return.
ipmitool mc infoCheck that the BMC respondsAnswers again once the BMC has finished restarting.
sudo ipmitool chassis power cycleCold-reset the systemActivates BIOS and GPU-tray firmware; PCIe switch, retimer, EROT and CPLD updates need an AC power cycle.
ipmitool raw 0x3c 0x80 0x05Read the system power limitDGX B200: two bytes come back, and a value above 12,000 means power capping is off.
ipmitool raw 0x3c 0x81 0x05 0x70 0x17Cap system power at 6 kWDGX B200 example that sets a limit below the budget the power supplies can carry.
sudo ifconfig enx9638a3b292ec 169.254.0.18 netmask 255.255.0.0Reach the BMC from the hostenx9638a3b292ec is the guide's example interface name; the host joins the internal network where the BMC answers Redfish at 169.254.0.17.

Firmware 11

Stop all GPU activity, nvidia-smi included, before a firmware update; it can block the VBIOS update.

CommandWhat it does
nvfwupd -t ip=<BMC-IP-ADDRESS> user=<BMC-USERNAME> password=<BMC-PASSWORD> show_version -p MB-TRAY-PACKAGE GPU-TRAY-LATEST-PACKAGECompare installed and package firmwareCompares running versions with those in the packages; run without package arguments it shows the installed versions only.
nvfwupd -t ip=<BMC-IP-ADDRESS> user=<BMC-USERNAME> password=<BMC-PASSWORD> update_fw -p MB-TRAY-PACKAGE -y -s update_bmc.jsonUpdate the BMC firmwareThe JSON file names the Redfish target, here HostBMC_0; -y skips the confirmation prompt.
nvfwupd -t ip=<BMC-IP-ADDRESS> user=<BMC-USER-ID> password=<BMC-PASSWORD> update_fw -p FIRMWARE-PACKAGE-FILE -y -s updparameters.json -bUpdate firmware in the backgroundWith -b the command prints a task ID and exits instead of following the update to the end.
nvfwupd -t ip=<BMC-IP-ADDRESS> user=<BMC-USER-ID> password=<BMC-PASSWORD> show_update_progress -i 1Show update task progressPrints the task state and percent complete for the given task ID.
nvfwupd -c config.yaml show_versionCompare versions from a config fileThe YAML file carries the target address, credentials and packages, so none appear on the command line.
nvfwupd -c config.yaml update_fwUpdate from a config fileUpdates the motherboard tray as described in the YAML file.
sudo mlxfwmanager -u -D .Update NIC firmware from a folderRun in the network directory of the package; confirms once for all ConnectX-7 adapters and BlueField-3 DPUs.
sudo nvme fw-commit $DeviceName --action=3Activate downloaded NVMe firmwareAction 3 commits the image and activates it at once, with no reset.
sudo nvme id-ctrl $DeviceNameVerify the NVMe firmware version
sudo nvsm show firmwareShow firmware with an update containerInteractive walk-through: pick a firmware update container that is already loaded and it reports the installed versions.
sudo nvsm update firmwareUpdate firmware with a containerSame guided flow as show firmware, but it runs the container to apply the update.

OS updates and release upgrade 17

CommandWhat it does
more /etc/dgx-releaseShow DGX OS release informationGives the ISO build version and every over-the-network update with its date.
sudo apt updateRefresh the package listsAlways the first step; a GPG error here points to the DOCA repository key issue.
sudo apt full-upgrade -sPreview a package upgradeA simulation that shows which packages would change.
sudo apt full-upgradeUpgrade all packagesInstalls every available update, kernel included; keep the current GRUB configuration when asked.
sudo rebootReboot after an upgradeNeeded for driver upgrades: until then nvidia-smi fails with a driver/library version mismatch.
sudo apt install -y doca-repoAdd the DOCA repositoryNeeded once on a DGX OS 7.0.0 install, whose DOCA repository location changed; follow with apt update.
sudo apt install -y nvidia-release-upgradeInstall the release upgraderFirst bring DGX OS 6 fully up to date; only the latest 6.x can move to 7.
sudo nvidia-release-upgradeUpgrade DGX OS 6 to 7MOFED drivers are replaced by DOCA-OFED along the way; reboot whenever the upgrade asks.
sudo -E nvidia-release-upgradeRelease-upgrade behind a proxyThe -E option keeps the proxy environment variables.
dpkg -a --configureReconfigure after a failed upgradeFirst recovery step once power or network is back and the system reaches a command line.
apt -f install -yFix broken package installsSecond recovery step after an interrupted release upgrade.
sudo pro attach XXXXXEnable Extended Security MaintenanceXXXXX stands for the Ubuntu Pro token that NVIDIA Enterprise Support supplies to DGX customers.
sudo pro statusCheck the Ubuntu Pro subscriptionAlso shows the expiry date after a token refresh.
sudo pro refreshRenew an expired Pro tokenThe client does not renew by itself when the token expires.
apt list --installed nvidia-driver*-openCheck the installed driver branchAn OS upgrade stays on the same GPU driver branch, so check before moving to a newer one.
apt list nvidia-driver-*-openList available driver branchesRun after apt update; shows the open GPU kernel module branches in the repository.
apt list --installed cuda-toolkit-*Check the installed CUDA Toolkit

Containers 9

CommandWhat it does
docker run --gpus all --rm -it ubuntu nvidia-smi -LTest GPU access in a containerLists only the compute GPUs; on DGX Station A100 the display GPU is hidden from Docker.
docker run --gpus 2 --rm -it ubuntu nvidia-smi -LGive a container two GPUsThe number after --gpus is how many GPUs the container is assigned.
docker run --gpus '"device=1,2"' ...Run on specific GPUsThe device list takes GPU indexes or UUIDs, quoted as shown.
sudo docker --versionCheck the Docker versionPart of the quick health check after setup.
sudo usermod -aG docker user-login-idLet a user run DockerAdds an existing login to the docker group; NVIDIA warns that group members can escalate to root.
sudo systemctl daemon-reloadReload systemd unit filesNeeded after editing docker-override.conf, before Docker is restarted.
sudo systemctl restart dockerRestart DockerApplies a changed bridge address or other daemon override.
sudo systemctl restart nv-docker-gpusRefresh the Docker GPU allow listDGX Station A100: run it with a Docker restart whenever MIG instances are created, destroyed or changed.
wget https://nvcr.io/v2Test access to the NGC registryA working path connects and then answers 401 Unauthorized.

Crash dumps and kernel settings 8

CommandWhat it does
/usr/sbin/nvidia-kdump-config enable-dmesg-dumpEnable dmesg crash dumpsCaptures the kernel log only and reserves memory for the crash kernel.
/usr/sbin/nvidia-kdump-config enable-vmcore-dumpEnable dmesg and vmcore dumpsSaves the vmcore as well as the kernel log, and likewise reserves crash-kernel memory.
/usr/sbin/nvidia-kdump-config disableDisable crash dumpsTurns kdump off, so no memory is held back for the crash kernel.
cat /sys/devices/system/cpu/vulnerabilities/*Check CPU mitigation stateLines that start with Mitigation: mean the mitigations are on; Vulnerable lines mean they are off.
sudo apt install nv-mitigations-off -yDisable CPU mitigationsTrades the side-channel mitigations for workload performance; it applies after a reboot.
sudo apt purge nv-mitigations-offRe-enable CPU mitigations
sudo update-grubRegenerate the GRUB configurationApplies an edit to /etc/default/grub, such as init_on_alloc=0, at the next reboot.
cat /proc/cmdlineVerify kernel parametersAfter the reboot the added parameter should be listed here.

Notes

→ Related guide: GPU cluster management: the full command guide

NVSM first

sudo nvsm show health runs the NVSM health checks and ends with an overall status, and is the first command after any alert; sudo nvsm show alerts lists what is currently raised. sudo nvsm show gpus, sudo nvsm show nvswitches and sudo nvsm show memory drill into the accelerators, the NVSwitch interconnects and the DIMMs; sudo nvsm show psus and sudo nvsm show fans cover power and cooling. sudo nvsm dump health packs logs and system state into an archive under /tmp for NVIDIA Enterprise Support — run it before opening a case. NVSM talks to a privileged API server, so every command needs sudo, and the CLI answers only while all NVSM services are running: sudo nvsm status shows which one is not.

GPUs and interconnect

nvidia-smi topo -m prints the matrix of connections between GPUs and NICs with their CPU affinity; NV# marks a bonded set of NVLinks and the other codes describe the PCIe path. nvidia-smi nvlink -s shows each NVLink: active links with their bandwidth, the rest as Inactive or Sleep. NVSM reports GPU faults as health incidents fed by DCGM: in an interactive session show -display properties in the gpus target gives the rollup, and show -level all GPU14/health lists the incidents of one GPU. dcgmi diag -r 3 runs the long DCGM diagnostic, which takes about 15 minutes, and dcgmi health -g 1 -c reads the health watches of a GPU group once dcgmi health -g 1 -s mpi has set them.

Storage and encrypted drives

The data drives ship as a RAID 0 array meant for caching, not for long-term data. sudo configure_raid_array.py -m raid5 trades capacity for redundancy and, like the change back with sudo configure_raid_array.py -m raid0, erases what the array holds. After a failed OS drive is replaced the mirror normally rebuilds by itself at power-on; when it does not, start volumes/rebuild in the storage target of an interactive session starts it. Self-encrypting data drives are set up with sudo nv-disk-encrypt init -g followed by sudo nv-disk-encrypt lock. Export the keys with sudo nv-disk-encrypt export -k yourvaultpassword and keep them somewhere safe, because lost keys leave only a factory reset that destroys the data.

Network ports

sudo nvsm show networkadapters lists the adapters NVSM monitors, and sudo nvsm muteNIC shows which ports have their link-down alert muted, which suits ports that are left unused on purpose. ConnectX ports come configured for InfiniBand; the mlxconfig rows switch a port to Ethernet, and the change applies after the system is powered off and on. The DGX guides do not print fabric diagnostics such as ibstat or a kernel-log search for XID errors, so those commands are not on this page.

Updates and firmware

more /etc/dgx-release shows the installed DGX OS version and each over-the-network update. Routine updates are sudo apt update, a look at sudo apt full-upgrade -s, then sudo apt full-upgrade and a reboot; the move from DGX OS 6 to 7 runs through sudo nvidia-release-upgrade from the latest 6.x release. Firmware on a DGX B200 is updated with nvfwupd against the BMC, and an update takes effect only after activation: sudo ipmitool mc reset cold for the BMC, sudo ipmitool chassis power cycle for the BIOS and the GPU tray, and an AC power cycle for PCIe switches, retimers, EROT and CPLDs.

Questions

Which command should I run first when a DGX system raises an alert?

sudo nvsm show health. It runs the NVSM checks and closes with an overall status line; sudo nvsm show alerts then lists the warning and critical alerts for every subsystem.

How do I collect logs for NVIDIA Enterprise Support?

sudo nvsm dump health writes a compressed archive under /tmp that can be attached to a support case. nvsm dump health -qkd is the quick variant, at the cost of more CPU and memory while it runs. For out-of-band logs through the BMC of a DGX B200, the firmware guide uses nvdebug, for example nvdebug -l -t DGX to list the collectors.

Why does an nvsm command fail or hang?

NVSM needs root, so the guides prefix every command with sudo, and the CLI works only when all NVSM services are up. sudo nvsm status shows each service and an overall status; sudo systemctl status nvsm-core gives the systemd detail for the core service.

How do I upgrade a DGX system from DGX OS 6 to DGX OS 7?

Bring DGX OS 6 up to date with sudo apt update and sudo apt full-upgrade, install the upgrader with sudo apt install -y nvidia-release-upgrade, then run sudo nvidia-release-upgrade, or sudo -E nvidia-release-upgrade behind a proxy. Reboot when prompted; the Mellanox OFED drivers are replaced by DOCA-OFED during the upgrade.

Which DGX OS and NVSM versions does this page follow?

The DGX OS 7 User Guide (edition of 30 September 2026, covering releases up to DGX OS 7.6.0), the NVSM User Guide release 25.03, and the DGX B200 user guide and firmware update guide, whose nvfwupd help text is version 2.1.1. Five GPU rows come from the DCGM and nvidia-smi documentation. Power-mode, power-capping and firmware rows are tied to the system named in their description.

Sources and checks

5 more source pages

Every row on this page was matched to a line in the documentation above on . A binding that could not be found there is not listed.

4 rows from the earlier version of this page were removed in that check: duplicates, and bindings the documentation does not list or lists differently.

Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.