System health with NVSM 13
On DGX Station only show health and dump health are supported.
The nvsm CLI answers only while all NVSM services are up and running.
| Command | What it does |
|---|---|
sudo nvsm show health | Show system healthRuns the NVSM checks across subsystems and ends with an overall status line; the place to start after any alert. |
sudo nvsm show health -swh | Check software healthVerifies DGX OS and container-stack packages, services and settings; the long flag is --software_health. |
sudo nvsm show alerts | List active alertsOne list of the warning and critical alerts raised by every subsystem. |
sudo nvsm show versions | Show component versionsLists on-board hardware with its firmware plus the installed NVSM, DCGM and OS versions; expect a run of about three minutes. |
sudo nvsm status | Show NVSM service statusTabulates the mqtt, core, api-gateway and notifier services, then gives an overall health and status. |
sudo systemctl status nvsm-core | Check the NVSM core serviceNVSM runs as several systemd units; the nvsm CLI works only when all of them are up. |
sudo systemctl status -all nvsm* | List every NVSM serviceShows the state of each NVSM unit in one systemctl call. |
man nvsm | Open the NVSM man pageInstalled together with NVSM; each verb also has its own page, such as nvsm_show. |
sudo nvsm --help | Print NVSM helpPrints much the same overview as the man page: verbs, global options and examples. |
sudo nvsm show -h | Help for the show verbThe same -h flag works after the other verbs, for example after set, start or stress-test. |
sudo nvsm stress-test [--usage] [--force] [--no-prompt] [TEST...] [DURATION] | Stress-test syntaxLoads GPUs, PCIe, DIMMs, drives, CPUs and network cards together and closes with a pass or fail summary per component. |
sudo nvsm stress-test --usage | Show stress-test help |
sudo nvsm stress-test --force | Run the pre-flight stress testAbout 20 minutes over GPUs, processors, memory and storage; NVIDIA recommends it before production use and after servicing. |
Interactive NVSM session 17
| Command | What it does |
|---|---|
sudo nvsm | Start an interactive sessionOpens the nvsm-> prompt, where verbs are typed without the sudo nvsm prefix and cd moves between targets. |
exit | Leave the interactive sessionTyped at the nvsm-> prompt; returns to the system shell. |
show fans | Run a command at the promptAny NVSM command works here without the sudo nvsm prefix; the shell and prompt forms are interchangeable. |
cd /systems/localhost/storage/alerts | Go to past storage alertsEach earlier alert is a target such as alert0; a bare show lists them. |
show alert1 | Show one alertPrints the message details, component ID, event time and recommended action of that alert. |
cd /systems/localhost/processors/alerts | Go to past processor alerts |
cd /systems/localhost/memory/alerts | Go to past memory alerts |
cd /chassis/localhost/thermal/alerts | Go to past thermal alerts |
cd /chassis/localhost/power/alerts | Go to past power alerts |
cd /systems/localhost/storage/drives | Go to the drive targetsEvery drive is a target named after its device, such as nvme2n1. |
show nvme2n1 | Show one drive |
cd /systems/localhost/memory | Go to the memory targetsDIMMs appear by slot name, next to the alerts and policy targets. |
show CPU2_DIMM_B1 | Show one DIMM |
cd /chassis/localhost/thermal/fans | Go to the fan targetsChassis items live under /chassis/localhost, system items under /systems/localhost. |
show PDB_FAN2 | Show one fan |
cd /chassis/localhost/power | Go to the power supply targets |
show PSU4 | Show one power supplyPresence, health, last power output in watts, model and serial number. |
Health reports and support bundles 13
| Command | What it does |
|---|---|
sudo nvsm dump health | Create a health reportWrites a compressed tar archive under /tmp, named nvsm-health with host name and timestamp, for a support case. |
nvsm dump health -qkd | Create a quick health reportShort for --quick_dump: finishes sooner but uses more CPU and memory; the guide shows it at a root prompt. |
sudo tar xlf nvsm-health-dgx-1-20180907085048.tar.xz | Unpack a health reportThe archive holds sos reports plus outputs such as dmidecode, lspci, lsmod and nvsysinfo_log.txt. |
sudo nvsm set /callhome trigger=true | Send a Call Home report nowOn-demand mode opens a support case with a captured health dump; it needs internet access and a registered system. |
sudo nvsm set /callhome trigger=true diagtest=true | Test Call Home readinessChecks that the system can reach NVIDIA Enterprise Services without opening a ticket. |
sudo nvsm show /callhome | Show the last Call Home resultAfter a diagnostic run it confirms whether the setup is ready for Call Home. |
sudo nvsm set /callhome trigger=false | Cancel a Call Home in progress |
sudo nvsm set /policy callhome_enable=true | Enable automatic Call HomeAutomatic mode sends alerts in batches at set intervals to the NVIDIA Enterprise Support portal. |
sudo nvsm set /policy callhome_enable=false | Disable automatic Call HomeUse it before maintenance such as reseating parts, so the resulting critical alerts do not open support cases. |
sudo nvsm set /policy offline_callhome_enable=true | Enable offline Call HomeFor air-gapped sites: alert and system data stay on the local system for manual hand-over to NVIDIA. |
nvdebug -i BMCIP -u BMCUSER -p BMCPASS -t PLATFORM | Collect debug logs through the BMCOut-of-band collection into a zip under /tmp; -t names the platform type, which is DGX for a DGX system. |
nvdebug -l -t DGX | List DGX log collectorsPrints each collector ID, name and log location, grouped by method such as Redfish. |
nvdebug -i $BMC_IP -u $BMC_USER -p $BMC_PASS -t DGX -g Redfish | Collect only Redfish logsThe -g option limits the run to one log group. |
GPUs 13
| Command | What it does |
|---|---|
sudo nvsm show gpus | List GPUs and their healthInventory and health for each GPU; NVSM takes GPU health from continuous DCGM monitoring. |
cd /systems/localhost/gpus | Go to the GPU targetInteractive session only: later show commands then work relative to the GPU target. |
show -display properties | Show the GPU health rollupIn the gpus target, Status_HealthRollup sums up all GPUs; Critical means at least one has a health incident. |
show -display properties=*health /systems/localhost/gpus/* | List the health of every GPUWildcards narrow the output to the health properties of each GPU target, which singles out the failing one. |
show -level all GPU14/health | Show one GPU's health incidentsDescends into the incident targets under that GPU and prints each message, for example an NVLink link that is down. |
show GPU6 | Show one GPURun inside the gpus target; prints the properties of that single GPU. |
nvidia-smi | Check the GPU driverIts header carries the NVIDIA driver and CUDA versions; the guide uses it to confirm an upgrade. |
nvidia-smi -L | List GPUs with UUIDsOne line per GPU or MIG device with its UUID, the identifier that Docker accepts in device=. |
nvidia-smi topo -m | Show the GPU topology matrixMatrix of GPU and NIC connections with CPU affinity; NV# marks a bonded set of NVLinks. |
nvidia-smi nvlink -s | Show NVLink statusActive links show their bandwidth; a link that is present but down is reported as Inactive. |
dcgmi diag -r 3 | Run the long DCGM diagnosticLevel 3 takes about 15 minutes and adds memory, PCIe/NVLink and targeted stress and power tests. |
dcgmi health -g 1 -c | Check DCGM group healthGives an overall health score for group 1 and the problems found on each affected GPU. |
dcgmi health -g 1 -s mpi | Set DCGM health watchesTurns on the memory, PCIe and InfoROM watches for group 1; the letter a selects every watch. |
Processors, memory, cooling and power 11
| Command | What it does |
|---|---|
sudo nvsm show processors | List CPUsHealth and properties for each installed CPU; show cpus is an alias. |
sudo nvsm show memory | List memory modulesOne entry per installed DIMM with its health; show dimms is an alias. |
sudo nvsm show fans | List fansSpeed in RPM, thresholds and health for each fan. |
sudo nvsm show temperatures | List temperature sensorsReadings from every sensor NVSM knows about; show temps is the short alias. |
sudo nvsm show power | List power suppliesHealth and last output wattage for each power supply unit. |
sudo nvsm show psus | List power supplies (alias)Same output as show power under a PSU-named alias. |
sudo nvsm show powermode | Show the power modeReports the current system power mode, MaxQ or MaxP on a DGX-2. |
sudo nvsm set powermode=maxq | Switch to MaxQ modeDGX-2: efficiency mode that lets two systems share an 18 kW rack; the setting survives reboots. |
sudo nvsm set powermode=maxp | Return to MaxP modeDGX-2 default: GPUs run unconstrained up to their thermal design power. |
sudo nvsm show nvswitches | List NVSwitch interconnects |
ipmitool sdr elist | grep -I fan | List fan sensors from the BMCA reading of exactly 1848 RPM usually means the fan module is missing or unseated. |
Storage and RAID 18
Changing the RAID level erases the data stored on the data array.
| Command | What it does |
|---|---|
sudo nvsm show storage | Show all storage informationCovers the storage target, which groups drives, volumes, alerts and policy. |
sudo nvsm show drives | List storage drives |
sudo nvsm show volumes | List storage volumesIncludes the RAID volumes and the EFI system partition volume with their health. |
sudo nvsm show controllers | List storage controllers |
cd /systems/localhost/storage | Go to the storage targetInteractive session: the starting point for viewing and rebuilding volumes. |
show volumes/md0 | Show the OS RAID volumeA degraded mirror shows one drive and Status_Health = Critical. |
start volumes/rebuild | Rebuild a RAID or ESP volumeAsks for raid-0, raid-1 or esp and, for RAID 1, the replaced drive such as nvme3; it cannot be stopped once started. |
mdadm --detail /dev/md0 | Show RAID array detailsPrints level, state, member devices and the UUID that the NVSM volume mute policies take. |
sudo configure_raid_array.py -m raid5 | Change the data array to RAID 5Erases the data array and rebuilds it with redundancy and less capacity; no unmount is needed. |
sudo configure_raid_array.py -m raid0 | Change the data array to RAID 0Factory level: maximum capacity and no redundancy; changing level erases what the array holds. |
sudo /usr/bin/configure_raid_array.py -c -f | Recreate the RAID 0 data arrayRebuilds the /raid array, for example after nv-disk-encrypt erase removed it. |
lsblk | Confirm the RAID levelThe TYPE column of each data drive names the level of the array. |
sudo systemctl status nvidia-raid-config | Check first-boot RAID setupWait for the Finished message, typically about 10 minutes after first boot, before rebooting. |
sudo nvme list | List NVMe drivesNode, serial number, model and firmware revision of every NVMe device. |
sudo systemctl stop cachefilesd | Stop NFS cachingcachefilesd caches NFS mounts on the data drives; the next row keeps it from starting again. |
sudo systemctl disable cachefilesd | Disable NFS caching permanently |
cat /proc/fs/nfsfs/volumes | Verify NFS cachingFSC=yes in the output means the mount is cached on the data drives. |
nvme format -s1 DEVICE-PATH | Secure-erase an NVMe driveB200 data-deletion step, run from the installer rescue shell on a node such as /dev/nvme0n1; it destroys the data. |
Drive encryption 11
| Command | What it does |
|---|---|
sudo apt install -y nv-disk-encrypt | Install the SED management toolManages self-encrypting data drives only, never OS drives; reboot after installing. |
sudo nv-disk-encrypt info | Show which drives can be encryptedPer drive: SED capable, boot disk, locked, lock enabled; only non-boot SEDs can be managed. |
sudo nv-disk-encrypt info -j | Show drive status as JSONEligible drives read sed_capable true and used_for_boot false. |
sudo nv-disk-encrypt init -g | Initialise, typing each passwordPrompts for a vault password and then one per eligible drive; -g adds random salt values. |
sudo nv-disk-encrypt init -k YOUR-VAULT-PASSWORD -g -r | Initialise with random passwordsNothing is prompted: -k passes the vault password and -r generates a password for each drive. |
sudo nv-disk-encrypt lock | Enable drive lockingRun after init: drives lock when they lose power and unlock by themselves once the system boots again. |
sudo nv-disk-encrypt disable | Disable drive lockingTurns locking off on all drives; the first step before changing passwords or adding or replacing disks. |
sudo nv-disk-encrypt export -k yourvaultpassword | Export the drive keysWrites the vault contents to /tmp/secrets.out; back the keys up, because lost keys force a data-destroying factory reset. |
sudo umount /raid | Unmount the data arrayPart of stopping the RAID before an erase, after cachefilesd has been stopped. |
sudo mdadm --stop /dev/md1 | Stop the data RAID arrayLast step before the erase; md1 is the data array in the guide. |
sudo nv-disk-encrypt erase | Erase the encrypted drivesDestroys all data: unlocks the drives, disables locking and removes the RAID 0 array. |
Network adapters and InfiniBand 18
| Command | What it does |
|---|---|
sudo nvsm show networkadapters | List physical network adaptersThe adapter IDs it prints, such as PCI0000_0c_00, are what the port and policy targets use. |
sudo nvsm show networkinterfaces | List logical network interfaces |
sudo nvsm show networkports | List network ports |
sudo nvsm show networkdevicefunctions | List adapter PCIe functions |
sudo nvsm show /chassis/localhost/NetworkAdapters | List adapter IDsPrints the adapter targets by PCI address, the IDs used when muting notifications. |
sudo nvsm muteNIC | Show link-down alert mutingWith no arguments it lists each NIC, its PCI interface and whether its NV-NET-01 alert is muted. |
sudo nvsm muteNIC ibp220s0=enable,ibp41s0f1=enable | Mute link-down alertsFor unused adapters: comma-separated NIC=enable pairs stop their NV-NET-01 alerts. |
sudo nvsm muteNIC ibp220s0=disable,ibp41s0f1=disable | Unmute link-down alertsRestores alerting; the alert database must also be cleared before the alerts show again. |
sudo nvsm_database_rotate.sh -d alert | Clear the NVSM alert databaseStops NVSM, empties the stored alerts and starts it again. |
sudo nvsm show /chassis/localhost/NetworkAdapters/policy | Show muted adaptersThe mute_monitoring and mute_notification properties read NOT_SET when nothing is muted. |
sudo nvsm set chassis/localhost/NetworkAdapters/policy mute_notification=PCI0000_0c_00,PCI0000_12_00 | Mute adapter notificationsTakes a comma-separated list of adapter IDs. |
sudo mlxconfig -e -d aa:00.0 q | grep -e 'Device\|Configurations\|LINK_TYPE' | Query a port's link typeIB(1) is InfiniBand and ETH(2) Ethernet, shown for the default, current and next-boot configuration. |
sudo mlxconfig -y -d aa:00.0 set LINK_TYPE_P1=2 | Set a ConnectX port to EthernetThe value 2 selects Ethernet; power the system off and on, and reconfigure the switch side as well. |
mlxconfig -d DEVICE set NUM_OF_PLANES_P1=0 | Set ConnectX-8 planes to zeroA step in splitting one 800 Gbps InfiniBand port into two 400 Gb Ethernet ports. |
mlxconfig -d DEVICE set NUM_OF_PF=2 | Expose two physical functionsFinal split step: the OS then sees two 400 GbE interfaces after a power cycle. |
mlxconfig -d DEVICE reset | Reset a ConnectX-8 deviceReturns the port to a single 800 Gbps InfiniBand link and reverts every other setting on the device. |
sudo flint -d 05:00.2 q full | grep "FW Version" | Query ConnectX firmware versionDGX B200 example; 28.47.1026 or later resolves the ACCESS_REG errors in the kernel log. |
sudo netplan apply | Apply the network configurationAfter editing /etc/netplan/01-netcfg.yaml for a static address; an SSH session drops here, so use the BMC console. |
Alerts and monitoring policies 19
| Command | What it does |
|---|---|
sudo nvsm show policy | Show alert policiesPolicy settings for the subsystems, such as poll intervals and muted devices. |
sudo nvsm show /policy | Show the global policyThe /policy target holds the email settings used for alert notification. |
sudo nvsm set /policy [email protected] | Set the alert sender addressIt has to be a valid email address, or no alert mail goes out. |
sudo nvsm set /policy email_smtp_server_name=smtpserver.nvidia.com | Set the SMTP relay server |
sudo nvsm set /policy [email protected],[email protected] | Set the alert recipientsSeveral addresses are separated by commas; each new alert is mailed to all of them. |
sudo nvsm set /policy email_smtp_server_port=465 | Set the SMTP port |
sudo nvsm create testalert | Create a test alertRaises a dummy Warning alert on component Test0, which exercises the email or Call Home notification path. |
sudo nvsm create testalert Email1 Critical | Create a custom test alertThe two arguments are a component ID and a severity. |
sudo nvsm clear testalert | Clear test alertsDismisses every test alert on component Test0; append a component ID to clear a different one. |
sudo nvsm show testalerts | List test alerts |
sudo nvsm show /systems/localhost/gpus/policy | Show the GPU policyIts mute_notification and mute_monitoring lists suppress alerts or monitoring for chosen GPUs. |
sudo nvsm show /systems/localhost/memory/policy | Show the memory policyCovers monitoring of DIMM correctable and uncorrectable errors. |
sudo nvsm show /systems/localhost/processors/policy | Show the processor policyCovers monitoring of CPU machine-check exceptions. |
sudo nvsm show /systems/localhost/storage/policy | Show the storage policyPoll intervals and mute lists for controllers, drives and volumes. |
sudo nvsm show /systems/localhost/pcie/policy | Show the PCIe policy |
sudo nvsm show /chassis/localhost/thermal/policy | Show the thermal policyHolds separate mute lists for fans and for PDB sensors. |
sudo nvsm show /chassis/localhost/power/policy | Show the power policyIts mute lists exclude chosen power supplies from monitoring or from notification. |
nvsm set /systems/localhost/storage/policy volume_mute_monitoring=<UUID> | Stop monitoring a volumePut the array UUID that mdadm --detail reports in place of <UUID>; the guide shows the command at a root prompt. |
nvsm set /systems/localhost/storage/policy volume_mute_notification=<UUID> | Silence alerts for a volumeSuppresses alert notification for the array with that UUID, which mdadm --detail reports. |
BMC and remote console 13
| Command | What it does |
|---|---|
sudo nvsm set -bmccred | Store BMC credentials for NVSMDGX H100: prompts for a BMC user and password and keeps them encrypted on the host for Redfish queries. |
sudo ipmitool lan print 1 | Show BMC network settingsRun it first to see the current settings before switching to a static address. |
sudo ipmitool lan set 1 ipsrc static | Use a static BMC addressFirst step; the address, netmask and gateway follow. |
sudo ipmitool lan set 1 ipaddr MY-IP-ADDRESS | Set the BMC IP addressThe BIOS setup calls this value the Station IP address. |
sudo ipmitool lan set 1 netmask MY-NETMASK-ADDRESS | Set the BMC subnet mask |
sudo ipmitool lan set 1 defgw ipaddr MY-DEFAULT-GATEWAY-IP-ADDRESS | Set the BMC default gatewayThe BIOS setup calls this value the Router IP address. |
ipmitool -I lanplus -H BMC-IP -U USERNAME -P PASSWORD sol activate | Open the serial-over-LAN consoleKeeps console output visible during a vmcore crash dump, when the BMC screen goes blank. |
sudo ipmitool mc reset cold | Reboot the BMCActivates a BMC firmware update; allow a couple of minutes for the BMC to return. |
ipmitool mc info | Check that the BMC respondsAnswers again once the BMC has finished restarting. |
sudo ipmitool chassis power cycle | Cold-reset the systemActivates BIOS and GPU-tray firmware; PCIe switch, retimer, EROT and CPLD updates need an AC power cycle. |
ipmitool raw 0x3c 0x80 0x05 | Read the system power limitDGX B200: two bytes come back, and a value above 12,000 means power capping is off. |
ipmitool raw 0x3c 0x81 0x05 0x70 0x17 | Cap system power at 6 kWDGX B200 example that sets a limit below the budget the power supplies can carry. |
sudo ifconfig enx9638a3b292ec 169.254.0.18 netmask 255.255.0.0 | Reach the BMC from the hostenx9638a3b292ec is the guide's example interface name; the host joins the internal network where the BMC answers Redfish at 169.254.0.17. |
Firmware 11
Stop all GPU activity, nvidia-smi included, before a firmware update; it can block the VBIOS update.
| Command | What it does |
|---|---|
nvfwupd -t ip=<BMC-IP-ADDRESS> user=<BMC-USERNAME> password=<BMC-PASSWORD> show_version -p MB-TRAY-PACKAGE GPU-TRAY-LATEST-PACKAGE | Compare installed and package firmwareCompares running versions with those in the packages; run without package arguments it shows the installed versions only. |
nvfwupd -t ip=<BMC-IP-ADDRESS> user=<BMC-USERNAME> password=<BMC-PASSWORD> update_fw -p MB-TRAY-PACKAGE -y -s update_bmc.json | Update the BMC firmwareThe JSON file names the Redfish target, here HostBMC_0; -y skips the confirmation prompt. |
nvfwupd -t ip=<BMC-IP-ADDRESS> user=<BMC-USER-ID> password=<BMC-PASSWORD> update_fw -p FIRMWARE-PACKAGE-FILE -y -s updparameters.json -b | Update firmware in the backgroundWith -b the command prints a task ID and exits instead of following the update to the end. |
nvfwupd -t ip=<BMC-IP-ADDRESS> user=<BMC-USER-ID> password=<BMC-PASSWORD> show_update_progress -i 1 | Show update task progressPrints the task state and percent complete for the given task ID. |
nvfwupd -c config.yaml show_version | Compare versions from a config fileThe YAML file carries the target address, credentials and packages, so none appear on the command line. |
nvfwupd -c config.yaml update_fw | Update from a config fileUpdates the motherboard tray as described in the YAML file. |
sudo mlxfwmanager -u -D . | Update NIC firmware from a folderRun in the network directory of the package; confirms once for all ConnectX-7 adapters and BlueField-3 DPUs. |
sudo nvme fw-commit $DeviceName --action=3 | Activate downloaded NVMe firmwareAction 3 commits the image and activates it at once, with no reset. |
sudo nvme id-ctrl $DeviceName | Verify the NVMe firmware version |
sudo nvsm show firmware | Show firmware with an update containerInteractive walk-through: pick a firmware update container that is already loaded and it reports the installed versions. |
sudo nvsm update firmware | Update firmware with a containerSame guided flow as show firmware, but it runs the container to apply the update. |
OS updates and release upgrade 17
| Command | What it does |
|---|---|
more /etc/dgx-release | Show DGX OS release informationGives the ISO build version and every over-the-network update with its date. |
sudo apt update | Refresh the package listsAlways the first step; a GPG error here points to the DOCA repository key issue. |
sudo apt full-upgrade -s | Preview a package upgradeA simulation that shows which packages would change. |
sudo apt full-upgrade | Upgrade all packagesInstalls every available update, kernel included; keep the current GRUB configuration when asked. |
sudo reboot | Reboot after an upgradeNeeded for driver upgrades: until then nvidia-smi fails with a driver/library version mismatch. |
sudo apt install -y doca-repo | Add the DOCA repositoryNeeded once on a DGX OS 7.0.0 install, whose DOCA repository location changed; follow with apt update. |
sudo apt install -y nvidia-release-upgrade | Install the release upgraderFirst bring DGX OS 6 fully up to date; only the latest 6.x can move to 7. |
sudo nvidia-release-upgrade | Upgrade DGX OS 6 to 7MOFED drivers are replaced by DOCA-OFED along the way; reboot whenever the upgrade asks. |
sudo -E nvidia-release-upgrade | Release-upgrade behind a proxyThe -E option keeps the proxy environment variables. |
dpkg -a --configure | Reconfigure after a failed upgradeFirst recovery step once power or network is back and the system reaches a command line. |
apt -f install -y | Fix broken package installsSecond recovery step after an interrupted release upgrade. |
sudo pro attach XXXXX | Enable Extended Security MaintenanceXXXXX stands for the Ubuntu Pro token that NVIDIA Enterprise Support supplies to DGX customers. |
sudo pro status | Check the Ubuntu Pro subscriptionAlso shows the expiry date after a token refresh. |
sudo pro refresh | Renew an expired Pro tokenThe client does not renew by itself when the token expires. |
apt list --installed nvidia-driver*-open | Check the installed driver branchAn OS upgrade stays on the same GPU driver branch, so check before moving to a newer one. |
apt list nvidia-driver-*-open | List available driver branchesRun after apt update; shows the open GPU kernel module branches in the repository. |
apt list --installed cuda-toolkit-* | Check the installed CUDA Toolkit |
Containers 9
| Command | What it does |
|---|---|
docker run --gpus all --rm -it ubuntu nvidia-smi -L | Test GPU access in a containerLists only the compute GPUs; on DGX Station A100 the display GPU is hidden from Docker. |
docker run --gpus 2 --rm -it ubuntu nvidia-smi -L | Give a container two GPUsThe number after --gpus is how many GPUs the container is assigned. |
docker run --gpus '"device=1,2"' ... | Run on specific GPUsThe device list takes GPU indexes or UUIDs, quoted as shown. |
sudo docker --version | Check the Docker versionPart of the quick health check after setup. |
sudo usermod -aG docker user-login-id | Let a user run DockerAdds an existing login to the docker group; NVIDIA warns that group members can escalate to root. |
sudo systemctl daemon-reload | Reload systemd unit filesNeeded after editing docker-override.conf, before Docker is restarted. |
sudo systemctl restart docker | Restart DockerApplies a changed bridge address or other daemon override. |
sudo systemctl restart nv-docker-gpus | Refresh the Docker GPU allow listDGX Station A100: run it with a Docker restart whenever MIG instances are created, destroyed or changed. |
wget https://nvcr.io/v2 | Test access to the NGC registryA working path connects and then answers 401 Unauthorized. |
Crash dumps and kernel settings 8
| Command | What it does |
|---|---|
/usr/sbin/nvidia-kdump-config enable-dmesg-dump | Enable dmesg crash dumpsCaptures the kernel log only and reserves memory for the crash kernel. |
/usr/sbin/nvidia-kdump-config enable-vmcore-dump | Enable dmesg and vmcore dumpsSaves the vmcore as well as the kernel log, and likewise reserves crash-kernel memory. |
/usr/sbin/nvidia-kdump-config disable | Disable crash dumpsTurns kdump off, so no memory is held back for the crash kernel. |
cat /sys/devices/system/cpu/vulnerabilities/* | Check CPU mitigation stateLines that start with Mitigation: mean the mitigations are on; Vulnerable lines mean they are off. |
sudo apt install nv-mitigations-off -y | Disable CPU mitigationsTrades the side-channel mitigations for workload performance; it applies after a reboot. |
sudo apt purge nv-mitigations-off | Re-enable CPU mitigations |
sudo update-grub | Regenerate the GRUB configurationApplies an edit to /etc/default/grub, such as init_on_alloc=0, at the next reboot. |
cat /proc/cmdline | Verify kernel parametersAfter the reboot the added parameter should be listed here. |
No row on this page matches that filter.
Notes
→ Related guide: GPU cluster management: the full command guide
NVSM first
sudo nvsm show health runs the NVSM health checks and ends with an overall status, and is the first command after any alert; sudo nvsm show alerts lists what is currently raised. sudo nvsm show gpus, sudo nvsm show nvswitches and sudo nvsm show memory drill into the accelerators, the NVSwitch interconnects and the DIMMs; sudo nvsm show psus and sudo nvsm show fans cover power and cooling. sudo nvsm dump health packs logs and system state into an archive under /tmp for NVIDIA Enterprise Support — run it before opening a case. NVSM talks to a privileged API server, so every command needs sudo, and the CLI answers only while all NVSM services are running: sudo nvsm status shows which one is not.
GPUs and interconnect
nvidia-smi topo -m prints the matrix of connections between GPUs and NICs with their CPU affinity; NV# marks a bonded set of NVLinks and the other codes describe the PCIe path. nvidia-smi nvlink -s shows each NVLink: active links with their bandwidth, the rest as Inactive or Sleep. NVSM reports GPU faults as health incidents fed by DCGM: in an interactive session show -display properties in the gpus target gives the rollup, and show -level all GPU14/health lists the incidents of one GPU. dcgmi diag -r 3 runs the long DCGM diagnostic, which takes about 15 minutes, and dcgmi health -g 1 -c reads the health watches of a GPU group once dcgmi health -g 1 -s mpi has set them.
Storage and encrypted drives
The data drives ship as a RAID 0 array meant for caching, not for long-term data. sudo configure_raid_array.py -m raid5 trades capacity for redundancy and, like the change back with sudo configure_raid_array.py -m raid0, erases what the array holds. After a failed OS drive is replaced the mirror normally rebuilds by itself at power-on; when it does not, start volumes/rebuild in the storage target of an interactive session starts it. Self-encrypting data drives are set up with sudo nv-disk-encrypt init -g followed by sudo nv-disk-encrypt lock. Export the keys with sudo nv-disk-encrypt export -k yourvaultpassword and keep them somewhere safe, because lost keys leave only a factory reset that destroys the data.
Network ports
sudo nvsm show networkadapters lists the adapters NVSM monitors, and sudo nvsm muteNIC shows which ports have their link-down alert muted, which suits ports that are left unused on purpose. ConnectX ports come configured for InfiniBand; the mlxconfig rows switch a port to Ethernet, and the change applies after the system is powered off and on. The DGX guides do not print fabric diagnostics such as ibstat or a kernel-log search for XID errors, so those commands are not on this page.
Updates and firmware
more /etc/dgx-release shows the installed DGX OS version and each over-the-network update. Routine updates are sudo apt update, a look at sudo apt full-upgrade -s, then sudo apt full-upgrade and a reboot; the move from DGX OS 6 to 7 runs through sudo nvidia-release-upgrade from the latest 6.x release. Firmware on a DGX B200 is updated with nvfwupd against the BMC, and an update takes effect only after activation: sudo ipmitool mc reset cold for the BMC, sudo ipmitool chassis power cycle for the BIOS and the GPU tray, and an AC power cycle for PCIe switches, retimers, EROT and CPLDs.
Questions
Which command should I run first when a DGX system raises an alert?
sudo nvsm show health. It runs the NVSM checks and closes with an overall status line; sudo nvsm show alerts then lists the warning and critical alerts for every subsystem.
How do I collect logs for NVIDIA Enterprise Support?
sudo nvsm dump health writes a compressed archive under /tmp that can be attached to a support case. nvsm dump health -qkd is the quick variant, at the cost of more CPU and memory while it runs. For out-of-band logs through the BMC of a DGX B200, the firmware guide uses nvdebug, for example nvdebug -l -t DGX to list the collectors.
Why does an nvsm command fail or hang?
NVSM needs root, so the guides prefix every command with sudo, and the CLI works only when all NVSM services are up. sudo nvsm status shows each service and an overall status; sudo systemctl status nvsm-core gives the systemd detail for the core service.
How do I upgrade a DGX system from DGX OS 6 to DGX OS 7?
Bring DGX OS 6 up to date with sudo apt update and sudo apt full-upgrade, install the upgrader with sudo apt install -y nvidia-release-upgrade, then run sudo nvidia-release-upgrade, or sudo -E nvidia-release-upgrade behind a proxy. Reboot when prompted; the Mellanox OFED drivers are replaced by DOCA-OFED during the upgrade.
Which DGX OS and NVSM versions does this page follow?
The DGX OS 7 User Guide (edition of 30 September 2026, covering releases up to DGX OS 7.6.0), the NVSM User Guide release 25.03, and the DGX B200 user guide and firmware update guide, whose nvfwupd help text is version 2.1.1. Five GPU rows come from the DCGM and nvidia-smi documentation. Power-mode, power-capping and firmware rows are tied to the system named in their description.
Sources and checks
- NVIDIA System Management (NVSM) User Guide, release 25.03 — Using the NVSM CLI nvsm show, dump, set, stress-test, alerts, policies and RAID rebuild
- NVSM User Guide, release 25.03 — Introduction Verifying the NVSM services
- NVSM User Guide, release 25.03 — NVSM Call Home Call Home modes and commands
- NVIDIA DGX OS 7 User Guide (PDF edition of 30 Sep 2026, releases up to DGX OS 7.6.0) Package and release upgrades, RAID, drive encryption, ConnectX ports, Docker
- DGX OS 7 User Guide — System Configurations Crash dumps, CPU mitigations, NFS caching, registry check, port link type
- DGX OS 7 User Guide — Managing Self-Encrypting Drives Stopping and recreating the data RAID around an erase
5 more source pages
- DGX OS 7 User Guide — Upgrading the OS Release information file
- NVIDIA DGX B200 Firmware Update Guide, version 26.07.4 (PDF of 13 Sep 2026, nvfwupd 2.1.1) nvfwupd, nvdebug, firmware activation, NVMe and NIC firmware
- NVIDIA DGX B200 User Guide (PDF of 8 Sep 2026) BMC network settings, power capping, pre-flight test, secure erase
- NVIDIA DCGM User Guide — Feature Overview dcgmi health and diagnostic examples
- NVIDIA System Management Interface (nvidia-smi) documentation topo and nvlink options
Every row on this page was matched to a line in the documentation above on . A binding that could not be found there is not listed.
4 rows from the earlier version of this page were removed in that check: duplicates, and bindings the documentation does not list or lists differently.
Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.