torchrun 67
torchrun is PyTorch's launcher for distributed training and an entry point into python -m torch.distributed.run. Its table moves from single-node and multi-node launch lines through the rendezvous, restart, worker start-up and logging options to the environment variables each worker receives and the ones you can set.
Single-node launches 7
Every multi-word option is also registered with underscores, so --nproc_per_node and --rdzv_backend, as many tutorials write them, behave the same.
| Command | What it does |
|---|---|
torchrun --standalone --nnodes=1 --nproc-per-node=$NUM_TRAINERS YOUR_TRAINING_SCRIPT.py | Single-node multi-worker jobThe reference form for one machine: torchrun hosts its own rendezvous, so no address, port or job id is needed. |
torchrun --rdzv-backend=c10d --rdzv-endpoint=localhost:0 --nnodes=1 --nproc-per-node=$NUM_TRAINERS YOUR_TRAINING_SCRIPT.py | Several jobs on one hostPort 0 lets each job take its own free port, so parallel runs on the machine neither clash nor merge. |
--standalone | Standalone modeStarts a c10d store on a free local port and fills in backend, endpoint and id itself, overriding values you pass. |
--nproc-per-node NPROC_PER_NODE | Processes per nodeWorkers started on this node, 1 by default; for GPU training keep it at or below the number of GPUs. |
--nproc-per-node gpu | One worker per GPUTakes the CUDA device count of the node and stops with an error when CUDA is not available. |
--nproc-per-node auto | Worker count from the hardwareUses the GPU count when CUDA is present, else the XPU count, and otherwise one worker per CPU. |
torchrun --numa-binding=node --nproc-per-node=8 train.py | Eight workers with NUMA bindingPins each of the eight workers to CPUs close to its assigned GPU, which helps on NUMA multi-GPU machines. |
Multi-node launches 8
| Command | What it does |
|---|---|
torchrun --nnodes=$NUM_NODES --nproc-per-node=$NUM_TRAINERS --max-restarts=3 --rdzv-id=$JOB_ID --rdzv-backend=c10d --rdzv-endpoint=$HOST_NODE_ADDR YOUR_TRAINING_SCRIPT.py | Fault-tolerant multi-node jobRun the identical line on every node; the fixed-size group survives three failures by restarting all workers. |
torchrun --nnodes=1:4 --nproc-per-node=$NUM_TRAINERS --max-restarts=3 --rdzv-id=$JOB_ID --rdzv-backend=c10d --rdzv-endpoint=$HOST_NODE_ADDR YOUR_TRAINING_SCRIPT.py | Elastic job of one to four nodesTrains with anywhere from one to four nodes and restarts the group on up to three joins, departures or crashes. |
torchrun --nnodes=2 --nproc_per_node=8 --rdzv_id=100 --rdzv_backend=c10d --rdzv_endpoint=$MASTER_ADDR:29400 elastic_ddp.py | Two nodes with eight GPUs eachThe DDP tutorial runs this on both hosts with the same MASTER_ADDR, giving 16 workers in total. |
--nnodes NNODES | Node countEither a fixed number or a MIN:MAX range for elastic jobs; the default 1:1 means a single node. |
--nnodes=1:4 | Elastic node rangeLets the job start with a single node and admit up to four, restarting the workers whenever membership changes. |
--node-rank NODE_RANK | Node rankDifferent on each machine, 0 for the first; only the static rendezvous reads it, other backends assign ranks themselves. |
export MASTER_ADDR=$(scontrol show hostname ${SLURM_NODELIST} | head -n 1) | First Slurm node as endpoint hostPlaced in the batch script so every node points its rendezvous endpoint at the first host of the allocation. |
srun --nodes=2 ./torchrun_script.sh | Start the wrapper on two nodesSlurm runs the script that holds the torchrun line once on each of the two allocated nodes. |
Rendezvous 8
Rendezvous timeouts are backend settings for --rdzv-conf: join_timeout defaults to 600 seconds and last_call_timeout to 30 seconds.
| Command | What it does |
|---|---|
--rdzv-backend RDZV_BACKEND | Rendezvous backendDefaults to static; c10d is recommended, while etcd-v2 and the legacy etcd need an etcd server with its v2 API on. |
--rdzv-backend=c10d | Use the c10d rendezvousThe recommended backend: a TCP store hosted by torchrun on the endpoint node, with no separate server to start. |
--rdzv-endpoint RDZV_ENDPOINT | Rendezvous endpointUsually HOST:PORT of a well-connected node in the cluster; for c10d a missing port means 29400. |
--rdzv-id RDZV_ID | Rendezvous IDA user-chosen job id that every node must share; the log directory also uses it as the prefix of the job folder. |
--rdzv-conf RDZV_CONF | Extra rendezvous settingsComma-separated KEY=VALUE pairs handed to the backend, such as the timeouts of the c10d rendezvous. |
--master-addr MASTER_ADDR | Master addressHost of rank 0 for the static rendezvous only; 127.0.0.1 by default, which is enough on one machine. |
--master-port MASTER_PORT | Master portPort on the rank 0 host for the static rendezvous; torchrun falls back to 29500 when it is not given. |
--local-addr LOCAL_ADDR | Address of this nodeOverrides the address this node uses for connections; otherwise torchrun looks it up, falling back to the FQDN. |
Restarts and shutdown 4
| Command | What it does |
|---|---|
--max-restarts MAX_RESTARTS | Max restartsDefaults to 0; each failure or membership change stops all workers and restarts the whole group, using up one attempt. |
--monitor-interval MONITOR_INTERVAL | Worker check intervalSeconds between the agent's looks at worker state; the default is 0.1. |
--shutdown-timeout SHUTDOWN_TIMEOUT | Grace period before SIGKILLWithout it torchrun reads TORCH_ELASTIC_SHUTDOWN_TIMEOUT, and failing that waits 30 seconds. |
--signals-to-handle SIGNALS | Signals forwarded to workersComma-separated; defaults to SIGTERM,SIGINT,SIGHUP,SIGQUIT, and Slurm sites often add SIGUSR1 or SIGUSR2. |
Worker start-up 7
--numa-binding accepts node, socket, exclusive or core-complex; the NUMA Binding page advises node when in doubt.
| Command | What it does |
|---|---|
--start-method START_METHOD | Worker start methodHow worker processes are created: spawn, the default, or fork or forkserver. |
--role ROLE | Worker role nameDefaults to default; it sets ROLE_RANK and ROLE_WORLD_SIZE and opens the default console prefix, as in [default3]. |
--module | Launch a Python moduleShort form -m; give a module name instead of a script file, and each worker runs it as python -m would. |
--no-python | Run a non-Python programStarts the target directly instead of through the Python interpreter, for shell scripts or binaries. |
--run-path | Run script via runpyWorkers execute the script with runpy rather than a new python command; it needs an absolute path and wins over --no-python. |
--numa-binding NUMA_BINDING | Bind workers to nearby CPUsKeeps each worker on CPUs close to the GPU it is assigned; the available modes are defined in torch.numa.binding. |
--virtual-local-rank | Virtual local rank modeEvery worker then sees LOCAL_RANK 0 and its own GPU as device 0, through an adjusted CUDA_VISIBLE_DEVICES. |
Logs and console output 11
| Command | What it does |
|---|---|
--log-dir LOG_DIR | Log directoryBase folder for the files written by --redirects or --tee; each job gets its own sub-folder prefixed with the rendezvous id. |
--redirects 3 | Send all output to filesMoves stdout and stderr of every worker off the console into files; 1 takes only stdout and 2 only stderr. |
-r 0:1,1:2 | Redirect per local rankShort form of --redirects with a mapping: here local rank 0 sends stdout and local rank 1 stderr to files. |
--tee 3 | Copy output to files and consoleKeeps both streams of every worker on screen while also saving them; each console line gets a role and local rank prefix. |
--log-line-prefix-template '${hostname}:${rank}: ' | Prefix lines with host and rankApplies to tee'd console lines; adding ${hostname} shows which node a message came from in multi-node jobs. |
torchrun --nnodes 2 --nproc-per-node 8 --tee 3 --log-line-prefix-template "${hostname}:${rank}: " train.py | Two nodes with host-tagged outputEach console line starts with the node name and global rank, so a crash message points to the failing host. |
--local-ranks-filter 0,1,2 | Show only some ranks on screenHides console output of the other local ranks; files written by --redirects or --tee still hold everything. |
--duplicate-stdout-filters FILTERS | Copy matching stdout linesLines that contain any of the comma-separated terms also go to an extra file; a double comma escapes a comma. |
--duplicate-stderr-filters FILTERS | Copy matching stderr linesSame filter syntax as the stdout variant, applied to the error stream of the workers. |
--logs-specs LOGS_SPECS | Custom log handling plug-inNames an entry point in the torchrun.logs_specs group whose LogsSpecs class replaces the built-in log handling. |
--event-log-handler EVENT_LOG_HANDLER | Event logging handlerPicks a registered handler for the launcher's elastic events; when not set, the value is null. |
Launcher setup 4
| Command | What it does |
|---|---|
python -m torch.distributed.run | Launch through the moduleEquivalent to torchrun, whose console script is only an entry point into this module. |
pip install shtab | Install the completion helperNeeded only for --print-completion; torchrun loads the package solely when that option is used. |
torchrun --print-completion zsh > ~/.zsh/completions/_torchrun | Write zsh completion scriptThe script is built from the option parser itself, so completions match the installed torchrun; tcsh works too. |
torchrun --print-completion bash > ~/.local/share/bash-completion/completions/torchrun | Write bash completion scriptSaves to the usual per-user bash-completion folder; check your shell documentation if it loads completions elsewhere. |
Variables in each worker 13
| Command | What it does |
|---|---|
LOCAL_RANK | Rank on this nodeIndex of the worker among those on its node; use it to choose the GPU, as in device_ids=[local_rank]. |
RANK | Global rankUnique across the whole job but not stable: after a restart a worker can get a different number. |
GROUP_RANK | Worker group rankEquals the node's rank when each node runs one group, and ranges from 0 to the maximum node count. |
ROLE_RANK | Rank within the roleCounts only the workers that were started with the same --role value as this one. |
LOCAL_WORLD_SIZE | Workers on this nodeEquals the --nproc-per-node value, which torchrun expects to be the same on every node. |
WORLD_SIZE | Total workers in the jobCan change between restarts of an elastic job, so read it at start-up instead of hard-coding it. |
ROLE_WORLD_SIZE | Workers sharing the roleCounts every worker in the job that was launched with this worker's role. |
MASTER_ADDR | Host of rank 0Fully qualified name of the machine running rank 0, read when the process group is initialised. |
MASTER_PORT | Store port on rank 0 hostPort on MASTER_ADDR where the C10d TCP store for the workers can be hosted. |
TORCHELASTIC_RESTART_COUNT | Restarts so farGrows by one each time the agent restarts the worker group after a failure or membership change. |
TORCHELASTIC_MAX_RESTARTS | Restart limitHolds the --max-restarts value that the job was launched with. |
TORCHELASTIC_RUN_ID | Run id of the jobMatches the --rdzv-id value, or the random id torchrun generates under --standalone. |
OMP_NUM_THREADS | OpenMP threads per workerIf it is unset and a node runs more than one worker, torchrun sets it to 1 and prints a warning. |
Variables you set 5
Options that take a value can also be preset with a PET_ variable named after the option, such as PET_NPROC_PER_NODE; a value on the command line wins.
| Command | What it does |
|---|---|
PYTHON_EXEC | Python interpreter overrideWorkers start the training script with this executable instead of the interpreter running torchrun. |
TORCHELASTIC_LOG_LINE_PREFIX_TEMPLATE | Default console line prefixSets the tee prefix template without the option; --log-line-prefix-template wins when both are given. |
TORCH_ELASTIC_SHUTDOWN_TIMEOUT | Default shutdown grace periodSeconds to wait before SIGKILL when --shutdown-timeout is not given; 30 if neither is set. |
export NCCL_DEBUG=INFO | Verbose NCCL loggingMakes NCCL print detailed logs, the multinode tutorial's first suggestion when nodes fail to communicate. |
export NCCL_SOCKET_IFNAME=eth0 | Choose the NCCL network interfaceSometimes needed so the distributed backend uses the right network card; eth0 is the tutorial's example name. |
One node, many GPUs
torchrun --standalone --nnodes=1 --nproc-per-node=$NUM_TRAINERS YOUR_TRAINING_SCRIPT.py is the reference's single-machine form: --standalone starts a c10d rendezvous on a free local port, so no address, port or job id is needed. --nproc-per-node gpu starts one worker per GPU. Each worker finds RANK, LOCAL_RANK and WORLD_SIZE in its environment, so the script calls dist.init_process_group() with only a backend (NCCL is the recommended one for GPUs) and picks its device from LOCAL_RANK. Two jobs on the same host need separate rendezvous ports; a c10d endpoint of localhost:0 lets torchrun choose a free one.
Multiple nodes
Every node runs the same torchrun command. With the recommended --rdzv-backend=c10d, --rdzv-endpoint RDZV_ENDPOINT names HOST:PORT of one node (port 29400 when omitted) and --rdzv-id RDZV_ID is a job id shared by all nodes; ranks are assigned at the rendezvous. The older static form instead gives each node --node-rank NODE_RANK plus --master-addr MASTER_ADDR and --master-port MASTER_PORT (29500 by default). A range such as --nnodes=1:4 makes the job elastic. Under Slurm, the DDP tutorial puts the command in a script, sets MASTER_ADDR to the first host of SLURM_NODELIST and starts it with srun --nodes=2 ./torchrun_script.sh.
Failures and logs
--max-restarts MAX_RESTARTS defaults to 0. When a worker fails, torchrun stops all workers and restarts the whole group while attempts remain; ranks can change between attempts, so the script should reload its latest checkpoint at start-up. Decorating the script's main function with @record from torch.distributed.elastic.multiprocessing.errors adds the traceback to the error summary. Worker output goes to the console by default: --tee 3 also saves both streams under --log-dir LOG_DIR, --redirects 3 saves them instead, and --local-ranks-filter 0,1,2 limits what reaches the screen. For NCCL connection problems the multinode tutorial suggests export NCCL_DEBUG=INFO.
Should I write --nproc-per-node or --nproc_per_node?
Either works. Every multi-word torchrun option is registered under both spellings in torch/distributed/run.py; the PyTorch 2.14 reference writes dashes, while many tutorials still use underscores.
How does a training script know which GPU to use under torchrun?
It reads LOCAL_RANK from the environment and uses it as the device index, for example device_ids=[local_rank] for DistributedDataParallel. The train-script guide says to take the local rank from this variable rather than parse a --local-rank argument.
Which ports does torchrun use by default?
The c10d rendezvous uses port 29400 on the endpoint host when --rdzv-endpoint RDZV_ENDPOINT gives no port; the static rendezvous uses --master-port MASTER_PORT, 29500 unless set; and --standalone picks a free local port.
What happens when one torchrun worker crashes?
All workers are stopped. If attempts remain under --max-restarts MAX_RESTARTS (0 by default), the whole worker group starts again, possibly with different ranks; otherwise the job fails. Work since the last checkpoint is lost.
How do I run two torchrun jobs on the same machine?
Give each its own c10d rendezvous; port 0 lets torchrun pick a free one: torchrun --rdzv-backend=c10d --rdzv-endpoint=localhost:0 --nnodes=1 --nproc-per-node=$NUM_TRAINERS YOUR_TRAINING_SCRIPT.py. Jobs that share a port can conflict or be merged into a single job.
Source: PyTorch 2.14 documentation: torchrun (Elastic Launch). Checked row by row on .
6 more source pages
- PyTorch v2.14.1 source: torch/distributed/run.py, the argument parser behind torchrun
- PyTorch v2.14.1 source: torch/distributed/argparse_util.py
- PyTorch 2.14 documentation: NUMA Binding
- PyTorch Tutorials: Getting Started with Distributed Data Parallel (section on torchrun)
- PyTorch Tutorials: Multinode Training (troubleshooting)
- PyTorch 2.14 documentation: Rendezvous (Torch Distributed Elastic)
Dell iDRAC (racadm) 80
racadm is the command-line interface to the Integrated Dell Remote Access Controller (iDRAC) and runs either on the host itself or from another machine with -r, -u and -p. Its table covers inventory, power control, event and Lifecycle logs, sensors, support collections, storage, configuration profiles, jobs and firmware updates, and the iDRAC's own network, reset, sessions and virtual media.
Connecting and help 4
When iDRAC does not use the default port 443, give the port after the address with -r, separated by a colon.
| Command | What it does |
|---|---|
racadm help | List all subcommandsAdding a subcommand, group, object or FQDD name after help narrows the output to that item. |
racadm help SUBCOMMAND | Help for one subcommandPrints the syntax and options of that subcommand alone, for example racadm help get. |
racadm -r RACIPADDR -u USERNAME -p PASSWORD SUBCOMMAND | Run a subcommand remotelyRemote RACADM from another machine; append :PORT to the address when iDRAC is not listening on 443. |
--nocertwarn | Hide certificate warningsSuppresses the Security Alert that remote RACADM prints when the iDRAC certificate is not signed by a trusted third party. |
Identity and inventory 12
| Command | What it does |
|---|---|
racadm getsysinfo | System infoOne report on the iDRAC, host system and watchdog, with firmware version, service tag and IP settings. |
racadm getsysinfo -d | Show iDRAC information onlyOther filters: -c for common settings, -4 and -6 for IPv4 and IPv6, -A to drop the headers. |
racadm getsvctag | Service tagPrints just the host's service tag, without the rest of the getsysinfo report. |
racadm getversion | Show firmware versionsLists current software versions with model and generation, and whether each target can be updated. |
racadm getversion -b | Show BIOS version-c gives the CPLD version and -i the IDSDM version instead; -f filters by bios, idrac, lc or idsdm. |
racadm hwinventory | Hardware inventoryLists the current hardware inventory device by device, with the properties of each FQDD such as controllers and NICs. |
racadm hwinventory nic | List NIC FQDDsPass one of the returned FQDDs to hwinventory for that port's full details, including its MAC addresses. |
racadm swinventory | Software inventoryLists each software object installed on the server together with its properties. |
racadm license view | View licensesShows status, license type and expiration per device; add -c to limit it to one component. |
racadm getled | Check the ID LED stateAnswers Blinking or Not-Blinking, the two states that setled switches between. |
racadm setled -l 1 | Start ID LED blinkingThe reply RAC0907 confirms that the system ID LED is blinking. |
racadm setled -l 0 | Stop ID LED blinkingThe reply RAC0908 confirms that blinking has been switched off. |
Power control 6
| Command | What it does |
|---|---|
racadm serveraction powerstatus | Show power statusReports ON, OFF or a standby state such as Hibernate or ModernStandby. |
racadm serveraction powerup | Power onThe action word is powerup, not poweron; if the host is already on, the reply says so. |
racadm serveraction powerdown | Power offTurns the managed system off; graceshutdown is the variant that lets the operating system shut down. |
racadm serveraction graceshutdown | Graceful shutdownSkipped when the operating system cannot finish shutting down, so the server may stay on. |
racadm serveraction powercycle | Power cycleA cold reboot: power goes off and back on, like turning the server off and on at the front panel. |
racadm serveraction hardreset | Hard resetForces a reboot; listed in the iDRAC9 6.xx guide, while the 7.xx action list no longer includes it. |
Logs and sensors 13
| Command | What it does |
|---|---|
racadm getsel | Event logPrints every SEL entry; -s and -c pick a start record and count, -o puts each record on one line. |
racadm getsel -i | Number of SEL records |
racadm clrsel | Clear the SELDeletes every SEL record; the account needs the Clear Logs permission. |
racadm getraclog | iDRAC logShows RAC log entries such as session logins, each with sequence number, message ID, category and severity. |
racadm getraclog -c COUNT | Limit iDRAC log recordsLocal RACADM returns at most 100 records by default; -s sets the starting record. |
racadm lclog view | Lifecycle logPrints every record in the active Lifecycle log; over remote RACADM, raise iDRAC.Racadm.Timeout if nothing comes back. |
racadm lclog view -i | Count Lifecycle log records |
racadm lclog view -k STRING | Search records for textThe guide's example filters on the word session to find login records. |
racadm lclog view -c storage,system -s warning,critical | Filter by category and severityCategories include System, Storage, Worknotes, Config, Updates and Audit; both lists are comma-separated and case-insensitive. |
racadm lclog export -f FILENAME -u USERNAME -p PASSWORD -l SHARE | Export the Lifecycle logWrites the log history to a CIFS, NFS, TFTP or FTP share; HTTP and HTTPS shares also take -port. |
racadm lclog export -f FILENAME -u USERNAME -p PASSWORD -l SHARE --complete | Export the full log archiveExports the whole Lifecycle log in compressed .xml.gz form instead of the default export. |
racadm getsensorinfo | Sensor statusCovers sensor types such as power, temperature, fans, voltage, memory, battery and intrusion. |
racadm getsensorinfo -c | Compact sensor list |
Support collections 5
| Command | What it does |
|---|---|
racadm techsupreport collect | Support reportStarts a SupportAssist collection, the new name for the Tech Support Report, and returns a job ID to track. |
racadm techsupreport collect -t SysInfo,TTYLog | Collect system info and TTY logs-t also accepts OSAppNoPII or OSAppAll; OS data needs iSM installed and running on the host. |
racadm techsupreport export -f FILENAME | Save the report locallyWrites the collected report to a .zip file; -l, -u and -p send it to a network share instead. |
racadm supportassist accepteula | Accept the SupportAssist EULANeeded before supportassist collect will run; geteulastatus shows whether it has been accepted. |
racadm supportassist collect | Collect SupportAssist data-t picks log types such as sysinfo, Debug or TTYLog; -f, -l or -upload choose where the data goes. |
Storage inventory 7
Dell's guide says raid can be typed in place of storage, so racadm raid get status and racadm storage get status are the same command.
| Command | What it does |
|---|---|
racadm storage get status | Storage root statusHealth of the storage root node; typing raid in place of storage runs the same command. |
racadm storage get controllers | List storage controllers |
racadm storage get controllers -o | Controller details-o prints every property; add -p with up to ten comma-separated property names to filter. |
racadm storage get vdisks | List virtual disksAdd --refkey with a controller FQDD to list only the disks on that controller. |
racadm storage get vdisks -o -p layout,status | Virtual disk layout and statusThe same -o -p pattern selects properties for controllers, pdisks and enclosures. |
racadm storage get pdisks | List physical disksAdd -vdkey: followed by a virtual disk FQDD to list only the drives in that virtual disk. |
racadm storage get enclosures | List enclosures |
Reading and changing settings 8
| Command | What it does |
|---|---|
racadm get system.location | Read a settings groupPrints every object in the group; this example is the System.Location topology group. |
racadm get system.location.rack.name | Read a single objectGroup and object names follow the FQDD alias, joined by dots. |
racadm set iDRAC.Info.Name NAME | Set the iDRAC nameA real-time object, so it applies at once; staged objects such as BIOS or NIC settings need a job and reboot. |
racadm get -f FILENAME -t xml -l SHARE | Export a configuration profileSaves a Server Configuration Profile; -t json writes JSON and -c limits it to the listed FQDDs. |
racadm get -f FILENAME -t xml -l SHARE --clone | Export a clone profileLeaves out system-specific details such as the service tag; --replace keeps them. |
racadm set -f FILENAME -t xml -l SHARE | Import a configuration profileApplies an SCP file and creates its own job, so no separate jobqueue create is needed. |
racadm set -f FILENAME -t xml -l SHARE --preview | Preview a profile importValidates the file and returns a job ID to check; the preview never restarts the system. |
racadm clearpending FQDD | Clear pending valuesDrops staged changes on a BIOS, NIC, FC, InfiniBand or storage controller FQDD unless a job is already scheduled. |
Jobs and firmware updates 7
| Command | What it does |
|---|---|
racadm jobqueue view | View jobsLists the jobs in the current queue with their status. |
racadm jobqueue view -i JOBID | Check one job's status |
racadm jobqueue create FQDD -r pwrcycle -s TIME_NOW | Create a job for a deviceApplies pending changes on that FQDD now; -r also takes none, graceful or forced. |
racadm jobqueue delete -i JOBID | Delete a jobWith JID_CLEARALL as the ID it force-deletes every job in the queue. |
racadm update -f UPDATEFILE | Update firmware from a local fileRuns through local RACADM on the host; takes a Windows DUP or a .d9, .pm or .sc image. |
racadm update -f UPDATEFILE -l LOCATION -u USERNAME -p PASSWORD | Update firmware from a shareFetches the file from a CIFS, NFS, HTTP or HTTPS share; add --reboot for a graceful restart afterwards. |
racadm update viewreport | View catalog comparison report |
Controller network and reset 10
| Command | What it does |
|---|---|
racadm getniccfg | iDRAC network configShows both the current and the static IP settings of the iDRAC network port. |
racadm setniccfg -s IPV4ADDRESS NETMASK GATEWAY | Set static IPType all three as dotted strings; -s6 does the same for IPv6 with a prefix length. |
racadm setniccfg -d | Switch to DHCPDHCP is the default mode; -d6 turns on IPv6 autoconfiguration instead. |
racadm racreset | Reset iDRACThe reset is recorded in the iDRAC log, and the controller can take up to two minutes to come back. |
racadm racresetcfg | Factory reset iDRACThe DNS RAC name object is kept, and iDRAC stops responding for about 30 seconds while processes restart. |
racadm racresetcfg -all | Factory reset including usersAlso returns the user accounts to their shipping values. |
racadm racresetcfg -rc | Reset to default credentialsDiscards the settings and restores the default user name and password. |
racadm getractime | Show iDRAC timeWith -d the time comes back as YYYYMMDDhhmmss. |
racadm sslcertview -t 1 | View SSL certType 1 is the web server certificate; other -t values select CA, Active Directory and 802.1X certificates. |
racadm sslresetcfg | Reset web server certificateRestores the factory certificate and restarts the web server; the change takes effect after about 30 seconds. |
Sessions and virtual media 8
| Command | What it does |
|---|---|
racadm getssninfo | List sessionsShows session ID, user, IP address, session type and login time for everyone connected. |
racadm closessn -i SESSION_ID | Close a sessionTake the ID from getssninfo; the session running the command cannot be closed. |
racadm closessn -u USERNAME | Close a user's sessions |
racadm closessn -a | Close all other sessionsNeeds Administrator permission; the session that runs the command stays open. |
racadm remoteimage -c -u USERNAME -p PASSWORD -l IMAGE_PATH | Attach a remote imageMounts an image from a network share as virtual media; wrap the path in single quotes and use forward slashes. |
racadm remoteimage -s | Virtual media statusShows whether a remote file share is attached and the image path it points to. |
racadm remoteimage -d | Detach the remote image |
racadm vmdisconnect | End virtual media sessionsCloses every active Virtual Media session; needs the Access Virtual Media permission. |
Identify and power
racadm getsysinfo prints the iDRAC, host system and watchdog details in one report, including firmware version, service tag and network settings, and racadm getsvctag returns just the tag for support tickets. The power actions are racadm serveraction powerup and racadm serveraction powerdown, with racadm serveraction graceshutdown for an orderly stop that the operating system must complete and racadm serveraction powercycle for a cold reboot. racadm serveraction hardreset forces a reboot; it is listed in the 6.xx edition of Dell's guide but no longer in the 7.xx action list.
What happened
racadm getsel prints the System Event Log and racadm lclog view the Lifecycle log, whose categories also cover configuration changes, updates and audit records; racadm getraclog holds the iDRAC's own entries such as session logins. For disks, racadm storage get status gives the health of the storage root node and racadm storage get vdisks lists the virtual disks; Dell's guide notes that raid works in place of storage. racadm techsupreport collect gathers a SupportAssist collection for Dell support; save it with racadm techsupreport export -f FILENAME before opening a case.
The iDRAC itself
racadm getniccfg shows the iDRAC's IP configuration; racadm setniccfg -s IPV4ADDRESS NETMASK GATEWAY sets a static address from the host OS and racadm setniccfg -d returns the port to DHCP. racadm racreset restarts the controller, which can take up to two minutes to respond again. racadm racresetcfg restores factory settings, and racadm racresetcfg -all and racadm racresetcfg -rc also reset the user accounts, so use them deliberately. racadm jobqueue view lists pending firmware and configuration jobs, and racadm sslcertview -t 1 shows the certificate the web interface presents.
Is it racadm raid get status or racadm storage get status?
Dell's guide documents the storage form, racadm storage get status, and notes that raid can be typed in place of storage, so both spellings run the same command. A plain raid status without get is not in the guide.
How do I run racadm against a remote iDRAC?
Put the connection options before the subcommand: racadm -r RACIPADDR -u USERNAME -p PASSWORD SUBCOMMAND. If iDRAC does not use port 443, add the port after the address with a colon, and add --nocertwarn to hide the warning shown when the iDRAC certificate is not signed by a trusted third party.
Which serveraction words turn the server on and off?
The guide names them powerup and powerdown: racadm serveraction powerup turns the host on, racadm serveraction powerdown turns it off and racadm serveraction powerstatus reports the current state. For an orderly stop through the operating system use racadm serveraction graceshutdown.
Which iDRAC version does this page follow?
Dell's iDRAC9 RACADM CLI Guide for the 7.xx firmware series, Rev. A08, which covers releases up to 7.30.30.54. racadm serveraction hardreset comes from the 6.xx edition of the same guide, because the 7.xx action list no longer includes it.
Source: Dell — Integrated Dell Remote Access Controller 9 RACADM CLI Guide, 7.xx series (Rev. A08, September 2026, PDF). Checked row by row on .
1 more source page
No row on this page matches that filter.
Questions
A multi-node torchrun job fails on one host. Which commands help find the cause?
Launch with --tee 3 and --log-line-prefix-template '${hostname}:${rank}: ' so that each console line starts with the node name and global rank, as in torchrun --nnodes 2 --nproc-per-node 8 --tee 3 --log-line-prefix-template "${hostname}:${rank}: " train.py. When nodes fail to communicate, PyTorch's multinode tutorial first suggests export NCCL_DEBUG=INFO. If the host is a Dell server, its iDRAC reports on the hardware: racadm getsel prints the System Event Log and racadm getsensorinfo covers sensors such as power, temperature, fans and memory.
Which racadm commands identify a server and its location?
racadm setled -l 1 makes the system ID LED blink, racadm setled -l 0 stops it, and racadm getled answers Blinking or Not-Blinking. racadm getsvctag prints the service tag alone. racadm get system.location shows the System.Location topology group, and racadm get system.location.rack.name reads a single object from it, the rack name.
How many workers should torchrun start on a GPU node?
--nproc-per-node NPROC_PER_NODE starts one worker unless set, and for GPU training it should stay at or below the number of GPUs. --nproc-per-node gpu uses the CUDA device count and stops with an error when CUDA is not available, while --nproc-per-node auto falls back to the XPU count and then to one worker per CPU. torchrun expects the same value on every node, and when a node runs more than one worker with OMP_NUM_THREADS unset, it sets that variable to 1 and prints a warning.
Sources and checks
Each tool on this page keeps its own source line under its table. The documented lists of these 2 tools are short enough to read together, so they share this page; each keeps its own section, and its old address redirects to that section.
Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.