Run:ai commands

Command lines for the NVIDIA Run:ai CLI v2, the runai client documented for Run:ai 2.26, taken from the CLI reference and examples in NVIDIA's Run:ai documentation and grouped by task. Words in capitals are placeholders; test, ubuntu and proj1 are the docs' sample values. Rows that begin with a dash are flags for the submit or list commands. The old v1 syntax (runai submit, runai list jobs) is left out: NVIDIA supports it only for clusters below 2.18.

133 commands in 13 sections. Checked row by row against NVIDIA Run:ai 2.26 documentation (self-hosted): runai (CLI Commands Reference) on . Edited by Jay.

Installing and signing in 15

This is CLI v2, which needs a cluster of version 2.18 or later; the old v1 client is supported only for clusters below 2.18.

CommandWhat it does
runai versionCheck the installed CLIThe install guide runs it as the test that the client was set up correctly.
runai version --wideShow full version details
runai config set --cp-url CONTROL_PLANE_URLSet the control plane URLOnly Windows users run this by hand; the Linux and Mac installers do it, and it also writes config.json.
runai loginSign in through the browserOpens the Run:ai sign-in page, which handles SSO as well as user name and password accounts.
runai login --listen-port=43121 --listen-host=localhostSet the login callback portMoves the local listener that receives the browser's sign-in callback; 43121 and localhost are the defaults.
runai login user -u USERNAMESign in without a browserThe password is then asked for on standard input, which the reference prefers to passing -p on the command line.
runai login ssoSign in with SSO, no local browserFor SSO accounts on a machine that cannot open a browser itself.
runai login access-keySign in with an access keyAsks for the key interactively; with --client-id, --secret and --interactive=disabled it runs unattended, and --save keeps the credentials.
runai logoutSign out of Run:ai
runai whoamiShow the signed-in user
runai auth get-tokenPrint an access tokenPrints it in kubeconfig form by default; -o plaintext gives the bare token.
runai kubeconfig setAdd Run:ai auth to kubeconfigLets third-party tools work under Run:ai authorization; the runai CLI itself does not need this step.
runai upgradeUpgrade the CLIUpdates the client by hand; --force skips the check for a newer version.
runai config set --auto-update trueTurn on automatic updatesSaves you from running runai upgrade by hand whenever a new client version is released.
runai config set --auto-update falseTurn off automatic updatesFrom then on every command prints a warning that automatic upgrades are off.

Clusters and projects 14

Every command that takes -p falls back to the default project from runai project set when -p is left out.

CommandWhat it does
runai clusterPick a cluster from a menuOnly needed when several clusters are connected; a single cluster becomes the default at the first sign-in.
runai cluster listList clustersShows the names that runai cluster set accepts.
runai cluster set CLUSTER_NAMESet the default clusterWorkloads you submit go to this cluster until you choose another.
runai projectPick a project from a menuChooses the default project interactively, so later commands can leave out -p.
runai project listList projects
runai project list --yamlList projects as YAML
runai project set PROJECT_NAMESet the default projectCommands then use this project whenever -p is missing; the CLI confirms that the project was configured.
runai project create NAMECreate a projectWithout --resources it uses the default node pool, 0 deserved GPUs and a GPU limit of -1.
runai node listList nodes
runai node list --jsonList nodes as JSON
runai nodepool listList node pools
runai pvc list -p PROJECT-NAMEList a project's PVCsAdd --json or --yaml to the same command for other output formats.
runai template list -p PROJECT-NAMEList workload templatesTemplates defined at higher scopes are shown too, not only the project's own.
runai environment list -AList environment assets

Submitting workloads 15

runai training submit and the other plain runai training verbs act on standard training; the reference prints their examples in the longer runai training standard form.

CommandWhat it does
runai training submitSubmit a training workloadThe reference's own examples write the same command as runai training standard submit.
runai training standard submit WORKLOAD-NAME -p PROJECT-NAME -i runai.jfrog.io/demo/quickstart-demoSubmit a named training jobNames the workload, picks the project with -p and the container image with -i.
runai training standard submit WORKLOAD-NAME -p PROJECT-NAME --template TEMPLATE-NAMESubmit from a templateTemplate values and command-line flags are merged; a flag for the same key overrides the template.
runai workspace submit WORKLOAD-NAME -p PROJECT-NAME -i runai.jfrog.io/demo/quickstart-demoSubmit a workspace
runai workspace submit -p test -i ubuntuSubmit with a generated nameLeaving out the name gives a random one that starts with the workload type.
runai workspace submit WORKLOAD-NAME -p PROJECT-NAME -i ubuntu -- ls -laPass arguments to the imageThe words after -- become arguments, and the image's entrypoint stays because --command is not set.
runai workspace submit WORKLOAD-NAME -p PROJECT-NAME -i ubuntu --command -- echo "Hello, World"Override the entrypointWith --command the words after -- replace the image's entrypoint instead of being handed to it.
runai workspace submit WORKLOAD-NAME -p PROJECT-NAME --environment ENVIRONMENT-ASSET-NAMESubmit with an environment asset
runai workspace submit -p PROJECT-NAME -i python --attach -- python3Submit and attachSubmits the workload and connects your terminal to its python3 process straight away.
runai workspace submit --image jupyter/scipy-notebook -p PROJECT-NAME --gpu-devices-request 1 --external-url container=8888 --name-prefix jupyter --command -- start-notebook.sh --NotebookApp.base_url='/${RUNAI_PROJECT}/${RUNAI_JOB_NAME}' --NotebookApp.token=''Start a Jupyter notebookRequests one GPU and publishes the notebook's port 8888 at an external URL.
runai workspace submit -p test -i ubuntu -e name=value -e name2=value2Set environment variablesRepeat -e once for each variable the container should see.
runai workspace submit -p test -i ubuntu --label name=value --label name2=value2Add labels
runai inference submit WORKLOAD-NAME -p PROJECT-NAME -i ghcr.io/knative/helloworld-go --gpu-devices-request 1 --serving-port=8080 --min-replicas=0 --max-replicas=1Deploy inference with scale to zero--min-replicas=0 allows scaling down to no replicas, and --serving-port names the port that answers requests.
runai inference update WORKLOAD-NAME -p PROJECT-NAME -i runai.jfrog.io/demo/quickstart-demoChange an inference image
runai inference update WORKLOAD-NAME -p PROJECT-NAME --max-replicas=5 --min-replicas=3 --metric=concurrency --metric-threshold=10Change inference autoscalingSets three to five replicas, scaled on the concurrency metric with a threshold of 10.

Distributed training 9

The top-level runai pytorch, runai mpi and runai tensorflow groups (and runai xgboost and runai jax) are aliases for the framework groups under runai training.

CommandWhat it does
runai training pytorch submit -p PROJECT-NAME -i nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu20.04 -g 1 --workers 3 --command -- bash -c 'trap : TERM INT; sleep infinity & wait'PyTorch job with three workersThe sleep loop keeps the pods alive so that you can open a shell in them afterwards.
runai training pytorch bash pytorch-06027b585626 -p PROJECT-NAMEOpen a shell in a PyTorch jobpytorch-06027b585626 is a sample generated name; use the name of your own workload.
runai mpi submit dist1 --workers=2 -g 1 -i runai.jfrog.io/demo/quickstart-distributed:v0.3.0 -e RUNAI_SLEEP_SECS=60 -p PROJECT-NAMESubmit an MPI jobStarts the job dist1 with two workers and passes RUNAI_SLEEP_SECS to the containers.
runai tensorflow submit --workers=5 --no-master -g 1 -i kubeflow/tf-mnist-with-summaries:latest -p PROJECT-NAME --command -- python /var/tf_mnist/mnist_with_summaries.py --max_steps 1000000Submit a TensorFlow jobRuns the MNIST sample on five workers without a separate master pod.
--workersSet the number of workers
--no-masterRun without a master pod
--master-argsArguments for the master onlyThe arguments after -- then go to the workers, so leader and workers can differ.
--master-commandCommand for the master onlyCan be combined with --master-args; --command with -- still sets the workers' command.
--clean-pod-policy AllRemove pods when the job endsDeletes the pods once the job has completed or failed; None and Running are the other values.

Resource requests 11

CommandWhat it does
--gpu-devices-request 1Request whole GPUsShort form -g; counts complete devices, unlike the portion and memory requests.
--gpu-portion-request 0.5Request part of a GPUA value between 0 and 1 asks for that share of a single device.
--gpu-portion-limitCap the GPU fractionUpper bound for the share, also given as a number between 0 and 1.
--gpu-memory-request 1GRequest GPU memorySizes the GPU share by memory, written like 1G or 500M.
--gpu-memory-limitCap GPU memory
--cpu-core-request 0.5Request CPU coresFractions of a core are allowed; --cpu-core-limit sets the ceiling.
--cpu-core-limit 1Cap CPU cores
--cpu-memory-request 1GRequest CPU memory
--cpu-memory-limit 1GCap CPU memory
--large-shmUse a large /dev/shmGives the container a bigger shared-memory device for memory-heavy work.
runai workspace submit -p PROJECT-NAME -i runai.jfrog.io/demo/quickstart-demo --cpu-core-request 0.3 --cpu-core-limit 1 --cpu-memory-request 50M --cpu-memory-limit 1G --gpu-devices-request 1 --gpu-memory-request 1GSet requests and limits togetherOne submit line combining CPU, memory and GPU requests with their limits.

Scheduling and placement 7

CommandWhat it does
--node-pools pool-a --node-pools pool-bChoose node poolsThe order of the flags is the order of preference.
--exclude-node node-a --exclude-node node-bKeep off certain nodesThe scheduler will not place the workload on the named nodes.
--node-typeRequire a node typeSets node affinity through a node-type label.
--prioritySet scheduling priorityValues run from very-low to very-high and replace the workload type's default; preemptibility is not changed with it.
--preemptibility non-preemptibleMake a workload non-preemptibleHigher-priority workloads can then not displace it; preemptible is the other value.
--backoff-limitSet retries for a failed jobTraining jobs are marked failed only after this many retries.
--auto-deletion-time-after-completionDelete a finished job laterTakes a duration such as 5s, 2m or 3h counted from completion.

Container and storage options 11

CommandWhat it does
--imageChoose the container imageShort form -i, which is how the reference examples pass it.
--image-pull-policy AlwaysSet the image pull policyIfNotPresent and Never are the other accepted values.
--working-dirSet the working directoryReplaces the working directory that the image defines.
--run-as-userRun as your own userTakes the user and group IDs from your local terminal credentials when none are given.
runai workspace submit -i ubuntu --existing-pvc claimname=test-pvc-2-project-mn2xs,path=/home/testMount an existing PVCThe claim stays in place after the workload is removed.
runai workspace submit -i ubuntu --new-pvc claimname=yuval-3,size=10M,path=/tmp/testCreate a PVC for the workloadThe new claim is removed together with the workload.
runai workspace submit -i ubuntu --new-pvc claimname=yuval2,size=10M,path=/tmp/test,ephemeralCreate an ephemeral PVCGone as soon as the workload is paused, not only when it is deleted.
--git-sync name=NAME,repository=REPO,path=PATH,secret=SECRET,rev=REVISIONMount a Git repository
--nfs path=PATH,server=SERVER,mountpath=MOUNT_PATH,readwriteMount an NFS volume
--external-url container=8888Expose a port at a URLPublishes a container port, here the notebook's 8888, at an external address.
--port service-type=NodePort,container=80,external=8080Expose a portMaps container port 80 to 8080 through a NodePort service.

Listing and describing workloads 12

CommandWhat it does
runai workload listList workloadsShows training, workspace and inference workloads together; --type narrows the list to one of them.
runai workload list -AList workloads in all projectsCovers every workload within your scope instead of the default project only.
runai training list -p PROJECT-NAMEList a project's training jobs
runai training list --status STATUSFilter training by stateStates include Pending, Running, Completed, Failed and Stopped.
runai training list --framework FRAMEWORKFilter training by frameworkFrameworks are TensorFlow, PyTorch, MPI, XGBoost and JAX.
runai workspace list --yamlList workspaces as YAML
runai inference list -AList all inference workloads
runai workload describe WORKLOAD_NAMEDescribe a workloadPrints general, compute, pod, event, network and storage sections, all on by default.
runai workspace describe WORKLOAD-NAME -o jsonDescribe as JSONOutput can be table, json or yaml; table is the default.
runai workspace describe WORKLOAD-NAME --containers --pod-limit 20 --event-limit 100Describe with container detailsAdds container-level detail and raises the pod and event limits from 10 and 50.
--columnsChoose list columnsTakes a comma-separated set of column names for the list commands.
--no-headersPrint a table without headers

Viewing logs 7

CommandWhat it does
runai training logs WORKLOAD_NAMEView training logs
runai workspace logs WORKLOAD-NAME --followStream logs liveShort form -f; keeps printing new lines as they arrive.
runai workspace logs WORKLOAD-NAME --tail=100Show the last 100 linesShort form -t; the default of -1 prints every line.
runai workspace logs WORKLOAD-NAME --since=300sLogs from the last 5 minutes--since-time takes an RFC3339 timestamp instead of a duration.
runai workspace logs WORKLOAD-NAME --previousLogs of the previous instanceReads the output of the pod's earlier run after a restart.
runai workspace logs WORKLOAD-NAME --timestampsAdd timestamps to logs
runai workspace logs WORKLOAD-NAME --limit-bytes=1024Cap the log output sizeStops after 1024 bytes; without the flag there is no limit.

Shells, attach and port forwarding 7

CommandWhat it does
runai workspace bash WORKLOAD-NAMEOpen a bash shellOpens it in the main worker; --pod picks a different pod.
runai workspace exec WORKLOAD-NAME --tty --stdin -- /bin/bashRun bash through exec--tty and --stdin, short -t and -i, make the session interactive.
runai workspace exec WORKLOAD-NAME -- lsRun a single commandThe command after -- runs in the workload's main worker.
runai workspace exec WORKLOAD-NAME --pod POD-NAME -- nvidia-smiRun a command in one podUseful for checking the GPUs of a particular worker in a multi-pod workload.
runai workspace attach WORKLOAD-NAME --tty --stdinAttach to the main workerConnects your terminal to the running container; add --pod for another pod.
runai workspace port-forward WORKLOAD-NAME --port 8080:8090 --address localhostForward a local portConnections to localhost:8080 reach port 8090 inside the workload.
runai workspace port-forward WORKLOAD-NAME --port 8080:8090 --port 6443:443 --address localhostForward several portsRepeat --port for each local and remote pair.

Suspending and deleting 9

CommandWhat it does
runai workspace suspend WORKSPACE-NAMESuspend a workspace
runai workspace resume WORKSPACE-NAMEResume a workspace
runai training suspend WORKLOAD_NAMESuspend a training job
runai training resume WORKLOAD_NAMEResume a training job
runai training delete WORKLOAD_NAMEDelete a training job-p picks the project, and --uuid deletes by ID instead of by name.
runai workspace delete WORKLOAD-NAMEDelete a workspace
runai workload delete -p proj1 workload01 workload02 workload03Delete several workloadsAsks for confirmation first unless -y is given.
runai workload delete -p proj1 -y workload01 workload02 workload03Delete without confirming
runai workload delete -p proj1 --type training workload01 workload02 workload03Delete only training workloads--type accepts training, workspace or inference.

Configuration and global flags 11

CommandWhat it does
runai config describeShow the CLI configurationAdd --json or --yaml for structured output.
runai config generate --jsonPrint the current configurationThe install guide uses it to check that configuration worked.
runai config generate -f path/to/config.jsonWrite a config fileA file name ending in .yaml produces YAML instead.
runai config set --status-timeout-duration 5sChange the status timeoutThe cluster status call gives up after 3s by default; values such as 300ms work too.
--config-fileUse another config fileThe RUNAI_CLI_CONFIG_FILE variable does the same; the default name is config.json.
--config-pathUse another config folderCan also come from the RUNAI_CLI_CONFIG_PATH environment variable.
--debugTurn on debug outputShort form -d; accepted by every runai command.
--quietShow errors onlyShort form -q; everything except error messages is suppressed.
--verboseTurn on verbose output
--helpShow help for a commandShort form -h; works after any command or subcommand.
--jsonOutput as JSONThe list commands also take --yaml and --table.

Completion and diagnostics 5

CommandWhat it does
source <(runai completion bash)Load Bash completionAdd the line to ~/.bashrc after installing the bash-completion package.
source <(runai completion zsh)Load Zsh completionGoes in ~/.zshrc after the compinit line.
runai.exe completion powershell | Out-String | Invoke-ExpressionLoad PowerShell completionAdd it to your PowerShell profile on Windows.
runai diagnostics collect-logsCollect logs for supportGathers pod logs from the Run:ai and related namespaces into an archive in the current directory.
runai diagnostics collect-logs --output-dir /tmp/logsSave the logs archive elsewhereThe archive is written to the current directory unless --output-dir names another.

Notes

Setup once

runai version confirms that the client is installed. runai login opens the sign-in page in a browser; on a machine without one, runai login user -u USERNAME asks for the password on standard input. Windows users first point the client at the control plane with runai config set --cp-url CONTROL_PLANE_URL; the Linux and Mac installers do that themselves. runai cluster set CLUSTER_NAME and runai project set PROJECT_NAME store defaults, so later commands can leave out -p. With only one cluster connected, the first sign-in already makes it the default.

Submitting and watching

Each workload type has its own command group, runai workspace, runai training and runai inference, with the same verbs: submit, list, describe, logs, bash, exec and port-forward, plus attach, suspend and resume for workspaces and training. Resources are flags on the submit line: --gpu-devices-request 1 (short -g) for whole GPUs, --gpu-portion-request 0.5 for part of one. runai workload list shows all types together, and runai workload describe WORKLOAD_NAME prints a workload's pods and recent events. runai workspace logs WORKLOAD-NAME --follow streams output, and runai workspace bash WORKLOAD-NAME opens a shell in the main worker.

Coming from CLI v1

Scripts written for the first CLI use runai submit, runai list jobs and runai config project. In CLI v2 the workload type comes first: runai training submit, runai workload list and runai project set PROJECT_NAME. NVIDIA supports v1 only on clusters older than version 2.18, and the v2 reference has no counterpart for runai top node or runai delete project; runai node list and runai nodepool list are the nearest listing commands.

Questions

How do I request a GPU when submitting a Run:ai workload?

Add --gpu-devices-request 1 (short form -g) to a submit command for whole GPUs. For a share of one GPU use --gpu-portion-request 0.5, a value between 0 and 1, or ask for an amount of GPU memory with --gpu-memory-request 1G.

Why does runai submit no longer work?

runai submit belongs to CLI v1, which NVIDIA supports only for clusters below version 2.18. CLI v2 names the workload type first: runai training submit for training jobs, runai workspace submit for workspaces and runai inference submit for inference.

How do I open a shell inside a running Run:ai workload?

runai workspace bash WORKLOAD-NAME opens bash in the main worker, and --pod picks another pod. runai workspace exec WORKLOAD-NAME --tty --stdin -- /bin/bash does the same through exec. The training and inference groups have the same bash and exec commands.

How do I set a default project in the Run:ai CLI?

Run runai project set PROJECT_NAME, or runai project to choose from a menu. Commands then use that project whenever -p is left out; runai project list shows the projects available to you.

Sources and checks

42 more source pages

Every row on this page was matched to a line in the documentation above on . A binding that could not be found there is not listed.

2 rows from the earlier version of this page were removed in that check: duplicates, and bindings the documentation does not list or lists differently.

Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.