Installing and signing in 15
This is CLI v2, which needs a cluster of version 2.18 or later; the old v1 client is supported only for clusters below 2.18.
| Command | What it does |
|---|---|
runai version | Check the installed CLIThe install guide runs it as the test that the client was set up correctly. |
runai version --wide | Show full version details |
runai config set --cp-url CONTROL_PLANE_URL | Set the control plane URLOnly Windows users run this by hand; the Linux and Mac installers do it, and it also writes config.json. |
runai login | Sign in through the browserOpens the Run:ai sign-in page, which handles SSO as well as user name and password accounts. |
runai login --listen-port=43121 --listen-host=localhost | Set the login callback portMoves the local listener that receives the browser's sign-in callback; 43121 and localhost are the defaults. |
runai login user -u USERNAME | Sign in without a browserThe password is then asked for on standard input, which the reference prefers to passing -p on the command line. |
runai login sso | Sign in with SSO, no local browserFor SSO accounts on a machine that cannot open a browser itself. |
runai login access-key | Sign in with an access keyAsks for the key interactively; with --client-id, --secret and --interactive=disabled it runs unattended, and --save keeps the credentials. |
runai logout | Sign out of Run:ai |
runai whoami | Show the signed-in user |
runai auth get-token | Print an access tokenPrints it in kubeconfig form by default; -o plaintext gives the bare token. |
runai kubeconfig set | Add Run:ai auth to kubeconfigLets third-party tools work under Run:ai authorization; the runai CLI itself does not need this step. |
runai upgrade | Upgrade the CLIUpdates the client by hand; --force skips the check for a newer version. |
runai config set --auto-update true | Turn on automatic updatesSaves you from running runai upgrade by hand whenever a new client version is released. |
runai config set --auto-update false | Turn off automatic updatesFrom then on every command prints a warning that automatic upgrades are off. |
Clusters and projects 14
Every command that takes -p falls back to the default project from runai project set when -p is left out.
| Command | What it does |
|---|---|
runai cluster | Pick a cluster from a menuOnly needed when several clusters are connected; a single cluster becomes the default at the first sign-in. |
runai cluster list | List clustersShows the names that runai cluster set accepts. |
runai cluster set CLUSTER_NAME | Set the default clusterWorkloads you submit go to this cluster until you choose another. |
runai project | Pick a project from a menuChooses the default project interactively, so later commands can leave out -p. |
runai project list | List projects |
runai project list --yaml | List projects as YAML |
runai project set PROJECT_NAME | Set the default projectCommands then use this project whenever -p is missing; the CLI confirms that the project was configured. |
runai project create NAME | Create a projectWithout --resources it uses the default node pool, 0 deserved GPUs and a GPU limit of -1. |
runai node list | List nodes |
runai node list --json | List nodes as JSON |
runai nodepool list | List node pools |
runai pvc list -p PROJECT-NAME | List a project's PVCsAdd --json or --yaml to the same command for other output formats. |
runai template list -p PROJECT-NAME | List workload templatesTemplates defined at higher scopes are shown too, not only the project's own. |
runai environment list -A | List environment assets |
Submitting workloads 15
runai training submit and the other plain runai training verbs act on standard training; the reference prints their examples in the longer runai training standard form.
| Command | What it does |
|---|---|
runai training submit | Submit a training workloadThe reference's own examples write the same command as runai training standard submit. |
runai training standard submit WORKLOAD-NAME -p PROJECT-NAME -i runai.jfrog.io/demo/quickstart-demo | Submit a named training jobNames the workload, picks the project with -p and the container image with -i. |
runai training standard submit WORKLOAD-NAME -p PROJECT-NAME --template TEMPLATE-NAME | Submit from a templateTemplate values and command-line flags are merged; a flag for the same key overrides the template. |
runai workspace submit WORKLOAD-NAME -p PROJECT-NAME -i runai.jfrog.io/demo/quickstart-demo | Submit a workspace |
runai workspace submit -p test -i ubuntu | Submit with a generated nameLeaving out the name gives a random one that starts with the workload type. |
runai workspace submit WORKLOAD-NAME -p PROJECT-NAME -i ubuntu -- ls -la | Pass arguments to the imageThe words after -- become arguments, and the image's entrypoint stays because --command is not set. |
runai workspace submit WORKLOAD-NAME -p PROJECT-NAME -i ubuntu --command -- echo "Hello, World" | Override the entrypointWith --command the words after -- replace the image's entrypoint instead of being handed to it. |
runai workspace submit WORKLOAD-NAME -p PROJECT-NAME --environment ENVIRONMENT-ASSET-NAME | Submit with an environment asset |
runai workspace submit -p PROJECT-NAME -i python --attach -- python3 | Submit and attachSubmits the workload and connects your terminal to its python3 process straight away. |
runai workspace submit --image jupyter/scipy-notebook -p PROJECT-NAME --gpu-devices-request 1 --external-url container=8888 --name-prefix jupyter --command -- start-notebook.sh --NotebookApp.base_url='/${RUNAI_PROJECT}/${RUNAI_JOB_NAME}' --NotebookApp.token='' | Start a Jupyter notebookRequests one GPU and publishes the notebook's port 8888 at an external URL. |
runai workspace submit -p test -i ubuntu -e name=value -e name2=value2 | Set environment variablesRepeat -e once for each variable the container should see. |
runai workspace submit -p test -i ubuntu --label name=value --label name2=value2 | Add labels |
runai inference submit WORKLOAD-NAME -p PROJECT-NAME -i ghcr.io/knative/helloworld-go --gpu-devices-request 1 --serving-port=8080 --min-replicas=0 --max-replicas=1 | Deploy inference with scale to zero--min-replicas=0 allows scaling down to no replicas, and --serving-port names the port that answers requests. |
runai inference update WORKLOAD-NAME -p PROJECT-NAME -i runai.jfrog.io/demo/quickstart-demo | Change an inference image |
runai inference update WORKLOAD-NAME -p PROJECT-NAME --max-replicas=5 --min-replicas=3 --metric=concurrency --metric-threshold=10 | Change inference autoscalingSets three to five replicas, scaled on the concurrency metric with a threshold of 10. |
Distributed training 9
The top-level runai pytorch, runai mpi and runai tensorflow groups (and runai xgboost and runai jax) are aliases for the framework groups under runai training.
| Command | What it does |
|---|---|
runai training pytorch submit -p PROJECT-NAME -i nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu20.04 -g 1 --workers 3 --command -- bash -c 'trap : TERM INT; sleep infinity & wait' | PyTorch job with three workersThe sleep loop keeps the pods alive so that you can open a shell in them afterwards. |
runai training pytorch bash pytorch-06027b585626 -p PROJECT-NAME | Open a shell in a PyTorch jobpytorch-06027b585626 is a sample generated name; use the name of your own workload. |
runai mpi submit dist1 --workers=2 -g 1 -i runai.jfrog.io/demo/quickstart-distributed:v0.3.0 -e RUNAI_SLEEP_SECS=60 -p PROJECT-NAME | Submit an MPI jobStarts the job dist1 with two workers and passes RUNAI_SLEEP_SECS to the containers. |
runai tensorflow submit --workers=5 --no-master -g 1 -i kubeflow/tf-mnist-with-summaries:latest -p PROJECT-NAME --command -- python /var/tf_mnist/mnist_with_summaries.py --max_steps 1000000 | Submit a TensorFlow jobRuns the MNIST sample on five workers without a separate master pod. |
--workers | Set the number of workers |
--no-master | Run without a master pod |
--master-args | Arguments for the master onlyThe arguments after -- then go to the workers, so leader and workers can differ. |
--master-command | Command for the master onlyCan be combined with --master-args; --command with -- still sets the workers' command. |
--clean-pod-policy All | Remove pods when the job endsDeletes the pods once the job has completed or failed; None and Running are the other values. |
Resource requests 11
| Command | What it does |
|---|---|
--gpu-devices-request 1 | Request whole GPUsShort form -g; counts complete devices, unlike the portion and memory requests. |
--gpu-portion-request 0.5 | Request part of a GPUA value between 0 and 1 asks for that share of a single device. |
--gpu-portion-limit | Cap the GPU fractionUpper bound for the share, also given as a number between 0 and 1. |
--gpu-memory-request 1G | Request GPU memorySizes the GPU share by memory, written like 1G or 500M. |
--gpu-memory-limit | Cap GPU memory |
--cpu-core-request 0.5 | Request CPU coresFractions of a core are allowed; --cpu-core-limit sets the ceiling. |
--cpu-core-limit 1 | Cap CPU cores |
--cpu-memory-request 1G | Request CPU memory |
--cpu-memory-limit 1G | Cap CPU memory |
--large-shm | Use a large /dev/shmGives the container a bigger shared-memory device for memory-heavy work. |
runai workspace submit -p PROJECT-NAME -i runai.jfrog.io/demo/quickstart-demo --cpu-core-request 0.3 --cpu-core-limit 1 --cpu-memory-request 50M --cpu-memory-limit 1G --gpu-devices-request 1 --gpu-memory-request 1G | Set requests and limits togetherOne submit line combining CPU, memory and GPU requests with their limits. |
Scheduling and placement 7
| Command | What it does |
|---|---|
--node-pools pool-a --node-pools pool-b | Choose node poolsThe order of the flags is the order of preference. |
--exclude-node node-a --exclude-node node-b | Keep off certain nodesThe scheduler will not place the workload on the named nodes. |
--node-type | Require a node typeSets node affinity through a node-type label. |
--priority | Set scheduling priorityValues run from very-low to very-high and replace the workload type's default; preemptibility is not changed with it. |
--preemptibility non-preemptible | Make a workload non-preemptibleHigher-priority workloads can then not displace it; preemptible is the other value. |
--backoff-limit | Set retries for a failed jobTraining jobs are marked failed only after this many retries. |
--auto-deletion-time-after-completion | Delete a finished job laterTakes a duration such as 5s, 2m or 3h counted from completion. |
Container and storage options 11
| Command | What it does |
|---|---|
--image | Choose the container imageShort form -i, which is how the reference examples pass it. |
--image-pull-policy Always | Set the image pull policyIfNotPresent and Never are the other accepted values. |
--working-dir | Set the working directoryReplaces the working directory that the image defines. |
--run-as-user | Run as your own userTakes the user and group IDs from your local terminal credentials when none are given. |
runai workspace submit -i ubuntu --existing-pvc claimname=test-pvc-2-project-mn2xs,path=/home/test | Mount an existing PVCThe claim stays in place after the workload is removed. |
runai workspace submit -i ubuntu --new-pvc claimname=yuval-3,size=10M,path=/tmp/test | Create a PVC for the workloadThe new claim is removed together with the workload. |
runai workspace submit -i ubuntu --new-pvc claimname=yuval2,size=10M,path=/tmp/test,ephemeral | Create an ephemeral PVCGone as soon as the workload is paused, not only when it is deleted. |
--git-sync name=NAME,repository=REPO,path=PATH,secret=SECRET,rev=REVISION | Mount a Git repository |
--nfs path=PATH,server=SERVER,mountpath=MOUNT_PATH,readwrite | Mount an NFS volume |
--external-url container=8888 | Expose a port at a URLPublishes a container port, here the notebook's 8888, at an external address. |
--port service-type=NodePort,container=80,external=8080 | Expose a portMaps container port 80 to 8080 through a NodePort service. |
Listing and describing workloads 12
| Command | What it does |
|---|---|
runai workload list | List workloadsShows training, workspace and inference workloads together; --type narrows the list to one of them. |
runai workload list -A | List workloads in all projectsCovers every workload within your scope instead of the default project only. |
runai training list -p PROJECT-NAME | List a project's training jobs |
runai training list --status STATUS | Filter training by stateStates include Pending, Running, Completed, Failed and Stopped. |
runai training list --framework FRAMEWORK | Filter training by frameworkFrameworks are TensorFlow, PyTorch, MPI, XGBoost and JAX. |
runai workspace list --yaml | List workspaces as YAML |
runai inference list -A | List all inference workloads |
runai workload describe WORKLOAD_NAME | Describe a workloadPrints general, compute, pod, event, network and storage sections, all on by default. |
runai workspace describe WORKLOAD-NAME -o json | Describe as JSONOutput can be table, json or yaml; table is the default. |
runai workspace describe WORKLOAD-NAME --containers --pod-limit 20 --event-limit 100 | Describe with container detailsAdds container-level detail and raises the pod and event limits from 10 and 50. |
--columns | Choose list columnsTakes a comma-separated set of column names for the list commands. |
--no-headers | Print a table without headers |
Viewing logs 7
| Command | What it does |
|---|---|
runai training logs WORKLOAD_NAME | View training logs |
runai workspace logs WORKLOAD-NAME --follow | Stream logs liveShort form -f; keeps printing new lines as they arrive. |
runai workspace logs WORKLOAD-NAME --tail=100 | Show the last 100 linesShort form -t; the default of -1 prints every line. |
runai workspace logs WORKLOAD-NAME --since=300s | Logs from the last 5 minutes--since-time takes an RFC3339 timestamp instead of a duration. |
runai workspace logs WORKLOAD-NAME --previous | Logs of the previous instanceReads the output of the pod's earlier run after a restart. |
runai workspace logs WORKLOAD-NAME --timestamps | Add timestamps to logs |
runai workspace logs WORKLOAD-NAME --limit-bytes=1024 | Cap the log output sizeStops after 1024 bytes; without the flag there is no limit. |
Shells, attach and port forwarding 7
| Command | What it does |
|---|---|
runai workspace bash WORKLOAD-NAME | Open a bash shellOpens it in the main worker; --pod picks a different pod. |
runai workspace exec WORKLOAD-NAME --tty --stdin -- /bin/bash | Run bash through exec--tty and --stdin, short -t and -i, make the session interactive. |
runai workspace exec WORKLOAD-NAME -- ls | Run a single commandThe command after -- runs in the workload's main worker. |
runai workspace exec WORKLOAD-NAME --pod POD-NAME -- nvidia-smi | Run a command in one podUseful for checking the GPUs of a particular worker in a multi-pod workload. |
runai workspace attach WORKLOAD-NAME --tty --stdin | Attach to the main workerConnects your terminal to the running container; add --pod for another pod. |
runai workspace port-forward WORKLOAD-NAME --port 8080:8090 --address localhost | Forward a local portConnections to localhost:8080 reach port 8090 inside the workload. |
runai workspace port-forward WORKLOAD-NAME --port 8080:8090 --port 6443:443 --address localhost | Forward several portsRepeat --port for each local and remote pair. |
Suspending and deleting 9
| Command | What it does |
|---|---|
runai workspace suspend WORKSPACE-NAME | Suspend a workspace |
runai workspace resume WORKSPACE-NAME | Resume a workspace |
runai training suspend WORKLOAD_NAME | Suspend a training job |
runai training resume WORKLOAD_NAME | Resume a training job |
runai training delete WORKLOAD_NAME | Delete a training job-p picks the project, and --uuid deletes by ID instead of by name. |
runai workspace delete WORKLOAD-NAME | Delete a workspace |
runai workload delete -p proj1 workload01 workload02 workload03 | Delete several workloadsAsks for confirmation first unless -y is given. |
runai workload delete -p proj1 -y workload01 workload02 workload03 | Delete without confirming |
runai workload delete -p proj1 --type training workload01 workload02 workload03 | Delete only training workloads--type accepts training, workspace or inference. |
Configuration and global flags 11
| Command | What it does |
|---|---|
runai config describe | Show the CLI configurationAdd --json or --yaml for structured output. |
runai config generate --json | Print the current configurationThe install guide uses it to check that configuration worked. |
runai config generate -f path/to/config.json | Write a config fileA file name ending in .yaml produces YAML instead. |
runai config set --status-timeout-duration 5s | Change the status timeoutThe cluster status call gives up after 3s by default; values such as 300ms work too. |
--config-file | Use another config fileThe RUNAI_CLI_CONFIG_FILE variable does the same; the default name is config.json. |
--config-path | Use another config folderCan also come from the RUNAI_CLI_CONFIG_PATH environment variable. |
--debug | Turn on debug outputShort form -d; accepted by every runai command. |
--quiet | Show errors onlyShort form -q; everything except error messages is suppressed. |
--verbose | Turn on verbose output |
--help | Show help for a commandShort form -h; works after any command or subcommand. |
--json | Output as JSONThe list commands also take --yaml and --table. |
Completion and diagnostics 5
| Command | What it does |
|---|---|
source <(runai completion bash) | Load Bash completionAdd the line to ~/.bashrc after installing the bash-completion package. |
source <(runai completion zsh) | Load Zsh completionGoes in ~/.zshrc after the compinit line. |
runai.exe completion powershell | Out-String | Invoke-Expression | Load PowerShell completionAdd it to your PowerShell profile on Windows. |
runai diagnostics collect-logs | Collect logs for supportGathers pod logs from the Run:ai and related namespaces into an archive in the current directory. |
runai diagnostics collect-logs --output-dir /tmp/logs | Save the logs archive elsewhereThe archive is written to the current directory unless --output-dir names another. |
No row on this page matches that filter.
Notes
Setup once
runai version confirms that the client is installed. runai login opens the sign-in page in a browser; on a machine without one, runai login user -u USERNAME asks for the password on standard input. Windows users first point the client at the control plane with runai config set --cp-url CONTROL_PLANE_URL; the Linux and Mac installers do that themselves. runai cluster set CLUSTER_NAME and runai project set PROJECT_NAME store defaults, so later commands can leave out -p. With only one cluster connected, the first sign-in already makes it the default.
Submitting and watching
Each workload type has its own command group, runai workspace, runai training and runai inference, with the same verbs: submit, list, describe, logs, bash, exec and port-forward, plus attach, suspend and resume for workspaces and training. Resources are flags on the submit line: --gpu-devices-request 1 (short -g) for whole GPUs, --gpu-portion-request 0.5 for part of one. runai workload list shows all types together, and runai workload describe WORKLOAD_NAME prints a workload's pods and recent events. runai workspace logs WORKLOAD-NAME --follow streams output, and runai workspace bash WORKLOAD-NAME opens a shell in the main worker.
Coming from CLI v1
Scripts written for the first CLI use runai submit, runai list jobs and runai config project. In CLI v2 the workload type comes first: runai training submit, runai workload list and runai project set PROJECT_NAME. NVIDIA supports v1 only on clusters older than version 2.18, and the v2 reference has no counterpart for runai top node or runai delete project; runai node list and runai nodepool list are the nearest listing commands.
Questions
How do I request a GPU when submitting a Run:ai workload?
Add --gpu-devices-request 1 (short form -g) to a submit command for whole GPUs. For a share of one GPU use --gpu-portion-request 0.5, a value between 0 and 1, or ask for an amount of GPU memory with --gpu-memory-request 1G.
Why does runai submit no longer work?
runai submit belongs to CLI v1, which NVIDIA supports only for clusters below version 2.18. CLI v2 names the workload type first: runai training submit for training jobs, runai workspace submit for workspaces and runai inference submit for inference.
How do I open a shell inside a running Run:ai workload?
runai workspace bash WORKLOAD-NAME opens bash in the main worker, and --pod picks another pod. runai workspace exec WORKLOAD-NAME --tty --stdin -- /bin/bash does the same through exec. The training and inference groups have the same bash and exec commands.
How do I set a default project in the Run:ai CLI?
Run runai project set PROJECT_NAME, or runai project to choose from a menu. Commands then use that project whenever -p is left out; runai project list shows the projects available to you.
Sources and checks
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai (CLI Commands Reference) Command groups and global flags
- NVIDIA Run:ai 2.26 documentation (self-hosted): CLI Commands Examples Example command lines: login, submitting, PVCs, distributed jobs, listing
- NVIDIA Run:ai 2.26 documentation (self-hosted): Install and Configure CLI Setup: version check, control plane URL, login, default cluster and project, completion
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai login Browser and SSO sign-in
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai login user Sign-in without a browser
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai login access-key Access-key sign-in
42 more source pages
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai logout Sign-out
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai whoami Current user
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai version Version flags
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai upgrade Manual upgrade
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai config set Configuration values
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai config generate Configuration file
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai config describe Showing the configuration
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai project list Project listing
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai node list Node listing
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai nodepool list Node pool listing
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai workload list Listing workloads of every type
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai workload describe Describing a workload
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai workload delete Deleting several workloads
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai training submit Submit flags: resources, scheduling, image, storage, ports
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai training list Listing training workloads and list output flags
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai training logs Training logs
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai training delete Deleting training workloads
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai training suspend Suspending training workloads
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai training resume Resuming training workloads
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai workspace submit Submitting workspaces
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai workspace list Listing workspaces
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai inference submit Submitting inference workloads
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai inference list Listing inference workloads
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai training pytorch submit Distributed training flags
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai pvc list PVC listing
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai workspace describe Describe options
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai workspace logs Log options
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai workspace bash Opening a shell
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai workspace exec Running commands in a workload
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai workspace attach Attaching to a workload
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai workspace delete Deleting workspaces
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai workspace suspend Suspending workspaces
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai workspace resume Resuming workspaces
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai workspace port-forward Port forwarding
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai inference update Updating inference workloads
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai project create Creating projects
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai template list Template listing
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai environment list Environment asset listing
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai diagnostics collect-logs Collecting diagnostic logs
- NVIDIA Run:ai 2.26 documentation (self-hosted): runai auth get-token Access tokens
- NVIDIA Run:ai 2.26 documentation (self-hosted): Add NVIDIA Run:ai Authorization to Kubeconfig runai kubeconfig set
- NVIDIA Run:ai 2.26 documentation (self-hosted): Auto Update CLI Mechanism Automatic CLI updates
Every row on this page was matched to a line in the documentation above on . A binding that could not be found there is not listed.
2 rows from the earlier version of this page were removed in that check: duplicates, and bindings the documentation does not list or lists differently.
Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.