Submitting jobs 27
| Command | What it does |
|---|---|
sbatch SCRIPT_FILE | Submit a batch scriptReturns as soon as the script is queued and a job ID assigned; output goes to slurm-JOBID.out unless redirected. |
sbatch -N4 myscript | Submit with a node countOptions given on the command line, here four nodes, come before the script name. |
#SBATCH --time=1 | Set an option inside the scriptDirective lines are read only before the first command of the script; this one sets a one-minute limit. |
sbatch -N4 <<EOF | Submit a script from standard inputWith no file name on the command line, sbatch reads the script from standard input, here a shell here-document. |
sbatch -n4 -w "adev[9-10]" -o my.stdout my.script | Submit with tasks, nodes and output filePins the job to nodes adev9 and adev10, sizes its steps at four tasks and writes output to my.stdout. |
sbatch -w node[2-3] : -w node4 : -w node[5-7] work.bash | Submit a heterogeneous jobColons separate the components, each with its own options; this one has three components on different node sets. |
--wrap | Submit a command without a scriptsbatch builds a one-line sh script around the quoted command, so no script file is named. |
--parsable | Print only the job IDPrints the job ID, plus a semicolon and the cluster name when there is one, which suits shell scripting. |
--test-only | Validate without submittingChecks the script and estimates when it would start under the present queue; nothing is queued. |
--output | Set the output fileShort form -o; takes a filename pattern, and without it stdout and stderr both land in slurm-%j.out. |
--error | Set the error fileShort form -e; splits standard error into its own file instead of sharing the output file. |
--input | Set the input fileShort form -i; when it is absent the script gets /dev/null as standard input. |
--open-mode | Append to or truncate output filesTakes append or truncate; the JobFileAppend setting of the cluster decides the default. |
%j | Job ID in a file nameA replacement symbol for the output, error and input patterns; a number after the percent sign zero-pads it, as in job%4j.out. |
%x | Job name in a file nameExpands to the name given with --job-name, or to the default job name. |
%N | Host name in a file nameExpands to the short host name, so each node writes its own file. |
--job-name | Name the jobShort form -J; the name appears beside the job ID in queue listings and defaults to the script name. |
--chdir | Set the working directoryShort form -D; the script starts there, and a relative path is taken from where sbatch was run. |
--mail-type | Email on job eventsValues include BEGIN, END, FAIL, REQUEUE, ALL and TIME_LIMIT_90, combined with commas; no mail is sent by default. |
--mail-user | Set the notification addressAccepts a full address or a user name; the submitting user is the default recipient. |
--begin=now+1hour | Defer the start timeShort form -b; also accepts a clock time such as 16:00, an ISO date-time, or a bare number of seconds after now+. |
--hold | Submit in held stateShort form -H; the job sits at priority zero until scontrol release is run on it. |
--wait | Wait for the job to endShort form -W; sbatch blocks until the job finishes and exits with the exit code of the job. |
--export=EDITOR | Pass chosen environment variablesLimits the exported environment to the SLURM_* variables plus the names listed; NAME=value sets a value, and ALL is the default mode. |
--requeue | Allow the job to be requeuedAfter a node failure or preemption the script restarts from its beginning under the same job ID; --no-requeue forbids that. |
--signal | Signal before the time limitSends the chosen signal ahead of the end time, 60 seconds by default; a B: prefix signals only the batch shell. |
--nice | Lower the scheduling priorityWith no value the priority drops by 100; only privileged users may pass a negative adjustment to raise it. |
Interactive jobs and steps 29
Since Slurm 20.11 steps created by srun are exclusive, so the formerly recommended srun --pty $SHELL no longer works as an interactive shell; SchedMD recommends salloc instead.
| Command | What it does |
|---|---|
salloc -N1024 bash | Start a shell inside an allocationsalloc waits for the allocation, runs the named command and gives the nodes back when that command exits. |
salloc -N16 xterm | Open an xterm with an allocationsrun commands typed in the new terminal run on the 16 allocated nodes; closing it releases them. |
salloc -N5 srun -n10 myprogram | Allocate and launch in one lineGrabs five nodes and starts ten tasks of the program on them as a single command. |
salloc -N4 test.sh | Run a script in a new allocationFrom the srun manual examples: each srun line of the script becomes a job step on the four allocated nodes. |
--no-shell | Allocate without running a commandA salloc option: the job exists and holds its resources with no process attached, ready for srun --jobid. |
srun --pty bash -i | Interactive shell through srunSchedMD calls this the not-recommended route: the shell step takes every resource of the job, so later steps need --overlap. |
--pty | Run task zero in a pseudo terminalAn srun option that also turns on unbuffered I/O and sends the output of every other task to /dev/null. |
srun -n1 --pty --x11 xclock | Run an X11 program on a nodeNeeds Slurm built with X11 support and PrologFlags=x11 set in slurm.conf. |
srun -N3 -l /bin/hostname | Run a command on three nodesCreates the allocation and launches the step in one command; -l prefixes each output line with its task number. |
srun -n4 -l /bin/hostname | Run a command as four tasksNo node count is given, so the four tasks use one processor each wherever they fit. |
srun -N2 -pdebug uptime | Put options before the commandAnything after the executable name is passed to the program, so srun options must come first. |
srun -n4 prog1 & | Run steps in parallel in a scriptEach backgrounded srun is one job step; a final wait keeps the batch script alive until all of them finish. |
srun -n1 -c8 --mem-per-cpu=2gb server : -n16 --mem-per-cpu=1gb client | Launch two programs in one stepA colon separates the components: one server task with eight CPUs and sixteen client tasks with their own memory setting. |
srun -n8 -l --multi-prog silly.conf | Run different programs per taskThe file named last is a configuration file that lists the executable and arguments for each task rank. |
srun -lN2 -r2 hostname | Start a step on later nodesInside an allocation -r2 begins the step at the third node, so two steps can use disjoint nodes. |
--overlap | Share resources between stepsBy default parallel steps do not share CPUs, memory or GRES; with this option a step may overlap every other step. |
--exact | Limit a step to what it requestsWithout it a step gets all non-GRES resources on its nodes; other steps then block or fail unless they overlap. |
--exclusive | Request exclusive resourcesOn srun inside a running job it is a synonym for --exact; when creating a job it asks for whole nodes. |
--label | Prefix output with the task numberShort form -l; applies to the output and error streams of a step. |
--unbuffered | Turn off output bufferingShort form -u; tasks run with a pseudo terminal so their output is not held back by glibc. |
--immediate | Give up if resources are busyShort form -I; exits unless the resources are free within the given seconds, one second when no value is given. |
--jobid | Start a step in an existing jobAn srun option that behaves exactly as if SLURM_JOB_ID were set to that job. |
--kill-on-bad-exit | End the step when a task failsShort form -K; a value of 0 keeps the step running, and without the option the KillOnBadExit setting applies. |
--mpi=pmi2 | Choose the MPI typeSelects the PMI2 plugin for the step; the MPI library itself must implement the PMI2 client side. |
--cpu-bind=none | Disable CPU bindingTurns off the automatic binding that the task/affinity plugin otherwise applies to the tasks. |
sattach 15.0 | Attach to a running job stepConnects the terminal to the input, output and error streams of step 0 of job 15. |
sattach --output-filter 5 65386.15 | Show output from one task onlyThe filtering is done locally by sattach, here for the fifth task of step 15. |
sbcast my.prog /tmp/my.prog | Copy a file to every allocated nodeRun it inside a batch script or allocation; the copy lands on local disk of each node, ready for srun. |
--send-libs | Send shared libraries with a programAn sbcast option: detects the shared objects the executable depends on and places them in a directory beside it. |
Resource requests 26
| Command | What it does |
|---|---|
--nodes | Set the node countShort form -N; one number is both minimum and maximum, and MIN-MAX gives a range. |
--ntasks | Set the number of tasksShort form -n; sbatch launches nothing itself, so this sizes the allocation for the steps that srun starts later. |
--cpus-per-task=3 | Set CPUs per taskShort form -c; tells the controller that every task needs three processors on the same node. |
--ntasks-per-node | Set tasks per nodeMeant to go with --nodes; when --ntasks is also given it becomes a per-node maximum instead. |
--mem | Set memory per nodeA number is read as mebibytes unless it carries a K, M, G or T suffix. |
--mem=0 | Request all memory of a nodeMentioned under --exclusive, which hands the job every CPU and GRES of its nodes but only the memory it requested. |
--mem-per-cpu | Set memory per CPUScales with the number of allocated CPUs; it excludes --mem and --mem-per-gpu on the same job. |
salloc -n3 --mem-per-cpu=100 | Allocate three tasks with 100 MB eachThe manual pairs it with sacct to compare requested and allocated memory when cores have several threads. |
--tmp | Set temporary disk per nodeA minimum amount of scratch space, in mebibytes unless suffixed K, M, G or T. |
--time | Set the time limitShort form -t; written as minutes, hours:minutes:seconds or days-hours, and zero asks for no limit. |
--time-min | Allow a shorter time limitLets the backfill scheduler shorten the job down to this value when that allows an earlier start. |
--partition | Choose the partitionShort form -p; with a comma-separated list the job runs in whichever partition can start it first. |
--account | Charge the job to an accountShort form -A; the account can still be changed after submission with scontrol. |
--qos | Choose a quality of serviceShort form -q; users are held to the QOS set defined for their association in the Slurm database. |
--constraint="intel" | Require a node featureShort form -C; features are labels the administrator assigns to nodes, and only matching nodes are used. |
--nodes=16 --constraint="graphics*4" | Require a feature on some nodesThe count after the asterisk says how many of the sixteen nodes must carry the feature. |
--prefer | Prefer a node featureA soft version of --constraint: the preferred features are tried first and dropped if no such nodes are free. |
--nodelist | Request specific nodesShort form -w; other nodes are added when the listed ones cannot satisfy the node or processor count. |
--exclude | Keep the job off certain nodesShort form -x; takes a list of node names. |
--exclusive | Do not share nodesWith no value no other job runs on the allocated nodes; =user only keeps other users away. |
--oversubscribe | Allow sharing of resourcesShort form -s; the OverSubscribe setting of the partition takes precedence over the job option. |
--licenses=foo:4,bar | Request licensesShort form -L; a colon sets the count, a comma means all are needed and a pipe means any one of them. |
sbatch --reservation=alan_6 -N4 my.script | Run in a reservationThe job runs on nodes held by the named reservation; scontrol show reservation lists the reservations and their names. |
--threads-per-core | Limit threads used per coreSelects nodes with at least that many threads per core and uses at most that many in the task layout. |
--overcommit | Allow more tasks than CPUsShort form -O; resources are allocated as though one task per node had been requested. |
srun --ntasks=4 --nodes=1 a.out | See how a node minimum behavesOn two-processor nodes this gets two nodes, because --nodes is a minimum and each task wants its own processor. |
GPU requests 26
Every --gpu* option needs the select/cons_tres plugin; on a cluster configured without it, jobs that use these options are rejected.
| Command | What it does |
|---|---|
--gres=gpu:2 | Request GPUs per nodeGeneric resources are counted per node and the count defaults to 1; it cannot be combined with --gpus-per-node. |
--gres=gpu:kepler:2 | Request GPUs of one typeThe middle field is a GPU type set up by the administrator, here two kepler cards on every node. |
--gres=help | List the requestable GRESPrints the generic resources available on the cluster and exits without submitting anything. |
--gpus=volta:3 | Request GPUs for the whole jobShort form -G; a total across all nodes with an optional type, and every node of the job must still receive one. |
--gpus-per-node=volta:3 | Request GPUs on each nodeThe GPU-specific spelling of --gres; several types can be listed with commas. |
--gpus-per-task=volta:1 | Request GPUs for each taskNeeds an explicit task count such as --ntasks, not just a range of nodes. |
--gpus-per-socket=volta:3 | Request GPUs on each socketOnly valid when the job also states a sockets-per-node count. |
--cpus-per-gpu | Set CPUs per GPUCannot be used together with --cpus-per-task; steps that inherit the value behave as if --exact were set. |
--mem-per-gpu | Set memory per GPUIn mebibytes unless suffixed K, M, G or T; the default comes from DefMemPerGPU. |
--ntasks-per-gpu | Set tasks per GPUGive either the task count or the GPUs and Slurm works out the other, raising the CPU count when needed. |
--gpu-bind | Bind tasks to GPUsShorthand for --tres-bind=gres/gpu: and takes the same types: closest, map:, mask:, none, per_task: and single:. |
--tres-bind=gres/gpu:verbose,map:0,1,2,3+gres/nic:closest | Bind tasks to listed GPU IDsmap: hands the listed IDs to task ranks in order, and verbose reports the binding that was applied. |
--gres-flags | Set GRES binding flagsenforce-binding limits the job to the CPUs tied to its GPUs in gres.conf, which can delay the start. |
--gpu-freq=medium,memory=high | Set GPU frequencySets the GPU clock and, with memory=, the GPU memory clock; both return to the highest values after the job. |
--tres-per-task=gres/gpu=1 | Request GPUs as a trackable resourceThe general per-task form, written type/name=count; gres/gpu:a100=2 asks for a typed GPU. |
--gres=mps:50 | Request part of a GPU through MPSMPS counts are percentages of a GPU, and a request is met from a single GPU on a node, never split over two. |
--gres=none | Run a step without generic resourcesAn srun setting for a step inside a job; by default a step receives every generic resource the job requested. |
sbatch --gres=gpu:4 -n4 -N1-1 gres_test.bash | Request four GPUs on one nodeThe node range 1-1 fixes the job to exactly one node, and the script then divides the GPUs among its steps. |
srun --gres=gpu:2 -n2 --exclusive show_device.sh & | Give a step two GPUsA step may ask for fewer GPUs than the job holds, so several steps can run side by side on separate devices. |
srun --tres-per-task=gres/gpu:1 -n2 --gpus=2 --mem=2G gpu_burn | Run two tasks with one GPU eachThe guide uses this job to show per-task GPU accounting: two tasks, two GPUs in total. |
sacct -j 1277.0 --format=tresusageinave -p | Show average GPU memory and utilisationLists gres/gpumem and gres/gpuutil per step when gres/gpu is tracked; needs NVML or RSMI autodetection. |
sacct -j 1277.0 --format=tresusageintot -p | Show GPU usage summed over tasksThe Tot variant adds the values of all task ranks in the step instead of averaging them. |
sinfo -O NodeList:10,Gres:20,GresUsed:0 | List configured and allocated GPUsOne line per node; the GRES_USED column shows the allocated count and the index of each GPU in use. |
scontrol show node -d | Show GRES allocation of nodesThe detail flag adds the generic resources currently allocated on each node, shared ones included. |
CUDA_VISIBLE_DEVICES | GPUs visible to a job stepSlurm sets it per step on the node where tasks start; salloc sets no global value. |
CUDA_DEVICE_ORDER=PCI_BUS_ID | Match CUDA numbering to NVMLNVML numbers GPUs by PCI bus ID, so CUDA needs this setting to report them in the same order. |
Job arrays and dependencies 28
| Command | What it does |
|---|---|
sbatch --array=0-31 -N1 tmp | Submit a job arrayCreates 32 tasks from one script with index values 0 to 31; arrays exist only for batch jobs. |
sbatch --array=1,3,5,7 -N1 tmp | Submit listed array indexesA comma list names the exact index values to create. |
sbatch --array=1-7:2 -N1 tmp | Submit an array with a stepThe number after the colon is the step size, giving indexes 1, 3, 5 and 7. |
--array=0-15%4 | Limit running array tasksThe number after the percent sign caps how many tasks of the array run at the same time. |
--array=0,6,16-32 | Mix single indexes and rangesShort form -a; the lowest index is 0 and the highest is one less than MaxArraySize. |
sbatch -o slurm-%A_%a.out --array=1-3 -N1 tmp | Name array output filesGives one file per task, such as slurm-36_1.out; this pattern is also the default for arrays. |
%A | Array job ID in a file nameExpands to the master job ID shared by all tasks of the array. |
%a | Array index in a file nameExpands to the index of the task, the same value as SLURM_ARRAY_TASK_ID. |
SLURM_ARRAY_TASK_ID | Index of the current array taskEach task of the array finds its own index value here, between SLURM_ARRAY_TASK_MIN and SLURM_ARRAY_TASK_MAX. |
SLURM_ARRAY_JOB_ID | Job ID of the whole arrayThe first job ID of the array; with an underscore and the task index it identifies one element, such as 36_1. |
SLURM_ARRAY_TASK_COUNT | Number of tasks in the arrayThe same total in every task of the array. |
squeue -r | List array tasks one per lineBy default pending tasks of an array share one line with an index expression; --array is the long form. |
squeue -j 1234_2,1234_3 | Show chosen array tasksArray elements are written as the array job ID, an underscore and the index. |
scancel 20_[1-3] | Cancel a range of array tasksThe bracket expression selects indexes 1 to 3 of array 20 and leaves the other tasks alone. |
scancel 20_4 20_5 | Cancel two array tasksSeveral task IDs can be listed with spaces between them. |
scancel 20 | Cancel a whole job arrayGiving only the array job ID cancels every element of that array. |
scancel ${SLURM_ARRAY_JOB_ID}_${SLURM_ARRAY_TASK_ID} | Cancel the current array taskFor use inside the batch script, where both variables are set for an array element. |
scontrol update JobID=21845_2 name=arturo | Change one array taskAn update addressed to the array job ID alone would apply to all elements instead. |
--dependency | Make a job wait for othersShort form -d; types are after, afterany (the default), afterok, afternotok, aftercorr and singleton. |
-d afterok:20:21,afterany:23 | Require all dependenciesWith a comma every condition must hold: jobs 20 and 21 exit with code 0 and job 23 ends. |
-d afterok:20:21?afterany:23 | Accept any one dependencyWith a question mark the job is released as soon as one of the conditions is met. |
sbatch --depend=afterok:123 my.job | Run after an array succeedsWaits until every task of array 123 has finished with exit code zero. |
sbatch --depend=afterany:123 my.job | Run after an array finishesSatisfied once all tasks have completed, whatever their exit codes. |
sbatch --depend=afternotok:123 my.job | Run when an array task failsReleased after the whole array has completed with at least one failed task. |
sbatch --depend=aftercorr:123 my.job | Chain matching array tasksEach task of the new array starts after the task with the same index in job 123 has succeeded. |
sbatch --depend=after:123_4 my.job | Run after an array task startsThe after type is met when the named element has started, not when it has finished. |
sbatch --depend=afterok:123_4:123_8 my.job2 | Wait for two array tasksMore job IDs are appended to one dependency type with colons. |
--kill-on-invalid-dep | Cancel on an impossible dependencyTakes yes or no; otherwise such a job normally stays pending with the reason DependencyNeverSatisfied. |
Monitoring the queue 30
| Command | What it does |
|---|---|
squeue | View the job queueShows pending, running and completing jobs; ST holds the state code and the last column the nodes or the reason for waiting. |
squeue -u USER_NAME | View the jobs of one userThe long form is --user, and it also accepts several names or numeric user IDs separated by commas. |
--me | Show only my jobsA squeue option with the same effect as --user plus your own name; --notme lists the jobs of everyone else. |
--states | Filter by job stateShort form -t; accepts state names or all, and squeue shows pending, running and completing jobs when it is absent. |
squeue -p debug -t COMPLETED -o "%.6i %p" | Custom columns for one partitionLists completed jobs of the debug partition as a six-character right-justified job ID followed by the priority. |
squeue -s -p debug -S u | List job steps sorted by userThe -s flag switches the listing from jobs to job steps and -S takes the sort field. |
squeue --jobs 12345,12346,12348 | Show specific jobsListed jobs appear even when they sit in a hidden partition. |
squeue --steps 65552.1 | Show one job stepSteps are named job_id.step_id, with an optional array index before the dot. |
--start | Show expected start timesLists pending jobs ordered by their estimated start, the same as a preset --format with --sort=S and --states=PENDING. |
--long | Show more columnsShort form -l; the long layout spells the state out and adds the time limit of each job. |
--iterate | Refresh the listing repeatedlyShort form -i; reprints at the interval given in seconds, with a time stamp above the header. |
--partition | Filter the queue by partitionShort form -p on squeue; takes one partition name or several separated by commas. |
--nodelist | Show jobs on given nodesShort form -w on squeue; localhost stands for the machine the command runs on. |
--name | Filter by job nameShort form -n; matches jobs or steps whose name is in the comma-separated list. |
--sort | Sort the listingShort form -S; uses the field letters of the output format, and several fields can be listed. |
--format | Choose the output columnsShort form -o; each field is written %[[.]size]type, and the default is "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R". |
--Format | Choose columns by field nameShort form -O; less control over layout than -o, but it reaches every field, including those without a letter. |
%R | Reason or node list columnA squeue format field: the reason in parentheses for a pending job, the allocated nodes for a running one. |
%S | Start time columnA squeue format field that holds the actual start, or the expected one while the job is pending. |
%L | Time left columnA squeue format field: time limit minus time used, printed as days-hours:minutes:seconds. |
%E | Remaining dependencies columnA squeue format field that lists the jobs this one is still waiting for. |
--noheader | Omit the header lineShort form -h; handy when the output is piped into another command. |
--json | Print the queue as JSONDumps all job information and ignores the sorting and formatting options; --yaml does the same in YAML. |
scontrol show job 71701 | Show full details of one jobPrints every field of the job record: state, reason, time limit, node list, CPU and memory figures. |
scontrol show job | Show details of all jobsWith no job ID the record of each job in the system is printed in turn. |
sprio | Show priority factors of pending jobsOne line per pending job with the weighted age, fair-share, job size, partition and QOS components. |
sprio -n | Show normalized priority factorsPrints each factor as a fraction of 1 before the configured weights are applied. |
sprio --jobs=65548,65547 | Show priority of specific jobsTakes numeric job IDs separated by commas; all pending jobs are listed without it. |
sprio --users=fred,sally | Show priority for given usersAdds a USER column and keeps only the pending jobs of the named users. |
sprio -w | Show configured priority weightsPrints the weight of each factor and no job data. |
Node and partition state 20
| Command | What it does |
|---|---|
sinfo | Show partitions and node statesGroups nodes by partition and state; an asterisk after a partition name marks the default partition. |
sinfo -s | Summarise partitionsOne line per partition, with node counts written as allocated/idle/other/total. |
sinfo --long --partition=debug | Detailed view of one partitionAdds job size limits, root-only and oversubscribe settings and the allowed groups to the columns. |
sinfo --states=drained | List nodes in a given stateState names are case insensitive and can be combined with commas; a ~ prefix inverts the match. |
sinfo -Nel | List every node with detailsNode-oriented long listing with CPUs, memory, temporary disk, weight, features and reason; -e keeps unlike nodes on separate lines. |
sinfo -R | Show why nodes are unavailableLists the reason text an administrator recorded for down, drained, failing or fail nodes, with the nodes it applies to. |
--nodes | Limit sinfo to named nodesShort form -n on sinfo; accepts a range expression such as linux[00-17] and makes the query faster on large clusters. |
--dead | Show non-responding nodesShort form -d on sinfo; --responding is the opposite and hides those nodes. |
--reservation | List reservations with sinfoShort form -T; sinfo then ignores most of its node and partition options. |
--all | Include hidden partitionsShort form -a on sinfo, squeue and scontrol; also shows partitions closed to your group. |
%G | GRES column in sinfoA sinfo format field with the generic resources of the nodes, which is where configured GPUs appear. |
%E | Node reason column in sinfoA sinfo format field that explains why a node is down, drained or draining. |
scontrol show node adev1 | Show one node in fullPrints state, CPU and memory figures, features and, for an unavailable node, the recorded reason. |
scontrol show nodes | Show all nodes in fullThe workload-manager comparison sheet lists it beside sinfo -N as the node list command. |
scontrol show partition | Show partition configurationPrints node and CPU totals, MaxTime, state and the node list of every partition. |
scontrol show reservation | Show reservations in fullPrints start and end time, nodes, users and flags of each reservation; res is accepted as an abbreviation. |
scontrol show hostnames tux[1-3] | Expand a node rangeWrites one host name per line; with no argument it expands SLURM_JOB_NODELIST. |
scontrol show config | grep SchedulerType | Look up a configuration valueshow config prints the running configuration, filtered here for the scheduler plugin in use. |
scontrol show slurmd | Check the slurmd on a nodeRun it on the node in question; the last slurmctld message time shows whether the controller reaches that daemon. |
scontrol ping | Check that the controller respondsReports whether the primary and backup slurmctld daemons answer. |
Accounting and efficiency 29
| Command | What it does |
|---|---|
sacct | List accounting recordsWith no options it shows your own jobs and steps since midnight, with state and exit code. |
sacct -j $SLURM_JOB_ID -X -o jobid%7,reqtres%35,alloctres%35 | Compare requested and allocated TRESOne line for the job itself; the number after each percent sign sets the column width. |
--jobs | Select jobs or stepsShort form -j on sacct; takes a comma-separated list of job or job.step IDs, and the step ID batch selects the batch script. |
sacct --brief | Brief accounting listingPrints only the job ID, state and exit code columns; -b is the short form. |
sacct --allocations | One line per job without stepsShort form -X; step lines are dropped, so utilisation figures for the allocation read as zero. |
sacct --format=jobid,elapsed,ncpus,ntasks,state | Choose accounting columnsFields print in the order given, and a %NUMBER after a field name sets how many characters it gets. |
--helpformat | List the available fieldsShort form -e; prints every field name that the format option of sacct or sstat accepts. |
sacct -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,start,end,state | Jobs within a time windowShows jobs that were eligible between the start and end time, with their real start and end. |
sacct -T -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,jobname,user,start,end,state | Clip job times to the windowWith -T a job that began before the window is shown as starting at the window start, which gives exact run time inside a period. |
sacct --state=pending -S11:20:00 -E11:25:00 -X -ojobid,start,end,state | Jobs in a state during a windowSelects jobs that were pending in that period; the State column still prints what the job is now. |
--starttime | Set the start of the periodShort form -S on sacct; defaults to midnight today and accepts dates, clock times, today, or offsets such as now-1hours. |
--endtime | Set the end of the periodShort form -E on sacct; takes the same time formats as the start time. |
--user | Select accounting records by userShort form -u on sacct, also spelled --uid; without it only the jobs of the user running sacct are listed. |
--allusers | Show jobs of all usersShort form -a on sacct; has effect for root, or when PrivateData does not hide job records. |
--parsable2 | Pipe-delimited outputShort form -P; fields are separated by | with none at the line end, and --delimiter swaps the character. |
--batch-script | Show the script of a jobShort form -B on sacct; needs -j and a cluster that stores scripts through AccountingStoreFlags=job_script. |
MaxRSS | Peak memory fieldA sacct field with the largest resident set size among the tasks; compare it with ReqMem. |
ReqMem | Requested memory fieldA sacct field taken from the job allocation, not from the steps; a letter suffix gives the unit. |
Elapsed | Wall-clock time fieldA sacct field printed as [days-]hours:minutes:seconds. |
TotalCPU | CPU time used fieldA sacct field adding user and system CPU time; set against CPUTime it shows how busy the allocated cores were. |
CPUTime | Allocated CPU time fieldA sacct field equal to the elapsed time multiplied by the number of CPUs. |
ExitCode | Exit code fieldA sacct field written code:signal, where the second number is the signal that ended the process, if any. |
sstat --format=AveCPU,AvePages,AveRSS,AveVMSize,JobID -j 11 | Usage of a running job stepWorks while the job runs; with a bare job ID only the lowest-numbered regular step is reported. |
sstat -p --format=AveCPU,AvePages,AveRSS,AveVMSize,JobID -j 11 | Running step usage, parsableThe -p flag separates the values with | and ends each line with one. |
--allsteps | Report every step of a jobShort form -a on sstat, used when the job ID is given without a step. |
--pidformat | List process IDs of each stepShort form -i on sstat; a preset layout with the job ID, nodes and PIDs. |
sshare | Show fair-share usageLists shares, usage and the FairShare factor by account and user; it applies only with the priority/multifactor plugin. |
sshare -A ACCOUNT | Fair-share of one accountShows the users in that account; several account names can be given with commas. |
sshare -l | Long fair-share listingAdds the normalized usage column, and Level FS when the Fair Tree algorithm is in use. |
Modifying and cancelling jobs 23
| Command | What it does |
|---|---|
scancel 1234 | Cancel a jobEnds the job and every step that belongs to it. |
scancel --state=PENDING --user=bob --partition=debug | Cancel jobs by filterNo job ID is needed: every pending job of that user in that partition is cancelled. |
--me | Restrict scancel to my jobsA scancel filter: only jobs that belong to the user running the command are touched. |
scancel --signal=TERM 1234.1 1234.3 | Signal chosen job stepsDelivers the TERM signal to the two named steps only; without --signal the listed job or step is terminated. |
scancel --signal=KILL 1235 | Kill all steps but keep the jobSIGKILL goes to every step of job 1235 while the allocation itself stays. |
scancel --signal=USR1 --batch 1236 | Signal only the batch shellFor scripts that trap a signal: only the batch step receives SIGUSR1, not the other steps. |
--full | Signal the script and its childrenShort form -f on scancel; normally only SIGKILL reaches the batch step, and this adds the shell script and its child processes. |
--interactive | Confirm each cancellationShort form -i on scancel; asks about every job or step before acting. |
scontrol hold JOBID | Hold a pending jobSets the priority to zero so the job cannot start; a hold placed by an administrator needs one to lift it. |
scontrol release JOBID | Release a held jobHold followed by release also hands a manually set priority back to the multifactor plugin. |
scontrol update JobId=1234 StartTime=now+30days | Postpone a jobMoves the earliest start 30 days ahead, which the FAQ gives as the easy way to park a job. |
scontrol update JobId=1234 StartTime=now | Let a postponed job startResets the earliest start time so the scheduler can consider the job again. |
scontrol update JobId=71701 TimeLimit=30:00 Priority=500 | Change time limit and prioritySeveral fields go in one update, named as show job prints them; a + or - before the time adjusts the present limit. |
scontrol update JobId=$SLURM_JOB_ID NumNodes=2 | Shrink a running jobA started job can only get smaller; steps on the nodes given up are killed unless launched with --no-kill. |
scontrol requeue 10 | Requeue a finished or failed jobPuts the batch job back into the pending state so it is scheduled and run again. |
scontrol requeuehold 10 | Requeue a job in held stateAs requeue, but the job waits at priority zero until it is released. |
scontrol requeuehold State=SpecialExit 10 | Requeue and flag a jobMarks the held job SPECIAL_EXIT, shown as SE by squeue, to single out a particular failure. |
scontrol suspend 21845 | Suspend a running jobProcesses are stopped with SIGSTOP and the suspended time is not counted against the time limit. |
scontrol resume 21845 | Resume a suspended jobThe CPUs were free for other jobs meanwhile, so resuming can leave two jobs on the same cores. |
squeue -u adam -h -o "scontrol hold %i" >hold_script | Build a hold script for a userscontrol cannot filter by user or partition, so squeue prints one hold command per job into a file. |
squeue -tpd -h -o "scontrol update jobid=%i priority=1000" >my.script | Build an update scriptOne scontrol command changes one job, so the same technique writes an update line per pending job. |
scontrol write batch_script | Save the script of a jobFollowed by the job ID; writes slurm-JOBID.sh, or standard output when - is given as the file name. |
scontrol notify | Send a message to a jobFollowed by the job ID and the text, which appears on standard error of its srun, salloc or batch job. |
Administration with scontrol 19
| Command | What it does |
|---|---|
scontrol | Start an interactive sessionWith no command scontrol shows a prompt and runs each line typed, as in the manual page examples, until quit. |
scontrol update nodename='%N' state=drain reason='removing nodes' | Drain a nodeNo new job starts on the node while running jobs finish; replace %N with the node name. |
scontrol update NodeName=whatever State=RESUME | Return a node to serviceNeeded for a DOWN node when ReturnToService is zero, the default, even after its slurmd has registered again. |
sinfo -t drain -h -o "scontrol update nodename='%N' state=drain reason='%E'" | Save drain states as commandsPrints one restore command per drained node with its reason, to replay after a cold start of a major update. |
scontrol cancel_reboot NODE | Cancel a pending node rebootHas the same effect as setting the node state to CANCEL_REBOOT. |
scontrol update PartitionName=foo State=DOWN | Close a partitionA partition in the DOWN state keeps its queued jobs but starts none of them. |
scontrol update PartitionName=bar State=UP | Reopen a partitionLets jobs be queued and started in the partition again. |
scontrol update PartitionName=debug MaxTime=60:00 MaxNodes=4 | Change partition limitsThe manual page example: caps jobs in the debug partition at 60 minutes and four nodes. |
scontrol delete partition | Remove a partitionWritten scontrol delete PartitionName=NAME; refused while jobs use the partition, and it must also leave slurm.conf. |
scontrol create reservation starttime=2009-02-06T16:00:00 duration=120 user=root flags=maint,ignore_jobs nodes=ALL | Reserve all nodes for maintenanceA two-hour window on every node; ignore_jobs lets it be created although jobs are still running. |
scontrol create reservation user=alan,brenda starttime=noon duration=60 flags=daily nodecnt=10 | Create a daily reservationTen nodes for two users for an hour at noon, repeated every day. |
scontrol create reservation StartTime=now Duration=60 NodeCnt=1 CoreCnt=2 User=alan | Reserve cores instead of nodesReserves two cores rather than a whole node; core reservations are not available with the select/linear plugin. |
scontrol create reservation starttime=2009-04-06T16:00:00 duration=120 user=root flags=license_only licenses=lustre:1000 | Reserve licenses onlyHolds 1000 lustre licenses for two hours, so jobs that need one are not scheduled in that window. |
scontrol update ReservationName=root_3 duration=150 users=admin | Modify a reservationChanges the length and the users allowed in; the reply is Reservation updated. |
scontrol delete ReservationName=alan_6 | Delete a reservationRefused while jobs are running in it; reservations are also purged automatically after their end time. |
scontrol reconfigure | Re-read the configurationTells slurmctld and the slurmd daemons to load slurm.conf again without interrupting running jobs. |
scontrol shutdown | Stop the Slurm daemonsDaemons save their state and exit; the option slurmctld stops only the controller and leaves slurmd running. |
scontrol takeover | Switch to a backup controllerMakes a backup slurmctld request control from the primary and become the active controller. |
scontrol show assoc_mgr | Show cached limits and usagePrints the controller cache of users, associations and QOS; flags=qos or accounts=NAME narrows it. |
Administration with sacctmgr 29
| Command | What it does |
|---|---|
sacctmgr list cluster | List registered clustersA quick check that the cluster exists in the accounting database before accounts are added. |
sacctmgr add cluster snowflake | Register a clusterClusters are defined before accounts and accounts before users; since 20.02 slurmctld adds its own cluster at start-up. |
sacctmgr add account none,test Cluster=snowflake Description="none" Organization="none" | Add accounts to a clusterCommas add several accounts at once, and leaving Cluster= out adds them on every cluster. |
sacctmgr add account chemistry,physics parent=science Description="physical sciences" Organization=science | Add sub-accounts under a parentparent= places the new accounts below an existing one in the account tree. |
sacctmgr add user brian Account=physics | Add a user to an accountCreates the association that lets the user run jobs charged to that account. |
sacctmgr add user da DefaultAccount=test | Add a user with a default accountThe default account is charged whenever a job is submitted without --account. |
sacctmgr create account name=science fairshare=50 | Create an account with sharescreate is identical to add; fairshare sets the shares the account gets in priority calculations. |
sacctmgr create user name=adam cluster=tux account=physics fairshare=10 | Create a user associationTies the user to one account on one cluster with a share value of its own. |
sacctmgr delete user name=adam cluster=tux account=physics | Remove a user from an accountDeletes only that association; to move a user, add the new account first and then remove the old one. |
sacctmgr delete account name=physics cluster=tux | Delete an accountLimited here to one cluster by the cluster= condition. |
sacctmgr modify user where name=adam cluster=tux account=physics set maxjobs=2 maxwall=30:00 | Limit a user's jobs and wall timeThe where part picks the association and the set part writes the limits: two jobs, 30 minutes each. |
sacctmgr modify user name=adam set fairshare=10 where cluster=tux | Add a condition after setWords before set select the records; a condition placed after set must be introduced by where. |
sacctmgr modify user name=adam set qos+=expedite | Add a QOS to a userThe += operator appends to the QOS list and -= removes from it, while a plain = replaces the list. |
sacctmgr modify account name=G set qos=expedite | Set the only QOS of an accountWith = nothing else stays usable, and accounts later added below it inherit that single QOS. |
sacctmgr show assoc format=cluster,account,qos | Show QOS per associationformat= keeps the listing to the named columns. |
sacctmgr list associations cluster=tux format=Account,Cluster,User,Fairshare tree withd | List associations as a treetree indents account names by hierarchy and withd includes deleted records. |
sacctmgr show qos format=name,priority | List QOS names and prioritiesThe unfiltered show qos output is very wide, so the guide always picks columns. |
sacctmgr add qos zebra | Create a QOSThe listing that follows shows the new QOS with priority 0. |
sacctmgr modify qos zebra set priority=10 | Set the priority of a QOSAffects scheduling only with the multifactor priority plugin and a non-zero PriorityWeightQOS. |
sacctmgr modify qos zebra set GrpTRES=cpu=24 | Cap CPUs for a QOSA group limit: the total held by all running jobs of the QOS together. |
sacctmgr modify user crock set qos=zebra | Give a user one QOSReplaces whatever QOS list the user had; qos+=NAME would add to it. |
sacctmgr delete qos alligator | Delete a QOS |
sacctmgr modify user bob set GrpTRES=cpu=1500,mem=200,gres/gpu=50 | Cap a user's CPUs, memory and GPUsEach TRES must be named in the value; gres/gpu limits the number of GPUs. |
sacctmgr modify user bob set GrpTRES=cpu=-1,mem=-1,gres/gpu=-1 | Clear TRES limits of a userA value of -1 removes the limit for that resource. |
sacctmgr show tres | List trackable resourcesShows the TRES types and names, such as gres/gpu, that limits and --tres-per-task accept. |
sacctmgr list transactions Action="Add Users" Start=11/03-10:30:00 format=Where,Time | Audit accounting changesLists what was changed in the database since the given time, here only the additions of users. |
sacctmgr dump cluster=tux file=tux_data_file | Dump cluster associations to a fileWrites a flat file that sacctmgr load can read back; needs administrator or root rights. |
sacctmgr load tux_data_file | Load associations from a dumpReads a file made by sacctmgr dump; archive data is loaded with archive load instead. |
--immediate | Commit sacctmgr changes at onceShort form -i on sacctmgr, which otherwise asks before it commits a change. |
Environment variables in job scripts 32
Input variables such as SBATCH_PARTITION override options set in the batch script, and command line options override the variables.
| Command | What it does |
|---|---|
SLURM_JOB_ID | ID of the running jobSet by the controller in the batch script; handy for naming scratch directories and for sacct or scancel calls. |
SLURM_JOB_NAME | Name of the jobHolds the value of --job-name, or the script name when none was given. |
SLURM_JOB_NODELIST | Nodes allocated to the jobA compact range expression such as dev[7-10]; scontrol show hostnames expands it to one name per line. |
SLURM_JOB_NUM_NODES | Number of allocated nodesThe older name SLURM_NNODES carries the same number for backwards compatibility. |
SLURM_JOB_PARTITION | Partition the job runs inUseful when a job was offered several partitions and the script needs to know which one it got. |
SLURM_JOB_ACCOUNT | Account charged for the job |
SLURM_JOB_QOS | QOS of the job |
SLURM_SUBMIT_DIR | Directory sbatch was run fromThe batch script starts in this directory unless --chdir names another one. |
SLURM_SUBMIT_HOST | Host the job was submitted from |
SLURM_NTASKS | Number of tasks requestedOnly present when --ntasks or a --ntasks-per-* option was given; srun inside the script reads it as its own task count. |
SLURM_CPUS_PER_TASK | CPUs requested per taskExists only when --cpus-per-task or --tres-per-task=cpu=# was given, so a script should not assume it is there. |
SLURM_CPUS_ON_NODE | CPUs allocated on this nodeCounts the CPUs of the batch step on the current node; with select/linear it is the whole node. |
SLURM_JOB_CPUS_PER_NODE | CPU count for each nodeWritten compactly, so 72(x2),36 means 72 CPUs on the first two nodes and 36 on the third. |
SLURM_TASKS_PER_NODE | Task count for each nodeSame node order as SLURM_JOB_NODELIST; 2(x3),1 means two tasks on each of three nodes, then one. |
SLURM_MEM_PER_NODE | Memory requested per nodeMirrors --mem; SLURM_MEM_PER_CPU and SLURM_MEM_PER_GPU exist for the other two memory options. |
SLURM_GPUS | Number of GPUs requestedPresent only when the job used -G or --gpus; the per-node form sets SLURM_GPUS_PER_NODE instead. |
SLURM_GPUS_ON_NODE | GPUs allocated on this nodeCounts the devices given to the batch step on the node where the script runs. |
SLURM_JOB_GPUS | Global IDs of the allocated GPUsUnlike CUDA_VISIBLE_DEVICES the numbers are not relative to a device cgroup; set in batch and interactive jobs only. |
SLURM_TRES_PER_TASK | Per-task resources requestedCarries --tres-per-task, and also --cpus-per-task and --gpus-per-task rewritten in that form. |
SLURM_PROCID | Rank of the current taskThe MPI rank, or relative process ID, of the process that reads it. |
SLURM_NODEID | Index of the current nodeStarts at 0, which lets one script do different work on each node of an srun. |
SLURM_STEP_ID | Step ID within the jobSet by srun for the tasks it launches, next to SLURM_STEP_NODELIST and SLURM_STEP_NUM_TASKS. |
SLURMD_NODENAME | Node running the batch script |
SLURM_RESTART_COUNT | Times the job was restartedPresent after an explicit requeue or a restart caused by a system failure. |
SLURM_JOB_END_TIME | Projected end of the jobA UNIX timestamp; SLURM_JOB_START_TIME holds the start in the same form. |
SLURM_JOB_DEPENDENCY | Dependency list of the job |
SLURM_CLUSTER_NAME | Cluster the job runs on |
SBATCH_PARTITION | Default partition for sbatchAn input variable: it overrides #SBATCH lines in the script and is itself overridden by the command line. |
SBATCH_ACCOUNT | Default account for sbatchRead by sbatch at start-up with the same meaning as -A. |
SBATCH_TIMELIMIT | Default time limit for sbatchRead by sbatch at start-up with the same meaning as -t. |
SQUEUE_FORMAT | Default squeue columnsTakes the same format string as -o, so a preferred layout applies to every squeue call. |
SACCT_FORMAT | Default sacct columnsTakes the comma-separated field list that --format would take. |
The SchedMD documentation site states that its pages describe Slurm 26.05; older releases are documented in the archive and with the source.
No row on this page matches that filter.
Notes
Getting work onto the cluster
sbatch SCRIPT_FILE submits a batch script whose #SBATCH lines request nodes, GPUs, time and a partition; it returns a job ID immediately and the job runs when resources free up. Directives are read only up to the first command of the script, and a line such as #SBATCH --time=1 loses to the same option given on the command line. For interactive work SchedMD recommends salloc: salloc -N1024 bash holds an allocation while the shell runs, and several srun steps can share it. srun --pty bash -i still works, but its step takes every resource of the job, so later steps need --overlap. sbcast my.prog /tmp/my.prog copies a file to local disk on every node of an allocation.
Asking for GPUs
--gres=gpu:2 asks for two GPUs on each node and --gres=gpu:kepler:2 names a type; --gpus=volta:3 is a total for the whole job, and --gpus-per-task=volta:1 needs an explicit task count. The --gpus family only works with the select/cons_tres plugin, and --gres=gpu cannot be mixed with --gpus-per-node in one request. Typed and untyped requests must also match between a job and its steps. Size the rest of the job from the GPU count with --cpus-per-gpu and --mem-per-gpu, and see what the nodes offer with sinfo -O NodeList:10,Gres:20,GresUsed:0. Inside the job, CUDA_VISIBLE_DEVICES tells each step which devices it may use.
Watching and stopping
squeue shows the whole queue and --me only your jobs, with state (PD pending, R running) and, in the last column, the reason a pending job waits. --start lists expected start times and sprio breaks the priority of pending jobs into its factors. scontrol show job 71701 prints everything about one job, including the exact nodes. scancel 1234 kills one job, scancel --state=PENDING --user=bob --partition=debug cancels by filter, and sattach 15.0 connects to the I/O of a running step.
Cluster state and accounting
sinfo shows partitions and node states and sinfo -Nel lists nodes individually with CPUs, memory and features; sinfo -R gives the reason a node is down or drained, and scontrol show node adev1 shows one node in full. sacct lists the jobs of the current day; add --jobs for one job, or a field list as in sacct --format=jobid,elapsed,ncpus,ntasks,state to choose columns. Setting MaxRSS against ReqMem, and TotalCPU against CPUTime, shows whether a job used the memory and cores it asked for. sstat --format=AveCPU,AvePages,AveRSS,AveVMSize,JobID -j 11 reports usage while a step is still running, and sshare shows fair-share usage.
Questions
Where do Slurm options go: on the command line or in the script?
Type the commands in a terminal on the cluster. Options go on the command line or, for batch jobs, in #SBATCH lines at the top of the script; when both set the same option, the command line wins.
What is the Slurm command to submit a batch job?
sbatch SCRIPT_FILE. sbatch returns as soon as the script is queued and prints the job ID; the script later runs on one of the allocated nodes, and its output goes to slurm-%j.out unless --output names another file.
How do I request a GPU in Slurm?
Add --gres=gpu:2 for GPUs on each node, or --gpus=volta:3 for a total across the job with an optional type, to sbatch, srun or salloc. The --gpus options require the select/cons_tres plugin. Slurm sets CUDA_VISIBLE_DEVICES for each job step.
How do I get an interactive shell on a compute node?
SchedMD recommends salloc: on a cluster configured with LaunchParameters=use_interactive_step, salloc run without a command opens a shell on a node of the allocation. srun --pty bash -i also works, but the FAQ calls it not recommended because that step consumes all resources of the job.
Why is my Slurm job still pending?
squeue prints the reason in its last column, for example Resources or Priority. --start shows the expected start time of pending jobs, and sprio lists the priority factors of each one.
Sources and checks
- SchedMD: sbatch manual page (Slurm 26.05) Batch submission options, #SBATCH directives, job environment variables
- SchedMD: srun manual page (Slurm 26.05) Job steps and interactive runs
- SchedMD: salloc manual page (Slurm 26.05) Interactive allocations
- SchedMD: squeue manual page (Slurm 26.05) Queue monitoring
- SchedMD: scancel manual page (Slurm 26.05) Cancelling and signalling jobs
- SchedMD: sinfo manual page (Slurm 26.05) Node and partition state
19 more source pages
- SchedMD: sacct manual page (Slurm 26.05) Job accounting
- SchedMD: scontrol manual page (Slurm 26.05) Viewing and changing jobs, nodes, partitions and reservations
- SchedMD: sacctmgr manual page (Slurm 26.05) Accounts, users, QOS and limits
- SchedMD: sstat manual page (Slurm 26.05) Status of running steps
- SchedMD: sprio manual page (Slurm 26.05) Priority factors
- SchedMD: sshare manual page (Slurm 26.05) Fair-share information
- SchedMD: sattach manual page (Slurm 26.05) Attaching to a job step
- SchedMD: sbcast manual page (Slurm 26.05) Copying files to allocated nodes
- SchedMD: Quick Start User Guide Worked examples of the main commands
- SchedMD: Generic Resource (GRES) Scheduling GPU request options and examples
- SchedMD: Job Array Support Job arrays and array dependencies
- SchedMD: Accounting and Resource Limits sacctmgr examples
- SchedMD: Slurm documentation index Version of the documentation set
- SchedMD: Rosetta Stone of Workload Managers (PDF)
- SchedMD: Frequently Asked Questions Interactive shells, holding and requeueing jobs
- SchedMD: Slurm Troubleshooting Guide Controller and node checks
- SchedMD: Advanced Resource Reservation Guide Reservation examples
- SchedMD: Quality of Service (QOS) QOS examples
- SchedMD: Resource Limits Limit examples
Every row on this page was matched to a line in the documentation above on . A binding that could not be found there is not listed.
1 row from the earlier version of this page was removed in that check: duplicates, and bindings the documentation does not list or lists differently.
Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.