Launching 14
| Command | What it does |
|---|---|
mpirun -n 6 mpi-hello-world | Run N processesStarts six copies of the executable on the local machine; without -n, one process per processor core is launched. |
mpiexec -n 2 mpi-hello-world | Launch with mpiexecmpiexec and mpirun are the same executable under two names, so every option works with either. |
mpirun -n 2 a.out : -n 2 b.out | Launch two programs as one jobBoth executables share one MPI_COMM_WORLD: a.out gets ranks 0 and 1, b.out ranks 2 and 3. |
mpirun --app my_appfile | Launch from an application fileThe file lists one sub-application per line, such as -n 2 a.out, and may contain # comments. |
mpirun -H aa -n 1 hostname : -H bb,cc -n 2 uptime | Different programs on different hostsProcess 0 runs hostname on aa while processes 1 and 2 run uptime on bb and cc. |
mpirun -n 2 --host a,b uptime | Run a non-MPI commandmpirun also starts ordinary programs; here the Unix uptime command runs on hosts a and b. |
-np | Older spelling of -nSets how many copies to run; the manual accepts -np, --n and -c but prefers -n. |
--help | Show usage and option summaryOpen MPI's launch guide treats this output as the definitive list of options. |
-V | Print the versionPrints the Open MPI version and exits; --version is the long form. |
--verbose | Verbose launcher outputShort form -v; turns on additional messages from mpirun itself. |
--show-progress | Report launch progressPrints a short status at intervals while daemons and processes start, mainly useful on large systems. |
--allow-run-as-root | Allow rootPermit mpirun to launch as the root user; discouraged outside containers. |
OMPI_ALLOW_RUN_AS_ROOT=1 | Allow root from the environmentOnly takes effect when the companion CONFIRM variable is set as well. |
OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 | Confirm running as rootSecond variable of the pair that replaces --allow-run-as-root, added for container setups where root is the default user. |
Hosts and slots 19
| Command | What it does |
|---|---|
--hostfile FILENAME | Host fileNames a file with one node per line and an optional slots=N count. |
--machinefile FILENAME | Hostfile under its other nameAccepted for compatibility with other MPI implementations; behaves exactly like --hostfile. |
--default-hostfile FILENAME | Set a default hostfileLowest-priority host list: --hostfile and then --host filter the nodes it names. |
mpirun --hostfile my-hostfile.txt mpi-hello-world | Launch across a hostfileWith no -n, one process starts per slot: 44 in the example, where two nodes default to their 16 cores. |
node3.example.com slots=2 | Hostfile line with a slot countslots is the most processes Open MPI will place on that node; left out, the node's core count is used. |
aa slots=4 max_slots=4 | Hostfile line with a hard limitmax_slots stops this job from oversubscribing the node even when -n asks for more processes. |
-H host1,host2 | Host listComma-delimited list of hosts to launch processes on directly. |
--host node01:5,node02 | Hosts with slot countsA :N suffix sets the slots for that entry; without it the node gets one slot. |
mpirun --host a,b:2,c:3 hostname | Start a set number per hostLaunches one copy on a, two on b and three on c. |
mpirun -H aa,aa,bb ./a.out | Repeat a host for more slotsNaming aa twice puts two processes there and one on bb. |
mpirun --hostfile myhostfile --host aa ./a.out | Filter a hostfile by hostOnly nodes named by --host are used; naming a host that is not in the file makes mpirun abort. |
mpirun --hostfile myhostfile -n 6 ./a.out | Use only part of the slotsWith three 4-slot nodes, ranks 0-3 land on aa and 4-5 on bb; cc is left unused. |
mpirun -n 1 --hostfile hostfile_1 hostname : -n 1 --hostfile hostfile_2 uptime | Separate hostfile per programEach application context of a colon-separated command line can name its own hostfile. |
mpirun -n 4 --hostfile my-hostfile --map-by :OVERSUBSCRIBE a.out | Run more processes than slotsFour processes on a host declared with two slots fail unless this qualifier is added. |
--oversubscribe | Allow oversubscriptionAllow launching more processes than available CPU slots. |
--nooversubscribe | Forbid oversubscriptionThe default behaviour: mpirun stops with an error before starting anything if the request exceeds the slots. |
--nolocal | Keep processes off the launch nodeAlias for --map-by :nolocal; it wins even when the local host is listed with --host. |
--use-hwthread-cpus | Count hardware threads as CPUsChanges the default slot count of a node and makes a processing element a hardware thread, not a core. |
--hetero-nodes | Assume differing node topologiesTells the launcher not to assume that every allocated node has the same hardware layout. |
Mapping processes 24
| Command | What it does |
|---|---|
--map-by VALUE | Set the mapping policyDirectives include slot, hwthread, core, numa, package, node, seq, ppr, rankfile and pe-list, each with optional colon-separated qualifiers. |
--map-by node | Map round-robin by nodeDistribute processes round-robin across nodes instead of filling one node first. |
--map-by slot | Fill each node in turnA node's slots are used up before the next node gets a process, so neighbouring ranks share a node. |
--map-by core | Map one process per coreSame as setting the PRRTE parameter rmaps_default_mapping_policy to core. |
--map-by package | Map round-robin by packageHands processes to processor packages in turn; socket is accepted as an alias for package. |
--map-by numa | Map round-robin by NUMA regionPlaces one process per NUMA region in turn until the node's slots are used. |
mpirun --np 4 --map-by hwthread --host hostA:4,hostB:2 ./a.out | Map to hardware threadsWith only --map-by given, binding follows the same object: each rank is pinned to one hardware thread. |
--map-by ppr:N:OBJ | Processes per resourceGeneral form: N processes on every object of the named type, such as node or package. |
mpirun -H aa,bb --map-by ppr:2:package ./a.out | Two processes per packageIn the manual's example ranks 0-3 start on aa and ranks 4-7 on bb. |
mpirun -H aa,bb --map-by ppr:2:node ./a.out | Two processes per nodeRanks 0 and 1 start on aa, ranks 2 and 3 on bb. |
mpirun -H aa,bb --map-by ppr:1:node ./a.out | One process per nodeGives the same layout as the deprecated --pernode and --npernode 1 options. |
-N NUM | Processes per nodeShort form of --npernode, which the manual marks as deprecated in favour of --map-by ppr:n:node. |
mpirun --np 4 --report-bindings --map-by slot:PE=2 --bind-to core | Two cores per processThe PE=2 qualifier gives each rank two processing elements, and the rank is bound to both cores. |
mpirun --np 4 --report-bindings --map-by slot:PE=2 --use-hwthread-cpus | Two hardware threads per processHere a processing element is a hardware thread, so each rank is bound to two of them. |
--map-by pe-list=0,2,5 | Map to listed coresAll three processes get the same binding: the logical cores named in the list. |
--map-by pe-list=0,2,5:ordered | One listed core per processProcesses take the listed CPUs in order: 0 for the first one on each node, then 2, then 5. |
mpirun -hostfile myhostfile --map-by seq ./a.out | Map in hostfile orderSequential mapper: one process per hostfile line in file order, whatever the slot counts say. |
mpirun -H aa,bb,cc,dd --map-by rankfile:file=myrankfile ./a.out | Place ranks from a rankfileEach line of the file fixes one rank's host and the logical package and cores it is bound to. |
rank 0=aa slot=1:0-2 | Rankfile linePuts rank 0 on node aa, bound to cores 0 to 2 of logical package 1. |
rank 0=+n0 slot=1:0-2 | Rankfile line with a relative host+n0 is the first host of the list supplied by --host, a hostfile or the scheduler. |
--map-by rankfile:file=myrankfile:hwtcpus | Rankfile slots as hardware threadsWithout this qualifier the locations in a rankfile are read as cores. |
--rankfile FILENAME | Name a rankfile directlyListed among the launch options; the manual's older option text prefers --map-by rankfile:file=FILE. |
--runtime-options donotlaunch | Map without launchingThe job is mapped but not started, a way to explore placement patterns before a real run. |
--help placement | Explain mapping, ranking and bindingPrints the launcher's own longer description of the three placement phases. |
Binding and ranking 19
mpirun binds processes even when no binding option is given: to a core for one or two processes, to a package for more, and not at all on an oversubscribed node.
| Command | What it does |
|---|---|
--bind-to VALUE | Set the binding policyDirectives are none, hwthread, core, l1cache to l3cache, numa and package; qualifiers such as overload follow a colon. |
--bind-to core | CPU bindingBind each process to a specific CPU core for consistent performance. |
--bind-to package | Bind to a whole packageA rank may then run on any core of the package it was mapped to. |
--bind-to none | Turn binding offWhat the manual suggests for threaded applications that are not bound to several cores. |
--bind-to l3cache | Bind to an L3 cacheThe rank may use every processor sharing one L3 cache inside the package it was mapped to. |
--bind-to hwthread | Bind to one hardware threadNeeds hardware threads to count as CPUs, which --use-hwthread-cpus or the hwtcpus mapping qualifier arranges. |
--bind-to numa | Bind to a NUMA regionThe rank may use all CPUs of the NUMA region it was mapped to. |
--bind-to :overload-allowed | Allow overlapping bindingsLets ranks share a processing unit; otherwise overlapping rankfile bindings are an error. |
mpirun --np 4 --report-bindings --map-by core --bind-to core | Bind ranks to successive coresThe report shows rank 0 on core 0, rank 1 on core 1, and so on inside package 0. |
mpirun --np 4 --report-bindings --map-by package --bind-to package | Map and bind by packageRanks 0 and 1 get all cores of package 0, ranks 2 and 3 those of package 1. |
mpirun --np 4 --report-bindings --map-by package --bind-to package --rank-by span | Alternate ranks across packagesWith span ranking, even ranks sit on package 0 and odd ranks on package 1. |
mpirun --np 4 --report-bindings --bind-to none | Check that ranks are unboundThe report then lists every rank as unbound instead of naming cores. |
mpirun --np 4 --bind-to core --host hostA:4,hostB:2 ./a.out | Bind to cores, map by packageWith only --bind-to given, mapping stays by package: ranks 0-1 take cores of package 0, ranks 2-3 of package 1. |
--rank-by VALUE | Set the ranking policyTakes slot, node, fill or span and has no qualifiers; it changes rank numbers, not process locations. |
--rank-by fill | Rank densely per resourceNumbers every process on one package before moving to the next; marked as the default in the manual's table. |
--rank-by span | Rank across all resourcesHands out rank numbers round-robin over the packages of all nodes before a second pass. |
--rank-by node | Rank round-robin by nodeConsecutive rank numbers alternate between the nodes. |
--map-by core --bind-to core --rank-by span | Default for one or two processesWhat mpirun applies when no placement option is given and the job has at most two processes. |
--map-by package --bind-to package --rank-by fill | Default for three or more processesApplied when no placement option is given and the job is larger than two processes: everything by package. |
Environment and working directory 17
| Command | What it does |
|---|---|
-x VAR | Export env variableExport an environment variable from the launching shell to every process. |
-x foo=bar | Export a variable with a valueThe started processes see foo set to bar, whatever the launching shell holds. |
-x foo* | Export variables by prefixThe trailing asterisk matches every variable in the current environment whose name begins with foo. |
mpirun -x DISPLAY -x OFILE=/tmp/out | Export several variablesEach -x option carries one variable; the manual advises exporting existing variables instead of defining values here. |
--set-env | Set a variable for the jobTakes NAME=VALUE and overwrites any existing value, like -x with a value. |
--unset-env VAR | Remove a variableA name ending in an asterisk, as in foo*, removes every variable with that prefix. |
--prepend-env | Prepend to a list variableWrite the separator in brackets after the name, as in LD_LIBRARY_PATH[:], followed by the value. |
--append-env | Append to a list variableSame syntax as --prepend-env; the new value goes after the existing content of the variable. |
--wdir DIR | Set the working directorympirun aborts if a node cannot change to it; by default nodes try the directory mpirun was started in. |
--set-cwd-to-session-dir | Run in the session directoryEach started process begins in its own session directory instead of the launch directory. |
--path PATH | Search path for executablesLooked in before the PATH environment variable when the program name has no directory part. |
--preload-binary | Copy the executable to the nodesShort form -s; the binary is sent to each remote machine before processes start. |
--preload-files LIST | Copy data files to the nodesTakes a comma-separated list; the files appear in the working directory of the remote processes. |
mpirun --prefix /opt/openmpi-VERSION -n 4 a.out | Point remote nodes at an installPrepends that installation's bin and lib directories to PATH and LD_LIBRARY_PATH; mostly needed for ssh launches. |
/opt/openmpi-VERSION/bin/mpirun -n 4 a.out | Invoke mpirun by absolute pathHas the same effect as passing --prefix with the directory the installation lives in. |
--noprefix | Disable automatic prefixStops mpirun from setting a prefix for remote daemons when it was built or invoked to do so. |
--tmpdir DIR | Move the session directorySets the root under which the launcher creates its session directory tree. |
MCA parameters 18
--mca is meant for Open MPI's own parameters; for PRRTE and PMIx parameters the manual advises --prtemca and --pmixmca, because mpirun otherwise has to guess the project from the parameter name.
| Command | What it does |
|---|---|
--mca KEY VALUE | Set MCA parameterPass a Modular Component Architecture parameter to tune Open MPI's behavior. |
mpirun --mca mpi_show_handle_leaks 1 -np 4 a.out | Set a parameter on the command lineThe command line has the highest precedence, above environment variables, tuning files and configuration files. |
mpirun --mca param "value with multiple words" | Quote a multi-word valueQuotes make the shell pass the words as one value; in parameter files quotes would become part of the value. |
export OMPI_MCA_mpi_show_handle_leaks=1 | Set a parameter in the environmentAny variable named OMPI_MCA_ followed by the parameter name is read; command-line values override it. |
--prtemca KEY VALUE | Set a PRRTE runtime parameterFor launcher settings such as default mapping and binding; note the single r in the option name. |
--pmixmca KEY VALUE | Set a PMIx parameterPMIx keeps its own parameter system, so its settings go through this option, not --mca. |
--tune FILENAME | Load parameters from a fileAccepts one file or a comma-separated list of files. |
mpirun -np 2 --tune foo.conf a.out | Apply a tuning fileValues from the file beat mca-params.conf but lose to the command line and the environment. |
mpirun -np 2 --tune foo.conf,bar.conf a.out | Apply several tuning filesFile names are joined with commas in a single --tune option. |
mpi_show_handle_leaks = 1 | Line of mca-params.confOne parameter per line in $HOME/.openmpi/mca-params.conf or the system-wide openmpi-mca-params.conf; # starts a comment. |
mpirun --mca mpi_show_mca_params enviro hello_c | Show parameters set by the userRank 0 prints every parameter that came from the environment or the command line, with its value. |
mpirun --mca btl self,sm,usnic | Use only the listed componentsA framework name used as a parameter takes an include list; components not named are ignored. |
mpirun --mca btl ^tcp,uct | Exclude componentsThe caret applies to the whole list: tcp and uct are left out and every other BTL stays eligible. |
export PRTE_MCA_rmaps_default_mapping_policy=core | Default mapping from the environmentPRRTE settings take the PRTE_MCA_ prefix; this one replaces the 4.x variable OMPI_MCA_rmaps_base_bycore. |
export PRTE_MCA_hwloc_default_binding_policy=package | Default binding from the environmentTakes over from the 4.x bind-to-socket setting of the hwloc_base framework. |
export PRTE_MCA_rmaps_default_mapping_policy=:oversubscribe | Oversubscribe by defaultA qualifier on its own is a valid value; this replaces OMPI_MCA_rmaps_base_oversubscribe from 4.x. |
export PRTE_MCA_display=map-devel,bind | Default display optionsShows the detailed map and the bindings for every job started from this environment. |
export PRTE_MCA_output=file=output.txt | Send output to files by defaultEnvironment form of the output file setting; it replaces the 4.x parameter orte_output_filename. |
Transports and networks 16
| Command | What it does |
|---|---|
mpirun -n 4 --mca btl tcp,sm,self prog1 | Pick TCP, shared memory and loopbackRuns four copies that exchange messages only through the tcp, sm and self byte transfer layers. |
mpirun --mca pml ob1 --mca btl tcp,sm,self | Force the TCP transport stackLeaving out sm slows traffic between ranks on one host; leaving out self breaks sends to the same process. |
mpirun --mca btl_tcp_if_include eth1,eth2 | Restrict TCP to interfacesOnly the named interfaces carry MPI traffic; an include list cannot be combined with an exclude list. |
mpirun --mca btl_tcp_if_exclude lo,eth0 | Keep TCP off interfacesAn exclude list has to name loopback and other local-only devices itself. |
mpirun --mca btl_tcp_if_include 192.168.1.0/24,10.10.0.0/16 | Restrict TCP to subnetsInclude and exclude lists also accept networks written in CIDR notation. |
mpirun --mca btl ^tcp | Disable the TCP BTLAll other BTLs stay eligible; the runtime still uses TCP for its own control messages. |
mpirun --mca btl self,sm,tcp --mca btl_base_verbose 30 -n 2 -host NodeA,NodeB a.out | Trace TCP connection attemptsThe verbose output lists each address and port the TCP BTL tries to connect to. |
mpirun --mca pml ucx --mca osc ucx | Force UCXSelects UCX for two-sided and one-sided MPI communication; on InfiniBand and RoCE devices it is already the default. |
mpirun -x UCX_NET_DEVICES=mlx5_0:1 | Choose the UCX network devicePins UCX to port 1 of device mlx5_0; by default it takes the port with the highest bandwidth. |
mpirun --mca pml ucx -x UCX_NET_DEVICES=mlx5_0:1 -x UCX_IB_GID_INDEX=1 | Set the RoCE GID indexUCX picks IPv4 RoCEv2 by default; the second variable selects a different GID. |
mpirun --mca pml ucx -x UCX_IB_SL=N | Set the InfiniBand service levelN is a value from 0 to 15; 0 is the default. |
mpirun --mca pml ucx --mca osc ucx --mca scoll ucx --mca atomic ucx | Force UCX for OpenSHMEM tooThe scoll and atomic settings route OpenSHMEM memory access and atomics through UCX as well. |
mpirun --mca pml cm --mca mtl ofi --mca mtl_ofi_provider_include psm2 mpi_hello | Use Libfabric with one providerSelects the OFI matching transport layer and limits it to the psm2 provider. |
--prtemca ras_base_launch_orted_on_hn 1 | Launch a daemon on the head nodeRecommended with the HPE CXI provider so processes on the first node of an allocation can use it. |
--display-comm | Show transports between ranksPrints which communication method each pair of ranks uses, at the start of the run. |
--display-comm-finalize | Show transports at shutdownSame table, printed when the program reaches MPI_Finalize. |
GPU-aware communication 11
| Command | What it does |
|---|---|
ompi_info | grep "MPI extensions" | Check for the CUDA extensioncuda appears in the list of MPI extensions when the build has CUDA support. |
ompi_info --parsable --all | grep mpi_built_with_cuda_support:value | Confirm the CUDA build flagThe matching line ends in true on a build with CUDA support. |
ucx_info -v | Check how UCX was configuredCUDA through UCX is the preferred route, so UCX itself must have been built with CUDA. |
mpirun -n 2 --mca pml ucx -x UCX_TLS=rc,sm,cuda_copy,gdr_copy,cuda_ipc ./osu_latency D D | GPU-to-GPU latency benchmarkRuns the OSU latency test with CUDA buffers on both sides over UCX with its CUDA transports enabled. |
mpirun --mca opal_cuda_verbose 10 | Basic CUDA debug outputThis parameter has a single verbosity level. |
mpirun --mca mpi_common_cuda_verbose 100 | Most detailed CUDA tracingValues 10, 20 and 100 give progressively more information about CUDA-aware activity. |
mpirun --mca btl_smcuda_use_cuda_ipc 0 | Turn CUDA IPC offBy default the smcuda BTL uses CUDA IPC wherever it can; 0 switches that off. |
mpirun --mca btl_smcuda_cuda_ipc_verbose 100 | Report whether CUDA IPC is usedShows for each pair of GPUs whether CUDA IPC was enabled. |
mpirun -x PSM2_CUDA=1 -x PSM2_GPUDIRECT=1 --mca mtl psm2 mpi_hello | CUDA over Omni-Path PSM2Both PSM2 variables must be in the environment before MPI_Init is called. |
mpirun -x HCOLL_GPU_ENABLE=1 -x HCOLL_ENABLE_NBC=1 | GPU buffers in HCOLL collectivesEnables CUDA buffer support in the HCOLL versions of collectives such as MPI_Allreduce and MPI_Bcast. |
--gpu-support VAL | Toggle library GPU supporttrue or false tells the application's MPI library to switch its internal GPU support on or off. |
Output and display 17
| Command | What it does |
|---|---|
--output LIST | Control how output is printedValues include tag, timestamp, xml, merge-stderr-to-stdout, dir= and file=; nocopy and raw are qualifiers. |
--output tag | Label lines with job and rankEach line is prefixed with [job,rank] and the stream it came from. |
--output timestamp | Timestamp every output lineThe stamp is the time the launcher printed the line, not when the process wrote it. |
--output merge | Merge stderr into stdoutApplies to each process; replaces the older --merge-stderr-to-stdout option. |
--output file=FILENAME | Write output to per-rank filesCreates one file per process, named with the rank as suffix; the name is made an absolute path. |
--output dir=DIRNAME | Write output to a directory treeOutput lands in DIRNAME/job/rank/ as separate stdout, stderr and stddiag files. |
--display VALUE | Report allocation, map or bindingsTakes a comma-separated list; values can be shortened as long as they stay unambiguous. |
--display map | Show process mapPrint the mapped location of every process before launch, useful for debugging placement. |
--display bindings | Show where each rank is boundCurrent spelling of --report-bindings; the report is written to stderr. |
--display alloc | Show the detected allocationLists the hosts and slot counts the launcher found for the job. |
--display map-devel | Show the detailed process mapAdds local and node ranks and the assigned bindings to the map. |
--display topo | Show node topologyPrints the hardware topology along with the process map when the job starts. |
--stream-buffering VALUE | Set stdout and stderr buffering0 means unbuffered, 1 line buffered and 2 fully buffered. |
--stdin LIST | Choose which ranks read stdinTakes ranks, all or none; by default only rank 0 receives standard input. |
mpirun -n 2 my_app < my_input > my_output | Redirect input and outputOnly rank 0 reads the input file, while standard output of all processes goes to the output file. |
--report-pid VALUE | Print the launcher's PIDA minus sign sends it to stdout, a plus sign to stderr; anything else is a file name. |
--report-uri VALUE | Print the launcher's contact URISame channel choices as --report-pid, applied to the contact URI of the running mpirun. |
Debugging and launch checks 19
| Command | What it does |
|---|---|
mpirun -n 4 xterm -e gdb my_mpi_application | One gdb window per rankLikely to work on a personal computer; on a cluster the X display has to reach the nodes first. |
mpirun -n 4 --hostfile my_hostfile -x DISPLAY=arcade.example.com:0 xterm -e gdb my_mpi_application | Debugger windows from remote nodesExporting DISPLAY makes the xterm windows opened on the nodes appear on your own screen. |
--xterm LIST | Show ranks in xterm windowsOpens a separate window for the output of each listed rank. |
--stop-on-exec VALUE | Stop processes at execHalts the named processes before they run any code so a debugger can attach, where supported. |
--stop-in-init VALUE | Stop processes in MPI start-upThe named processes pause inside MPI initialization, a point where a debugger can attach. |
mpirun -n 2 valgrind --suppressions=$PREFIX/share/openmpi/openmpi-valgrind.supp | Valgrind with Open MPI suppressionsThe suppression file shipped with Open MPI hides known false positives raised inside the library. |
--debug-daemons | Debug the launch daemonsGives a limited stream of daemon output, normally just enough to confirm that the daemons started. |
--debug-daemons-file | Log daemon debug output to filesEach daemon writes to a file named output-prted- plus its namespace and node name, in the session directory. |
--leave-session-attached | Keep daemon error outputWithout it the daemons detach after launch and close the streams that carry their error messages. |
mpirun --mca plm_base_verbose 10 --host remotehost hostname | Trace the remote launchLevel 10 prints extra detail on how Open MPI starts its daemons on the remote hosts. |
mpirun --mca opal_base_help_aggregate 0 | Show every copy of an errorStops identical help messages from many processes being merged into one, which can hide the real error. |
mpi_abort_delay | Wait before abortingSet with --mca: MPI_Abort prints host and PID, then waits that many seconds so a debugger can attach. |
mpi_abort_print_stack | Stack trace on MPI_AbortA nonzero value makes an aborting program print its call stack, on systems that support it. |
mpi_param_check | Check MPI call argumentsWhen true, bad arguments such as NULL handles raise an MPI error at run time; false skips the checks. |
mpirun --host remotehost,otherhost hostname | Test launching across hostsFirst check of an installation: a non-MPI program must start on every listed node. |
mpirun --host remotehost,otherhost hello_c | Test MPI start-uphello_c from the examples directory initializes and finalizes MPI without sending messages. |
mpirun --host remotehost,otherhost ring_c | Test message passingring_c passes a message around the processes, which proves MPI traffic crosses the network. |
ssh remotehost env | grep -i path | Inspect the remote environmentShows the PATH and LD_LIBRARY_PATH that a non-interactive ssh login gets on that host. |
ssh remotenode which ompi_info | Check which install a node findsMust print the ompi_info of the intended installation, otherwise the remote PATH is wrong. |
Timeouts and failure handling 13
| Command | What it does |
|---|---|
--timeout SECONDS | Abort after a time limitmpirun kills the job and exits with the ETIMEDOUT status, typically 110 on Linux. |
MPIEXEC_TIMEOUT | Time limit from the environmentEnvironment variable with the same meaning as the --timeout option. |
--get-stack-traces | Stack traces on timeoutCombined with --timeout, prints a trace of every process still running when time runs out. |
--report-state-on-timeout | Process states on timeoutPrints what the runtime knows about the job and each process when the limit is hit. |
--spawn-timeout SECONDS | Time limit for start-upEnds the job when launching it takes longer than the given number of seconds. |
--runtime-options LIST | Pass runtime optionsComma-separated NAME=VALUE settings such as timeout, max-restarts or autorestart; a boolean given without a value is true. |
--prtemca state_base_error_non_zero_exit false | Keep running after a non-zero exitWithout it, one process ending with a non-zero status makes mpirun abort the whole job. |
--report-child-jobs-separately | Ignore child jobs in exit statusNon-zero exits of jobs spawned by the application then appear only in a summary message. |
--forward-signals LIST | Forward extra signalsSIGTSTP, SIGUSR1, SIGUSR2, SIGABRT, SIGALRM and SIGCONT are forwarded already; none forwards nothing. |
--enable-recovery | Enable recovery from failuresOff by default, and only present when Open MPI and PRRTE were built with fault tolerance. |
--max-restarts | Limit restarts of a failed processTakes the highest count of restarts allowed for any one process. |
--continuous | Keep the job runningFailed processes are restarted automatically up to the restart limit, and the job ends only when it is terminated. |
mpirun -n 4 --with-ft ulfm my-ft-program | Run with ULFM fault toleranceFault tolerance stays inactive at run time unless this option is given; mpi is a synonym for ulfm. |
Launching under schedulers 13
| Command | What it does |
|---|---|
mpirun mpi-hello-world | Fill a scheduler allocationInside a Slurm, PBS or LSF job mpirun gets the process count and hosts from the scheduler. |
salloc -n 4 | Allocate four Slurm slotsInteractive allocation; a plain mpirun inside it then starts four processes. |
sbatch -n 4 my_script.sh | Submit a script that calls mpirunThe script needs no -n or host options because mpirun reads them from Slurm at run time. |
srun --mpi=list | List Slurm's MPI pluginspmix must appear in the list before srun can launch Open MPI programs directly. |
srun -N 4 --mpi=pmix mpi-hello-world | Direct launch with srunStarts the program without mpirun through Slurm's PMIx plugin; PMI-2 is no longer supported in 5.0. |
bsub -n 8 -R "span[ptile=2]" "mpirun mpi-hello-world" | Submit an LSF jobRequests four nodes with two processors each; mpirun takes hosts and counts from LSF. |
prte_info | grep lsf | Check for LSF supportScheduler support lives in PRRTE, so prte_info, not ompi_info, shows it. |
qsub -I -lnodes=4 | Interactive PBS jobAllocates four nodes; mpirun started inside it uses all of them. |
qsub -l nodes=4 my_script.sh | Submit a PBS scriptThe script simply calls mpirun; Open MPI warns against editing $PBS_NODEFILE. |
prte_info | grep ras | Check for PBS or Torque supportA line naming the tm component means the build can read PBS allocations. |
qrsh -V -pe ompi 4 mpirun hostname | Grid Engine job in one lineRequests four slots from the parallel environment named ompi and runs mpirun in them. |
qsub -pe ompi 4 my_mpirun_job.csh | Submit a Grid Engine batch jobThe mpirun command sits inside the submitted script. |
prte_info | grep gridengine | Check for Grid Engine supportSupport has to be requested with --with-sge when Open MPI is configured. |
Inspecting the installation 20
| Command | What it does |
|---|---|
ompi_info | Summarise the installationPrints version, build options and the list of installed components in readable form. |
ompi_info --all | Show everythingLists all components and their MCA parameters at every level, 1 through 9. |
ompi_info --parsable | Machine-readable outputSame information in a colon-separated form meant for scripts. |
ompi_info --param btl tcp | Parameters of one componentShows only level 1 parameters of the tcp BTL unless a level is added. |
ompi_info --param btl tcp --level 6 | Parameters up to a levelLevels 1 to 6 cover end-user and application-tuner parameters. |
ompi_info --param FRAMEWORK COMPONENT --level 9 | All parameters of a componentLevel 9 is the maximum; without --level only parameters that decide whether jobs run are shown. |
ompi_info --param all all | Parameters of every componentall is accepted for the framework and for the component; add --level 9 to see more than level 1. |
ompi_info --type string --pretty-print --level 3 | Parameters of one data typeLists the string-valued parameters of levels 1 to 3. |
ompi_info --path bindir | Show an installation pathOther accepted names are prefix, libdir, incdir, pkglibdir and sysconfdir. |
ompi_info --version | Show Open MPI version numbersPrints the release numbers in pretty-print format; -V is the short form. |
ompi_info --config | Show configuration optionsShort form -c; reports how this build of Open MPI was configured. |
ompi_info --level LEVEL | Raise the parameter levelParameters of levels 1 through LEVEL are shown; the default is 1, or 9 together with --all. |
--selected-only | Only selected componentsompi_info option: limits the variables shown to components that are actually selected. |
--internal | Include internal parametersompi_info option: also lists parameters that users are not meant to change. |
ompi_info --param btl tcp --level 9 | All TCP BTL parametersLists every tuning parameter of the component that carries MPI traffic over TCP. |
ompi_info --param pml ucx --level 9 | All UCX PML parametersThe UCX PML is tuned mainly through UCX environment variables, not through these MCA parameters. |
ompi_info --param mtl ofi --level 9 | All OFI MTL parametersIncludes the settings that include or exclude Libfabric providers. |
ompi_info --param mpi all --level 9 | MPI-layer parametersLists the debugging parameters of the MPI layer, such as mpi_show_handle_leaks. |
ompi_info | grep smsc | List single-copy mechanismsShows which shared-memory single-copy components, such as cma, the build contains. |
prte_info | List PRRTE parametersCounterpart of ompi_info for the PRRTE runtime; pmix_info does the same for PMIx. |
Compiling with wrappers 21
| Command | What it does |
|---|---|
mpicc hello_world_mpi.c -o hello_world_mpi -g | Compile a C MPI programThe wrapper adds Open MPI's compiler and linker flags and then calls the underlying C compiler. |
mpicxx | C++ wrapper compilermpic++ and, where file names are case-sensitive, mpiCC call the same compiler with the same options. |
mpifort | Fortran wrapper compilerHandles mpif.h, the mpi module and the mpi_f08 module; mpif77 and mpif90 are deprecated aliases. |
mpicc --showme | Show the full compiler commandPrints what would be run without running it. |
mpicc --showme:compile | Show the compile flagsPrints only the flags the wrapper would pass when compiling. |
mpicc --showme:link | Show the link flagsPrints only the linker flags and libraries the wrapper would add. |
--showme:command | Show the underlying compilerWrapper option: prints the back-end compiler command, which may be more than one word. |
--showme:incdirs | List header directoriesWrapper option: prints the include directories as a plain space-separated list. |
--showme:libdirs | List library directoriesWrapper option: prints the directories given to the linker as a plain space-separated list. |
--showme:libs | List linked librariesWrapper option: prints the library names used to link an application. |
--showme:version | Show the wrapper's Open MPI versionWrapper option: prints the release number of the installation the wrapper belongs to. |
cc -c file1.c `mpicc --showme:compile` | Compile without the wrapperCommand substitution inserts the wrapper's compile flags into a plain compiler call. |
cc file1.o file2.o `mpicc --showme:link` -o my_mpi_program | Link without the wrapperSame technique for the link step, using the wrapper's linker flags. |
pkg-config ompi-c --cflags --libs | Get flags from pkg-configompi-c is the C package; ompi-cxx and ompi-fort exist for the other languages. |
export PKG_CONFIG_PATH=/opt/openmpi/lib/pkgconfig | Let pkg-config find Open MPIThe .pc files are installed in the pkgconfig directory under the library directory. |
MPI_COMPILE_FLAGS = $(shell mpicc --showme:compile) | Query flags from a MakefileOpen MPI advises querying the wrapper at build time instead of hard-coding its output. |
OMPI_CC | Override the C compilerEnvironment variable read by mpicc at run time in place of the compiler chosen at configure time. |
OMPI_CFLAGS | Replace the C compiler flagsRead by mpicc; OMPI_CPPFLAGS, OMPI_LDFLAGS and OMPI_LIBS replace the other flag groups. |
OMPI_CXX | Override the C++ compilerRead by mpic++ and mpiCC; Open MPI cautions that a different compiler may be binary incompatible. |
OMPI_FC | Override the Fortran compilerRead by mpifort in place of the Fortran compiler selected when Open MPI was configured. |
ompi_info --all | grep -i wrapper | Show extra flags the wrappers addLeaves out the -I and -L flags for Open MPI's own headers and libraries. |
Variables set in each process 6
| Command | What it does |
|---|---|
OMPI_COMM_WORLD_SIZE | Number of processes in the jobHolds the size of the MPI_COMM_WORLD the process belongs to. |
OMPI_COMM_WORLD_RANK | Rank of this processIts rank in MPI_COMM_WORLD, available to the program as an ordinary environment variable. |
OMPI_COMM_WORLD_LOCAL_SIZE | Ranks of the job on this nodeCounts only processes of the same job that share the node. |
OMPI_COMM_WORLD_LOCAL_RANK | Rank within the nodeFour processes of one job sharing a node get local ranks 0 to 3. |
OMPI_COMM_WORLD_NODE_RANK | Rank on the node across all jobsUnlike the local rank, this numbering counts processes of every job on the node. |
OMPI_UNIVERSE_SIZE | Slots allocated to the jobMay differ from the count of processes that were actually started. |
Deprecated options 21
Deprecated options are still listed in the 5.0 manual, which says they should generally not be used and may be removed in a future release.
| Command | What it does |
|---|---|
--report-bindings | Old form of --display bindingsStill used throughout the manual's own examples, but listed as deprecated. |
--display-map | Old form of --display mapPrinted the process layout ahead of the launch. |
--display-allocation | Old form of --display allocShowed which hosts and slots the job had been given. |
--display-devel-map | Old form of --display map-develDeveloper-oriented, more detailed variant of the process map. |
--display-topo | Old form of --display topoPrinted the node topology together with the process map. |
--tag-output | Old way to tag output linesPrefixed all output with [job,rank]; the manual now points to --output. |
--timestamp-output | Old form of --output timestampStamped every line of application output with the time. |
--merge-stderr-to-stdout | Old form of --output mergeFolded the error stream of each process into its standard output. |
--output-filename | Old form of --output file=Redirected the output of each process to a file named filename.rank. |
--output-directory | Old form of --output dir=Accepted NOJOBID and NOCOPY directives after a colon. |
--xml | Old way to get XML outputThe manual now points to --output, whose xml value gives pseudo-XML formatting. |
--bind-to-core | Old form of --bind-to coreGave every process a core of its own. |
--gmca KEY VALUE | Global generic MCA parameterApplied to all application contexts; the manual recommends the project-specific --gprtemca or --gpmixmca. |
--npernode | Old per-node count optionUse the ppr:n:node mapping directive instead; it launched n processes on every node. |
--pernode | Old one-per-node optionUse the ppr:1:node mapping directive instead. |
--npersocket | Old per-socket count optionUse the ppr:n:package mapping directive instead; it also switched on binding to sockets. |
--bynode | Old form of --map-by nodeSpread processes over the nodes in turn and ranked them by node as well. |
--bycore | Old form of --map-by core |
--byslot | Old form of --map-by slotMapped and ranked processes by slot with a single option. |
--bind-to-socket | Old form of --bind-to packageSocket is the older word for package, which mapping options still accept as an alias. |
--cpus-per-proc | Old form of the PE=n qualifierBound each process to a number of CPUs; --cpus-per-rank was an alias. |
The maintainers mark the hand-written half of the mpirun manual page as older content still to be audited; rows taken from it, such as -N, --nolocal and the rankfile lines, can lag behind the generated option list.
No row on this page matches that filter.
Notes
Where processes run
mpirun -n 6 mpi-hello-world starts six ranks on the local machine; without -n it starts one per processor core, or one per slot of the hosts it was given. --hostfile FILENAME names the hosts (with an optional slots= count per line) and -H host1,host2 lists them inline, where a :N suffix as in --host node01:5,node02 sets the slots. Asking for more ranks than slots is an error unless --oversubscribe is given. Inside a Slurm, PBS, LSF or Grid Engine job none of this is needed: mpirun mpi-hello-world reads the allocation from the scheduler. Slurm can also launch directly with srun -N 4 --mpi=pmix mpi-hello-world, but the Open MPI team recommends mpirun there.
Mapping and binding
Open MPI places processes in three steps: mapping assigns each one a location, ranking gives it its number in MPI_COMM_WORLD, and binding restricts it to particular processors. --map-by node deals ranks out round-robin across nodes instead of filling one node first; --map-by ppr:N:OBJ puts N processes on every node, package or other object, and a PE=n qualifier gives each rank n cores for threaded codes. Binding happens even when you ask for nothing: to a core for one or two processes, to a package for more, and not at all on an oversubscribed node. --bind-to core pins each rank to one core, which keeps the operating system from migrating it, and --bind-to none switches binding off. --display map and --display bindings print the resulting layout; they replace --display-map and --report-bindings, which the 5.0 manual lists as deprecated.
Environment and MCA
-x VAR exports an environment variable to every rank and -x foo=bar sets one, which is how UCX settings such as UCX_NET_DEVICES reach remote hosts. --mca KEY VALUE sets an Open MPI parameter: mpirun --mca btl_tcp_if_include eth1,eth2 chooses the TCP interfaces and mpirun --mca pml ucx --mca osc ucx forces UCX. The same parameter can come from an OMPI_MCA_ environment variable or from mca-params.conf, with the command line winning. Parameters of the launcher itself belong to PRRTE and are passed with --prtemca KEY VALUE. ompi_info --param FRAMEWORK COMPONENT --level 9 lists every parameter of a component; without --level only the first of nine levels is shown. --allow-run-as-root, or the two OMPI_ALLOW_RUN_AS_ROOT variables, lets mpirun run as root, which Open MPI strongly discourages outside cases such as containers.
Transports and GPU support
Open MPI normally chooses the transport itself: UCX on InfiniBand and RoCE devices when the build includes it, otherwise TCP together with shared memory. --display-comm prints which communication method each pair of ranks ended up with. For GPU buffers the preferred route is UCX built with CUDA: ucx_info -v shows how UCX was configured, and ompi_info --parsable --all | grep mpi_built_with_cuda_support:value tells whether Open MPI itself was built with CUDA support.
Questions
What is the difference between mpirun and mpiexec in Open MPI?
None. Both names are symbolic links to the same launcher, so mpiexec -n 2 mpi-hello-world behaves exactly like the mpirun form.
How do I run more MPI processes than cores with Open MPI?
Add --oversubscribe, an alias for the :oversubscribe qualifier of --map-by. Without it mpirun refuses to start more processes than the slots it knows about. Processes are not bound on an oversubscribed node.
Do I need a hostfile when running mpirun under Slurm?
No. Inside an salloc or sbatch allocation mpirun mpi-hello-world takes the hosts and the process count from Slurm. Direct launch with srun -N 4 --mpi=pmix mpi-hello-world also works when Slurm has the PMIx plugin, but Open MPI recommends mpirun; PMI-2 is not supported from 5.0 on.
What replaced --report-bindings and --display-map in Open MPI 5?
--display bindings and --display map. The 5.0 manual lists the old spellings, along with --tag-output, --timestamp-output and --output-filename, as deprecated in favour of --display VALUE and --output LIST.
How do I see which MCA parameters a component has?
ompi_info --param btl tcp --level 9 lists all parameters of the tcp BTL; put another framework and component in place of btl and tcp. Without --level only level 1 parameters appear. To see what a job actually received, set mpi_show_mca_params, as in mpirun --mca mpi_show_mca_params enviro hello_c.
How can I run mpirun as root inside a container?
Pass --allow-run-as-root, or set both OMPI_ALLOW_RUN_AS_ROOT=1 and OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 in the environment. mpirun refuses to run as root by default, and the Open MPI team strongly advises against it wherever it can be avoided.
Sources and checks
- Open MPI 5.0.x documentation: mpirun(1) manual page Current option lists, per-option details and deprecated options
- Open MPI v5.0.11 docs source: mpirun(1) manual page text Hand-written sections of the same manual page: older option list, hosts, placement examples, rankfiles, MCA parameters, exit status
- Open MPI v5.0.11 docs source: Quick start: Launching MPI applications First launches, hostfiles, scheduled environments
- Open MPI v5.0.11 docs source: Scheduling processes across hosts Hostfiles, --host, slots and oversubscription
- Open MPI v5.0.11 docs source: Launching only on the local node Single-machine runs
- Open MPI v5.0.11 docs source: Launching with SSH Remote PATH checks and prefix behaviour
25 more source pages
- Open MPI v5.0.11 docs source: Launching with Slurm mpirun inside Slurm jobs and srun direct launch
- Open MPI v5.0.11 docs source: Launching with LSF LSF jobs
- Open MPI v5.0.11 docs source: Launching with PBS / Torque PBS and Torque jobs
- Open MPI v5.0.11 docs source: Launching with Grid Engine Grid Engine jobs
- Open MPI v5.0.11 docs source: Unusual jobs Non-MPI programs, MPMD launches, application files
- Open MPI v5.0.11 docs source: Troubleshooting Step-by-step launch checks
- Open MPI v5.0.11 docs source: Prerequisites The --prefix option
- Open MPI v5.0.11 docs source: The Modular Component Architecture (MCA) Setting, showing and selecting MCA parameters; 4.x to 5.x parameter changes
- Open MPI v5.0.11 docs source: Quick start: Building MPI applications
- Open MPI v5.0.11 docs source: Extracting flags from the wrapper compilers
- Open MPI v5.0.11 docs source: Customizing wrapper compiler behavior OMPI_CC and related environment variables
- Open MPI v5.0.11 docs source: ompi_info(1) manual page ompi_info options and examples
- Open MPI v5.0.11 docs source: Open MPI Wrapper Compilers manual page mpicc, mpicxx, mpifort and the --showme options
- Open MPI v5.0.11 docs source: Environment variables set for MPI applications OMPI_COMM_WORLD_* variables
- Open MPI v5.0.11 docs source: CUDA CUDA-aware builds, verbosity and transport settings
- Open MPI v5.0.11 docs source: InfiniBand / RoCE support UCX selection and UCX environment variables
- Open MPI v5.0.11 docs source: OpenFabrics Interfaces (OFI) / Libfabric support OFI MTL and provider selection
- Open MPI v5.0.11 docs source: Shared Memory Single-copy mechanisms
- Open MPI v5.0.11 docs source: TCP TCP BTL selection, interface lists, connection tracing
- Open MPI v5.0.11 docs source: Open MPI Runtime Debugging Options MPI-layer debugging parameters
- Open MPI v5.0.11 docs source: Application Output Lost with Abnormal Termination Help message aggregation
- Open MPI v5.0.11 docs source: Using Serial Debuggers to Debug Open MPI Applications gdb in xterm windows
- Open MPI v5.0.11 docs source: Validating your installation ompi_info options for checking an installation
- Open MPI v5.0.11 docs source: Using Valgrind to Find Open MPI Application Errors Valgrind suppression file
- Open MPI v5.0.11 docs source: User-Level Fault Mitigation (ULFM) Running with fault tolerance
Every row on this page was matched to a line in the documentation above on . A binding that could not be found there is not listed.
Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.