Mellanox / InfiniBand commands

Command lines for checking NVIDIA (formerly Mellanox) InfiniBand fabrics on Linux: adapter and port status, discovery, counters, ibdiagnet health checks, the subnet manager, perftest benchmarks and the MFT tools mst, mlxconfig, flint and mlxlink, as NVIDIA's MLNX_OFED, MFT 4.37 and ibdiagnet manuals and the tools' own help print them. Numbers such as LID 32 are the manuals' examples and capitals mark your own values. The firmware tools need mst start first; UFM and the Windows tools are not covered.

125 commands in 10 sections. Checked row by row against NVIDIA MLNX_OFED Documentation v24.10-5.1.6.1 (2024 LTS U5): InfiniBand Fabric Utilities on . Edited by Jay.

Adapter and port status 15

CommandWhat it does
ibstatPort statusReads LID, SM LID, logical and physical state and active width of every local HCA port from the driver.
ibstat -lList adapter namesPrints only the channel adapter names, which other diagnostics accept after -C.
ibstat -pList port GUIDsPrints the GUID of each local port, one per line, instead of the full status.
ibstat CA_NAME PORTNUMOne port's statusLimits the report to a single adapter and port, for example port 2 of a named CA.
ibstatusStatus summaryPrints LID, SM LID, logical and physical state, width and rate for the local ports.
ibv_devicesList verbs devicesPrints each device name with its node GUID, as user-space verbs programs see it.
ibv_devinfoDevice infoReports the attributes that user-space verbs programs see for every device and port.
ibv_devinfo -vVerbose device attributesPrints every attribute instead of the short default listing.
ibv_devinfo -d DEV -i PORTOne device and portWithout these options the report covers all devices and all of their ports.
ibdev2netdevMap devices to netdevsPairs each RDMA device port with its Linux network interface and shows whether that interface's link is up.
ibdev2netdev -vMapping with details
ibaddrLocal port LID and GIDPrints the LID range and GID of the port this host uses; give a LID to resolve another port.
hca_self_test.ofedHCA self-testChecks firmware and driver versions, kernel architecture, active ports and node GUID after an MLNX_OFED install.
/etc/init.d/openibd startLoad the driverFixes tools such as ibv_devinfo failing with Function not implemented, which means the driver is not loaded.
/etc/init.d/openibd restartRestart the driver stackReloads the MLNX_OFED driver after installation when the installer did not update the adapter firmware.

Port counters and errors 12

PortXmitData and PortRcvData count units of four octets, so multiply them by 4 to get bytes.

CommandWhat it does
perfqueryPerf countersWith no address it reads the port through which this host is attached.
perfquery 32 1Counters of a remote portThe first number is the target LID and the second its port number.
perfquery -x 32 1Extended countersSwitches from the basic counter set to the extended port counters.
perfquery -a 32All ports aggregatedSums the counters over every port of the node at LID 32.
perfquery -l 32 1-10Each port separatelyPrints ports 1 to 10 of LID 32 one by one instead of a single total.
perfquery -r 32 1Read then resetPrints the counters of LID 32 port 1 and clears them afterwards.
perfquery -R 32 2 0x0fffReset error counters onlyThe mask 0x0fff selects the error counters; 0xf000 would clear the traffic counters instead.
perfquery -R -a 32Reset all portsClears the counters on every port of LID 32 without printing them.
ibqueryerrorsFabric-wide error scanLists every port in the fabric whose error counters exceed the threshold, which defaults to zero.
ibqueryerrors --detailsAdd discard details
ibqueryerrors -kClear errors after readingThe capital -K does the same for the data counters.
ibqueryerrors -rAdd port link information

Fabric health checks 12

CommandWhat it does
ibdiagnetFull fabric diagnosticChecks duplicate GUIDs, links, counters, routing, speed and width, the SM and partitions in one run.
ibdiagnet -i DEV-NAME -p PORT-NUMChoose the local portNeeded when the host has several adapters, to pick the one attached to the fabric under test.
ibdiagnet --pcReset fabric countersClears the performance counters on every node of the fabric, not just on the local adapter.
ibdiagnet --pc --reset_phy_info -i MLX_DEVStart a BER baselineStep one of NVIDIA's health check: clear counters and PHY data, then let traffic run 5 to 10 minutes.
ibdiagnet --get_phy_info -i MLX_DEVCollect BER after trafficFlags ports whose symbol or effective BER exceeds the threshold; review ibdiagnet2.log afterwards.
ibdiagnet -P allPrint all nonzero counters-P sets a threshold per counter, for example vl15_dropped=1; all applies one threshold, zero by default.
ibdiagnet --pm_pause_time 60Sample counters over 60 sTakes two counter samples 60 seconds apart and writes the difference to the PM_DELTA section of the db_csv file.
ibdiagnet --pm_get_allCollect every PM counterTurns on per-SL/VL counters, NVIDIA diagnostic counters, extended speed counters and the per-lane listing together.
ibdiagnet --ls 50Expect HDR link speedLogs every link below the expected speed in ibdiagnet.log; 50 is HDR in NVIDIA's example.
ibdiagnet --lw 4xExpect 4x link widthAny link that trained to another width is reported as an unexpected width.
ibdiagnet -rRouting quality reportTune this check with --r_opt, which takes options such as skip_vs or sl=N.
ibdiagnet -o DIRECTORYChoose output directory

Subnet manager and SA queries 15

The sminfo manual warns that anything beyond a plain query, such as its -s, -p and -a set options, can make the target SM malfunction.

CommandWhat it does
sminfoFind the master SMPrints the LID, GUID, priority and state of the subnet manager, for example SMINFO_MASTER.
smpquery nd 8Node description by LIDPrints the node description of LID 8, here the host running the master SM in NVIDIA's example.
smpquery portinfo LID PORTPort detailReads the PortInfo attribute of one port, addressed by the node's LID and the port number.
saqueryQuery the SAWith no options it asks the subnet administrator for the node records of the subnet.
saquery -sPorts running an SMMatches ports with the IsSM or IsSMdisabled capability bit, so disabled SMs show up too.
saquery --src-to-dst SRC:DSTPath record between nodesBoth ends can be given as node names or as LIDs.
opensmSubnet managerInitializes the subnet's switches and adapters; one SM has to run on every InfiniBand subnet.
opensm --helpList SM options
service opensmd startStart SM as a serviceStarts the SM in daemon mode through the opensmd service.
opensm -BRun as daemon
opensm -g GUIDBind to a port GUIDSelects the local port the SM serves; NVIDIA SM accepts up to eight GUIDs separated by commas.
opensm -p PRIORITYSet SM priorityIn a handover the master is chosen by priority, then GUID; values run from 0 to 15.
opensm -R ENGINEChoose routing engineWithout it NVIDIA SM routes with AR UPDN; several engines can be listed, separated by commas.
opensm -c FILE-NAMEWrite a config templateWrites OpenSM's configuration to the file and exits, giving a template for opensm.conf.
pkill -HUP opensmReload partitionsA HUP makes the SM start a heavy sweep, so partitions.conf changes apply without a restart.

Port and routing control 6

CommandWhat it does
ibportstate -D 0 1Query a port's stateWithout an operation the port is only queried; here it is reached by direct route 0, port 1.
ibportstate 3 1 resetReset a portResets port 1 of the node at LID 3; disable and enable take the same arguments.
ibportstate -C qib0 -P 1 3 1 disableDisable a port-C and -P pick the local CA and port to send from; 3 and 1 are the target LID and port.
ibportstate -C qib0 -P 1 3 1 enableEnable a port
ibroute 4Unicast forwarding tableDumps the LFT of the switch at LID 4, listing only LIDs that have a valid out port.
ibroute -M 4Multicast forwarding tableLists the multicast LIDs with entries on the switch at LID 4.

Connectivity and performance tests 22

Start the perftest server and client with the same options; only the client adds the server address.

CommandWhat it does
ibping -SStart a ping serverKeeps running on this node and answers clients that ping its LID.
ibping LIDIB pingTakes the destination LID, or a GUID together with -G, and prints ping-style results.
ibv_rc_pingpongVerbs ping-pong serverStart this on one node first, then point the client form at its host name.
ibv_rc_pingpong HOSTVerbs ping-pong clientExchanges 1000 messages of 4096 bytes unless -n and -s say otherwise.
ib_write_bwWrite bandwidth testRun without a host name it starts the server side and waits for a client to connect.
ib_write_bw HOSTWrite bandwidth clientOptions marked SYMMETRIC in the help, such as -a, -s and -q, must match the server's.
ib_write_bw -d mlx5_0Server on a chosen deviceWithout -d the test uses the first RDMA device it finds.
ib_write_bw -d mlx5_0 SERVER_IPClient on a chosen deviceThe client names the server's address; an IPoIB address works when IPoIB is configured.
ib_write_bw -aSweep all message sizesSteps through message sizes from 2 bytes to 8 MiB; give it to server and client alike.
ib_write_bw -D SECONDSRun for a fixed timeRuns for that many seconds instead of the default 5000 iterations.
ib_write_bw -q NUMUse several queue pairsSpreads the traffic over that many QPs; the value must be the same on both sides.
ib_write_bw -bBidirectional bandwidthBoth sides send at the same time; the default measures one direction.
ib_write_bw --report_gbitsReport in Gbit/sThe default unit is MiB/s; dividing a MiB/s figure by 119.2 also gives Gb/s.
ib_write_bw --numa_node=1 -d mlx5_0Bind to a NUMA nodeOverrides perftest's automatic binding to the adapter's own NUMA node, which it reads from sysfs.
ib_send_bw -d mlx5_0 --use_cuda=0 --use_cuda_dmabufGPUDirect RDMA testBuffers sit in GPU 0 memory exported through DMA-BUF; needs CUDA 11.7 and open GPU kernel modules 515 or later.
ib_read_bwRead bandwidth testRDMA reads need RC transport; one machine runs the server side and the other the client.
ib_send_bwSend bandwidth testThe server posts receives for the client's packets and both ends compute the throughput.
ib_write_latWrite latency testPing-pong of RDMA writes in which each side writes only after the peer's write has arrived.
ib_read_latRead latency testLike the read bandwidth test it needs RC transport; the sides take turns reading each other's memory.
ib_send_latSend latency testPing-pong test in which each side sends only after receiving, timing every exchange with the CPU clock.
mlnx_tunePerformance tuningReports hardware and software status and known issues; a tuning profile is applied only when named with -p.
mlnx_tune -p HIGH_THROUGHPUTApply throughput profileOther profiles in the list cover IP forwarding, low latency with VMA and multicast.

Device access and configuration 11

CommandWhat it does
mst startStart MSTLoads the needed kernel modules and creates a /dev/mst file for each NVIDIA device found.
mst statusMST statusLists the /dev/mst device paths, for example /dev/mst/mt4115_pciconf0, used to address each adapter.
mst status -vDevices with RDMA namesAdds PCI address, RDMA device such as mlx5_0, network interface and NUMA node to each MST device.
mst stopStop MSTRemoves the /dev/mst files and unloads the modules; --force stops it even while in use.
mst restartRestart MSTSame as a stop followed by a start, with the same start options.
mst ib addAdd in-band fabric devicesMakes devices found across the fabric reachable in-band; needs OFED and an active IB link.
mst cable addAdd cable devices
mlxconfig -d DEVICE queryQuery NIC configShows each setting with its value for the next boot, for example SRIOV_EN.
mlxconfig -d DEVICE set LINK_TYPE_P1=1 LINK_TYPE_P2=1Set both ports to InfiniBandOn VPI cards 1 means IB and 2 Ethernet; reboot the machine to load the new setting.
mlxconfig -d DEVICE set SRIOV_EN=1 NUM_OF_VFS=8Enable SR-IOV with 8 VFsTurns on SR-IOV and creates eight virtual functions per physical function after a reboot.
mlxconfig -d DEVICE resetRestore default configAsks for confirmation, then returns every setting to its default; power-cycle the device to apply.

Firmware query and update 10

CommandWhat it does
flint -d DEVICE qCheck firmwareq is short for query; the output also lists security attributes of the firmware, such as secure-fw or debug.
flint -i IMAGE qQuery a firmware fileSame report for an image file on disk instead of the adapter.
flint -d DEVICE -i FW-FILE burnBurn firmware imageThe usual form keeps the GUIDs and VSD already on the device; -guid or -mac override them.
flint -d lid-XXX swresetReset an unmanaged switchRestarts an unmanaged switch in-band after a firmware burn, addressed by its LID.
mlxfwmanager --queryQuery firmware of all adaptersLists every NVIDIA adapter in the host with part number, PSID and its firmware, PXE and UEFI versions.
mlxfwmanager -d DEVICE --queryQuery one adapter
mlxfwmanager -u -d DEVICE -i MFAFILEUpdate from an MFA fileCompares current and available versions and burns the matching image; add -f when only the PXE ROM needs updating.
mlxfwmanager --online -u -d DEVICEUpdate from NVIDIA onlineDownloads the image from NVIDIA's website instead of using a local file; add -y to skip the prompt.
mlxfwreset -d DEVICE queryShow supported reset levelsLists which reset levels, types and sync modes the device supports, with the default marked.
mlxfwreset -d DEVICE resetLoad new firmware nowRestarts the driver and resets PCI, level 3 by default, so new firmware runs without a reboot.

Notes

Is the fabric up

ibstat shows each local HCA port with its logical and physical state, LID and the subnet manager's LID. A port that stays in Initializing is waiting for a subnet manager to assign its LID, so check with sminfo that one answers; opensm or service opensmd start runs NVIDIA SM on a host when no managed switch provides one. ibstatus prints a similar summary that includes the port rate, ibv_devinfo shows what verbs applications see and ibdev2netdev maps each device port to its network interface. ibnetdiscover, ibhosts and ibswitches list what the fabric contains, iblinkinfo prints every link node by node, and ibdiagnet --ls 50 or ibdiagnet --lw 4x reports each link that trained to an unexpected speed or width.

Is it fast

Start ib_write_bw -d mlx5_0 on one node and ib_write_bw -d mlx5_0 SERVER_IP on the other; both sides need the same options, and ib_write_bw --report_gbits prints the result in Gbit/s instead of MiB/s. ib_read_bw, ib_send_bw and latency tests such as ib_send_lat work the same way, and ib_send_bw -d mlx5_0 --use_cuda=0 --use_cuda_dmabuf runs from GPU memory. Perftest binds itself to the adapter's NUMA node; ib_write_bw --numa_node=1 -d mlx5_0 overrides that. When results vary, perfquery and ibqueryerrors show port error counters, and NVIDIA's bit error rate check is ibdiagnet --pc --reset_phy_info -i MLX_DEV, 5 to 10 minutes of traffic, then ibdiagnet --get_phy_info -i MLX_DEV. mlnx_tune reports host settings and known issues that affect performance.

Firmware and configuration

The MFT tools address adapters through device files: mst start creates them, mst status lists paths such as /dev/mst/mt4115_pciconf0 and mst status -v adds the matching RDMA device and network interface. mlxconfig -d DEVICE query shows the settings for the next boot, and mlxconfig -d DEVICE set LINK_TYPE_P1=1 LINK_TYPE_P2=1 switches both ports of a VPI card to InfiniBand after a reboot (2 selects Ethernet). mlxfwmanager --query reports firmware, PXE and UEFI versions with the PSID, mlxfwmanager --online -u -d DEVICE updates from NVIDIA's website and mlxfwreset -d DEVICE reset loads new firmware without a reboot where the card supports it. mlxlink -d DEVICE -c shows the physical counters and BER of one port.

Questions

Why does ibstat show a port stuck in Initializing?

The port is waiting for a subnet manager to assign its LID. Run sminfo from any host on the fabric; if no SM answers, start NVIDIA SM on a node with service opensmd start or enable the SM on a managed switch.

How do I measure RDMA bandwidth between two nodes?

Run ib_write_bw -d mlx5_0 on the first node, which waits as the server, then ib_write_bw -d mlx5_0 SERVER_IP on the second. Give both sides the same options, for example ib_write_bw -a for a sweep over message sizes, and add ib_write_bw --report_gbits to read the result in Gbit/s.

Which command checks the whole fabric for bad links and errors?

ibdiagnet discovers the fabric and checks links, counters, speed and width, routing, partitions and the subnet manager in one run, writing its files under /var/tmp/ibdiagnet2. ibqueryerrors lists every port whose error counters exceed the threshold, and perfquery 32 1 reads the counters of a single port.

How do I switch a ConnectX VPI card between InfiniBand and Ethernet?

Load the tools driver with mst start, find the device path with mst status, then run mlxconfig -d DEVICE set LINK_TYPE_P1=1 LINK_TYPE_P2=1. Value 1 selects InfiniBand and 2 Ethernet; the new link type is used after the machine reboots.

Does ibping need a server running on the target node?

No. A default ping server is built into the kernel, so ibping LID gets answers without extra setup; ibping -S starts a ping server in user space instead.

Sources and checks

19 more source pages

Every row on this page was matched to a line in the documentation above on . A binding that could not be found there is not listed.

Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.