Adapter and port status 15
| Command | What it does |
|---|---|
ibstat | Port statusReads LID, SM LID, logical and physical state and active width of every local HCA port from the driver. |
ibstat -l | List adapter namesPrints only the channel adapter names, which other diagnostics accept after -C. |
ibstat -p | List port GUIDsPrints the GUID of each local port, one per line, instead of the full status. |
ibstat CA_NAME PORTNUM | One port's statusLimits the report to a single adapter and port, for example port 2 of a named CA. |
ibstatus | Status summaryPrints LID, SM LID, logical and physical state, width and rate for the local ports. |
ibv_devices | List verbs devicesPrints each device name with its node GUID, as user-space verbs programs see it. |
ibv_devinfo | Device infoReports the attributes that user-space verbs programs see for every device and port. |
ibv_devinfo -v | Verbose device attributesPrints every attribute instead of the short default listing. |
ibv_devinfo -d DEV -i PORT | One device and portWithout these options the report covers all devices and all of their ports. |
ibdev2netdev | Map devices to netdevsPairs each RDMA device port with its Linux network interface and shows whether that interface's link is up. |
ibdev2netdev -v | Mapping with details |
ibaddr | Local port LID and GIDPrints the LID range and GID of the port this host uses; give a LID to resolve another port. |
hca_self_test.ofed | HCA self-testChecks firmware and driver versions, kernel architecture, active ports and node GUID after an MLNX_OFED install. |
/etc/init.d/openibd start | Load the driverFixes tools such as ibv_devinfo failing with Function not implemented, which means the driver is not loaded. |
/etc/init.d/openibd restart | Restart the driver stackReloads the MLNX_OFED driver after installation when the installer did not update the adapter firmware. |
Fabric discovery and links 13
| Command | What it does |
|---|---|
ibnetdiscover | Discover the subnetPrints every node and link with GUIDs, node types, port numbers, LIDs and node descriptions. |
ibnetdiscover TOPOLOGY-FILE | Save the topology to a fileWrites the discovered topology into that file instead of standard output. |
ibnetdiscover -p | Ports report |
ibhosts | List hostsAccepts a saved topology file in place of a live scan; -C and -P choose the local CA port. |
ibswitches | List switchesLists every switch with its GUID, port count, description and LID, the address an in-band firmware update needs. |
ibnodes | List all nodesCovers channel adapters and switches together, from the live subnet or a saved topology file. |
ibrouters | List routers |
iblinkinfo | Link infoWalks the fabric and prints the link of every port, grouped by node; options restrict it to part of the fabric. |
iblinkinfo --down | Only down links |
iblinkinfo --line | One line per linkPuts all details of a link on a single line instead of the per-node layout. |
iblinkinfo --switches-only | Switch links only |
ibtracert 4 16 | Trace route between LIDsShows every hop from LID 4 to LID 16; add -n for the short output format. |
ibtracert -m 0xc000 4 16 | Trace a multicast path |
Port counters and errors 12
PortXmitData and PortRcvData count units of four octets, so multiply them by 4 to get bytes.
| Command | What it does |
|---|---|
perfquery | Perf countersWith no address it reads the port through which this host is attached. |
perfquery 32 1 | Counters of a remote portThe first number is the target LID and the second its port number. |
perfquery -x 32 1 | Extended countersSwitches from the basic counter set to the extended port counters. |
perfquery -a 32 | All ports aggregatedSums the counters over every port of the node at LID 32. |
perfquery -l 32 1-10 | Each port separatelyPrints ports 1 to 10 of LID 32 one by one instead of a single total. |
perfquery -r 32 1 | Read then resetPrints the counters of LID 32 port 1 and clears them afterwards. |
perfquery -R 32 2 0x0fff | Reset error counters onlyThe mask 0x0fff selects the error counters; 0xf000 would clear the traffic counters instead. |
perfquery -R -a 32 | Reset all portsClears the counters on every port of LID 32 without printing them. |
ibqueryerrors | Fabric-wide error scanLists every port in the fabric whose error counters exceed the threshold, which defaults to zero. |
ibqueryerrors --details | Add discard details |
ibqueryerrors -k | Clear errors after readingThe capital -K does the same for the data counters. |
ibqueryerrors -r | Add port link information |
Fabric health checks 12
| Command | What it does |
|---|---|
ibdiagnet | Full fabric diagnosticChecks duplicate GUIDs, links, counters, routing, speed and width, the SM and partitions in one run. |
ibdiagnet -i DEV-NAME -p PORT-NUM | Choose the local portNeeded when the host has several adapters, to pick the one attached to the fabric under test. |
ibdiagnet --pc | Reset fabric countersClears the performance counters on every node of the fabric, not just on the local adapter. |
ibdiagnet --pc --reset_phy_info -i MLX_DEV | Start a BER baselineStep one of NVIDIA's health check: clear counters and PHY data, then let traffic run 5 to 10 minutes. |
ibdiagnet --get_phy_info -i MLX_DEV | Collect BER after trafficFlags ports whose symbol or effective BER exceeds the threshold; review ibdiagnet2.log afterwards. |
ibdiagnet -P all | Print all nonzero counters-P sets a threshold per counter, for example vl15_dropped=1; all applies one threshold, zero by default. |
ibdiagnet --pm_pause_time 60 | Sample counters over 60 sTakes two counter samples 60 seconds apart and writes the difference to the PM_DELTA section of the db_csv file. |
ibdiagnet --pm_get_all | Collect every PM counterTurns on per-SL/VL counters, NVIDIA diagnostic counters, extended speed counters and the per-lane listing together. |
ibdiagnet --ls 50 | Expect HDR link speedLogs every link below the expected speed in ibdiagnet.log; 50 is HDR in NVIDIA's example. |
ibdiagnet --lw 4x | Expect 4x link widthAny link that trained to another width is reported as an unexpected width. |
ibdiagnet -r | Routing quality reportTune this check with --r_opt, which takes options such as skip_vs or sl=N. |
ibdiagnet -o DIRECTORY | Choose output directory |
Subnet manager and SA queries 15
The sminfo manual warns that anything beyond a plain query, such as its -s, -p and -a set options, can make the target SM malfunction.
| Command | What it does |
|---|---|
sminfo | Find the master SMPrints the LID, GUID, priority and state of the subnet manager, for example SMINFO_MASTER. |
smpquery nd 8 | Node description by LIDPrints the node description of LID 8, here the host running the master SM in NVIDIA's example. |
smpquery portinfo LID PORT | Port detailReads the PortInfo attribute of one port, addressed by the node's LID and the port number. |
saquery | Query the SAWith no options it asks the subnet administrator for the node records of the subnet. |
saquery -s | Ports running an SMMatches ports with the IsSM or IsSMdisabled capability bit, so disabled SMs show up too. |
saquery --src-to-dst SRC:DST | Path record between nodesBoth ends can be given as node names or as LIDs. |
opensm | Subnet managerInitializes the subnet's switches and adapters; one SM has to run on every InfiniBand subnet. |
opensm --help | List SM options |
service opensmd start | Start SM as a serviceStarts the SM in daemon mode through the opensmd service. |
opensm -B | Run as daemon |
opensm -g GUID | Bind to a port GUIDSelects the local port the SM serves; NVIDIA SM accepts up to eight GUIDs separated by commas. |
opensm -p PRIORITY | Set SM priorityIn a handover the master is chosen by priority, then GUID; values run from 0 to 15. |
opensm -R ENGINE | Choose routing engineWithout it NVIDIA SM routes with AR UPDN; several engines can be listed, separated by commas. |
opensm -c FILE-NAME | Write a config templateWrites OpenSM's configuration to the file and exits, giving a template for opensm.conf. |
pkill -HUP opensm | Reload partitionsA HUP makes the SM start a heavy sweep, so partitions.conf changes apply without a restart. |
Port and routing control 6
| Command | What it does |
|---|---|
ibportstate -D 0 1 | Query a port's stateWithout an operation the port is only queried; here it is reached by direct route 0, port 1. |
ibportstate 3 1 reset | Reset a portResets port 1 of the node at LID 3; disable and enable take the same arguments. |
ibportstate -C qib0 -P 1 3 1 disable | Disable a port-C and -P pick the local CA and port to send from; 3 and 1 are the target LID and port. |
ibportstate -C qib0 -P 1 3 1 enable | Enable a port |
ibroute 4 | Unicast forwarding tableDumps the LFT of the switch at LID 4, listing only LIDs that have a valid out port. |
ibroute -M 4 | Multicast forwarding tableLists the multicast LIDs with entries on the switch at LID 4. |
Connectivity and performance tests 22
Start the perftest server and client with the same options; only the client adds the server address.
| Command | What it does |
|---|---|
ibping -S | Start a ping serverKeeps running on this node and answers clients that ping its LID. |
ibping LID | IB pingTakes the destination LID, or a GUID together with -G, and prints ping-style results. |
ibv_rc_pingpong | Verbs ping-pong serverStart this on one node first, then point the client form at its host name. |
ibv_rc_pingpong HOST | Verbs ping-pong clientExchanges 1000 messages of 4096 bytes unless -n and -s say otherwise. |
ib_write_bw | Write bandwidth testRun without a host name it starts the server side and waits for a client to connect. |
ib_write_bw HOST | Write bandwidth clientOptions marked SYMMETRIC in the help, such as -a, -s and -q, must match the server's. |
ib_write_bw -d mlx5_0 | Server on a chosen deviceWithout -d the test uses the first RDMA device it finds. |
ib_write_bw -d mlx5_0 SERVER_IP | Client on a chosen deviceThe client names the server's address; an IPoIB address works when IPoIB is configured. |
ib_write_bw -a | Sweep all message sizesSteps through message sizes from 2 bytes to 8 MiB; give it to server and client alike. |
ib_write_bw -D SECONDS | Run for a fixed timeRuns for that many seconds instead of the default 5000 iterations. |
ib_write_bw -q NUM | Use several queue pairsSpreads the traffic over that many QPs; the value must be the same on both sides. |
ib_write_bw -b | Bidirectional bandwidthBoth sides send at the same time; the default measures one direction. |
ib_write_bw --report_gbits | Report in Gbit/sThe default unit is MiB/s; dividing a MiB/s figure by 119.2 also gives Gb/s. |
ib_write_bw --numa_node=1 -d mlx5_0 | Bind to a NUMA nodeOverrides perftest's automatic binding to the adapter's own NUMA node, which it reads from sysfs. |
ib_send_bw -d mlx5_0 --use_cuda=0 --use_cuda_dmabuf | GPUDirect RDMA testBuffers sit in GPU 0 memory exported through DMA-BUF; needs CUDA 11.7 and open GPU kernel modules 515 or later. |
ib_read_bw | Read bandwidth testRDMA reads need RC transport; one machine runs the server side and the other the client. |
ib_send_bw | Send bandwidth testThe server posts receives for the client's packets and both ends compute the throughput. |
ib_write_lat | Write latency testPing-pong of RDMA writes in which each side writes only after the peer's write has arrived. |
ib_read_lat | Read latency testLike the read bandwidth test it needs RC transport; the sides take turns reading each other's memory. |
ib_send_lat | Send latency testPing-pong test in which each side sends only after receiving, timing every exchange with the CPU clock. |
mlnx_tune | Performance tuningReports hardware and software status and known issues; a tuning profile is applied only when named with -p. |
mlnx_tune -p HIGH_THROUGHPUT | Apply throughput profileOther profiles in the list cover IP forwarding, low latency with VMA and multicast. |
Device access and configuration 11
| Command | What it does |
|---|---|
mst start | Start MSTLoads the needed kernel modules and creates a /dev/mst file for each NVIDIA device found. |
mst status | MST statusLists the /dev/mst device paths, for example /dev/mst/mt4115_pciconf0, used to address each adapter. |
mst status -v | Devices with RDMA namesAdds PCI address, RDMA device such as mlx5_0, network interface and NUMA node to each MST device. |
mst stop | Stop MSTRemoves the /dev/mst files and unloads the modules; --force stops it even while in use. |
mst restart | Restart MSTSame as a stop followed by a start, with the same start options. |
mst ib add | Add in-band fabric devicesMakes devices found across the fabric reachable in-band; needs OFED and an active IB link. |
mst cable add | Add cable devices |
mlxconfig -d DEVICE query | Query NIC configShows each setting with its value for the next boot, for example SRIOV_EN. |
mlxconfig -d DEVICE set LINK_TYPE_P1=1 LINK_TYPE_P2=1 | Set both ports to InfiniBandOn VPI cards 1 means IB and 2 Ethernet; reboot the machine to load the new setting. |
mlxconfig -d DEVICE set SRIOV_EN=1 NUM_OF_VFS=8 | Enable SR-IOV with 8 VFsTurns on SR-IOV and creates eight virtual functions per physical function after a reboot. |
mlxconfig -d DEVICE reset | Restore default configAsks for confirmation, then returns every setting to its default; power-cycle the device to apply. |
Firmware query and update 10
| Command | What it does |
|---|---|
flint -d DEVICE q | Check firmwareq is short for query; the output also lists security attributes of the firmware, such as secure-fw or debug. |
flint -i IMAGE q | Query a firmware fileSame report for an image file on disk instead of the adapter. |
flint -d DEVICE -i FW-FILE burn | Burn firmware imageThe usual form keeps the GUIDs and VSD already on the device; -guid or -mac override them. |
flint -d lid-XXX swreset | Reset an unmanaged switchRestarts an unmanaged switch in-band after a firmware burn, addressed by its LID. |
mlxfwmanager --query | Query firmware of all adaptersLists every NVIDIA adapter in the host with part number, PSID and its firmware, PXE and UEFI versions. |
mlxfwmanager -d DEVICE --query | Query one adapter |
mlxfwmanager -u -d DEVICE -i MFAFILE | Update from an MFA fileCompares current and available versions and burns the matching image; add -f when only the PXE ROM needs updating. |
mlxfwmanager --online -u -d DEVICE | Update from NVIDIA onlineDownloads the image from NVIDIA's website instead of using a local file; add -y to skip the prompt. |
mlxfwreset -d DEVICE query | Show supported reset levelsLists which reset levels, types and sync modes the device supports, with the default marked. |
mlxfwreset -d DEVICE reset | Load new firmware nowRestarts the driver and resets PCI, level 3 by default, so new firmware runs without a reboot. |
Link and cable diagnostics 9
On an adapter card leave out -p: port 1 is the pciconf0 device and port 2 the pciconf0.1 device, as in NVIDIA's ConnectX-4 example.
| Command | What it does |
|---|---|
mlxlink -d DEVICE | Link status of a portOn an adapter each port has its own MST device, so the switch-only -p option is left out. |
mlxlink -d DEVICE -c | Counters and BERAdds physical-layer error counters and the bit error rate to the link report. |
mlxlink -d DEVICE --show_module --json | Module info as JSON |
mlxlink -d DEVICE --port_state UP | Bring a port upDN takes the port down and TG toggles it. |
mlxlink -d DEVICE --cable --ddm | Cable DDM readings |
mlxlink -d DEVICE --cable --dump | Dump cable EEPROM |
mlxcables -d mt4115_pciconf0_cable_0 -q | Query a cableShows vendor, part and serial number, length and temperature of the cable or transceiver. |
mlxcables -d mt4115_pciconf0_cable_0 --DDM | Cable diagnostics |
mget_temp -d DEV | Read adapter temperaturePrints the device temperature in degrees Celsius; -v adds a table of all thermal diodes. |
No row on this page matches that filter.
Notes
Is the fabric up
ibstat shows each local HCA port with its logical and physical state, LID and the subnet manager's LID. A port that stays in Initializing is waiting for a subnet manager to assign its LID, so check with sminfo that one answers; opensm or service opensmd start runs NVIDIA SM on a host when no managed switch provides one. ibstatus prints a similar summary that includes the port rate, ibv_devinfo shows what verbs applications see and ibdev2netdev maps each device port to its network interface. ibnetdiscover, ibhosts and ibswitches list what the fabric contains, iblinkinfo prints every link node by node, and ibdiagnet --ls 50 or ibdiagnet --lw 4x reports each link that trained to an unexpected speed or width.
Is it fast
Start ib_write_bw -d mlx5_0 on one node and ib_write_bw -d mlx5_0 SERVER_IP on the other; both sides need the same options, and ib_write_bw --report_gbits prints the result in Gbit/s instead of MiB/s. ib_read_bw, ib_send_bw and latency tests such as ib_send_lat work the same way, and ib_send_bw -d mlx5_0 --use_cuda=0 --use_cuda_dmabuf runs from GPU memory. Perftest binds itself to the adapter's NUMA node; ib_write_bw --numa_node=1 -d mlx5_0 overrides that. When results vary, perfquery and ibqueryerrors show port error counters, and NVIDIA's bit error rate check is ibdiagnet --pc --reset_phy_info -i MLX_DEV, 5 to 10 minutes of traffic, then ibdiagnet --get_phy_info -i MLX_DEV. mlnx_tune reports host settings and known issues that affect performance.
Firmware and configuration
The MFT tools address adapters through device files: mst start creates them, mst status lists paths such as /dev/mst/mt4115_pciconf0 and mst status -v adds the matching RDMA device and network interface. mlxconfig -d DEVICE query shows the settings for the next boot, and mlxconfig -d DEVICE set LINK_TYPE_P1=1 LINK_TYPE_P2=1 switches both ports of a VPI card to InfiniBand after a reboot (2 selects Ethernet). mlxfwmanager --query reports firmware, PXE and UEFI versions with the PSID, mlxfwmanager --online -u -d DEVICE updates from NVIDIA's website and mlxfwreset -d DEVICE reset loads new firmware without a reboot where the card supports it. mlxlink -d DEVICE -c shows the physical counters and BER of one port.
Questions
Why does ibstat show a port stuck in Initializing?
The port is waiting for a subnet manager to assign its LID. Run sminfo from any host on the fabric; if no SM answers, start NVIDIA SM on a node with service opensmd start or enable the SM on a managed switch.
How do I measure RDMA bandwidth between two nodes?
Run ib_write_bw -d mlx5_0 on the first node, which waits as the server, then ib_write_bw -d mlx5_0 SERVER_IP on the second. Give both sides the same options, for example ib_write_bw -a for a sweep over message sizes, and add ib_write_bw --report_gbits to read the result in Gbit/s.
Which command checks the whole fabric for bad links and errors?
ibdiagnet discovers the fabric and checks links, counters, speed and width, routing, partitions and the subnet manager in one run, writing its files under /var/tmp/ibdiagnet2. ibqueryerrors lists every port whose error counters exceed the threshold, and perfquery 32 1 reads the counters of a single port.
How do I switch a ConnectX VPI card between InfiniBand and Ethernet?
Load the tools driver with mst start, find the device path with mst status, then run mlxconfig -d DEVICE set LINK_TYPE_P1=1 LINK_TYPE_P2=1. Value 1 selects InfiniBand and 2 Ethernet; the new link type is used after the machine reboots.
Does ibping need a server running on the target node?
No. A default ping server is built into the kernel, so ibping LID gets answers without extra setup; ibping -S starts a ping server in user space instead.
Sources and checks
- NVIDIA MLNX_OFED Documentation v24.10-5.1.6.1 (2024 LTS U5): InfiniBand Fabric Utilities What each diagnostic and performance utility does
- NVIDIA DOCA-Host packages (latest repository, July 2026 builds for Ubuntu 24.04): built-in help of the InfiniBand tools
- rdma-core v65.0 (linux-rdma): ibstat(8) manual page ibstat examples
- NVIDIA MLNX_OFED Documentation v24.10-5.1.6.1 (2024 LTS U5): Installing the Driver Driver restart and self-test after installation
- NVIDIA MLNX_OFED Documentation v24.10-5.1.6.1 (2024 LTS U5): InfiniBand Related Issues Loading the driver
- rdma-core v65.0 (linux-rdma): ibnetdiscover(8) manual page Synopsis and output file
19 more source pages
- NVIDIA InfiniBand Cluster Bring-up Procedure: HCA Firmware Installation Finding switch LIDs and resetting an unmanaged switch
- NVIDIA ibdiagnet InfiniBand Fabric Diagnostic Tool User Manual v2.27.0: Bit Error Rate (BER) BER collection and the fabric health method
- NVIDIA ibdiagnet InfiniBand Fabric Diagnostic Tool User Manual v2.27.0: Port Counters Counter options with examples
- NVIDIA ibdiagnet InfiniBand Fabric Diagnostic Tool User Manual v2.27.0: Fabric Links Validation Expected speed and width checks
- NVIDIA InfiniBand Cluster Bring-up Procedure: SM Logs Locating the subnet manager
- NVIDIA MLNX_OFED Documentation v24.10-5.1.6.1 (2024 LTS U5): NVIDIA SM OpenSM syntax, daemon start and partition reload
- perftest 26.07.8 (linux-rdma/perftest): README Server and client examples, GPUDirect and NUMA binding
- NVIDIA Firmware Tools (MFT) Documentation v4.37.0: mst Service on Linux mst commands and example output
- NVIDIA Firmware Tools (MFT) Documentation v4.37.0: Examples of mlxconfig Usage Query and reset
- NVIDIA Firmware Tools (MFT) Documentation v4.37.0: Using mlxconfig Link type and SR-IOV settings
- NVIDIA Firmware Tools (MFT) Documentation v4.37.0: Querying the Firmware Image flint query of a device or image file
- NVIDIA Firmware Tools (MFT) Documentation v4.37.0: Burning a Firmware Image flint burn
- NVIDIA Firmware Tools (MFT) Documentation v4.37.0: mlxfwmanager – Firmware Update and Query Tool Querying adapters
- NVIDIA Firmware Tools (MFT) Documentation v4.37.0: Updating the Device Updating from an MFA file or online
- NVIDIA Firmware Tools (MFT) Documentation v4.37.0: mlxfwreset – Loading Firmware on 5th Generation Devices Tool Query and reset
- NVIDIA Firmware Tools (MFT) Documentation v4.37.0: mlxlink Utility Link, counter, port state and cable examples
- NVIDIA Firmware Tools (MFT) Documentation v4.37.0: mlxcables - Cables Tool Cable query and DDM
- rdma-core v65.0 (linux-rdma): perfquery(8) manual page Units of the data counters
- rdma-core v65.0 (linux-rdma): sminfo(8) manual page Warning about the set options
Every row on this page was matched to a line in the documentation above on . A binding that could not be found there is not listed.
Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.