Cluster nodes and configuration 18
Listing commands whose synopsis offers -Y, mmlscluster among them, print colon-delimited output with a header naming each column, which suits scripts.
| Command | What it does |
|---|---|
mmlscluster | Cluster infoPrints cluster name and ID, repository type, remote shell and copy commands, then every node with its designation. |
mmlscluster --ces | List protocol nodesLimits the listing to CES protocol nodes; --cnfs and --cloud-gateway do the same for clustered NFS and cloud tiering. |
mmgetstate -a | Daemon state on all nodesEach node reports active, arbitrating, down, unknown or unresponsive; active means the daemon is ready for work. |
mmgetstate -a -L | Quorum and node countsAdds quorum, nodes up and total nodes to each line; an asterisk in the Quorum column means tiebreaker disks. |
mmgetstate -s | Cluster state summaryCounts the local and remote nodes that have joined the cluster and how many of them are quorum nodes. |
mmlsnodeclass --all | List node classesIncludes the system-defined classes; without an option only user-defined node classes appear. |
mmlsmgr | Show manager nodesNames the file system manager node of each file system and, on the last line, the cluster manager. |
mmlsmgr -c | Show cluster manager |
mmchmgr -c NODE | Move the cluster managerAppoints NODE as cluster manager; give a file system name instead of -c to move that file system's manager. |
mmlslicense -L | License type per nodeAn asterisk after a license type means the node's license is too low for the roles it performs. |
mmchlicense server --accept -N NODE | Assign server licensesQuorum, manager and NSD server roles need a server license; --accept skips the license prompt. |
mmaddnode -N NODEDESC | Add nodes to the clusterA descriptor such as k164n06:quorum-manager gives the node name and its roles; every file system gets a mount point there. |
mmchnode --quorum --manager -N NODE | Make nodes quorum managersDesignates them quorum nodes and adds them to the pool that file system and token managers are picked from. |
mmdelnode -N NODE | Remove nodes from the clusterUnmount the file systems and stop GPFS on those nodes first, and run it from a node that stays. |
mmdsh -N NODE COMMAND | Run a command on many nodesRuns on 64 nodes at a time by default and prefixes output lines with the host name unless -s is given. |
mmlsconfig | Show cluster configurationPrints the attributes set explicitly plus a default list, as they would apply after a restart. |
mmlsconfig ATTRIBUTE | Show one settingSeveral names can be joined with commas, as in maxblocksize,pagepool. |
mmchconfig ATTRIBUTE=VALUE -N NODE | Change a settingApplies to all nodes unless -N is given; -i takes effect at once and persists, -I only until restart. |
Starting and stopping GPFS 5
-N accepts a comma-separated list of nodes, a node file or a node class; for mmstartup the mount node class is not allowed.
| Command | What it does |
|---|---|
mmstartup -a | Start GPFS on all nodesWithout -a or -N only the node where you type the command starts its daemon. |
mmstartup -N NODE | Start GPFS on chosen nodesNODE can be a comma-separated list, a node file or a node class other than mount. |
mmshutdown -a | Shut down GPFS on all nodesFile systems are unmounted first; a hanging unmount times out after 60 seconds plus 3 per node. |
mmshutdown -N NODE | Stop GPFS on chosen nodesAsks for confirmation when stopping these nodes would cause a harmful condition; --accept skips that question. |
mmbuildgpl | Build the portability layerNeeds the kernel development files and a compiler; --build-package also produces an installable gpfs.gplbin package. |
Health, diagnostics and performance 16
| Command | What it does |
|---|---|
mmhealth cluster show | Cluster healthCounts total, failed, degraded and healthy entities for each component across the cluster. |
mmhealth node show | Node healthReports the local node and each monitored component; add a component such as ces or filesystem for detail. |
mmhealth node show -N all | Health of every node |
mmhealth node show --unhealthy | Unhealthy components onlyHides components that are fine so only the problems remain; --not-ok is a synonym. |
mmhealth node eventlog --hour | Events of the last hour--day, --week and --month widen the window; with no period the whole event history is listed. |
mmhealth event show EVENTNAME | Explain an eventDescribes what an event such as quorum_down means; an event ID works in place of the name. |
mmhealth event resolve EVENTNAME | Clear a resolvable eventWorks for fserr* and hc_ events and the few others whose user action says they can be cleared by hand. |
mmdiag --waiters | Show waiting threadsGives each waiting mmfsd thread with its wait time and reason, which helps with deadlocks and performance problems. |
mmdiag --version | Running GPFS buildDetails the build that the daemon on this node is running, which helps with installation problems. |
mmdiag --config | Active daemon settingsValues in force in the running mmfsd, undocumented ones included; mmlsconfig shows what would apply after a restart. |
mmdiag --iohist | Recent I/O historyLists recent GPFS I/O requests on this node with data or metadata type, size and completion time. |
mmnetverify | Network checks, all nodesSkips only the data-large and flood tests, and the local node probes all nodes of the cluster. |
mmnetverify connectivity | Connectivity checksCovers name resolution, ping, remote shell and remote copy between this node and the others. |
mmpmon -i INFILE -p -r 10 -d 5000 | I/O monitoringINFILE holds requests such as io_s or fs_io_s; they run 10 times, 5 seconds apart, as script-friendly output. |
mmperfmon config show | Show sensor configuration--config-file saves the active performance monitoring setup to a file you name. |
mmperfmon query compareNodes cpu_user | Compare nodes on a metricBuilds one column per node for a single metric, here cpu_user; only one metric can be compared at a time. |
File systems and mounts 14
A file system name does not have to be fully qualified: fs0 works as well as /dev/fs0 wherever DEVICE appears.
| Command | What it does |
|---|---|
mmlsfs all | List filesystemsPrints every attribute flag with its value, such as -B block size, -Q quota state and -T mount point. |
mmlsmount all -L | Who has file systems mounted-L adds the names of the nodes that mount each one; all_remote limits the list to remote file systems. |
mmmount all -a | Mount all file systems everywhereWithout -a or -N only the node you run it on mounts the file system. |
mmmount fs2 -o ro | Mount read-onlyfs2 is the file system name; -o hands options such as ro to the mount command on the local node. |
mmumount DEVICE -a | Unmount on all nodesall in place of DEVICE unmounts every file system; -N picks particular nodes instead of all of them. |
mmumount DEVICE -f | Force an unmountGoes ahead while files are still open, so outstanding writes can be lost; IBM urges extreme caution. |
mmdf DEVICE | Disk usageShows size, failure group, data or metadata role and free space per disk and pool; it is I/O heavy. |
mmdf DEVICE --block-size auto | Readable capacity unitsScales the block counts automatically to a readable unit such as G or T. |
mmdf DEVICE -F | Inode usageReports used, free and allocated inode counts for the file system. |
mmcrfs DEVICE -F STANZAFILE | Create a file systemWithout -T the mount point is /gpfs/DEVICE, and by default it mounts whenever the GPFS daemon starts. |
mmchfs DEVICE -A yes | Mount at daemon startno makes mounting manual, and automount mounts on first access on non-Windows nodes. |
mmfsck DEVICE -n | Check without repairingReports inconsistencies only; any mode except online -o needs the file system unmounted. |
mmfsck DEVICE -y | Check and repairAnswers yes to every repair prompt; IBM keeps it for severely damaged file systems. |
mmfsck DEVICE -n -o | Online checkRuns on a mounted file system and finds a subset of problems, such as block allocation map corruption. |
Disks and NSDs 11
| Command | What it does |
|---|---|
mmlsdisk DEVICE -L | Disk detailsAdds disk IDs and a remarks column, where desc marks a holder of a file system descriptor. |
mmlsdisk DEVICE -e | Disks not up and readyShows only the problem disks; when every disk is up and ready it prints message 6027-623. |
mmlsnsd | List NSDsShows each NSD with its file system and NSD servers; -F lists only those not in use. |
mmlsnsd -m | Map NSDs to local devicesPairs each NSD name with its device name on this node and its NSD servers; -M checks every node but is slow. |
mmcrnsd -F STANZAFILE | Create NSDsEach %nsd stanza gives a disk's device, NSD servers, usage and failure group. |
mmadddisk DEVICE -F STANZAFILE -r | Add disks and rebalanceThe file system can stay mounted, and -r spreads existing blocks over the new disks as well. |
mmdeldisk DEVICE DISKNAME | Remove a diskMoves data off and keeps every file's replication by default (-r); -m copies the minimum and finishes faster. |
mmchdisk DEVICE suspend -d DISKDESC | Stop allocating on a diskData on the disk can still be read and updated; resume makes the disk take new blocks again. |
mmchdisk DEVICE start -a | Start all down disksEach disk goes from down to recovering while missed updates are repaired, then becomes up. |
mmrestripefs DEVICE -b | Rebalance a file systemSpreads file blocks evenly over all disks that are neither stopped nor suspended. |
mmrestripefs DEVICE -r | Restore replicationEmpties suspended disks and brings replicated files back to their set replica count after a disk failure. |
Filesets and snapshots 10
| Command | What it does |
|---|---|
mmlsfileset DEVICE | List filesetsShows each fileset with its status, such as Linked, and its junction path. |
mmlsfileset DEVICE -L | Fileset IDs and inode spacesAdds the fileset ID, root inode, parent, creation time, inode space and comment. |
mmlsfileset DEVICE -d -i | Fileset space and inode useCounts blocks and inodes in use per fileset; the scan can take minutes on a large file system. |
mmcrfileset DEVICE FILESETNAME --inode-space new | Create an independent filesetnew gives the fileset its own inode space; naming an existing fileset instead creates a dependent one. |
mmlinkfileset DEVICE FILESETNAME -J JUNCTIONPATH | Link a filesetA new fileset stays invisible until linked, and the junction path must not exist yet. |
mmunlinkfileset DEVICE FILESETNAME | Unlink a filesetFails while files are open in it unless -f, which closes them so later use returns ESTALE. |
mmlssnapshot DEVICE | List snapshotsShows each snapshot's ID, status, creation and expiration time and fileset; -d adds the space it uses. |
mmcrsnapshot DEVICE SNAPSHOT | Create a global snapshotRead-only, visible under .snapshots in the file system root, and no protection against media failure. |
mmcrsnapshot DEVICE SNAPSHOT -j FILESETNAME | Snapshot a filesetCovers an independent fileset with its dependent filesets and appears under .snapshots in the fileset root. |
mmdelsnapshot DEVICE SNAPSHOT | Delete a snapshotWithout -j FILESETNAME a global snapshot of that name is removed; with it, the fileset snapshot. |
Quotas 10
| Command | What it does |
|---|---|
mmlsquota | My quota usageNon-root users see their own quotas, those of their groups and fileset quotas. |
mmlsquota -j FILESET DEVICE | Fileset quotaShows block and file limits for one fileset; -e first collects current usage from all nodes. |
mmrepquota -j DEVICE | Fileset quota reportLists usage and limits for every fileset; -e collects fresh usage from all nodes before printing. |
mmrepquota -u DEVICE | User quota reportAdd -v for a header and entry types, or name DEVICE:FILESET to report on one fileset. |
mmrepquota -q DEVICE | Quota enforcement stateSays whether user, group and fileset quotas and their default quotas are being enforced. |
mmcheckquota DEVICE | Verify quotaRecounts usage by user, group and fileset and rewrites the quota files; run it when load is light. |
mmsetquota fs1 --user user234 --block 25G:30G --files 10K:11K | Set user quota limitsValues are soft:hard pairs, here for user234 on fs1; --group sets a group's limits the same way. |
mmsetquota fs1 --grace user --block 10days | Set the grace periodApplies to users' block limits; use --files for file counts and group or fileset instead of user. |
mmquotaon DEVICE | Turn on quota enforcementCovers user, group and fileset quotas unless -u, -g or -j picks one; -a does every file system. |
mmchfs DEVICE -Q yes | Enable quotas at mountQuotas then switch on automatically at every mount; -Q no keeps the quota files but ignores them. |
Storage pools and policies 6
| Command | What it does |
|---|---|
mmlspool DEVICE all -L | Storage pool detailsPrints each pool's ID, block size, data or metadata usage and maximum disk size. |
mmlspolicy DEVICE -L | Show installed policy-L prints the whole policy file; without it only the first line and who installed it appear. |
mmapplypolicy DEVICE -P POLICYFILE -I test | Dry-run a policyEvaluates every rule and reports what would happen without deleting or moving any data. |
mmapplypolicy DEVICE -P POLICYFILE | Apply a policy fileRuns the rules now and moves data between pools during the scan; -I defer leaves moves for mmrestripefs. |
mmchpolicy DEVICE POLICYFILENAME -I test | Validate a policy fileChecks the rules and reports what it parsed, but installs nothing. |
mmchpolicy DEVICE POLICYFILENAME | Install a policyValidates and activates the rules at once; DEFAULT in place of a file restores the default placement policy. |
Protocol nodes and exports 13
mmces state, still shown in older guides, has been deprecated since IBM Storage Scale 5.1.9.
| Command | What it does |
|---|---|
mmces service list -a | Protocol services by nodeNames the enabled protocols, such as NFS, SMB and S3, then the ones running on each CES node. |
mmces service start smb -a | Start a protocol serviceService names are NFS, SMB, OBJ, BLOCK, HDFS and S3; without -N or -a only the local node is affected. |
mmces service stop smb -N NODE | Stop a protocol serviceClients lose access through that node's CES addresses, and the stop does not survive a reboot. |
mmces node list | List CES nodesShows each protocol node with its state, such as suspended; --verbose adds the addresses it holds. |
mmces node suspend -N NODE | Suspend protocol nodesThe node's CES addresses move to other nodes; --stop also stops its protocol services. |
mmces node resume --start | Resume a protocol nodeThe node can take CES addresses again, and --start brings up all enabled protocols. |
mmces address list | List CES addressesShows each CES address with its group, attributes and the node that holds it. |
mmces address move --rebalance | Rebalance CES addressesRedistributes addresses now under the configured policy, or even-coverage when none is set. |
mmnfs export list | List NFS exportsShows exported paths with delegations and clients; --nfsdefs PATH adds access type, squash and other client settings. |
mmnfs export add PATH --client "192.0.2.0/20(Access_Type=RW)" | Create an NFS exportThe client string sets access per network or netgroup, here read-write for 192.0.2.0/20. |
mmsmb export list | List SMB sharesWithout options it prints every share with common settings such as browseable; --option all shows every setting. |
mmsmb export add SMBEXPORT PATH | Create an SMB shareThe path must be on a GPFS file system with NFSv4 ACLs enforced, which mmlsfs with -k shows. |
mmuserauth service list | Show protocol authenticationPrints the authentication method set up for file and object access; service check tests it against the servers. |
Recovery groups and pdisks 10
| Command | What it does |
|---|---|
mmvdisk nodeclass list | List mmvdisk node classesShows each mmvdisk node class with its recovery groups; --node-class all adds the member servers. |
mmvdisk server list --node-class NCNAME | Server status by node classLists each server's node number and whether it is active; --disk-topology reports the RAID disk topology it found. |
mmvdisk recoverygroup list | List recovery groupsCovers every recovery group, active or not and mmvdisk-managed or not, with its current server. |
mmvdisk recoverygroup list --not-ok | Recovery groups with problemsKeeps only groups that are down, have inactive servers or hold pdisks that need replacing. |
mmvdisk recoverygroup list --declustered-array | Declustered array summaryGives a one-line overview of each declustered array in every recovery group. |
mmvdisk pdisk list --recovery-group all --not-ok | Failing pdisksLists pdisks that are not working correctly in any recovery group; name one group instead of all to narrow it. |
mmvdisk pdisk list --replace --recovery-group all | Pdisks due for replacementA lower priority value in the output means the disk needs replacing more urgently. |
mmvdisk pdisk replace --prepare --recovery-group RGNAME --pdisk PDISKNAME | Prepare a disk swapSuspends the pdisk and prints its enclosure, drawer and slot so the carrier can be pulled. |
mmvdisk pdisk replace --recovery-group RGNAME --pdisk PDISKNAME | Finish a disk swapRun it once the new drive sits in the slot that the prepare step reported. |
mmvdisk filesystem list | List RAID file systemsShows mmvdisk file systems with the vdisk sets and NSDs behind them; --file-system narrows it to one. |
No row on this page matches that filter.
Notes
Cluster and nodes
mmlscluster prints the cluster name and ID and every node with its designation (quorum, manager, gateway, perfmon), and mmgetstate -a shows whether the daemon on each node is active, arbitrating, down or unresponsive. mmhealth cluster show counts failed and degraded entities per component across the cluster, mmhealth node show drills into one node, and mmhealth node show --unhealthy trims that report to the components that need attention.
Filesystems and disks
mmlsfs all shows every filesystem's attributes (block size, replication, quota enablement) and mmdf DEVICE its capacity per disk and pool, with free space in full blocks and in fragments. mmlsdisk DEVICE -L lists the disks behind a filesystem with their status and availability, and mmlsdisk DEVICE -e keeps only those that are not up and ready; once the fault is fixed, mmchdisk DEVICE start -a brings stopped disks back. For filesets, mmlsfileset DEVICE -d -i reports the space and inodes each one uses.
Performance and quotas
mmpmon -i INFILE -p -r 10 -d 5000 runs the requests listed in INFILE, such as io_s for the whole node or fs_io_s per mounted filesystem, ten times with a five-second pause and in parseable form, while mmperfmon config show prints the active configuration of the performance monitoring tool. mmrepquota -j DEVICE reports fileset quotas with usage and limits, the command to run when a project says it cannot write, and mmcheckquota DEVICE recounts usage and rewrites the quota files, for example after a node failure has left the quota data wrong.
Questions
Which command shows whether GPFS is running on every node?
mmgetstate -a prints the daemon state of each node: active, arbitrating, down, unknown or unresponsive. mmgetstate -a -L adds the quorum and node counts, and mmstartup -N NODE starts the daemon on nodes that are down.
Which command lists the nodes of a Storage Scale cluster?
mmlscluster lists every node with its daemon and admin interface names and its designation. The mmlsnode command that older versions of this page showed has no entry in the 6.0.1 command reference. mmlsnodeclass --all lists the node classes that many commands accept after -N.
How do I find and restart disks that are down?
mmlsdisk DEVICE -e lists only the disks that are not up and ready. After the cause is fixed, mmchdisk DEVICE start -a takes them from down through recovering to up while missed updates are repaired, and mmrestripefs DEVICE -r restores full replication of files whose replicas became unavailable.
Why does mmdiag --config show different values from mmlsconfig?
mmdiag --config reads the values in force in the daemon running on that node, undocumented parameters included, while mmlsconfig shows the configured values, which are the ones that would apply after GPFS restarts.
How do I replace a failed drive in an IBM Storage Scale System (ESS)?
mmvdisk pdisk list --replace --recovery-group all lists the pdisks marked for replacement. mmvdisk pdisk replace --prepare --recovery-group RGNAME --pdisk PDISKNAME suspends the pdisk and reports its location; with the new drive inserted, mmvdisk pdisk replace --recovery-group RGNAME --pdisk PDISKNAME completes the replacement.
Sources and checks
- IBM Documentation: IBM Storage Scale 6.0.1 Command reference (the command pages cited here) Every mm command row except the mmvdisk rows: synopsis, parameters and examples, one page per command
- IBM Documentation: IBM Storage Scale System Software 7.0.1, IBM Storage Scale RAID commands (mmvdisk) Recovery group, pdisk, node class, server and file system rows of mmvdisk
Every row on this page was matched to a line in the documentation above on . A binding that could not be found there is not listed.
2 rows from the earlier version of this page were removed in that check: duplicates, and bindings the documentation does not list or lists differently.
Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.