Storage CLI commands: mdadm, Ceph, NetApp ONTAP, Pure Storage, VAST Data

Reading the state of an array

cat /proc/mdstat is the quickest view: every running array with its level and members, failed members flagged with (F), and a progress line with a finish estimate while a recovery runs. mdadm --detail DEVICE reports on one active array, while mdadm --examine DEVICE reads the superblock stored on a member disk; the manual draws the line there, components for --examine and whole running arrays for --detail. In scripts, mdadm --detail --test DEVICE turns the state into an exit status. echo check > /sys/block/mdX/md/sync_action starts a scrub through sysfs. The /proc/mdstat and sysfs rows are quoted from the Linux RAID wiki on kernel.org, which mdadm(8) links to and which is now archived.

Replacing a disk

Manage mode names the array first and the member devices after each operation. mdadm /dev/md0 --fail /dev/sdb1 marks a member faulty, mdadm /dev/md0 --remove /dev/sdb1 takes it out (only failed members and spares can be removed), and mdadm /dev/md0 --add /dev/sda1 adds the new disk; on a degraded array the rebuild onto it starts at once. --replace avoids running degraded: the old disk keeps serving until a spare has been rebuilt in its place, and --with picks that spare. A disk that has already vanished is handled with mdadm /dev/md4 --fail detached --remove detached.

Creating, recording and assembling

mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/hd[ac]1 is the manual's mirror example; new arrays get version 1.2 metadata and, for the levels that use one, a 512 KB chunk. mdadm --detail --scan >> mdadm.conf writes config lines for the running arrays into a file to review before it becomes /etc/mdadm.conf or /etc/mdadm/mdadm.conf. mdadm --assemble --scan, the command for a startup file, assembles the arrays listed there; with none listed it searches the devices, and arrays that do not obviously belong to this host get non-conflicting names and start read-auto. mdadm --stop DEVICE stops one array once nothing uses it. Level changes such as mdadm --grow /dev/md4 --level=6 --backup-file=/root/backup-md4 need a backup file on a separate device, as do shrinks and layout changes.

Status and health

ceph status (or ceph -s) is the first look: cluster ID, health, monitor quorum and the daemons of each service. When it reports HEALTH_WARN or HEALTH_ERR, ceph health detail names every active alert and what is behind it, for example which OSDs are down. ceph -w prints the status and then keeps following the cluster log. With several monitors, check that they form a quorum: ceph quorum_status names the monitors in quorum and the leader, and ceph mon stat shows the monitor map.

Capacity and placement

ceph df shows raw capacity and usage per pool, and ceph osd df shows how full each OSD is. That matters because one OSD above the full ratio (95% by default) stops clients from writing. ceph osd tree shows the CRUSH hierarchy with each OSD's up or down state, ceph pg stat summarises placement group states, and ceph pg dump_stuck lists the ones stuck in a state such as unclean or inactive. ceph osd pool ls detail shows each pool's replica size, PG count and autoscale mode, rados lspools lists pools, and ceph osd pool create POOL-NAME makes a new one.

OSDs in and out

ceph osd out OSD-NUM takes an OSD out of data placement so that Ceph rebalances and copies its data to other OSDs, the first step in removing one, and ceph osd in OSD-NUM reverses it. For maintenance where the OSDs come back, ceph osd set noout keeps CRUSH from rebalancing while they are stopped; clear it with ceph osd unset noout. On a cephadm cluster, ceph orch osd rm OSD_ID evacuates and removes an OSD in one command. ceph auth ls lists every cephx user with its key and capabilities.

Cluster, nodes and network

cluster show lists the nodes with their health and eligibility, and cluster image show gives the ONTAP version each node runs. system node image show adds which image is the default and which is running, but needs advanced privilege. storage failover show reports for each node whether takeover is possible and, when takeover is disabled, the reason. network interface show lists LIFs with their current node and port and whether they are home, and network interface revert -vserver * -lif * sends every stray LIF back. event log show is the EMS log, filterable by severity and time.

Aggregates and disks

storage aggregate show lists aggregates with size, used percentage, state and RAID status, and storage aggregate show-space breaks down where the space went, from volume footprints to metadata. storage disk show lists disks with container type and owner, while storage disk show -spare and storage disk show -broken pick out spares and failed drives. storage aggregate relocation start -node node0 -destination node1 -aggregate-list aggr1, aggr2 hands aggregates to the HA partner node, and vserver show lists the storage virtual machines that own the volumes.

Volumes and replication

volume show lists volumes per SVM with size and state, and volume show-space splits a volume's usage into user data, metadata and snapshot reserve. volume create -vserver VSERVER -volume VOLUME -aggregate AGGREGATE -size SIZE -junction-path PATH makes a FlexVol volume: the SVM and volume name are the required parameters (the SVM may be left out when only one data SVM exists), the aggregate places it and the junction path mounts it. snapmirror show lists replication relationships with their state, status and health, and lag time is one of its fields. Tab completion works at every level of the hierarchy, and ? lists what can follow.

Data objects

On FlashArray, purevol create NAME --size SIZE makes a block volume and purevol list lists them; on FlashBlade, purefs create --size 1T purefb creates a file system and purefs list shows each one's size, protocols and writable state. purevol snap --suffix golden-v1 golden-rhel9-boot takes a snapshot named after the volume plus the suffix, and purevol copy golden-rhel9-boot.golden-v1 ec2-instance-01-boot turns it into a new volume. Sizes carry a unit suffix such as 2TB, 100G or 32T, and a volume can grow but never shrink.

Hosts and protection groups

A volume reaches a server through a host object: purehost create --iqnlist IQN HOSTNAME registers the initiator, and purevol connect NAME --host HOSTNAME, or purevol connect VOLUME --hgroup GROUP for a cluster, maps the volume with a LUN. Volumes that must be snapshotted together go into one protection group, for example with purevol add orcl26-p1-data orcl26-p1-logs --pgroup pg-orcl26-prod1; a group snapshot keeps write order consistent across them, and each member is cloned under its group, snapshot and volume names.

Array, hardware and health

purearray list prints the array ID and purearray list --space its capacity, data reduction and snapshot space. purehw list reports the status of every component, down to single Ethernet ports, and puredrive list narrows that to the drives. puremessage list lists alert events with severity and component, and purealert list --report 20001 prints one alert with its suggested action. purevol list --csv gives comma-separated output for scripts.

Network and access

FlashBlade clients mount file systems through data VIPs, which the array spreads across its blades and fails over by itself. A subnet comes first with puresubnet create, then each VIP with purenetwork vip create --address 100.127.199.4 --servicelist data data-34-90, and purenetwork list --filter 'services="data"' shows the result. pureadmin create --api-token issues an API token for integrations such as the Splunk add-on, and pureadmin list --api-token --expose shows it again later. purelog create --uri URI NAME forwards the array's syslog messages to a remote server.

Views and how clients see data

A view exposes a directory to clients over the protocols named in view create --path PATH --protocols NFS; a list such as NFS,SMB opens the same data to several protocols, as in view create --path PATH --protocols NFS,SMB --share SHARE --policy-id ID --create-dir. Every view has a view policy, made with viewpolicy create --name NAME --flavor NFS --auth-source PROVIDERS, and one policy can serve any number of views. Later changes go through view modify --id VIEW_ID --protocols PROTOCOLS, which selects the view by ID rather than by path. Clients connect to virtual IPs from a pool created with vippool create --ip-ranges IP_RANGES --subnet-cidr CIDR, and vms modify --mgmt-data-vip MGMT_DATA_VIP adds an optional VMS management address on the data subnet.

Tenants, roles and limits

tenant create --name NAME --capacity-rules CAPACITY_RULES creates a tenant with soft and hard capacity limits. Administrative access is built from roles and realms: role create --name NAME --tenant-ids IDs scopes a role to one or more tenants, realm create --name NAME --object-types OBJECT_TYPES groups object types into a permission realm, and manager create --username NAME --role-id ID adds a VMS manager account under a role. quota create --name NAME --path PATH --hard-limit LIMIT caps a directory, and qospolicy create --name NAME --policy-type VIEW --is-gold gives the workloads of a view QoS policy priority for CPU and memory. Per-client NFS metrics are switched on and charted on the Analytics page of the VAST Web UI; the 5.5 CLI reference lists no command for them.

Snapshots and clones

snapshot create --path PATH --name NAME records everything under a directory at that moment, and snapshot create --path PATH --name NAME --enable-lock makes the snapshot indestructible until the cluster's indestructibility mechanism is unlocked. snapshot clone --id ID --name NAME --target-path PATH --target-tenant-id ID turns a local snapshot into a new directory that is writable at once, so data comes back without waiting for a copy. Scheduled snapshots come from a protection policy, protectionpolicy create --clone-type LOCAL --name POLICY_NAME --schedule SCHEDULE --snapshot-prefix PREFIX, applied to a directory with protectedpath create --name NAME --source-dir SOURCE_DIR --local-tenant-id ID --protection-policy-id PROTECTION_POLICY_ID; the replication clone types also send the snapshots to an async or S3 replication peer.