Space and inode usage 4
Command What it does
lfs df -hFilesystem usage Shows space per OST and MDT; one nearly full OST is enough to make writes fail while others have room. lfs df -iInode usage Counts used and free inodes on each MDT and OST rather than bytes. lfs df --pool ssd /mnt/testfsUsage of one OST pool Restricts the report to the OSTs that belong to the named pool. lfs df -v /mnt/testfsInclude disconnected targets Lists every MDT and OST of that filesystem, also targets the client has no connection to at the moment.
Servers and targets 10
lctl dl is the short name of lctl device_list; both list the local Lustre devices.
Command What it does
lfs check serversCheck server status lfs check allCheck MGT, MDT and OST status Sends an RPC to each server in scope; give a mount path to limit the check to that filesystem. lfs ostsList OSTs Answers from cached status, usually under 30 seconds old, instead of asking each server. lfs mdtsList MDTs lfs getnameShow filesystem instances Without a path every Lustre mount point is reported; pass paths to ask about particular mounts. lctl dlList devices Device names, for example the OST names on an OSS, appear in column four of the listing. lctl get_param osc.*.ost_conn_uuidWhich server serves each OST Run on a client; each line ends with the NID of the server that currently serves that OST. lctl get_param versionLustre version on this node Works on clients and servers alike and prints a single version= line. lfs --versionVersion of the lfs tool Reports the build of the lfs utility, which can differ from the loaded kernel modules. lctl lustre_build_versionKernel module build version Shows the build of the Lustre kernel modules, which lfs --version does not report.
Reading file layouts 5
Command What it does
lfs getstripe /mnt/lustre/fooShow striping Lists the stripe count and stripe size, then one line per object naming the OST that holds it. lfs getstripe -v /mnt/lustre/file1Detailed object allocation lfs getstripe --mdt-index /mnt/lustre/remote_dir1MDT holding a directory Prints only the index number of the MDT that stores the remote sub-directory. lfs getstripe -d /mnt/lustre/domdirShow a directory's default layout Shows the layout that files created in the directory from now on will inherit. lfs getstripe -I /mnt/testfs/del_compList component IDs Prints the ID of each component of a composite file, one per line.
Setting file layouts 7
Command What it does
lfs setstripe -c STRIPE_COUNT DIRNAMESet striping On an existing directory this becomes the default layout for files created in it afterwards. lfs setstripe -s 4M /mnt/lustre/new_fileCreate file with 4 MB stripes The file must not exist yet; stripe sizes have to be multiples of 64 KB. lfs setstripe -c -1 /mnt/lustre/full_stripeStripe across all OSTs A count of -1 uses every available OST and skips any that are full. lfs setstripe -i START_OST FILENAMEStart striping on a given OST The default of -1 lets the MDS choose the first OST, which keeps space and load balanced. lfs setstripe -C STRIPE_COUNT FILENAMEOverstripe a file Like -c, but places several stripes on one OST when the count is higher than the number of OSTs. lfs setstripe -d /mnt/lustre/dirRemove directory default striping Files created there afterwards fall back to the filesystem-wide default layout. lfs migrate -c 2 -S 2M /mnt/testfs/3comp_to_normalRe-layout an existing file Copies the data into a plain layout of two 2 MB stripes, replacing its three components.
Composite layouts 4
Command What it does
lfs setstripe -E 256M -c 1 -E 16G -c 4 -E -1 -S 4M -c -1 /mnt/testfs/pfldirProgressive layout on a directory New files start on one OST, widen to four after 256 MB and use all OSTs with 4 MB stripes beyond 16 GB. lfs setstripe --component-del -I 5 /mnt/testfs/del_compDelete a layout component Remove components from the last one backwards, because a hole in the middle of a layout is refused. lfs setstripe -E 1M -L mdt -E -1 /mnt/lustre/domdir/Default Data-on-MDT layout Files created in the directory afterwards keep their first 1 MB on the MDT and the rest on OSTs. lfs setstripe -E 1G -z 64M -E -1 -z 256M /mnt/lustre/fileSelf-extending layout Up to 1 GB the extendable component grows in 64 MB steps, beyond that in 256 MB steps.
Directories across MDTs 9
Command What it does
lfs mkdir -i MDT_INDEX /MOUNT_POINT/REMOTE_DIRCreate a remote directory Everything later created below that directory is stored on the same MDT. lfs mkdir -c MDT_COUNT /MOUNT_POINT/NEW_DIRECTORYCreate a striped directory Meant for single huge directories of 50k entries or more, since striped directories cost extra overhead. lfs mkdir -c 1 -i -1 dir1Place directory by MDT usage With -i -1 the new directory goes to an MDT that has less space in use. lfs setdirstripe -D -c 1 -i -1 dir1Default layout for subdirectories Plain mkdir inside dir1 then places each new subdirectory the same way. lfs getdirstripe testdir1 -DShow default directory striping Prints the stripe count, offset, hash type and max-inherit depth that new subdirectories receive. lfs getdirstripe -m ./remotedirMDT index of a directory Prints just the MDT index, handy for checking a directory before and after migrating it. lfs migrate -m 0 ./remotedirMove directory to another MDT Works on whole directories only and moves their inodes to MDT0000. lfs migrate -m 1,3 /testfs/largedirMigrate onto two MDTs Moves directory entries and inodes to MDT1 and MDT3; file data on the OSTs stays where it is. lfs setdirstripe -c 2 /testfs/testdirRestripe an existing directory Changes how many MDTs an already populated directory spans, available since Lustre 2.14.
Finding files 7
Command What it does
lfs find /mnt/lustreList a whole tree lfs find /mnt/lustre -mtime +30 -type f -printFind files older than 30 days A leading + means that value or more; -type f keeps only regular files. lfs find --obd OST2-UUID /mnt/lustre/Files with objects on an OST Matches every file that has at least one stripe object on the named OST. lfs find /mnt/lustre --pool poolAFind entries in a pool Matches files and directories whose layout names the pool poolA. lfs find /mnt/lustre ! --pool ""Entries in any pool The ! negates the empty pool name, so everything assigned to some pool matches. lfs find -L mdt /mnt/lustreFind Data-on-MDT files Matches entries whose layout includes a component stored on the MDT. lfs find --mirror-count +2 --type f /mnt/testfsFiles with more than two mirrors
Mirrored files 6
Command What it does
lfs mirror create -N -S 4M -c 2 -p flash -N -c -1 -p archive /mnt/testfs/file1Create a two-mirror file Each -N starts a mirror with its own layout, here one in the flash pool and one in archive. lfs mirror extend -N -S 8M -c -1 -p archive /mnt/testfs/file1Add a mirror to a file Works on a plain file too, turning it into a mirrored one; directories are not accepted. lfs mirror split --mirror-id 1 /mnt/testfs/file1Split off a mirror The detached copy becomes a new file named file1.mirror~1 with that mirror as its layout. lfs mirror resync /mnt/testfs/file1Resync all stale mirrors Clears the stale flag on every mirror that missed writes made through another mirror. lfs mirror resync --only 2 /mnt/testfs/file1Resync one mirror Limits the resync to the mirror with ID 2. lfs mirror verify /mnt/testfs/file1Verify mirror contents Checks that all mirrors hold identical data; add -v to see where they differ.
OST pools 8
Pool commands run on the MGS, and that node also needs either the MDT on the same host or a mounted Lustre client.
Command What it does
lctl pool_new FSNAME.POOLNAMECreate an OST pool Run on the MGS; pool names are limited to 15 ASCII characters. lctl pool_add testfs.pool1 OST[0-10/2]Add OSTs to a pool The range with a step adds the even-numbered OSTs 0 to 10 in a single command. lctl pool_remove testfs.flash OST0008 OST000a OST000cRemove OSTs from a pool OST indices are hexadecimal, so OST000a is the eleventh target. lctl pool_destroy testfs.flashDelete a pool Every OST must be removed from the pool before it can be destroyed. lctl pool_list FSNAMEList pools Add .POOLNAME to the filesystem name to list the OSTs inside one pool instead. lfs pool_list my_fs.my_poolList OSTs in a pool Runs on a client and prints the member OSTs of the pool my_pool. lfs setstripe --pool my_pool /mnt/lustre/dirTie a directory to a pool New files and subdirectories in it are then created on OSTs of my_pool. lfs setstripe --pool my_pool -c 2 /mnt/lustre/fileCreate file in a pool Stripes the new file over two OSTs chosen from my_pool only.
Quotas 14
Command What it does
lfs quota /mnt/testfsYour own quota Shows usage and limits for the calling user and that user’s primary group. lfs quota -u bob /mnt/lustreShow user quota Gives totals for the whole filesystem; add -v for a per-target breakdown. lfs quota -u bob -v /mnt/testfsPer-target quota detail Adds a breakdown of usage and limits for each MDT and OST. lfs quota -g eng /mnt/testfsShow group quota lfs quota -p 1 /mnt/testfsShow project quota lfs quota -t -u /mnt/testfsShow grace times Lists the block and inode grace periods that apply to user quotas. lfs setquota -u bob -b 307200 -B 309200 -i 10000 -I 11000 /mnt/testfsSet user limits Lowercase -b and -i set soft limits, uppercase -B and -I hard limits. lfs project -s -p 1 -r /mnt/testfs/dirAssign a project ID Tags the directory with project ID 1 so that a project quota can then limit it. lfs setquota -p 1 -b 307200 -B 309200 -i 10000 -I 11000 /mnt/testfsSet project limits Applies the limits to project 1, which caps the tagged directory tree. lfs setquota -t -u --block-grace 1000 --inode-grace 1w4d /mnt/lustreSet grace times Values are plain seconds or the XXwXXdXXhXXmXXs form, such as 1w4d. lfs setquota -U -b 10G -B 11G -i 100K -I 105K /mnt/testfsDefault quota for all users Sets the default that users fall back to; -G and -P set the group and project defaults. lfs setquota -u bob --default /mnt/testfsReturn user to default quota lctl set_param -P osd-*.testfs1*.quota_slave_dt.enabled=ugpEnable block quotas Run on the MGS; u, g and p switch on user, group and project enforcement for blocks. lctl get_param osd-*.*.quota_slave_*.enabledCheck quota enforcement Shows, per target, which quota types are enforced for blocks and for inodes.
Hierarchical storage 7
Command What it does
lctl set_param mdt.$FSNAME-MDT0000.hsm_control=enabledStart the HSM coordinator Needed on every MDT before a filesystem can be bound to an HSM back end. lfs hsm_archive --archive=5 /mnt/lustre/fooArchive a file Copies the file to the back end with archive ID 5 instead of the default one. lfs hsm_state /mnt/lustre/fooShow HSM state Reports flags such as exists, archived or released together with the archive ID. lfs hsm_release FILE1Release file data Frees the space on Lustre; the only request handled synchronously, without the coordinator. lfs hsm_restore FILE1Restore file data Brings released data back; reading or writing a released file also triggers it. lfs hsm_remove FILE1Remove archived copy Deletes the copy held by the HSM system, not the Lustre file. lctl get_param -n mdt.lustre-MDT0000.hsm.actionsList HSM requests Shows the registered requests of one MDT and their status.
FIDs and changelogs 7
Command What it does
lfs path2fid /mnt/chip/v1_2Get FID of a path Prints the identifier in square brackets, the form that lfs fid2path takes back. lfs fid2path /myth [0x200003ab4:0x435:0x0] [0x20001e863:0x1c1:0x0] [0x20001e863:0x1c2:0x0]Resolve FIDs to paths Run as root on a client, for example to name the files an MDS reports as open. lctl --device lustre-MDT0000 changelog_register --user monitor --mask UNLNK,CLOSERegister a changelog user Run on the MDS; records stay on the MDT until every registered user has cleared them. lfs changelog lustre-MDT0000Read changelog records Optional start and end record numbers after the MDT name narrow the range. lfs changelog_clear lustre-MDT0000 monitor-cl1 3Clear processed records Marks records up to number 3 as consumed by monitor-cl1 so the MDT can purge them. lctl --device lustre-MDT0000 changelog_deregister --user monitorRemove a changelog user lctl get_param mdd.lustre-MDT0000.changelog_usersChangelog users and index Shows the current record index and how far each registered user has read.
Reading and setting parameters 7
Command What it does
lctl get_param OBDTYPE.OBDNAME.PARAMETERGet parameter Use the same name as set_param; -n prints only the value without the parameter name. lctl set_param osc.*.max_dirty_mb=1024Set a parameter until restart Changes the value on this node only and is lost when the target or client is restarted. lctl set_param -P llite.testfs-*.max_read_ahead_mb=16Set a parameter permanently Issued on the MGS and applied to every node the parameter concerns, also after remounts. lctl set_param -P -d osc.*.max_dirty_mbDelete a permanent setting Removes the stored -P value so the default applies again. lctl list_param -RF mdtList MDT parameters -R recurses and -F marks directories, symlinks and writable files with /, @ and =. lctl conf_param testfs.llite.max_read_ahead_mb=16Permanent setting via conf_param Must run on the MGS; the value is stored permanently in the filesystem configuration. lctl --device MGS llog_print paramsList stored -P settings Prints the params log on the MGS, which nodes apply when they mount.
Statistics and tuning 10
Command What it does
lctl get_param llite.*.statsClient operation statistics Counts VFS-level operations for each mounted filesystem; counters that are still zero are left out. lctl get_param osc.testfs-OST0000*.rpc_statsRPC histograms for one OST Includes counts of read and write RPCs in flight from this client to OST0000. lctl get_param mdc.*.rpc_statsMetadata RPC histogram Shows how many modifying metadata RPCs ran in parallel, a clue to application parallelism. lctl get_param osd-*.*.brw_statsBlock I/O histograms Run on an OSS or MDS for histograms of the I/O requests reaching its disks. lctl set_param mdc.*.max_rpcs_in_flight=16Raise metadata RPC concurrency Run on the client; caps parallel metadata RPCs from it to each MDT. lctl get_param osc.*.cur_grant_bytesWriteback space granted per OST Shows how much cache space this client holds in reserve with each OST for buffered writes. lctl get_param jobid_varCheck job stats setting Reports disable while job statistics are off, which is the default. lctl set_param -P jobid_var=SLURM_JOB_IDEnable job stats for Slurm Set on the MGS so every client tags its I/O with the Slurm job ID. lctl get_param mdt.*.job_statsMetadata stats per job Run on an MDS to see metadata operations broken down by job. lctl get_param obdfilter.*.job_statsData stats per job Run on an OSS to see read and write activity broken down by job.
LNet networking 6
Command What it does
lctl list_nidsList network IDs Needs LNet to be running and prints the NIDs of the local node only. lctl ping NIDPing Lustre node Gives up after 120 seconds by default; a negative timeout disables that limit. lctl network upStart LNet Run after loading the lnet module; lctl list_nids then shows the local NIDs. lctl which_nid NIDSPick best remote NID From a node’s list of NIDs, shows the one this host would communicate through. lctl get_param peersShow LNet peers Lists every NID known to this node along with its send-queue state. lctl get_param nisShow network interfaces Reports the queue health of this node’s own network interfaces.
Target maintenance 5
Command What it does
lctl set_param osp.FSNAME-OSTNNNN*.max_create_count=0Stop new objects on a full OST Run on the MDS; new files then land on the remaining OSTs while space is rebalanced. lctl set_param osc.FSNAME-OSTNUMBER-*.active=1Reactivate an OST on clients Needed on clients after a temporary deactivation; the MDS uses the osp form of the parameter. lctl set_param -P osc.testfs-OST000[45].active=0Mark OSTs permanently inactive Targets deactivated this way no longer appear in lfs df output at all. lctl set_param obdfilter.{OST_name}.degraded=1Flag an OST as degraded Run on the OSS during a RAID rebuild so the MDS avoids that OST for new objects; set 0 afterwards. lctl set_param mdt.fs-MDT0000.readonly=1Make an MDT read-only Takes effect at once: clients get EROFS until the value is set back to 0.
Recovery and debugging 6
Command What it does
lctl get_param mdt.testfs-MDT0000.recovery_statusShow recovery status Run on the server that has just mounted the target, for example after a failover. lctl --device LUSTRE_DEVICE_NUMBER abort_recoveryAbort target recovery Use when a client that crashed will never reconnect; lctl dl gives the device number. lctl get_param mdt.*.exports.*.open_filesFiles held open per client Run on the MDS to find processes keeping deleted or growing files open; the output lists FIDs. lctl debug_kernel FILEDump the debug log Writes the in-kernel Lustre debug buffer to the file as text for analysis. lctl get_param debugShow debug mask lctl set_param debug_mb=SIZEResize the debug buffer Sets the upper limit of the in-kernel buffer, counted in MiB.
No row on this page matches that filter.
Notes
Capacity and servers lfs df -h shows used and available space per OST and MDT with a filesystem summary, and lfs df -i the inode counts of the same targets. Because each file's objects sit on particular OSTs, one nearly full OST can make writes fail while the summary still shows free space; lctl set_param osp.FSNAME-OSTNNNN*.max_create_count=0, run on the MDS, keeps new objects off that OST while data is moved elsewhere. lfs osts and lfs mdts list the targets, lfs check servers asks every server for its status, and lfs quota -u bob /mnt/lustre shows one user's usage against the limits.
Striping A Lustre file is spread over OSTs in a layout set when the file is created, and a new file takes the default of its directory. lfs getstripe /mnt/lustre/foo shows a file's stripe count, stripe size and the OST of each object. lfs setstripe -c STRIPE_COUNT DIRNAME sets a directory's default for files created afterwards; a count of -1 stripes over every available OST, as in lfs setstripe -c -1 /mnt/lustre/full_stripe. Files that already exist keep their layout until their data is copied into a new one, which is what lfs migrate -c 2 -S 2M /mnt/testfs/3comp_to_normal does. lfs find /mnt/lustre -mtime +30 -type f -print searches by age and type, and the same command also matches by OST, pool or layout.
Network and parameters lctl list_nids shows this node's Lustre network IDs (NIDs), and lctl ping NID tests whether another node answers over LNet on the fabric of that NID, the quick check of an LNet configuration. lctl dl lists the Lustre devices set up on the node. lctl get_param OBDTYPE.OBDNAME.PARAMETER reads tunables and statistics such as lctl get_param llite.*.stats. A value set with lctl set_param osc.*.max_dirty_mb=1024 lasts until the node restarts; the -P form, issued on the MGS as in lctl set_param -P llite.testfs-*.max_read_ahead_mb=16, is stored and applied again when nodes mount.
Questions
Which command shows how full each OST is? lfs df -h on any client lists every OST and MDT with used and available space and a filesystem summary; lfs df -i reports inodes instead.
Where do OST pool commands and permanent lctl settings have to run? On the MGS. Pool commands such as lctl pool_new FSNAME.POOLNAME must be run there, and settings made with lctl set_param -P llite.testfs-*.max_read_ahead_mb=16 are issued there and passed on to the nodes they concern.
How can I see which Lustre version a node runs? lctl get_param version works on clients and servers. lfs --version reports the lfs tool itself and lctl lustre_build_version the loaded kernel modules.
Are DDN's own EXAScaler management commands listed? No. DDN's product documentation is reached through its customer support portal, which requires a login, so this page keeps to the lfs and lctl commands documented in the public Lustre Operations Manual and the Lustre 2.17 man pages.
Sources and checks
Lustre Software Release 2.x Operations Manual (current edition, covers features through Lustre 2.17) Most rows: lfs and lctl synopses, examples and procedures lfs(1) man page, Lustre 2.17.0 release lfs check all, lfs osts, lfs mdts, version commands lctl(8) man page, Lustre 2.17.0 release lctl ping and device_list lctl-pool_remove(8) man page, Lustre 2.17.0 release Removing OSTs from a pool lctl-pool_destroy(8) man page, Lustre 2.17.0 release Deleting a pool lfs-df(1) man page, Lustre 2.17.0 release lfs df pool and verbose forms, deactivated OSTs Every row on this page was matched to a line in the documentation above on 9 October 2026 . A binding that could not be found there is not listed.
Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.