awk commands

Reference for GNU awk (gawk) 5.4.1, built from the GNU manual "GAWK: Effective AWK Programming" and the gawk(1) man page: command-line options, patterns, field and record splitting, built-in variables, string, numeric and time functions, output and control statements, plus short programs printed in the manual. Options are listed without the leading awk; rows that start with awk are whole commands. Entries marked as gawk extensions may not work in other awk implementations.

172 commands in 11 sections. Checked row by row against GAWK: Effective AWK Programming, Edition 5.4, the GNU manual for gawk 5.4.1 (Info edition shipped in the gawk-5.4.1 release) on . Edited by Jay.

Invoking awk 19

A value given with -v exists before BEGIN runs; a VAR=VALUE argument among the file names is assigned only when awk reaches it, so BEGIN never sees it.

CommandWhat it does
awk 'PROGRAM' INPUT-FILE1 INPUT-FILE2 ...Run a program on filesSingle quotes stop the shell from interpreting the program and hand it to awk as one argument.
awk 'PROGRAM'Read from standard inputWith no file names awk processes whatever is typed or piped in until end-of-file, normally Ctrl-d.
awk -f SOURCE-FILE INPUT-FILE1 INPUT-FILE2 ...Run a program fileSuits programs too long for the command line; any file name can hold the source.
-f SOURCE-FILERead program from a fileLong form --file; repeat it to combine several source files into one program, the usual way to load function libraries.
-F FSSet the field separatorLong form --field-separator; the value is treated like an assignment to FS, so it can be a regular expression.
-v VAR=VALAssign a variable before startLong form --assign; the value is ready inside BEGIN rules, and each further variable needs its own -v.
-e PROGRAM-TEXTPass program text as an optionLong form --source; mixes command-line code with files given by -f, and each piece must be a syntactically whole unit.
-i SOURCE-FILEInclude a source libraryLong form --include; equivalent to an @include line, loads each file only once, and the main program must still come from elsewhere.
-l EXTLoad a dynamic extensionLong form --load; the shared library is located through AWKLIBPATH and the platform's file suffix is added automatically.
-E FILERun script, end option parsingLong form --exec; for #! and CGI scripts: later arguments go straight to the program and VAR=VALUE assignments are refused.
--End of optionsLater arguments go into ARGV even when they begin with a dash, which protects such file names.
-kParse input as CSVLong form --csv; applies RFC 4180 quoting rules in place of RS and FS, and is a fatal error together with --posix.
-bTreat characters as bytesLong form --characters-as-bytes; input and print output are handled as single-byte data rather than by the locale's multibyte rules.
VAR=VALUEAssign a variable between filesSuch an argument is not opened as a file; it is applied when awk reaches it, too late for BEGIN.
SOME_COMMAND | awk -f myprog.awk file1 - file2Read standard input between filesA lone dash names standard input, so the piped data is read after file1 and before file2.
#! /bin/awk -fStart a self-contained scriptGoes on the first line; once the file is executable, typing its name runs it as though awk -f had been used.
-hPrint a usage summaryLong form --help; shows the short and long options gawk understands, then quits.
-VPrint version informationLong form --version; shows which gawk release is installed, the detail a bug report should quote.
-CPrint the licence noticeLong form --copyright; shows the short form of the GNU General Public License, then exits.

Debugging and compatibility options 17

CommandWhat it does
-cCompatibility modeLong form --traditional; switches off the GNU extensions so that gawk acts like Brian Kernighan's awk.
-PStrict POSIX modeLong form --posix; removes every extension POSIX does not allow and takes precedence over --traditional.
-L[VALUE]Warn about dubious constructsLong form --lint[=VALUE]; fatal makes warnings errors, invalid reports only invalid code, no-ext mutes extension warnings.
-tWarn about non-V7 constructsLong form --lint-old; flags anything missing from the original awk of Version 7 Unix.
-d[FILE]Dump global variablesLong form --dump-variables[=FILE]; writes each one's type and last value, sorted, to awkvars.out unless a file name is attached.
-D[FILE]Start the debuggerLong form --debug[=FILE]; interactive by default, or reads debugger commands from the attached file to run unattended.
-gGenerate a gettext templateLong form --gen-pot; writes a portable object template for the strings marked as translatable to standard output.
-ITrace executed byte codesLong form --trace; each internal op code is written to standard error as it runs, with a plus sign in front.
-o[FILE]Pretty-print the programLong form --pretty-print[=FILE]; saves a formatted listing to awkprof.out by default, without running the program or optimizing it.
-p[FILE]Profile the programLong form --profile[=FILE]; records how often each statement and function ran in awkprof.out by default, with optimization off.
-OEnable optimizationsLong form --optimize; already on by default, so it only serves to cancel an earlier -s.
-sDisable optimizationsLong form --no-optimize; turns off the default optimizer pass, which currently does simple constant folding.
-SSandbox modeLong form --sandbox; blocks system(), getline and print redirections and extensions, for running scripts you do not trust.
-MArbitrary-precision arithmeticLong form --bignum; does nothing when gawk was built without MPFR support, a feature the manual calls on parole.
-nRead octal and hex inputLong form --non-decimal-data; input values written in octal or hexadecimal are converted automatically, which can break old programs.
-NUse locale decimal pointLong form --use-lc-numeric; numeric input is parsed with the decimal character of the current locale.
-rAllow interval expressionsLong form --re-interval; intervals in regexps are now the default, so the flag only remains for older scripts.

Patterns and actions 14

CommandWhat it does
[PATTERN] { ACTION }Rule with pattern and actionWithout a pattern the action runs on every record; a pattern alone prints each record it matches.
/REGULAR EXPRESSION/Match records by regexpSelects each record that contains text fitting the expression anywhere within it.
BEGINRun before any inputIts action runs once before the first record is read; a program with only BEGIN rules reads no input.
ENDRun after all inputIts action runs once when the input is used up, so totals and counts are printed here.
BEGINFILERun before each filegawk extension: runs before a file's first record with FILENAME already set; test ERRNO here to skip an unreadable file.
ENDFILERun after each filegawk extension: runs when the last record of a file has been processed, even if the file was empty.
BEGPAT, ENDPATMatch a range of recordsTurns on at a record matching the first pattern and off after one matching the second, both included.
awk '$1 == "on", $1 == "off"' myfilePrint on-to-off blocksRange example: prints every stretch of records from a first field of on through the next off.
EXP ~ /REGEXP/Test a value against a regexpChecks one string, such as a field, rather than the whole record; valid as a pattern or inside if and while.
EXP !~ /REGEXP/Test that a regexp failsTrue when the string on the left holds no match for the expression.
awk '/edu/ && /li/' mail-listRequire two matchesPrints only the records that contain both strings; any Boolean expression can serve as a pattern.
awk '! /li/' mail-listInvert a matchPrints the records in which the string does not occur.
#Start a commentEverything from the number sign to the end of that line is ignored; it need not be in the first column.
awk -v pat="$pattern" '$0 ~ pat { nmatches++ } END { print nmatches, "found" }' /path/to/dataMatch a shell variablePasses the shell value with -v and uses it as a dynamic regexp, avoiding quoting tricks inside the program.

Fields and records 18

Setting FS or OFS does not change a record that has already been read: a plain print still shows the old separators until a field assignment such as $1 = $1 rebuilds it.

With -k (--csv) gawk accepts RFC 4180 data only, so a semicolon-separated file cannot be read that way, and assigning FS, FIELDWIDTHS, FPAT or RS merely produces a warning.

CommandWhat it does
$0Whole input recordHolds the record as read, separators included; assigning to it splits the fields again with the current FS.
$1First fieldA dollar sign plus a number picks a field, and the number may have several digits, as in $127.
$NFLast fieldNF is the field count, so this names the final field however long the record is.
$1 = $1Rebuild the recordForces $0 to be reassembled from its fields joined by OFS, needed before print shows a new output separator.
NFNumber of fieldsUpdated for every record; raising it creates empty fields and lowering it drops the trailing ones and rebuilds $0.
NRTotal records readCounts every record since the program started and keeps counting across all input files.
FNRRecord number in current fileStarts over at each new input file, whereas NR keeps the running total.
FSInput field separatorThe default single space means runs of blanks, TABs and newlines; other values are one character or a regexp.
awk 'BEGIN { FS = ":" } ; { print $2 }'Set FS inside the programAssigning in a BEGIN rule makes the new separator apply from the very first record.
awk -F, 'PROGRAM' INPUT-FILESSplit fields on commasPlain comma splitting from the command line; quoted fields with embedded commas need -k instead.
-F '\t'Split fields on TABsKeep the quotes: unquoted, the shell strips the backslash and awk is handed a plain t.
RSInput record separatorA newline by default; an empty string splits on blank lines, and gawk treats a longer value as a regexp.
BEGIN { RS = "" ; FS = "\n" }Read blank-line-separated recordsEach paragraph becomes one record and every line inside it a field.
awk '{ print $0 }' RS="u" mail-listSet RS on the command lineA VAR=VALUE argument placed before the file name sets the separator for that file and the ones after it.
RTText that ended the recordgawk extension, refreshed on every read with the input that actually matched RS.
FIELDWIDTHSFixed-width field listgawk extension: space-separated column widths, each optionally preceded by a skip count and colon; overrides FS and FPAT.
FPATField content patterngawk extension: a regexp describing what a field contains instead of what separates fields; overrides FS and FIELDWIDTHS.
FPAT = "([^,]*)|(\"[^\"]+\")"Match quoted CSV fieldsA field is either text without commas or a double-quoted string, so commas inside quotes stay in one field.

Reading input with getline 8

CommandWhat it does
getlineRead the next record nowReplaces $0 and the fields and updates NF, NR and FNR; the rest of the current action then continues.
getline VARRead next record into a variableOnly the variable, NR, FNR and RT change; $0 and NF stay as they are, and no rule sees the line.
getline < FILERead a record from a fileSets $0 and NF but leaves NR and FNR alone; the file name is a string expression.
getline VAR < FILERead file line into variableOnly the variable and RT are set; every getline form returns 1 for a record, 0 at end of file, -1 on error.
COMMAND | getlineRead a record from a commandThe left side is executed by the shell and each call fetches the following record of its output into $0.
COMMAND | getline VARRead command output into a variableStores one record of the output without touching $0, as in "date" | getline current_time.
while ((getline line < "file.txt") > 0)Loop over a file's linesThe manual's read-until-EOF idiom; it advises parenthesizing getline in every control expression because some awk versions require it.
close(FILENAME [, HOW])Close a file or pipeNeeded before rereading a file or rerunning a command; the argument must be the exact string used to open it.

Built-in variables 17

CommandWhat it does
FILENAMECurrent input file nameSet to - when reading standard input and empty inside BEGIN, where no file has been opened yet.
ARGCCount of command-line argumentsOne more than the last ARGV index; options and the program text are not counted.
ARGVCommand-line argument arrayStarts at index 0 and ends one below ARGC; holds file names and VAR=VALUE assignments, not options or program text.
ENVIRONEnvironment variable arrayIndexed by variable name, so ENVIRON["HOME"] returns the home directory path.
ERRNOLast system error textgawk extension: filled in when getline, a redirection or close() fails, and cleared before each input file is opened.
RSTARTStart of the last matchSet by match() to the position where the matched text begins, or 0 when nothing matched.
RLENGTHLength of the last matchSet by match() to the number of characters matched, or -1 when nothing matched.
IGNORECASECase-insensitive matching switchgawk extension: when nonzero, regexp matches and string comparisons ignore case; array subscripts and single-character FS do not.
CONVFMTNumber-to-string conversion formatUsed when a number is turned into a string in an expression; the default is %.6g.
SUBSEPMultidimensional subscript separatorJoins the parts of an index such as foo["A", "B"] into one string; the default is \034.
PROCINFORunning program informationgawk extension: an array with elements such as "pid", "version", "FS", "strftime" and "sorted_in".
FUNCTABTable of function namesgawk extension covering built-in, your own and extension-provided functions; deleting or assigning its elements is a fatal error.
BINMODEBinary I/O modegawk extension for non-POSIX systems: 1, 2 or 3 selects binary input, output or both; "r", "w" and "rw" also work.
LINTRuntime lint switchgawk extension: a true value acts like --lint from inside the program, and "fatal" turns the warnings into errors.
PRECWorking precision in bitsgawk extension for -M arithmetic; floating-point values are computed to this many bits, 53 unless changed.
ROUNDMODEArbitrary-precision rounding modegawk extension for -M arithmetic; the default "N" is the IEEE 754 round-to-nearest-even mode.
TEXTDOMAINText domain for translationsgawk extension: names the message catalog used for translatable strings and dcgettext(); the default is "messages".

String functions 15

CommandWhat it does
length([STRING])Count charactersWithout an argument it measures $0; given an array, it returns the number of elements.
substr(STRING, START [, LENGTH ])Extract a substringPositions count from 1; without a length the result runs to the end of the string.
index(IN, FIND)Find a substring positionGives the character position of the first occurrence, or 0 when the text is absent; no regexps.
split(STRING, ARRAY [, FIELDSEP [, SEPS ] ])Split a string into an arrayReturns the number of pieces; the separator defaults to FS, and gawk can store the separators in a fourth array.
sub(REGEXP, REPLACEMENT [, TARGET])Replace the first matchEdits the target in place, $0 by default, and returns 0 or 1; & in the replacement stands for the matched text.
gsub(REGEXP, REPLACEMENT [, TARGET])Replace every matchWorks on $0 unless a target is named and returns how many substitutions were made.
{ gsub(/Britain/, "United Kingdom"); print }Replace text on every lineManual example: swaps the string in each record, then prints the edited record.
{ sub(/candidate/, "& and his wife"); print }Reuse the matched textThe ampersand inserts whatever the regexp matched, so the first candidate gains the added words.
gensub(REGEXP, REPLACEMENT, HOW [, TARGET])Substitute and return a copygawk extension: the original stays unchanged, HOW is "g" or a match number, and \1 to \9 reuse parenthesized groups.
match(STRING, REGEXP [, ARRAY])Locate a regexp matchReturns the starting position or 0 and sets RSTART and RLENGTH; gawk can fill an array with parenthesized groups.
sprintf(FORMAT, EXPRESSION1, ...)Format into a stringBuilds the text printf would produce and returns it without printing anything.
tolower(STRING)Convert to lowercaseReturns a copy with uppercase letters changed; other characters are left alone.
toupper(STRING)Convert to uppercaseReturns a copy with lowercase letters raised; digits and punctuation pass through.
strtonum(STR)Parse a number with base prefixgawk extension: a leading 0 means octal and 0x hexadecimal, which plain string-to-number conversion ignores.
asort(SOURCE [, DEST [, HOW ] ])Sort array valuesgawk extension: renumbers the sorted values from 1 and returns the element count; asorti() sorts the indices instead.

Numeric and time functions 12

CommandWhat it does
int(X)Truncate to an integerCuts off the fractional part, rounding toward zero for negative values as well as positive ones.
sqrt(X)Square rootReturns the positive root; a negative argument makes gawk print a warning.
exp(X)Exponential functionRaises e to the given power and reports an error when the result is out of range.
log(X)Natural logarithmNeeds a positive argument; otherwise the result is NaN and gawk warns for negative input.
sin(X)Sine of an angleThe argument is in radians, not degrees.
cos(X)Cosine of an angle
atan2(Y, X)Arctangent of a quotientResult is in radians; atan2(0, -1) is a handy way to obtain pi.
rand()Random number below oneUniformly distributed from 0 up to but not including 1; the sequence repeats on every run unless srand() is called.
srand([X])Seed the random generatorWithout an argument the current date and time become the seed; the previous seed is returned.
systime()Current time as a timestampgawk extension: counts seconds from the system epoch, the start of 1970 in UTC on POSIX systems.
strftime([FORMAT [, TIMESTAMP [, UTC-FLAG] ] ])Format a timestampgawk extension: defaults to the current time and a date-like format; a true third argument gives UTC.
mktime(DATESPEC [, UTC-FLAG ])Build a timestamp from a dategawk extension: the date string lists year, month, day, hour, minute and second; the result is -1 if it cannot be converted.

Output 19

printf adds no newline of its own and ignores OFS and ORS; only print uses the two output separators.

CommandWhat it does
print ITEM1, ITEM2, ...Print a list of itemsCommas put OFS between the items and ORS is added at the end; parenthesize the list when an item uses >.
printPrint the current recordWith no items it behaves as print $0.
printf FORMAT, ITEM1, ITEM2, ...Print with a format stringNo newline is added and OFS and ORS are ignored, so end the format with \n when a line break is wanted.
%dDecimal integer format%i is an equivalent spelling kept for ISO C compatibility.
%sString formatA precision such as %.3s caps how many characters of the string are printed.
%fFloating-point formatThe precision sets the digits after the decimal point, so printf "%4.3f", 1950 gives 1950.000.
%gCompact number formatChooses scientific or floating-point notation, whichever is shorter; %G uses an uppercase E.
%%Literal percent signTakes no argument from the list, and modifiers have no effect on it.
awk '{ printf "%-10s %s\n", $1, $2 }' mail-listPrint an aligned tableThe minus sign left-justifies the first column in a width of 10 characters.
OFSOutput field separatorPlaced between the items of a print statement that are separated by commas; a single space by default.
ORSOutput record separatorWritten after every print statement; a newline unless changed.
OFMTNumber format for printControls how print turns numbers into text; the default is %.6g.
awk 'BEGIN { OFS = ";"; ORS = "\n\n" } { print $1, $2 }' mail-listChange both output separatorsOutput fields are joined with a semicolon and each record is followed by an empty line.
print ITEMS > OUTPUT-FILEWrite output to a fileThe file is emptied on the first write only; later prints to the same name append while it stays open.
print ITEMS >> OUTPUT-FILEAppend output to a fileExisting contents are kept, and the file is created when it does not exist.
print ITEMS | COMMANDPipe output to a commandThe command is a string run by the shell; the pipe stays open until awk exits or close() is called with that string.
print "Serious error detected!" > "/dev/stderr"Print to standard errorgawk handles this file name itself; the double quotes around it are required.
fflush([FILENAME])Flush buffered outputWith no argument every open output file and pipe is flushed; returns 0 on success.
system(COMMAND)Run a shell commandWaits for the command to finish and returns its exit status; disabled by --sandbox.

Control flow and arrays 17

CommandWhat it does
if (CONDITION) THEN-BODY [else ELSE-BODY]Conditional statementThe condition counts as false when it is zero or the empty string; the else branch may be left out.
while (CONDITION) BODYLoop while a condition holdsThe test comes first, so the body may run zero times.
do BODY while (CONDITION)Run a loop at least onceThe body executes before the condition is first checked, then repeats while it stays true.
for (INITIALIZATION; CONDITION; INCREMENT) BODYCounted loopRuns the initialization once, then repeats body and increment for as long as the condition is true.
for (VAR in ARRAY) BODYLoop over array indicesVisits every index once in an order awk does not define; gawk can fix it through PROCINFO["sorted_in"].
switch (EXPRESSION) { case VALUE OR REGULAR EXPRESSION: CASE-BODY default: DEFAULT-BODY }Branch on a valuegawk extension: cases are constants or regexps tried in order, and execution falls through until a break.
breakLeave the innermost loopJumps out of the enclosing for, while or do loop, and is also what ends a switch case.
continueSkip to the next iterationThe remaining statements of the current pass are not executed and the loop moves on at once.
nextSkip to the next recordStops all processing of the current record, so no later rule sees it.
nextfileSkip to the next fileAbandons the current data file, updates FILENAME and resets FNR before the first rule runs again.
exit [RETURN CODE]Stop the programRemaining input is ignored but an END rule still runs; the optional value becomes the exit status.
SELECTOR ? IF-TRUE-EXP : IF-FALSE-EXPConditional expressionEvaluates only one of the two branches, as in C; x >= 0 ? x : -x gives an absolute value.
function NAME([PARAMETER-LIST])Define a functionDefinitions may sit anywhere between rules, even after the first call, because awk reads the whole program first.
return [EXPRESSION]Return from a functionHands control, and optionally a value, back to the caller.
INDX in ARRAYTest for an array indexYields 1 or 0 without creating the element, unlike a comparison against ARRAY[INDX].
delete ARRAY[INDEX-EXPRESSION]Remove one array elementAfterwards the element is gone as if it had never been assigned, and an in test returns 0.
delete ARRAYClear a whole arrayRemoves every element in one statement, faster than deleting them in a loop.

One-liners from the manual 16

CommandWhat it does
awk '/li/ { print $0 }' mail-listPrint lines containing a stringThe slashes mark li as a regexp; since printing is the default action, the braces could be dropped.
awk 'length($0) > 80' dataPrint lines over 80 charactersA comparison used as the pattern with no action, so matching records are simply printed.
awk '{ if (length($0) > max) max = length($0) } END { print max }' dataPrint the longest line lengthTracks the maximum while reading and reports it from the END rule.
awk 'NF > 0' dataRemove blank linesA line with no fields has NF equal to 0, so only non-empty lines are printed.
awk 'BEGIN { for (i = 1; i <= 7; i++) print int(101 * rand()) }'Print seven random integersEach value lies between 0 and 100 inclusive; needs no input because everything happens in BEGIN.
ls -l FILES | awk '{ x += $5 } END { print "total bytes: " x }'Sum file sizesAdds up the fifth column of the listing, which holds each file's size in bytes.
awk -F: '{ print $1 }' /etc/passwd | sortList login names in orderSplits each password-file entry on colons, prints the first field and lets sort arrange the output.
awk 'END { print NR }' dataCount lines in a fileNR still holds the number of records read when the END rule runs.
awk 'NR % 2 == 0' dataPrint even-numbered linesChange the test to NR % 2 == 1 to get the odd-numbered lines instead.
awk '/li/ { print $1, $NF }' mail-listPrint first and last fieldsSearches the whole record for the string, then prints two fields of each matching line.
awk -F: '$5 == ""' /etc/passwdFind users without a full namePrints the password-file entries whose fifth colon-separated field is empty.
awk '{ print $n }' n=4 inventory-shipped n=2 mail-listPrint a different field per fileEach n=VALUE argument takes effect just before the file that follows it is opened.
awk 'BEGIN { print "Don\47t Panic!" }'Print an apostrophe\47 is an escape that produces a single quote without fighting the shell's quoting.
awk '{ print "Field number one: " $1 }' mail-listJoin text and a fieldTwo expressions written side by side are concatenated; awk has no operator symbol for it.
{ print FNR, $0 }Number lines per filePuts the line's number within its own file in front of each line.
BEGIN { FS = ":" } { print $1 | "sort" }Sort login names inside awkPipes the first colon-separated field of each line to sort from within the program.

Notes

Fields, separators and the two options that matter

awk splits each line into fields on whitespace, numbered $1, $2… with $0 the whole line. -F FS changes the input separator — -F: for /etc/passwd, -F, for simple comma-separated data — and sets the FS variable. -v VAR=VAL passes a shell value into the program before it runs, which is the correct way to use a shell variable inside single quotes. -f SOURCE-FILE reads the program from a file once a one-liner has grown past a line. Output fields are joined by OFS, which defaults to a space, so -v OFS=, turns a print of several fields into comma-separated output.

Variables you read inside the program

FILENAME is the current input file — useful with several files on the command line — and the record counters NR and FNR (total and per-file) let NR==FNR distinguish the first file from the rest in two-file joins. RT holds the text that terminated the current record. ENVIRON is an array of the environment, so ENVIRON["HOME"] reads a variable without -v. RSTART and RLENGTH hold the position and length of the last match(), and IGNORECASE set to 1 makes string comparisons and regex matches case-insensitive in gawk.

CSV and gawk extensions

Proper CSV with quoted commas defeats -F,; gawk 5.3 added -k to enable CSV parsing, and older versions use FPAT with a pattern that matches quoted fields. FIELDWIDTHS splits fixed-width records by column counts instead of separators. -M switches to arbitrary-precision arithmetic when gawk is built with MPFR, -L[VALUE] warns about non-portable constructs when a script must also run on mawk or BSD awk, and -c runs in compatibility mode with the extensions off.

Questions

How do I use a shell variable inside an awk program?

Pass it with -v: awk -v name="$NAME" '$1 == name' file. Do not interpolate it into the quoted program.

How do I print a column with a different separator?

Set OFS: awk -F: -v OFS="\t" '{print $1, $3}' /etc/passwd. The comma in print uses OFS between fields.

How do I handle CSV with quoted commas?

Use gawk's -k option (5.3 and later) or set FPAT to a pattern for quoted fields. Plain -F, splits inside quotes.

What is the difference between NR and FNR?

NR counts records across all input files; FNR resets for each file. NR==FNR is true only while reading the first file.

Sources and checks

Every row on this page was matched to a line in the documentation above on . A binding that could not be found there is not listed.

Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.