awk Basics

awk is a full pattern-scanning and text-processing language, named after its creators Aho, Weinberg, and Kernighan. Where grep finds lines and sed transforms text, awk's superpower is treating each input line as a record automatically split into fields, which makes it exceptionally good at working with column-oriented data: log files, CSVs, ps or df output, and any whitespace- or delimiter-separated text. A typical awk program is a series of pattern { action } pairs — for every input line, awk tests each pattern and runs the corresponding action if it matches, with an empty pattern matching every line.

By default awk splits each line (a 'record') on runs of whitespace into fields referenced as $1, $2, and so on, with $0 representing the entire line. NF (Number of Fields) holds the count of fields in the current record, so $NF conveniently refers to the last field regardless of how many there are. NR (Number of Records) is a running counter of lines processed so far, useful for numbering output or skipping headers. The field separator defaults to whitespace but is changed with -F, e.g. -F, for comma-separated data, or by setting the FS variable inside a BEGIN block. BEGIN and END blocks run once before any input is read and once after all input has been processed respectively, making them the natural place to initialize variables or print summary totals.

A pattern before an action can be a regex (/ERROR/ {print}), a comparison ($3 > 100), a range, or combinations with &&/||. Omitting the action defaults to {print $0}, so awk '/ERROR/' app.log behaves much like grep ERROR app.log. awk's arrays are associative (keyed by string, similar to a hash map/dictionary) rather than strictly numeric-indexed, which is what makes the counting and grouping idiom count[$1]++ so natural — it works whether $1 is a number, a username, or an IP address. printf gives formatted output control (%s, %d, %.2f, field widths) that plain print lacks, which matters when producing aligned reports or fixed-decimal numbers.