Combining grep, awk, and sed

Analogy categoryAnalogies

Picking the Right Tool for Each Stage

grep, awk, and sed overlap in capability but each has a sweet spot: grep excels at fast line filtering with regex matching (especially with -E, -v, -c, and -o), sed excels at targeted line-level or block-level text substitution and extraction, and awk excels at anything involving fields, arithmetic, or aggregation across records. A well-designed pipeline uses grep early to cheaply discard irrelevant lines before the more expensive field-splitting or stateful work in awk or sed runs, which reduces the amount of data later stages have to process.

🏏

Cricket analogy: A team's fielding coach positions a first slip to catch the easy edges (grep filtering obvious matches) before the more specialized wicketkeeper (awk) handles the nuanced decisions requiring judgment, an efficient division of labor.

bash

# Pipeline: grep filters cheaply first, awk aggregates, sed reformats the final output
grep 'ERROR' app.log \
  | awk -F'|' '{ count[$2]++ } END { for (svc in count) print svc, count[svc] }' \
  | sort -k2 -rn \
  | sed 's/^/service: /'

# grep -c for a quick count vs awk for grouped counts
grep -c 'ERROR' app.log             # total ERROR lines
awk '/ERROR/{c[$2]++} END{for(k in c) print k, c[k]}' app.log  # grouped by field 2

When One Tool Alone Would Be Awkward

Some tasks are technically possible in a single tool but far clearer as a short pipeline: extracting only the lines inside a specific config block with sed, then using awk to sum a numeric field within just that block, avoids writing complex sed arithmetic or awk range-tracking logic in one program. Similarly, grep -l to find which files contain a pattern, followed by a loop running sed -i only on those specific files, is clearer and safer than trying to make a single sed invocation conditionally skip entire files based on content it hasn't fully read yet.

🏏

Cricket analogy: Rather than asking one all-rounder to open the batting, bowl the death overs, and captain the side, a team splits those roles among specialists, just as a pipeline splits filtering (grep), transforming (sed), and aggregating (awk) among specialized tools.

A useful rule of thumb: use grep to decide WHICH lines or files matter, sed to reshape TEXT on those lines (substitution, deletion, extraction), and awk when you need to reason about FIELDS, numbers, or accumulate state ACROSS records. If a one-liner starts fighting the tool (deeply nested sed branching, or awk regex gymnastics that grep would do trivially), that's a signal to split it into a pipeline.

Performance and Correctness in Pipelines

Each stage in a shell pipeline runs as a separate process connected by pipes, so grep pattern file | awk '...' streams data and starts processing before grep even finishes reading the whole file, which is efficient for large files, but it also means error handling needs care: a failure in an early pipeline stage doesn't automatically stop later stages, and by default only the exit status of the last command in the pipeline is checked unless set -o pipefail is enabled. When performance matters on very large files, doing the filtering with grep's optimized string-matching engine before handing a much smaller stream to awk's more general (and slower) field-splitting regex engine is measurably faster than making awk do all the filtering and field work itself.

🏏

Cricket analogy: A relay of fielders returning the ball from the boundary to the keeper, where each fielder starts moving before the previous throw fully lands, mirrors how pipeline stages run concurrently, streaming data rather than waiting for the prior stage to finish entirely.

By default, bash only reports the exit status of the LAST command in a pipeline ($? after grep ... | awk ... reflects awk's exit code, not grep's). If an early stage like grep fails (e.g., file not found) but a later stage still exits 0, the pipeline as a whole appears to succeed. Enable set -o pipefail in scripts where any stage failing should cause the whole pipeline to be treated as failed.

grep is best for fast line filtering; sed for targeted text substitution/extraction; awk for field-based logic and aggregation.
Filtering early with grep before handing a smaller stream to awk or sed reduces the work later stages must do.
Some tasks are technically possible in one tool but clearer and safer as a short multi-tool pipeline.
grep -l plus a loop with sed -i is often safer than trying to make one sed invocation conditionally skip whole files.
Pipeline stages run as concurrent, streaming processes connected by pipes, not sequentially one-at-a-time.
By default only the last command's exit status is checked in a pipeline; set -o pipefail makes any stage's failure propagate.
Choosing the right tool per stage, rather than forcing one tool to do everything, usually yields clearer and faster scripts.