Sorting and Cutting Text with sort, cut, uniq

Analogy categoryAnalogies

Many real shell tasks — finding the most frequent IP address in a log, extracting a single column from a CSV, or listing unique values in a field — don't need a full programming language like awk; they're solved faster and more readably by chaining three small, single-purpose tools: sort to reorder lines, cut to slice out columns, and uniq to collapse or count adjacent duplicate lines. Understanding how these three compose is one of the highest-leverage skills in everyday shell work, embodying the Unix philosophy of small tools joined by pipes.

🏏

Cricket analogy: Finding the most-dismissed batsman in a season doesn't need a full statistics package — chaining sort to order deliveries, cut to slice out the batsman column, and uniq to count repeats is the small-tools Unix philosophy, like a scorer using three simple ledgers instead of one giant spreadsheet.

sort: ordering lines

sort orders input lines, alphabetically by default. -n switches to numeric comparison (essential — without it, '10' sorts before '9' because plain sort compares character by character); -r reverses the order; -k selects which field to sort by (e.g. -k2,2 sorts by the second field only); and -u removes duplicate lines as part of sorting, similar to piping through uniq afterward but slightly more efficient since it doesn't require a separate pass. -t sets the field delimiter for -k, since sort's default field splitting is whitespace, much like awk's. Combining -k with a numeric suffix like -k5,5n sorts specifically by field 5 as a number, which is common when sorting du or ps output by size or CPU.

🏏

Cricket analogy: sort -n on over counts avoids '10' sorting before '9' as plain text would; -r reverses to show the highest wicket-takers first, -k2,2 sorts by the batsman column, -t, sets a comma delimiter, and -k5,5n sorts a CSV of strike rates numerically by the fifth field, like sorting ps output by CPU.

bash

# Numeric sort (without -n, '10' would sort before '9')
sort -n numbers.txt

# Reverse numeric sort: largest first
sort -rn scores.txt

# Sort by the 3rd colon-delimited field, numerically
sort -t: -k3,3n /etc/passwd

# Sort du output by size (human-readable, so use -h not -n)
du -sh */ | sort -rh | head -10

# cut: extract fields 1 and 3 from a colon-delimited file
cut -d: -f1,3 /etc/passwd

# cut: extract a fixed character range (e.g. columns 1-10)
cut -c1-10 fixedwidth.txt

# uniq: count occurrences of each unique line (input MUST be sorted first)
sort access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -5

# uniq: show only lines that appear more than once
sort emails.txt | uniq -d

cut and uniq: slicing and collapsing

cut extracts portions of each line, either by delimiter-separated field number (-d sets the delimiter, -f selects fields) or by fixed character position (-c). Unlike awk, cut has no concept of collapsing repeated delimiters — cut -d' ' -f1 on text with multiple consecutive spaces produces empty fields, which is often the reason people reach for awk instead when dealing with irregularly-spaced command output. uniq collapses consecutive identical lines into one, but critically it only compares adjacent lines — non-adjacent duplicates are left alone entirely. This is why uniq is almost always preceded by sort in a pipeline: sorting guarantees identical lines become adjacent, so uniq (and its -c count flag, or -d/-u to show only duplicated/unique lines) actually works as intended.

🏏

Cricket analogy: cut -d' ' -f1 on a scorecard with irregular spacing between overs produces empty fields since cut doesn't collapse repeated delimiters like awk would; and uniq -c only counts adjacent identical entries, which is why you must sort the list of dismissed batsmen first so repeats become adjacent before counting.

The classic pattern sort | uniq -c | sort -rn — sort to group duplicates, uniq -c to count them, sort -rn to rank by frequency — is one of the most reused idioms in shell scripting, appearing everywhere from analyzing web server logs for top requesters to counting word frequency in text corpora. It's worth committing to memory.

Running uniq on unsorted input silently produces wrong results without any error — it will only merge duplicates that happen to be adjacent, leaving every other repeated occurrence untouched and uncounted. Always sort before uniq unless you specifically know the input is already grouped.

sort defaults to lexical ordering; use -n for numeric, -r to reverse, -k to sort by a specific field, -t to set the field delimiter.
cut extracts fields with -d (delimiter) and -f (field list), or fixed character ranges with -c; it does not collapse repeated delimiters like awk does.
uniq only removes/counts ADJACENT duplicate lines, so input must be sorted first for correct deduplication or counting.
The idiom sort | uniq -c | sort -rn counts occurrences and ranks them from most to least frequent — extremely common in log analysis.
uniq -d shows only lines that had duplicates; uniq -u shows only lines that appeared exactly once.
For irregularly whitespace-separated data, prefer awk's field splitting over cut, which treats every delimiter occurrence literally.