Chapter 27 — Text processing: `grep`, `sed`, `awk`, and pipes

In one sentence

grep finds lines, sed transforms them, and awk treats each line as structured columns you can compute over — and chained together with pipes, these three tools can replace an enormous amount of custom code.

Why this matters

Logs are text. Config files are text. Command output is text. Fluency with these three tools turns "I'd need to write a script for that" into a one-liner you type in ten seconds, constantly, for the rest of your career.

How it actually works

   grep "pattern" file        # print lines matching a pattern
   grep -i "pattern" file      # case-insensitive
   grep -v "pattern" file      # INVERT -- print lines that DON'T match
   grep -r "pattern" dir/      # search recursively through a directory
   grep -c "pattern" file      # just COUNT matching lines
   grep -E "regex|pattern" file # extended regex (alternation, etc.)

   sed 's/old/new/' file       # substitute the FIRST match per line
   sed 's/old/new/g' file      # substitute ALL matches per line (global)
   sed -n '5,10p' file         # print only lines 5 through 10

   awk '{print $1}' file       # print the FIRST whitespace-separated
                               # column of every line
   awk -F, '{print $2}' file    # same, but split on commas instead
   awk '$3 > 100 {print $1}' file  # print column 1, but only for lines
                               # where column 3 is greater than 100

Chained together, this is where the real power shows up:

   # Top 5 most common words in a text file
   cat file.txt | tr ' ' '\n' | sort | uniq -c | sort -rn | head -5

   # Every unique IP address in a web server log, with request counts
   awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10

   # Every ERROR line from today's log, with the message only
   grep "ERROR" app.log | awk -F'ERROR: ' '{print $2}'

Real-world example

   A now-famous 1986 exchange, documented in the essay collection "A
   Quarter Century of Unix" and widely retold since, involved Donald
   Knuth (asked to demonstrate his "literate programming" style)
   writing a multi-PAGE, carefully structured program to solve a simple
   task: read a text file, find the N most frequently occurring words,
   print them sorted by frequency. Doug McIlroy -- one of Unix's original
   designers, and the person credited with inventing the PIPE itself --
   was asked to review it, and responded with a SIX-LINE shell pipeline
   that did the exact same job:

     tr -cs A-Za-z '\n' | tr A-Z a-z | sort | uniq -c | sort -rn | sed ${1}q

   (translate non-letters to newlines, one word per line -> lowercase
   everything -> sort -> count unique occurrences -> sort by count,
   descending -> take the top N)

   This became one of the most-cited illustrations of the Unix
   philosophy IN EXISTENCE precisely because it's real, it's short, and
   it demonstrates EXACTLY this chapter's lesson: small, focused tools
   (`tr`, `sort`, `uniq`), each doing ONE thing well, chained together
   with pipes (Chapter 19), can replace substantial custom programs --
   and the resulting pipeline is often easier to read, debug, and modify
   piece by piece than the equivalent bespoke code.

Try it yourself

cd ~/lab/playground
cat > sample.log << 'EOF'
2024-01-01 10:00:01 INFO  request from 10.0.0.1
2024-01-01 10:00:02 ERROR request from 10.0.0.2 failed: timeout
2024-01-01 10:00:03 INFO  request from 10.0.0.1
2024-01-01 10:00:04 ERROR request from 10.0.0.3 failed: refused
2024-01-01 10:00:05 INFO  request from 10.0.0.1
EOF

grep ERROR sample.log
grep -c ERROR sample.log
sed 's/ERROR/PROBLEM/' sample.log
awk '{print $4}' sample.log             # just the "from"/IP-ish column
awk '/ERROR/ {print $4, $NF}' sample.log  # IP + last field, ERROR lines only

# Reproduce a version of the McIlroy pipeline yourself
echo "the quick brown fox the lazy dog the fox" | \
  tr ' ' '\n' | sort | uniq -c | sort -rn

Common mistakes

Reaching for a full script before trying a one-line pipeline. For text-shaped problems, try grep/sed/awk/sort/uniq first — it's often genuinely faster to write AND to read later.
Forgetting sed's substitution is per-line by default. Without the g flag, only the FIRST match on each line is replaced — a very common source of "why didn't this replace everything" confusion.
Using awk for arbitrarily complex logic. It's brilliant for column-oriented, line-by-line processing — past a certain complexity, a proper script (Chapter 39) is more maintainable.

Check yourself

What does grep -v do, and when is it useful?
Why does sed 's/a/b/' sometimes seem to "not work" on a line with multiple matches, and what fixes it?
What made the McIlroy/Knuth story such an enduring illustration of the Unix philosophy?

Chapter 27 — Text processing: grep, sed, awk, and pipes