Chapter 70 -- Observability and performance: perf, flame graphs, eBPF, and OpenTelemetry

In one sentence

Expert performance work starts with a symptom, measures the bottleneck, and only then reaches for deeper tools like perf, flame graphs, or eBPF.

Why this matters

Most bad performance fixes are plausible guesses. Linux gives you enough visibility to avoid guessing, but the tools have different costs and levels of intrusion. top and vmstat are cheap; strace can slow a busy process; profilers and eBPF tools require care and usually elevated privileges.

How it actually works

Use a ladder:

   1. User symptom: slow endpoint, timeout, error rate, cost spike
   2. Golden signals: latency, traffic, errors, saturation
   3. Host view: CPU, memory, disk, network, run queue
   4. Process view: threads, file descriptors, syscalls, allocations
   5. Code/path view: profiles, flame graphs, traces
   6. Kernel view: scheduler, block I/O, TCP, locks, eBPF probes

Examples:

uptime
vmstat 1 5
pidstat -durh 1 5 2>/dev/null || true
perf top 2>/dev/null

perf answers "where is CPU time going?" eBPF-based tools can answer questions like "which process is opening this file?", "which TCP connects are slow?", or "which kernel function is hot?" without permanently instrumenting the application. OpenTelemetry answers a different question: how requests move through distributed services.

Real-world scenario

A service is slow and CPU is high. Logs say nothing. A CPU profile shows most time in JSON serialization, not the database. The fix is a code/data-shape change, not more replicas. Another day, the same symptom is caused by disk I/O wait from synchronous logging. Same symptom, different measurement, different fix.

Try it yourself

Profile a harmless CPU-heavy command:

openssl speed sha256 2>/dev/null &
DEMO_PID=$!
ps -p "$DEMO_PID" -o pid,comm,%cpu
wait "$DEMO_PID"

If your lab has perf installed and permissions allow it, run:

perf stat openssl speed sha256

Build it in Go

Go has profiling built in: import net/http/pprof and every running service can produce CPU, heap, goroutine, and block profiles on demand. Chapter 81 plants two bugs (a regexp compiled on every request, and a goroutine leak) and finds both: the regexp turned out to be 86% of the handler's CPU, and the leak showed up as 489 goroutines stuck on one line.

Common mistakes

Optimizing the function you dislike instead of the function the profile identifies.
Looking only at average latency and missing tail latency.
Treating observability as "logs only." Metrics, traces, profiles, and events answer different questions.
Running heavy tracing on production without understanding overhead and data sensitivity.

Check yourself

Why should you define the symptom before picking a tool?
What does a flame graph help you see?
How are distributed traces different from host metrics?