37. Performance tuning: pprof, trace, and benchmark methodology
37.1 The four profile types that matter most
| Profile | Endpoint / flag | Answers |
|---|---|---|
| CPU | /debug/pprof/profile?seconds=30 |
Where is CPU time going? |
| Heap | /debug/pprof/heap |
What's allocating memory, and how much is live? |
| Goroutine | /debug/pprof/goroutine |
How many goroutines, and where are they stuck? |
| Block/Mutex | /debug/pprof/block, /mutex |
Where is contention (lock waits) happening? |
37.2 go tool trace: the scheduler's-eye view
go test -trace=trace.out ./...
go tool trace trace.out
Where pprof aggregates "how much time in this function," trace
shows a literal timeline of every goroutine's state transitions
(running, runnable, blocked, GC) — the right tool when the question is
"why did this specific request stall for 200ms" rather than "what's
hot."
37.3 Benchmark methodology that doesn't lie to you
- Run with
-benchtime=3xor longer for stable numbers; a single quick run on a laptop with other processes competing for CPU is noise. - Always include
-benchmem; allocation count is often the more actionable number than raw nanoseconds. - Use
benchstat(golang.org/x/perf/cmd/benchstat) to compare two runs statistically instead of eyeballing a percentage difference that might be within noise.
go test -bench=. -benchmem -count=10 ./... > old.txt
# make your change
go test -bench=. -benchmem -count=10 ./... > new.txt
benchstat old.txt new.txt
37.4 Real-world example
"We think the new caching layer made things faster" is a hypothesis, not
a fact, until benchstat on ten runs each of old vs. new shows a
statistically significant delta rather than two numbers that happen to
differ once.
Applied in the series: Linux guide, Chapter 81: two planted bugs found with pprof. A regexp compiled per request turned out to be 86% of the handler's CPU, and a goroutine leak showed up as 489 goroutines stuck on one line.