Part VIII — Professional Go: debugging, refactoring, operations

This Part is what turns "I know Go" into "I can own Go systems." The language is only half the job; the professional skill is diagnosing production behavior, changing code safely, proving behavior with tests, shipping repeatable releases, and operating services with enough telemetry that incidents become debuggable instead of mysterious.

58. Debugging Go in production: Delve, pprof, traces, stack dumps, GODEBUG

58.1 Read the panic stack trace first

A Go panic already tells you the failing goroutine, the call stack, and usually the exact nil pointer / bounds / type assertion problem. Start at the first frame in your code, not at runtime/panic.go.

panic: runtime error: invalid memory address or nil pointer dereference

goroutine 42 [running]:
example.com/app/user.(*Service).Get(...)
	/app/user/service.go:87
example.com/app/httpapi.handleGetUser(...)
	/app/httpapi/users.go:33

Debugging rule: fix the earliest invalid assumption, not the last function on the stack. If handleGetUser passed a nil service, the bug is probably construction/wiring, not the line that finally dereferenced it.

58.2 Delve for local debugging

go install github.com/go-delve/delve/cmd/dlv@latest
dlv debug ./cmd/server -- --config ./dev.yaml

Useful commands inside Delve:

break user/service.go:87
continue
goroutines
goroutine 42
stack
locals
print req.UserID
next
step

Use Delve when a failure is easy to reproduce locally and the question is "which branch/value got me here?" Use logs, traces, and pprof when the failure only appears under concurrency, load, or production traffic.

58.3 Goroutine dumps for hangs and leaks

When a Go service is "stuck," you want to know where goroutines are parked. Expose net/http/pprof on an internal-only port:

import _ "net/http/pprof"

go func() {
	log.Println(http.ListenAndServe("127.0.0.1:6060", nil))
}()

Then capture:

curl -s http://127.0.0.1:6060/debug/pprof/goroutine?debug=2 > goroutines.txt
go tool pprof http://127.0.0.1:6060/debug/pprof/profile?seconds=30
go tool trace trace.out

Patterns to recognize:

Symptom	Likely cause
Many goroutines blocked on `<-ch`	producer exited, channel never closed, or missing cancellation
Many goroutines in `net/http.(*persistConn).roundTrip`	outbound HTTP calls lack timeout or downstream is slow
Many goroutines waiting on `sync.Mutex.Lock`	lock contention or code doing slow work while holding a lock
Heap grows while goroutine count grows	goroutine leak retaining request data, buffers, or closures

58.4 `GODEBUG`, `GOTRACEBACK`, and runtime knobs

GOTRACEBACK=all ./server         # include all goroutines in crash output
GODEBUG=gctrace=1 ./server       # print GC cycle summaries
GOMEMLIMIT=512MiB ./server       # soft memory limit for the Go runtime
GOGC=50 ./server                 # collect more aggressively than default

These are diagnostic and operational tools, not magic fixes. If gctrace shows constant GC work, the long-term fix is usually to reduce allocations or retained heap, then set GOMEMLIMIT/GOGC intentionally for the deployment environment.

War story. A service "randomly froze" every few hours. CPU was low, memory was fine, and logs stopped. A goroutine dump showed hundreds of goroutines blocked on jobs <- item: the worker pool had returned after one worker hit an error, but the producer kept sending forever. The fix was not "increase channel buffer"; it was errgroup.WithContext, so failure canceled the whole pipeline and all senders selected on ctx.Done().