65. Observability in Go services: logs, metrics, traces, health, profiling

65.1 Logs: structured and boring

logger.InfoContext(ctx, "request completed",
	"method", r.Method,
	"path", r.URL.Path,
	"status", status,
	"duration_ms", time.Since(start).Milliseconds(),
	"request_id", requestIDFrom(ctx),
)

Good logs answer: what happened, to whom, where, when, how long, and with which correlation ID. They should not leak secrets.

65.2 Metrics: RED and USE

For request/response services, start with RED:

• Rate: requests per second.
• Errors: error count/rate by route/status class.
• Duration: latency histogram, not just average.

For resources, use USE:

• Utilization: CPU, memory, disk, connection pools.
• Saturation: queue length, goroutines, blocked workers.
• Errors: failed DB calls, rejected jobs, dropped messages.

Avoid high-cardinality labels like user ID, raw URL, email, token, request ID, or arbitrary error string.

65.3 Health, readiness, and liveness are different

mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
	w.WriteHeader(http.StatusOK) // process is alive
})

mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
	ctx, cancel := context.WithTimeout(r.Context(), 500*time.Millisecond)
	defer cancel()
	if err := db.PingContext(ctx); err != nil {
		http.Error(w, "not ready", http.StatusServiceUnavailable)
		return
	}
	w.WriteHeader(http.StatusOK)
})

Liveness answers "should the orchestrator restart me?" Readiness answers "should I receive traffic?" Do not make liveness depend on every downstream service, or a database outage can cause restart storms.

65.4 Tracing and pprof

Trace IDs connect logs across services; pprof explains where CPU, memory, goroutines, blocking, and mutex contention are happening inside one process. Keep pprof internal-only and access-controlled.

Real-world example. A p99 latency regression with normal CPU often shows up in a block or mutex profile, not a CPU profile: goroutines are waiting, not burning cycles. That distinction changes the fix from "optimize code" to "reduce lock scope / increase pool / remove shared state."