Frontend Labs
⋯
Read as one page Markdown source ↗

29. Instrumenting Orbit with OpenTelemetry

go get go.opentelemetry.io/otel \
       go.opentelemetry.io/otel/sdk \
       go.opentelemetry.io/otel/sdk/metric \
       go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc \
       go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc \
       go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp \
       go.opentelemetry.io/contrib/instrumentation/runtime \
       github.com/exaring/otelpgx \
       github.com/redis/go-redis/extra/redisotel/v9

29.1 SDK setup, configured entirely by standard env vars

internal/platform/telemetry.go:

package platform

import (
	"context"
	"errors"
	"time"

	"go.opentelemetry.io/contrib/instrumentation/runtime"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/propagation"
	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

// SetupOTel installs global tracer/meter providers. Everything is driven by standard env vars:
//   OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, OTEL_EXPORTER_OTLP_ENDPOINT,
//   OTEL_TRACES_SAMPLER(_ARG), OTEL_SDK_DISABLED ...
// so the same binary works in Compose, kind and prod without code changes.
func SetupOTel(ctx context.Context) (func(context.Context) error, error) {
	res, err := resource.New(ctx,
		resource.WithFromEnv(),
		resource.WithTelemetrySDK(),
		resource.WithHost(),
	)
	if err != nil {
		return nil, err
	}

	traceExp, err := otlptracegrpc.New(ctx)
	if err != nil {
		return nil, err
	}
	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(traceExp), // async, batched: never block a request on export
		sdktrace.WithResource(res),
	)

	metricExp, err := otlpmetricgrpc.New(ctx)
	if err != nil {
		return nil, err
	}
	mp := sdkmetric.NewMeterProvider(
		sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metricExp, sdkmetric.WithInterval(15*time.Second))),
		sdkmetric.WithResource(res),
	)

	otel.SetTracerProvider(tp)
	otel.SetMeterProvider(mp)
	// W3C traceparent + baggage: what every other OTel-instrumented hop understands.
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
		propagation.TraceContext{}, propagation.Baggage{}))

	// Go runtime metrics: goroutines, GC, heap, scheduler latency.
	if err := runtime.Start(); err != nil {
		return nil, err
	}

	return func(ctx context.Context) error {
		return errors.Join(tp.Shutdown(ctx), mp.Shutdown(ctx)) // flush on exit
	}, nil
}

⚠️ Caveat (verify): the OTel Go packages move fast. Run go build against current versions and check whether runtime.Start() options or exporter constructors changed. All exporter settings come from the OTEL_* env vars; if spans don't arrive, set OTEL_EXPORTER_OTLP_INSECURE=true and confirm the endpoint scheme.

29.2 Logs that carry the trace

Applications log structured JSON to stdout. The node agent ships it (§30). The only addition is putting the active trace_id/span_id on every log line:

package platform

import (
	"context"
	"log/slog"
	"os"

	"go.opentelemetry.io/otel/trace"
)

type traceHandler struct{ slog.Handler }

func (h traceHandler) Handle(ctx context.Context, r slog.Record) error {
	if sc := trace.SpanContextFromContext(ctx); sc.IsValid() {
		r.AddAttrs(slog.String("trace_id", sc.TraceID().String()),
			slog.String("span_id", sc.SpanID().String()))
	}
	return h.Handler.Handle(ctx, r)
}
func (h traceHandler) WithAttrs(a []slog.Attr) slog.Handler { return traceHandler{h.Handler.WithAttrs(a)} }
func (h traceHandler) WithGroup(n string) slog.Handler       { return traceHandler{h.Handler.WithGroup(n)} }

// NewLogger: JSON to stdout, level from LOG_LEVEL, trace-correlated.
func NewLogger() *slog.Logger {
	var lvl slog.Level
	_ = lvl.UnmarshalText([]byte(Env("LOG_LEVEL", "info")))
	return slog.New(traceHandler{slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: lvl})})
}

Use slog.InfoContext(ctx, ...) / slog.ErrorContext(ctx, ...), which pass the context in. Plain slog.Info can't see the span.

Logging rules that keep log bills and incidents sane:

  • One event per line, JSON, with stable keys (order_id, not "order 123 failed" prose).
  • Never log secrets or PII: tokens, passwords, card numbers, full emails. Redact at the source; the Collector can redact as a second line of defense (§30.3).
  • INFO for state changes, ERROR for things a human may need to act on. Don't log every successful request at INFO in a high-RPS service; metrics and traces cover that.

29.3 Auto-instrumentation: HTTP, Postgres, Redis

// internal/platform/http.go
// Route registers a handler wrapped in an OTel span + http.server.* metrics named after the pattern.
func Route(mux *http.ServeMux, pattern string, h http.HandlerFunc) {
	mux.Handle(pattern, otelhttp.NewHandler(h, pattern))
}

cmd/orders/main.go, the changed lines only:

func main() {
	slog.SetDefault(platform.NewLogger())
	ctx := context.Background()

	shutdownOTel, err := platform.SetupOTel(ctx)
	if err != nil {
		slog.Error("otel setup", "err", err) // telemetry failure must not stop the service
	} else {
		defer func() {
			c, cancel := context.WithTimeout(context.Background(), 5*time.Second)
			defer cancel()
			_ = shutdownOTel(c)
		}()
	}

	pcfg, _ := pgxpool.ParseConfig(platform.MustEnv("DATABASE_URL"))
	pcfg.ConnConfig.Tracer = otelpgx.NewTracer()           // a span per query
	db, _ := pgxpool.NewWithConfig(ctx, pcfg)
	_ = otelpgx.RecordStats(db)                             // pool metrics: acquired, idle, wait time

	rdb := redis.NewClient(&redis.Options{Addr: platform.MustEnv("REDIS_ADDR")})
	_ = redisotel.InstrumentTracing(rdb)
	_ = redisotel.InstrumentMetrics(rdb)

	// ...
	platform.Route(mux, "POST /orders", h.Create)          // instead of mux.HandleFunc
	mux.HandleFunc("GET /livez", health.Livez)              // probes: deliberately NOT traced (noise)
	mux.HandleFunc("GET /readyz", health.Readyz)
	// ...
}

Outbound HTTP calls get the client-side half: &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)} injects traceparent into outgoing requests.

29.4 Business metrics and spans

package orders

import (
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/metric"
)

var (
	tracer = otel.Tracer("github.com/acme/orbit/orders")
	meter  = otel.Meter("github.com/acme/orbit/orders")

	ordersCreated, _ = meter.Int64Counter("orbit.orders.created",
		metric.WithDescription("Orders successfully created"), metric.WithUnit("{order}"))
	orderQuantity, _ = meter.Int64Histogram("orbit.orders.quantity",
		metric.WithDescription("Items per order"), metric.WithExplicitBucketBoundaries(1, 2, 5, 10, 50))
)

// inside Create, after a successful insert:
//   ordersCreated.Add(ctx, 1, metric.WithAttributes(attribute.Bool("idempotent_replay", !created)))
//   orderQuantity.Record(ctx, int64(req.Quantity))
//
// a custom span around a meaningful unit of work:
//   ctx, span := tracer.Start(ctx, "orders.insert")
//   defer span.End()
//   span.SetAttributes(attribute.Int64("orbit.product_id", req.ProductID))
//   if err != nil { span.RecordError(err); span.SetStatus(codes.Error, "insert failed") }

War story: cardinality. Someone added attribute.String("user_id", ...) to a request counter. With 2 million users × 20 routes × 5 status codes, Prometheus went from 300k to 200M active series and ran out of memory, taking alerting down with it. Metric attributes must be bounded (route, method, status class, region). Unbounded values (user, order, request IDs) belong on spans and logs.

⚠️ Caveat (verify): metric names depend on the otelhttp version. Releases following the current HTTP semantic conventions emit http.server.request.duration in seconds (Prometheus: http_server_request_duration_seconds_*). Older releases emitted http.server.duration in milliseconds. Every PromQL query, alert, SLO, KEDA trigger and canary analysis in this guide assumes the first. Check what you actually get: curl -s localhost:9090/api/v1/label/__name__/values | jq -r '.data[]' | grep http_server (port-forward Prometheus first). Also check that otelpgx.RecordStats and redisotel.InstrumentMetrics exist in the versions you pin.

29.5 Propagating traces through the queue

HTTP propagation is automatic. A message queue is not: inject the context into the message and extract it on the consumer side, so one trace covers POST /orders → XADD → worker → UPDATE.

// producer (orders): add the W3C trace context to the stream entry
carrier := propagation.MapCarrier{}
otel.GetTextMapPropagator().Inject(ctx, carrier)
values := map[string]any{"order_id": id}
for k, v := range carrier { // "traceparent", optionally "tracestate"/"baggage"
	values[k] = v
}
h.RDB.XAdd(ctx, &redis.XAddArgs{Stream: Stream, Values: values, MaxLen: 100_000, Approx: true})

// consumer (worker): continue the trace
carrier := propagation.MapCarrier{}
for k, v := range m.Values {
	if s, ok := v.(string); ok {
		carrier[k] = s
	}
}
ctx = otel.GetTextMapPropagator().Extract(ctx, carrier)
ctx, span := tracer.Start(ctx, "orders.created process",
	trace.WithSpanKind(trace.SpanKindConsumer),
	trace.WithAttributes(
		attribute.String("messaging.system", "redis"),
		attribute.String("messaging.destination.name", Stream),
		attribute.String("messaging.message.id", m.ID)))
defer span.End()

Worker metrics to add: orbit.worker.processed (counter, attribute outcome=paid|failed|retry) and orbit.worker.process.duration (histogram, seconds).

29.6 Configuration in Kubernetes

Added to every service's base manifest (and to the Compose file):

env:
  - name: HOST_IP
    valueFrom: { fieldRef: { fieldPath: status.hostIP } }
  - name: POD_NAME
    valueFrom: { fieldRef: { fieldPath: metadata.name } }
  - { name: OTEL_SERVICE_NAME, value: orders }
  - { name: OTEL_EXPORTER_OTLP_ENDPOINT, value: "http://$(HOST_IP):4317" }     # node-local agent
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: "service.namespace=orbit,service.version=$(APP_VERSION),deployment.environment.name=prod,k8s.pod.name=$(POD_NAME)"

(APP_VERSION is set by Kustomize from the image tag; service.version is what lets dashboards compare canary vs stable in §37.)

29.7 Zero-code instrumentation (for services you don't own)

The OpenTelemetry Operator injects auto-instrumentation agents (Java, Python, Node.js, .NET; Go via eBPF) using a pod annotation. That's useful for third-party or legacy apps:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata: { name: default, namespace: legacy }
spec:
  exporter: { endpoint: http://otel-agent.monitoring:4318 }
  propagators: [tracecontext, baggage]
  sampler: { type: parentbased_traceidratio, argument: "1" }
---
# on the Deployment's pod template:
#   annotations: { instrumentation.opentelemetry.io/inject-java: "true" }

Grafana Beyla / OTel eBPF instrumentation can produce RED metrics and traces for any HTTP/gRPC service at the kernel level, with no code and no restarts.

⚠️ Caveat (verify): the Operator Instrumentation exporter endpoint, ports and Go eBPF support vary by Operator version. Grafana Beyla is being folded into the OpenTelemetry eBPF instrumentation project, so check which one is current.

Generated from real-life-k8s-guide.md · 8 Oct 2026 16:36