Part IV — Observability

28. Observability strategy

28.1 Signals and what each one answers

Signal	Answers	Strength	Cost driver
Metrics	"Is something wrong? How much? Since when?"	cheap aggregates, alerting, trends	cardinality (unique label combinations)
Logs	"What exactly happened in this event?"	detail, context, audit	volume (bytes ingested)
Traces	"Where did the time go across services for this request?"	causality across hops	volume (sample!)
Profiles	"Which line of code burns CPU/allocates memory?"	continuous pprof (Pyroscope, Parca)	low
K8s events	"What did the platform do?" (OOMKill, eviction, scaling)	explains restarts	low, but they expire after 1 h

The workflow they enable: an alert fires on a metric → a dashboard narrows it to a service and time → an exemplar or trace shows the slow hop → that span's logs (same trace_id) show the error. Correlation by shared resource attributes (service.name, k8s.pod.name) and trace_id is what turns three data stores into one investigation.

28.2 What to measure: Golden Signals, RED, USE

RED (for every service): Rate, Errors, Duration.
USE (for every resource: CPU, memory, disk, connection pools, queues): Utilization, Saturation, Errors.
Four Golden Signals (Google SRE book): latency, traffic, errors, saturation, which is RED + saturation.

For Orbit:

Component	Signals
catalog, orders	RED per route; DB pool in-use/wait; cache hit ratio
worker	messages processed/s, processing duration, failures, stream lag (saturation)
Postgres	connections vs max, replication lag, TPS, slow queries, disk
Redis	memory vs maxmemory, ops/s, stream length
Kubernetes	restarts, OOMKills, not-ready pods, HPA at max, pending pods, node pressure, PVC usage

28.3 Why OpenTelemetry

OpenTelemetry (OTel) is the CNCF standard for generating and shipping telemetry: one SDK per language, one wire protocol (OTLP), one Collector. The backend becomes a swappable detail: Prometheus/Loki/Tempo today, Datadog/Honeycomb/New Relic/Grafana Cloud tomorrow, by changing Collector config, not application code.

 app (OTel SDK) ──OTLP──► Collector AGENT (DaemonSet, per node) ──OTLP──► Collector GATEWAY (Deployment)
 app stdout logs ──files──►   │  + k8s metadata, batching                  │ tail sampling, routing, redaction
                              │                                            ├──► Prometheus (metrics, OTLP receiver)
                              │                                            ├──► Loki       (logs, OTLP)
                              │                                            └──► Tempo      (traces, OTLP)
 kube-state-metrics, node-exporter, kubelet/cAdvisor ◄──scrape── Prometheus

Best practice: apps send to a local agent, never directly to a SaaS endpoint. The agent buffers through backend outages, adds K8s metadata, and lets you change backends, sampling and redaction without redeploying 200 services.