30. Deploying the observability stack

All components go into the monitoring namespace. Values files live in platform/ and are deployed by Argo CD in Part VII. Here we install them by hand once to understand them.

kubectl create namespace monitoring
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

30.1 Prometheus, Alertmanager, Grafana: kube-prometheus-stack

One chart provides the Prometheus Operator, Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics, and a battle-tested set of Kubernetes alerts and dashboards.

# platform/kube-prometheus-stack/values.yaml
fullnameOverride: kps
prometheus:
  prometheusSpec:
    retention: 15d
    retentionSize: 40GB
    enableOTLPReceiver: true                 # accept OTLP metrics at /api/v1/otlp (Prometheus 3)
    otlp:
      promoteResourceAttributes:             # resource attrs -> labels (default: only in target_info)
        - service.version
        - deployment.environment.name
        - k8s.namespace.name
        - k8s.pod.name
        - k8s.deployment.name
    enableFeatures: [exemplar-storage]       # trace IDs attached to histogram samples
    # pick up ServiceMonitors/PodMonitors/PrometheusRules from ALL namespaces, not just this release's
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    resources:
      requests: { cpu: 500m, memory: 2Gi }
      limits:   { memory: 2Gi }
    storageSpec:
      volumeClaimTemplate:
        spec: { accessModes: [ReadWriteOnce], resources: { requests: { storage: 50Gi } } }
alertmanager:
  config:
    route:
      receiver: default
      group_by: [alertname, namespace, service]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      routes:
        - matchers: [severity="page"]
          receiver: pagerduty
        - matchers: [alertname="Watchdog"]   # dead-man's switch: must ALWAYS fire (see §31.4)
          receiver: deadmansswitch
          repeat_interval: 1m
    inhibit_rules:
      - source_matchers: [severity="page"]
        target_matchers: [severity="ticket"]
        equal: [namespace, service]
    receivers:
      - name: default
        slack_configs:
          - api_url_file: /etc/alertmanager/secrets/slack/url   # from a Secret, not inline
            channel: "#orbit-alerts"
            send_resolved: true
            title: '{{ .CommonLabels.alertname }} ({{ .Status }})'
            text: '{{ range .Alerts }}{{ .Annotations.summary }} - runbook: {{ .Annotations.runbook_url }}{{ "\n" }}{{ end }}'
      - name: pagerduty
        pagerduty_configs: [{ routing_key_file: /etc/alertmanager/secrets/pagerduty/key }]
      - name: deadmansswitch
        webhook_configs: [{ url: "https://hc-ping.com/<uuid>" }]
  alertmanagerSpec:
    secrets: [slack, pagerduty]
grafana:
  adminPassword: change-me                  # prod: admin.existingSecret + SSO
  sidecar:
    dashboards: { enabled: true, searchNamespace: ALL }   # dashboards as ConfigMaps (label grafana_dashboard=1)
  additionalDataSources:
    - name: Loki
      type: loki
      uid: loki
      url: http://loki-gateway.monitoring.svc
      jsonData:
        derivedFields:                       # log line -> trace
          - name: trace_id
            matcherType: label
            matcherRegex: trace_id
            datasourceUid: tempo
            url: "$${__value.raw}"
    - name: Tempo
      type: tempo
      uid: tempo
      url: http://tempo.monitoring.svc:3200
      jsonData:
        tracesToLogsV2:                      # trace -> its logs
          datasourceUid: loki
          filterByTraceID: true
          spanStartTimeShift: "-5m"
          spanEndTimeShift: "5m"
        serviceMap: { datasourceUid: prometheus }
        nodeGraph: { enabled: true }

kubectl create secret generic slack -n monitoring --from-literal=url=https://hooks.slack.com/services/...
kubectl create secret generic pagerduty -n monitoring --from-literal=key=...
helm upgrade --install kps prometheus-community/kube-prometheus-stack -n monitoring \
  -f platform/kube-prometheus-stack/values.yaml --wait
kubectl port-forward -n monitoring svc/kps-grafana 3000:80     # http://localhost:3000

⚠️ Caveat (verify): the least certain part of the stack. Check against the kube-prometheus-stack/Prometheus Operator version you install. (1) The field names enableOTLPReceiver and otlp.promoteResourceAttributes in prometheusSpec: run kubectl explain prometheus.spec.otlp after installing. If they're absent, use enableFeatures: [otlp-write-receiver] on older versions, or the Collector's prometheusremotewrite exporter with enableRemoteWriteReceiver: true. (2) The label mapping: OTLP service.namespace/service.name → job="orbit/orders", and dots → underscores. (3) Grafana derivedFields with matcherType: label (structured metadata) needs a recent Grafana; otherwise use a regex matcher on "trace_id":"(\w+)". (4) The fullnameOverride: kps → service kps-prometheus naming; confirm with kubectl get svc -n monitoring.

30.2 Loki (logs) and Tempo (traces)

# platform/loki/values.yaml: single-binary mode for the lab; use "SimpleScalable"/"Distributed" + object storage in prod
deploymentMode: SingleBinary
loki:
  auth_enabled: false
  commonConfig: { replication_factor: 1 }
  storage: { type: filesystem }             # prod: s3/gcs/azure bucket
  schemaConfig:
    configs:
      - from: "2024-01-01"
        store: tsdb
        object_store: filesystem
        schema: v13
        index: { prefix: index_, period: 24h }
  limits_config:
    allow_structured_metadata: true          # required for OTLP ingestion
    retention_period: 168h
singleBinary: { replicas: 1, persistence: { size: 20Gi } }
read: { replicas: 0 }
write: { replicas: 0 }
backend: { replicas: 0 }
chunksCache: { enabled: false }
resultsCache: { enabled: false }

# platform/tempo/values.yaml (single binary; tempo-distributed + object storage in prod)
tempo:
  retention: 72h
  receivers:
    otlp:
      protocols:
        grpc: { endpoint: 0.0.0.0:4317 }
        http: { endpoint: 0.0.0.0:4318 }
persistence: { enabled: true, size: 20Gi }

helm upgrade --install loki  grafana/loki  -n monitoring -f platform/loki/values.yaml --wait
helm upgrade --install tempo grafana/tempo -n monitoring -f platform/tempo/values.yaml --wait

⚠️ Caveat (verify): Loki and Tempo chart values change between chart majors, and these were written against Loki chart v6-style values. Grafana has been reorganizing the Tempo charts (single-binary vs tempo-distributed). Render with helm template and check kubectl get svc -n monitoring for the actual service names (loki-gateway, tempo) that §30.1 and §30.3 point to.

30.3 OpenTelemetry Collector: agent (DaemonSet) + gateway (Deployment)

Agent: receives OTLP from pods on its node (hostPort 4317), tails container log files, enriches everything with Kubernetes metadata, and forwards it to the gateway.

# platform/otel-agent/values.yaml
mode: daemonset
fullnameOverride: otel-agent
image: { repository: otel/opentelemetry-collector-contrib }
presets:
  kubernetesAttributes: { enabled: true }   # k8sattributes processor + RBAC
  logsCollection: { enabled: true }         # filelog receiver on /var/log/pods
resources:
  requests: { cpu: 100m, memory: 256Mi }
  limits:   { memory: 512Mi }
config:
  receivers:
    otlp:
      protocols:
        grpc: { endpoint: ${env:MY_POD_IP}:4317 }
        http: { endpoint: ${env:MY_POD_IP}:4318 }
  processors:
    memory_limiter: { check_interval: 1s, limit_percentage: 80, spike_limit_percentage: 25 }
    batch: { send_batch_size: 8192, timeout: 2s }
    resourcedetection: { detectors: [env, system], override: false }
    transform/json-logs:                      # lift JSON fields + trace_id out of the log body
      error_mode: ignore
      log_statements:
        - context: log
          statements:
            - merge_maps(attributes, ParseJSON(body), "upsert") where IsMatch(body, "^\\{")
            - set(trace_id.string, attributes["trace_id"]) where attributes["trace_id"] != nil
            - set(span_id.string, attributes["span_id"]) where attributes["span_id"] != nil
            - set(severity_text, attributes["level"]) where attributes["level"] != nil
  exporters:
    otlp/gateway:
      endpoint: otel-gateway.monitoring.svc:4317
      tls: { insecure: true }                 # mTLS in prod, or rely on a mesh
      sending_queue: { enabled: true, queue_size: 5000 }
      retry_on_failure: { enabled: true, max_elapsed_time: 300s }
    loadbalancing:                            # traces: all spans of a trace -> same gateway pod (tail sampling)
      routing_key: traceID
      protocol: { otlp: { tls: { insecure: true } } }
      resolver:
        k8s: { service: otel-gateway-headless.monitoring }
  service:
    pipelines:
      traces:  { receivers: [otlp], processors: [memory_limiter, k8sattributes, resourcedetection, batch], exporters: [loadbalancing] }
      metrics: { receivers: [otlp], processors: [memory_limiter, k8sattributes, resourcedetection, batch], exporters: [otlp/gateway] }
      logs:    { receivers: [otlp, filelog], processors: [memory_limiter, k8sattributes, transform/json-logs, batch], exporters: [otlp/gateway] }

(The loadbalancing exporter with the k8s resolver needs RBAC to watch EndpointSlices; add a small Role, or use the dns resolver against the headless service.)

Gateway: central processing, with tail sampling, redaction and fan-out to backends.

# platform/otel-gateway/values.yaml
mode: deployment
fullnameOverride: otel-gateway
replicaCount: 2
image: { repository: otel/opentelemetry-collector-contrib }
resources:
  requests: { cpu: 250m, memory: 512Mi }
  limits:   { memory: 1Gi }
podDisruptionBudget: { enabled: true, minAvailable: 1 }
config:
  receivers:
    otlp: { protocols: { grpc: { endpoint: ${env:MY_POD_IP}:4317 } } }
  processors:
    memory_limiter: { check_interval: 1s, limit_percentage: 80, spike_limit_percentage: 25 }
    batch: {}
    tail_sampling:                            # decide AFTER seeing the whole trace
      decision_wait: 10s
      policies:
        - { name: errors,   type: status_code, status_code: { status_codes: [ERROR] } }
        - { name: slow,     type: latency,     latency: { threshold_ms: 500 } }
        - { name: baseline, type: probabilistic, probabilistic: { sampling_percentage: 10 } }
    attributes/redact:
      actions:
        - { key: http.request.header.authorization, action: delete }
        - { key: db.query.text, action: hash }     # or keep, if queries are parameterized (they should be)
  exporters:
    otlphttp/prometheus:
      endpoint: http://kps-prometheus.monitoring.svc:9090/api/v1/otlp
      tls: { insecure: true }
    otlphttp/loki:
      endpoint: http://loki-gateway.monitoring.svc/otlp
    otlp/tempo:
      endpoint: tempo.monitoring.svc:4317
      tls: { insecure: true }
  service:
    pipelines:
      traces:  { receivers: [otlp], processors: [memory_limiter, tail_sampling, attributes/redact, batch], exporters: [otlp/tempo] }
      metrics: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/prometheus] }
      logs:    { receivers: [otlp], processors: [memory_limiter, attributes/redact, batch], exporters: [otlphttp/loki] }

helm upgrade --install otel-gateway open-telemetry/opentelemetry-collector -n monitoring -f platform/otel-gateway/values.yaml
helm upgrade --install otel-agent   open-telemetry/opentelemetry-collector -n monitoring -f platform/otel-agent/values.yaml
kubectl logs -n monitoring ds/otel-agent | grep -i error

Sampling strategy explained:

Head sampling (decided at the first span, in the SDK) is cheap but blind: at 10% you drop 90% of your errors.
Tail sampling (decided in the gateway after the trace completes) keeps 100% of errors and slow traces plus a 10% baseline. That's the standard production choice. It costs gateway memory and requires that all spans of a trace reach the same gateway replica, hence the loadbalancing exporter.
Metrics are never sampled. They come from the SDK before sampling, so RED dashboards and SLOs stay exact.

Best practice: monitor the monitoring. Alert on Collector otelcol_exporter_send_failed_* and otelcol_processor_refused_*, Prometheus up == 0 for important targets, and the Watchdog alert (§31.4). An observability pipeline that silently drops data fails in the worst possible moment: during the incident.

⚠️ Caveat (verify): (1) OTTL syntax changed in recent Collector releases (paths such as log.body with context inference). The context: log + body form shown here should still work but may print deprecation warnings. (2) Check the loadbalancing exporter's resolver.k8s RBAC needs and config shape in its README. (3) The chart's presets and the daemonset hostPort: 4317 default depend on the chart version. Validate every config with docker run --rm -v $PWD:/c otel/opentelemetry-collector-contrib validate --config=/c/config.yaml (extract the config: block first), and pin the chart and image versions.

30.4 Scraping things that expose `/metrics`

Not everything speaks OTLP. Postgres (CNPG), Redis exporters and Envoy expose Prometheus endpoints. Point the Prometheus Operator at them with ServiceMonitor/PodMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata: { name: redis, namespace: orbit }
spec:
  selector: { matchLabels: { app.kubernetes.io/name: redis-exporter } }
  endpoints: [{ port: metrics, interval: 30s }]

(Run oliver006/redis_exporter as a sidecar in the Redis StatefulSet, or as a small Deployment, to get Redis metrics. CNPG's enablePodMonitor: true from §20 is already scraped.)

30. Deploying the observability stack

30.1 Prometheus, Alertmanager, Grafana: kube-prometheus-stack

30.2 Loki (logs) and Tempo (traces)

30.3 OpenTelemetry Collector: agent (DaemonSet) + gateway (Deployment)

30.4 Scraping things that expose /metrics

30.4 Scraping things that expose `/metrics`