39. Backup and disaster recovery

39.1 What actually needs backing up

Best practice: if your cluster can be rebuilt from Git, then cluster DR = data DR. Treat clusters as cattle: "restore the cluster" becomes "create a new cluster, point Argo CD at the repo, restore the databases".

Define targets per service before choosing tools. RPO: how much data can we lose? RTO: how long can we be down? Orbit's targets: RPO 5 min, RTO 1 h.

39.2 Postgres: continuous backup and point-in-time recovery (CNPG)

CloudNativePG ships base backups plus continuous WAL archiving to object storage through the Barman Cloud plugin. The RPO is the WAL archive interval, which is seconds to minutes.

apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata: { name: orbit-backups, namespace: orbit }
spec:
  retentionPolicy: "30d"
  configuration:
    destinationPath: s3://acme-orbit-pg-backups/       # lab: a MinIO bucket
    s3Credentials: { inheritFromIAMRole: true }         # Pod Identity / IRSA in the cloud
    wal: { compression: gzip }
    data: { compression: gzip }
---
# added to the Cluster spec (§20.1)
#   plugins:
#     - name: barman-cloud.cloudnative-pg.io
#       isWALArchiver: true
#       parameters: { barmanObjectName: orbit-backups }
---
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata: { name: orbit-db-daily, namespace: orbit }
spec:
  schedule: "0 0 2 * * *"           # six fields: sec min hour dom mon dow
  backupOwnerReference: self
  cluster: { name: orbit-db }
  method: plugin
  pluginConfiguration: { name: barman-cloud.cloudnative-pg.io }

Point-in-time restore into a new cluster (never over the broken one):

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata: { name: orbit-db-restore, namespace: orbit }
spec:
  instances: 3
  storage: { size: 2Gi }
  bootstrap:
    recovery:
      source: orbit-db
      recoveryTarget: { targetTime: "2026-10-04 09:41:00+00" }   # just before the bad DELETE
  externalClusters:
    - name: orbit-db
      plugin:
        name: barman-cloud.cloudnative-pg.io
        parameters: { barmanObjectName: orbit-backups, serverName: orbit-db }

War story: GitLab's 2017 outage. An engineer ran rm -rf on the primary's data directory during a replication incident. Then the team found that none of their five backup mechanisms worked: pg_dump had been failing silently (version mismatch), and other mechanisms had never been tested. They restored from a 6-hour-old staging copy. Lesson burned into every SRE team since: a backup you haven't restored is not a backup. Automate restore tests (§39.4).

⚠️ Caveat (verify): the Barman Cloud plugin is a separate install (it needs cert-manager). Its API group, ObjectStore fields, and the Cluster plugins: / recovery externalClusters.plugin shapes follow the plugin docs for recent CNPG (1.26+). Older CNPG versions use the in-tree spec.backup.barmanObjectStore instead. Check against the CNPG and plugin docs for your versions, and use a MinIO bucket with explicit credentials in the lab (no IAM role on kind).

39.3 Velero: Kubernetes objects and volumes

# lab: MinIO as the S3 target
helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
helm upgrade --install velero vmware-tanzu/velero -n velero --create-namespace -f platform/velero/values.yaml

velero schedule create orbit-hourly --schedule="@every 1h" --include-namespaces orbit --ttl 168h
velero backup create orbit-manual --include-namespaces orbit --wait
velero backup describe orbit-manual --details
velero restore create --from-backup orbit-manual --namespace-mappings orbit:orbit-restore-test

For databases, prefer the DB-native backup (consistent, PITR-capable) and use Velero for the surrounding objects and for simple PVs (file-system backup via Kopia, or CSI snapshots).

⚠️ Caveat (verify): the Velero values file isn't shown. It must configure the AWS object-store plugin (velero/velero-plugin-for-aws), the MinIO URL with s3ForcePathStyle, and credentials. Follow the Velero chart README for your chart version.

39.4 Prove it: an automated restore test

A weekly CronJob (or CI job) that restores the latest backup into a scratch namespace, runs sanity queries, and reports success as a metric:

#!/usr/bin/env bash
# restore-test.sh: runs weekly; alert if orbit_restore_test_last_success_timestamp is older than 8 days
set -euo pipefail
ns=restore-test-$(date +%s)
kubectl create ns "$ns"
sed "s/namespace: orbit/namespace: $ns/" restore-cluster.yaml | kubectl apply -f -
kubectl wait -n "$ns" cluster/orbit-db-restore --for=condition=Ready --timeout=30m
kubectl exec -n "$ns" orbit-db-restore-1 -- psql -U postgres -d orbit -tAc \
  "select count(*) > 0 from orders where created_at > now() - interval '1 day'" | grep -q t
curl -s --data-binary "orbit_restore_test_last_success_timestamp $(date +%s)" \
  http://pushgateway.monitoring:9091/metrics/job/restore_test
kubectl delete ns "$ns"

39.5 DR strategies and the full-cluster drill

The drill (do it on kind; it maps 1:1 to the cloud):

1. kind delete cluster --name orbit. The "region" is gone.
2. kind create cluster --config kind-orbit.yaml, install Argo CD, apply platform/root.yaml.
3. Argo CD rebuilds the platform and apps. The DB Cluster bootstraps from backup (a recovery bootstrap in a DR overlay).
4. Verify with the k6 smoke test plus data checks. Write down the wall-clock time: that is your real RTO.