Part IX — Capstone and Reference

43. Capstone: assemble and verify the whole system

43.1 Bootstrap order (kind)

# Makefile (excerpt)
cluster:        ## 1. cluster + LB
	kind create cluster --config kind-orbit.yaml
images:         ## 2. build + load images (CI pushes to GHCR in the real flow)
	for s in catalog orders worker; do \
	  docker build --build-arg SERVICE=$$s -t orbit-$$s:dev . && kind load docker-image orbit-$$s:dev --name orbit; \
	done
gitops:         ## 3. Argo CD + root app: everything else is pulled from Git
	helm upgrade --install argocd argo/argo-cd -n argocd --create-namespace --wait
	kubectl apply -n argocd -f platform/root.yaml
verify:         ## 4. smoke + load + chaos
	k6 run -e BASE=https://orbit.localtest.me load/smoke.js
	k6 run load/orders.js

Sync waves (from §36.4) give the platform its order: CRDs and operators (cert-manager, CNPG, KEDA, Rollouts, Kyverno, ESO) → gateways, monitoring and Collectors → policies → the orbit-* ApplicationSet (DB → migration hook → services → routes → ScaledObjects).

macOS note: if the Gateway's LoadBalancer address isn't reachable from the host, port-forward the Envoy service instead: kubectl -n envoy-gateway-system port-forward svc/$(kubectl -n envoy-gateway-system get svc -l gateway.envoyproxy.io/owning-gateway-name=public -o name | cut -d/ -f2) 8443:443 and use https://orbit.localtest.me:8443.

Lab VM note (§0): inside the VM the LoadBalancer IP works directly (cloud-provider-kind runs as a systemd service). To use the browser on your Mac, run the same port-forward inside the VM: Lima forwards 8443 to your Mac's localhost automatically (§0.5).

43.2 Acceptance tests: the system is "done" when all of these pass

#	Test	Proves
1	`git push` to main → new digest running in dev in < 10 min, no human steps	CI + GitOps
2	Unsigned image applied by hand → rejected by admission	supply chain policy
3	`kubectl run` with root user in `orbit` → rejected	Pod Security
4	k6 800 RPS: p95 < 300 ms, errors < 1%; orders and worker replicas scale out, then back in (worker to 0)	autoscaling
5	Rolling restart of all services during k6 → 0 failed requests	probes, preStop, PDB, `maxUnavailable: 0`
6	Kill the Postgres primary → failover < 30 s; orders not-ready, then ready; no pod restarts	liveness/readiness design, CNPG
7	Bad canary (5% 500s) → automatic abort at ≤ 25% weight	progressive delivery
8	Burn-rate alert fires for #7 if you force-promote; Slack message has a runbook link	SLO alerting
9	From that alert: dashboard → exemplar → trace → logs in < 5 min	observability correlation
10	Drain a node / stop a kind "zone" during k6 → SLO holds	topology spread, tolerations, PDB
11	Delete the `orbit` namespace → restore DB (PITR) + Argo CD resync; measure RTO; verify orders exist	backup & DR
12	Catalog pod cannot open a connection to Postgres admin port / metadata IP	NetworkPolicy
13	NetworkChaos 200 ms to DB → p95 degrades but error rate stays < 1%, alerts are tickets, not pages	timeouts, alert design
14	A poison message (unknown `order_id` that fails forever) → moved to `orders.dlq` after 5 attempts; alert on DLQ length > 0	queue hygiene

43.3 The dead-letter queue (test #14)

// in worker.handle, on failure: check delivery count from XPENDING; after maxAttempts, park it
const maxAttempts = 5

func (w *Worker) deadLetterIfExhausted(ctx context.Context, m redis.XMessage, cause error) bool {
	p, err := w.RDB.XPendingExt(ctx, &redis.XPendingExtArgs{
		Stream: w.Stream, Group: w.Group, Start: m.ID, End: m.ID, Count: 1,
	}).Result()
	if err != nil || len(p) == 0 || p[0].RetryCount < maxAttempts {
		return false
	}
	values := map[string]any{"original_id": m.ID, "error": cause.Error(), "attempts": p[0].RetryCount}
	for k, v := range m.Values {
		values[k] = v
	}
	pipe := w.RDB.TxPipeline()                       // MULTI: add to DLQ + ACK atomically
	pipe.XAdd(ctx, &redis.XAddArgs{Stream: w.Stream + ".dlq", Values: values})
	pipe.XAck(ctx, w.Stream, w.Group, m.ID)
	_, err = pipe.Exec(ctx)
	slog.ErrorContext(ctx, "message dead-lettered", "order_id", m.Values["order_id"], "attempts", p[0].RetryCount)
	return err == nil
}

Alert on XLEN orders.created.dlq > 0 (via redis_exporter's --check-streams). A DLQ nobody watches just loses messages slowly.

⚠️ Caveat (verify): check the XPendingExt result field (RetryCount) against your go-redis version, and check whether redis_exporter's --check-streams flag exists in yours (redis_exporter --help).

43.4 What you built

Git (app + deploy + platform)
  └─► CI: test → build multi-arch → scan → sign → SBOM → bump digest
        └─► Argo CD: platform add-ons + orbit (dev → staging → prod via PR)
              └─► Kubernetes: Envoy Gateway (TLS) → catalog / orders (Rollout canary) → CNPG Postgres, Redis
                    worker (KEDA 0..N on stream lag), CronJobs, PDBs, spread, NetworkPolicies, PSA restricted
                    Kyverno (signed images), ESO (secrets), Velero + CNPG PITR (backups)
                    OTel agent/gateway → Prometheus / Loki / Tempo → Grafana, SLO burn alerts → Alertmanager