18. StatefulSets, DaemonSets, Jobs, and CronJobs

18.1 StatefulSet: stable identity + stable storage

Use a StatefulSet when each replica needs its own disk and a stable name: databases, Kafka, ZooKeeper, Elasticsearch, Redis.

Orbit's Redis (single instance, AOF persistence; Redis HA is a separate topic: use a managed service or an operator in production):

apiVersion: v1
kind: Service
metadata: { name: redis, labels: { app.kubernetes.io/name: redis } }
spec:
  clusterIP: None                 # headless: required for StatefulSet pod DNS
  selector: { app.kubernetes.io/name: redis }
  ports: [{ name: redis, port: 6379 }]
---
apiVersion: apps/v1
kind: StatefulSet
metadata: { name: redis }
spec:
  serviceName: redis
  replicas: 1
  selector: { matchLabels: { app.kubernetes.io/name: redis } }
  template:
    metadata: { labels: { app.kubernetes.io/name: redis } }
    spec:
      securityContext: { runAsUser: 999, runAsGroup: 999, fsGroup: 999, runAsNonRoot: true }
      containers:
        - name: redis
          image: redis:7.4-alpine
          args: ["--appendonly", "yes", "--maxmemory", "200mb", "--maxmemory-policy", "noeviction"]
          ports: [{ name: redis, containerPort: 6379 }]
          resources:
            requests: { cpu: 100m, memory: 256Mi }
            limits:   { memory: 256Mi }
          readinessProbe: { exec: { command: ["redis-cli", "ping"] }, periodSeconds: 5 }
          livenessProbe:  { tcpSocket: { port: redis }, periodSeconds: 10, failureThreshold: 6 }
          volumeMounts: [{ name: data, mountPath: /data }]
  volumeClaimTemplates:
    - metadata: { name: data }
      spec:
        accessModes: [ReadWriteOnce]
        resources: { requests: { storage: 1Gi } }

noeviction matters because Redis is our queue too. With an LRU policy under memory pressure, Redis would silently drop unprocessed orders.

Best practice: for databases, use an operator (CloudNativePG, Percona, Strimzi for Kafka) or a managed service (RDS, Cloud SQL). A StatefulSet gives you identity and disks. It does not give you failover, backups, replication, minor-version upgrades, or connection pooling. Operators encode a DBA's runbook as a controller. Orbit uses CloudNativePG (§20).

18.2 DaemonSet: one pod per node

For node agents: log collectors, the OTel Collector agent, CNI, CSI node plugins, node-exporter, security agents.

apiVersion: apps/v1
kind: DaemonSet
metadata: { name: node-agent, namespace: monitoring }
spec:
  selector: { matchLabels: { app: node-agent } }
  updateStrategy: { type: RollingUpdate, rollingUpdate: { maxUnavailable: 10% } }
  template:
    metadata: { labels: { app: node-agent } }
    spec:
      tolerations: [{ operator: Exists }]        # also run on tainted nodes (control plane, GPU, spot)
      priorityClassName: system-node-critical     # never evicted for app pods
      containers:
        - name: agent
          image: busybox:1.36
          command: ["sh", "-c", "while true; do date; sleep 60; done"]
          resources: { requests: { cpu: 10m, memory: 16Mi }, limits: { memory: 32Mi } }

You'll deploy a real one, the OTel Collector agent, in §30.

18.3 Jobs and CronJobs

apiVersion: batch/v1
kind: Job
metadata: { name: reprice-catalog }
spec:
  backoffLimit: 3                 # retries before Failed
  activeDeadlineSeconds: 600      # hard timeout for the whole job
  ttlSecondsAfterFinished: 86400  # garbage-collect after a day
  completions: 10                 # 10 work items...
  parallelism: 3                  # ...3 at a time
  completionMode: Indexed         # each pod gets JOB_COMPLETION_INDEX 0..9 (shard the work)
  podFailurePolicy:               # don't retry bugs; do retry infra disruptions
    rules:
      - action: FailJob
        onExitCodes: { containerName: job, operator: In, values: [42] }
      - action: Ignore
        onPodConditions: [{ type: DisruptionTarget }]
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: job
          image: orbit-catalog:dev
          command: ["/app", "reprice", "--shard=$(JOB_COMPLETION_INDEX)"]
---
apiVersion: batch/v1
kind: CronJob
metadata: { name: expire-pending-orders }
spec:
  schedule: "*/15 * * * *"
  timeZone: "Etc/UTC"
  concurrencyPolicy: Forbid       # don't start a run while the last is still going
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 300
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: expire
              image: postgres:17
              envFrom: [{ secretRef: { name: orbit-db-app } }]
              command: ["sh", "-c", "psql \"$uri\" -c \"UPDATE orders SET status='failed' WHERE status='pending' AND created_at < now() - interval '1 hour'\""]

Best practice: CronJob runs are at-least-once and may overlap or be skipped (controller downtime, startingDeadlineSeconds). Make every job idempotent, set concurrencyPolicy: Forbid unless overlap is safe, and alert on "last successful run too old", not just on failure. A job that silently stops being scheduled never fails.