24. Scheduling: placement for resilience

24.1 Spread replicas across zones and nodes

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway        # prefer spread, don't block scheduling
    labelSelector: { matchLabels: { app.kubernetes.io/name: orders } }
    matchLabelKeys: [pod-template-hash]      # compute skew per rollout revision
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector: { matchLabels: { app.kubernetes.io/name: orders } }
    matchLabelKeys: [pod-template-hash]

Without this, the scheduler can put all 3 replicas on one node. That node reboots and the service is down even though "replicas: 3".

24.2 Affinity, taints, and tolerations

# node affinity: only arm64 nodes, prefer spot
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions: [{ key: kubernetes.io/arch, operator: In, values: [arm64] }]
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 50
        preference: { matchExpressions: [{ key: karpenter.sh/capacity-type, operator: In, values: [spot] }] }
  # pod affinity: put the worker near Redis (same zone) to cut cross-zone traffic cost and latency
  podAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          topologyKey: topology.kubernetes.io/zone
          labelSelector: { matchLabels: { app.kubernetes.io/name: redis } }

Taints repel, tolerations permit. A dedicated GPU pool:

kubectl taint nodes gpu-node-1 nvidia.com/gpu=true:NoSchedule

tolerations: [{ key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }]
nodeSelector: { nvidia.com/gpu.present: "true" }   # toleration alone doesn't ATTRACT, it only allows

24.3 The golden Deployment (everything so far, in one manifest)

This is the reference manifest for a stateless HTTP service. Every field is here for a reason covered above. Copy it as your template.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders
  labels: &labels
    app.kubernetes.io/name: orders
    app.kubernetes.io/part-of: orbit
    app.kubernetes.io/component: api
spec:
  replicas: 3                               # removed when an HPA/KEDA manages replicas (§27)
  revisionHistoryLimit: 5
  selector: { matchLabels: { app.kubernetes.io/name: orders } }
  strategy: { type: RollingUpdate, rollingUpdate: { maxSurge: 25%, maxUnavailable: 0 } }
  minReadySeconds: 5
  progressDeadlineSeconds: 300
  template:
    metadata:
      labels: *labels
    spec:
      serviceAccountName: orders            # its own identity (§34.1), never "default"
      automountServiceAccountToken: false   # it never calls the K8s API
      priorityClassName: orbit-critical
      terminationGracePeriodSeconds: 45
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        runAsGroup: 65532
        fsGroup: 65532
        seccompProfile: { type: RuntimeDefault }
      topologySpreadConstraints:
        - { maxSkew: 1, topologyKey: topology.kubernetes.io/zone, whenUnsatisfiable: ScheduleAnyway,
            labelSelector: { matchLabels: { app.kubernetes.io/name: orders } }, matchLabelKeys: [pod-template-hash] }
        - { maxSkew: 1, topologyKey: kubernetes.io/hostname, whenUnsatisfiable: ScheduleAnyway,
            labelSelector: { matchLabels: { app.kubernetes.io/name: orders } }, matchLabelKeys: [pod-template-hash] }
      tolerations:
        - { key: node.kubernetes.io/unreachable, operator: Exists, effect: NoExecute, tolerationSeconds: 30 }
        - { key: node.kubernetes.io/not-ready,   operator: Exists, effect: NoExecute, tolerationSeconds: 30 }
      containers:
        - name: orders
          image: ghcr.io/acme/orbit-orders:sha-3f9c2a1   # immutable tag (GitOps pins the digest)
          imagePullPolicy: IfNotPresent
          ports: [{ name: http, containerPort: 8080 }]
          envFrom: [{ configMapRef: { name: orders-config } }]
          env:
            - name: DATABASE_URL
              valueFrom: { secretKeyRef: { name: orbit-db-app, key: uri } }
            - { name: GOMEMLIMIT, value: "230MiB" }
            - name: POD_NAME
              valueFrom: { fieldRef: { fieldPath: metadata.name } }
            - { name: OTEL_SERVICE_NAME, value: orders }
            - name: OTEL_EXPORTER_OTLP_ENDPOINT     # node-local collector agent (§30)
              value: "http://$(HOST_IP):4317"
          resources:
            requests: { cpu: 250m, memory: 256Mi }
            limits:   { memory: 256Mi }
          startupProbe:   { httpGet: { path: /livez,  port: http }, periodSeconds: 2,  failureThreshold: 30 }
          livenessProbe:  { httpGet: { path: /livez,  port: http }, periodSeconds: 10, timeoutSeconds: 2, failureThreshold: 3 }
          readinessProbe: { httpGet: { path: /readyz, port: http }, periodSeconds: 5,  timeoutSeconds: 2, failureThreshold: 2 }
          lifecycle:
            preStop: { sleep: { seconds: 5 } }
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
          volumeMounts: [{ name: tmp, mountPath: /tmp }]
      volumes:
        - { name: tmp, emptyDir: { sizeLimit: 64Mi } }

(HOST_IP comes from fieldRef: status.hostIP; it's omitted above for brevity and included in the Kustomize base in §26.)