27. Autoscaling: pods, sizes, and nodes

27.1 Three layers

 load ↑ ──► HPA / KEDA: more PODS ──► pods Pending (no room) ──► Cluster Autoscaler / Karpenter: more NODES
           VPA: right-size each pod's requests (vertical)

They depend on each other: HPA percentages are relative to requests, and node autoscalers react to Pending pods' requests, not actual usage. Wrong requests break all three layers.

27.2 HPA on CPU (catalog)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: catalog }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: catalog }
  minReplicas: 3                    # >= number of zones: survive a zone loss
  maxReplicas: 30
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 65 } }
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - { type: Percent, value: 100, periodSeconds: 30 }   # at most double every 30s
        - { type: Pods,    value: 4,   periodSeconds: 30 }
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 300                         # wait 5 min of low load: no flapping
      policies: [{ type: Percent, value: 20, periodSeconds: 60 }]

The algorithm: desired = ceil(current × currentMetric / target). With 4 pods at 90% CPU and a 65% target, ceil(4 × 90/65) = 6. The HPA syncs every 15 s and ignores changes within a 10% tolerance.

Best practice: target 60–70%, not 90%. Scaling takes time (HPA sync + scheduling + image pull + startup + readiness ≈ 30–120 s). The headroom absorbs the spike while new pods arrive. Remove replicas: from the Deployment manifest when an HPA owns it, or every GitOps sync resets the count.

27.3 KEDA: scale on what actually matters

CPU is a lagging, indirect signal. A queue consumer should scale on backlog, and an API on request rate or latency. KEDA (CNCF-graduated) feeds external metrics to an HPA it manages, and can scale to zero.

helm repo add kedacore https://kedacore.github.io/charts
helm upgrade --install keda kedacore/keda -n keda --create-namespace --wait

Worker scales on Redis Stream lag (0 → 20):

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: worker, namespace: orbit }
spec:
  scaleTargetRef: { name: worker }
  minReplicaCount: 0              # scale to zero when there's nothing to do
  maxReplicaCount: 20
  pollingInterval: 10
  cooldownPeriod: 120             # seconds of zero activity before going to 0
  triggers:
    - type: redis-streams
      metadata:
        address: redis.orbit.svc.cluster.local:6379
        stream: orders.created
        consumerGroup: workers
        lagCount: "50"            # target: ~50 unprocessed entries per replica
        activationLagCount: "1"   # wake from 0 when anything arrives

Orders scales on request rate from Prometheus (set up in §31):

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: orders, namespace: orbit }
spec:
  scaleTargetRef: { name: orders }
  minReplicaCount: 3
  maxReplicaCount: 30
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown: { stabilizationWindowSeconds: 300 }
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://kps-prometheus.monitoring.svc:9090
        query: sum(rate(http_server_request_duration_seconds_count{job="orbit/orders"}[1m]))
        threshold: "100"          # ~100 RPS per pod (measured by load test, §27.7)
    - type: cpu                   # safety net: CPU too
      metricType: Utilization
      metadata: { value: "70" }

KEDA has 70+ scalers: Kafka lag, SQS/PubSub/Service Bus depth, RabbitMQ, NATS, Postgres queries, cron schedules (pre-scale before a known daily peak), and more. ScaledJob runs one Job per message for long tasks such as video transcoding.

War story: a team autoscaled Kafka consumers on CPU. Consumers were I/O-bound (waiting on a slow downstream), so CPU stayed at 20% while lag grew to 6 hours. Scaling on consumer lag fixed it. Note that for Kafka, replicas beyond the partition count do nothing, so set maxReplicaCount ≤ partitions.

⚠️ Caveat (verify): the KEDA redis-streams trigger's lagCount/activationLagCount need a recent KEDA and Redis 7+. On older versions use pendingEntriesCount, which measures delivered-but-unacked entries, not backlog. Check the KEDA scaler docs for your version. The Prometheus trigger assumes the metric and label names in §30; check that the query returns data in Prometheus first.

27.4 Scaling on latency, carefully

Scaling on p95 latency sounds ideal, but latency rises for reasons that more pods don't fix (a slow DB, lock contention). Adding pods then adds DB connections and makes it worse. Prefer throughput or saturation signals (RPS, in-flight requests, queue depth, CPU) for scaling, and use latency for alerting.

Also check connection math: maxReplicas × DB_MAX_CONNS must stay below Postgres max_connections (30 × 10 = 300 > 200 in our config!). Use a pooler (PgBouncer, which CNPG provides via its Pooler resource) or lower per-pod pools.

27.5 VPA: right-sizing recommendations

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata: { name: orders }
spec:
  targetRef: { apiVersion: apps/v1, kind: Deployment, name: orders }
  updatePolicy: { updateMode: "Off" }     # recommend only; a human/PR applies it

kubectl describe vpa orders     # Target / Lower Bound / Upper Bound per container

Don't let VPA (in Auto/Recreate mode) and HPA act on the same metric (CPU/memory); they fight. The common pattern is HPA on RPS/queue, plus VPA in Off mode feeding periodic right-sizing PRs.

27.6 Node autoscaling: Cluster Autoscaler vs Karpenter

apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: general }
spec:
  template:
    spec:
      nodeClassRef: { group: karpenter.k8s.aws, kind: EC2NodeClass, name: default }
      requirements:
        - { key: kubernetes.io/arch,             operator: In, values: [amd64, arm64] }
        - { key: karpenter.sh/capacity-type,     operator: In, values: [spot, on-demand] }
        - { key: karpenter.k8s.aws/instance-category, operator: In, values: [c, m, r] }
      expireAfter: 720h                # recycle nodes monthly: patched AMIs, no snowflakes
  limits: { cpu: "500" }               # cost ceiling
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m
    budgets: [{ nodes: "10%" }]        # disrupt at most 10% of nodes at once

Overprovisioning hides node spin-up time (60–120 s). Run low-priority "balloon" pods that reserve spare capacity. Real pods preempt them instantly, and the evicted balloons trigger a new node in the background.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: { name: overprovisioning }
value: -10
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: overprovisioning, namespace: kube-system }
spec:
  replicas: 2
  selector: { matchLabels: { app: overprovisioning } }
  template:
    metadata: { labels: { app: overprovisioning } }
    spec:
      priorityClassName: overprovisioning
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.10
          resources: { requests: { cpu: "1", memory: 2Gi } }

⚠️ Caveat (verify): Karpenter field names follow the karpenter.sh/v1 API (v1.0+). The instance-category label and EC2NodeClass details are AWS-specific and change with releases. Validate with kubectl apply --dry-run=server on a cluster that has Karpenter installed.

27.7 Prove it: a load test that triggers scaling

// load/orders.js
import http from 'k6/http';
import { check } from 'k6';
import { uuidv4 } from 'https://jslib.k6.io/k6-utils/1.4.0/index.js';

export const options = {
  scenarios: {
    ramp: {
      executor: 'ramping-arrival-rate',        // open model: fixed RPS regardless of latency
      startRate: 20, timeUnit: '1s', preAllocatedVUs: 200, maxVUs: 1000,
      stages: [
        { target: 200, duration: '2m' },
        { target: 800, duration: '5m' },       // should force scale-out
        { target: 800, duration: '5m' },
        { target: 0,   duration: '2m' },
      ],
    },
  },
  thresholds: {
    http_req_failed:   ['rate<0.01'],          // < 1% errors
    http_req_duration: ['p(95)<300'],          // p95 < 300 ms
  },
  insecureSkipTLSVerify: true,
};

const BASE = __ENV.BASE || 'https://orbit.localtest.me';

export default function () {
  const res = http.post(`${BASE}/api/orders`,
    JSON.stringify({ product_id: 1 + Math.floor(Math.random() * 3), quantity: 1 }),
    { headers: { 'Content-Type': 'application/json', 'Idempotency-Key': uuidv4() } });
  check(res, { created: (r) => r.status === 201 });
}

k6 run load/orders.js &
watch -n2 'kubectl get hpa,scaledobject -n orbit; kubectl get pods -n orbit | grep -c Running'
kubectl get events -n orbit --field-selector reason=SuccessfulRescale -w

What you should observe: orders replicas climb as RPS crosses ~100/pod; worker goes 0 → N as lag builds and back to 0 after the cooldown; p95 holds below threshold. If it doesn't, Part IV gives you the tools to see why.

Part III checkpoint: zero-error rolling deploys (rerun the §20 exercise 1 during a k6 run: 0 failures), pods spread across zones, autoscaling on meaningful signals, and one golden manifest you can justify line by line.