12. Kubernetes architecture: the control loop

12.1 The one idea: declarative desired state + reconciliation

You don't tell Kubernetes what to do. You tell it what should be true ("3 replicas of orders:1.4.2 exist"). Controllers then loop forever:

        ┌──────────────────────────────────────────────┐
        │  observe actual state  (watch the API server) │
        │            │                                  │
        │            ▼                                  │
        │  diff against desired state (spec vs status)  │
        │            │                                  │
        │            ▼                                  │
        │  act to converge (create/delete/update)       │
        └────────────┴──────────────────────────────────┘
                  repeat forever

Every "self-healing" behavior is just this loop. A pod dies, so actual is 2, desired is 3, and a new pod is created. A node disappears, its pods are marked gone, and they get replaced elsewhere. When you read Kubernetes objects, read spec (what you asked for) vs status (what the controller observed).

12.2 Components

                       CONTROL PLANE                                          WORKER NODE (xN)
  ┌───────────────────────────────────────────────────────┐        ┌──────────────────────────────────┐
  │ kube-apiserver  ◄──── kubectl / controllers / kubelets │        │ kubelet   ── CRI ──► containerd  │
  │      │  (the only thing that talks to etcd)            │◄──────►│   │ runs pods, probes, reports   │
  │      ▼                                                 │        │ kube-proxy (or eBPF CNI)         │
  │ etcd (Raft, 3 or 5 members: the cluster's database)    │        │   │ Service VIP -> pod IPs       │
  │ kube-scheduler (assigns pods to nodes)                 │        │ CNI plugin (pod network)         │
  │ kube-controller-manager (Deployment, ReplicaSet, Node, │        │ CSI node plugin (volumes)        │
  │   Job, EndpointSlice, ... controllers)                 │        └──────────────────────────────────┘
  │ cloud-controller-manager (LBs, node lifecycle on cloud)│
  └───────────────────────────────────────────────────────┘

Real-world example: managed Kubernetes (EKS, GKE, AKS) runs the control plane for you. You operate the nodes, the add-ons, and your workloads. Etcd backup, API server scaling and upgrades become the provider's problem. That's why almost everyone uses managed control planes in production.

12.3 What happens on kubectl apply -f deployment.yaml

1. kubectl sends the object to the API server: authn → authz (RBAC) → mutating admission (defaults, sidecar injection) → schema validation → validating admission (policies) → persisted to etcd.
2. The Deployment controller sees a new Deployment and creates a ReplicaSet.
3. The ReplicaSet controller creates 3 Pod objects (with no node assigned yet).
4. The scheduler filters nodes (resources, taints, affinity), scores them, and binds each pod to a node.
5. That node's kubelet sees the pod, asks the CNI for an IP, the CSI for volumes, and containerd to pull and start containers. Then it runs probes.
6. Once the readiness probe passes, the EndpointSlice controller adds the pod IP to the Service, and kube-proxy programs the node's routing.

Every step is an independent controller watching the API. Nothing calls anything directly. When something is stuck, ask "which controller should have acted, and what does the object's status/events say?"

12.4 A production-like local cluster with kind

kind ("Kubernetes in Docker") runs each node as a container. Use a multi-node cluster with zone labels so scheduling, anti-affinity and topology spread behave like a real cloud.

# kind-orbit.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: orbit
nodes:
  - role: control-plane
  - role: worker
    labels: { topology.kubernetes.io/zone: zone-a }
  - role: worker
    labels: { topology.kubernetes.io/zone: zone-b }
  - role: worker
    labels: { topology.kubernetes.io/zone: zone-c }

kind create cluster --config kind-orbit.yaml
kubectl get nodes -L topology.kubernetes.io/zone

# LoadBalancer Services on kind: run cloud-provider-kind in another terminal
go install sigs.k8s.io/cloud-provider-kind@latest
sudo cloud-provider-kind

# metrics-server (kubectl top, HPA). kind's kubelet certs are self-signed, hence the flag.
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm upgrade --install metrics-server metrics-server/metrics-server -n kube-system \
  --set args='{--kubelet-insecure-tls}'

# load locally built images into the cluster nodes (no registry needed)
for s in catalog orders worker; do
  docker build --build-arg SERVICE=$s -t orbit-$s:dev . && kind load docker-image orbit-$s:dev --name orbit
done

Alternatives: k3d (k3s in Docker, very fast), minikube (many drivers and add-ons), OrbStack/Docker Desktop's built-in Kubernetes (single node). Use kind because it is what Kubernetes itself uses for CI and supports multi-node.

⚠️ Caveat (verify): cloud-provider-kind flags and its macOS behavior change between releases. On macOS the LoadBalancer IP may not be reachable from the host; port-forward instead (§43.1), or use the Linux lab VM (§0), where it works natively. The metrics-server --set args='{...}' syntax varies by chart version; if it's rejected, put args: in a values file. Multi-node kind needs raised inotify limits on Linux hosts (fs.inotify.max_user_watches=524288, max_user_instances=512); the §0 bootstrap sets them.

12.5 kubectl, efficiently

# contexts and namespaces
kubectl config get-contexts
kubectx kind-orbit           # switch cluster
kubens orbit                 # switch default namespace

# discovery: the built-in documentation
kubectl api-resources                        # every type, short names, namespaced?
kubectl explain deployment.spec.strategy --recursive
kubectl explain pod.spec.containers.livenessProbe

# read
kubectl get pods -o wide                     # node + IP
kubectl get pods -l app.kubernetes.io/name=orders -w   # watch by label
kubectl get deploy orders -o yaml | yq '.status'
kubectl describe pod <pod>                   # EVENTS at the bottom: read them first
kubectl get events --sort-by=.lastTimestamp -A | tail -30
kubectl get pods -o custom-columns=NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount,NODE:.spec.nodeName

# write: always declarative in real life
kubectl apply -f manifests/ --server-side
kubectl diff -f manifests/                   # what WOULD change (use before every apply)

# generate YAML instead of hand-writing from memory
kubectl create deployment demo --image=nginx --replicas=2 --dry-run=client -o yaml > demo.yaml

# interact
kubectl logs deploy/orders -f --all-containers --since=10m
stern orders -n orbit                        # tail all pods matching "orders", color-coded
kubectl exec -it deploy/orders -- sh         # (not on distroless; see §33.3)
kubectl port-forward svc/orders 8082:80
k9s                                          # terminal UI; learn it, you'll live in it

Best practice: in production, humans don't kubectl apply. Git + Argo CD does (§36). Humans use kubectl read-only, plus break-glass actions that are audited. Practice that discipline on kind too.