Chapter 58 — Kubernetes networking: Services, Ingress, NetworkPolicy, CNI

In one sentence

Kubernetes networking is normal networking with more layers: pod IPs, Service virtual IPs, kube-proxy or eBPF rules, ingress controllers, cluster DNS, and a CNI plugin, each with its own failure modes.

The problem

A request that fails inside a cluster could have died at any of six layers, most of them invisible to kubectl get pods. Teams new to Kubernetes tend to restart pods until the problem goes away. Teams that understand the packet path find the broken layer in minutes.

The model: four rules

Kubernetes requires every network plugin (CNI) to provide:

Every pod gets its own IP address.
Pods can reach all other pods without NAT.
Nodes can reach all pods (and vice versa) without NAT.
A pod sees itself with the same IP that others use to reach it.

How the CNI meets these rules varies: routing pod CIDRs directly (Calico in BGP mode, cloud VPC CNIs that give pods real VPC addresses) or an overlay (VXLAN/Geneve, Chapter 54). Overlays explain many MTU problems in clusters.

The packet path to remember

client
  -> cloud LoadBalancer / Ingress controller / Gateway   (L7: host + path routing)
  -> Service (a virtual ClusterIP, e.g. 10.96.12.34:80)   (exists only as rules!)
  -> kube-proxy rules (iptables / IPVS / nftables) or eBPF (Cilium): DNAT to ...
  -> EndpointSlice: the list of READY pod IPs
  -> Pod IP:containerPort (through the node's veth into the pod namespace)

The key insight: a ClusterIP isn't bound to any interface. No process listens on it, and you can't ping it in most clusters. It exists only as NAT rules on every node that rewrite the destination to a pod IP chosen from the EndpointSlice. That's why a Service with zero ready endpoints answers with "connection refused" or a timeout, depending on the implementation.

Debug in this order, or reverse it from inside the pod.

How it actually works: the details that cause incidents

1. Cluster DNS and ndots:5. A pod's /etc/resolv.conf usually looks like:

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

ndots:5 means "a name with fewer than 5 dots is tried with every search suffix first". So api.github.com (2 dots) becomes, in order:

api.github.com.default.svc.cluster.local   -> NXDOMAIN
api.github.com.svc.cluster.local           -> NXDOMAIN
api.github.com.cluster.local               -> NXDOMAIN
api.github.com                             -> answer

That's 4 lookups (×2 for A and AAAA) for every external name. On a busy service this is a large share of CoreDNS load and adds latency. Fixes: use a trailing dot (api.github.com.), lower ndots in the pod's dnsConfig, or run NodeLocal DNSCache.

2. The conntrack DNS race. glibc sends the A and AAAA queries in parallel from the same UDP socket. Both packets race through conntrack's NAT on the node, one can lose the insert, and it is silently dropped. The client then waits its full 5-second DNS timeout. If a cluster shows requests taking "exactly 5 s more" now and then, suspect this. Mitigations: NodeLocal DNSCache, options single-request-reopen, or a newer kernel/CNI that fixes the race.

3. externalTrafficPolicy. For LoadBalancer/NodePort Services:

	`Cluster` (default)	`Local`
Which nodes accept traffic	all	only nodes running a ready pod
Extra hop	possibly (node → another node's pod)	no
Client source IP	lost (SNAT to the node's IP)	preserved
Balance	even across pods	uneven if pods are unevenly spread

If your app logs show node IPs instead of client IPs, this is why.

4. Headless Services (clusterIP: None) skip the virtual IP. DNS returns the pod IPs directly. StatefulSets use them for stable per-pod names, and gRPC clients use them for client-side load balancing (Chapter 57's long-lived connection problem).

5. NetworkPolicy. If no policy selects a pod, all traffic is allowed. Once any policy selects a pod, only traffic that some policy allows gets through. And the CNI must enforce NetworkPolicy, which not every CNI does. A default-deny starting point for a namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: shop
spec:
  podSelector: {}                 # every pod in the namespace
  policyTypes: [Ingress, Egress]
  egress:                         # ...but still allow DNS, or nothing will resolve
  - to:
    - namespaceSelector: {}
      podSelector:
        matchLabels: {k8s-app: kube-dns}
    ports:
    - {protocol: UDP, port: 53}
    - {protocol: TCP, port: 53}

Forgetting the DNS exception is the most common NetworkPolicy outage: everything "times out", because nothing can resolve a name.

Commands that answer real questions

kubectl get pod -o wide                          # pod IPs and nodes
kubectl get svc,endpointslices -l app=web        # does the Service have READY endpoints?
kubectl describe ingress web                     # rules, backends, events
kubectl get networkpolicy -A                     # anything selecting these pods?
kubectl logs -n kube-system -l k8s-app=kube-dns  # CoreDNS errors

A throwaway debugging pod with every network tool in it:

kubectl run -it --rm netshoot --image=nicolaka/netshoot -- bash
# inside:
cat /etc/resolv.conf                              # search domains and ndots
dig web.default.svc.cluster.local +short          # Service ClusterIP
dig +search web                                   # same, via search path
curl -sv http://web:80/healthz                    # through the Service
curl -sv http://<pod-ip>:8080/healthz             # bypass the Service: is it the pod or the Service?

On a node (or a kind node container): iptables-save | grep KUBE-SVC | head shows the NAT rules that are the Services.

Practice (45 min): see a Service is just NAT rules

With kind (Linux guide, Chapter 53 has the setup):

kind create cluster --name net
kubectl create deployment web --image=nginx --replicas=3
kubectl expose deployment web --port=80
kubectl get svc web                                      # note the CLUSTER-IP
docker exec net-control-plane iptables-save | grep -E 'KUBE-SVC|KUBE-SEP' | grep -m6 web
kubectl scale deployment web --replicas=1
docker exec net-control-plane iptables-save | grep -c KUBE-SEP   # fewer endpoint rules
kind delete cluster --name net

You'll see one KUBE-SVC-… chain for the Service and one KUBE-SEP-… ("service endpoint") chain per pod, chosen with --probability rules: random load balancing implemented in the firewall.

Common fingerprints

Symptom	Likely cause
Service has no endpoints	label selector doesn't match pods, or pods not Ready
Pod can call by IP but not by name	CoreDNS down, wrong namespace in the name, NetworkPolicy blocking port 53
Random +5 s latency on outbound calls	conntrack DNS race; `ndots:5` amplification
Ingress returns 404	host/path rule mismatch, wrong ingress class
Connection times out only across namespaces	NetworkPolicy
Client IPs in logs are all node IPs	`externalTrafficPolicy: Cluster` SNAT
Large responses fail in cluster	overlay MTU / PMTU issue (Chapter 54)
New pods get no gRPC traffic	long-lived HTTP/2 connections through a ClusterIP (L4) Service

Build it in Go (20 min) — client-side balancing over a headless Service

Use what you built in Chapters 18 and 57: resolve a headless Service with net.DefaultResolver.LookupHost(ctx, "web-headless.default.svc.cluster.local") to get every ready pod IP, then round-robin requests across them yourself and re-resolve every 30 s. This is what gRPC's dns:/// resolver with the round_robin policy does, and it fixes the "new pods get no traffic" row above. Run it in-cluster from a golang pod, scale the Deployment, and watch the IP list change.

Common confusions

"I can ping the Service IP." Usually you can't. There's no interface behind it to answer ICMP. Test with the real port (curl, nc -vz).
"Kubernetes load-balances my gRPC calls." A ClusterIP Service balances connections. One long-lived HTTP/2 connection carries all calls to one pod.
"NetworkPolicy is enforced everywhere." Only if the CNI supports it. Test a deny rule before relying on it.

Check yourself

Why can't a ClusterIP be pinged, and what actually makes it work?
Explain ndots:5 and the cost of resolving api.github.com from a pod.
When would you choose externalTrafficPolicy: Local, and what does it cost?
Write a NetworkPolicy that denies all egress except DNS.