Part 12 — Inside the server: from the NIC to your handler
"Moving up the stack" on the receiving side means a real sequence of hardware
and kernel events. Here's what happens on the Kubernetes node hosting
orders-service:
L1 Signal arrives at the (virtual) NIC; PHY decodes symbols → bits
L2 NIC checks FCS, matches dst MAC, DMA-copies the frame into a RAM ring buffer
── NIC raises an interrupt; kernel switches to polling (NAPI) to batch packets
── (offloads: the NIC may verify checksums, combine segments (GRO/LRO),
and spread flows across CPU cores by hashing the 5-tuple (RSS))
L2→3 Driver builds an sk_buff; EtherType 0x0800 → hand to IPv4
L3 Validate header; routing decision: "this is for a local pod address"
── netfilter/iptables hooks: conntrack lookup, kube-proxy DNAT
(if addressed to a Service ClusterIP → rewritten to a pod IP)
── forwarded across the node's bridge/veth pair into the pod's network namespace
L4 TCP: find the socket by 5-tuple; check seq numbers; queue in the socket's
receive buffer; send ACK (possibly delayed)
── wakes the process blocked in epoll/kqueue: "socket readable"
L5-7 User space: the runtime's event loop reads bytes → TLS decrypt (if mTLS) →
HTTP parser → router → your handler(req)
Each step has a queue, and every queue can overflow and drop:
| Queue | Overflow symptom | Where to look |
|---|---|---|
| NIC ring buffer | rx_missed / rx_no_buffer counters |
ethtool -S |
Kernel backlog (netdev_max_backlog) |
drops in /proc/net/softnet_stat |
softnet stats |
| Conntrack table | "table full, dropping packet" | dmesg, conntrack -S |
| SYN / accept backlog | ListenOverflows, ListenDrops |
netstat -s, nstat |
| Socket receive buffer | RcvbufErrors, zero-window advertisements |
ss -tm, nstat |
| Application worker pool / event loop | latency rises, then timeouts | APM, event-loop lag metrics |
Rule of thumb: when a service is "slow but CPU is low", look for one of these queues being full, or for the app waiting on a downstream call.
Go deeper in the series
- The kernel receive path, netfilter hooks, and queue overflows, with a lab that fills an accept queue to 4,097: TCP/IP guide, Chapter 52.
- From the program's side: threads vs the netpoller (Linux Chapter 73), file-descriptor limits (Chapter 78), and profiling a live Go service (Chapter 81).