Part XV — Kernel and NIC internals

48. The packet's path through the Linux kernel

48.1 Receive path (RX), top to bottom

 wire
  |
  v
NIC hardware
  - matches dst MAC (or promisc), verifies Ethernet FCS
  - RSS: hashes the packet's 5-tuple -> picks one of N RX queues
  - DMA-writes the frame into a ring-buffer slot (rx ring) in RAM
  - raises an IRQ (or is polled)
  |
  v
Hardware IRQ handler (tiny)  ->  schedules NAPI softirq (NET_RX_SOFTIRQ) on that CPU
  |
  v
NAPI poll (softirq, per RX queue / per CPU)
  - pulls a budget of frames from the ring
  - GRO: merges consecutive same-flow segments into one big super-frame
  - builds sk_buff (skb) for each
  |
  v
netif_receive_skb  ->  tc ingress (qdisc/clsact, XDP already ran earlier at driver)
  |
  v
IP layer:  ip_rcv  -> netfilter PREROUTING (conntrack, DNAT)
  - routing decision: local? -> ip_local_deliver ;  forward? -> ip_forward
  - netfilter: FORWARD or INPUT chains
  |
  v
L4:  tcp_v4_rcv / udp_rcv
  - 5-tuple hash -> find the socket
  - TCP: sequence/ACK processing, into the socket receive queue (or out-of-order queue)
  |
  v
Socket receive buffer (sk_rcvbuf)  ->  wakes the process
  |
  v
recv()/read() syscall copies bytes into user space (or splice/mmap/io_uring)

48.2 Transmit path (TX), top to bottom

send()/write()  -> copies user bytes into sk_sndbuf
  |
  v
TCP: segmentation into MSS units (or defer via TSO), builds headers, seq numbers,
     congestion window check, pacing timestamp
  |
  v
IP: route lookup (dst cache), fills IP header, netfilter OUTPUT/POSTROUTING (SNAT)
  |
  v
Neighbor (ARP/ND): resolve next-hop MAC, build Ethernet header
  |
  v
tc egress qdisc  (fq_codel / fq / cake / htb ...)  <- where shaping & AQM live
  |
  v
Driver xmit -> TX ring slot -> NIC DMA-reads the frame
  - NIC offloads: TSO splits the big segment into MTU-sized frames on the wire,
    computes L3/L4 checksums, optionally inserts VLAN tag
  |
  v
wire

48.3 Where packets get dropped (and the counter that shows it)

Stage	Drop reason	Where it shows
NIC RX ring full	softirq/CPU can't keep up; RX ring too small	`ethtool -S`: `rx_no_buffer`, `rx_missed_errors`, `rx_fifo_errors`; `ip -s link` `RX dropped`
`netdev_max_backlog` full	per-CPU backlog between NAPI and stack overflowed	`/proc/net/softnet_stat` col 2 (drops); raise `net.core.netdev_max_backlog`
softirq budget exhausted	`time_squeeze` — poll ran out of budget/time	`/proc/net/softnet_stat` col 3; tune `net.core.netdev_budget`/`netdev_budget_usecs`
conntrack table full	NAT/stateful firewall state exhausted	`dmesg`: `nf_conntrack: table full, dropping packet`; raise `nf_conntrack_max`
socket rcvbuf full (UDP)	app too slow / buffer too small	`nstat` `UdpRcvbufErrors`; `ss -u` Recv-Q
socket rcvbuf / ofo prune (TCP)	memory pressure, lots of out-of-order	`nstat` `TCPRcvQDrop`, `PruneCalled`, `OfoPruned`
accept queue full	app not `accept()`ing fast enough	`nstat` `ListenOverflows`/`ListenDrops`
qdisc drop	shaper/AQM dropped (this is often intentional — CoDel)	`tc -s qdisc show` `dropped N`
tc/nftables/iptables rule	explicit policy DROP	rule counters (`iptables -nvL`, `nft list ruleset`)

48.4 IRQ affinity, RPS/RFS, and XPS

RSS (hardware) spreads RX across queues by 5-tuple hash → each queue's IRQ is pinned to a CPU (/proc/interrupts, irqbalance or manual smp_affinity). Goal: spread load, keep a flow on one CPU (cache locality), don't land NIC IRQs on the CPUs your app threads run on.
RPS (software RSS) — spread to more CPUs when the NIC has few queues.
RFS — steer a flow's processing to the CPU where the application thread that owns the socket runs (best cache behavior). rps_flow_cnt, rps_sock_flow_entries.
XPS — pick the TX queue based on the sending CPU.
aRFS — hardware-accelerated RFS (NIC steers based on the kernel's hint).

Misconfigured affinity = one CPU at 100% si (softirq) while others idle, and packet drops at moderate pps. Check mpstat -P ALL 1 for a lopsided %soft.

48.5 War story

A 25 GbE cache server dropped packets and spiked p99 at only ~4 Mpps. mpstat showed CPU0 at 100% softirq, others idle. irqbalance had parked all 16 NIC queue IRQs on CPU0-3, and RPS was off. Pinning each queue's IRQ to a distinct core, enabling RFS, and moving the app off the IRQ cores took it to ~14 Mpps with p99 halved and zero drops. Lesson: at high pps the bottleneck is usually per-CPU softirq, not bandwidth; fix IRQ affinity + RFS before buying NICs.

Try it yourself

ethtool -S eth0 | grep -Ei 'drop|miss|err|nobuf|fifo'
ethtool -g eth0                          # ring sizes; ethtool -G eth0 rx 4096
ethtool -l eth0                          # queue counts;  -L to change
cat /proc/net/softnet_stat               # per-CPU: processed, dropped, time_squeeze (hex)
grep eth0 /proc/interrupts               # which CPUs take NIC IRQs
mpstat -P ALL 1                          # look for one CPU pegged on %soft
cat /sys/class/net/eth0/queues/rx-0/rps_cpus