49. Offloads: checksum, TSO/GSO/GRO, LRO, RSS, and XDP
49.1 The offloads and what they do
| Offload | Direction | What it does | Effect |
|---|---|---|---|
| Checksum offload (CSO) | TX/RX | NIC computes/verifies IP & TCP/UDP checksums | frees CPU; local captures show "checksum incorrect" because tcpdump sees the packet before the NIC fills it in — normal, ignore |
| TSO (TCP Segmentation Offload) | TX | kernel hands the NIC one big (up to 64 KB) TCP "segment"; NIC chops it into MTU-sized frames + per-frame headers/checksums | massively fewer trips through the stack; ~2–4× throughput per core |
| GSO (Generic Segmentation Offload) | TX | same idea done in software just before the driver — for NICs/protocols without hardware TSO (also USO for UDP, GSO for GRE/VXLAN) | most of TSO's benefit, hardware-independent |
| GRO (Generic Receive Offload) | RX | merges consecutive same-flow segments into one large skb before the stack processes them (strict rules so it's reversible) | fewer stack traversals; also inflates what a capture shows as one 40 KB "segment" |
| LRO (Large Receive Offload) | RX | hardware version of GRO, but lossy/aggressive (can't always be un-merged) | disable LRO on routers/bridges/hypervisors — it corrupts forwarding; GRO is the safe choice |
| RSS (Receive Side Scaling) | RX | hardware hashes 5-tuple → spreads flows across RX queues/CPUs | multicore scaling |
| RFS/aRFS | RX | steer a flow to the CPU running its socket's app thread | cache locality |
| VLAN offload | TX/RX | NIC inserts/strips 802.1Q tag | |
| UDP GSO/GRO | TX/RX | segmentation/aggregation for UDP — critical for QUIC performance | closes much of the QUIC-vs-TCP CPU gap |
| Tunnel offloads | TX/RX | checksum/TSO awareness inside VXLAN/GENEVE/GRE | needed for overlay networks (Ch. 56) to not tank throughput |
| TLS offload (kTLS + NIC) | TX/RX | kernel (and optionally NIC) does record encryption | high-throughput HTTPS/CDN |
49.2 Managing them with ethtool -k
ethtool -k eth0 # show all offload settings
ethtool -K eth0 tso off gso off gro off # turn off (for debugging / capture accuracy)
ethtool -K eth0 lro off # keep LRO OFF on anything that forwards
ethtool -K eth0 rx-udp-gro-forwarding on # helps forwarded QUIC/UDP throughput
When to disable offloads:
- Accurate packet capture / IDS / timing analysis — GRO/LRO/TSO merge or split packets so a capture no longer reflects the wire. Disable GRO+LRO (and optionally TSO/GSO) on the box while capturing, or capture on a mirror/TAP port.
- A router / bridge / firewall / hypervisor host — LRO must be off; GRO is usually fine and re-segmented correctly on forward.
- Debugging weird MTU/checksum bugs.
- Otherwise: leave offloads on — they're a large multiple of throughput per CPU.
49.3 XDP and AF_XDP (kernel bypass, sort of)
XDP (eXpress Data Path): run an eBPF program in the driver, on the raw
RX descriptor, before an sk_buff is even allocated. Verdicts: XDP_DROP
(discard — line-rate DDoS filtering, tens of Mpps per core), XDP_TX (bounce
back out — load balancer, e.g. Katran, Cilium), XDP_PASS (into the
normal stack), XDP_REDIRECT (to another NIC or into an AF_XDP userspace
socket for full DPDK-like bypass without leaving the kernel framework).
Uses: volumetric DDoS scrubbing, L4 load balancing at the host, packet-level observability, custom forwarding — all without the cost of the full netstack.
49.4 Full kernel bypass: DPDK / netmap
For the extreme end (NFV, trading, 100G+ single-box routing), DPDK takes the NIC away from the kernel entirely: userspace poll-mode drivers, hugepages, busy- polling cores, zero interrupts. You get ~100+ Mpps but you also reimplement ARP, routing, TCP (or use a userspace stack like VPP, F-Stack, mTCP), and you burn whole cores at 100%. Only worth it when the kernel genuinely can't keep up and XDP isn't enough.
49.5 War story
An IDS was "missing attacks" that a PCAP replay clearly contained. On the live
tap host, GRO was on, so the IDS saw coalesced 40 KB pseudo-segments and its
signature engine (expecting wire-sized packets, checking TCP flags per segment)
mis-parsed streams. ethtool -K eth0 gro off lro off on the capture interface
restored detection. Throughput of the monitored traffic was unaffected (it's a
passive tap). Lesson: any tool that reasons about individual packets —
IDS/IPS, latency measurement, forensic capture — needs GRO/LRO off on its
capture path, or a hardware TAP that sees the true wire.
Try it yourself
ethtool -k eth0 | grep -E 'tcp-segmentation|generic-.*-offload|large-receive|rx-checksumming|scatter'
# Before a capture on the host itself:
sudo ethtool -K eth0 gro off lro off ; sudo tcpdump ... ; sudo ethtool -K eth0 gro on
# See GRO in action: capture with offloads on vs off, compare 'length' values
ip -s link show eth0 # RX/TX packets, dropped, errors
bpftool prog show # any XDP/eBPF programs attached?
ip link show eth0 | grep -o 'xdp.*'