45. Emulating bad networks with tc/netem

tc (traffic control) with the netem qdisc lets you inject latency, jitter, loss, reordering, duplication, and corruption on an interface — essential for testing how your app behaves on real-world links before users find out.

45.1 Basic recipes

IFACE=eth0    # apply to the EGRESS of this interface (affects packets leaving)

# 100 ms fixed latency:
sudo tc qdisc add dev $IFACE root netem delay 100ms

# 100 ms +/- 20 ms jitter, correlation 25% (successive delays are related):
sudo tc qdisc change dev $IFACE root netem delay 100ms 20ms 25%

# 1% random packet loss:
sudo tc qdisc change dev $IFACE root netem loss 1%

# Correlated (bursty) loss - Gilbert-Elliott-ish:
sudo tc qdisc change dev $IFACE root netem loss 1% 25%

# Reordering: 25% of packets sent 10ms early (needs a delay baseline):
sudo tc qdisc change dev $IFACE root netem delay 10ms reorder 25% 50%

# Duplication / corruption:
sudo tc qdisc change dev $IFACE root netem duplicate 1%
sudo tc qdisc change dev $IFACE root netem corrupt 0.1%

# Rate limit to 1 Mbit with a small buffer (emulate a slow uplink):
sudo tc qdisc change dev $IFACE root netem rate 1mbit limit 20

# Combine: "bad hotel wifi" = 80ms+/-40ms jitter, 2% bursty loss, 5Mbit:
sudo tc qdisc replace dev $IFACE root netem delay 80ms 40ms rate 5mbit loss 2% 25%

# Remove everything:
sudo tc qdisc del dev $IFACE root

45.2 Applying it to only some traffic

netem on root hits everything (including your SSH session — do this on a test box or over a console). To scope it, use a classful qdisc + filter:

sudo tc qdisc add dev $IFACE root handle 1: prio
sudo tc qdisc add dev $IFACE parent 1:3 handle 30: netem delay 200ms loss 3%
# send only traffic to 10.0.0.9:5432 through the netem band:
sudo tc filter add dev $IFACE parent 1:0 protocol ip u32 \
     match ip dst 10.0.0.9/32 match ip dport 5432 0xffff flowid 1:3

Or, cleaner, use a network namespace / veth pair and shape the veth so only the app-under-test is affected (Ch. 55).

45.3 Emulating asymmetric and both-direction impairment

tc only shapes egress. To impair the return path too, either run netem on the peer's egress, or use an IFB (Intermediate Functional Block) device to redirect ingress through a qdisc:

sudo modprobe ifb numifbs=1
sudo ip link set ifb0 up
sudo tc qdisc add dev $IFACE handle ffff: ingress
sudo tc filter add dev $IFACE parent ffff: protocol ip u32 match u32 0 0 \
     action mirred egress redirect dev ifb0
sudo tc qdisc add dev ifb0 root netem delay 50ms loss 1%

45.4 What to test with it

• Timeouts & retries: does the client back off, or hammer? Does it retry non-idempotent requests (bad)?
• Connection pools: do they detect and evict dead connections under loss?
• Keepalive/heartbeat intervals vs a 30 s "NAT" timeout.
• HTTP/2 vs HTTP/3 under 2% loss (HOL blocking — Ch. 39).
• TLS handshake under high latency + loss (does it complete in bounded time?).
• Congestion control choice: iperf3 -C cubic vs -C bbr under netem loss 1% delay 100ms.
• User-facing SLOs: run the real UI through a "3G" profile.

45.5 Reference impairment profiles

45.6 War story

A mobile app's sync feature passed every test in the office and generated a storm of duplicate writes in the field. Reproduced instantly with tc qdisc add dev eth0 root netem delay 300ms 100ms loss 3%: the client's 2 s request timeout fired while the server was still processing, the client retried a non-idempotent POST /transactions, and both eventually succeeded. Fixes: idempotency keys, exponential backoff with jitter, and a timeout ≥ the server's p99 processing time + 2×RTT. Lesson: most "works in the office" distributed bugs reproduce in one command with netem; add it to CI.

Try it yourself

sudo tc qdisc replace dev eth0 root netem delay 100ms 20ms loss 1%
ping -c10 <peer>            # see the injected delay/jitter/loss
curl -w 'ttfb %{time_starttransfer}\n' -o /dev/null -s https://<peer>/
sudo tc -s qdisc show dev eth0     # sent/dropped/reordered counters
sudo tc qdisc del dev eth0 root    # clean up!