8. Simulating Network Conditions

Load (concurrency/throughput) is only half the story — real users are on imperfect networks. Simulating latency, jitter, and packet loss surfaces bugs that a pristine localhost/datacenter test never will (timeout tuning, retry storms, TCP behavior under loss).

8.1 tc + netem (Linux) — kernel-level network emulation

# add 100ms latency +/- 20ms jitter (normal distribution) on eth0 egress
sudo tc qdisc add dev eth0 root netem delay 100ms 20ms distribution normal

# add 2% packet loss on top
sudo tc qdisc change dev eth0 root netem delay 100ms 20ms loss 2%

# simulate a constrained mobile link: 100ms delay, 1% loss, 1mbit cap
sudo tc qdisc add dev eth0 root handle 1: netem delay 100ms loss 1%
sudo tc qdisc add dev eth0 parent 1: handle 10: tbf rate 1mbit burst 32kbit latency 400ms

# remove all shaping when done
sudo tc qdisc del dev eth0 root

For containerized targets, apply netem inside the container's network namespace (or on a dedicated bridge) so you don't degrade the host's other traffic:

# apply to a specific docker container's interface
CID=$(docker inspect -f '{{.State.Pid}}' my_service_container)
sudo nsenter -t $CID -n tc qdisc add dev eth0 root netem delay 150ms 30ms loss 1%

Common real-world profiles to script as reusable presets:

8.2 Toxiproxy — application-level fault injection proxy

Unlike tc (kernel-wide, needs root, per-interface), Toxiproxy sits as a TCP proxy in front of a specific dependency (DB, downstream API) and lets you inject/ remove faults at runtime via HTTP API — ideal for automated test suites.

docker run -d --name toxiproxy -p 8474:8474 -p 5433:5433 ghcr.io/shopify/toxiproxy

# create a proxy: app connects to localhost:5433 instead of the real DB directly
curl -X POST http://localhost:8474/proxies \
  -d '{"name":"postgres","listen":"0.0.0.0:5433","upstream":"postgres_real:5432"}'

# inject 200ms latency with 50ms jitter on all upstream traffic
curl -X POST http://localhost:8474/proxies/postgres/toxics \
  -d '{"type":"latency","attributes":{"latency":200,"jitter":50}}'

# simulate the DB link going completely dark (timeout testing)
curl -X POST http://localhost:8474/proxies/postgres/toxics \
  -d '{"type":"timeout","attributes":{"timeout":0}}'

Run your standard load test (k6/Locust/wrk) against the app while Toxiproxy injects DB latency — this directly validates the timeout/circuit-breaker/bulkhead behavior described in §5.3, under real concurrent load rather than a single manual request.

8.3 Comcast (Go) — simpler netem wrapper, cross-platform-ish CLI

go install github.com/tylertreat/comcast@latest
comcast --latency=150 --packet-loss=5% --target-bw=1000 --device=eth0
comcast --stop   # remove all rules