37. Tuning knobs on Linux (sysctl reference)
Apply with sysctl -w key=value (runtime) and persist in
/etc/sysctl.d/99-net.conf. Measure before and after. Defaults are good for
most hosts; tune deliberately for a known workload (WAN bulk transfer, high-
connection-count edge proxy, low-latency RPC).
37.1 Core buffers & backlog
| sysctl |
Typical default |
When to raise |
Notes |
net.core.rmem_max / wmem_max |
208 KB–4 MB |
WAN bulk transfer |
absolute cap for SO_RCVBUF/SO_SNDBUF and TCP autotune max |
net.core.netdev_max_backlog |
1000 |
10G+ NICs, softirq drops |
per-CPU queue between NIC and stack; raise to 5000–30000 if nstat shows netdev_backlog drops |
net.core.somaxconn |
4096 (older: 128) |
busy accept()ers |
must also pass a large backlog to listen() |
net.core.default_qdisc |
fq_codel |
fq for BBR / pacing |
bufferbloat control |
37.2 TCP memory & windows
| sysctl |
Meaning |
net.ipv4.tcp_rmem = min default max |
per-socket receive buffer autotuning bounds (bytes). Raise max to ~2×BDP for WAN. |
net.ipv4.tcp_wmem = min default max |
per-socket send buffer autotuning bounds. |
net.ipv4.tcp_mem = low pressure high (pages!) |
system-wide TCP memory limits in 4 KB pages. Under memory pressure the stack shrinks windows and prunes. Raise on hosts with millions of sockets. |
net.ipv4.tcp_moderate_rcvbuf = 1 |
enable receive-buffer autotuning (keep on). |
net.ipv4.tcp_window_scaling = 1 |
keep on always. |
net.ipv4.tcp_adv_win_scale |
app-data vs window split of the rcvbuf. |
37.3 Connection setup / teardown
| sysctl |
Meaning |
net.ipv4.tcp_max_syn_backlog |
SYN-queue size (half-open). Raise to 8192–65536 on public-facing servers. |
net.ipv4.tcp_syncookies = 1 |
SYN-flood protection; keep on. |
net.ipv4.tcp_syn_retries / tcp_synack_retries |
initial SYN / SYN-ACK retransmit count (~6/5 → ~127 s / ~31 s). Lower for faster failover. |
net.ipv4.tcp_abort_on_overflow = 0 |
on accept-queue overflow, drop the ACK (client retries) rather than RST. Keep 0 unless you prefer fast failure. |
net.ipv4.tcp_fin_timeout = 60 |
bounds FIN_WAIT_2. |
net.ipv4.tcp_tw_reuse = 2 |
reuse TIME_WAIT for outbound connects when safe (timestamps). Set 1 on busy clients. Never tcp_tw_recycle (removed/broken). |
net.ipv4.ip_local_port_range = 32768 60999 |
ephemeral source ports. Widen to 1024 65535 on heavy outbound-connection hosts. |
net.ipv4.tcp_max_tw_buckets |
cap on TIME_WAIT sockets (excess are killed early). |
net.ipv4.tcp_max_orphans |
cap on orphaned (no fd) sockets; DoS guard. |
37.4 Reliability / recovery
| sysctl |
Meaning |
net.ipv4.tcp_sack / tcp_dsack = 1 |
keep on — essential on lossy/long paths. |
net.ipv4.tcp_fack |
deprecated (RACK supersedes). |
net.ipv4.tcp_recovery = 1 |
enable RACK-TLP loss detection (default on modern kernels). |
net.ipv4.tcp_early_retrans |
early-retransmit / TLP behavior. |
net.ipv4.tcp_frto = 2 |
spurious-RTO detection & undo. |
net.ipv4.tcp_retries2 = 15 |
~13–30 min before killing an established connection. Lower (e.g. 8 ≈ ~100 s) for faster failover clusters; or use TCP_USER_TIMEOUT per socket. |
net.ipv4.tcp_slow_start_after_idle = 1 |
reset cwnd after an idle period. Set 0 for long-lived keep-alive connections / HTTP/2 origins to avoid re-ramping after every idle gap. |
net.ipv4.tcp_notsent_lowat |
limit unsent bytes in the send buffer → lower latency for apps that write large buffers then want to react (e.g. adaptive video). Try 131072. |
37.5 Congestion control / pacing / ECN
| sysctl |
Meaning |
net.ipv4.tcp_congestion_control |
cubic (default) / bbr / dctcp (DC only). |
net.ipv4.tcp_available_congestion_control |
loaded modules. |
net.core.default_qdisc = fq |
needed for BBR pacing (or fq_codel). |
net.ipv4.tcp_ecn = 2 |
accept ECN if peer requests; 1 also actively requests it. |
net.ipv4.tcp_pacing_ss_ratio / tcp_pacing_ca_ratio |
pacing aggressiveness in slow start / congestion avoidance. |
37.6 Offloads, routing, misc
| sysctl / tool |
Meaning |
net.ipv4.tcp_mtu_probing = 1 (or 2) |
PLPMTUD: probe for MTU when a black hole is suspected — mitigates ICMP-filtered paths (Ch. 8). 2 with a set base MSS is aggressive. |
net.ipv4.tcp_timestamps = 1 |
keep on (RTT accuracy + PAWS); 0 only if a middlebox mangles them. |
net.ipv4.tcp_fastopen = 3 |
enable TFO client+server. |
net.ipv4.ip_forward = 1 |
make the host a router. |
net.ipv4.conf.all.rp_filter = 1/2 |
reverse-path filter: 1 strict (drops asymmetric), 2 loose (multi-homed hosts). |
net.netfilter.nf_conntrack_max |
firewall state table size; raise on NAT gateways/LBs or new flows get dropped (nf_conntrack: table full). |
net.ipv4.neigh.default.gc_thresh1/2/3 |
ARP/ND cache GC thresholds; raise on hosts with thousands of neighbors or you get neighbour table overflow. |
fs.file-max, ulimit -n |
fd ceilings — a million connections needs a million fds. |
37.7 A sample profile: WAN bulk-transfer host
# /etc/sysctl.d/99-wan-transfer.conf
net.core.rmem_max = 268435456
net.core.wmem_max = 268435456
net.ipv4.tcp_rmem = 4096 131072 268435456
net.ipv4.tcp_wmem = 4096 131072 268435456
net.ipv4.tcp_mem = 786432 1048576 26777216
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_mtu_probing = 1
net.core.netdev_max_backlog = 30000
37.8 A sample profile: high-connection-count edge proxy
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_slow_start_after_idle = 0
net.netfilter.nf_conntrack_max = 2000000
fs.file-max = 3000000
# plus: ulimit -n 1048576 for the proxy process
37.9 War story
An SRE copied a "10 Gbit tuning" blog's sysctls onto every host in the fleet,
including tiny 512 MB edge boxes. net.ipv4.tcp_mem was set to huge values and
tcp_rmem max to 64 MB. Under a connection surge those boxes OOM-killed the proxy
because a few thousand connections × tens of MB of window blew past RAM.
Lesson: buffer maxima are per socket; on memory-constrained or
high-fan-in hosts, keep tcp_rmem/wmem max modest and cap total with
tcp_mem. Tune per role, not per blog post.
Try it yourself
sysctl -a --pattern 'net.(ipv4.tcp|core)' # dump all current values
nstat -az # the counters to watch before/after
watch -n1 'nstat -az | grep -E "ListenDrops|ListenOverflows|TCPBacklogDrop|PruneCalled|RcvPruned|OfoPruned|TW"'
cat /proc/net/sockstat # TCP mem pages in use vs tcp_mem
