34. Keepalive, and detecting dead peers

34.1 The problem

An idle ESTABLISHED TCP connection sends nothing. If the peer host crashes, loses power, or a NAT/firewall silently drops the mapping, neither side knows until it next tries to send data (and then eats retransmits + RTO for minutes) or forever (if it's only reading).

34.2 TCP keepalive

When enabled (SO_KEEPALIVE on the socket), after the connection is idle for tcp_keepalive_time, the kernel sends a keepalive probe (a segment with seq = snd_nxt − 1 and no data — deliberately "old" so the peer must ACK it).

Defaults ⇒ ~2 h 11 min to detect a dead peer. Almost always too slow. The sysctls are system-wide; the TCP_KEEP* socket options let each app choose (e.g. 60 s idle, 10 s interval, 3 probes ⇒ ~90 s detection).

34.3 TCP_USER_TIMEOUT — the better knob

TCP_USER_TIMEOUT (ms) bounds how long transmitted but unacked data may remain before the connection is killed — this also caps the retransmission sequence for active connections (not just idle ones), overriding tcp_retries2. Combined with keepalive it gives predictable failure timing: "this connection will error out within N seconds of the peer becoming unreachable, whether idle or busy." Widely used by databases and gRPC.

34.4 Application-level keepalive / heartbeats

Often better than TCP keepalive because:

• It traverses proxies/load balancers that terminate TCP (each hop has its own connection; TCP keepalive on segment A tells you nothing about segment C).
• It can carry liveness of the application, not just the socket (an app can be deadlocked while its kernel still ACKs).
• Protocol examples: HTTP/2 PING frames, gRPC keepalive pings, WebSocket ping/pong, SSH ServerAliveInterval, MQTT PINGREQ, Redis TCP keepalive + PING.

34.5 Keepalive and NAT/firewall idle timeouts

Stateful middleboxes drop idle flows: common TCP idle timeouts are 5 min (consumer NAT), 30–60 min (enterprise firewalls), and AWS NLB is 350 s, ELB idle default 60 s. If your keepalive/heartbeat interval is longer than the shortest middlebox timeout on the path, the flow gets silently reaped and the next send fails. Rule: heartbeat interval < min middlebox idle timeout (e.g. ≤ 60 s to be safe behind unknown LBs; ≤ 30 s for UDP).

34.6 War story

A microservice held a pool of 50 long-lived gRPC connections to a database proxy through an AWS NLB. Every night during a traffic lull, the first queries after ~6 minutes of idle failed with connection reset, then recovered. NLB's 350 s idle timeout silently dropped the flows; the client only found out on the next RPC. Fix: gRPC keepalive time=60s, timeout=20s, permitWithoutStream=true so the flows never idle past 350 s. Lesson: long-lived connections need a heartbeat shorter than every idle timeout on the path, or they die quietly during quiet periods.

Try it yourself

sysctl net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_intvl net.ipv4.tcp_keepalive_probes
ss -tieo dst <peer>        # 'timer:(keepalive,...)' shows keepalive countdown per socket
# ssh keepalive:
grep -E 'ServerAlive' ~/.ssh/config     # ServerAliveInterval 30 / ServerAliveCountMax 3
# Set aggressive keepalive in Python:
python3 - <<'EOF'
import socket
s = socket.create_connection(("example.com", 80))
s.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 30)
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 10)
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 3)
EOF