43. Socket statistics: ss, /proc, and connection states
43.1 ss — the modern netstat
ss -tan # all TCP, numeric
ss -tanp # + owning process (needs root for others' sockets)
ss -tl # listening TCP; Recv-Q = accept backlog used, Send-Q = its max
ss -tan state established
ss -tan state time-wait
ss -tan '( sport = :443 )'
ss -tan dst 10.0.0.0/8
ss -s # summary counts by state / mem
ss -tie # -e extended + -i internal TCP info <-- the good one
ss -tim # + memory (rmem/wmem/... per socket)
ss -tio # + timer info (retransmit/keepalive countdown)
ss -natp | grep -c ESTAB
43.2 Decoding ss -tie (the per-socket TCP report)
ESTAB 0 0 10.0.0.5:443 203.0.113.9:54321
ts sack cubic wscale:7,9 rto:212 rtt:11.5/5.2 ato:40 mss:1448 pmtu:1500
rcvmss:536 advmss:1448 cwnd:42 ssthresh:35 bytes_sent:1048576 bytes_acked:1047000
bytes_received:900 segs_out:730 segs_in:20 data_segs_out:729
send 42.3Mbps lastsnd:4 lastrcv:1200 lastack:4 pacing_rate 84.6Mbps
delivery_rate 40.1Mbps busy:980ms retrans:0/3 dsack_dups:1 reordering:5
rcv_rtt:12 rcv_space:65535 minrtt:10.9
| Field | Meaning / use |
|---|---|
ts sack cubic |
options negotiated + congestion control in use |
wscale:7,9 |
send/recv window scale shifts (both nonzero = scaling active) |
rto:212 |
current retransmission timeout (ms). Near TCP_RTO_MIN (200) = healthy low-RTT path |
rtt:11.5/5.2 |
SRTT / RTTVAR (ms). Rising RTTVAR = jittery path |
minrtt |
best RTT ever seen — the path's true propagation delay; rtt ≫ minrtt = queueing/bufferbloat |
ato:40 |
delayed-ACK timeout in effect (ms) |
mss / pmtu / advmss |
segment sizing; pmtu < 1500 = a smaller link on the path |
cwnd / ssthresh |
congestion window / threshold, in segments. cwnd small + retrans climbing = lossy |
retrans:0/3 |
current unrecovered / total retransmits this connection |
send X, delivery_rate Y |
sender's rate estimate vs actually delivered rate |
pacing_rate |
rate the qdisc/stack is pacing at (BBR/fq) |
bytes_acked vs bytes_sent |
gap = data in flight or lost |
rcv_space |
receiver window autotuning target; grows on fast paths |
busy: |
time the socket had data to send (not app-limited) |
app_limited (flag) |
throughput is bounded by the app not writing, not the network |
43.3 /proc and nstat
nstat -az # every TCP/IP/UDP/ICMP kernel counter, since boot, non-zero-friendly
nstat # deltas since last run (great for before/after)
cat /proc/net/snmp # classic SNMP-style counters (Tcp:, Udp:, Ip:, Icmp:)
cat /proc/net/netstat # TcpExt: (the interesting ones)
cat /proc/net/sockstat # sockets in use, TCP mem pages vs tcp_mem
cat /proc/net/tcp /proc/net/tcp6 # raw socket table (hex); ss parses this
Counters worth alerting on:
Counter (nstat/netstat -s) |
Meaning |
|---|---|
TcpExtListenOverflows / ListenDrops |
accept-queue full → dropped connections (Ch. 24) |
TcpExtSyncookiesSent |
under SYN-queue pressure / SYN flood |
TcpExtTCPBacklogDrop |
packets dropped because the socket backlog was full |
TcpExtPruneCalled / RcvPruned / OfoPruned |
out-of-order/receive memory pressure → data discarded |
TcpRetransSegs / TcpExtTCPLostRetransmit |
retransmissions / retransmits themselves lost |
TcpExtTCPTimeouts / TCPSpuriousRTOs |
RTO fired / RTO was a mistake |
TcpExtTCPFastRetrans / TCPSACKRecovery |
fast-path recovery events (good — better than timeouts) |
UdpInErrors / UdpRcvbufErrors / UdpInDatagrams |
UDP receive drops (Ch. 22) |
IpReasmFails / IpReasmReqds |
IP fragment reassembly failures (Ch. 15) |
TcpExtTW / TCPTimeWaitOverflow |
TIME_WAIT churn / hitting tcp_max_tw_buckets |
TcpExtTCPAbortOnMemory / TCPAbortOnTimeout |
connections killed by memory limits / retransmit exhaustion |
nf_conntrack: table full (dmesg) |
firewall state table exhausted → new flows dropped |
43.4 War story
A service's error rate rose with traffic but CPU, memory, and app latency looked
fine. nstat before/after a load test showed TcpExtListenOverflows +14000 and
TcpExtTCPReqQFullDrop. The app's framework defaulted listen() backlog to 128;
bursts overflowed the accept queue and clients saw connect timeouts that the app
never logged (they never became connections). Raising backlog to somaxconn
(4096) + adding SO_REUSEPORT workers cleared it. Lesson: connection-level
failures are invisible to app logs; watch nstat's Listen* / *Drop counters
as first-class SLIs.
Try it yourself
watch -n1 'ss -s'
ss -tie state established | grep -E 'minrtt|retrans|app_limited' | head
nstat -n && sleep 10 && nstat # 10-second delta of every counter
nstat -az | grep -E 'ListenOverflows|ListenDrops|TCPBacklogDrop|SyncookiesSent|PruneCalled'
cat /proc/net/sockstat # 'mem' pages vs `sysctl net.ipv4.tcp_mem`