27. Sequence numbers, cumulative ACKs, and retransmission

27.1 The model

Every byte in each direction has a sequence number. The receiver returns an acknowledgment number = "the next byte I expect" = one past the highest contiguous byte received. This is a cumulative ACK: ack=5001 means "I have everything through byte 5000," regardless of what arrived in between.

Sender sends 4 segments, 1000 B each, starting seq=1:
  seg A: seq 1     .. 1000
  seg B: seq 1001  .. 2000
  seg C: seq 2001  .. 3000   <-- LOST
  seg D: seq 3001  .. 4000

Receiver:
  gets A -> ACK 1001
  gets B -> ACK 2001
  gets D -> still ACK 2001   (can't ack past the hole; this is a "duplicate ACK")

27.2 Two ways a loss is detected

1. Retransmission timeout (RTO): the sender's timer for the oldest unacked segment expires with no progress → resend it. Slow (RTO ≥ 200 ms, often 1 s on the first go) and drastic (congestion window collapses to 1 — Ch. 31).
2. Duplicate ACKs → fast retransmit: three ACKs carrying the same ack number tell the sender "a later segment arrived but there's a gap at ack." The sender resends that one segment immediately, without waiting for the RTO (Ch. 29). Much faster.

27.3 What the receiver does with out-of-order data

It buffers it (up to the receive window) and keeps ACKing the last in-order byte. When the missing segment finally arrives, the ACK jumps forward past all the buffered data in one step. Without SACK, the sender can't tell which later segments arrived, so a naive sender might resend B, C, and D ("go-back-N"–ish). SACK (Ch. 29) fixes exactly this.

27.4 Spurious retransmissions

If an ACK is merely delayed (not lost) and the RTO is too tight, the sender retransmits data the receiver already has. The receiver drops the duplicate and ACKs normally, but the sender needlessly cut its window. Mitigations:

• Karn's algorithm (Ch. 28): don't measure RTT from a retransmitted segment.
• TCP timestamps: let the receiver echo which send the ACK corresponds to → the sender can tell a retransmit was spurious.
• F-RTO (net.ipv4.tcp_frto, on by default): a heuristic that detects a spurious timeout from the pattern of the next ACKs and un-does the window reduction.
• DSACK (duplicate SACK): the receiver explicitly reports "you sent me this block twice," letting the sender learn its RTO/reordering estimate was wrong.

27.5 Reordering vs loss

The internet reorders packets (ECMP path changes, link bonding, parallel switch fabrics). Three dup-ACKs from reordering (not loss) trigger an unnecessary fast retransmit. Linux adapts: tcp_reordering starts at 3 and RACK-TLP (tcp_recovery, default on modern kernels) replaces the dup-ACK counting with a time-based notion — "a segment is lost if a later-sent segment was ACKed and enough time has passed" — which is far more robust to reordering.

27.6 War story

A storage replication link across two data centers showed 6% "retransmissions" in nstat but zero actual packet loss on the carrier's reports. tcpdump + Wireshark's Expert Info showed spurious retransmissions and lots of out-of-order segments: the pair of 10G links was hashing the flow's packets across both members inconsistently, reordering them. Fixing the LAG hash to be per-flow (5-tuple) dropped "retransmissions" to ~0.02% and doubled throughput. Lesson: high "retransmit" counters with no carrier loss = reordering; look at out-of-order and spurious-retransmit stats, and check LACP/ECMP hashing.

Try it yourself

nstat -az | grep -Ei 'retrans|reorder|dsack|spurious|lostretransmit'
# e.g. TcpRetransSegs, TcpExtTCPSpuriousRTOs, TcpExtTCPDSACKRecv, TcpExtTCPReordering
ss -tie dst <peer>          # per-socket: retrans:X/Y  (current/total), reordering N
sysctl net.ipv4.tcp_recovery net.ipv4.tcp_frto net.ipv4.tcp_reordering