47. Why VoIP and video use UDP, jitter buffers, and RTP

47.1 The timeliness-over-reliability trade

For a live voice/video stream, a packet that arrives after its playout deadline is useless — worse than useless, because waiting for its retransmission stalls everything after it (head-of-line blocking). So real-time media:

• rides UDP (no retransmission, no in-order guarantee, no connection stall),
• accepts loss and conceals it (interpolate a lost 20 ms audio frame; freeze or blur a lost video block until the next keyframe),
• does its own congestion control tuned for low latency.

47.2 RTP / RTCP

RTP (Real-time Transport Protocol, RFC 3550) over UDP carries the media:

RTP header (12 B): V|P|X|CC|M|PT | sequence number(16) | timestamp(32) | SSRC(32)

• Sequence number: detect loss and reordering (RTP itself doesn't fix them; the app does).
• Timestamp: sampling instant (e.g. +160 per 20 ms G.711 frame) → drives playout timing and lets the receiver compute jitter; independent of arrival time.
• SSRC: identifies the stream source (each participant/camera).
• Payload Type (PT): codec (Opus, VP8/VP9/AV1, H.264…).
• Marker (M): e.g. first packet after silence, or last packet of a video frame.

RTCP (the control sibling, usually next UDP port up, or multiplexed via RTCP-MUX on the same port) carries Sender/Receiver Reports: packet counts, cumulative loss, interarrival jitter, and round-trip time (via LSR/DLSR timestamps). These reports are the feedback signal for congestion control and for the sender to switch bitrate/resolution.

Modern stacks also use RTP Extensions: transport-wide congestion control feedback (transport-cc), abs-send-time, NACK (selective retransmit for video), RED/FEC (forward error correction), PLI/FIR (please send a keyframe).

47.3 The jitter buffer

The network delivers packets with variable delay (jitter). If you played each packet the instant it arrived, audio would stutter. The jitter buffer (a.k.a. de-jitter buffer) holds arriving packets briefly and releases them on a smooth clock derived from RTP timestamps.

• Too small → late packets miss playout → dropouts/glitches.
• Too large → added mouth-to-ear latency (conversation feels laggy; >150 ms one-way is noticeable, >400 ms is bad).
• Adaptive jitter buffers grow during turbulence and shrink during calm, trading latency for concealment dynamically. Typical target: cover ~p95–p99 of observed jitter, often 20–120 ms.

Mouth-to-ear budget ≈ capture + encode + packetize + network + jitter buffer + decode + playout. The jitter buffer and network are where the fight is.

47.4 Loss concealment & recovery

47.5 Congestion control for media

Media CC must keep latency low (can't fill buffers like CUBIC) and adapt the encoder bitrate, not just the send window:

• GCC (Google Congestion Control) — WebRTC's default: a delay-based controller (watches one-way delay gradient from transport-cc feedback / abs- send-time) combined with a loss-based controller; output is a target bitrate the encoder must hit. Reacts before queues build.
• NADA, SCReAM — IETF alternatives, similar goals.
• On congestion the app drops resolution/framerate/bitrate (and may pause video to protect audio) — graceful degradation instead of freeze.

47.6 Transport choices in the wild

Rule: conversational (two-way, <400 ms) → UDP + RTP + jitter buffer + concealment. One-way broadcast/VOD where seconds of latency are fine → TCP + HTTP + CDN.

47.7 War story

A telehealth product had "robotic audio" for clinics on a particular ISP. RTCP Receiver Reports showed 2–4% loss and interarrival jitter ~90 ms on that path; the app's jitter buffer was fixed at 60 ms, so ~p80 of packets were "late" and discarded. Two changes: (1) enable Opus in-band FEC + raise packet loss perceptual robustness, (2) switch to an adaptive jitter buffer (60–200 ms). Audio MOS went from ~2.8 to ~4.1 with only ~25 ms added latency. Lesson: for real-time media, watch RTCP loss & jitter, and match the jitter buffer to the measured jitter distribution — plus FEC for the residual loss.

Try it yourself

# Inspect RTP/RTCP in a capture:
#   Wireshark: Telephony > RTP > RTP Streams  (shows lost %, max jitter, delta)
#              select a stream > Analyze  > "Play Streams" for audio
tshark -r call.pcap -q -z rtp,streams
tshark -r call.pcap -Y rtcp -T fields -e rtcp.ssrc.fraction -e rtcp.ssrc.jitter
# Chrome: chrome://webrtc-internals  -> per-stream packetsLost, jitter, jitterBufferDelay,
#         framesDropped, availableOutgoingBitrate, googCurrentDelayMs