Chapter 44 — Compression, range requests, and proxies

Three smaller topics that round out practical HTTP.

Compression

   Client says what it can handle:   Accept-Encoding: br, gzip
   Server says what it used:         Content-Encoding: br
   Server MUST also send:            Vary: Accept-Encoding

Rules that matter:

• Don't compress already-compressed files (JPEG, PNG, MP4, ZIP). You burn CPU and may make them bigger.
• Set a minimum size (~1 KB). Compressing a 50-byte response makes it larger.
• The Vary header is mandatory, or a cache will serve compressed bytes to a client that asked for plain ones.

Range requests

Lets a client ask for part of a resource.

   Server advertises:  Accept-Ranges: bytes
   Client asks:        Range: bytes=1000-1999
   Server replies:     206 Partial Content
                       Content-Range: bytes 1000-1999/50000

This is how: video seeking works, downloads resume after a drop, and download managers fetch several pieces in parallel.

The one detail that matters for correctness: use If-Range.

   Range: bytes=5000-
   If-Range: "v3-abc123"        <-- "only if it's still the version I started"

   -> unchanged?  206 with the requested range
   -> changed?    200 with the WHOLE new file (safe restart)

Without If-Range, a resumed download can splice together bytes from two different versions of a file and produce silent corruption.

Proxies — the forward kind

Chapter 36 covered reverse proxies (in front of servers). A forward proxy sits in front of clients — typically a corporate egress gateway.

   FORWARD:  your machine -> [corporate proxy] -> the internet
   REVERSE:  the internet -> [load balancer]   -> the company's servers

For HTTPS, a forward proxy can't read the traffic, so it builds a blind tunnel:

   CLIENT -> PROXY:   CONNECT example.com:443 HTTP/1.1
   PROXY  -> CLIENT:  HTTP/1.1 200 Connection Established
   ... then it just relays bytes; TLS is end-to-end ...

The proxy sees the hostname and the volume, but not the content.

TLS interception is when a company does want to see the content: the proxy terminates TLS with its own certificate and re-encrypts to the real server. This requires installing a corporate root CA on every device, and it causes a very specific, very common problem:

"SSL certificate problem: unable to get local issuer certificate" — but only at work, and only in some tools.

The reason: the corporate CA got installed in the OS trust store (so browsers and curl work), but Python, Node, Java, Go, and Docker each use their own trust store. Each needs the CA added separately.

Also learn the environment variables, because everything respects them:

export http_proxy=http://proxy.corp:3128
export https_proxy=http://proxy.corp:3128
export no_proxy=localhost,127.0.0.1,.internal.example.com

# compression -- see it applied, and check Vary
curl -sI -H 'Accept-Encoding: gzip, br' https://example.com \
  | grep -iE 'content-encoding|vary|content-length'

# compare sizes
curl -s https://example.com | wc -c
curl -s -H 'Accept-Encoding: gzip' https://example.com --output - | wc -c

# range requests
curl -sI https://example.com | grep -i accept-ranges
curl -s -r 0-99 https://example.com | wc -c            # expect ~100 bytes
curl -sI -H 'Range: bytes=0-99' https://example.com | grep -iE '^HTTP|content-range'

# what does your environment think about proxies?
env | grep -i proxy

# which CA bundle is each tool using?
curl -v https://example.com 2>&1 | grep -i 'CAfile'
python3 -c "import certifi; print(certifi.where())"

# are you behind TLS interception? check who signed the certificate
echo | openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -issuer