Part 10 — HTTP in depth

Part 7 taught you what an HTTP request looks like. This part is what you actually need to operate the web: caching, cookies, CORS, and the streaming protocols. If you build web applications, this is the most immediately useful part of the guide.

Chapter 39 — How HTTP messages are framed (and how it goes wrong)

In one sentence

The receiver needs to know where one request ends and the next begins, and HTTP gives two ways to say it — which is exactly why they can be made to disagree.

The problem

You're reading bytes off a TCP connection (Part 5). TCP gives you a stream with no message boundaries. So HTTP must mark them itself.

The two ways

   METHOD 1: COUNT THE BYTES
   ------------------------------------
   POST /upload HTTP/1.1
   Host: example.com
   Content-Length: 27
                                <-- blank line
   {"name":"Ada","age":36}
   \_____________________/
     exactly 27 bytes; then the message is over


   METHOD 2: CHUNKS (when you don't know the length in advance)
   ------------------------------------
   HTTP/1.1 200 OK
   Transfer-Encoding: chunked

   1a                           <-- 0x1a = 26 bytes of data next
   {"status":"still working"}
   9                            <-- 9 bytes next
   more data
   0                            <-- a zero-size chunk means THE END
                                <-- blank line

Chunked is what lets a server start sending before it knows the total size — streaming a log, generating a report, or an LLM producing tokens one at a time.

The rule when both appear

If a message has both Content-Length and Transfer-Encoding: chunked, the standard says: Transfer-Encoding wins, and Content-Length must be ignored — or better, the message rejected.

Why does the standard have to say that? Because if two servers in a chain disagree, you get a genuine security hole.

Request smuggling, in plain language

Imagine two people reading the same letter through a slot, and disagreeing about where the letter ends.

   A single stream of bytes arrives containing what LOOKS like one request.

   The front proxy reads it using Transfer-Encoding
        -> it thinks the request ends HERE  ------+
                                                  |
   The back-end server reads it using Content-Length
        -> it thinks the request ends HERE  --+   |
                                              |   |
   The bytes in between belong to nobody -- so the back end treats them as
   the BEGINNING OF THE NEXT PERSON'S REQUEST.

The attacker's leftover bytes get glued onto the front of the next customer's request. That can steal their session, poison a cache, or bypass access controls.

You don't need to memorise the variants (CL.TE, TE.CL, TE.TE). You need to know the defence:

Reject any request containing both headers, at the edge.
Reject weird Transfer-Encoding values — including ones with odd spacing like Transfer-Encoding : chunked, which one parser may accept and another ignore.
Prefer HTTP/2 end-to-end, where framing is explicit and length-prefixed.
Keep your proxy and your server on current versions; this class of bug is found and fixed regularly.

Other framing rules worth knowing

Case	Rule
`HEAD` response	Has a `Content-Length` but no body at all. Don't wait for one.
`204 No Content` and `304 Not Modified`	Never have a body, whatever the headers say
`100 Continue`	An interim reply meaning "go ahead, send the body" — so a client can ask permission before uploading 2 GB. You'll see two status lines for one request.
No `Content-Length`, no chunked (HTTP/1.0)	The body ends when the connection closes

Practice (20 min)

A. Watch chunked encoding happen:

curl -sv --raw http://httpbin.org/stream/3 2>&1 | tail -30

The --raw flag stops curl from decoding chunks, so you see the hex sizes and the terminating 0.

B. Compare with a fixed-length response:

curl -sI https://example.com | grep -iE 'content-length|transfer-encoding'

C. Watch a streamed response arrive over time (chunked in action):

curl -sN https://httpbin.org/drip?duration=5\&numbytes=10

Bytes trickle in over 5 seconds rather than arriving at once.

D. See 100 Continue:

head -c 1000000 /dev/urandom > /tmp/big
curl -sv -H 'Expect: 100-continue' --data-binary @/tmp/big \
     https://httpbin.org/post 2>&1 | grep '< HTTP'

You should see HTTP/1.1 100 Continue and then HTTP/1.1 200 OK.

Common confusions

"Is chunked slower?" No — it lets the server start sending immediately, which usually makes the user experience faster.
"Why would a proxy and server disagree?" Different codebases, different authors, different decades, different tolerance for malformed input. That's why strictness at the edge matters.
"Does this affect HTTP/2 and HTTP/3?" Their native framing is unambiguous. But a proxy that translates HTTP/2 to HTTP/1.1 downstream can reintroduce the problem if it trusts a client-supplied length.

Check yourself

What are the two ways HTTP marks the end of a message body?
When would you need chunked encoding?
In plain words, how does request smuggling work?
What should an edge proxy do with a request containing both headers?