Part 2 — Encoding: the thing that is not security

This Part is short and it prevents a whole category of career-damaging mistakes.

Chapter 5 — How text becomes bytes

In one sentence

Computers store numbers, so every character must be mapped to a number, and the mapping is called a character encoding.

The problem

A file, a network packet, and memory all hold bytes — numbers from 0 to 255. The letter A is not a number. Something must decide that A = 65.

ASCII: the original 128

   char  decimal  hex   binary
   ----  -------  ---   --------
   'A'      65    0x41  01000001
   'B'      66    0x42  01000010
   'a'      97    0x61  01100001
   '0'      48    0x30  00110000
   ' '      32    0x20  00100000
   '\n'     10    0x0A  00001010

Notice '0' is 48, not 0. The character zero and the number zero are different things — a distinction behind many parsing bugs.

ASCII uses 7 bits, so 128 characters. Enough for American English. Not enough for the world.

UTF-8: how the world actually encodes text

UTF-8 represents over a million characters using 1 to 4 bytes each, and — the crucial design win — ASCII text is byte-identical in UTF-8.

   'A'    -> 41                     1 byte   (identical to ASCII)
   'e'    -> 65                     1 byte
   'é'    -> C3 A9                  2 bytes
   '€'    -> E2 82 AC               3 bytes
   emoji  -> F0 9F 98 80            4 bytes

Two consequences that cause security bugs:

Character count ≠ byte count. A "10-character" username can be 40 bytes. Length limits that count one and buffers that hold the other is a classic overflow.
Invalid byte sequences exist. Not every byte string is valid UTF-8. Different parsers handle invalid sequences differently — one may reject, one may substitute, one may silently drop bytes. Two parsers disagreeing about the same input is the root of an entire attack class (Chapter 7).

Practice (10 min)

# see the bytes behind text
printf 'Hello' | xxd
printf 'Héllo' | xxd          # note the two bytes for é
printf '€' | xxd

# character count vs byte count
python3 -c "s='Héllo€'; print('chars:', len(s), ' bytes:', len(s.encode()))"

# what happens with invalid UTF-8?
printf '\xc3\x28' | python3 -c "import sys; print(sys.stdin.buffer.read().decode('utf-8', errors='replace'))"

Check yourself

Why is the character '0' stored as 48?
Give one security-relevant consequence of characters taking a variable number of bytes.
Why does it matter that not every byte sequence is valid UTF-8?