Chapter 7 — Unicode, normalisation, and the bugs they cause

   'é' can be encoded TWO ways in Unicode:

     U+00E9              LATIN SMALL LETTER E WITH ACUTE   -> C3 A9
     U+0065 U+0301       'e' + COMBINING ACUTE ACCENT      -> 65 CC 81

   They render identically. They are different bytes.
   A byte comparison says they are different. A human says they are the same.

Why an attacker cares

Attack 1 — Account confusion. If registration normalises but login doesn't (or vice versa), you can register a "different" account that logs in as an existing one, or vice versa.

Attack 2 — Filter bypass through NFKC. This is the sharp one:

   A filter blocks the string "<script>".

   An attacker sends:   ＜script＞      (full-width characters, U+FF1C/U+FF1E)

   The filter compares bytes -> no match -> ALLOWED.
   Later, some component normalises with NFKC:
        ＜  ->  <
        ＞  ->  >
   The output is now  <script>  in the page.

The rule that prevents this: normalise before you validate, never after.

Attack 3 — Homoglyphs. Different characters that look the same:

   'a'  U+0061  Latin small letter A
   'а'  U+0430  CYRILLIC small letter A     <-- visually identical

   раypal.com   is not   paypal.com

This is the basis of IDN homograph phishing. Browsers defend by showing punycode (xn--...) when a domain mixes scripts suspiciously.

Attack 4 — Invisible characters. Zero-width spaces (U+200B), bidirectional overrides (U+202E), and Unicode "tag" characters (U+E0000 block) are invisible in most renderings but present in the bytes. Uses:

• Bypassing keyword filters
• "Trojan Source" — bidi overrides make source code display differently from how it compiles, hiding a backdoor in plain sight during code review
• Hiding prompt-injection instructions in text an AI system reads

The defensive rules

   1. NORMALISE ON INPUT, at the trust boundary. NFC for general text.
   2. Normalise BEFORE validating, not after.
   3. Strip invisible/control characters explicitly. Don't hope.
   4. For usernames and identifiers: restrict to an ALLOWLIST of characters
      (e.g. a-z 0-9 . _ -) rather than trying to block bad ones.
   5. Compare identifiers case-folded AND normalised, consistently, everywhere.
   6. Set explicit encodings: Content-Type with charset, database column
      collation, and your language's string handling.

Rule 4 is worth emphasising: allowlists beat blocklists, always. You can enumerate what's permitted; you cannot enumerate what's malicious.

cd ~/sec-lab/scratch

python3 - <<'PY'
import unicodedata as ud

# 1. Two 'identical' strings
a = "café"                      # composed
b = "café"                # decomposed
print("look the same? ", a, b)
print("equal bytes?   ", a == b)
print("after NFC?     ", ud.normalize("NFC", a) == ud.normalize("NFC", b))
print("byte lengths:  ", len(a.encode()), len(b.encode()))

# 2. NFKC bypass -- the dangerous one
attack = "＜script＞"    # full-width < and >
print("\nfilter sees:   ", repr(attack), " blocked? ", "<script>" in attack)
print("after NFKC:    ", repr(ud.normalize("NFKC", attack)),
      " now dangerous? ", "<script>" in ud.normalize("NFKC", attack))

# 3. Homoglyphs
print("\npaypal (latin):  ", "paypal.com".encode())
print("paypal (cyrillic a):", "pаypal.com".encode())
print("equal? ", "paypal.com" == "pаypal.com")

# 4. Invisible characters
sneaky = "admin​"           # trailing zero-width space
print("\nlooks like 'admin'? ", sneaky)
print("is 'admin'?         ", sneaky == "admin")
print("stripped:           ", "".join(c for c in sneaky if ud.category(c)[0] != "C"))
PY