Chapter 26 — Hands-on: post-quantum with OpenSSL and liboqs

Why this chapter exists

You should touch the real thing once: generate an ML-KEM keypair, do an encapsulation, sign with ML-DSA, and negotiate a hybrid TLS handshake. Then it stops being abstract.

Getting a PQC-capable toolchain

   OPTION A -- OpenSSL 3.5+ (LTS, released April 2025)
     Native ML-KEM, ML-DSA, and SLH-DSA. No extra provider needed.
       brew install openssl@3        # ensure >= 3.5
       openssl list -kem-algorithms       | grep -i mlkem
       openssl list -signature-algorithms | grep -i -E 'ml-dsa|slh-dsa'

   OPTION B -- Open Quantum Safe provider (works with OpenSSL 3.0-3.4,
               and adds HQC, FN-DSA, and hybrids not yet in core)
       brew install liboqs oqs-provider     # or build from source
       openssl list -providers -provider oqsprovider
       # add to openssl.cnf, or pass:  -provider oqsprovider -provider default

   OPTION C -- language libraries
       Python:  pip install quantcrypt      (wraps liboqs)
       Go:      filippo.io/mlkem768, or Go 1.24+ crypto/mlkem
       Rust:    the `ml-kem` and `ml-dsa` crates (RustCrypto)

This machine: openssl version here reports LibreSSL 3.3.6, which has no PQC support at all. Every command below needs OpenSSL 3.5+ or the OQS provider. Install one before running them — this is the §0.4 trap in its final form.

1. ML-KEM: generate, encapsulate, decapsulate

mkdir -p ~/sec-lab/pqc && cd ~/sec-lab/pqc

# Recipient generates a keypair
openssl genpkey -algorithm ML-KEM-768 -out mlkem_sk.pem
openssl pkey -in mlkem_sk.pem -pubout -out mlkem_pk.pem
wc -c mlkem_sk.pem mlkem_pk.pem          # note the ~1.2 KB public key

# Sender encapsulates: produces a ciphertext + a shared secret
openssl pkeyutl -encap -inkey mlkem_pk.pem -pubin \
        -out ct.bin -secret sender_secret.bin

# Recipient decapsulates: recovers the SAME shared secret
openssl pkeyutl -decap -inkey mlkem_sk.pem \
        -in ct.bin -secret recipient_secret.bin

xxd sender_secret.bin
xxd recipient_secret.bin
cmp sender_secret.bin recipient_secret.bin && echo "SHARED SECRETS MATCH"
ls -l ct.bin                              # ~1088 bytes for ML-KEM-768

Feed that 32-byte secret through HKDF (Chapter 22, step 4) to get AEAD keys — exactly as you would with an X25519 secret.

2. ML-DSA: sign and verify

cd ~/sec-lab/pqc
openssl genpkey -algorithm ML-DSA-65 -out mldsa_sk.pem
openssl pkey -in mldsa_sk.pem -pubout -out mldsa_pk.pem

echo "release 6.0.0 -- first post-quantum build" > rel.txt
openssl pkeyutl -sign   -inkey mldsa_sk.pem -rawin -in rel.txt -out rel.sig
openssl pkeyutl -verify -pubin -inkey mldsa_pk.pem -rawin -in rel.txt \
        -sigfile rel.sig

ls -l rel.sig            # ~3.3 KB -- compare to 64 bytes for Ed25519
# tamper -> verification fails
echo "release 6.0.0 -- backdoored" > rel.txt
openssl pkeyutl -verify -pubin -inkey mldsa_pk.pem -rawin -in rel.txt \
        -sigfile rel.sig

3. SLH-DSA: the conservative signer (feel the cost)

cd ~/sec-lab/pqc
time openssl genpkey -algorithm SLH-DSA-SHA2-128s -out slh_sk.pem
openssl pkey -in slh_sk.pem -pubout -out slh_pk.pem
wc -c slh_pk.pem                         # tiny -- ~32 bytes of key material

echo "firmware image v3" > fw.txt
time openssl pkeyutl -sign -inkey slh_sk.pem -rawin -in fw.txt -out fw.sig
ls -l fw.sig                             # ~7.8 KB signature
time openssl pkeyutl -verify -pubin -inkey slh_pk.pem -rawin -in fw.txt \
        -sigfile fw.sig

Note the numbers: sub-32-byte key, ~8 KB signature, slow sign, fast-ish verify. That profile is why SLH-DSA belongs on firmware, not TLS.

4. Hybrid post-quantum TLS handshake

# --- server: a throwaway self-signed cert is fine for this test ---
cd ~/sec-lab/pqc
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
        -keyout srv.key -out srv.crt -days 1 -subj "/CN=localhost"

openssl s_server -accept 4433 -cert srv.crt -key srv.key \
        -groups X25519MLKEM768 -tls1_3 -www &
SERVER=$!
sleep 1

# --- client: force the hybrid group and report what was negotiated ---
echo | openssl s_client -connect localhost:4433 \
        -groups X25519MLKEM768 -tls1_3 2>/dev/null \
        | grep -E "Negotiated TLS1.3 group|Cipher|Protocol"

kill $SERVER

# --- against the real internet ---
echo | openssl s_client -connect cloudflare.com:443 \
        -groups X25519MLKEM768 2>/dev/null \
        | grep -E "Negotiated TLS1.3 group|New, TLSv1.3"

You should see Negotiated TLS1.3 group: X25519MLKEM768. You just did a post-quantum key exchange.

5. Benchmark PQC vs classical

openssl speed -provider default ml-kem-768 x25519 2>/dev/null | tail -6
openssl speed ml-dsa-65 ed25519 ecdsap256 2>/dev/null | tail -8

Typical findings: ML-KEM-768 is faster than X25519 for the operation itself; the cost is bytes on the wire, not CPU. ML-DSA-65 signing is fast; its signatures are large. SLH-DSA signing is genuinely slow.

   1. Add an algorithm field to the manifest header:
        {"sig_alg": "ed25519", ...}   or   {"sig_alg": "ml-dsa-65", ...}
        or  {"sig_alg": "hybrid-ed25519-mldsa65", ...}
   2. `keygen --alg <alg>` generates the right keypair(s).
   3. `sign` writes the sig_alg into the manifest and produces the
      matching signature(s). For hybrid: TWO signatures, both over the
      same manifest bytes.
   4. `verify`:
        - reads sig_alg from the manifest
        - for hybrid, requires BOTH signatures to verify
        - refuses unknown algorithms (fail closed)
   5. Test migration: sign a release with ed25519, then re-sign the SAME
      artifacts as hybrid, and confirm an old ed25519-only verifier still
      accepts the ed25519 signature while a new verifier checks both.
   6. Record signature sizes and sign/verify timings for each alg in a
      README table.

   Keep it under ~150 lines. This is a genuine pattern -- Sigstore,
   apt, and TLS libraries all carry an algorithm identifier for exactly
   this reason.