Chapter 73 — Quantization: from float32 to int4, the actual algorithm

In one sentence

Quantization maps a range of real-valued weights onto a small set of integers using a scale factor computed from that range, so storing and computing with the integers approximates the original floats at a fraction of the memory — Chapter 66 gave you the resulting byte counts; this chapter gives you the arithmetic that produces them.

The problem

Chapter 66's precision table said int8 uses 1 byte/weight and int4 uses half a byte, but treated "storing weights in fewer bits" as a black box. Given a real weight like 0.4514, how does it actually become an 8-bit integer, and how do you get a usable approximation back?

The idea, in plain language

Find the largest-magnitude weight in a group of weights, and use it to define a scale: the ratio that stretches the integer type's full range (e.g. -127 to 127 for int8) to cover that group's actual range. Divide every weight by the scale and round to the nearest integer to quantize; multiply back by the scale to dequantize. The error you introduce is bounded by how coarse that scale is — which depends entirely on how you chose the group of weights that share one scale.

How it actually works: symmetric quantization

Weight distributions are roughly zero-centered, so symmetric quantization (one scale, no offset) is standard for weights:

scale = max(|w|) / 127                  (int8 range: -127..127)
q     = round(w / scale), clipped to [-127, 127]
dequantize:  w_approx = q * scale

Worked example, real numbers, for w = [0.02, -0.87, 0.45, -0.13, 0.91]:

scale = max(|w|) / 127 = 0.91 / 127 = 0.0071654

w        quantized int   dequantized   abs error
0.02  ->      3       ->    0.0215     ->  0.0015
-0.87 ->    -121       ->   -0.8670    ->  0.0030
0.45  ->     63        ->    0.4514    ->  0.0014
-0.13 ->    -18        ->   -0.1290    ->  0.0010
0.91  ->     127       ->    0.9100    ->  0.0000

The weight that defined the scale (0.91) round-trips with zero error; everything smaller accumulates a little rounding error proportional to how far below the scale's max it sits. This is inherent to the method, not a bug — it's the direct price of representing a continuous range with 255 discrete buckets.

Asymmetric quantization (common for activations, e.g. post-ReLU values that are never negative) adds a zero_point so the integer range doesn't waste half its buckets on unused negative values:

scale      = (max(w) - min(w)) / 255
zero_point = round(-min(w) / scale)
q          = round(w / scale) + zero_point, clipped to [0, 255]
dequantize:  w_approx = (q - zero_point) * scale

Granularity is the real lever: what one outlier does

A single unusually large weight in a tensor forces the scale to stretch to cover it — crushing the precision of every other weight in that same scale group. Worked example: a small group of weights, one outlier (25.0) among values near 0.01–0.03:

group = [0.01, -0.02, 0.015, -0.01, 0.03, 25.0, 0.02, -0.015]

Per-tensor scale (one scale for the whole group, outlier included):
  scale = 25.0 / 127 = 0.1969
  every small value (0.01, -0.02, 0.015, -0.01, 0.03) quantizes to 0
  -- 100% of their information is gone, because 0.1969 is bigger than they are

Per-group scale (outlier excluded -- e.g. a separate scale per 128-weight block):
  scale = 0.03 / 127 = 0.000236
  abs errors: [0.00008, 0.00008, 0.00012, 0.00008, 0.0, ...]
  -- three orders of magnitude more precise for the values that matter

This is exactly why real quantization schemes (GPTQ, AWQ, GGUF) never use one scale for an entire multi-million-weight tensor — they compute a separate scale per row, per channel, or per small group (often 32–128 weights), specifically to contain outlier damage to the group the outlier actually lives in.

Post-training quantization vs. quantization-aware training

PTQ (post-training quantization) — quantize an already-trained model directly. Fast (no retraining), the standard choice for shrinking an existing checkpoint (what Ollama does when you pull a q4_0 model).
QAT (quantization-aware training) — simulate the rounding during training or fine-tuning, so the model's weights adapt around the precision loss. Better accuracy at the same bit width, but costs an actual training run — worth it mainly when you're already fine-tuning (Chapter 42) and can fold QAT into that same run.

Two named algorithms worth knowing

GPTQ — quantizes a layer's weights column by column; after quantizing each column, it adjusts the remaining unquantized columns to compensate for the error just introduced, using a small amount of representative input data (calibration data) run through the layer to measure which directions in weight-space matter most for that layer's actual outputs. This is why GPTQ needs calibration data and plain round-to-nearest quantization doesn't — it's actively minimising output error, not just minimising weight-value error.
AWQ (Activation-aware Weight Quantization) — a different insight: not all weights matter equally, and the ones that matter most are the ones multiplied by large-magnitude activations, not necessarily the ones with the largest weight values themselves. AWQ inspects activation statistics to find the ~1% of "salient" weight channels and protects them (keeps them at higher precision or rescales them) rather than quantizing every weight uniformly.

The nuance Chapter 66's table glossed over

"4-bit" doesn't mean every number in the file is 4 bits. Each scale (and zero-point, for asymmetric schemes) is itself typically stored at higher precision (often fp16), once per group — so a "4-bit" GGUF model, accounting for that per-group overhead, averages closer to 4.5–5 bits per weight in practice, not exactly 4. The smaller the group size, the better the accuracy and the more this overhead matters — a real, load-bearing tradeoff, not a rounding footnote.

Practice (20 min) — quantize a matrix yourself, measure the difference

import random

random.seed(0)
weights = [random.gauss(0, 0.02) for _ in range(1000)]
weights[500] = 3.5   # inject one outlier, like a real trained layer often has

def quantize_symmetric(vals, bits=8):
    qmax = 2 ** (bits - 1) - 1
    scale = max(abs(v) for v in vals) / qmax
    q = [max(-qmax, min(qmax, round(v / scale))) for v in vals]
    deq = [qi * scale for qi in q]
    return deq

def mse(a, b):
    return sum((x - y) ** 2 for x, y in zip(a, b)) / len(a)

# One scale for everything (outlier included)
deq_all = quantize_symmetric(weights)
print("per-tensor MSE (outlier included):", mse(weights, deq_all))

# Per-group scale, group size 100 -- the outlier only damages its own group
group_size = 100
deq_grouped = []
for i in range(0, len(weights), group_size):
    deq_grouped.extend(quantize_symmetric(weights[i:i+group_size]))
print("per-group (size 100) MSE:", mse(weights, deq_grouped))

Run it: the per-tensor MSE will be dominated by the 999 non-outlier weights all losing precision to accommodate the one outlier; the per-group MSE will be dramatically lower for those same 999 weights, because only the one group containing the outlier pays the cost.

Common confusions

Quantization is not distillation (Chapter 67) — quantization keeps the exact same architecture and the exact same learned function, just stored and computed at lower precision; distillation trains a genuinely different (smaller) set of weights. They stack: distill first, then quantize the smaller result (Chapter 68's student could be quantized further, though at 76 parameters there's essentially nothing left to save).
Lower bit-width isn't free precision loss everywhere equally. Chapter 66's ~1–3% benchmark-score drop for int4 vs. bf16 is a typical range for weight-only quantization with a decent scheme (GPTQ/AWQ, sensible group size) — naive per-tensor int4 with no calibration can be dramatically worse, as the outlier example above demonstrates.
Quantizing activations is harder than quantizing weights. Weights are static and can be analysed offline (as GPTQ does); activations vary per input, which is exactly the problem AWQ's activation-aware weight selection works around without having to quantize activations directly.

Check yourself

Write the formula for symmetric int8 quantization and explain what each term does.
Why does a single outlier weight damage every other weight's precision under per-tensor quantization, and how does per-group quantization fix that?
What does GPTQ's calibration data actually get used for?
What is AWQ protecting, and why does it look at activations rather than weight magnitudes to decide what to protect?
Why is "4-bit" model size slightly larger in practice than params x 4 bits would suggest?

End of Part 19 — Milestone check

I can derive AdamW's update rule from Chapter 10's plain gradient descent, term by term, and explain what each addition fixes.
I can compute a KV cache size for a given n_kv_heads and explain why GQA's quality holds up better than MQA's.
I can quantize a small weight vector to int8 by hand and explain why per-group scaling beats per-tensor scaling on data with outliers.
I know, precisely, what each of AdamW, GQA, and quantization does and does not have in common with distillation (Chapter 67).