Chapter 66 — What model weights actually are

Where weights live, concretely

Every Linear layer you've seen since Chapter 14 is one weight matrix W and one bias vector b:

y = W @ x + b

• x — the input vector (length = in_features)
• W — a matrix of shape [out_features, in_features]
• b — a vector of length out_features
• y — the output vector (length = out_features)

A Transformer block (Chapter 25) is a handful of these: W_q, W_k, W_v, W_o for attention, plus two or three bigger matrices for the feed-forward network. Stack ~30–100 blocks and add one embedding matrix ([vocab_size, d_model]) at the start, and that's the entire "weights" of an LLM — there is no other kind of stored knowledge. Everything the model "knows" is encoded as the specific numbers in these matrices.

Parameter count, worked out: for one feed-forward sub-layer with d_model = 4096 and hidden size 4 x d_model = 16384:

up-projection:   4096 x 16384 = 67,108,864 weights
down-projection: 16384 x 4096 = 67,108,864 weights

That's ~134M weights in one sub-layer of one block. A 32-block model repeats this (plus attention) 32 times — this is how you get from "a matrix multiply" to "70 billion numbers" without anything mysterious happening in between. It is repetition of the same small idea, at scale (Chapter 16's theme again).

Precision: the same number, different numbers of bits

A weight is a real number, and a computer has to pick how many bits to spend storing each one. This choice is independent of the model's architecture — you can take the exact same trained weights and re-save them at a different precision.

bf16 and fp16 are both "16-bit floats" but split those bits differently: bf16 keeps fp32's wide exponent range (fewer mantissa bits, so less precision per number but the same dynamic range training needs) — this is why bf16 became the default for training large models, while fp16 was more prone to overflow.

The memory formula that matters in practice:

memory_bytes = num_parameters x bytes_per_parameter

For a 7B model at bf16: 7,000,000,000 x 2 = 14,000,000,000 bytes ≈ 14 GB. This is before you add activation memory or the KV cache (Chapter 38) — the weights are usually the smaller part of a serving system's memory budget once requests are flowing, but they're the fixed cost you pay just to load the model at all.

File formats: what you actually download

safetensors mattering is worth pausing on: loading a model is mostly memory-mapping a file, not "running a program." The header tells the runtime "bytes 0–67108863 are a [4096, 16384] float32 tensor named layers.0.mlp.up_proj.weight," and the runtime points a tensor directly at that region of the file. This is also why big models are slow to load from a network but fast to load from local disk — it's I/O-bound, not compute-bound.

import json, struct

# Point this at any .safetensors file you have locally
# (e.g. one downloaded by a HuggingFace `transformers` model).
path = "model.safetensors"

with open(path, "rb") as f:
    header_len = struct.unpack("<Q", f.read(8))[0]   # first 8 bytes: header length
    header = json.loads(f.read(header_len))

for name, info in list(header.items())[:10]:
    if name == "__metadata__":
        continue
    print(f"{name:50s} {info['dtype']:8s} {info['shape']}")