10. Linear Algebra I — Vectors & Matrices

Level: Intermediate You'll use this for: ML embeddings, graphics transforms, recommendation systems, and any "list of numbers as one object" problem.

Vectors

A vector is an ordered list of numbers — a point in space, a feature list, or a text embedding. Vectors have magnitude (length) and direction.

|v| = sqrt(v1² + v2² + ... + vn²)     <- this is just Pythagoras (Ch. 7) generalized

Dot product

A · B = a1*b1 + a2*b2 + ... + an*bn

The dot product measures how much two vectors point in the same direction — it's the numerator of cosine similarity (Ch. 7), and it's the single operation a neuron performs on its inputs before an activation function (Ch. 21).

import numpy as np
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
np.dot(a, b)   # 1*4 + 2*5 + 3*6 = 32

Matrices and matrix multiplication

A matrix is a grid of numbers — a batch of vectors, an image, or a linear transformation. Multiplying a matrix by a vector applies that transformation.

[a b] [x]   [ax + by]
[c d] [y] = [cx + dy]

Rule that trips everyone up: matrix multiplication is not commutative (AB ≠ BA in general), and the inner dimensions must match: (m×n) · (n×p) = (m×p).

A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
A @ B          # matrix multiply (NOT A * B, which is element-wise!)

Computing A @ B by hand, entry by entry: each output entry is the dot product of a row from A and a column from B.

A = [1 2]     B = [5 6]
    [3 4]         [7 8]

result[0][0] = (row 0 of A) · (col 0 of B) = 1×5 + 2×7 = 5+14 = 19
result[0][1] = (row 0 of A) · (col 1 of B) = 1×6 + 2×8 = 6+16 = 22
result[1][0] = (row 1 of A) · (col 0 of B) = 3×5 + 4×7 = 15+28 = 43
result[1][1] = (row 1 of A) · (col 1 of B) = 3×6 + 4×8 = 18+32 = 50

A @ B = [19 22]
        [43 50]

Swap the order (B @ A) and you'd pair B's rows with A's columns instead — different numbers entirely, which is the mechanical reason matrix multiplication isn't commutative.

This exact bug (* vs. @/.dot()) is one of the most common silent correctness bugs in numpy/PyTorch code — * does element-wise multiplication, @ does true matrix multiplication, and both run without error on compatible shapes, silently producing wrong numbers.

[x']   [cos θ  -sin θ] [x]
[y'] = [sin θ   cos θ] [y]

theta = math.radians(90)
R = np.array([[math.cos(theta), -math.sin(theta)],
              [math.sin(theta),  math.cos(theta)]])
R @ np.array([1, 0])   # -> [0, 1], a 90° rotation

Norms: three ways to measure "how big"

‖x‖₁ = |x₁| + |x₂| + ... + |xₙ|               L1 / Manhattan norm
‖x‖₂ = sqrt(x₁² + x₂² + ... + xₙ²)            L2 / Euclidean norm (the |v| above)
‖x‖∞ = max(|x₁|, |x₂|, ..., |xₙ|)             L∞ / max norm

Example: x = (3, 4) has ‖x‖₁ = 7, ‖x‖₂ = 5, ‖x‖∞ = 4. In a city grid you'd walk 7 blocks (L1), a crow would fly 5 (L2), and the longest single leg of the walk is 4 (L∞).

• AI: L2 regularization ("weight decay") adds λ‖w‖₂² to the loss and shrinks all weights a little. L1 regularization adds λ‖w‖₁ and drives many weights to exactly zero, so it selects features. Adversarial-robustness papers measure attack budgets in L∞: "no pixel changed by more than 8/255."
• Network: gradient clipping in distributed training, and "max link utilization" in traffic engineering, are both L∞ thinking: you care about the worst coordinate, not the total.

The dot product, geometrically

The algebraic formula (multiply pairwise, then add) hides a geometric fact:

A · B = ‖A‖ · ‖B‖ · cos θ

Example: A = (3, 4), B = (4, 3).

A · B = 3·4 + 4·3 = 24
‖A‖ = ‖B‖ = 5
cos θ = 24 / (5·5) = 0.96   ->   θ ≈ 16.3°

Rearranged, this is cosine similarity (Ch. 7). Why does the algebraic sum equal a cosine? Expand ‖A - B‖² both as a sum of squares and by the law of cosines from trigonometry, and the cross term -2·A·B has to equal -2‖A‖‖B‖cos θ.

Projection, "how much of A lies along B":

proj_B(A) = (A · B / B · B) · B  =  (24/25) · (4, 3)  =  (3.84, 2.88)

Projection is how PCA (Ch. 20) squashes data onto its principal axes, and how the Gram-Schmidt process builds orthogonal bases.

Matrix × vector = a weighted mix of columns

There are two equally correct ways to read W·x, and the second one is the more useful:

W·x = x₁·(column 1 of W) + x₂·(column 2 of W) + ... + xₙ·(column n of W)

The vector x tells you how much of each column to mix in. The set of every possible mix is called the column space (or span). It's every output the matrix can ever produce.

AI in the wild: one neural network layer, by hand. A dense layer computes h = ReLU(W·x + b).

x = [1]     W = [0.5  -1  ]     b = [ 0.1]
    [2]         [2     0.25]        [-0.5]

W·x = [0.5·1 + (-1)·2 ] = [-1.5]       (each row dotted with x)
      [2·1   + 0.25·2 ]   [ 2.5]

W·x + b = [-1.4]     ReLU  ->   h = [0.0]
          [ 2.0]                    [2.0]

Neuron 1's template points away from this input, so it stays silent. Neuron 2 matches it and fires. GPT-3 does the same thing with feed-forward matrices of 49,152 × 12,288 and repeats it across 96 layers for every token it generates. Inference is mostly W·x, which is why GPUs (matrix-multiply machines) took over AI.

Networking in the wild: the routing matrix

Put a network's routing into a matrix and many traffic-engineering questions become y = R·d:

Three flows over three links. Flow A uses links 1 and 2, flow B uses links 2 and 3, and flow C uses only link 3:

        A  B  C
R = [   1  0  0 ]  link 1          d = [10]  Mbps (A)
    [   1  1  0 ]  link 2              [20]       (B)
    [   0  1  1 ]  link 3              [ 5]       (C)

y = R·d = [10, 10+20, 20+5] = [10, 30, 25] Mbps

The reverse question is harder and more useful. SNMP gives you link loads y for free, but you want the per-flow demand d. This is network tomography, solving R·d = y for d. Here R is square and invertible, so d = R⁻¹y recovers [10, 20, 5] exactly. Real networks have far more flows than links, so R is wide and has no inverse. Operators then use least squares (below) plus priors, which is what ISP traffic-matrix estimation tools do.

Counting paths with matrix powers

For a network's adjacency matrix A (Ch. 17), there's a remarkable theorem:

(Aᵏ)[i][j] = number of walks of exactly k hops from node i to node j

Four routers: R1–R2, R1–R3, R2–R3, R3–R4.

     R1 R2 R3 R4                  A² = [2 1 1 1]
A = [0  1  1  0]                       [1 2 1 1]
    [1  0  1  0]                       [1 1 3 0]
    [1  1  0  1]                       [1 1 0 1]
    [0  0  1  0]

A²[R1][R4] = 1: there's exactly one 2-hop walk (R1→R3→R4). The diagonal A²[i][i] is node i's degree, since every neighbour gives a walk out and straight back. A³[R1][R1] = 2 counts the triangle R1→R2→R3→R1 in both directions. Fraud-ring detection on payment graphs counts triangles with exactly this trace-of-A³ trick.

Solving A·x = b, and the inverse

A system of linear equations is one matrix equation. For a 2 × 2 matrix the inverse has a closed form worth knowing by heart:

A = [a b]       A⁻¹ = 1/(ad - bc) · [ d  -b]        x = A⁻¹·b
    [c d]                           [-c   a]

In code you almost never form A⁻¹ explicitly. np.linalg.solve(A, b) is faster and more accurate numerically (Ch. 1's floating-point issues compound inside an explicit inverse).

Least squares: fitting a line through noisy data

When there are more equations than unknowns, as with noisy measurements, no x satisfies them all. Least squares picks the w that minimizes total squared error, and calculus (Ch. 13: set the gradient to zero) gives a closed form called the normal equations:

minimize  ‖X·w - y‖₂²     ==>     w = (Xᵀ X)⁻¹ Xᵀ y

Worked example: latency vs. load. Four measurements of load (thousand req/s) and p50 latency (ms). We fit latency = w₀ + w₁·load:

load: 1, 2, 3, 4        latency: 12, 15, 19, 22

X = [1 1]     y = [12]       XᵀX = [ 4  10]      Xᵀy = [ 68]
    [1 2]         [15]             [10  30]            [187]
    [1 3]         [19]
    [1 4]         [22]

(XᵀX)⁻¹ = 1/(4·30 - 10·10) · [ 30 -10]  = 1/20 · [ 30 -10]
                             [-10   4]           [-10   4]

w₀ = (30·68 - 10·187) / 20 = 170 / 20 = 8.5 ms     (latency at zero load)
w₁ = (-10·68 + 4·187) / 20 =  68 / 20 = 3.4 ms     (per extra 1k req/s)

Forecast: at 6k req/s, latency ≈ 8.5 + 3.4·6 = 28.9 ms. Be careful with that number, because queueing (Ch. 25) bends this line upward near saturation. A linear fit is only valid in the range you measured.

X = np.array([[1, 1], [1, 2], [1, 3], [1, 4]]); y = np.array([12, 15, 19, 22])
w, *_ = np.linalg.lstsq(X, y, rcond=None)
print(w)   # [8.5 3.4]

Linear regression, the first model in every ML course, is this equation. A neural network's last layer, trained with squared error, is solving it too.

Real-life engineering ties

• SWE: any spreadsheet-like data (rows × columns) is a matrix; game/UI transforms (rotate, scale, translate) are matrix multiplications.
• AI: embeddings are vectors; a transformer's attention mechanism is, at its core, a sequence of matrix multiplications (Ch. 20–21); a neural network layer is output = activation(W @ input + b).
• Network: representing a network topology as an adjacency matrix (Ch. 17) lets you use matrix operations to answer reachability questions.
• Security: feature vectors for ML-based malware/intrusion detection; representing permission matrices (subjects × objects) in access-control models.

More real-life examples:

• Every 3D game engine (Unity, Unreal) represents object position, rotation, and scale as 4x4 transform matrices, and renders a frame by multiplying a vertex through a chain of these matrices (model → view → projection) — a direct, large-scale use of Ch. 10's rotation-matrix idea.
• Search engines historically represented documents as TF-IDF vectors (one dimension per vocabulary word) and ranked relevance using vector operations like the dot product and cosine similarity from Ch. 7.
• Recommendation engines (Netflix, Amazon) represent the entire user-item interaction history as one giant sparse matrix, and factorize it (Ch. 20) to predict missing entries — "would this user like this item."
• Spreadsheet formulas that reference ranges (SUMPRODUCT, array formulas in Excel/Sheets) are literally performing vector/matrix operations under a friendlier UI.

Common mistake

Confusing * (element-wise) with @/np.dot/np.matmul (true matrix multiply) in numpy/PyTorch — both are legal, only one is usually correct, and there's no error to warn you.