Chapter 2 — What "data", "features", and "learning" mean here

In one sentence

Data is examples; features are the numbers you feed the model; learning is adjusting knobs to reduce mistakes.

Data

Data = a collection of examples. In tables, one example per row.

| age | income | owns_home | ...  |  defaulted_on_loan   |
|-----|--------|-----------|------|----------------------|
| 34  | 52000  | yes       | ...  |  no                  |   <- one example
| 51  | 98000  | yes       | ...  |  no                  |
| 22  | 21000  | no        | ...  |  yes                 |
  \___________________________/       \________________/
        the INPUT (features X)         the ANSWER (label y)

Two big families:

Labelled data — you have the answers (this email is spam). Learning from it is supervised learning.
Unlabelled data — just the raw stuff (a billion web pages, no answers). Learning from it is unsupervised or self-supervised learning.

This distinction is the key to understanding LLMs. Labelled data is expensive (a human must label each item). Unlabelled text is nearly free and nearly infinite. LLMs work because someone found a way to learn from unlabelled text: hide the next word and make the model guess it — the text labels itself. That's self-supervised learning, and it's why LLMs could be trained on the whole internet. More in Chapter 33.

Features

A feature is one measurable property, expressed as a number.

For a house: square_metres, bedrooms, year_built, distance_to_station
For an email: number_of_links, contains_word_free, sender_is_known
For a photo: originally hand-designed edge detectors; now learned automatically (that's what "deep learning" changed)

Feature engineering — hand-crafting good features — was 80% of an ML practitioner's job until ~2012. Deep learning largely automated it for images, audio, and text. This is a recurring theme: each era of AI specifies less and learns more.

Learning

Learning = an optimisation loop:

   +--------------------------------------------------+
   |  guess  ->  measure error  ->  adjust  ->  repeat |
   +--------------------------------------------------+
              (the "loss")      (the "gradient")

Loss — a single number saying how wrong the model is. Lower is better. You choose it. (Chapter 10.)
Gradient — which direction to nudge each knob to reduce the loss. (Chapter 15.)
Epoch — one full pass through all your training data.
Convergence — the loss has stopped improving; you're done.

The three flavours of learning

Flavour	You give it	It learns to	Example
Supervised	inputs + correct answers	predict the answer	spam detection, price prediction
Unsupervised	inputs only	find structure	grouping customers, compressing data
Self-supervised	inputs only, but you create labels from the input itself	predict a hidden part of the input	LLMs (hide the next word); image models (hide a patch)
Reinforcement (RL)	an environment + rewards	act to maximise reward	game AI; the "RLHF" stage of an LLM (Chapter 34)

Worked example: the same problem, three ways

Task: you have 10,000 customer support emails.

Supervised: you pay someone to tag 2,000 of them (billing / technical / sales), then train a model to tag the rest.
Unsupervised: you cluster all 10,000 by similarity and discover there are five natural groups — you didn't know the categories in advance.
Self-supervised: you take each email, hide a random word, and train a model to predict it. That model learns language in general, and can then be adapted cheaply to the tagging task with only 200 labelled examples.

The third one sounds indirect. It is also what made the last decade of AI work.

Practice (10 min)

# Look at real data before modelling it -- always do this first.
from sklearn.datasets import fetch_20newsgroups

data = fetch_20newsgroups(subset="train",
                          categories=["sci.space", "rec.sport.baseball"],
                          remove=("headers", "footers", "quotes"))

print("number of examples:", len(data.data))
print("label names:", data.target_names)
print("\n--- first example ---")
print(data.data[0][:400])
print("--- its label:", data.target_names[data.target[0]])

# how balanced are the classes?
import collections
print("\nlabel counts:", collections.Counter(data.target))

Then: print examples 1, 2, and 3. Can you tell which category they belong to? If a human struggles, a model will too — that tells you something about the ceiling on accuracy.

Common confusions

"Feature" vs "parameter": a feature is part of your input data; a parameter is a knob inside the model. Height is a feature; the 100 in weight = 100*height - 105 is a parameter.
"Label" vs "prediction": the label is the truth from your dataset; the prediction is what the model says.
More data always helps? More relevant, clean, diverse data helps. Duplicated or mislabelled data can hurt.

Check yourself

Give an example of a labelled dataset and an unlabelled one.
Why is self-supervised learning so important for LLMs?
What's the difference between a feature and a parameter?