59. Hugging Face: tokenizers, models, and datasets

The Hugging Face libraries are how most people load open-weight models: tokenizers (Rust-backed), transformers (model code and weights), datasets (Arrow-backed, memory-mapped datasets), and the Hub.

# uv add transformers tokenizers torch
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

name = "HuggingFaceTB/SmolLM2-135M-Instruct"        # a tiny open model that runs on a laptop CPU
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)   # float32 on CPU by default

ids = tok("Python is the language of AI because", return_tensors="pt")
print(ids["input_ids"][0].tolist())                 # tokens: integers, as in AI Part 7
print([tok.decode([i]) for i in ids["input_ids"][0][:8]])

messages = [{"role": "user", "content": "In one sentence, what is a Python generator?"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True,
                                 return_tensors="pt", return_dict=True)    # input_ids + attention_mask
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=60, do_sample=False)
new_tokens = out[0][inputs["input_ids"].shape[1]:]                         # drop the prompt
print(tok.decode(new_tokens, skip_special_tokens=True))

Map each line to the AI guide: the tokenizer is BPE from AI Ch 28; the chat template is AI Ch 30; generate is the next-token loop with the KV cache and sampling settings.

# datasets: load, stream, and transform without loading everything into RAM
from datasets import load_dataset

ds = load_dataset("fancyzhx/ag_news", split="train", streaming=True)   # iterable: Ch 13's lazy pipeline
for row in ds.take(2):
    print(row["label"], row["text"][:70])

Security and hygiene for the Hub: model repositories can contain code (trust_remote_code=True runs Python from the repository, so read it first or don't), and pickled weights (prefer repositories with .safetensors). Pin a specific revision= (a commit hash) in anything reproducible; a repository's main branch can change under you, which is the model-world version of the missing lockfile in Ch 21.4.

For fine-tuning, the AI guide's LoRA project uses peft and trl on top of these same classes.