68. Logging and observability

68.1 Structured logs

Plain text logs are for humans; production logs are read by machines (Loki, Elasticsearch, CloudWatch). Emit JSON, one object per line, with consistent fields:

import json
import logging
import sys
import time
import contextvars
import uuid

request_id: contextvars.ContextVar[str] = contextvars.ContextVar("request_id", default="-")

class JsonFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        doc = {
            "ts": time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(record.created)) + f".{int(record.msecs):03d}Z",
            "level": record.levelname.lower(),
            "logger": record.name,
            "msg": record.getMessage(),
            "request_id": request_id.get(),
        }
        doc.update(getattr(record, "fields", {}))
        if record.exc_info:
            doc["exc"] = self.formatException(record.exc_info)
        return json.dumps(doc, default=str)

handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JsonFormatter())
logging.basicConfig(level=logging.INFO, handlers=[handler], force=True)
log = logging.getLogger("orders")

def handle_request(order_id: int) -> None:
    request_id.set(uuid.uuid4().hex[:8])          # per request; asyncio tasks each get their own copy
    log.info("order received", extra={"fields": {"order_id": order_id}})
    try:
        raise TimeoutError("payment provider")
    except TimeoutError:
        log.exception("payment failed", extra={"fields": {"order_id": order_id, "provider": "psp-1"}})

handle_request(7)

contextvars is how a request ID follows a request through every function and every await without passing it around: each asyncio task and each thread sees its own value. structlog and python-json-logger are mature libraries for the same idea.

Rules that save incidents: log events with fields, not prose ("order received", order_id=7); never log secrets, tokens, or full request bodies with personal data (Security Ch 50); log at the boundaries (request in, request out, external call, error), not inside every loop.

68.2 Metrics and traces

• Metrics (counts, latencies, queue depths) with prometheus_client: a Histogram for request latency, a Counter for errors, scraped from /metrics. Use them for dashboards and alerts (Net Ch 45–47).
• Traces with OpenTelemetry: auto-instrumentation for FastAPI, requests, httpx, database drivers, and many ML-serving stacks, so one request's path through services shows up as a timeline (OS Ch 70):

uv add opentelemetry-distro opentelemetry-exporter-otlp
uv run opentelemetry-bootstrap -a install       # installs instrumentations for the libraries you use
OTEL_SERVICE_NAME=tickets OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
  uv run opentelemetry-instrument uvicorn labs.api:app

For training jobs, the equivalent "observability" is loss curves, GPU utilization, data-loader wait time, and throughput (samples or tokens per second), logged per step.