54. pandas and Polars: tables, groupby, joins, and time series

54.1 pandas in ten operations

import numpy as np
import pandas as pd

rng = np.random.default_rng(0)
df = pd.DataFrame({
    "ts": pd.date_range("2026-10-04 10:00", periods=1000, freq="7s", tz="UTC"),
    "path": rng.choice(["/", "/api/orders", "/api/users", "/login"], size=1000),
    "status": rng.choice([200, 200, 200, 404, 500], size=1000),
    "ms": rng.lognormal(3, 0.6, size=1000).round(1),
})

print(df.head(3))                                   # look at the data first
print(df.dtypes, df.shape)
print(df.describe())                                # count, mean, std, quartiles
print(df[df["status"] >= 500].shape[0], "errors")   # boolean filter
print(df.groupby("path")["ms"].agg(["count", "median", lambda s: s.quantile(0.95)]))
print(df["status"].value_counts(normalize=True).round(3))
df["is_error"] = df["status"] >= 500                # new column, vectorized
per_min = df.set_index("ts").resample("1min").agg({"is_error": "mean", "ms": "median"})
print(per_min.head())                               # time-series resampling
owners = pd.DataFrame({"path": ["/api/orders", "/api/users"], "team": ["checkout", "identity"]})
print(df.merge(owners, on="path", how="left")["team"].fillna("web").value_counts())   # join

This is the log analyzer from Ch 12, reduced to a dozen vectorized operations. pandas is the right tool when the data fits in memory and you are exploring.

54.2 The pandas traps

• Chained assignment. df[df.x > 0]["y"] = 1 may modify a temporary copy and do nothing. Use df.loc[df.x > 0, "y"] = 1. pandas 3 enables copy-on-write by default, which makes the behaviour consistent: every indexing result acts as a copy.
• apply with a Python function is a Python loop in disguise. Prefer vectorized column operations, .str and .dt accessors, and np.where.
• dtypes. A column of numbers with one bad value becomes object (a column of Python objects: slow and error-prone). Use pd.to_numeric(errors="coerce") and check df.dtypes after loading.
• Memory. A CSV of 1 GB can take 5–10 GB as a DataFrame. Pass dtype= and usecols= to read_csv, use category for repeated strings, and read Parquet instead of CSV.

54.3 Polars: the fast, lazy alternative

Polars is a DataFrame library written in Rust. It is multi-threaded by default, uses Apache Arrow memory, and has a lazy API that optimizes the whole query before running it, and can stream data larger than memory.

import polars as pl

lf = pl.LazyFrame({
    "path": ["/", "/api/orders", "/api/orders", "/login"] * 250,
    "status": [200, 500, 200, 401] * 250,
    "ms": [12.0, 340.0, 95.0, 20.0] * 250,
})
out = (
    lf.filter(pl.col("status") < 500)
      .group_by("path")
      .agg(pl.len().alias("n"), pl.col("ms").median().alias("p50"), pl.col("ms").quantile(0.95).alias("p95"))
      .sort("p95", descending=True)
      .collect()                       # nothing runs until collect(): the plan is optimized first
)
print(out)
# For files: pl.scan_parquet("logs/*.parquet") / pl.scan_csv(...) read lazily, in parallel, in chunks.

Which to use: pandas for exploration and compatibility (every library accepts it); Polars when performance or memory matters, or for pipelines over many files. Both read and write Parquet, the columnar file format you should use for any table bigger than a spreadsheet.