65. The design lessons in requests

65.1 "HTTP for Humans": design the API from the call site

The README's first example has not really changed in over a decade:

r = requests.get("https://api.github.com/user", auth=("user", "pass"))
r.status_code; r.headers["content-type"]; r.json()

Compare that with urllib.request in 2011: build an opener, a password manager, an auth handler, a Request object, then call urlopen. Kenneth Reitz wrote the call he wanted first and implemented backwards from it. The lessons for your own libraries (Ch 67):

• Make the common case one call (requests.get) and the advanced case possible (Session, adapters, hooks, PreparedRequest).
• Accept friendly inputs, normalize once. params may be a dict, a list of tuples, or bytes; auth may be a tuple or an AuthBase; timeout a float or a tuple. prepare_* converts all of them into one canonical form, so the rest of the code deals with one shape.
• Return rich objects. Response has .json(), .text (decoded), .content (bytes), .ok, .raise_for_status(), .elapsed, .history, and .request, so debugging needs no extra tools.

65.2 An exception hierarchy users can catch at the right level

# requests/exceptions.py  (abridged)
class RequestException(IOError): ...                        # catch-all for this library
class HTTPError(RequestException): ...                      # raise_for_status()
class ConnectionError(RequestException): ...
class ProxyError(ConnectionError): ...
class SSLError(ConnectionError): ...
class Timeout(RequestException): ...
class ConnectTimeout(ConnectionError, Timeout):
    """The request timed out while trying to connect to the remote server.
    Requests that produced this error are safe to retry."""
class ReadTimeout(Timeout): ...
class MissingSchema(RequestException, ValueError): ...      # also a ValueError: it IS a bad argument
class JSONDecodeError(InvalidJSONError, CompatJSONDecodeError): ...

This is Ch 7.4 done well:

• One base class (RequestException) for "anything requests raised".
• Multiple inheritance to express two truths at once. ConnectTimeout is both a ConnectionError and a Timeout, so code catching either handles it. MissingSchema is also a ValueError, so generic argument-validation code catches it. RequestException subclasses IOError (OSError), so code written for I/O errors keeps working.
• The docstring carries operational knowledge: a ConnectTimeout is safe to retry (nothing reached the server); a ReadTimeout may not be (the server may have processed the request). That distinction is the idempotency rule from Ch 43.2, encoded in the type system.

65.3 Extension points instead of options

requests has few configuration flags. It has three extension points instead:

import requests
from requests.auth import AuthBase

class BearerAuth(AuthBase):                    # 1. auth: a callable that edits the PreparedRequest
    def __init__(self, token: str) -> None:
        self.token = token
    def __call__(self, r: requests.PreparedRequest) -> requests.PreparedRequest:
        r.headers["Authorization"] = f"Bearer {self.token}"
        return r

def log_response(r: requests.Response, *args, **kwargs) -> None:   # 2. hooks: observe or replace responses
    print(f"{r.request.method} {r.url} -> {r.status_code} in {r.elapsed.total_seconds() * 1000:.0f} ms")

s = requests.Session()
s.auth = BearerAuth("example-token")
s.hooks["response"].append(log_response)
# 3. transport adapters: s.mount("https://internal.example/", MyAdapter()) for mTLS, retries, mocking, unix sockets
s.get("https://example.com", timeout=5)

Auth objects, hooks, and adapters are small interfaces (one method each), so third parties built requests-oauthlib, requests-aws4auth, requests-unixsocket, and the responses mocking library without changing requests. Prefer a small interface over a growing list of keyword arguments.

65.4 What requests got wrong, or cannot change

Mature projects carry decisions they would make differently today, and reading them is as useful as reading the good parts:

• No default timeout. Changing it would break code that relies on long downloads, so it stays None; linters (ruff rule S113) flag calls without timeout=.
• requests.get() creates a session per call, which is convenient and slow (§64.1).
• Synchronous only. Adding async would mean two code paths through every layer. httpx was built to have both from the start (and HTTP/2).
• Feature-frozen. The project accepts bug fixes but very few new features. For an API used by millions of programs, stability is the feature.