64. The call path: requests.get() to the socket

64.1 api.py: every call gets a throwaway Session

# requests/api.py
def request(method: str, url: _t.UriType, **kwargs: Unpack[_t.RequestKwargs]) -> Response:
    # By using the 'with' statement we are sure the session is closed, thus we
    # avoid leaving sockets open which can trigger a ResourceWarning in some
    # cases, and look like a memory leak in others.
    with sessions.Session() as session:
        return session.request(method=method, url=url, **kwargs)

def get(url, params=None, **kwargs):
    return request("get", url, params=params, **kwargs)

Here is the performance fact from Ch 43.1 in the source: requests.get() builds a new Session, which builds new connection pools, uses them once, and closes them. Every module-level call pays for a fresh TCP and TLS handshake. A long-lived Session does not. Notice also the with statement (Ch 8) and the typed **kwargs (Unpack[...] of a TypedDict, Ch 18): recent releases added full type hints.

64.2 Session.__init__: defaults and mounted adapters

# requests/sessions.py  (comments trimmed)
class Session(SessionRedirectMixin):
    def __init__(self) -> None:
        self.headers = default_headers()      # User-Agent, Accept-Encoding, Accept, Connection: keep-alive
        self.auth = None
        self.proxies = {}
        self.hooks = default_hooks()          # {"response": []}
        self.params = {}
        self.stream = False
        self.verify = True                    # TLS verification ON by default
        self.cert = None
        self.max_redirects = DEFAULT_REDIRECT_LIMIT   # 30
        self.trust_env = True                 # read proxies/CA bundle/netrc from the environment
        self.cookies = cookiejar_from_dict({})
        self.adapters = OrderedDict()
        self.mount("https://", HTTPAdapter())
        self.mount("http://", HTTPAdapter())

A Session is a bag of defaults plus a prefix → adapter table. That table is a composition point (Ch 17.2): you can mount a different adapter for a specific host or scheme:

# requests/sessions.py
def mount(self, prefix: str, adapter: BaseAdapter) -> None:
    """Registers a connection adapter to a prefix.
    Adapters are sorted in descending order by prefix length."""
    self.adapters[prefix] = adapter
    keys_to_move = [k for k in self.adapters if len(k) < len(prefix)]
    for key in keys_to_move:
        self.adapters[key] = self.adapters.pop(key)

def get_adapter(self, url: str) -> BaseAdapter:
    for prefix, adapter in self.adapters.items():
        if url.lower().startswith(prefix.lower()):
            return adapter
    raise InvalidSchema(f"No connection adapters were found for {url!r}")

mount keeps the OrderedDict sorted longest-prefix-first by moving every shorter key to the end, so get_adapter can return the first match. It is longest-prefix matching, the same rule a router uses to pick a route (Net Ch 13), done with a dict that remembers insertion order (Ch 6.4).

64.3 Session.request: build, prepare, merge, send

# requests/sessions.py  (docstring trimmed)
def request(self, method, url, params=None, data=None, headers=None, cookies=None, files=None,
            auth=None, timeout=None, allow_redirects=True, proxies=None, hooks=None,
            stream=None, verify=None, cert=None, json=None) -> Response:
    if isinstance(url, bytes):
        url = url.decode("utf-8")

    # Create the Request.
    req = Request(
        method=method.upper(), url=url, headers=headers, files=files, data=data or {},
        json=json, params=params or {}, auth=auth, cookies=cookies, hooks=hooks,
    )
    prep = self.prepare_request(req)

    proxies = proxies or {}
    settings = self.merge_environment_settings(prep.url, proxies, stream, verify, cert)

    # Send the request.
    send_kwargs = {"timeout": timeout, "allow_redirects": allow_redirects}
    send_kwargs.update(settings)
    resp = self.send(prep, **send_kwargs)
    return resp

Note timeout=None in the signature. That default is the "requests has no timeout" fact from Ch 43.1. It is passed all the way down to the socket, where None means "block forever".

Three objects, three jobs:

prepare_request merges session-level settings (headers, cookies, auth) into the request, then calls PreparedRequest.prepare:

# requests/models.py
def prepare(self, method=None, url=None, headers=None, files=None, data=None,
            params=None, auth=None, cookies=None, hooks=None, json=None) -> None:
    """Prepares the entire request with the given parameters."""
    self.prepare_method(method)
    self.prepare_url(url, params)          # IDNA-encode the host, merge params into the query string
    self.prepare_headers(headers)          # CaseInsensitiveDict; validates header values
    self.prepare_cookies(cookies)
    self.prepare_body(data, files, json)   # json -> bytes + Content-Type; files -> multipart
    self.prepare_auth(auth, url)

    # Note that prepare_auth must be last to enable authentication schemes
    # such as OAuth to work on a fully prepared request.

    # This MUST go after prepare_auth. Authenticators could add a hook
    self.prepare_hooks(hooks)

The comments are design notes worth copying: the order of steps is a contract. An OAuth 1 signer must see the final URL, headers, and body, because it signs them; anything that changed them afterwards would break the signature.

merge_environment_settings is where trust_env takes effect:

# requests/sessions.py  (simplified)
if self.trust_env:
    env_proxies = get_environ_proxies(url, no_proxy=no_proxy)    # HTTP_PROXY, HTTPS_PROXY, NO_PROXY
    for k, v in env_proxies.items():
        proxies.setdefault(k, v)
    if verify is True or verify is None:
        verify = os.environ.get("REQUESTS_CA_BUNDLE") or os.environ.get("CURL_CA_BUNDLE") or verify

That is why REQUESTS_CA_BUNDLE=/path/corp-ca.pem fixes certificate errors behind a corporate TLS-inspecting proxy (Ch 45.1), and why a stray HTTPS_PROXY in a container's environment silently routes your traffic through a proxy. When a request behaves differently on one machine, check the environment.

64.4 Session.send: adapter, hooks, cookies, redirects

# requests/sessions.py  (comments trimmed)
def send(self, request: PreparedRequest, **kwargs) -> Response:
    kwargs.setdefault("stream", self.stream)
    kwargs.setdefault("verify", self.verify)
    kwargs.setdefault("cert", self.cert)
    if "proxies" not in kwargs:
        kwargs["proxies"] = resolve_proxies(request, self.proxies, self.trust_env)

    if isinstance(request, Request):
        raise ValueError("You can only send PreparedRequests.")

    allow_redirects = kwargs.pop("allow_redirects", True)
    stream = kwargs.get("stream")
    hooks = request.hooks

    adapter = self.get_adapter(url=request.url)
    start = preferred_clock()
    r = adapter.send(request, **kwargs)                  # ◀── down into the transport
    elapsed = preferred_clock() - start
    r.elapsed = timedelta(seconds=elapsed)

    r = dispatch_hook("response", hooks, r, **kwargs)    # user hooks can replace the response

    if r.history:
        for resp in r.history:
            extract_cookies_to_jar(self.cookies, resp.request, resp.raw)
    extract_cookies_to_jar(self.cookies, request, r.raw)

    if allow_redirects:
        gen = self.resolve_redirects(r, request, **kwargs)   # a GENERATOR of responses
        history = [resp for resp in gen]
    else:
        history = []

    if history:
        history.insert(0, r)
        r = history.pop()
        r.history = history
    ...
    if not stream:
        r.content            # read the whole body now (a property that downloads on first access)
    return r

Things to notice:

• r.elapsed is measured with preferred_clock() (time.perf_counter), a monotonic clock, not wall time (Ch 19.1). It measures until the headers arrived, not the body.
• resolve_redirects is a generator (Ch 13) that yields each redirect response; send consumes it with a list comprehension. The same generator is reused with yield_requests=True to compute r.next without following it.
• r.content on a line by itself looks like a no-op. It is a property whose getter downloads the body. Without stream=True, the body is read into memory before send returns; with stream=True, the connection stays checked out of the pool until you read or close the response, which is why streaming code must use with session.get(..., stream=True) as r:.

Redirects and security. resolve_redirects calls rebuild_auth, which deletes the Authorization header when a redirect goes to a different host (should_strip_auth), so a malicious redirect cannot harvest your credentials; and rebuild_method, which turns a POST into a GET on 302/303 the way browsers do. Both encode lessons from past bug reports.

64.5 HTTPAdapter.send: the boundary to urllib3

# requests/adapters.py  (abridged)
DEFAULT_POOLBLOCK = False
DEFAULT_POOLSIZE = 10
DEFAULT_RETRIES = 0

class HTTPAdapter(BaseAdapter):
    def __init__(self, pool_connections=DEFAULT_POOLSIZE, pool_maxsize=DEFAULT_POOLSIZE,
                 max_retries=DEFAULT_RETRIES, pool_block=DEFAULT_POOLBLOCK):
        if max_retries == DEFAULT_RETRIES:
            self.max_retries = Retry(0, read=False)
        else:
            self.max_retries = Retry.from_int(max_retries)
        ...
        self.init_poolmanager(pool_connections, pool_maxsize, block=pool_block)

    def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None):
        try:
            conn = self.get_connection_with_tls_context(request, verify, proxies=proxies, cert=cert)
        except LocationValueError as e:
            raise InvalidURL(e, request=request)

        self.cert_verify(conn, request.url, verify, cert)
        url = self.request_url(request, proxies)        # path only, or the full URL for an HTTP proxy
        chunked = not (request.body is None or "Content-Length" in request.headers)

        if isinstance(timeout, tuple):
            connect, read = timeout
            resolved_timeout = TimeoutSauce(connect=connect, read=read)
        elif isinstance(timeout, TimeoutSauce):
            resolved_timeout = timeout
        else:
            resolved_timeout = TimeoutSauce(connect=timeout, read=timeout)

        try:
            resp = conn.urlopen(
                method=request.method, url=url, body=request.body, headers=request.headers,
                redirect=False,               # requests handles redirects itself (64.4)
                assert_same_host=False,
                preload_content=False,        # requests decides when to read the body
                decode_content=False,
                retries=self.max_retries,     # Retry(0, read=False) by default: no retries
                timeout=resolved_timeout,
                chunked=chunked,
            )
        except (ProtocolError, OSError) as err:
            raise ConnectionError(err, request=request)
        except MaxRetryError as e:
            if isinstance(e.reason, ConnectTimeoutError):
                if not isinstance(e.reason, NewConnectionError):
                    raise ConnectTimeout(e, request=request)
            if isinstance(e.reason, ResponseError):
                raise RetryError(e, request=request)
            if isinstance(e.reason, _ProxyError):
                raise ProxyError(e, request=request)
            if isinstance(e.reason, _SSLError):
                raise SSLError(e, request=request)
            raise ConnectionError(e, request=request)
        ...
        return self.build_response(request, resp)

This method is an anti-corruption layer: urllib3's exceptions (MaxRetryError, ProtocolError, _SSLError) never escape; they are translated into requests' own hierarchy (Ch 65.2), so users depend on requests' API, not on its engine. requests once bundled its own copy of urllib3 and later switched to depending on it as a separate package; this layer is why user code did not have to change.

Defaults visible here: 10 pooled connections per host, no retries (Retry(0, read=False)), and the (connect, read) timeout tuple. To get status-based retries, mount an adapter with urllib3's Retry:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

retry = Retry(total=4, backoff_factor=0.5, status_forcelist=[429, 502, 503, 504],
              allowed_methods={"GET", "HEAD", "PUT", "DELETE"},   # idempotent methods only (Ch 14.2)
              respect_retry_after_header=True)
s = requests.Session()
s.mount("https://", HTTPAdapter(max_retries=retry, pool_maxsize=32))
r = s.get("https://example.com", timeout=(3.05, 10))
print(r.status_code, r.elapsed)

build_response turns urllib3's response into a requests Response: status code, a CaseInsensitiveDict of headers (HTTP header names are case insensitive, Ch 16 shows how to build such a container), encoding from Content-Type, cookies, and a back-reference to the request.

64.6 urllib3: the pool, the connection, and the socket

get_connection_with_tls_context asks urllib3's PoolManager for a connection pool keyed by (scheme, host, port, TLS settings). Each HTTPConnectionPool holds idle connections in a thread-safe queue.LifoQueue:

# urllib3/connectionpool.py  (abridged)
def _get_conn(self, timeout=None):
    ...
    try:
        conn = self.pool.get(block=self.block, timeout=timeout)   # reuse an idle connection
    except queue.Empty:
        if self.block:
            raise EmptyPoolError(self, "Pool is empty and a new connection can't be opened due to blocking mode.")
        pass  # Oh well, we'll create a new connection then

    # If this is a persistent connection, check if it got disconnected
    if conn and is_connection_dropped(conn):
        log.debug("Resetting dropped connection: %s", self.host)
        conn.close()

    return conn or self._new_conn()

LIFO, not FIFO: the most recently used connection is the most likely to still be open (servers close idle keep-alive connections after a timeout), so reusing it first avoids dead sockets. is_connection_dropped checks the socket anyway (a zero-timeout poll for a pending FIN, Net Ch 23) and discards a connection the server already closed. After the response is read, _put_conn returns the connection to the queue. That is the entire mechanism behind "a Session reuses connections".

urlopen gets a connection, sends the request with _make_request (which writes the request line and headers through Python's http.client), and reads the status line and headers. When there is no idle connection, HTTPSConnection.connect() opens one, and the networking finally happens:

# urllib3/util/connection.py  (abridged)
def create_connection(address, timeout=_DEFAULT_TIMEOUT, source_address=None, socket_options=None):
    host, port = address
    ...
    family = allowed_gai_family()                     # IPv4, IPv6, or both
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):   # ◀── DNS
        af, socktype, proto, canonname, sa = res
        sock = None
        try:
            sock = socket.socket(af, socktype, proto)   # ◀── socket(2)
            _set_socket_options(sock, socket_options)   # TCP_NODELAY=1 by default: Nagle off (Net Ch 26)
            if timeout is not _DEFAULT_TIMEOUT:
                sock.settimeout(timeout)                # the connect timeout from requests
            if source_address:
                sock.bind(source_address)
            sock.connect(sa)                            # ◀── connect(2): the TCP three-way handshake
            err = None
            return sock
        except OSError as _:
            err = _
            if sock is not None:
                sock.close()
    ...

Then, for HTTPS, HTTPSConnection.connect() wraps that socket with ssl_wrap_socket(...), an ssl.SSLContext loaded with the CA bundle from certifi (requests/certs.py is literally from certifi import where), passing server_hostname for SNI and hostname verification. That is the TLS handshake from Sec Ch 28 and the ssl code from Ch 45.1.

Notice the loop over getaddrinfo results: if a name resolves to several addresses (IPv6 and IPv4, or several IPv4 addresses) and the first one fails, urllib3 tries the next. The connect timeout applies per address, so a host with four dead addresses and timeout=(5, 30) can take 20 seconds to fail. Knowing that one detail explains a whole class of "why did the timeout take four times longer than I set?" incidents.

64.7 The whole path on one page

requests.get(url, timeout=(3, 10))                                  api.py
 └─ Session().request("GET", url, ...)                              sessions.py
     ├─ Request(...) → prepare_request() → PreparedRequest          models.py  (exact bytes)
     ├─ merge_environment_settings()                                env: proxies, CA bundle
     └─ Session.send(prep)
         ├─ get_adapter(url) → HTTPAdapter  (longest prefix)
         ├─ HTTPAdapter.send(prep)                                  adapters.py
         │   ├─ PoolManager.connection_from_host(...)               urllib3: pool per (scheme, host, port, TLS)
         │   └─ HTTPSConnectionPool.urlopen(...)
         │       ├─ _get_conn()  → idle connection from LifoQueue, or
         │       │   HTTPSConnection.connect()
         │       │    ├─ create_connection()
         │       │    │   ├─ socket.getaddrinfo()   ── DNS ────────────── Net Ch 18
         │       │    │   ├─ socket.socket()        ── socket(2) ───────── OS Ch 19
         │       │    │   └─ sock.connect()         ── TCP handshake ───── Net Ch 21
         │       │    └─ ssl_wrap_socket()          ── TLS 1.3 handshake ─ Sec Ch 28
         │       ├─ _make_request() → http.client writes "GET / HTTP/1.1" ── Net Ch 39
         │       └─ read status line + headers
         │   └─ build_response() → Response; urllib3 errors → requests errors
         ├─ dispatch_hook("response"), extract cookies
         ├─ resolve_redirects() (generator), strip auth on host change
         └─ r.content  (read the body unless stream=True), connection back to the pool

Run the traceback.print_stack experiment from §63.2 again and match each frame to a line of this diagram.