48. Outbound requests: timeouts, retries, idempotency, and an SSRF guard

Implements: HTTPS Ch 21 and Ch 22; Security in Depth Ch 33.

48.1 Retries with backoff, jitter, and an idempotency key

A retry is only safe if repeating the request cannot do the work twice. GET is safe by definition. A POST /payments is safe only if the server deduplicates by an idempotency key that the client generates once, before the first attempt. The demo below simulates the worst case: the server charges the card, then the response is lost (here, a 503 sent after the charge).

// src/bin/retries.rs
use axum::extract::State;
use axum::http::{HeaderMap, StatusCode};
use axum::{routing::post, Router};
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
use std::time::Duration;

#[derive(Default)]
struct Ledger { charges: u32, attempts: u32, seen: HashMap<String, String> }

async fn pay(State(l): State<Arc<Mutex<Ledger>>>, headers: HeaderMap) -> (StatusCode, String) {
    let Some(key) = headers.get("idempotency-key").and_then(|v| v.to_str().ok()).map(String::from) else {
        return (StatusCode::BAD_REQUEST, "Idempotency-Key required".into());
    };
    let mut l = l.lock().unwrap();
    l.attempts += 1;
    if let Some(prev) = l.seen.get(&key) {
        return (StatusCode::CREATED, prev.clone());          // replay the stored result; no second charge
    }
    l.charges += 1;
    let receipt = format!("receipt-{}", l.charges);
    l.seen.insert(key, receipt.clone());
    if l.attempts == 1 {
        return (StatusCode::SERVICE_UNAVAILABLE, "lost response".into()); // charged, but the client sees a failure
    }
    (StatusCode::CREATED, receipt)
}

fn backoff(attempt: u32) -> Duration {
    let cap_ms = 2_000u64;
    let exp_ms = 100u64.saturating_mul(1 << attempt.min(10)).min(cap_ms);
    Duration::from_millis(rand::random_range(0..=exp_ms))     // "full jitter": avoid synchronized retry storms
}

fn retryable(status: StatusCode) -> bool {
    matches!(status, StatusCode::TOO_MANY_REQUESTS | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT)
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let ledger = Arc::new(Mutex::new(Ledger::default()));
    let app = Router::new().route("/payments", post(pay)).with_state(Arc::clone(&ledger));
    let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await?;
    let url = format!("http://{}/payments", listener.local_addr()?);
    tokio::spawn(async move { axum::serve(listener, app).await });

    let client = reqwest::Client::builder()
        .connect_timeout(Duration::from_secs(2))
        .timeout(Duration::from_secs(5))                       // total per-attempt deadline
        .build()?;
    let key = format!("{:032x}", rand::random::<u128>());       // generated ONCE, reused by every attempt

    let mut attempt = 0;
    let receipt = loop {
        let result = client.post(&url).header("Idempotency-Key", &key).body("amount=500").send().await;
        match result {
            Ok(r) if r.status().is_success() => break r.text().await?,
            Ok(r) if retryable(r.status()) && attempt < 4 => println!("attempt {attempt}: {}, retrying", r.status()),
            Err(e) if (e.is_timeout() || e.is_connect()) && attempt < 4 => println!("attempt {attempt}: {e}, retrying"),
            Ok(r) => anyhow::bail!("giving up: {}", r.status()),
            Err(e) => return Err(e.into()),
        }
        tokio::time::sleep(backoff(attempt)).await;
        attempt += 1;
    };
    let l = ledger.lock().unwrap();
    println!("{receipt}; server saw {} attempts and charged {} time(s)", l.attempts, l.charges);
    Ok(())
}

attempt 0: 503 Service Unavailable, retrying
receipt-1; server saw 2 attempts and charged 1 time(s)

Without the key, the retry would have charged the card twice. In a real service, store the key with the result in the same database transaction as the charge, and expire keys after a retention period. Add a retry budget (for example, retries may be at most 10% of requests) so a struggling dependency does not receive three times its normal load. That is the retry-storm failure described in HTTPS Ch 21.

48.2 An SSRF guard that survives DNS rebinding

A server that fetches user-supplied URLs (webhooks, link previews, image imports) can be pointed at http://169.254.169.254/ (cloud credentials), http://localhost:6379/ (an internal Redis), or a private network. Checking the hostname string is not enough. evil.example can resolve to 127.0.0.1, or resolve to a public IP during your check and a private one when the HTTP client connects (DNS rebinding). The fix: resolve once, check every address, and connect only to the address you checked.

// src/bin/safefetch.rs
use std::net::{IpAddr, SocketAddr};
use std::time::Duration;

fn is_forbidden(ip: IpAddr) -> bool {
    match ip {
        IpAddr::V4(v4) => {
            let o = v4.octets();
            v4.is_private() || v4.is_loopback() || v4.is_link_local()  // 10/8 172.16/12 192.168/16 127/8 169.254/16
                || v4.is_unspecified() || v4.is_broadcast() || v4.is_documentation() || v4.is_multicast()
                || o[0] == 0                                           // 0.0.0.0/8 ("this network")
                || (o[0] == 100 && (o[1] & 0xC0) == 64)                // 100.64/10 carrier-grade NAT
                || (o[0] == 198 && (o[1] & 0xFE) == 18)                // 198.18/15 benchmarking
        }
        IpAddr::V6(v6) => {
            let s = v6.segments();
            v6.is_loopback() || v6.is_unspecified() || v6.is_multicast()
                || (s[0] & 0xFE00) == 0xFC00                           // fc00::/7 unique local
                || (s[0] & 0xFFC0) == 0xFE80                           // fe80::/10 link local
                || v6.to_ipv4_mapped().is_some_and(|v4| is_forbidden(IpAddr::V4(v4))) // ::ffff:127.0.0.1
        }
    }
}

async fn safe_get(raw_url: &str, max_bytes: usize) -> anyhow::Result<String> {
    let url = reqwest::Url::parse(raw_url)?;
    anyhow::ensure!(matches!(url.scheme(), "http" | "https"), "scheme {} not allowed", url.scheme());
    anyhow::ensure!(url.username().is_empty() && url.password().is_none(), "credentials in URL not allowed");
    let host = url.host_str().ok_or_else(|| anyhow::anyhow!("no host"))?;
    let port = url.port_or_known_default().unwrap_or(80);
    let bare = host.trim_start_matches('[').trim_end_matches(']');   // IPv6 literals come bracketed

    // 1. Resolve ONCE, with a deadline.
    let addrs: Vec<SocketAddr> = tokio::time::timeout(Duration::from_secs(2), tokio::net::lookup_host((bare, port)))
        .await??.collect();
    // 2. EVERY address must be allowed: attackers return [public, private] and hope you check only the first.
    anyhow::ensure!(!addrs.is_empty(), "{host} did not resolve");
    if let Some(bad) = addrs.iter().find(|a| is_forbidden(a.ip())) {
        anyhow::bail!("{host} resolves to forbidden address {}", bad.ip());
    }
    // 3. PIN the connection to the vetted address, so a second DNS answer cannot redirect it.
    let client = reqwest::Client::builder()
        .resolve(bare, addrs[0])
        .redirect(reqwest::redirect::Policy::none())   // a redirect to http://127.0.0.1 would skip all checks
        .connect_timeout(Duration::from_secs(3))
        .timeout(Duration::from_secs(10))
        .build()?;
    let mut resp = client.get(url).send().await?;
    anyhow::ensure!(resp.status().is_success(), "status {}", resp.status());

    // 4. Cap the body while streaming. Content-Length can be missing or false.
    let mut body = Vec::new();
    while let Some(chunk) = resp.chunk().await? {
        anyhow::ensure!(body.len() + chunk.len() <= max_bytes, "response larger than {max_bytes} bytes");
        body.extend_from_slice(&chunk);
    }
    Ok(String::from_utf8_lossy(&body).into_owned())
}

#[tokio::main]
async fn main() {
    for url in [
        "http://169.254.169.254/latest/meta-data/",   // cloud metadata
        "http://localhost:6379/",                      // resolves to loopback
        "http://[::ffff:127.0.0.1]/",                  // IPv4-mapped IPv6 loopback
        "http://0x7f000001/",                          // 127.0.0.1 in hex: the URL parser normalizes it
        "http://10.1.2.3/admin",                       // private network
        "file:///etc/passwd",                          // wrong scheme
        "http://user:pw@example.com/",                 // credentials in the URL
    ] {
        match safe_get(url, 1 << 20).await {
            Ok(body) => println!("ALLOWED {url} ({} bytes)", body.len()),
            Err(e) => println!("blocked {url:<42} {e}"),
        }
    }
}

blocked http://169.254.169.254/latest/meta-data/   169.254.169.254 resolves to forbidden address 169.254.169.254
blocked http://localhost:6379/                     localhost resolves to forbidden address ::1
blocked http://[::ffff:127.0.0.1]/                 [::ffff:7f00:1] resolves to forbidden address ::ffff:127.0.0.1
blocked http://0x7f000001/                         127.0.0.1 resolves to forbidden address 127.0.0.1
blocked http://10.1.2.3/admin                      10.1.2.3 resolves to forbidden address 10.1.2.3
blocked file:///etc/passwd                         scheme file not allowed
blocked http://user:pw@example.com/                credentials in URL not allowed

Note the 0x7f000001 line. The WHATWG URL parser that reqwest::Url uses normalizes hex, octal, and short-form IPv4 hosts before you see them, so the check runs on the real address. Hand-written string checks such as host.starts_with("127.") miss all of these encodings. This is why every SSRF guide says to parse first and then check addresses, never strings.

A production version should also apply the check to every redirect hop (implement redirect::Policy::custom), and run the fetcher in a network segment with egress firewall rules as the second layer (Security in Depth Ch 8).