65. Debugging: backtraces, lldb/gdb, tokio-console, Miri

65.1 Panics and backtraces

RUST_BACKTRACE=1 cargo run      # backtrace on panic
RUST_BACKTRACE=full cargo run   # with every frame, including std internals
RUST_LIB_BACKTRACE=1            # anyhow: capture a backtrace when an error is created

In production, install a panic hook that sends the panic to your logs, so it is not lost on a stderr nobody reads:

fn main() {
    let default_hook = std::panic::take_hook();
    std::panic::set_hook(Box::new(move |info| {
        let location = info.location().map(|l| format!("{}:{}", l.file(), l.line())).unwrap_or_default();
        let msg = info.payload().downcast_ref::<&str>().copied()
            .or_else(|| info.payload().downcast_ref::<String>().map(String::as_str))
            .unwrap_or("<non-string panic>");
        // In a service: tracing::error!(%location, msg, "panic") so it reaches your log pipeline.
        eprintln!(r#"{{"level":"error","event":"panic","location":"{location}","message":"{msg}"}}"#);
        default_hook(info);
    }));
    let v: Vec<u8> = Vec::new();
    let _ = std::panic::catch_unwind(|| v[3]);
}

65.2 A debugger session

cargo build                         # debug info is on by default in the dev profile
rust-lldb target/debug/kv           # or rust-gdb on Linux; both add pretty-printers for Vec, String, Option...
(lldb) b kv.rs:120                  # breakpoint at a line
(lldb) r serve
(lldb) frame variable args          # prints Vec<Vec<u8>> readably thanks to the Rust formatters
(lldb) bt                           # backtrace

VS Code with rust-analyzer plus the CodeLLDB extension gives you the same thing with a UI: click "Debug" above any fn main or #[test]. Delve in the Go guide §58 and gdb in the C guide §14.4 are the equivalents.

65.3 Async debugging

A stuck async service usually has one of three causes: a task blocked in synchronous code (§31.5), a lock held across .await, or a future that is never woken. Two tools help:

• tracing spans around each request and each important .await, so the logs show where a request stopped making progress.
• tokio-console, which works like top for tasks. It shows every task, how long it has been idle or busy, and its wake-up count. Enable it with the console-subscriber crate and RUSTFLAGS="--cfg tokio_unstable". A task with a large "busy" time and no .await is blocking the executor.

65.4 The rest of the toolbox