54. Fuzzing and property testing

54.1 Coverage-guided fuzzing with cargo-fuzz

A fuzzer runs your parser millions of times with mutated inputs and keeps the inputs that reach new code paths. In Rust it mostly finds panics (slice indexing, unwrap, overflow) and infinite loops. In unsafe code it also finds memory bugs, because the target is compiled with AddressSanitizer. Same idea as C §31's libFuzzer chapter.

cargo install cargo-fuzz          # needs a nightly toolchain: rustup toolchain install nightly
cargo fuzz init                   # creates fuzz/ with its own Cargo.toml
cargo fuzz add dns_parse

// fragment: fuzz/fuzz_targets/dns_parse.rs
#![no_main]
use libfuzzer_sys::fuzz_target;

fuzz_target!(|data: &[u8]| {
    // The only property: the parser must never panic or hang, whatever the bytes.
    let _ = rust_labs::dns::parse_response(data, u16::from_be_bytes([data.first().copied().unwrap_or(0), 0]));
});

cargo +nightly fuzz run dns_parse -- -max_total_time=300   # 5 minutes
# On a crash: fuzz/artifacts/dns_parse/crash-<hash> holds the input. Reproduce it with:
cargo +nightly fuzz run dns_parse fuzz/artifacts/dns_parse/crash-<hash>

Move the parser into src/lib.rs (as pub mod dns) so both the binary and the fuzz target can use it. Try it on the Ch 45 parser with the jumps > 16 check removed. The fuzzer finds the pointer loop, reported as a timeout, within seconds.

54.2 Property tests for invariants

Fuzzing asks "does it crash?". Property tests check that a rule holds, for example that parsing a value you serialized gives back the same value:

#[derive(Debug, Clone, PartialEq)]
struct Frame { stream: u32, payload: Vec<u8> }

fn encode(f: &Frame) -> Vec<u8> {
    let mut out = f.stream.to_be_bytes().to_vec();
    out.extend_from_slice(&(f.payload.len() as u32).to_be_bytes());
    out.extend_from_slice(&f.payload);
    out
}

fn decode(b: &[u8]) -> Option<(Frame, &[u8])> {
    let (stream, rest) = b.split_at_checked(4)?;
    let (len, rest) = rest.split_at_checked(4)?;
    let len = u32::from_be_bytes(len.try_into().ok()?) as usize;
    let (payload, rest) = rest.split_at_checked(len)?;
    Some((Frame { stream: u32::from_be_bytes(stream.try_into().ok()?), payload: payload.to_vec() }, rest))
}

#[cfg(test)]
mod tests {
    use super::*;
    use proptest::prelude::*;

    proptest! {
        #[test]
        fn roundtrip(stream: u32, payload in proptest::collection::vec(any::<u8>(), 0..2048)) {
            let f = Frame { stream, payload };
            let wire = encode(&f);
            prop_assert_eq!(decode(&wire), Some((f, &[][..])));
        }

        #[test]
        fn never_panics_and_never_overreads(bytes in proptest::collection::vec(any::<u8>(), 0..64)) {
            if let Some((f, rest)) = decode(&bytes) {
                prop_assert_eq!(8 + f.payload.len() + rest.len(), bytes.len());
            }
        }
    }
}

fn main() {
    let f = Frame { stream: 1, payload: b"hi".to_vec() };
    println!("{:?}", decode(&encode(&f)));
}

Use both. Property tests run in milliseconds in cargo test on every commit. Fuzzers run for hours in CI or on OSS-Fuzz, which accepts Rust projects.