44. A container in 100 lines: namespaces with `nix`

A container is a process with its own namespaces (what it can see), cgroups (what it can use), and root filesystem (OS Ch 44). The Go version of this lab is OS Ch 80. Run this one as root inside the --privileged lab container.

// linux-only
// usage: minibox HOSTNAME CMD [ARGS...]     e.g. minibox box1 sh
use nix::mount::{mount, MsFlags};
use nix::sched::{unshare, CloneFlags};
use nix::sys::wait::{waitpid, WaitStatus};
use nix::unistd::{execvp, fork, pipe, sethostname, ForkResult};
use std::ffi::CString;
use std::fs::File;
use std::io::{Read, Write};

fn setup_cgroup(pid: i32) -> std::io::Result<()> {
    // cgroup v2: a child group with a 64 MiB memory cap and a 20-process cap.
    let cg = "/sys/fs/cgroup/minibox";
    std::fs::create_dir_all(cg)?;
    for (file, value) in [("memory.max", "67108864".to_string()), ("pids.max", "20".into()), ("cgroup.procs", pid.to_string())] {
        // memory.max only exists if the parent delegates the memory controller (cgroup.subtree_control).
        std::fs::write(format!("{cg}/{file}"), value)
            .map_err(|e| std::io::Error::new(e.kind(), format!("{cg}/{file}: {e}")))?;
    }
    Ok(())
}

fn child(hostname: &str, argv: &[CString]) -> nix::Result<()> {
    sethostname(hostname)?;                                   // visible only inside the new UTS namespace
    // Make every mount private so our /proc mount does not propagate back to the host.
    mount(None::<&str>, "/", None::<&str>, MsFlags::MS_REC | MsFlags::MS_PRIVATE, None::<&str>)?;
    // A fresh /proc that shows only this PID namespace: `ps` inside sees PID 1 = our command.
    mount(Some("proc"), "/proc", Some("proc"), MsFlags::empty(), None::<&str>)?;
    execvp(&argv[0], argv)?;                                  // replaces this process; returns only on error
    unreachable!()
}

fn main() {
    let args: Vec<String> = std::env::args().skip(1).collect();
    if args.len() < 2 { eprintln!("usage: minibox HOSTNAME CMD [ARGS...]"); std::process::exit(2); }
    let argv: Vec<CString> = args[1..].iter().map(|a| CString::new(a.as_str()).expect("no NUL")).collect();

    // New UTS (hostname), PID, and mount namespaces. CLONE_NEWPID affects our CHILDREN:
    // the next process we fork becomes PID 1 of the new namespace.
    unshare(CloneFlags::CLONE_NEWUTS | CloneFlags::CLONE_NEWPID | CloneFlags::CLONE_NEWNS)
        .expect("unshare (are you root, in a --privileged container?)");

    // The child must not run the command until the parent has put it in the cgroup.
    // Otherwise the command can start (and fork its own children) before the limits apply.
    let (ready_r, ready_w) = pipe().expect("pipe");

    // SAFETY: the program is single-threaded at this point, so fork() is sound.
    match unsafe { fork() }.expect("fork") {
        ForkResult::Child => {
            drop(ready_w);
            let mut go = [0u8; 1];
            let _ = File::from(ready_r).read(&mut go);   // blocks until the parent writes (or exits: EOF)
            if let Err(e) = child(&args[0], &argv) { eprintln!("child setup failed: {e}"); std::process::exit(127); }
        }
        ForkResult::Parent { child } => {
            drop(ready_r);
            if let Err(e) = setup_cgroup(child.as_raw()) { eprintln!("cgroup limits skipped: {e}"); }
            let _ = File::from(ready_w).write_all(b"1"); // release the child
            match waitpid(child, None) {
                Ok(WaitStatus::Exited(_, code)) => std::process::exit(code),
                Ok(WaitStatus::Signaled(_, sig, _)) => std::process::exit(128 + sig as i32),
                other => { eprintln!("unexpected wait result: {other:?}"); std::process::exit(1) }
            }
        }
    }
}

# --cgroupns=host so the memory and pids controllers are delegated to new groups
docker run --rm -it --privileged --cgroupns=host -v "$PWD":/work -w /work rust:1 bash
cargo build --release --bin minibox
./target/release/minibox box1 sh -c 'hostname; echo "pid=$$"; ls /proc | grep -E "^[0-9]+$" | tr "\n" " "; cat /proc/self/cgroup'
# box1
# pid=1                    <- PID 1 in its own namespace
# 1 3 4 5                  <- only this namespace's processes are visible in the new /proc
# 0::/minibox              <- running inside the limited cgroup (because of the sync pipe)
hostname                   # the host's hostname is unchanged
cat /sys/fs/cgroup/minibox/memory.max   # 67108864

Without --cgroupns=host, Docker gives the container its own cgroup namespace in which the memory controller is not delegated to child groups. memory.max then does not exist, and the program prints cgroup limits skipped: /sys/fs/cgroup/minibox/memory.max: Permission denied. Real runtimes handle this by moving processes into a leaf group and enabling controllers in cgroup.subtree_control. The "no internal processes" rule of cgroup v2 requires this.

The pipe is not optional. The first version of this lab, written without it, printed the Docker cgroup instead of /minibox. The parent wrote the child's PID into cgroup.procs after fork, and by then the child had already started sh. A process that starts before its limits are applied can fork children outside them. runc solves this the same way, with a sync pipe between parent and child.

Why fork is unsafe: after fork in a multi-threaded program, the child contains only the calling thread. Any lock another thread held at that moment stays locked forever in the child, including the allocator's lock. Command uses posix_spawn or a careful fork+exec for that reason. Call fork yourself only in a single-threaded program, as here.

What this toy omits, and real runtimes (runc, youki, which is written in Rust) add: pivot_root into an image root filesystem, network namespaces with veth pairs (OS Ch 69), user namespaces, dropping capabilities, and seccomp filters (Security in Depth Ch 15).

44. A container in 100 lines: namespaces with nix

44. A container in 100 lines: namespaces with `nix`