9. Debugging containers

9.1 Exit codes you must recognize

Code	Meaning	Typical cause
0	clean exit	a "server" that exits 0 means it thought it was done (wrong CMD, batch mode)
1	app error	panic, unhandled exception, bad config (check logs)
125	Docker itself failed	bad `docker run` flag
126	command not executable	missing `+x`, wrong architecture binary
127	command not found	typo in ENTRYPOINT, missing shell in distroless (`sh -c` on distroless!)
137	128 + 9 = SIGKILL	OOM killed (check `OOMKilled: true`) or killed after the stop grace period
139	128 + 11 = SIGSEGV	native crash, CGO, wrong libc (glibc binary on Alpine/musl)
143	128 + 15 = SIGTERM	stopped normally by `docker stop`/Kubernetes

docker inspect orders --format '{{.State.ExitCode}} OOMKilled={{.State.OOMKilled}} {{.State.Error}}'

9.2 The debugging ladder

docker logs --tail 200 orders                  # 1. what did it say before dying?
docker inspect orders | jq '.[0].State, .[0].Config.Env, .[0].Mounts'   # 2. config as actually applied
docker events --since 10m --filter container=orders   # 3. die/oom/restart events
docker stats --no-stream                       # 4. resources
docker exec -it orders sh                      # 5. inside (not possible on distroless...)

# 6. ...so attach a toolbox that shares the target's namespaces
docker run --rm -it --pid container:orders --network container:orders nicolaka/netshoot
#   ps aux, ls /proc/1/root/app, ss -tnp, curl localhost:8080/readyz

# 7. container exits too fast to exec? override the entrypoint
docker run --rm -it --entrypoint sh orbit-orders:debug     # with a debug image variant
# 8. or snapshot its filesystem
docker create --name snap orbit-orders:0.1.0 && docker export snap | tar -t | head

Best practice: build a :debug variant only when you need it (gcr.io/distroless/static-debian12:debug-nonroot has busybox). Don't ship shells in production images "just in case". §33.3 shows ephemeral containers, which make that unnecessary.

9.3 "Works locally, fails in the container" checklist

Binding to 127.0.0.1 inside the container. Bind to 0.0.0.0 (or :8080 in Go) or the port publish can't reach it.
localhost used to reach another container.
Architecture mismatch: an arm64 Mac building images for amd64 clusters (exec format error). Build with --platform.
Missing CA certificates (x509: certificate signed by unknown authority) on scratch.
Timezone or locale assumptions (tzdata missing).
File permissions with a non-root user.
App writes to a read-only root FS. Give it a tmpfs/emptyDir for that path.