Chapter 79 -- Reading /proc from Go: build your own ps
In one sentence
ps, top, free, and most monitoring agents are parsers for files under
/proc. Reading them yourself shows exactly what each number means and where
the parsing traps are.
Why this matters
When a container has no ps, when you need one number in a health check, or
when you're building an exporter or debugging an agent, you read /proc
directly. Chapter 23 showed the files from the shell. This chapter shows the
format details that break naive parsers.
How it actually works
| File | What it holds | Trap |
|---|---|---|
/proc/<pid>/stat |
one line: state, CPU ticks, threads, start time... | field 2 is (comm) and may contain spaces and parentheses: split after the last ) |
/proc/<pid>/status |
human-readable: VmRSS, Threads, voluntary_ctxt_switches |
values carry units (kB) |
/proc/<pid>/cmdline |
argv, NUL-separated | empty for kernel threads |
/proc/<pid>/fd/ |
one symlink per open descriptor | needs permission for other users' processes |
/proc/<pid>/limits |
effective rlimits | the truth about Chapter 78's limits |
CPU% requires two samples: (ticks_after − ticks_before) / USER_HZ / interval.
USER_HZ is 100 on practically every Linux system.
Build it in Go (30 min) -- gops
//go:build linux
// gops: `ps` and a little of `top`, written from /proc alone -- everything
// those tools show comes from these files.
//
// go run ./gops # top 10 processes by CPU over 1 second
package main
import (
"fmt"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"time"
)
type proc struct {
pid int
comm string
state string
threads int
rssKB int64
cpuTicks int64 // utime + stime, in clock ticks
cmdline string
}
// read parses /proc/<pid>/stat (one line, space-separated, but the command
// name is in parentheses and may itself contain spaces) and /proc/<pid>/status.
func read(pid int) (*proc, error) {
b, err := os.ReadFile(fmt.Sprintf("/proc/%d/stat", pid))
if err != nil {
return nil, err
}
s := string(b)
l, r := strings.IndexByte(s, '('), strings.LastIndexByte(s, ')')
f := strings.Fields(s[r+2:]) // fields after the comm: f[0] is field 3 ("state")
p := &proc{pid: pid, comm: s[l+1 : r], state: f[0]}
utime, _ := strconv.ParseInt(f[11], 10, 64) // field 14
stime, _ := strconv.ParseInt(f[12], 10, 64) // field 15
p.threads, _ = strconv.Atoi(f[17]) // field 20
p.cpuTicks = utime + stime
st, _ := os.ReadFile(fmt.Sprintf("/proc/%d/status", pid))
for _, line := range strings.Split(string(st), "\n") {
if strings.HasPrefix(line, "VmRSS:") {
p.rssKB, _ = strconv.ParseInt(strings.Fields(line)[1], 10, 64)
}
}
cl, _ := os.ReadFile(fmt.Sprintf("/proc/%d/cmdline", pid)) // NUL-separated argv
p.cmdline = strings.TrimSpace(strings.ReplaceAll(string(cl), "\x00", " "))
if p.cmdline == "" {
p.cmdline = "[" + p.comm + "]" // kernel threads have no cmdline
}
return p, nil
}
func snapshot() map[int]*proc {
out := map[int]*proc{}
dirs, _ := filepath.Glob("/proc/[0-9]*")
for _, d := range dirs {
pid, _ := strconv.Atoi(filepath.Base(d))
if p, err := read(pid); err == nil { // processes can vanish mid-scan: ignore
out[pid] = p
}
}
return out
}
func main() {
const hz = 100 // USER_HZ: clock ticks per second on virtually all Linux systems
interval := time.Second
before := snapshot()
time.Sleep(interval)
after := snapshot()
type row struct {
p *proc
cpu float64
}
var rows []row
for pid, p := range after {
var delta int64
if b, ok := before[pid]; ok {
delta = p.cpuTicks - b.cpuTicks
}
rows = append(rows, row{p, float64(delta) / hz / interval.Seconds() * 100})
}
sort.Slice(rows, func(i, j int) bool { return rows[i].cpu > rows[j].cpu })
states := map[string]int{}
for _, r := range rows {
states[r.p.state]++
}
fmt.Printf("%d processes; states: %v (R running, S sleeping, D disk wait, Z zombie)\n\n", len(rows), states)
fmt.Printf("%7s %-1s %6s %5s %9s %s\n", "PID", "S", "%CPU", "THR", "RSS(MB)", "COMMAND")
for _, r := range rows[:min(10, len(rows))] {
cmd := r.p.cmdline
if len(cmd) > 50 {
cmd = cmd[:50] + "..."
}
fmt.Printf("%7d %-1s %6.1f %5d %9.1f %s\n", r.p.pid, r.p.state, r.cpu, r.p.threads, float64(r.p.rssKB)/1024, cmd)
}
}
Real output (in a container with --pid=host, a busy loop running):
311 processes; states: map[I:126 R:2 S:183] (R running, S sleeping, D disk wait, Z zombie)
PID S %CPU THR RSS(MB) COMMAND
82877 R 101.0 1 0.5 sh -c (while :; do :; done) & sleep 0.5; /b/gops
34206 S 0.0 1 3.5 nginx: worker process
359 I 0.0 1 0.0 [kworker/11:1H-kblockd]
1114 S 0.0 1 14.6 postgres: checkpointer
295 S 0.0 20 88.9 /usr/bin/containerd --config /etc/containerd/conta...
What to notice:
I(idle) is a kernel-thread state for idle workers: 126 of them here, all in square brackets because kernel threads have no command line.- 101% CPU is one fully busy core, plus sampling error.
topwould show the same thing. - Processes vanish mid-scan. The code ignores read errors for that reason:
any
/procwalker must tolerate processes exiting between listing and reading.
Exercises:
- Add a
-p PIDmode that prints the process's open descriptors (resolving/proc/PID/fd/*symlinks) and its/proc/PID/limits. - Add a zombie report: every
Zprocess with its parent (field 4,ppid). Run it next to Chapter 74'spid1lab. - Export the numbers in Prometheus format (the TCP/IP guide's
proberlab shows how).
Common mistakes
- Splitting
/proc/<pid>/staton spaces from the start of the line. - Computing CPU% from a single sample.
- Treating a failed read as fatal during a
/procscan.
Check yourself
- Why must the
statparser split after the last)? - How do you compute a process's CPU% from
/proc? - Why do some processes have an empty
cmdline?
Further reading
man 5 proc, the authoritative field-by-field reference.- Chapter 23 of this guide.