22. I/O: io.Reader/Writer, bufio, and os

22.1 The two interfaces underneath everything

type Reader interface {
	Read(p []byte) (n int, err error)
}
type Writer interface {
	Write(p []byte) (n int, err error)
}

Files, network connections, in-memory buffers, gzip streams, TLS connections, and os.Stdin/os.Stdout all implement these two interfaces — which is why io.Copy(dst, src) works identically no matter what dst and src actually are.

22.2 bufio for line-oriented and buffered work

f, err := os.Open("access.log")
if err != nil {
	log.Fatal(err)
}
defer f.Close()

scanner := bufio.NewScanner(f)
for scanner.Scan() {
	line := scanner.Text()
	process(line)
}
if err := scanner.Err(); err != nil {
	log.Fatal(err)
}

bufio.Scanner's default buffer caps at 64KB per line — for genuinely huge lines, use bufio.Reader.ReadString('\n') or raise scanner.Buffer(...) explicitly.

22.3 Streaming instead of loading whole files

// BAD for a multi-GB file: os.ReadFile loads it all into memory
data, err := os.ReadFile("huge.log")
if err != nil {
	log.Fatal(err)
}
_ = data

// GOOD: stream line by line, constant memory regardless of file size
f, err := os.Open("huge.log")
if err != nil {
	log.Fatal(err)
}
defer f.Close()
scanner := bufio.NewScanner(f)
for scanner.Scan() {
	handleLine(scanner.Text())
}
if err := scanner.Err(); err != nil {
	log.Fatal(err)
}

22.4 Real-world example

Chapter 42's log analyzer processes multi-gigabyte files in constant memory using exactly this streaming pattern — the difference between os.ReadFile and bufio.Scanner is the difference between a tool that OOMs on a large input and one that doesn't care how large the input is.

Applied in the series: the TCP/IP guide's pcapread decodes binary capture files with bufio, io.ReadFull, and encoding/binary; the Linux guide's atomicwrite makes file writes durable and atomic.