Chapter 19 — Pipes, sockets, shared memory, and message queues

In one sentence

When two separate processes need to exchange data — and they can't just share memory the way threads do — the OS provides several distinct mechanisms (pipes, sockets, shared memory segments, message queues), each with a different trade-off between speed, simplicity, and how far apart the two ends can be.

Why this matters

| (the pipe operator) is the single most-used piece of Unix philosophy in existence, and understanding what it actually IS (not just how to use it) explains why Unix command-line tools compose so beautifully.

How it actually works

   PIPES: a one-directional, in-kernel byte stream connecting two
   processes' file descriptors (Chapter 23) -- one writes, the other
   reads, and the kernel buffers the data between them. This is EXACTLY
   what `|` does in your shell:

     ls -la | grep ".txt"
     the shell: creates a pipe, forks two children (Chapter 6), connects
     `ls`'s stdout to the pipe's write end and `grep`'s stdin to the
     pipe's read end. `ls` never needs to know `grep` exists, or vice
     versa -- they just read/write file descriptors, and the KERNEL
     moves the bytes between them.

   Named pipes (FIFOs) work the same way but have a filesystem path,
   so unrelated processes (not just parent/child) can connect to them.

   SOCKETS: like pipes, but bidirectional and, crucially, can connect
   processes on DIFFERENT MACHINES (network sockets, TCP/UDP) or on
   the SAME machine (Unix domain sockets -- faster than network sockets
   for local-only communication, since they skip the whole network
   stack).

   SHARED MEMORY: the fastest IPC mechanism, because it skips COPYING
   data through the kernel entirely -- two processes map the SAME
   physical memory pages (Chapter 10) into their own virtual address
   spaces, and can read/write it directly. Fastest, but the RISKIEST:
   now you need explicit synchronisation (Chapter 18's locks) between
   processes, not just threads, because the OS gives you NO protection
   against both writing at once.

   MESSAGE QUEUES: the kernel maintains a queue of discrete MESSAGES
   (not a raw byte stream like a pipe) that processes can send to and
   receive from, with some ordering/priority semantics -- less commonly
   used directly today than the other three, but still real and
   available (POSIX message queues, System V message queues).

Real-world example

   Docker's own architecture makes constant, direct use of a Unix
   domain socket: the Docker CLI you run (`docker ps`, `docker build`)
   talks to the Docker DAEMON over a Unix socket at
   `/var/run/docker.sock` by default (documented in Docker's own
   architecture docs) -- not a network socket, even though the
   communication LOOKS like a network API call (it's HTTP underneath).
   This is a deliberate choice, for two of this chapter's exact reasons:
   it's FASTER than a loopback TCP connection for same-machine
   communication, and it can use FILESYSTEM PERMISSIONS (Chapter 24) to
   control who's allowed to talk to the daemon at all -- which is
   exactly why "the docker.sock file's permissions" is a well-known,
   frequently-discussed real security consideration (anyone who can
   write to that socket can effectively control the whole Docker daemon,
   which usually means root-equivalent access to the host -- a fact
   documented extensively in container security guidance). You'll meet
   this exact socket again in Phase 3.

Try it yourself

# See pipes in action, explicitly
mkfifo /tmp/mypipe
(echo "hello through a named pipe" > /tmp/mypipe &)
cat /tmp/mypipe
rm /tmp/mypipe

# See the anonymous pipe your shell creates for `|`
strace -f -e trace=pipe,pipe2 bash -c 'echo hi | cat' 2>&1 | grep pipe

# See the Docker socket, if you have Docker installed
ls -la /var/run/docker.sock 2>/dev/null || echo "Docker not installed yet -- see Phase 3"

Build it in Go

Sockets are the IPC mechanism you'll use most. The TCP/IP guide's Go labs build TCP and UDP clients and servers from scratch. Chapter 22 there shows why a socket is a byte stream that needs message framing, the same lesson as this chapter's pipes.

Common mistakes

Assuming a pipe has unlimited buffer space. Pipes have a FIXED kernel buffer size (commonly 64KB on Linux); a writer can BLOCK if the reader isn't keeping up and the buffer fills — this is exactly how | creates natural backpressure between commands.
Using shared memory "because it's fastest" without proper synchronisation. The speed comes with zero built-in safety — you've opted into Chapter 17's and 18's exact problems, across process boundaries.
Forgetting Unix domain sockets are LOCAL-ONLY. They cannot connect across machines — reach for a network socket the moment you need that.

Check yourself

Name four IPC mechanisms and one key trade-off for each.
Why is a Unix domain socket typically faster than a TCP loopback connection for local-only communication?
Why does controlling who can write to /var/run/docker.sock matter so much for security?