Chapter 42 — `strace`, `lsof`, and answering "why is this stuck"

In one sentence

strace shows you EVERY syscall a process makes, in real time, and lsof shows you every file (in the broad Chapter 23 sense — files, sockets, devices) a process has open — together, they turn "this process is doing something mysterious" into "here's exactly what it's doing, right now."

Why this matters

When a process is hung and you have NO idea why, these two tools are almost always the fastest path from "no idea" to "found it" — genuinely one of the highest-leverage skill pairs in all of Linux troubleshooting.

How it actually works

   strace -p PID                 # attach to an ALREADY-RUNNING process
                                # and watch its syscalls live (Chapter
                                # 2) -- needs appropriate permissions
   strace command                 # run a NEW command under strace from
                                # the start
   strace -c command                # just a SUMMARY: which syscalls,
                                # how many times, how much time spent
   strace -f command                 # follow child processes too
                                # (Chapter 6's fork/exec)
   strace -e trace=open,read,write command  # filter to SPECIFIC
                                # syscalls, cutting the noise

   lsof -p PID                   # every file/socket/device THIS
                                # process has open
   lsof /path/to/file             # every process with THIS SPECIFIC
                                # file open -- "who's using this?"
   lsof -i :8080                    # what's listening on/using port
                                # 8080
   lsof +L1                         # DELETED files that are still
                                # open (Chapter 15's exact "deleted but
                                # not actually freed" scenario) --
                                # genuinely useful for finding disk
                                # space that `du` can't explain

A practical diagnostic sequence, when a process seems "stuck":

   1. ps -o pid,stat,cmd -p PID    check its STATE (Chapter 5) -- is
                                  it actually running (R), or stuck in
                                  D (uninterruptible I/O wait)?
   2. strace -p PID                 see the LAST syscall it made, and
                                  whether it's making progress at all,
                                  or genuinely blocked on one call
   3. lsof -p PID                    see WHAT it has open -- a file on
                                  a hung network mount? A socket to an
                                  unresponsive service?

Real-world example

   `lsof +L1` (or the closely related `lsof | grep deleted`) is a
   genuinely standard, widely-documented, real fix for a specific,
   common production mystery: "`df` shows the disk is almost full, but
   `du` can't find anything that adds up to that much space" -- exactly
   the scenario Chapter 15 described conceptually (a deleted file whose
   space isn't freed because a process still has it open), made
   PRACTICALLY SOLVABLE with this one command. This exact combination
   -- `df` disagreeing with `du`, then `lsof` revealing a large,
   DELETED-but-still-open log file held by a long-running process
   (often one that was never restarted after a log-rotation tool
   deleted/rotated its log file out from under it) -- is documented
   across countless sysadmin troubleshooting guides and forum threads
   as one of the standard "disk space mystery" playbooks, precisely
   because it recurs constantly across real production systems whenever
   log rotation (Chapter 34) and long-running processes interact badly.

Try it yourself

# See strace reveal exactly what a command does, syscall by syscall
strace -c ls / > /dev/null    # summary: which syscalls, how many
strace ls / 2>&1 | head -15    # the actual sequence, in order

# See lsof answer "who's using this port"
python3 -m http.server 8765 &
sleep 1
lsof -i :8765
kill %1

# Reproduce and then SOLVE Chapter 15's "deleted but not freed" mystery
python3 -c "
import time, os
f = open('/tmp/bigfile.tmp', 'w')
f.write('x' * 20_000_000)
f.flush()
os.remove('/tmp/bigfile.tmp')   # 'deleted' -- but still open
time.sleep(8)
" &
sleep 2
lsof +L1 2>/dev/null | grep bigfile   # there it is -- a deleted file
                                       # still consuming real disk space
wait

Common mistakes

Reaching for strace on a process handling large volumes of traffic without filtering. It can slow the process down noticeably and produce an overwhelming amount of output — use -e trace= to filter to what you actually care about.
Forgetting lsof needs elevated permissions to see OTHER users' processes. sudo lsof ... is often necessary outside your own processes.
Only checking du when disk space doesn't add up. du reports space used by files STILL VISIBLE in the directory tree — it can't see deleted- but-open files at all; that's exactly where lsof +L1 comes in.

Check yourself

What does strace show you, and what does lsof show you — how are they complementary?
What's the standard three-step sequence for diagnosing a "stuck" process?
Why can df and du disagree about how much disk space is actually used, and what tool resolves the mystery?

Chapter 42 — strace, lsof, and answering "why is this stuck"