53. Firewalls, stateful inspection, and conntrack

53.1 Packet filter generations

1. Stateless ACL — match per-packet on 5-tuple + flags. Fast, but can't tell a reply from an unsolicited packet, so you must open both directions and high-port ranges (fragile, leaky).
2. Stateful firewall — tracks connections. Allow NEW outbound, then ESTABLISHED,RELATED return traffic is permitted automatically. This is the default model everywhere now (netfilter/nftables, pf, iptables, cloud security groups).
3. NGFW / L7 firewall — adds app identification (DPI), TLS inspection (MITM with a corporate CA), IDS/IPS signatures, user identity.
4. WAF — HTTP-semantic rules (SQLi/XSS/path traversal, request smuggling, rate limits).

53.2 Linux conntrack — the state table

netfilter's conntrack assigns every flow a state entry (the same table NAT uses, Ch. 12). Per-entry it stores the tuple(s), a state, and a timeout.

TCP timeouts (defaults, nf_conntrack_tcp_timeout_*): syn_sent 120 s, established 432000 s (5 days), time_wait 120 s, close 10 s, fin_wait 120 s. UDP: nf_conntrack_udp_timeout 30 s, *_stream 120 s.

53.3 The failure modes

• nf_conntrack: table full, dropping packet — table hit nf_conntrack_max. New flows are silently dropped; existing ones fine. Causes: a scan/flood, a connection-churning app, too-long timeouts holding dead entries, or just growth. Fixes: raise nf_conntrack_max (and the hash hashsize = max/8), shorten established timeout for your workload, drop INVALID early, or NOTRACK high-volume stateless traffic (e.g. a DNS server's UDP/53) in the raw table.
• Idle-timeout drops (Ch. 34) — the firewall reaps a legit idle connection; the next packet is INVALID → RST or drop. Fix with keepalives shorter than the timeout.
• Asymmetric routing (Ch. 13) — the firewall sees only one direction, so the flow never reaches ESTABLISHED and return packets look INVALID. Fix routing or the flow dies.
• Conntrack + ECMP/HA — two firewalls in a path must sync conntrack state (conntrackd, or cloud stateful-LB) or a failover/rehash drops every existing flow.
• INVALID + -j DROP vs default ACCEPT — many rulesets forget to drop INVALID, letting crafted out-of-state ACK/RST packets through filters that only checked "not NEW."

53.4 A sane host firewall shape (nftables)

table inet filter {
  chain input {
    type filter hook input priority 0; policy drop;
    ct state established,related accept
    ct state invalid drop
    iif "lo" accept
    ip protocol icmp accept
    ip6 nexthdr ipv6-icmp accept          # DO NOT blanket-drop ICMPv6 (Ch. 14/17)
    tcp dport 22 ct state new limit rate 15/minute accept
    tcp dport { 80, 443 } accept
    # everything else: dropped by policy
  }
  chain forward { type filter hook forward priority 0; policy drop; }
  chain output  { type filter hook output  priority 0; policy accept; }
}

53.5 Cloud security groups vs NACLs

• Security Groups (AWS/GCP/Azure NSG): stateful, attached to an instance/ENI, allow-only (implicit deny), evaluated as a set. Return traffic auto-allowed.
• Network ACLs (AWS subnet-level): stateless, ordered rules, explicit allow and deny, and you must open ephemeral return ports (e.g. 1024-65535 inbound) because it won't track state.
• Both are distributed enforcement (no single choke point), so "connection table full" isn't your problem — but connection tracking on an NLB/NAT Gateway is: AWS NAT Gateway ~55 000 simultaneous connections per destination, and per-flow idle timeout 350 s.

53.6 War story

A batch platform intermittently failed to reach an internal API; curl hung then Connection reset. The firewall logged INVALID drops. Cause: the API's TCP established conntrack entry on the firewall timed out after a long quiet phase of a job (no keepalive), then the job's next request arrived on the dead entry → INVALID → drop, and a later packet → RST. Fix: TCP_KEEPIDLE=120 on the client (under the firewall's timeout) and shortening the job's inter-request gaps. Also added ct state invalid drop with logging so it was diagnosable next time. Lesson: stateful firewalls silently expire idle flows; long-lived clients need keepalives, and you should log INVALID drops.

Try it yourself

sudo sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
sudo conntrack -L | wc -l
sudo conntrack -L -p tcp --state ESTABLISHED | head
sudo conntrack -S                         # per-CPU: found, invalid, insert_failed, drop
dmesg | grep -i nf_conntrack
sudo nft list ruleset          # or: iptables -nvL --line-numbers  (watch the counters)
sysctl net.netfilter.nf_conntrack_tcp_timeout_established