Chapter 50 — Logging, detection, and monitoring

What to log (security events)

   AUTHENTICATION   login success/failure (+ method, MFA used), logout,
                    password/'MFA change, token issue/refresh/revoke,
                    session creation, step-up auth. Include: user id,
                    source IP, user-agent, geo (derived), timestamp (UTC,
                    ISO 8601), request id.
   AUTHORIZATION    access-control DENIALS (esp.), privilege changes,
                    role/permission grants, admin actions, impersonation.
   INPUT VALIDATION rejections that look like attacks (SQLi/XSS patterns,
                    path traversal), WAF blocks.
   ACCOUNT LIFECYCLE creation, deletion, lockout, recovery, email/phone
                    change.
   DATA ACCESS       reads/exports of sensitive records (who, which, how
                    many), bulk operations, report generation.
   CONFIG / DEPLOY   feature-flag flips, config changes, deploys, IAM/
                    firewall/DNS changes, new API keys.
   INTEGRITY         signature verification failures, checksum mismatches,
                    unexpected file changes (FIM).
   SYSTEM            process start/stop, new listening ports, new cron,
                    outbound connections to new destinations, container
                    escapes/priv-esc signals.

What NOT to log

   * passwords, session tokens, API keys, full JWTs, MFA codes/secrets
   * full card numbers (PCI), CVV EVER, raw SSNs / national IDs
   * full request/response bodies on sensitive endpoints
   * more personal data than you need (GDPR data-minimisation; logs are
     a data store with retention obligations and breach exposure)
   -> REDACT at the logging boundary (allowlist fields, hash identifiers
      where you only need correlation, mask PANs to first6+last4).
   -> and beware LOG INJECTION (Ch 42): encode newlines/control chars so
      an attacker can't forge log lines or break your parser.

The pipeline

   app / host / network / cloud
        | structured logs (JSON), one event per line, stable schema
        v
   SHIP        Fluent Bit / Vector / journald+rsyslog / cloud agent
        | over TLS, buffered, with backpressure
        v
   CENTRAL STORE (append-only, access-controlled, time-synced via NTP)
        Elastic/OpenSearch, Loki, Splunk, cloud-native (CloudWatch,
        Cloud Logging), or a data lake + query engine
        |
        +--> SIEM / detections: correlation rules, threshold alerts,
        |      anomaly detection, threat-intel matching
        |      (Elastic Security, Splunk ES, Sentinel, Panther, Wazuh,
        |       Sigma rules as portable detection-as-code)
        +--> DASHBOARDS + scheduled reports
        +--> long-term cold storage for the retention window

Key properties: central (an attacker on one host can't erase the evidence), append-only / integrity-protected (WORM storage, or hash-chained), time- synced (correlation across systems needs trustworthy timestamps), retained long enough (dwell time is often months — 90 days is a floor, 1 year better).

Alerting: signal, not noise

   Alert on things that are ACTIONABLE and mean "attack in progress":
     - impossible travel / login from new country + MFA reset
     - spike in 401/403 from one source (enumeration / cred stuffing)
     - a single account hitting many object IDs (IDOR probing)
     - auth success after many failures (brute force succeeded)
     - admin action outside change windows / by a dormant account
     - outbound traffic to a new IP from a prod box (C2 / exfil)
     - your canary token used (Ch 49)
     - signature/integrity verification failure in the deploy path
     - disabling of logging/security tooling

   Every alert needs a RUNBOOK (what to check, how to triage, when to
   escalate). An alert with no runbook gets muted, then ignored.
   Tune ruthlessly: a channel that's 95% false positives is blind.

Detection engineering basics

   * Map detections to MITRE ATT&CK techniques -> know your coverage gaps.
   * Detection-as-code: rules in git, peer-reviewed, tested against
     recorded attack data, CI-linted (Sigma, Elastic detection rules).
   * Purple-team: run a known technique (Atomic Red Team, Caldera) ->
     did it fire? tune -> repeat.
   * Track metrics: MTTD (mean time to detect), MTTR (respond),
     alert precision, ATT&CK coverage.

# 1. Structured security logging in a tiny app (concept)
python3 - <<'PY'
import json, datetime, hashlib
def audit(event, **fields):
    rec = {"ts": datetime.datetime.now(datetime.UTC).isoformat(),
           "event": event, **fields}
    # redact: never log the token; log a short hash for correlation
    if "token" in rec:
        rec["token_fp"] = hashlib.sha256(rec.pop("token").encode()).hexdigest()[:12]
    # neutralise log injection
    line = json.dumps(rec).replace("\n", "\\n")
    print(line)
audit("auth.login.fail", user="alice", src_ip="203.0.113.9",
      ua="curl/8", reason="bad_password", request_id="req-abc123")
audit("authz.deny", user="bob", src_ip="198.51.100.4",
      object="orders/1002", action="read", request_id="req-def456")
audit("token.issue", user="alice", token="super-secret-value")
PY

# 2. Stand up a log stack and a detection (docker)
#    Loki + Grafana + Promtail, or the Elastic one-liner:
#    docker run -d --name es -p 9200:9200 -e discovery.type=single-node \
#      -e xpack.security.enabled=false docker.elastic.co/elasticsearch/elasticsearch:8.15.0

# 3. Detect brute force from auth logs with a threshold query (jq demo)
cat > auth.log <<'EOF'
{"ts":"2026-09-10T10:00:01Z","event":"auth.login.fail","user":"a","src_ip":"5.5.5.5"}
{"ts":"2026-09-10T10:00:02Z","event":"auth.login.fail","user":"b","src_ip":"5.5.5.5"}
{"ts":"2026-09-10T10:00:03Z","event":"auth.login.fail","user":"c","src_ip":"5.5.5.5"}
{"ts":"2026-09-10T10:00:04Z","event":"auth.login.fail","user":"d","src_ip":"5.5.5.5"}
{"ts":"2026-09-10T10:00:05Z","event":"auth.login.fail","user":"e","src_ip":"5.5.5.5"}
{"ts":"2026-09-10T10:00:06Z","event":"auth.login.ok","user":"e","src_ip":"5.5.5.5"}
EOF
jq -s 'group_by(.src_ip)[] | {ip: .[0].src_ip,
   fails: (map(select(.event=="auth.login.fail"))|length),
   then_success: (any(.event=="auth.login.ok"))}
   | select(.fails >= 5 and .then_success)
   | "ALERT: brute force likely succeeded from \(.ip) (\(.fails) fails)"' auth.log

# 4. Explore a real detection ruleset
git clone --depth 1 https://github.com/SigmaHQ/sigma
ls sigma/rules/web/ sigma/rules/application/    # read a few .yml rules

Mini-project (60 min): SecureShop detection baseline

   1. Define SecureShop's security log schema (JSON fields, event names)
      covering the "what to log" list; implement `audit()` in the app.
   2. Ship logs to a local Loki/Elastic; confirm they're queryable and
      that the app host cannot delete central copies.
   3. Write 5 detections (as Sigma rules or stack queries) with runbooks:
      credential stuffing, IDOR enumeration, admin action by dormant
      account, JWT signature failures, outbound to new IP from app tier.
   4. Simulate each (a script that generates the matching log pattern);
      confirm the alert fires; record MTTD.
   5. Redaction test: attempt to log a token/password/PAN; prove it's
      masked. Log-injection test: user field = "x\n{fake event}"; prove
      it's neutralised.
   6. Report -> ~/sec-lab/reports/part11-detection.md (schema, rules,
      simulation results, coverage vs ATT&CK).