Part 8 — Detection engineering and response at scale
G1 Ch 50-51 taught you what to log and the incident-response lifecycle. Part 1 Ch 2 taught you to validate detections by emulating real techniques. This Part makes detection a discipline — a lifecycle with hypotheses, tests, and metrics — scales the telemetry underneath it, adds threat hunting and intelligence on top, and extends incident response to the ephemeral, credential-centric world of cloud and containers, finishing with the threat that makes all of it matter most: ransomware.
Chapter 51 — The detection-engineering lifecycle
In one sentence
A detection is not "done" when a rule fires once in testing — it's a hypothesis about attacker behaviour that goes through design, implementation, validation against real attack emulation, tuning, and measured maintenance, the same discipline you'd apply to any production system.
Where we are
You've written detections ad hoc through this guide (Part 2 Ch 10, Part 4 Ch 22, Part 6 Ch 40). This chapter is the process that makes detection work sustainable across hundreds of rules and a changing environment, instead of a pile of unmaintained alerts nobody trusts.
How it works
The Pyramid of Pain (David Bianco) — target detections at what's expensive for the attacker to change:
TTPs (top -- HARDEST to change; forces the attacker
to rebuild their whole approach)
Tools
Network/Host Artifacts
Domain Names
IP Addresses
Hash Values (bottom -- TRIVIAL to change; one recompile)
A detection on a file HASH is defeated by recompiling the malware.
A detection on the TECHNIQUE ("a process spawned from an office
document reading LSASS memory", or "IMDSv1 metadata access followed by
an out-of-region API call") survives tool and infrastructure changes.
Spend your limited engineering time near the TOP of the pyramid; use
IOC feeds (bottom) as cheap, disposable, short-lived tripwires, not
your primary strategy.
The lifecycle (an Alerting and Detection Strategy, per detection):
1. IDENTIFY where does this detection idea come from? Threat
intel (Ch 53), an ATT&CK-technique gap in your coverage heat map
(Part 1 Ch 2), an attack-path chokepoint (Part 1 Ch 1, Ch 54), or a
real incident's root cause (the best source -- "this actually
happened to us").
2. DESIGN THE HYPOTHESIS write it as a sentence: "IAM principals
outside the CI role group calling iam:CreatePolicyVersion followed
within 5 minutes by iam:SetDefaultPolicyVersion indicates privilege-
escalation attempt (Part 2 Ch 6)." State the ATT&CK technique, the
data source needed, the expected false-positive sources, and the
priority/response if it fires.
3. DEVELOP the rule. Prefer a PORTABLE format (Sigma, below) over a
vendor-proprietary query when you can, so it survives a SIEM
migration and can be shared/reviewed as code.
4. TEST -- two kinds, both required:
UNIT TESTS: run the rule against a library of SAMPLE LOG
fixtures -- both POSITIVE (should fire) and NEGATIVE (common
legitimate patterns that should NOT fire) cases, in CI, on
every change to the rule.
LIVE VALIDATION: actually EXECUTE the technique (Atomic Red Team /
Stratus Red Team / CALDERA, Part 1 Ch 2) in a lab or sanctioned
window and confirm the rule fires against REAL telemetry, not
just a hand-crafted fixture.
5. DEPLOY through the same review + CI/CD pipeline as any other code
(Part 3) -- a detection rule is production code with the same
change-management discipline.
6. TUNE run for a burn-in period; track false-positive rate; add
exclusions for legitimate patterns found in step 4/6 WITHOUT
widening the rule so much it stops detecting the real technique
(document every exclusion with a reason and an owner).
7. MEASURE track PRECISION (of alerts fired, how many were real),
a recall PROXY (did it fire during every subsequent purple-team run
of the technique, Part 1 Ch 2), and MTTD contribution.
8. MAINTAIN environments change -- a log schema update, a new cloud
service, a refactored microservice can silently break a rule (it
stops matching, or starts matching everything). Re-validate rules
periodically with live re-emulation, not just "it hasn't alerted
recently" (which could mean "working" or "silently broken" --
indistinguishable without re-testing).
Detection-as-code, concretely — Sigma as the portable format:
title: AWS IAM Privilege Escalation via Policy Version Manipulation
status: stable
logsource: { product: aws, service: cloudtrail }
detection:
selection1:
eventName: CreatePolicyVersion
requestParameters.setAsDefault: true
selection2:
eventName: SetDefaultPolicyVersion
timeframe: 5m
condition: selection1 and selection2 | count() by userIdentity.arn > 0
falsepositives:
- Legitimate policy rollback by an authorized platform-team pipeline
level: high
tags: [attack.privilege_escalation, attack.t1078.004]
`sigma-cli` (or `pySigma` backends) COMPILES this ONE rule into native
queries for Splunk SPL, Elastic EQL/KQL, Microsoft Sentinel KQL, and
more -- write once, deploy to whatever SIEM you run today, and migrate
without rewriting the whole rule library later.
Detection Maturity and where most teams actually are (David Bianco's Hunting Maturity Model, adapted to detection generally):
HM0 no proactive detection; purely reactive to vendor/EDR alerts.
HM1 MINIMAL -- incorporates external threat intel as IOC matches.
HM2 PROCEDURAL -- follows other people's published detection
procedures (Sigma community rules, vendor content).
HM3 INNOVATIVE -- creates novel detections for YOUR environment's
actual attack paths (Ch 54) and threat model.
HM4 LEADING -- most new detections are AUTOMATED from repeatable
hunting procedures (Ch 53) with minimal manual rule-writing.
Most organisations should aim to be solidly HM2-3: use community
content as a baseline (don't reinvent well-known detections), but
build HM3-level custom detections for your specific attack-path
chokepoints (Ch 54) -- that's where generic content has nothing to say.
Worked example
Taking one Part 4 lab finding through the full lifecycle.
SOURCE: Part 4 Ch 22's lab showed an RBAC escalation via `create pods`
with a chosen `serviceAccountName` reaching a more-privileged SA.
1. IDENTIFY: ATT&CK T1078.004 (Valid Accounts: Cloud/Container
Accounts) + T1611 (Escape to Host) territory; sourced from our own
lab finding, the strongest kind of source.
2. HYPOTHESIS: "A pod is created specifying a serviceAccountName that
the CREATING identity does not itself hold (i.e. is more privileged
than the creator's own SA), outside a known deployment pipeline
identity, indicates a possible RBAC-escalation attempt."
3. DEVELOP: a k8s-audit-log-based Sigma rule matching `pods.create`
events where `requestObject.spec.serviceAccountName` != the
requesting `user.username`'s own default SA pattern AND the
requesting identity isn't the CI/CD deploy identity.
4. TEST: unit-test against a fixture of normal Helm/Argo CD deploy
audit events (should NOT fire) and a fixture of the actual lab
escalation audit event (SHOULD fire). Then re-run the Ch 22 lab
attack live against a cluster with the rule deployed -- confirm it
fires within the audit-log ingestion latency.
5. DEPLOY via the same PR + CI pipeline as the cluster's other policies
(Part 3), reviewed by a second engineer.
6. TUNE: burn in for two weeks; discover a legitimate operator that
creates helper pods with a different SA as part of normal
reconciliation -- add a scoped exclusion for THAT specific
operator's identity, with a comment and an owner, not a blanket
namespace exclusion.
7. MEASURE: over the next quarter, the rule fires 3 times, all 3
during scheduled purple-team re-runs of the Ch 22 technique (100%
recall against known re-emulation) and zero unexplained false
positives after tuning (precision improving).
8. MAINTAIN: six months later, the cluster's admission-webhook version
changes the audit log's field structure slightly -- the quarterly
re-emulation run catches that the rule SILENTLY STOPPED MATCHING
before any real incident would have relied on it. Fixed and
re-validated the same day.
Practice (75 min)
- Pick one real finding from an earlier Part's hands-on lab (cloud privesc, k8s escalation, a web-app chain). Write its full Alerting and Detection Strategy: hypothesis, ATT&CK mapping, data source, expected false positives, priority, response.
- Write it as a Sigma rule. Build two log fixtures (one that should match, one realistic-but-benign that shouldn't) and a unit test that asserts both.
- Compile the Sigma rule to your actual SIEM/log-query backend with
sigma-cli; deploy it. - Live-validate: re-run the original attack technique (Atomic Red Team / Stratus Red Team / your own lab repro) and confirm the deployed rule fires against real telemetry, recording latency.
- Deliberately introduce a schema change in your test log source (rename a field) and confirm the rule silently stops matching — this is the maintenance failure mode to design monitoring against (e.g. a "rule hasn't fired AND hasn't been re-validated in N days" meta-alert).
- Start (or add to) your ATT&CK coverage heat map (Part 1 Ch 2) with this detection's technique marked covered, with a link to its Sigma rule and its last live-validation date.
Build it in Go (40 min) — detections as code, with tests that gate deployment
The lifecycle above says detections are code: versioned, reviewed, tested, deployed. This lab makes that literal. Each rule is a Go type (including a stateful one), each ships with fixtures it must and must not fire on, and the program refuses to run if any rule fails its own tests.
// detect: detection-as-code (Chapter 51). Rules are Go functions over a stream
// of JSON events; every rule ships with fixtures it MUST fire on and MUST NOT
// fire on, and the program refuses to run if any rule fails its own tests --
// exactly the gate a CI pipeline for detections enforces.
//
// go run ./detect < events.jsonl (or with no input: runs the built-in demo stream)
package main
import (
"bufio"
"encoding/json"
"fmt"
"os"
"strings"
"time"
)
type event struct {
Time time.Time `json:"time"`
Action string `json:"action"` // login, kms:Decrypt, iam:CreateAccessKey, ...
Actor string `json:"actor"`
SrcIP string `json:"src_ip"`
Outcome string `json:"outcome"` // success | failure | denied
Context map[string]string `json:"context"`
}
type alert struct{ rule, severity, detail string }
// A rule sees events in time order and may keep state between them.
type rule interface {
name() string
observe(e event) *alert
}
// ---------------------------------------------------------------- rules --
// bruteThenSuccess: >= 5 failed logins for one actor within 2 minutes,
// followed by a success -- a guessed password, not just noise.
type bruteThenSuccess struct{ fails map[string][]time.Time }
func (r *bruteThenSuccess) name() string { return "brute-force-then-success" }
func (r *bruteThenSuccess) observe(e event) *alert {
if e.Action != "login" {
return nil
}
if r.fails == nil {
r.fails = map[string][]time.Time{}
}
if e.Outcome == "failure" {
r.fails[e.Actor] = append(r.fails[e.Actor], e.Time)
return nil
}
recent := 0
for _, t := range r.fails[e.Actor] {
if e.Time.Sub(t) <= 2*time.Minute {
recent++
}
}
delete(r.fails, e.Actor)
if recent >= 5 {
return &alert{r.name(), "high", fmt.Sprintf("%s logged in from %s after %d failures in 2m", e.Actor, e.SrcIP, recent)}
}
return nil
}
// crossTenantKey: a KMS decrypt DENIED because the caller tried another
// tenant's key (Chapter 9's worked example). Almost never benign.
type crossTenantKey struct{}
func (crossTenantKey) name() string { return "cross-tenant-key-access" }
func (crossTenantKey) observe(e event) *alert {
if e.Action == "kms:Decrypt" && e.Outcome == "denied" && e.Context["tenant"] != "" {
return &alert{"cross-tenant-key-access", "high",
fmt.Sprintf("%s was denied tenant %s's key", e.Actor, e.Context["tenant"])}
}
return nil
}
// newKeyForHuman: long-lived access keys created for a human user (policy:
// humans use SSO). Low severity on its own; high in an attack-path chain.
type newKeyForHuman struct{}
func (newKeyForHuman) name() string { return "access-key-for-human" }
func (newKeyForHuman) observe(e event) *alert {
if e.Action == "iam:CreateAccessKey" && e.Outcome == "success" && strings.HasPrefix(e.Context["user"], "human/") {
return &alert{"access-key-for-human", "medium", fmt.Sprintf("%s created a key for %s", e.Actor, e.Context["user"])}
}
return nil
}
// ------------------------------------------------- fixtures: rule tests --
type fixture struct {
rule func() rule
events []event
want bool // must the LAST event fire?
}
func at(min int) time.Time { return time.Date(2026, 10, 3, 9, min, 0, 0, time.UTC) }
func login(min int, actor, outcome string) event {
return event{Time: at(min), Action: "login", Actor: actor, SrcIP: "203.0.113.7", Outcome: outcome}
}
var fixtures = []fixture{
{func() rule { return &bruteThenSuccess{} }, append(repeat(login(0, "alice", "failure"), 5), login(1, "alice", "success")), true},
{func() rule { return &bruteThenSuccess{} }, append(repeat(login(0, "alice", "failure"), 2), login(1, "alice", "success")), false},
{func() rule { return &bruteThenSuccess{} }, append(repeat(login(0, "alice", "failure"), 5), login(9, "alice", "success")), false}, // too slow
{func() rule { return crossTenantKey{} }, []event{{Action: "kms:Decrypt", Outcome: "denied", Context: map[string]string{"tenant": "99"}}}, true},
{func() rule { return crossTenantKey{} }, []event{{Action: "kms:Decrypt", Outcome: "success", Context: map[string]string{"tenant": "99"}}}, false},
{func() rule { return newKeyForHuman{} }, []event{{Action: "iam:CreateAccessKey", Outcome: "success", Context: map[string]string{"user": "human/bob"}}}, true},
{func() rule { return newKeyForHuman{} }, []event{{Action: "iam:CreateAccessKey", Outcome: "success", Context: map[string]string{"user": "svc/ci"}}}, false},
}
func repeat(e event, n int) []event {
out := make([]event, n)
for i := range out {
out[i] = e
}
return out
}
func testRules() bool {
ok := true
for i, f := range fixtures {
r := f.rule()
var last *alert
for _, e := range f.events {
last = r.observe(e)
}
if (last != nil) != f.want {
fmt.Printf("FAIL fixture %d (%s): want fire=%v\n", i, r.name(), f.want)
ok = false
}
}
fmt.Printf("rule tests: %d fixtures, all passed=%v\n\n", len(fixtures), ok)
return ok
}
// ---------------------------------------------------------------- main --
func main() {
if !testRules() {
os.Exit(1) // never deploy a detection that fails its own fixtures
}
rules := []rule{&bruteThenSuccess{}, crossTenantKey{}, newKeyForHuman{}}
var events []event
if st, _ := os.Stdin.Stat(); st.Mode()&os.ModeCharDevice == 0 {
sc := bufio.NewScanner(os.Stdin)
for sc.Scan() {
var e event
if json.Unmarshal(sc.Bytes(), &e) == nil {
events = append(events, e)
}
}
} else { // a small attack story: guessing, a foothold, persistence, then data
events = append(repeat(login(0, "carol", "failure"), 6), login(1, "carol", "success"),
event{Time: at(3), Action: "iam:CreateAccessKey", Actor: "carol", Outcome: "success", Context: map[string]string{"user": "human/carol"}},
event{Time: at(5), Action: "kms:Decrypt", Actor: "carol", Outcome: "denied", Context: map[string]string{"tenant": "42"}},
event{Time: at(6), Action: "login", Actor: "dave", Outcome: "success"})
}
for _, e := range events {
for _, r := range rules {
if a := r.observe(e); a != nil {
fmt.Printf("%s [%-6s] %-26s %s\n", e.Time.Format("15:04"), a.severity, a.rule, a.detail)
}
}
}
}
$ go run ./detect
rule tests: 7 fixtures, all passed=true
09:01 [high ] brute-force-then-success carol logged in from 203.0.113.7 after 6 failures in 2m
09:03 [medium] access-key-for-human carol created a key for human/carol
09:05 [high ] cross-tenant-key-access carol was denied tenant 42's key
$ echo '{"time":"2026-10-03T10:00:00Z","action":"kms:Decrypt","actor":"analytics-role","outcome":"denied","context":{"tenant":"99"}}' | go run ./detect
rule tests: 7 fixtures, all passed=true
10:00 [high ] cross-tenant-key-access analytics-role was denied tenant 99's key
What to notice:
- The fixtures encode the edge cases, such as "5 failures, then success 8 minutes later" must not fire. Analysts argue about thresholds in fixtures, in code review, not in production pages.
- The demo stream reads as a story: credential guessing → persistence (a long-lived key) → probing data (a denied cross-tenant key). Three medium-to-high alerts in five minutes on one actor is the correlation Chapter 54 asks for.
- The second command is the Chapter 9 worked example, firing on real JSON.
Ship your CloudTrail, audit-log, or SIEM export to
stdinin the same shape.
Exercises:
- Add an impossible-travel rule (two logins for one user, with geo-IP distance ÷ time > 1,000 km/h) together with its fixtures.
- Add a correlation rule that raises one critical alert when one actor triggers three different rules within 15 minutes.
- Run the fixtures in CI as
go testand block merges that break them.
Common mistakes
- Writing detections only from vendor/community content. Necessary baseline, insufficient — your specific attack-path chokepoints (Ch 54) need custom detections nobody else can publish for you.
- "It fired once in testing" treated as validated. Unit tests catch regressions; only live emulation against real telemetry proves the rule actually works in your environment.
- Detections with no owner, no false-positive documentation, no last- validated date. They rot silently and nobody notices until an incident reveals the gap.
- Chasing hashes/IPs as a primary strategy. Cheap to produce, cheap for the attacker to invalidate. Spend engineering effort near the top of the pyramid.
- No unit tests for detection rules. Treating detection content as "just YAML" rather than code that needs the same CI discipline as anything else.
- Never re-validating after environment changes. A schema change, a new service, a refactor can silently break a rule; only re-emulation reveals it.
Check yourself
- What is the Pyramid of Pain, and why should most engineering effort target its top rather than its bottom?
- Name the eight stages of the detection-engineering lifecycle in this chapter.
- What are the two distinct kinds of "testing" a detection rule needs, and what does each catch that the other doesn't?
- Why write detections in Sigma rather than directly in your SIEM's native query language?
- What is the Hunting/Detection Maturity Model's HM3 level, and why can't community content alone get you there?
- Why can a detection "silently break" and how do you catch that before an incident does?
(Answers: Appendix G.)
Further reading
- Framework: Palantir's "Alerting and Detection Strategy Framework" (public write-up and template — the ADS document structure used above).
- Model: David Bianco, "The Pyramid of Pain" (2013, still the essential read) and "The Hunting Maturity Model".
- Project: SigmaHQ (
github.com/SigmaHQ/sigma),pySigmabackends,sigma-cli— read the rule-writing guide and the existing rule corpus. - Book: Detection Engineering — emerging practitioner books and the "Detection Engineering Maturity Matrix" (DEMM) community content.