Chapter 52 — Telemetry architecture

In one sentence

Detection is only as good as the telemetry underneath it, so the pipeline — what you collect, how you normalise it, where it's stored, and at what retention and cost — is itself a system you design deliberately, not an afterthought of "turn on all the logs."

Where we are

G1 Ch 50 covered what to log at the application layer. This chapter is the full-stack pipeline: every source across cloud, containers, network, identity, and SaaS, normalised into a common schema, and stored in a way that balances detection latency, hunting depth, and cost.

How it works

Sources, layered (recap and extend earlier Parts):

   ENDPOINT      EDR agents (CrowdStrike, SentinelOne, Microsoft Defender
                 for Endpoint) or open-source (osquery, Wazuh) --
                 process, file, and network events on hosts/workstations.
   CONTAINER/K8S Falco/Tetragon (Part 4 Ch 20) + Kubernetes AUDIT LOGS
                 (the API server's own record of every request -- distinct
                 from and complementary to runtime syscall telemetry).
   NETWORK       VPC/Flow Logs (Part 2 Ch 8), Zeek (rich protocol-level
                 metadata), packet capture at chokepoints for deep
                 investigation (not everywhere -- volume/cost).
   CLOUD CONTROL PLANE   CloudTrail / Cloud Audit Logs / Azure Activity
                 Log (Part 2 Ch 5) -- your PRIMARY timeline source for
                 anything identity/API-driven, including most of Part 2
                 and Part 3's attack classes.
   APPLICATION   structured security events (G1 Ch 50).
   IDENTITY      IdP sign-in logs, MFA challenge results, conditional-
                 access/risk decisions (Ch 63) -- often the FIRST signal
                 of account compromise, before any downstream system sees
                 anything.
   DNS           query logs -- cheap, high-signal for C2 beaconing,
                 tunnelling (G1 Ch 38), and DGA domains.
   EMAIL         phishing/malware detections, DLP hits (Ch 49).
   SAAS          increasingly essential and increasingly neglected: your
                 CI/CD platform's audit log (Part 3 Ch 12), your identity
                 provider's admin actions, your ticketing/wiki/chat
                 platforms' access logs. "SaaS Security Posture
                 Management" (SSPM) tools exist specifically because this
                 category is otherwise invisible.

The pipeline:

   AGENTS/COLLECTORS   Fluent Bit, Vector, the OpenTelemetry Collector --
     lightweight, ship over TLS, buffer through backpressure so a
     downstream outage doesn't silently drop events.
        |
   NORMALISATION   map every source into a COMMON SCHEMA so a detection
     rule written once works across sources and survives a tool swap.
     OCSF (Open Cybersecurity Schema Framework -- an open, vendor-backed
     standard with broad and growing adoption) or ECS (Elastic Common
     Schema) are the two real choices today; pick one and normalise at
     ingest, not per-query.
        |
   ENRICHMENT   add context the raw event doesn't carry: threat-intel
     matches (Ch 53), asset criticality/owner (from Ch 41's inventory),
     geolocation, and identity resolution (map a service-account ARN or a
     SPIFFE ID, Part 5 Ch 23, back to a human-readable owner).
        |
   ROUTING   not every event needs the same treatment. Split by VALUE:
     high-signal, low-volume sources (cloud control plane, identity) ->
     hot storage + real-time detection. High-volume, lower-per-event-
     value sources (raw network flow, verbose debug logs) -> a cheaper
     tier, sampled or filtered, still available for hunting but not
     driving real-time alerting on every event.
        |
   STORAGE, TIERED:
     HOT   the SIEM's own indexed storage -- fast queries, real-time
           correlation, expensive per GB. Weeks to a few months.
     WARM/COLD   a SECURITY DATA LAKE -- object storage (S3/GCS) in an
           open format (Parquet, OCSF-shaped) queried by a separate,
           cheaper engine (Athena, BigQuery, Snowflake) when a hunt or an
           investigation needs history the hot tier no longer holds.
           DECOUPLING the data lake from any single SIEM vendor's
           proprietary ingestion is an increasingly important
           architectural choice -- it avoids being cost-locked into one
           tool's per-GB pricing for data you're mostly not querying in
           real time, and it lets you point MULTIPLE analytics tools (a
           SIEM, a hunting notebook, an ML pipeline) at the same
           authoritative copy.
        |
   DETECTION/ANALYTICS LAYER   the SIEM's correlation engine, a streaming
     processor, or a scheduled query job -- this is where the rules from
     Ch 51 actually run, against whichever tier holds the relevant window.

Retention — driven by both compliance and reality:

   COMPLIANCE FLOORS: PCI-DSS requires at least a year of log retention
   with the most recent 3 months immediately available for analysis;
   other frameworks vary (Ch 61).
   REALITY CHECK: real breach dwell time (attacker-in-environment before
   detection) has AVERAGED WELL BEYOND 90 DAYS in industry reporting for
   years (Mandiant M-Trends, annually) -- a 30-90 day hot-storage window
   alone means many real intrusions are invisible to retrospective hunting
   by the time they're discovered through other means. This is the
   practical argument for the tiered hot/cold architecture above: keep the
   EXPENSIVE hot tier lean and fast, but keep a CHEAP cold tier deep enough
   (a year or more, for high-value sources like cloud control-plane and
   identity logs specifically) to hunt back through a realistic dwell-time
   window when needed.

Cost management (a real, ongoing architectural concern):

   Most SIEM pricing scales with INGESTION VOLUME, which creates a
   perverse incentive to under-log exactly the noisy-but-sometimes-
   critical sources (verbose debug output, full network flow). The
   tiered architecture above is the answer, not "just log less":
     [ ] route high-volume/lower-value sources to the cheap data-lake
         tier by default, with detection rules running there on a
         schedule rather than in real time where near-real-time isn't
         actually needed.
     [ ] reserve expensive hot-tier, real-time indexing for the sources
         that genuinely need sub-minute detection latency (identity,
         cloud control plane, EDR/runtime).
     [ ] periodically audit WHAT'S actually being queried/alerted-on vs
         what's just accumulating cost with no detection or hunting value
         -- not every field of every log needs to be searchable in the
         expensive tier forever.

Worked example

Redesigning SecureShop's telemetry to survive a real hunt.

   BEFORE: everything goes straight into the SIEM's hot, indexed storage
   with a 30-day retention (chosen because that's what fit the licensing
   budget). VPC Flow Logs alone are 60% of total ingestion cost.

   INCIDENT: eight weeks after a real intrusion (discovered via a
   third-party notification, a common real-world discovery path per
   Mandiant's reporting), the IR team needs to establish INITIAL ACCESS
   -- but it happened 65 days ago. The SIEM's 30-day retention has already
   aged it out. The investigation is now guessing from secondary
   evidence instead of the primary CloudTrail record.

   REDESIGN:
     1. Normalise everything to OCSF at ingest (Vector as the collector),
        so the SAME detection rules and hunting queries work whether the
        data currently lives in the hot SIEM or the cold data lake.
     2. Route CloudTrail, identity/IdP logs, and Kubernetes audit logs
        (the highest per-event VALUE, lowest per-event VOLUME) to hot
        SIEM storage with 90-day retention, PLUS an unconditional parallel
        copy to an S3 data lake in OCSF/Parquet with a 400-DAY retention
        (comfortably past typical dwell-time distributions).
     3. Route VPC Flow Logs and verbose application debug logs (highest
        VOLUME, lower per-event value for most investigations) DIRECTLY
        to the S3 data lake only, queried via Athena when a hunt actually
        needs them -- removing 60% of the ingestion cost from the
        expensive hot tier without losing the data for the cases that
        genuinely need it.
     4. Detection rules for time-sensitive techniques (Ch 54's privilege-
        escalation and exfiltration patterns) run in the SIEM against hot
        storage in real time; a SEPARATE weekly scheduled job re-runs a
        subset of hunting queries against the full data lake, catching
        anything the real-time rules missed or that only becomes
        significant in retrospect.
   RESULT: the total ingestion COST didn't rise (volume routing offset
   the retention increase on high-value sources), but the NEXT
   investigation like this one has 400 days of primary-source evidence
   instead of 30.

Practice (60 min)

Inventory your current telemetry sources against the list in this chapter — which are you NOT collecting at all (SaaS audit logs and identity-provider logs are the most commonly missing)?
Pick one high-volume, lower-per-event-value source (verbose app logs, full network flow) and one high-value, lower-volume source (cloud control plane, identity). Design (or implement, if you have a lab SIEM + object storage) the tiered routing: high-value to hot + long-retention cold copy, high-volume to cold-only with on-demand query.
Normalise one real log source into OCSF (or ECS) using a collector (Vector/Fluent Bit) transform; confirm a query written against the schema works regardless of the original source format.
Calculate your actual retention today for your two most important sources against Mandiant's most recently published median dwell-time figure — is your hot tier alone enough to investigate an intrusion discovered at that median?
Write a one-page telemetry architecture diagram for your system: sources, collectors, normalisation, routing decision, hot vs cold storage, and which layer runs real-time vs scheduled detection.

Build it in Go (25 min) — a tamper-evident audit log

Attackers who get far enough edit logs. A hash chain doesn't stop that, but it makes any edit, deletion, reordering, or truncation detectable, provided the newest link is anchored somewhere the attacker can't write.

// auditlog: a tamper-evident audit log (Chapter 52). Each entry stores an
// HMAC over (previous MAC + this entry), forming a chain. Editing, deleting,
// inserting, or reordering any line breaks verification from that point on.
// Anchor the latest MAC somewhere the attacker can't write (a WORM bucket,
// another account, a transparency log) and truncation is caught too.
//
//	go run ./auditlog
package main

import (
	"bufio"
	"bytes"
	"crypto/hmac"
	"crypto/sha256"
	"encoding/hex"
	"encoding/json"
	"fmt"
	"strings"
	"time"
)

type entry struct {
	Seq    int       `json:"seq"`
	Time   time.Time `json:"time"`
	Actor  string    `json:"actor"`
	Action string    `json:"action"`
	Prev   string    `json:"prev"` // MAC of the previous entry
	MAC    string    `json:"mac"`  // HMAC(key, prev || canonical entry without mac)
}

type logger struct {
	key  []byte // from a secrets manager; the log writer never shares it with readers
	last string
	seq  int
	out  bytes.Buffer
}

func (l *logger) mac(e entry) string {
	e.MAC = ""
	body, _ := json.Marshal(e) // struct field order is fixed: a stable encoding
	m := hmac.New(sha256.New, l.key)
	m.Write([]byte(e.Prev))
	m.Write(body)
	return hex.EncodeToString(m.Sum(nil))
}

func (l *logger) append(actor, action string) {
	l.seq++
	e := entry{Seq: l.seq, Time: time.Date(2026, 10, 3, 9, l.seq, 0, 0, time.UTC), Actor: actor, Action: action, Prev: l.last}
	e.MAC = l.mac(e)
	l.last = e.MAC
	line, _ := json.Marshal(e)
	l.out.Write(append(line, '\n'))
}

// verify replays the chain; it returns the first broken sequence number.
func verify(key []byte, log string, anchor string) string {
	l := &logger{key: key}
	prev, want := "", 1
	sc := bufio.NewScanner(strings.NewReader(log))
	for sc.Scan() {
		var e entry
		if err := json.Unmarshal(sc.Bytes(), &e); err != nil {
			return "unparseable line"
		}
		switch {
		case e.Seq != want:
			return fmt.Sprintf("BROKEN at seq %d: expected seq %d (deleted or reordered)", e.Seq, want)
		case e.Prev != prev:
			return fmt.Sprintf("BROKEN at seq %d: chain link mismatch", e.Seq)
		case !hmac.Equal([]byte(l.mac(e)), []byte(e.MAC)):
			return fmt.Sprintf("BROKEN at seq %d: entry content was modified", e.Seq)
		}
		prev, want = e.MAC, want+1
	}
	if prev != anchor {
		return "BROKEN: log ends before the anchored head (truncated)"
	}
	return "OK"
}

func main() {
	key := []byte("demo-key-from-a-secrets-manager!")
	l := &logger{key: key}
	l.append("alice", "login")
	l.append("alice", "kms:Decrypt tenant=42")
	l.append("mallory", "iam:CreateAccessKey user=human/mallory")
	l.append("mallory", "s3:GetObject exports/all.csv")
	l.append("alice", "logout")
	anchor := l.last // shipped to write-once storage after every append
	log := l.out.String()
	lines := strings.SplitAfter(log, "\n")

	fmt.Println("intact log:                ", verify(key, log, anchor))

	edited := strings.Replace(log, "s3:GetObject exports/all.csv", "s3:GetObject exports/one.csv", 1)
	fmt.Println("mallory edits entry 4:     ", verify(key, edited, anchor))

	deleted := strings.Join(append(lines[:2:2], lines[3:]...), "")
	fmt.Println("mallory deletes entry 3:   ", verify(key, deleted, anchor))

	truncated := strings.Join(lines[:3], "")
	fmt.Println("mallory drops the tail:    ", verify(key, truncated, anchor))
}

$ go run ./auditlog
intact log:                 OK
mallory edits entry 4:      BROKEN at seq 4: entry content was modified
mallory deletes entry 3:    BROKEN at seq 4: expected seq 3 (deleted or reordered)
mallory drops the tail:     BROKEN: log ends before the anchored head (truncated)

What to notice:

Each MAC covers the previous MAC, so changing entry 4 would also mean recomputing 5, 6, ..., which requires the HMAC key. That key lives with the writer, not with the storage or the readers.
Truncation is invisible inside the log itself. Only the separately stored anchor (anchor := l.last, shipped to write-once storage such as S3 Object Lock) catches it. That's why the chapter puts audit logs in a separate account.
This is the same idea as Certificate Transparency and as AWS CloudTrail's digest files, just smaller.

Exercises:

Replace HMAC with Ed25519 signatures so anyone can verify the log without holding the signing key.
Anchor the head every minute by writing it to a second, independently administered system, and alert when the anchors stop.
Run verify on a schedule from the detection pipeline and raise a Chapter 51 alert on any break.

Common mistakes

One retention setting for everything, chosen by licensing cost rather than investigative need. Tier deliberately: some sources need months of real-time indexing, others need a year-plus of cheap, queryable-on-demand storage.
No SaaS or identity-provider telemetry at all. These are exactly the sources that show account compromise first, and they're the most commonly missing category.
Locking the security data lake inside one SIEM vendor's proprietary ingestion. Makes future migration expensive and prevents pointing other tools (hunting notebooks, ML pipelines) at the same authoritative data.
Treating "we log everything" as equivalent to "we can investigate anything." Volume without normalisation, enrichment, and adequate retention on the RIGHT sources doesn't answer investigative questions.
No enrichment. A raw ARN or SPIFFE ID in an alert that nobody can quickly map to "which team owns this, and is this normal for them" slows every investigation.
Cost-cutting by dropping high-value sources instead of routing high-volume ones to a cheaper tier. Solves the budget line item while destroying investigative capability where it matters most.

Check yourself

Name six categories of telemetry source and which one is most commonly missing from real organisations' pipelines.
What does normalising to a common schema (OCSF/ECS) buy you that raw, per-source formats don't?
Why decouple the security data lake from a single SIEM vendor's storage?
What compliance floor exists for log retention under PCI-DSS, and why is real-world breach dwell time a stronger argument for longer retention than compliance alone?
Describe the tiered hot/cold routing strategy and what determines which tier a source goes to.
In the worked example, what specifically caused the 65-day-old evidence gap, and what architectural change fixed it without raising cost?