Chapter 26 — East-west security: internal APIs, gRPC, and event streams

In one sentence

Traffic between your own services ("east-west") must be authenticated, authorized, and encrypted exactly like traffic from the internet, because under assume-breach the internal network is hostile — and gRPC and message queues each have their own hardening.

Where we are

Ch 23-25 gave you service identity, safe token propagation, and an authz model. This chapter applies them to the actual internal transports and calls out the protocol-specific pitfalls.

How it works

Internal HTTP/gRPC APIs — the rules:

   [ ] mTLS on every internal call (SPIFFE, Ch 23). "It's internal" is not
       authentication.
   [ ] Every request carries an audience-scoped token (Ch 24); every
       service validates it (sig, iss, exp, AUD == self) and derives
       identity from it -- never from a header.
   [ ] Authorize every request against the model (Ch 25), at object level.
   [ ] NetworkPolicy default-deny (Ch 19) so a service is only reachable
       by its actual callers -- defense in depth if the mesh sidecar is
       bypassed.
   [ ] Internal endpoints are STILL endpoints: input validation, rate
       limits (Ch 28), no debug/admin routes reachable without auth, no
       verbose errors, no "internal" APIs that skip authz because "only
       our services call them".
   [ ] No "trusted subnet" that bypasses any of the above.

gRPC specifics:

   - CHANNEL credentials (TLS/mTLS) authenticate the connection; CALL
     credentials (a token in metadata, e.g. `authorization: Bearer ...`)
     authenticate the request. Use both.
   - INTERCEPTORS (server + client) are where you do authn/authz/logging
     centrally -- don't scatter checks in handlers.
   - DISABLE server reflection in production (it dumps your whole API
     surface to anyone who can connect).
   - Set limits: `MaxRecvMsgSize` / `MaxSendMsgSize` (default 4MB -- a
     huge message is a memory DoS), max concurrent streams, keepalive
     enforcement (reject abusive keepalive pings), connection age limits.
   - Metadata is attacker-influenced on ingress -- treat `x-*` headers
     from outside the trust boundary as untrusted; strip/normalize at the
     edge.
   - gRPC-Web / JSON transcoding at the edge: the same authz applies; don't
     let the transcoder bypass interceptors.
   - Deadlines/timeouts propagate via context -- set them (Ch 28); an
     unbounded call chains into a cascading hang.

Message queues and event streams (Kafka as the example; SQS/SNS/PubSub/ RabbitMQ analogous):

   TRANSPORT & AUTHN
     - TLS between clients and brokers (and inter-broker).
     - Client auth: mTLS, or SASL (SCRAM / OAUTHBEARER / GSSAPI). Not
       SASL/PLAIN over anything but TLS, ideally not at all.
   AUTHZ
     - Broker ACLs: per-topic READ/WRITE/DESCRIBE, per consumer GROUP,
       per TRANSACTIONAL-ID. Principle of least privilege: `orders-api`
       WRITES `orders.events`, `fulfillment` READS it, nobody else.
     - Multi-tenant: topic-name convention `t.<tenantId>.orders` + ACLs
       per tenant principal, OR a single topic with tenant_id in the
       event + consumer-side filtering (weaker -- a bug reads all tenants;
       prefer topic/prefix isolation + ACLs).
   DATA
     - Encrypt broker storage at rest (disk / per-topic if supported).
     - PII in events: minimize; consider field-level encryption (Part 7)
       so a consumer without the key gets ciphertext.
   EVENT AUTHENTICITY
     - A consumer should be able to trust an event's origin. Options:
       (a) rely on transport (mTLS + ACLs + broker trust) -- fine within
           one trust domain;
       (b) SIGN the event payload (producer signs with its key / SPIFFE
           JWT-SVID as an envelope) so authenticity survives replication,
           mirroring, tiered storage, and a compromised broker.
   REPLAY & POISON
     - Idempotency: every event has a stable ID; consumers dedupe.
     - Ordering/monotonicity where it matters (per-key sequence).
     - POISON MESSAGES / DLQs: a malformed or malicious event that a
       consumer can't process lands in a dead-letter queue -- which is
       often readable by MORE people and RETAINED LONGER than the main
       topic. Treat the DLQ as sensitive; scrub PII; alert on DLQ growth.
     - Message attributes / headers are an injection vector (a consumer
       that logs or `eval`s them, or uses them in a query) -- validate.
   SCHEMA
     - A schema registry with auth; validate on produce AND consume;
       schema evolution rules; a rogue producer shouldn't be able to
       register an incompatible schema.

The "internal API that skipped authz" anti-pattern: the single most common east-west finding. orders-api exposes /internal/orders/{id} with no auth "because only fulfillment calls it" — then an SSRF, a mislabeled NetworkPolicy, a test tool, or a compromised neighbour calls it directly and reads any order. Every internal endpoint gets identity + authz.

Worked example

Locking down an event pipeline so a compromised consumer can't forge or replay.

   Pipeline: orders-api --produces--> `orders.events` (Kafka)
             --consumed by--> fulfillment, analytics, email-service

   THREATS & CONTROLS:
     - A compromised `analytics` publishes a fake `order.cancelled` to
       trigger refunds.
         -> Kafka ACL: `analytics` has READ on `orders.events` only, no
            WRITE. `orders-api` is the only WRITE principal (mTLS
            identity). Forged publish -> broker rejects.
         -> Defense in depth: `fulfillment` verifies an envelope signature
            on each event (orders-api signs with its SPIFFE JWT-SVID;
            fulfillment checks sub == orders-api, aud == "orders.events",
            exp). A broker compromise that lets a write through still
            produces unsigned/wrong-signed events -> rejected.
     - `analytics` replays a captured `charge.succeeded` to double-count
       revenue / trigger downstream twice.
         -> every event has `event_id` (UUID) + `occurred_at`; consumers
            keep a dedup window keyed by `event_id`; `fulfillment` is
            idempotent on `order_id + event_type`.
     - PII (customer email, address) in `orders.events` read by
       `analytics` which shouldn't see it.
         -> the event carries those fields ENCRYPTED with the tenant DEK
            (Part 7); `email-service` has `kms:Decrypt` on tenant keys,
            `analytics` does not -> analytics sees ciphertext, processes
            only the non-PII fields.
     - A malformed event wedges `fulfillment`; it goes to a DLQ that the
       whole eng org can read.
         -> DLQ ACL = the on-call SRE role only; a scrubber strips PII
            before DLQ write; an alert fires on DLQ depth > 0.

Practice (75 min)

Two gRPC services with mTLS (SPIFFE) channel creds + a call-cred token; do authn/authz in interceptors; set MaxRecvMsgSize, deadlines, and disable reflection. Try: a 50MB message (rejected), a call with no token (rejected), a call with a wrong-aud token (rejected), grpcurl -list against the reflection endpoint (fails).
Add an /internal/orders/{id} endpoint with no auth "because internal". From a third service (simulating SSRF/compromise), call it and read data. Then add identity + authz; re-test.
Run Kafka (Redpanda/Strimzi) with mTLS + ACLs. Give producer WRITE on orders.events, consumer READ only. Try to publish as consumer (denied).
Add an envelope signature to events (sign with a key / JWT-SVID); have the consumer verify; publish a tampered event and watch it be rejected.
Implement idempotent consumption (dedupe by event_id); replay an event and confirm no double-processing.
Create a poison message; watch it hit the DLQ; add a PII scrubber before DLQ write and a depth alert.

Across the series

gRPC and HTTP/2 load balancing (why long-lived connections pin traffic to one pod) and protocol negotiation: HTTPS guide, Chapter 14 and TCP/IP guide, Chapter 57.
Retries, idempotency keys, and circuit breakers between services, measured: HTTPS guide, Chapter 21.

Common mistakes

"Internal" endpoints without auth. The #1 east-west finding. Identity + authz on every endpoint, always.
Trusting X-User-Id / X-Tenant-Id / X-Forwarded-* between services. Derive from the validated token; strip inbound x-* at the edge.
gRPC server reflection enabled in prod. Free API map for an attacker.
Default 4MB gRPC message limit left unchanged for untrusted callers, or raised without thought — memory DoS.
Kafka with no ACLs (or ACLs disabled / allow.everyone.if.no.acl.found= true). Any client reads/writes any topic.
One shared topic for all tenants with client-side filtering only. One missing filter = cross-tenant leak. Prefer topic/prefix isolation + ACLs.
DLQs treated as plumbing. They accumulate PII, are widely readable, and retained forever. Lock them down, scrub, alert.
No event idempotency. At-least-once delivery + a non-idempotent consumer = duplicate charges/emails/shipments on any retry or replay.

Check yourself

Why does "it's internal" fail as an authentication argument under assume-breach?
gRPC channel credentials vs call credentials — what does each authenticate, and where do you put authz logic?
Name three gRPC-specific hardening settings and why each matters.
For Kafka, what do ACLs gate, and what's the trade-off between topic-per- tenant and a shared topic with a tenant_id field?
When do you sign event payloads rather than rely on transport auth?
Why is a dead-letter queue a sensitive asset?