Part 5 — Microservices and distributed authorization
G1 taught mTLS and OAuth for one client and one server. A microservice system has hundreds of services, each of which must authenticate its callers, authorize every request, propagate the end-user's context safely across many hops, and keep tenants isolated — all without trusting the network. This Part is how.
Chapter 23 — Service identity: SPIFFE and SPIRE
In one sentence
SPIFFE gives every workload a cryptographic identity (a URI in a certificate or JWT) that it obtains automatically from a local API with no secret to distribute, and SPIRE is the implementation that issues and rotates those identities based on attested facts about the workload.
Where we are
G1 Ch 31 covered mTLS with certificates you issued by hand. That does not scale to hundreds of ephemeral services. SPIFFE/SPIRE is how you give every pod, VM, and function a short-lived, automatically-rotated identity that downstream services and policy engines can verify.
How it works
SPIFFE ID a URI: spiffe://securesh.op/ns/core/sa/orders-api
one per workload TYPE. The trust domain (securesh.op) is
the root of trust.
SVID "SPIFFE Verifiable Identity Document" -- the credential:
X.509-SVID a short-lived cert with the SPIFFE ID as a URI
SAN. Used for mTLS.
JWT-SVID a short-lived JWT with sub = the SPIFFE ID and
aud = the intended recipient. Used where you
can't do mTLS (through an L7 proxy, a queue).
TRUST BUNDLE the CA certs / JWKS for a trust domain -> how a verifier
checks an SVID. Federation = trust domains exchange
bundles so `securesh.op` workloads can verify
`partner.example` workloads.
WORKLOAD API a local gRPC API over a Unix domain socket. A workload
calls it to GET its SVID + the trust bundle, and to be
notified on rotation. NO token or secret is needed to call
it -- the fact that the process can open that socket, plus
kernel-level attestation of who the process is, IS the
authentication. This is the key trick: no bootstrap
secret.
SPIRE — the implementation:
SPIRE SERVER the CA + registrar. Signs SVIDs. Holds REGISTRATION
ENTRIES (selector-set -> SPIFFE ID mappings). Its CA can be
self-managed or "upstream'd" to Vault / AWS Private CA / disk / an
HSM. HA via a shared datastore.
SPIRE AGENT one per node. Two-step attestation:
1. NODE ATTESTATION -- the agent proves what node it runs on to the
server, using a platform mechanism:
aws_iid (EC2 instance identity doc), gcp_iit, azure_msi,
k8s_psat (a projected SA token the server validates against the
cluster), join_token, x509pop, tpm_devid.
2. WORKLOAD ATTESTATION -- when a local process calls the Workload
API, the agent inspects it via plugins:
unix (uid/gid/path/sha256), k8s (namespace/serviceaccount/
pod-label/pod-name/image-id), docker.
The resulting SELECTORS (e.g. k8s:ns:core, k8s:sa:orders-api) are
matched against registration entries -> the agent hands the
process the right SVID.
REGISTRATION ENTRY example:
spiffe ID : spiffe://securesh.op/ns/core/sa/orders-api
parent ID : spiffe://securesh.op/spire/agent/k8s_psat/<cluster>/<node>
selectors : k8s:ns:core, k8s:sa:orders-api, k8s:pod-label:app:orders-api
ttl : 1h (SVIDs auto-rotate at ~half-life)
How workloads use it:
- Directly: link the SPIFFE library (go-spiffe, py-spiffe, etc.), call
the Workload API, get an X.509-SVID, use it for mTLS; the library
validates the peer's SVID against the trust bundle.
- Via a mesh: Istio, Linkerd, Cilium, and Consul use SPIFFE identities
natively. Istio's `spiffe://<trust-domain>/ns/<ns>/sa/<sa>` is a
SPIFFE ID; its Citadel/istiod is effectively a SPIFFE CA. You get
mTLS + identity for free, and AuthorizationPolicy keys off
`source.principals` (the SPIFFE ID).
- Via SDS: the mesh's Envoy sidecars fetch SVIDs from a SPIRE agent
over the Envoy Secret Discovery Service.
Why this matters for the rest of the Part: every subsequent chapter assumes a service can answer "who is calling me?" with a verified SPIFFE ID. That is the foundation for service-to-service authz (Ch 25), safe token exchange (Ch 24), and "reachable but refused" (Ch 3).
Worked example
web cannot impersonate bff even with network reach.
SecureShop mesh, SPIFFE trust domain securesh.op.
Registration entries:
spiffe://securesh.op/ns/edge/sa/bff <- selectors k8s:ns:edge,
k8s:sa:bff
spiffe://securesh.op/ns/edge/sa/web <- k8s:ns:edge, k8s:sa:web
spiffe://securesh.op/ns/core/sa/orders-api
orders-api's AuthorizationPolicy (Istio):
action: ALLOW
rules:
- from: [{ source: { principals:
["spiffe://securesh.op/ns/edge/sa/bff"] } }]
to: [{ operation: { methods: ["POST"], paths: ["/orders"] } }]
ATTACK: `web` is popped. Attacker wants to call orders-api directly.
- `web`'s Envoy presents `web`'s X.509-SVID (it can't get bff's --
the SPIRE agent attests the calling process; a process in the
`web` pod matches k8s:sa:web selectors, not bff's).
- orders-api's sidecar checks source.principal == bff -> it's `web`
-> 403 RBAC: access denied. Logged with both identities.
- The attacker cannot forge bff's SVID: no CA key, SVIDs are
short-lived and node-attested, and the Workload API only gives a
process the identity it actually attests to.
Even with full network reach, `web`'s blast radius toward orders-api is
zero without a valid end-user context routed through bff.
Practice (60 min) — kind cluster
- Deploy SPIRE (server + agent DaemonSet) with the k8s_psat node attestor. Create registration entries for three workloads.
- Deploy a tiny Go/Python client and server using go-spiffe / py-spiffe that fetch X.509-SVIDs from the Workload API and do mTLS. Confirm the server logs the client's SPIFFE ID.
- Try to get workload A's SVID from workload B's pod — confirm the agent gives B its own identity, not A's (attestation is per-process).
- Install Istio (or Linkerd) with SPIRE integration; enable strict mTLS;
write an
AuthorizationPolicyscopingorders-apitobff's principal. Call fromweb(403) andbff(200). - Rotate: shorten the SVID TTL to 2 minutes; watch SVIDs auto-renew with no downtime.
- Set up SPIFFE Federation between two trust domains (two kind clusters) and verify a cross-domain mTLS call.
Build it in Go (40 min) — SPIFFE-style identity with only the standard library
SPIRE issues and rotates the certificates. What a service has to do is small and worth seeing in code: require mTLS, pull the SPIFFE ID out of the peer certificate's URI SAN, check the trust domain, and authorize by ID.
// spiffe: workload identity the SPIFFE way, with only the standard library.
// A trust domain's CA issues short-lived certificates whose identity is a URI
// SAN (spiffe://shop.example/ns/prod/sa/orders). Services authenticate each
// other with mTLS and authorize by that ID -- not by IP, not by hostname.
//
// go run ./spiffe
package main
import (
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"errors"
"fmt"
"io"
"log"
"math/big"
"net"
"net/http"
"net/url"
"slices"
"strings"
"time"
)
type ca struct {
cert *x509.Certificate
key *ecdsa.PrivateKey
}
func newCA(name string) *ca {
key, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
tmpl := &x509.Certificate{
SerialNumber: serial(), Subject: pkix.Name{CommonName: name},
NotBefore: time.Now().Add(-time.Minute), NotAfter: time.Now().Add(24 * time.Hour),
IsCA: true, BasicConstraintsValid: true,
KeyUsage: x509.KeyUsageCertSign,
}
der, _ := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
c, _ := x509.ParseCertificate(der)
return &ca{c, key}
}
// issue mints an SVID: a 1-hour certificate whose ONLY identity is the URI.
func (a *ca) issue(spiffeID string) tls.Certificate {
key, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
u, _ := url.Parse(spiffeID)
tmpl := &x509.Certificate{
SerialNumber: serial(),
NotBefore: time.Now().Add(-time.Minute),
NotAfter: time.Now().Add(time.Hour), // short-lived: rotation replaces revocation
URIs: []*url.URL{u},
KeyUsage: x509.KeyUsageDigitalSignature,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth, x509.ExtKeyUsageClientAuth},
}
der, _ := x509.CreateCertificate(rand.Reader, tmpl, a.cert, &key.PublicKey, a.key)
return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: key}
}
// spiffeID extracts the workload identity from a verified peer certificate.
func spiffeID(cs tls.ConnectionState) (string, error) {
if len(cs.PeerCertificates) == 0 || len(cs.PeerCertificates[0].URIs) != 1 {
return "", errors.New("no SPIFFE ID")
}
u := cs.PeerCertificates[0].URIs[0]
if u.Scheme != "spiffe" || u.Host != "shop.example" {
return "", fmt.Errorf("foreign trust domain %q", u.Host)
}
return u.String(), nil
}
func main() {
trust := newCA("shop.example trust domain")
pool := x509.NewCertPool()
pool.AddCert(trust.cert)
// The payments service: mTLS required, then an allow-list of callers.
allowed := []string{"spiffe://shop.example/ns/prod/sa/orders"}
srv := &http.Server{
Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
id, err := spiffeID(*r.TLS)
if err != nil || !slices.Contains(allowed, id) {
http.Error(w, "forbidden: "+id, http.StatusForbidden) // authenticated, not authorized
return
}
fmt.Fprintf(w, "charge accepted for caller %s", id)
}),
TLSConfig: &tls.Config{
Certificates: []tls.Certificate{trust.issue("spiffe://shop.example/ns/prod/sa/payments")},
ClientAuth: tls.RequireAndVerifyClientCert, // no client cert, no handshake
ClientCAs: pool,
MinVersion: tls.VersionTLS13,
},
ErrorLog: log.New(io.Discard, "", 0), // the clients below report the outcome
}
ln, _ := net.Listen("tcp", "127.0.0.1:0")
go srv.ServeTLS(ln, "", "")
addr := "https://" + ln.Addr().String() + "/charge"
call := func(label string, cert *tls.Certificate) {
cfg := &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS13,
// The client verifies the SERVER's SPIFFE ID too: mutual, not one-way.
InsecureSkipVerify: true, // hostname checks don't apply to SPIFFE IDs...
VerifyConnection: func(cs tls.ConnectionState) error { // ...so verify chain + ID ourselves
opts := x509.VerifyOptions{Roots: pool, KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}}
if _, err := cs.PeerCertificates[0].Verify(opts); err != nil {
return err
}
if id, err := spiffeID(cs); err != nil || id != "spiffe://shop.example/ns/prod/sa/payments" {
return fmt.Errorf("unexpected server identity %q", id)
}
return nil
},
}
if cert != nil {
// Send the certificate unconditionally. (With cfg.Certificates, Go's
// client quietly sends NOTHING if the cert isn't from a CA the server
// asked for -- an attacker's client won't be so polite.)
cfg.GetClientCertificate = func(*tls.CertificateRequestInfo) (*tls.Certificate, error) {
return cert, nil
}
}
c := &http.Client{Transport: &http.Transport{TLSClientConfig: cfg}, Timeout: 3 * time.Second}
resp, err := c.Post(addr, "text/plain", nil)
if err != nil {
var ue *url.Error
if errors.As(err, &ue) {
err = ue.Err
}
fmt.Printf("%-40s -> rejected in the TLS handshake: %v\n", label, err)
return
}
body, _ := io.ReadAll(resp.Body)
resp.Body.Close()
fmt.Printf("%-40s -> %d %s\n", label, resp.StatusCode, strings.TrimSpace(string(body)))
}
orders := trust.issue("spiffe://shop.example/ns/prod/sa/orders")
reports := trust.issue("spiffe://shop.example/ns/prod/sa/reports")
attacker := newCA("attacker CA").issue("spiffe://shop.example/ns/prod/sa/orders")
call("orders (allowed ID)", &orders)
call("reports (valid cert, wrong ID)", &reports)
call("forged orders ID from another CA", &attacker)
call("no client certificate", nil)
}
func serial() *big.Int {
n, _ := rand.Int(rand.Reader, big.NewInt(1<<62))
return n
}
$ go run ./spiffe
orders (allowed ID) -> 200 charge accepted for caller spiffe://shop.example/ns/prod/sa/orders
reports (valid cert, wrong ID) -> 403 forbidden: spiffe://shop.example/ns/prod/sa/reports
forged orders ID from another CA -> rejected in the TLS handshake: remote error: tls: unknown certificate authority
no client certificate -> rejected in the TLS handshake: remote error: tls: certificate required
What to notice:
- Authentication and authorization are separate outcomes.
reportshas a perfectly valid SVID from our trust domain, so the handshake succeeds, and it still gets a 403 because it isn't on the allow-list. A mesh that only "turns on mTLS" gives you the first half. - Identity is the URI, not the hostname or IP. That's why the client sets
InsecureSkipVerifyand re-implements verification inVerifyConnection: hostname checks don't apply to SPIFFE IDs, but chain verification and the ID check absolutely must. Copying only the first line of that pattern is a serious bug. - A finding from building this lab: given a certificate from a CA the server
didn't ask for, Go's TLS client silently sends no certificate, so the
server sees "certificate required" rather than "unknown authority". The lab
uses
GetClientCertificateso the forged certificate is really presented, as an attacker's client would. - One-hour lifetimes stand in for revocation, exactly as with SPIRE.
Exercises:
- Load the trust bundle and SVID from files and reload them on change
(rotation every 30 minutes), using the
GetCertificatepattern from the TCP/IP guide'scertreloadlab. - Replace the allow-list with a call to the Chapter 25 policy engine:
check("api:payments#caller@spiffe://..."). - Compare with
github.com/spiffe/go-spiffe/v2, which does the same against a live SPIRE agent's Workload API.
Common mistakes
- Treating SPIFFE IDs as secrets. They're identifiers, published freely. The SVID (the signed credential) is what matters, and it's short-lived.
- Loose registration selectors.
k8s:ns:corealone means any SA incoregets that identity. Addk8s:sa:(and label/image where useful). - Long SVID TTLs. Defeats the "leaked credential dies fast" property. Keep TTL short (minutes to an hour); rotation is automatic.
- Upstreaming the SPIRE CA to nothing. Its CA key is the trust domain. Back it with Vault/AWS PCA/HSM, protect the datastore, and plan CA rotation.
- mTLS without authorization. SPIFFE identity tells you who; you still need an AuthorizationPolicy / policy engine to decide what they may do (Ch 25).
- Skipping node attestation strength.
join_tokenis fine for a lab; use the platform attestor (aws_iid/gcp_iit/k8s_psat) in production.
Check yourself
- What is a SPIFFE ID vs an SVID, and why can the Workload API be called with no secret?
- Describe SPIRE's two-step attestation and give one node and one workload attestor.
- What's in a registration entry, and what goes wrong if the selectors are too loose?
- Why are short SVID TTLs a security property, and what makes them practical?
- How does a mesh like Istio use SPIFFE identities?
- SPIFFE identity establishes who; what still has to establish what they may do?
(Answers: Appendix G.)
Further reading
- Spec:
spiffe.io— the SPIFFE and SPIRE docs; the SPIFFE ID, SVID, and Workload API specs; SPIFFE Federation. - Book: Solving the Bottom Turtle (free, spiffe.io) — the "how do you bootstrap trust" problem and SPIFFE's answer.
- Integration: Istio "Identity and certificate management", Linkerd "mTLS", the SPIRE + Istio and SPIRE + Envoy SDS guides.
- Talks: "SPIFFE/SPIRE deep dive" from KubeCon; "Zero Trust workload identity" talks.