Chapter 21 — Multi-tenancy and isolation models

In one sentence

Namespaces plus RBAC, NetworkPolicy, quotas, and policy give you soft multi-tenancy for workloads you mostly trust; running genuinely untrusted tenants in the same cluster needs hard isolation — separate clusters, virtual clusters, dedicated nodes, or sandboxed runtimes — because the control plane and kernel are shared.

Where we are

SecureShop is SaaS: many customer tenants share the platform (Ch 0.6). This chapter is the isolation-model decision and its trade-offs, and Ch 27 does the data layer.

How it works

The tenancy spectrum, weakest to strongest isolation:

   NAMESPACE-PER-TENANT ("soft multi-tenancy")
     boundary: RBAC + NetworkPolicy + ResourceQuota/LimitRange + PSA
       restricted + a policy engine + per-tenant SAs.
     SHARED: the control plane (one API server, one etcd), the nodes, the
       kernel, the CNI, cluster-scoped resources (CRDs, some webhooks),
       DNS.
     good for: internal teams, environments, workloads you build.
     NOT sufficient for: untrusted code / customers running arbitrary
       workloads -- a control-plane bug, an admission bypass, a kernel
       CVE, or a noisy CRD crosses tenants.
     tools: Hierarchical Namespace Controller (HNC), Capsule, vcluster's
       "namespace" mode, Kiosk; "Multi-tenancy" working-group guidance.

   VIRTUAL CLUSTERS (vcluster)
     each tenant gets a syntactic "cluster": its own API server + control
       plane components running as pods inside a host namespace; workloads
       are synced down to the host.
     boundary: tenant can't touch host cluster-scoped resources or other
       tenants' API servers; still shares host nodes + kernel.
     good for: teams that need CRDs / cluster-admin-like power without a
       real cluster each; strong control-plane isolation, moderate cost.

   NODE ISOLATION (dedicated node pools)
     tenants get their own nodes via taints/tolerations + nodeSelector/
       affinity + a separate node pool. No pod co-tenancy -> kernel not
       shared between tenants.
     combine with the namespace controls above. Costs: idle capacity.

   SANDBOXED RUNTIMES (per-pod isolation)
     gVisor (runsc)  -- a user-space kernel intercepts syscalls; the
       container never talks to the host kernel directly. ~small overhead,
       some syscall-compat gaps.
     Kata Containers -- each pod in a lightweight VM (own kernel). Stronger
       isolation, more overhead.
     Set via a RuntimeClass; schedule untrusted tenant pods onto it.
     good for: running genuinely untrusted code (CI for customers,
       plugin/function platforms) on shared nodes.

   SEPARATE CLUSTERS PER TENANT ("hard multi-tenancy")
     the only model that shares nothing (except your cloud account, so
     multi-account too for the highest bar -- Part 2 Ch 7).
     good for: high-value / regulated / mutually-distrustful tenants.
     costs: N control planes, N upgrades, N cost floors -> automate
       ruthlessly (Cluster API, fleet management: Rancher, Anthos, EKS
       Blueprints, Argo CD ApplicationSets).

The decision:

   Is the tenant workload TRUSTED (you wrote it / it's an internal team)?
     yes -> namespace-per-tenant + full soft-MT controls is usually fine.
     no  -> you need hard isolation for the RUNTIME:
              arbitrary containers, some trust    -> sandboxed runtime
                                                     (gVisor/Kata) + node
                                                     isolation
              arbitrary code, regulated, or
                mutually hostile tenants          -> cluster-per-tenant
                                                     (+ account-per-tenant)
   Independently: the DATA layer needs its own isolation proof (Ch 27) --
   RLS / per-tenant keys / per-tenant indices -- regardless of the compute
   model.

Soft-MT hardening checklist (per tenant namespace):

   [ ] RBAC: tenant admins get a Role scoped to their namespace only; no
       cluster-scoped verbs; no `escalate`/`bind`/`impersonate`; can't
       edit their own quotas/policies.
   [ ] NetworkPolicy: default-deny; no cross-tenant traffic; egress
       allowlist; metadata blocked.
   [ ] ResourceQuota + LimitRange: cap CPU/mem/pods/PVCs/LoadBalancers/
       NodePorts; prevent one tenant starving others (a DoS = a security
       issue, Ch 28).
   [ ] PSA `restricted` enforced; policy engine denies privileged/hostPath/
       hostNetwork/host* and unapproved images.
   [ ] Separate SAs; `automountServiceAccountToken: false` default.
   [ ] No shared PVs; StorageClass with per-namespace encryption keys.
   [ ] CRD/operator access denied unless explicitly granted (a
       cluster-scoped CRD is a shared blast radius).
   [ ] Priority/preemption: tenants can't set `system-` priority classes.
   [ ] Audit + per-tenant cost/usage visibility.

Worked example

Choosing the model for two SecureShop features.

   FEATURE A: "each customer's storefront runs OUR code with THEIR config
   and theme". Trusted code, untrusted config/data.
     -> namespace-per-tenant, soft-MT controls, + the Ch 27 data layer.
        A control-plane CVE is a platform-wide incident, accepted with
        an aggressive patch SLA and monitoring. Cost-efficient; fine for
        this trust level.

   FEATURE B: "customers upload and run custom JavaScript 'automation
   functions' that react to order events". Arbitrary tenant code.
     -> soft-MT is NOT enough (their code + a kernel/container-escape CVE
        = other tenants). Options:
        - run each function in a gVisor pod on an isolated node pool,
          NetworkPolicy egress default-deny + FQDN allowlist, strict
          seccomp, no SA token, tight CPU/mem/time limits, per-invocation
          fresh pod. Cheaper, ~10-20% overhead, syscall-compat caveats.
        - or a per-tenant Kata pod (VM isolation) if the threat model /
          regulator demands a hypervisor boundary.
        - or push it out of the cluster entirely to a managed FaaS with
          per-tenant isolation and a denial-of-wallet cap.
     DECISION: gVisor + node isolation + the egress/seccomp/limits stack;
     revisit to Kata if a customer segment needs the stronger boundary.

Practice (60 min) — kind cluster

Build namespace-per-tenant soft-MT for two tenants: scoped RBAC, default-deny NetworkPolicy, ResourceQuota/LimitRange, PSA restricted, a Kyverno policy denying cross-tenant references and unapproved images.
As "tenant A admin", try to: read tenant B's pods/secrets, create a cluster-scoped resource, exceed the quota, run a privileged pod, set a system- priority class, reach tenant B over the network. Confirm each is blocked.
Install gVisor (containerd + runsc) and a RuntimeClass. Run a pod with runtimeClassName: gvisor; from inside, try a syscall-based escape (e.g. the ones from Ch 15) — confirm it fails differently (the host kernel isn't there).
Install vcluster; give "tenant A" a virtual cluster; confirm they can create CRDs and RBAC inside it without touching the host cluster.
Sketch the cost and operational model for cluster-per-tenant at 50 tenants (control-plane cost, upgrade cadence, tooling) — decide where your break-even is.

Common mistakes

Using namespaces for untrusted tenants. One shared API server, one etcd, one kernel. A control-plane or kernel CVE is cross-tenant.
No ResourceQuota. One tenant OOMs the nodes or exhausts LoadBalancers — availability impact on everyone (a security issue).
Cluster-scoped CRDs/operators exposed to tenants. A shared blast radius and often a privesc path.
Forgetting the data layer. Compute isolation without RLS / per-tenant keys / per-tenant indices means a bug in one service still leaks across tenants (Ch 27).
gVisor/Kata as a checkbox without testing the workload. Syscall-compat gaps and overhead need validation per workload.
Cluster-per-tenant without automation. N clusters you upgrade by hand is an operational (and therefore security) failure.

Check yourself

What is shared in namespace-per-tenant multi-tenancy, and why does that make it "soft"?
When is soft multi-tenancy sufficient, and when do you need hard isolation for the runtime?
gVisor vs Kata — how does each isolate, and what's the trade-off?
What does a virtual cluster (vcluster) isolate that a namespace does not?
Give five items on the per-tenant soft-MT hardening checklist.
Why is the data-layer isolation a separate decision from the compute model?