Chapter 35 — SSH certificates, hardening, and bastions

In one sentence

At more than a handful of hosts, replace known_hosts and authorized_keys sprawl with a small SSH CA that issues short-lived certificates, and put all access behind a hardened bastion.

Why `authorized_keys` doesn't scale

   50 engineers x 500 servers = up to 25,000 authorized_keys entries to
   keep in sync. When someone leaves, you must scrub their key from all
   500. When a host is rebuilt, every client re-TOFUs. Stale keys linger
   for years. This is a real, common source of breaches.

SSH certificates

An SSH certificate is a public key plus metadata (principals, validity window, options), signed by a CA key. It works for both directions:

   HOST CERTIFICATES  -- kills TOFU
     * Generate a CA keypair. Keep the CA private key offline / in an HSM /
       in a signing service.
     * Sign each server's host key -> host certificate with principal =
       the hostname(s).
     * Clients get ONE line in known_hosts:
         @cert-authority *.securesh.op ssh-ed25519 AAAA...<CA pubkey>
     * Now any host presenting a valid cert for its name is trusted.
       No prompts, no per-host pinning, rebuilds are painless.

   USER CERTIFICATES  -- kills authorized_keys sprawl
     * User authenticates to an identity provider (SSO/OIDC + MFA).
     * A signing service issues a SHORT-LIVED cert (e.g. 1-24 h) binding
       their key to principals (usernames/roles) they're entitled to.
     * Servers trust the CA:
         # /etc/ssh/sshd_config
         TrustedUserCAKeys /etc/ssh/user_ca.pub
     * No per-user keys on servers. Deprovisioning = stop issuing certs;
       access expires by itself within hours.

   # Issue a user cert valid 8h, principals alice+deployers, with restrictions
   ssh-keygen -s user_ca -I "alice@2026-09-10" \
     -n alice,deployers -V +8h \
     -O clear -O permit-pty -O permit-port-forwarding \
     -O source-address=10.0.0.0/8 \
     alice_ed25519.pub
   # -> alice_ed25519-cert.pub   (ssh presents it automatically alongside the key)

   ssh-keygen -Lf alice_ed25519-cert.pub    # inspect: principals, validity, options

Force expiry (-V) is the point. Short-lived certs make revocation a non-problem: a leaked cert dies on its own in hours. (There's also a RevokedKeys / KRL mechanism for the rare urgent case.)

Tools that do all this for you: HashiCorp Vault SSH secrets engine, smallstep step-ca, Teleport, Netflix BLESS, Cloudflare Access / GCP OS Login for the managed path.

Hardening `sshd_config`

A solid baseline (adapt paths; test with sshd -t before reload):

   # --- authentication ---
   PermitRootLogin no                    # or prohibit-password if you must
   PasswordAuthentication no
   KbdInteractiveAuthentication no       # unless you use it for TOTP MFA
   PubkeyAuthentication yes
   AuthenticationMethods publickey       # add ,keyboard-interactive for MFA
   MaxAuthTries 3
   LoginGraceTime 20
   AllowGroups ssh-users                 # allowlist, not everyone

   # --- trust anchors ---
   TrustedUserCAKeys /etc/ssh/user_ca.pub
   HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub
   HostKey /etc/ssh/ssh_host_ed25519_key

   # --- reduce attack surface ---
   AllowTcpForwarding no                 # yes only on bastions that need it
   AllowAgentForwarding no
   X11Forwarding no
   PermitTunnel no
   PermitUserEnvironment no
   AllowStreamLocalForwarding no

   # --- modern crypto only (OpenSSH 9+/10+) ---
   KexAlgorithms mlkem768x25519-sha256,sntrup761x25519-sha512@openssh.com,curve25519-sha256
   Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
   MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com
   HostKeyAlgorithms ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256
   PubkeyAcceptedAlgorithms ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256

   # --- logging / containment ---
   LogLevel VERBOSE                      # logs key fingerprints used to auth
   Banner none
   ClientAliveInterval 300
   ClientAliveCountMax 2
   # Match blocks for per-group restriction, chroot for SFTP-only users, etc.

Then: ssh-audit localhost and fix whatever it flags.

The bastion (jump host) pattern

                            internet
                               |
                        +------------+          hardened, minimal, heavily
                        |  BASTION   |          logged, MFA, no forwarding
                        | (public)   |          except what's needed, its
                        +------------+          own audited sshd
                               |  (private network only)
              +----------------+----------------+
              |                |                |
          app-01           app-02            db-01     (sshd listens on the
        no public IP     no public IP     no public IP   private interface only)

   Client:   ssh -J bastion.securesh.op app-01.internal
             (ProxyJump: bastion forwards TCP; never sees your keys/agent)

   Even better: bastion + user certificates + session recording
     -> every session is authenticated to a real identity, time-boxed,
        and replayable for audit. This is what Teleport / Vault / Boundary
        / AWS SSM Session Manager give you.

Modern alternative: agent-based / identity-aware access (AWS SSM Session Manager, GCP IAP, Tailscale SSH, Cloudflare Access) removes the public SSH port entirely — the host dials out or a control plane brokers the session.

Practice / mini-project (75 min): SecureShop SSH CA

   Build the whole thing locally with containers or VMs:

   1. CA: ssh-keygen -t ed25519 -f user_ca -C "SecureShop User CA"
          ssh-keygen -t ed25519 -f host_ca -C "SecureShop Host CA"
          (in reality user_ca lives in Vault/step-ca/an HSM)

   2. HOST side: on each "server", sign its host key with host_ca
          ssh-keygen -s host_ca -I app-01 -h -n app-01.securesh.op -V +52w \
              /etc/ssh/ssh_host_ed25519_key.pub
      Configure sshd: HostCertificate ..., TrustedUserCAKeys /etc/ssh/user_ca.pub

   3. CLIENT side: one line in known_hosts:
          @cert-authority *.securesh.op <contents of host_ca.pub>
      Connect -> NO TOFU prompt. Rebuild a host, re-sign -> still no prompt.

   4. USER side: write `ssh-issue <user> <principals>` that signs the
      user's key with user_ca, -V +8h, with source-address + permit-pty
      only. Connect with the cert.

   5. Prove the properties:
      - a user cert past its -V window is rejected (set -V +1m, wait).
      - a principal you weren't granted is refused by the server's
        AuthorizedPrincipalsFile / Match.
      - revoke a cert via a KRL (ssh-keygen -k) and confirm refusal.
      - `ssh -J bastion app-01` works; `ssh -A` is refused by config.

   6. Write it up in ~/sec-lab/reports/part8-ssh.md.

Common confusions

"SSH certs are X.509 certs." Different, simpler format (...-cert-v01@openssh.com). No chains, no DER, no CAs-of-CAs. One CA signs leaves.
"We need OCSP/CRL for SSH." Prefer short -V lifetimes; use a KRL only for urgent revocation. Expiry is the strategy, same as modern TLS.
"The bastion needs my agent forwarded." Use ProxyJump. The bastion should never be able to act as you.
"Disable TCP forwarding everywhere." Yes on app servers; the bastion often needs AllowTcpForwarding yes to do its job — scope it per host/group.
"Changing the port hardens SSH." It reduces scan noise. Key-only auth, MFA, modern crypto, and a bastion are the actual hardening.

Check yourself

Why does authorized_keys + known_hosts fail to scale, and what replaces each?
What single line in known_hosts eliminates TOFU for a whole domain?
Why do short certificate lifetimes make SSH revocation a near-non-issue?
Give five sshd_config settings from the hardening baseline and what each prevents.
What does ssh -J do that ssh -A does not (security-wise)?