Chapter 36 — Hands-on: SSH keys, tunnels, config, and signing

Why this chapter exists

SSH is a daily tool. Fluency with ~/.ssh/config, port forwarding, and ssh-keygen -Y signing pays off constantly. This is the drill.

~/.ssh/config — stop typing flags

   # Global-ish defaults
   Host *
       AddKeysToAgent yes
       IdentitiesOnly yes                 # only offer the key I name (avoids
                                          # "too many auth failures" lockouts)
       ServerAliveInterval 60
       HashKnownHosts yes
       ControlMaster auto
       ControlPath ~/.ssh/cm/%r@%h:%p
       ControlPersist 5m

   Host bastion
       HostName bastion.securesh.op
       User jump
       IdentityFile ~/.ssh/id_ed25519

   Host app-* db-*
       User deploy
       IdentityFile ~/.ssh/id_ed25519_work
       ProxyJump bastion                  # <-- transparently hop via bastion
       RequestTTY auto

   Host github.com
       User git
       IdentityFile ~/.ssh/id_ed25519_github
       IdentitiesOnly yes

   # Then just:  ssh app-01.internal   (user, key, and jump are automatic)

Port forwarding — the three kinds

   LOCAL forward   ssh -L 5433:db.internal:5432 bastion
     "expose the remote db's :5432 on my localhost:5433"
     -> psql -h 127.0.0.1 -p 5433 ... reaches db.internal through the tunnel.

   REMOTE forward  ssh -R 8080:localhost:3000 jump
     "expose MY localhost:3000 on the remote host's :8080"
     -> for demos, webhooks to a laptop. (GatewayPorts on the server
        controls whether :8080 is public or loopback-only there.)

   DYNAMIC forward ssh -D 1080 bastion
     "run a SOCKS5 proxy on localhost:1080 that exits via bastion"
     -> point a browser / curl --socks5 at it to reach an internal network.

   Add  -N  (no shell)  and  -f  (background) for tunnel-only sessions.
   Jump without forwarding anything:  ssh -J bastion target

scp / sftp / rsync over SSH

   # scp's syntax is legacy and footgun-prone; on OpenSSH 9+ it uses the
   # SFTP protocol under the hood. Prefer sftp or rsync for real work.
   sftp app-01.internal          # interactive: get/put/ls/lcd
   rsync -avz --delete ./site/ app-01.internal:/var/www/site/   # uses ssh
   rsync -e 'ssh -J bastion' ...  # rsync through a jump host

SSH message signing with ssh-keygen -Y (no GPG needed)

Since OpenSSH 8.0 you can sign arbitrary files with an SSH key — this is what Git uses for gpg.format ssh.

cd /tmp && mkdir -p ysign && cd ysign
ssh-keygen -t ed25519 -f signer -N '' -C release@securesh.op -q

# SIGN  (namespace scopes the signature's meaning; use "file" for files,
#        "git" is what git uses)
echo "SecureShop release 7.0.0" > NOTES
ssh-keygen -Y sign -f signer -n file NOTES          # -> NOTES.sig

# VERIFY needs an allowed-signers file mapping identity -> public key
printf '%s %s\n' release@securesh.op "$(cat signer.pub)" > allowed_signers
ssh-keygen -Y verify -f allowed_signers -I release@securesh.op \
           -n file -s NOTES.sig < NOTES             # -> "Good file signature..."

# tamper -> fails
echo tampered >> NOTES
ssh-keygen -Y verify -f allowed_signers -I release@securesh.op \
           -n file -s NOTES.sig < NOTES             # -> "Signature verification failed"

   # Git signed commits with your SSH key (no GPG):
   git config --global gpg.format ssh
   git config --global user.signingkey ~/.ssh/id_ed25519.pub
   git config --global commit.gpgsign true
   # verification needs an allowed_signers file:
   git config --global gpg.ssh.allowedSignersFile ~/.ssh/allowed_signers
   #   ~/.ssh/allowed_signers:  you@example.com ssh-ed25519 AAAA...
   git commit -S -m "signed"        # then: git log --show-signature

Debugging SSH

   ssh -v host        # one -v: auth method negotiation
   ssh -vv host       # + kex/cipher/mac, key offering
   ssh -vvv host      # packet-level

   # Common failures and the -v line that reveals them:
   #  "Permission denied (publickey)"  -> "Offering public key ..." then
   #     "Authentications that can continue" -> server didn't accept the key:
   #     wrong key, not in authorized_keys, bad perms (must be 700 ~/.ssh,
   #     600 the key, 644 .pub, 600 authorized_keys, and the HOME not
   #     group-writable -- sshd is strict).
   #  "too many authentication failures" -> agent offered many keys;
   #     set IdentitiesOnly yes + IdentityFile.
   #  "REMOTE HOST IDENTIFICATION HAS CHANGED" -> known_hosts mismatch (Ch 34).
   #  "no matching key exchange method" -> old server, modern client (or
   #     vice versa); check `ssh -Q kex` on both, add a KexAlgorithms line.

   sudo sshd -T            # dump the server's EFFECTIVE config
   sudo sshd -t            # validate config before reload (do this ALWAYS)
   journalctl -u ssh -f    # watch auth attempts (LogLevel VERBOSE helps)