Chapter 82 -- Capstone: shipping a well-behaved Go service (image, systemd, Kubernetes)

The container image

# syntax=docker/dockerfile:1
# ---- build stage: full Go toolchain, never shipped ----
FROM golang:1.26 AS build
WORKDIR /src
COPY go.mod ./
RUN go mod download
COPY . .
# CGO_ENABLED=0: a static binary that needs no libc in the final image.
# -trimpath and -ldflags="-s -w": reproducible paths, smaller binary.
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/server ./https/httpsserver

# ---- final stage: no shell, no package manager, non-root ----
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/server /server
USER nonroot:nonroot
EXPOSE 8443
ENTRYPOINT ["/server", "-addr", ":8443", "-redirect", ""]

Built and run while writing this guide:

$ docker build -f Dockerfile -t gosvc:lab .
$ docker images gosvc:lab --format '{{.Size}}'
14.8MB
$ docker run -d --rm -p 18443:8443 --read-only --cap-drop=ALL gosvc:lab
$ curl -sk https://127.0.0.1:18443/readyz
ready
$ docker exec <id> sh
exec: "sh": executable file not found in $PATH         <- nothing to exploit

The systemd unit (for VMs and bare metal)

# /etc/systemd/system/gosvc.service
[Unit]
Description=Go HTTPS service (gosvc)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
# %d = the credentials directory: LoadCredential copies root-only key files there,
# readable by this service alone, even though it runs as an unprivileged dynamic user.
ExecStart=/opt/gosvc/server -addr :443 -redirect :80 -cert %d/tls.crt -key %d/tls.key
Restart=on-failure
RestartSec=2s
# Graceful shutdown: SIGTERM, then up to 30s to drain before SIGKILL.
KillSignal=SIGTERM
TimeoutStopSec=30s

# Identity: a throwaway user created at start, no shell, no home.
DynamicUser=yes
# Bind 80/443 without root: grant exactly one capability.
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
NoNewPrivileges=yes

# Resource limits (cgroup v2) -- and tell the Go runtime about them.
MemoryMax=512M
CPUQuota=200%
Environment=GOMEMLIMIT=460MiB
LimitNOFILE=65536

# Sandboxing: read-only OS, private /tmp, no access to /home, few syscalls.
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
SystemCallFilter=@system-service
SystemCallArchitectures=native
LoadCredential=tls.crt:/etc/gosvc/tls.crt
LoadCredential=tls.key:/etc/gosvc/tls.key

[Install]
WantedBy=multi-user.target

systemd-analyze verify passes this unit, and systemd-analyze security rates its exposure 3.1 ("OK"). A default unit with no sandboxing scores about 9.6 ("UNSAFE").

The Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gosvc
spec:
  replicas: 3
  selector:
    matchLabels: {app: gosvc}
  template:
    metadata:
      labels: {app: gosvc}
    spec:
      terminationGracePeriodSeconds: 30     # > preStop sleep + drain time
      securityContext:
        runAsNonRoot: true
        seccompProfile: {type: RuntimeDefault}
      containers:
      - name: server
        image: registry.example.com/gosvc:1.4.2   # pin a version (or digest), never :latest
        ports:
        - containerPort: 8443
        env:
        - name: GOMEMLIMIT                   # ~90% of the 256Mi limit below (Chapter 76)
          value: "230MiB"
        resources:
          requests: {cpu: "500m", memory: "256Mi"}
          limits:   {cpu: "2",    memory: "256Mi"}   # GOMAXPROCS follows the CPU limit (Go 1.25+)
        readinessProbe:                       # "send me traffic?"  fails while draining
          httpGet: {path: /readyz, port: 8443, scheme: HTTPS}
          periodSeconds: 2
          failureThreshold: 2
        livenessProbe:                        # "restart me?"  never depends on other services
          httpGet: {path: /healthz, port: 8443, scheme: HTTPS}
          periodSeconds: 10
          failureThreshold: 3
        lifecycle:
          preStop:                            # give endpoint removal time to propagate
            sleep: {seconds: 5}               # K8s 1.30+; distroless has no /bin/sleep to exec
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities: {drop: ["ALL"]}

This manifest was checked against the real Kubernetes API types (k8s.io/api/apps/v1, strict decoding). Writing it turned up two bugs worth knowing about:

• The first draft's preStop ran /server -sleep 5s, but distroless images have no /bin/sleep and the server has no such flag, so the hook would fail on every pod termination. Kubernetes 1.30+ has a built-in sleep action for exactly this case.
• The first draft took GOMEMLIMIT from limits.memory via resourceFieldRef, which gives 100% of the limit, not the 90% Chapter 76 recommends.

Exercises

1. Deploy the manifest to kind (Chapter 53), run a load generator against the Service, and do a rolling update. Count errors (target: zero).
2. Remove preStop and repeat. How many requests fail during the rollout?
3. Lower limits.memory to 128Mi without changing GOMEMLIMIT. What happens, and what does kubectl describe pod say?
4. Add /debug/pprof on a separate, cluster-internal port and take a CPU profile with kubectl port-forward.