78. Flakiness, debugging and CI: war stories from this suite

The first green run of Support Desk's suite was not trustworthy yet. A suite is trustworthy when it is green repeatedly, in any order, in parallel. The flake detector is one command:

npx playwright test --repeat-each=10 --workers=8      # each test 10×, in parallel

Here is what it found, in order. Each problem is common in real projects.

1. The a11y scan failed on the first run. The chat transcript was <ol role="log">. An explicit role replaces the element's native role, so the <ol> was no longer a list, and its <li> children were "list items outside a list" (axe rule listitem). The fix was a <div role="log"> containing <div> messages. The test found a real problem for screen-reader users before any human tester.

2. A refund test broke other tests, depending on order. The test refunded A-1002, which other tests read, and a CI retry of the refund test would itself fail with "already in progress". The fix was the seedOrder fixture (§76): tests that write create their own data.

3. The suite rate-limited itself. All tests share Alice's session (via storageState), and the server allows 20 chats per minute per session. At --repeat-each=5 the 21st chat got a 429, and a dozen unrelated tests failed with a 5-second timeout. The fix was to make the limits configurable (CHAT_LIMIT in the webServer env) and give the rate-limit tests their own keys. The production default stays at 20.

4. Successful logins counted toward the login limit. The same run showed that the limiter counted every login. Six correct sign-ins in a minute locked a user out, which is an availability bug in the app, not the test. The fix was to count only failed attempts (check() before verifying, hit() on failure), which is also the correct production behavior.

5. Repeated wrong-password tests locked out Alice. After fix 4, the "wrong password" test still burned Alice's failure budget at --repeat-each=10, and later logins as Alice got 429s. This is the classic lockout trade-off, met in a test. The test now uses a throwaway email per run (nobody-${repeatEachIndex}-${retry}@example.com), and the rate-limit test does the same (ratelimit-${workerIndex}-…). It still proves what matters: the error message is generic.

6. Ownership assertions, not exact lists. "Alice has exactly [A-1001, A-1002]" failed once other tests had seeded orders for her. The fix asserts the property under test: Alice's list contains her orders and not Bob's.

After those fixes: 251 passed at --repeat-each=10.

The general rules these stories teach

1. No shared mutable state between tests. Seed per test, or isolate per worker (testInfo.workerIndex → its own user or tenant).
2. Test-only knobs come from the environment, never from code paths compiled into production by accident. Keep /__test__/* behind NODE_ENV=test and test that it is absent in production.
3. Wait on states, never on time. Use toHaveAttribute("aria-busy", "false"), not waitForTimeout(500).
4. Use unique keys for anything with memory: rate limiters, caches, lockouts, idempotency keys.
5. Run --repeat-each before trusting a new suite, and again whenever a test flakes in CI.

Debugging a failing test

A trace from CI is worth more than any log. trace: "on-first-retry" records one automatically whenever a test needed a retry.

CI: GitHub Actions with sharding

# .github/workflows/e2e.yml
name: e2e
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix: { shard: [1, 2, 3, 4] }
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: npm }
      - run: npm ci
      - run: npm audit signatures            # supply-chain check (§65 A03)
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test --shard=${{ matrix.shard }}/4 --reporter=blob --grep-invert @visual
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with: { name: blob-report-${{ matrix.shard }}, path: blob-report, retention-days: 7 }

  report:
    needs: test
    if: ${{ !cancelled() }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: npm }
      - run: npm ci
      - uses: actions/download-artifact@v4
        with: { path: all-blob-reports, pattern: blob-report-*, merge-multiple: true }
      - run: npx playwright merge-reports --reporter html ./all-blob-reports
      - uses: actions/upload-artifact@v4
        with: { name: html-report, path: playwright-report, retention-days: 14 }

• Sharding splits the suite across machines, and blob reports merge back into one HTML report with traces.
• Test preview deployments, not just localhost. Make baseURL read process.env.BASE_URL when it is set, skip webServer then, and run the same suite against every pull request's preview URL.
• Gate merges on it. A required status check, plus a local pre-commit hook for fast feedback, keeps the suite trusted. frontendlabs.xyz, the site this guide is published on, runs its own Playwright suite on every commit in the same way.

🎯 Challenges. (1) Add Firefox and WebKit projects and find what differs. Hint: <dialog> focus and CDP-only tests. (2) Write a test that runs the production build (NODE_ENV=production) and asserts /__test__/orders returns 404 and the cookie has Secure. (3) Add a network-throttled project (page.route with delays, or CDP's Network.emulateNetworkConditions) and assert the UI still shows progress. (4) Turn one of §77's mocks into a HAR recording with page.routeFromHAR and compare the trade-offs.