docs(testing): add resilient Playwright testing playbook

docs/testing/high-quality-playwright-tests.md

@@ -0,0 +1,148 @@
+# High-Quality, Low-Flake Playwright Testing Guide
+
+This guide defines how we write and evolve Playwright tests in bitvid so they stay reliable while increasing coverage.
+
+## Goals
+
+1. Catch user-visible regressions early.
+2. Keep CI signal trustworthy (low flake rate).
+3. Improve runtime coverage with deterministic tests, not brittle ones.
+
+## Core Principles
+
+1. Test behavior, not implementation details.
+2. Prefer deterministic setup over live-network dependence.
+3. Wait for explicit app state, not arbitrary sleep.
+4. Use resilient selectors and scoped locators.
+5. Keep assertions strict on outcomes, flexible on timing and non-critical internals.
+
+## Determinism First
+
+Use the existing harness and fixture stack:
+
+- `tests/e2e/helpers/bitvidTestFixture.ts`
+- `window.__bitvidTest__` APIs in `js/testHarness.js`
+
+Prefer these patterns:
+
+1. Use seeded relay data instead of remote relays.
+2. Use programmatic login (`loginAs`) instead of manual UI auth for most tests.
+3. Use harness APIs to force edge conditions (`setSignerDecryptBehavior`, relay overrides).
+4. Open complex modals via deterministic controller/harness methods when available.
+
+Why: nondeterministic setup is the top source of flaky tests.
+
+## Selector Strategy
+
+1. Prefer `data-testid` selectors.
+2. Scope locators to visible/active containers when duplicates can exist.
+3. Avoid broad selectors like `button` unless nested under a stable root.
+
+Example:
+
+- Better: `#playerModal #closeModal:visible`
+- Worse: `#closeModal` (can resolve hidden duplicate elements)
+
+## Waiting Strategy (Do This, Not That)
+
+Do:
+
+1. Wait for explicit state changes (`data-open="true"`, absence of `.hidden`, expected text/attribute).
+2. Use `waitForFunction` only with concrete conditions and clear timeouts.
+3. Add fallback action paths for known intermittent UI event misses when justified.
+
+Avoid:
+
+1. `waitForTimeout(...)` as primary synchronization.
+2. Assuming a click always opens a modal immediately.
+3. Chained interactions without state confirmation between steps.
+
+## Modal Testing Pattern
+
+Recommended flow:
+
+1. Trigger modal open via deterministic helper (harness/controller), fallback to UI click.
+2. Wait for modal-open condition.
+3. Assert visible + semantic state (`role`, `aria-modal`, `data-open`).
+4. Interact with modal.
+5. Close modal and verify closed state.
+
+If your tests repeatedly fail opening the same modal, add a harness helper for that modal instead of adding sleeps.
+
+## Accessibility Test Stability
+
+Accessibility tests are especially prone to timing flake. For keyboard and modal tests:
+
+1. Ensure focus target exists and is visible before key presses.
+2. Prefer element-scoped key actions (`locator.press("Enter")`) over global keyboard when possible.
+3. Verify modal opened before asserting internals.
+4. For Escape-close tests, first confirm modal open state deterministically.
+
+## Coverage Without Brittle UI Tests
+
+Not all coverage gains require full UI choreography.
+
+Use runtime module tests in browser context for pure/mostly-pure modules:
+
+1. `page.evaluate(async () => import("/js/module.js"))`
+2. Execute exported APIs with deterministic inputs.
+3. Assert normalized outputs and edge cases.
+
+This is effective for utilities, state transforms, and view-logic modules that are hard to reach through full flows.
+
+## Anti-Patterns to Avoid
+
+1. Duplicate clicks on close/open controls “just in case.”
+2. Assertions that only check element attachment but not visible state when visibility matters.
+3. Overly permissive assertions (`expect(x || true).toBe(true)`) unless documented as intentional.
+4. Tests that rely on console noise rather than asserting intended state transitions.
+
+## Flake Triage Workflow
+
+When a test flakes:
+
+1. Reproduce with repeats:
+- `npx playwright test <spec> --project=e2e --repeat-each=5`
+2. Identify failure type:
+- setup nondeterminism
+- selector ambiguity
+- missing state wait
+- race in app init
+3. Fix in this order:
+- add deterministic harness/controller hook
+- tighten selectors
+- add explicit state waits
+- only then consider timeout tuning
+4. Re-run repeats and then run coverage lane.
+
+## Quality Gates for Sustainable Improvement
+
+1. Keep `npm run test:e2e:coverage` green before raising coverage thresholds.
+2. Track coverage trend, but do not trade reliability for raw percentage.
+3. Ratchet coverage targets gradually (example: 40% -> 45% -> 50%).
+4. Require flaky-test fixes to include deterministic synchronization changes, not just longer timeouts.
+
+## PR Checklist for E2E Changes
+
+1. Does this test use deterministic fixture/harness setup?
+2. Are selectors stable (`data-testid` or scoped stable IDs)?
+3. Are waits condition-based (not sleep-based)?
+4. Does the test fail for real regressions and pass reliably under repeats?
+5. If adding coverage tests, do they increase useful coverage in high-risk paths?
+
+## Useful Commands
+
+1. Single spec:
+- `npx playwright test tests/e2e/<spec>.spec.ts --project=e2e`
+2. Repeat to detect flake:
+- `npx playwright test tests/e2e/<spec>.spec.ts --project=e2e --repeat-each=5`
+3. Full deterministic E2E coverage run:
+- `npm run test:e2e:coverage`
+4. Regenerate report from existing raw artifacts:
+- `npm run test:e2e:coverage:report`
+
+## Related Docs
+
+- `docs/testing/playwright-function-coverage.md`
+- `docs/testing/playwright-integration-recommendations-2026-02-22.md`
+- `AGENTS.md` (testing infrastructure and harness references)

docs/testing/playwright-function-coverage.md

@@ -32,6 +32,7 @@ This command:
 
 - Coverage is captured from Chromium runtime execution (`page.coverage`), then combined with static function estimates for source files in `js/` and `torrent/`.
 - Firefox coverage is not included in this report because Playwright JS coverage is Chromium-only.
+- For reliability patterns while increasing coverage, see `docs/testing/high-quality-playwright-tests.md`.
 
 ## Current Baseline (2026-02-22)