From 2f8282c570ae4af4af02445ead5287eb5639f3d6 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Sun, 22 Feb 2026 16:37:44 -0500 Subject: [PATCH] docs(testing): add resilient Playwright testing playbook --- docs/testing/high-quality-playwright-tests.md | 148 ++++++++++++++++++ docs/testing/playwright-function-coverage.md | 1 + 2 files changed, 149 insertions(+) create mode 100644 docs/testing/high-quality-playwright-tests.md diff --git a/docs/testing/high-quality-playwright-tests.md b/docs/testing/high-quality-playwright-tests.md new file mode 100644 index 00000000..62d46bc1 --- /dev/null +++ b/docs/testing/high-quality-playwright-tests.md @@ -0,0 +1,148 @@ +# High-Quality, Low-Flake Playwright Testing Guide + +This guide defines how we write and evolve Playwright tests in bitvid so they stay reliable while increasing coverage. + +## Goals + +1. Catch user-visible regressions early. +2. Keep CI signal trustworthy (low flake rate). +3. Improve runtime coverage with deterministic tests, not brittle ones. + +## Core Principles + +1. Test behavior, not implementation details. +2. Prefer deterministic setup over live-network dependence. +3. Wait for explicit app state, not arbitrary sleep. +4. Use resilient selectors and scoped locators. +5. Keep assertions strict on outcomes, flexible on timing and non-critical internals. + +## Determinism First + +Use the existing harness and fixture stack: + +- `tests/e2e/helpers/bitvidTestFixture.ts` +- `window.__bitvidTest__` APIs in `js/testHarness.js` + +Prefer these patterns: + +1. Use seeded relay data instead of remote relays. +2. Use programmatic login (`loginAs`) instead of manual UI auth for most tests. +3. Use harness APIs to force edge conditions (`setSignerDecryptBehavior`, relay overrides). +4. Open complex modals via deterministic controller/harness methods when available. + +Why: nondeterministic setup is the top source of flaky tests. + +## Selector Strategy + +1. Prefer `data-testid` selectors. +2. Scope locators to visible/active containers when duplicates can exist. +3. Avoid broad selectors like `button` unless nested under a stable root. + +Example: + +- Better: `#playerModal #closeModal:visible` +- Worse: `#closeModal` (can resolve hidden duplicate elements) + +## Waiting Strategy (Do This, Not That) + +Do: + +1. Wait for explicit state changes (`data-open="true"`, absence of `.hidden`, expected text/attribute). +2. Use `waitForFunction` only with concrete conditions and clear timeouts. +3. Add fallback action paths for known intermittent UI event misses when justified. + +Avoid: + +1. `waitForTimeout(...)` as primary synchronization. +2. Assuming a click always opens a modal immediately. +3. Chained interactions without state confirmation between steps. + +## Modal Testing Pattern + +Recommended flow: + +1. Trigger modal open via deterministic helper (harness/controller), fallback to UI click. +2. Wait for modal-open condition. +3. Assert visible + semantic state (`role`, `aria-modal`, `data-open`). +4. Interact with modal. +5. Close modal and verify closed state. + +If your tests repeatedly fail opening the same modal, add a harness helper for that modal instead of adding sleeps. + +## Accessibility Test Stability + +Accessibility tests are especially prone to timing flake. For keyboard and modal tests: + +1. Ensure focus target exists and is visible before key presses. +2. Prefer element-scoped key actions (`locator.press("Enter")`) over global keyboard when possible. +3. Verify modal opened before asserting internals. +4. For Escape-close tests, first confirm modal open state deterministically. + +## Coverage Without Brittle UI Tests + +Not all coverage gains require full UI choreography. + +Use runtime module tests in browser context for pure/mostly-pure modules: + +1. `page.evaluate(async () => import("/js/module.js"))` +2. Execute exported APIs with deterministic inputs. +3. Assert normalized outputs and edge cases. + +This is effective for utilities, state transforms, and view-logic modules that are hard to reach through full flows. + +## Anti-Patterns to Avoid + +1. Duplicate clicks on close/open controls “just in case.” +2. Assertions that only check element attachment but not visible state when visibility matters. +3. Overly permissive assertions (`expect(x || true).toBe(true)`) unless documented as intentional. +4. Tests that rely on console noise rather than asserting intended state transitions. + +## Flake Triage Workflow + +When a test flakes: + +1. Reproduce with repeats: +- `npx playwright test --project=e2e --repeat-each=5` +2. Identify failure type: +- setup nondeterminism +- selector ambiguity +- missing state wait +- race in app init +3. Fix in this order: +- add deterministic harness/controller hook +- tighten selectors +- add explicit state waits +- only then consider timeout tuning +4. Re-run repeats and then run coverage lane. + +## Quality Gates for Sustainable Improvement + +1. Keep `npm run test:e2e:coverage` green before raising coverage thresholds. +2. Track coverage trend, but do not trade reliability for raw percentage. +3. Ratchet coverage targets gradually (example: 40% -> 45% -> 50%). +4. Require flaky-test fixes to include deterministic synchronization changes, not just longer timeouts. + +## PR Checklist for E2E Changes + +1. Does this test use deterministic fixture/harness setup? +2. Are selectors stable (`data-testid` or scoped stable IDs)? +3. Are waits condition-based (not sleep-based)? +4. Does the test fail for real regressions and pass reliably under repeats? +5. If adding coverage tests, do they increase useful coverage in high-risk paths? + +## Useful Commands + +1. Single spec: +- `npx playwright test tests/e2e/.spec.ts --project=e2e` +2. Repeat to detect flake: +- `npx playwright test tests/e2e/.spec.ts --project=e2e --repeat-each=5` +3. Full deterministic E2E coverage run: +- `npm run test:e2e:coverage` +4. Regenerate report from existing raw artifacts: +- `npm run test:e2e:coverage:report` + +## Related Docs + +- `docs/testing/playwright-function-coverage.md` +- `docs/testing/playwright-integration-recommendations-2026-02-22.md` +- `AGENTS.md` (testing infrastructure and harness references) diff --git a/docs/testing/playwright-function-coverage.md b/docs/testing/playwright-function-coverage.md index d410a770..fde08ddb 100644 --- a/docs/testing/playwright-function-coverage.md +++ b/docs/testing/playwright-function-coverage.md @@ -32,6 +32,7 @@ This command: - Coverage is captured from Chromium runtime execution (`page.coverage`), then combined with static function estimates for source files in `js/` and `torrent/`. - Firefox coverage is not included in this report because Playwright JS coverage is Chromium-only. +- For reliability patterns while increasing coverage, see `docs/testing/high-quality-playwright-tests.md`. ## Current Baseline (2026-02-22)