mirror of
https://github.com/PR0M3TH3AN/bitvid.git
synced 2026-10-27 16:02:26 +00:00
149 lines
5.3 KiB
Markdown
149 lines
5.3 KiB
Markdown
# High-Quality, Low-Flake Playwright Testing Guide
|
|
|
|
This guide defines how we write and evolve Playwright tests in bitvid so they stay reliable while increasing coverage.
|
|
|
|
## Goals
|
|
|
|
1. Catch user-visible regressions early.
|
|
2. Keep CI signal trustworthy (low flake rate).
|
|
3. Improve runtime coverage with deterministic tests, not brittle ones.
|
|
|
|
## Core Principles
|
|
|
|
1. Test behavior, not implementation details.
|
|
2. Prefer deterministic setup over live-network dependence.
|
|
3. Wait for explicit app state, not arbitrary sleep.
|
|
4. Use resilient selectors and scoped locators.
|
|
5. Keep assertions strict on outcomes, flexible on timing and non-critical internals.
|
|
|
|
## Determinism First
|
|
|
|
Use the existing harness and fixture stack:
|
|
|
|
- `tests/e2e/helpers/bitvidTestFixture.ts`
|
|
- `window.__bitvidTest__` APIs in `js/testHarness.js`
|
|
|
|
Prefer these patterns:
|
|
|
|
1. Use seeded relay data instead of remote relays.
|
|
2. Use programmatic login (`loginAs`) instead of manual UI auth for most tests.
|
|
3. Use harness APIs to force edge conditions (`setSignerDecryptBehavior`, relay overrides).
|
|
4. Open complex modals via deterministic controller/harness methods when available.
|
|
|
|
Why: nondeterministic setup is the top source of flaky tests.
|
|
|
|
## Selector Strategy
|
|
|
|
1. Prefer `data-testid` selectors.
|
|
2. Scope locators to visible/active containers when duplicates can exist.
|
|
3. Avoid broad selectors like `button` unless nested under a stable root.
|
|
|
|
Example:
|
|
|
|
- Better: `#playerModal #closeModal:visible`
|
|
- Worse: `#closeModal` (can resolve hidden duplicate elements)
|
|
|
|
## Waiting Strategy (Do This, Not That)
|
|
|
|
Do:
|
|
|
|
1. Wait for explicit state changes (`data-open="true"`, absence of `.hidden`, expected text/attribute).
|
|
2. Use `waitForFunction` only with concrete conditions and clear timeouts.
|
|
3. Add fallback action paths for known intermittent UI event misses when justified.
|
|
|
|
Avoid:
|
|
|
|
1. `waitForTimeout(...)` as primary synchronization.
|
|
2. Assuming a click always opens a modal immediately.
|
|
3. Chained interactions without state confirmation between steps.
|
|
|
|
## Modal Testing Pattern
|
|
|
|
Recommended flow:
|
|
|
|
1. Trigger modal open via deterministic helper (harness/controller), fallback to UI click.
|
|
2. Wait for modal-open condition.
|
|
3. Assert visible + semantic state (`role`, `aria-modal`, `data-open`).
|
|
4. Interact with modal.
|
|
5. Close modal and verify closed state.
|
|
|
|
If your tests repeatedly fail opening the same modal, add a harness helper for that modal instead of adding sleeps.
|
|
|
|
## Accessibility Test Stability
|
|
|
|
Accessibility tests are especially prone to timing flake. For keyboard and modal tests:
|
|
|
|
1. Ensure focus target exists and is visible before key presses.
|
|
2. Prefer element-scoped key actions (`locator.press("Enter")`) over global keyboard when possible.
|
|
3. Verify modal opened before asserting internals.
|
|
4. For Escape-close tests, first confirm modal open state deterministically.
|
|
|
|
## Coverage Without Brittle UI Tests
|
|
|
|
Not all coverage gains require full UI choreography.
|
|
|
|
Use runtime module tests in browser context for pure/mostly-pure modules:
|
|
|
|
1. `page.evaluate(async () => import("/js/module.js"))`
|
|
2. Execute exported APIs with deterministic inputs.
|
|
3. Assert normalized outputs and edge cases.
|
|
|
|
This is effective for utilities, state transforms, and view-logic modules that are hard to reach through full flows.
|
|
|
|
## Anti-Patterns to Avoid
|
|
|
|
1. Duplicate clicks on close/open controls “just in case.”
|
|
2. Assertions that only check element attachment but not visible state when visibility matters.
|
|
3. Overly permissive assertions (`expect(x || true).toBe(true)`) unless documented as intentional.
|
|
4. Tests that rely on console noise rather than asserting intended state transitions.
|
|
|
|
## Flake Triage Workflow
|
|
|
|
When a test flakes:
|
|
|
|
1. Reproduce with repeats:
|
|
- `npx playwright test <spec> --project=e2e --repeat-each=5`
|
|
2. Identify failure type:
|
|
- setup nondeterminism
|
|
- selector ambiguity
|
|
- missing state wait
|
|
- race in app init
|
|
3. Fix in this order:
|
|
- add deterministic harness/controller hook
|
|
- tighten selectors
|
|
- add explicit state waits
|
|
- only then consider timeout tuning
|
|
4. Re-run repeats and then run coverage lane.
|
|
|
|
## Quality Gates for Sustainable Improvement
|
|
|
|
1. Keep `npm run test:e2e:coverage` green before raising coverage thresholds.
|
|
2. Track coverage trend, but do not trade reliability for raw percentage.
|
|
3. Ratchet coverage targets gradually (example: 40% -> 45% -> 50%).
|
|
4. Require flaky-test fixes to include deterministic synchronization changes, not just longer timeouts.
|
|
|
|
## PR Checklist for E2E Changes
|
|
|
|
1. Does this test use deterministic fixture/harness setup?
|
|
2. Are selectors stable (`data-testid` or scoped stable IDs)?
|
|
3. Are waits condition-based (not sleep-based)?
|
|
4. Does the test fail for real regressions and pass reliably under repeats?
|
|
5. If adding coverage tests, do they increase useful coverage in high-risk paths?
|
|
|
|
## Useful Commands
|
|
|
|
1. Single spec:
|
|
- `npx playwright test tests/e2e/<spec>.spec.ts --project=e2e`
|
|
2. Repeat to detect flake:
|
|
- `npx playwright test tests/e2e/<spec>.spec.ts --project=e2e --repeat-each=5`
|
|
3. Full deterministic E2E coverage run:
|
|
- `npm run test:e2e:coverage`
|
|
4. Regenerate report from existing raw artifacts:
|
|
- `npm run test:e2e:coverage:report`
|
|
|
|
## Related Docs
|
|
|
|
- `docs/testing/playwright-function-coverage.md`
|
|
- `docs/testing/playwright-integration-recommendations-2026-02-22.md`
|
|
- `AGENTS.md` (testing infrastructure and harness references)
|