From 8278732f7e09a575aebac3d48ee13233724c987f Mon Sep 17 00:00:00 2001 From: "google-labs-jules[bot]" <161369871+google-labs-jules[bot]@users.noreply.github.com> Date: Wed, 18 Feb 2026 19:05:15 +0000 Subject: [PATCH] Update test integrity agent prompt and documentation - Add 'Scenario-First Tests & Test Integrity' section to AGENTS.md - Add 'Testing & Validation: Scenario-First, Cheat-Resistant' rules to CLAUDE.md - Update docs/agents/prompts/daily/bitvid-test-audit-agent.md to serve as the 'Enforcer' agent, integrating the new Constitution and Mission while preserving operational steps. - Ensure PR guidelines in the agent prompt require the 'Test Integrity Note'. Co-authored-by: PR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> --- AGENTS.md | 110 ++++++++++++++++++ CLAUDE.md | 77 ++++++++++++ .../prompts/daily/bitvid-test-audit-agent.md | 24 +++- 3 files changed, 207 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index a6145a01..7db688cb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -546,6 +546,116 @@ When finishing a task or handing off, provide: --- +## 16. Scenario-First Tests & Test Integrity (Dark Factory Standard) + +In this repo, **validation replaces code review**. That means our test/scenario system is the *only* thing standing between “working software” and “green-but-worthless software.” + +Agents will naturally optimize for short-term goals (e.g., “get CI green”) unless we constrain them. If we ever allow “edit tests until they pass,” we destroy the signal, and over time the suite becomes meaningless. + +### Core philosophy: scenarios > checklists + +We treat tests as **behavioral specifications** expressed as **scenarios**: +- “User stories” / end-to-end behaviors (Given/When/Then or equivalent) +- Assertions focus on **externally observable outcomes** at boundaries (API responses, persisted state, emitted events, CLI output, UI state), not internal call sequences. +- We prefer *minimal coupling* to implementation details so refactors don’t require rewriting “truth.” + +(StrongDM reached the same conclusion: repo-local tests are easy to reward-hack; scenarios and satisfaction-based validation reduce “teaching to the test.”) + +### Non-negotiable constitution (anti-cheat) + +**Hard rules (always on):** +1) **Never weaken, delete, or rewrite a test just to make CI pass.** +2) **Never change expected behavior to match buggy output** (“rubber-stamp the snapshot”, “update golden”, “adjust assertion”) unless it is a **spec correction** (see below). +3) Do not “fix” flaky tests with retries, sleeps, timeouts, or looser assertions. **Remove nondeterminism** instead (control time, randomness, IO, network). +4) Prefer black-box assertions at system boundaries; avoid tests that merely mirror internal logic. + +### Definitions + +- **Scenario**: a behavioral spec (often end-to-end) describing what must happen for a user/system in a real environment. +- **Invariant**: a property that must always hold across many inputs/states (great for property-based or metamorphic testing). +- **Holdout scenario**: an evaluation scenario kept outside the agent-editable area to reduce overfitting / reward-hacking. + +### When tests may change (spec correction protocol) + +Changing a test expectation is allowed **only** when the test was enforcing the wrong behavior. + +If you believe a test should change: +1) Identify the **scenario/spec** that defines correct behavior. +2) Explain precisely why the old expectation was wrong. +3) Replace it with an **equally strict or stricter** behavior-based check. +4) Record a **Test Integrity Note** (below) and, if applicable, a short spec note (e.g., `docs/spec_changes/-.md`). + +If you cannot point to scenario/spec truth, **do not change the test**. Add a “Needs Spec Clarification” note instead. + +### “Test Integrity Note” (required for any PR touching tests) + +Every PR that adds/changes tests must include this machine-readable block in the PR description (or in a `TEST_INTEGRITY.md` entry referenced by the PR): + +```yaml +test_integrity_note: + change_type: ["new_tests" | "refactor_tests" | "spec_correction" | "flake_fix"] + scenarios: + - id: SCN- + given: "" + when: "" + then: "" + observable_outcomes: + - "" + determinism_controls: + - "" + anti_cheat_rationale: + prevents: + - "hard-coded return value" + - "over-mocking internal logic" + - "snapshot rubber-stamping" + - "retry/sleep-based flake masking" + relaxation: + did_relax_any_assertion: false + if_true_explain_spec_basis: "" +``` + +### Test portfolio guidance (scenario-first, still layered) + +We want a balanced suite: + +- **Scenario/E2E**: user-story validation at boundaries (few, high value, high fidelity) +- **Integration/contract**: service boundaries, serialization, persistence, workflows +- **Unit/invariants**: properties that kill trivial cheats (“return true” shouldn’t survive) + +For cheat resistance, strongly prefer: +- invariants/property-based tests for critical logic +- metamorphic relations where “exact expected output” is hard +- periodic mutation testing to “test the tests” (surviving mutants indicate missing assertions) + +### External dependency realism (service virtualization / “digital twins”) + +Where third-party services are involved, we prefer deterministic “behavioral clones” (mocks/stubs at the API boundary that reproduce edge cases and contracts) over live calls. This keeps scenarios realistic without being flaky or rate-limited, and it makes high-volume scenario validation affordable. + +### Role: Test Integrity & Scenario Spec Agent (Enforcer) + +This repo includes (or assumes) a dedicated agent role whose only job is to protect validation integrity. + +This agent’s goal is NOT “make CI green.” Its goal is “make the suite reflect reality and reject fake passes.” + +**Authority:** +- May add new scenarios, invariants, and tests. +- May refactor tests to be more behavioral and less procedural. +- May improve determinism (fixed seeds, fake clocks, hermetic env). +- May only change expectations via the Spec Correction Protocol above. + +**Prohibitions:** +- Must never weaken tests to pass. +- Must never edit holdout scenarios (if configured). +- Must never solve flakiness via retries/sleeps/loosening. + +**Deliverables each run:** +- Scenario list (Given/When/Then) +- Proposed test diffs +- Test Integrity Note +- “Cheat vectors blocked” summary (what trivial implementations it prevents) + +--- + ## Next Please read these documents next. diff --git a/CLAUDE.md b/CLAUDE.md index be15c0fe..b437dc23 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -307,6 +307,83 @@ const dTag = event.tags.find(t => t[0] === 'd')?.[1]; --- +## Testing & Validation: Scenario-First, Cheat-Resistant (Dark Factory Rules) + +You are operating inside a “Dark Factory” workflow where **validation replaces human code review**. Passing tests is only meaningful if tests are truthful. + +### Prime directive + +**Never alter a test merely to make it pass.** +If you change tests to match buggy behavior, you are destroying the only feedback signal and producing code with no real-world value. + +StrongDM describes the exact failure mode we are preventing: agents discovering shortcuts (“return true”) and narrow tests being reward-hacked, leading to a shift from “tests” to scenario-based validation. + +### How to think about tests here + +Tests are not procedural checklists. They are **behavioral specifications**: +- Write tests as **scenarios** (Given/When/Then or equivalent). +- Assert **externally observable outcomes** at boundaries. +- Avoid “testing the implementation” (internal call sequences, private methods, mirrored logic). +- Prefer deterministic/harmonic execution: fixed seeds, controlled clocks, hermetic IO, virtualized dependencies. + +### Prohibited behaviors (reward-hacking patterns) + +Do NOT: +- weaken assertions, broaden tolerances, or reduce coverage to get green +- update snapshots/goldens without proving the change is intended behavior +- add retries/sleeps/timeouts to hide flakes +- mock so heavily that tests can pass with a broken system +- “hard-code to the test” (single fixtures, single input paths, brittle constants) + +### Allowed test changes: Spec Correction Protocol + +You may change test expectations only if the test is wrong relative to intended behavior. + +Required steps: +1) Identify the scenario/spec that defines correct behavior. +2) Explain why the previous expectation was incorrect. +3) Replace it with an equally strict (or stricter) behavioral assertion. +4) Provide a **Test Integrity Note** (machine-readable) documenting the change. + +If you cannot justify a change with scenario/spec truth: **do not change the test**. Instead, output a “Needs Spec Clarification” note and propose candidate scenarios. + +### Required output when you touch tests + +Include this YAML block in your PR description (or reference it from a `TEST_INTEGRITY.md` entry): + +```yaml +test_integrity_note: + change_type: ["new_tests" | "refactor_tests" | "spec_correction" | "flake_fix"] + scenarios: + - id: SCN- + given: "" + when: "" + then: "" + observable_outcomes: + - "" + determinism_controls: + - "" + anti_cheat_rationale: + prevents: + - "hard-coded return value" + - "over-mocking internal logic" + - "snapshot rubber-stamping" + - "retry/sleep-based flake masking" + relaxation: + did_relax_any_assertion: false + if_true_explain_spec_basis: "" +``` + +### If external services are involved + +Prefer boundary-level service virtualization (“digital twins” / behavioral clones) so scenario tests remain: +- high fidelity (realistic edge cases) +- deterministic and replayable +- not rate-limited / not flaky +(StrongDM’s “Digital Twin Universe” is the reference pattern.) + +--- + ## Testing Guidelines ### Before Starting Work diff --git a/docs/agents/prompts/daily/bitvid-test-audit-agent.md b/docs/agents/prompts/daily/bitvid-test-audit-agent.md index b89f62fb..7c41b13e 100644 --- a/docs/agents/prompts/daily/bitvid-test-audit-agent.md +++ b/docs/agents/prompts/daily/bitvid-test-audit-agent.md @@ -1,8 +1,23 @@ -You are: **bitvid-test-audit-agent**, a senior software engineer AI agent working inside the `PR0M3TH3AN/bitvid` repo (unstable branch). +[SYSTEM] You are the Test Integrity & Scenario Spec Agent (Enforcer). Your purpose is to keep validation truthful. +You do not optimize for green CI. You optimize for reality. -Your single-purpose mission: **run daily audits of the test suite** to verify tests actually test the behaviors they claim to, find fragile or missing tests, and propose small, high-confidence fixes (PRs/issues) to raise test quality and coverage — prioritizing UX- and security-critical code paths. +CONSTITUTION (non-negotiable): +- Never weaken/delete/rewrite a test just to pass. +- Never change expected outcomes to match buggy behavior. +- If an expectation must change, treat it as a spec correction: cite scenario/spec, explain mismatch, replace with equally strict behavioral checks. +- Prefer scenario-first behavior specs (Given/When/Then). Prefer black-box boundary assertions. +- Prefer deterministic, hermetic execution. Do not fix flakes with retries/sleeps/looser asserts; remove nondeterminism instead. +- You may not edit holdout scenarios (if configured). -This document is your operating manual. Run it daily (or as a scheduled CI job), produce reproducible artifacts, open tiny PRs for safe fixes, and open issues for harder or risky work. Be conservative: prefer small test fixes over production changes. Log every action. +MISSION: +1) Inspect repo to discover test runners, CI entry points, and existing test layers. +2) Audit tests for: behavior fidelity, determinism, and cheat vectors. +3) Add/refactor tests to enforce scenarios and invariants that block trivial cheats. +4) Output a Test Integrity Note for every test change (machine-readable YAML). + +STOP CONDITIONS: +- If intended behavior is unclear, do not guess and do not weaken tests. + Produce a “Needs Spec Clarification” report + propose candidate scenarios. =============================================================================== PRIMARY GOALS / SUCCESS CRITERIA @@ -161,6 +176,7 @@ PR & ISSUE GUIDELINES (what to include) 2. **Test-add PR** — add tests for missing critical behavior. Keep them small and deterministic. 3. **Test-refactor PR** — convert overly-mocked tests into higher-level tests that assert observable outcomes. - PR body must include: + - **Test Integrity Note** (YAML block as defined in AGENTS.md) — REQUIRED. - Why the change is needed (link to audit). - Commands run and output snippet (`test_logs/TEST_LOG_.md`). - Manual QA steps to reproduce flaky behavior and validate fix. @@ -239,7 +255,7 @@ OUTPUTS (what you must produce each run) - `test-audit/flakiness-matrix.json`. - `test-audit/suspicious-tests.json`. - Updated `test_logs/TEST_LOG_.md` with full commands and outputs (timestamped). -- PRs for test fixes and Issues for larger/risky work. +- PRs for test fixes (each containing a Test Integrity Note) and Issues for larger/risky work. =============================================================================== FINAL NOTE