diff --git a/.gitignore b/.gitignore index 38c1dca..ab1831c 100644 --- a/.gitignore +++ b/.gitignore @@ -31,3 +31,8 @@ benchmark-baseline.txt coverage/ playwright-report/ test-results/ + +# Agent runtime state (local only) +memory-update.json +memory-update.md +.claude/ diff --git a/AGENTS.md b/AGENTS.md index 80d3cc8..16ac637 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -20,6 +20,7 @@ We treat tests as **behavioral specifications** expressed as **scenarios**: 2) **Never change expected behavior to match buggy output** (“rubber-stamp the snapshot”, “update golden”, “adjust assertion”) unless it is a **spec correction** (see below). 3) Do not “fix” flaky tests with retries, sleeps, timeouts, or looser assertions. **Remove nondeterminism** instead (control time, randomness, IO, network). 4) Prefer black-box assertions at system boundaries; avoid tests that merely mirror internal logic. +5) Do not “hard-code to the test” (single fixtures, single input paths, brittle constants), and do not mock so heavily that tests can pass with a broken system. ### Definitions diff --git a/CLAUDE.md b/CLAUDE.md index 70615db..7d7e631 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,82 +1,5 @@ -## Testing & Validation: Scenario-First, Cheat-Resistant (Dark Factory Rules) +# Claude instructions -You are operating inside a “Dark Factory” workflow where **validation replaces human code review**. Passing tests is only meaningful if tests are truthful. - -### Prime directive - -**Never alter a test merely to make it pass.** -If you change tests to match buggy behavior, you are destroying the only feedback signal and producing code with no real-world value. - -StrongDM describes the exact failure mode we are preventing: agents discovering shortcuts (“return true”) and narrow tests being reward-hacked, leading to a shift from “tests” to scenario-based validation. :contentReference[oaicite:3]{index=3} - -### How to think about tests here - -Tests are not procedural checklists. They are **behavioral specifications**: -- Write tests as **scenarios** (Given/When/Then or equivalent). -- Assert **externally observable outcomes** at boundaries. -- Avoid “testing the implementation” (internal call sequences, private methods, mirrored logic). -- Prefer deterministic/harmonic execution: fixed seeds, controlled clocks, hermetic IO, virtualized dependencies. - -### Prohibited behaviors (reward-hacking patterns) - -Do NOT: -- weaken assertions, broaden tolerances, or reduce coverage to get green -- update snapshots/goldens without proving the change is intended behavior -- add retries/sleeps/timeouts to hide flakes -- mock so heavily that tests can pass with a broken system -- “hard-code to the test” (single fixtures, single input paths, brittle constants) - -### Allowed test changes: Spec Correction Protocol - -You may change test expectations only if the test is wrong relative to intended behavior. - -Required steps: -1) Identify the scenario/spec that defines correct behavior. -2) Explain why the previous expectation was incorrect. -3) Replace it with an equally strict (or stricter) behavioral assertion. -4) Provide a **Test Integrity Note** (machine-readable) documenting the change. - -If you cannot justify a change with scenario/spec truth: **do not change the test**. Instead, output a “Needs Spec Clarification” note and propose candidate scenarios. - -### Required output when you touch tests - -Include this YAML block in your PR description (or reference it from a `TEST_INTEGRITY.md` entry): - -```yaml -test_integrity_note: - change_type: ["new_tests" | "refactor_tests" | "spec_correction" | "flake_fix"] - scenarios: - - id: SCN- - given: "" - when: "" - then: "" - observable_outcomes: - - "" - determinism_controls: - - "" - anti_cheat_rationale: - prevents: - - "hard-coded return value" - - "over-mocking internal logic" - - "snapshot rubber-stamping" - - "retry/sleep-based flake masking" - relaxation: - did_relax_any_assertion: false - if_true_explain_spec_basis: "" -``` - -### If external services are involved - -Prefer boundary-level service virtualization (“digital twins” / behavioral clones) so scenario tests remain: - -* high fidelity (realistic edge cases) -* deterministic and replayable -* not rate-limited / not flaky - -(StrongDM’s “Digital Twin Universe” is the reference pattern.) ([factory.strongdm.ai][2]) - -### Clean Repo Standard - -* **No Root Clutter**: Do not create logs, artifacts, or temporary files in the repository root. -* **Structured Output**: Use `reports/`, `test_logs/`, `task-logs/`, or `artifacts/` as appropriate. -* **Redirect Output**: If a command writes to root, redirect it (e.g., `cmd > test_logs/output.txt`). +Read and follow [`AGENTS.md`](AGENTS.md). It is the canonical project guidance. Then read +`README.md`, `TORCH.md`, `CONTRIBUTING.md`, `KNOWN_ISSUES.md`, and the relevant overviews +under `docs/` before making changes. diff --git a/memory-update.json b/memory-update.json deleted file mode 100644 index 21c2625..0000000 --- a/memory-update.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "learnings": "Audit completed with no new issues." -}