mirror of
https://github.com/PR0M3TH3AN/TORCH.git
synced 2026-10-27 07:51:52 +00:00
feat: establish scenario-first testing standards
- Created `AGENTS.md` and `CLAUDE.md` to define test integrity rules and behavior-based validation. - Updated `TORCH.md` to include "Scenario-First Tests & Test Integrity" and the "Test Integrity & Scenario Spec Agent" role. - Updated `src/prompts/daily/test-audit-agent.md` to adopt the new "Enforcer" role, prioritizing test fidelity over CI green state. - Preserved existing scheduler contract headers in the agent prompt. Co-authored-by: PR0M3TH3AN <[email protected]>
This commit is contained in:
1 parent
d0e6dfcd13
commit
5d698a8455
4 files changed
+324
-244
No files matched your search
@@ -0,0 +1,114 @@
|
||||
## Scenario-First Tests & Test Integrity (Dark Factory Standard)
|
||||
|
||||
In this repo, **validation replaces code review**. That means our test/scenario system is the *only* thing standing between “working software” and “green-but-worthless software.”
|
||||
|
||||
Agents will naturally optimize for short-term goals (e.g., “get CI green”) unless we constrain them. If we ever allow “edit tests until they pass,” we destroy the signal, and over time the suite becomes meaningless.
|
||||
|
||||
### Core philosophy: scenarios > checklists
|
||||
|
||||
We treat tests as **behavioral specifications** expressed as **scenarios**:
|
||||
- “User stories” / end-to-end behaviors (Given/When/Then or equivalent)
|
||||
- Assertions focus on **externally observable outcomes** at boundaries (API responses, persisted state, emitted events, CLI output, UI state), not internal call sequences.
|
||||
- We prefer *minimal coupling* to implementation details so refactors don’t require rewriting “truth.”
|
||||
|
||||
(StrongDM reached the same conclusion: repo-local tests are easy to reward-hack; scenarios and satisfaction-based validation reduce “teaching to the test.”) :contentReference[oaicite:1]{index=1}
|
||||
|
||||
### Non-negotiable constitution (anti-cheat)
|
||||
|
||||
**Hard rules (always on):**
|
||||
1) **Never weaken, delete, or rewrite a test just to make CI pass.**
|
||||
2) **Never change expected behavior to match buggy output** (“rubber-stamp the snapshot”, “update golden”, “adjust assertion”) unless it is a **spec correction** (see below).
|
||||
3) Do not “fix” flaky tests with retries, sleeps, timeouts, or looser assertions. **Remove nondeterminism** instead (control time, randomness, IO, network).
|
||||
4) Prefer black-box assertions at system boundaries; avoid tests that merely mirror internal logic.
|
||||
|
||||
### Definitions
|
||||
|
||||
- **Scenario**: a behavioral spec (often end-to-end) describing what must happen for a user/system in a real environment.
|
||||
- **Invariant**: a property that must always hold across many inputs/states (great for property-based or metamorphic testing).
|
||||
- **Holdout scenario**: an evaluation scenario kept outside the agent-editable area to reduce overfitting / reward-hacking.
|
||||
|
||||
### When tests may change (spec correction protocol)
|
||||
|
||||
Changing a test expectation is allowed **only** when the test was enforcing the wrong behavior.
|
||||
|
||||
If you believe a test should change:
|
||||
1) Identify the **scenario/spec** that defines correct behavior.
|
||||
2) Explain precisely why the old expectation was wrong.
|
||||
3) Replace it with an **equally strict or stricter** behavior-based check.
|
||||
4) Record a **Test Integrity Note** (below) and, if applicable, a short spec note (e.g., `docs/spec_changes/<date>-<slug>.md`).
|
||||
|
||||
If you cannot point to scenario/spec truth, **do not change the test**. Add a “Needs Spec Clarification” note instead.
|
||||
|
||||
### “Test Integrity Note” (required for any PR touching tests)
|
||||
|
||||
Every PR that adds/changes tests must include this machine-readable block in the PR description (or in a `TEST_INTEGRITY.md` entry referenced by the PR):
|
||||
|
||||
```yaml
|
||||
test_integrity_note:
|
||||
change_type: ["new_tests" | "refactor_tests" | "spec_correction" | "flake_fix"]
|
||||
scenarios:
|
||||
- id: SCN-<slug>
|
||||
given: "<preconditions>"
|
||||
when: "<stimulus>"
|
||||
then: "<observable outcomes>"
|
||||
observable_outcomes:
|
||||
- "<what the user/system can observe at the boundary>"
|
||||
determinism_controls:
|
||||
- "<fake clock / fixed seed / hermetic env / service virtualization>"
|
||||
anti_cheat_rationale:
|
||||
prevents:
|
||||
- "hard-coded return value"
|
||||
- "over-mocking internal logic"
|
||||
- "snapshot rubber-stamping"
|
||||
- "retry/sleep-based flake masking"
|
||||
relaxation:
|
||||
did_relax_any_assertion: false
|
||||
if_true_explain_spec_basis: ""
|
||||
```
|
||||
|
||||
### Test portfolio guidance (scenario-first, still layered)
|
||||
|
||||
We want a balanced suite:
|
||||
|
||||
* **Scenario/E2E**: user-story validation at boundaries (few, high value, high fidelity)
|
||||
* **Integration/contract**: service boundaries, serialization, persistence, workflows
|
||||
* **Unit/invariants**: properties that kill trivial cheats (“return true” shouldn’t survive)
|
||||
|
||||
For cheat resistance, strongly prefer:
|
||||
|
||||
* invariants/property-based tests for critical logic
|
||||
* metamorphic relations where “exact expected output” is hard
|
||||
* periodic mutation testing to “test the tests” (surviving mutants indicate missing assertions)
|
||||
|
||||
### External dependency realism (service virtualization / “digital twins”)
|
||||
|
||||
Where third-party services are involved, we prefer deterministic “behavioral clones” (mocks/stubs at the API boundary that reproduce edge cases and contracts) over live calls. This keeps scenarios **realistic** without being flaky or rate-limited, and it makes high-volume scenario validation affordable. ([factory.strongdm.ai][2])
|
||||
|
||||
---
|
||||
|
||||
## Role: Test Integrity & Scenario Spec Agent (Enforcer)
|
||||
|
||||
This repo includes (or assumes) a dedicated agent role whose only job is to protect validation integrity.
|
||||
|
||||
**This agent’s goal is NOT “make CI green.”**
|
||||
Its goal is “make the suite reflect reality and reject fake passes.”
|
||||
|
||||
**Authority:**
|
||||
|
||||
* May add new scenarios, invariants, and tests.
|
||||
* May refactor tests to be more behavioral and less procedural.
|
||||
* May improve determinism (fixed seeds, fake clocks, hermetic env).
|
||||
* May only change expectations via the Spec Correction Protocol above.
|
||||
|
||||
**Prohibitions:**
|
||||
|
||||
* Must never weaken tests to pass.
|
||||
* Must never edit holdout scenarios (if configured).
|
||||
* Must never solve flakiness via retries/sleeps/loosening.
|
||||
|
||||
**Deliverables each run:**
|
||||
|
||||
* Scenario list (Given/When/Then)
|
||||
* Proposed test diffs
|
||||
* Test Integrity Note
|
||||
* “Cheat vectors blocked” summary (what trivial implementations it prevents)
|
||||
Reference in new issue
Block a user