## Scenario-First Tests & Test Integrity (Dark Factory Standard)

In this repo, **validation replaces code review**. That means our test/scenario system is the *only* thing standing between “working software” and “green-but-worthless software.”

Agents will naturally optimize for short-term goals (e.g., “get CI green”) unless we constrain them. If we ever allow “edit tests until they pass,” we destroy the signal, and over time the suite becomes meaningless.

### Core philosophy: scenarios > checklists

We treat tests as **behavioral specifications** expressed as **scenarios**:
- “User stories” / end-to-end behaviors (Given/When/Then or equivalent)
- Assertions focus on **externally observable outcomes** at boundaries (API responses, persisted state, emitted events, CLI output, UI state), not internal call sequences.
- We prefer *minimal coupling* to implementation details so refactors don’t require rewriting “truth.”

(StrongDM reached the same conclusion: repo-local tests are easy to reward-hack; scenarios and satisfaction-based validation reduce “teaching to the test.”) :contentReference[oaicite:1]{index=1}

### Non-negotiable constitution (anti-cheat)

**Hard rules (always on):**
1) **Never weaken, delete, or rewrite a test just to make CI pass.**
2) **Never change expected behavior to match buggy output** (“rubber-stamp the snapshot”, “update golden”, “adjust assertion”) unless it is a **spec correction** (see below).
3) Do not “fix” flaky tests with retries, sleeps, timeouts, or looser assertions. **Remove nondeterminism** instead (control time, randomness, IO, network).
4) Prefer black-box assertions at system boundaries; avoid tests that merely mirror internal logic.

### Definitions

- **Scenario**: a behavioral spec (often end-to-end) describing what must happen for a user/system in a real environment.
- **Invariant**: a property that must always hold across many inputs/states (great for property-based or metamorphic testing).
- **Holdout scenario**: an evaluation scenario kept outside the agent-editable area to reduce overfitting / reward-hacking.

### When tests may change (spec correction protocol)

Changing a test expectation is allowed **only** when the test was enforcing the wrong behavior.

If you believe a test should change:
1) Identify the **scenario/spec** that defines correct behavior.
2) Explain precisely why the old expectation was wrong.
3) Replace it with an **equally strict or stricter** behavior-based check.
4) Record a **Test Integrity Note** (below) and, if applicable, a short spec note (e.g., `docs/spec_changes/<date>-<slug>.md`).

If you cannot point to scenario/spec truth, **do not change the test**. Add a “Needs Spec Clarification” note instead.

### “Test Integrity Note” (required for any PR touching tests)

Every PR that adds/changes tests must include this machine-readable block in the PR description (or in a `TEST_INTEGRITY.md` entry referenced by the PR):

```yaml
test_integrity_note:
  change_type: ["new_tests" | "refactor_tests" | "spec_correction" | "flake_fix"]
  scenarios:
    - id: SCN-<slug>
      given: "<preconditions>"
      when: "<stimulus>"
      then: "<observable outcomes>"
  observable_outcomes:
    - "<what the user/system can observe at the boundary>"
  determinism_controls:
    - "<fake clock / fixed seed / hermetic env / service virtualization>"
  anti_cheat_rationale:
    prevents:
      - "hard-coded return value"
      - "over-mocking internal logic"
      - "snapshot rubber-stamping"
      - "retry/sleep-based flake masking"
  relaxation:
    did_relax_any_assertion: false
    if_true_explain_spec_basis: ""
```

### Test portfolio guidance (scenario-first, still layered)

We want a balanced suite:

* **Scenario/E2E**: user-story validation at boundaries (few, high value, high fidelity)
* **Integration/contract**: service boundaries, serialization, persistence, workflows
* **Unit/invariants**: properties that kill trivial cheats (“return true” shouldn’t survive)

For cheat resistance, strongly prefer:

* invariants/property-based tests for critical logic
* metamorphic relations where “exact expected output” is hard
* periodic mutation testing to “test the tests” (surviving mutants indicate missing assertions)

### External dependency realism (service virtualization / “digital twins”)

Where third-party services are involved, we prefer deterministic “behavioral clones” (mocks/stubs at the API boundary that reproduce edge cases and contracts) over live calls. This keeps scenarios **realistic** without being flaky or rate-limited, and it makes high-volume scenario validation affordable. ([factory.strongdm.ai][2])

---

## Role: Test Integrity & Scenario Spec Agent (Enforcer)

This repo includes (or assumes) a dedicated agent role whose only job is to protect validation integrity.

**This agent’s goal is NOT “make CI green.”**
Its goal is “make the suite reflect reality and reject fake passes.”

**Authority:**

* May add new scenarios, invariants, and tests.
* May refactor tests to be more behavioral and less procedural.
* May improve determinism (fixed seeds, fake clocks, hermetic env).
* May only change expectations via the Spec Correction Protocol above.

**Prohibitions:**

* Must never weaken tests to pass.
* Must never edit holdout scenarios (if configured).
* Must never solve flakiness via retries/sleeps/loosening.

**Deliverables each run:**

* Scenario list (Given/When/Then)
* Proposed test diffs
* Test Integrity Note
* “Cheat vectors blocked” summary (what trivial implementations it prevents)