TORCH/AGENTS.md

Scenario-First Tests & Test Integrity (Dark Factory Standard)

In this repo, validation replaces code review. That means our test/scenario system is the only thing standing between “working software” and “green-but-worthless software.”

Agents will naturally optimize for short-term goals (e.g., “get CI green”) unless we constrain them. If we ever allow “edit tests until they pass,” we destroy the signal, and over time the suite becomes meaningless.

Core philosophy: scenarios > checklists

We treat tests as behavioral specifications expressed as scenarios:

• “User stories” / end-to-end behaviors (Given/When/Then or equivalent)
• Assertions focus on externally observable outcomes at boundaries (API responses, persisted state, emitted events, CLI output, UI state), not internal call sequences.
• We prefer minimal coupling to implementation details so refactors don’t require rewriting “truth.”

(StrongDM reached the same conclusion: repo-local tests are easy to reward-hack; scenarios and satisfaction-based validation reduce “teaching to the test.”) :contentReference[oaicite:1]{index=1}

Non-negotiable constitution (anti-cheat)

Hard rules (always on):

1. Never weaken, delete, or rewrite a test just to make CI pass.
2. Never change expected behavior to match buggy output (“rubber-stamp the snapshot”, “update golden”, “adjust assertion”) unless it is a spec correction (see below).
3. Do not “fix” flaky tests with retries, sleeps, timeouts, or looser assertions. Remove nondeterminism instead (control time, randomness, IO, network).
4. Prefer black-box assertions at system boundaries; avoid tests that merely mirror internal logic.
5. Do not “hard-code to the test” (single fixtures, single input paths, brittle constants), and do not mock so heavily that tests can pass with a broken system.

Definitions

• Scenario: a behavioral spec (often end-to-end) describing what must happen for a user/system in a real environment.
• Invariant: a property that must always hold across many inputs/states (great for property-based or metamorphic testing).
• Holdout scenario: an evaluation scenario kept outside the agent-editable area to reduce overfitting / reward-hacking.

When tests may change (spec correction protocol)

Changing a test expectation is allowed only when the test was enforcing the wrong behavior.

If you believe a test should change:

1. Identify the scenario/spec that defines correct behavior.
2. Explain precisely why the old expectation was wrong.
3. Replace it with an equally strict or stricter behavior-based check.
4. Record a Test Integrity Note (below) and, if applicable, a short spec note (e.g., docs/spec_changes/<date>-<slug>.md).

If you cannot point to scenario/spec truth, do not change the test. Add a “Needs Spec Clarification” note instead.

“Test Integrity Note” (required for any PR touching tests)

Every PR that adds/changes tests must include this machine-readable block in the PR description (or in a TEST_INTEGRITY.md entry referenced by the PR):

test_integrity_note:
  change_type: ["new_tests" | "refactor_tests" | "spec_correction" | "flake_fix"]
  scenarios:
    - id: SCN-<slug>
      given: "<preconditions>"
      when: "<stimulus>"
      then: "<observable outcomes>"
  observable_outcomes:
    - "<what the user/system can observe at the boundary>"
  determinism_controls:
    - "<fake clock / fixed seed / hermetic env / service virtualization>"
  anti_cheat_rationale:
    prevents:
      - "hard-coded return value"
      - "over-mocking internal logic"
      - "snapshot rubber-stamping"
      - "retry/sleep-based flake masking"
  relaxation:
    did_relax_any_assertion: false
    if_true_explain_spec_basis: ""

Test portfolio guidance (scenario-first, still layered)

We want a balanced suite:

• Scenario/E2E: user-story validation at boundaries (few, high value, high fidelity)
• Integration/contract: service boundaries, serialization, persistence, workflows
• Unit/invariants: properties that kill trivial cheats (“return true” shouldn’t survive)

For cheat resistance, strongly prefer:

• invariants/property-based tests for critical logic
• metamorphic relations where “exact expected output” is hard
• periodic mutation testing to “test the tests” (surviving mutants indicate missing assertions)

External dependency realism (service virtualization / “digital twins”)

Where third-party services are involved, we prefer deterministic “behavioral clones” (mocks/stubs at the API boundary that reproduce edge cases and contracts) over live calls. This keeps scenarios realistic without being flaky or rate-limited, and it makes high-volume scenario validation affordable. ([factory.strongdm.ai][2])

---

Role: Test Integrity & Scenario Spec Agent (Enforcer)

This repo includes (or assumes) a dedicated agent role whose only job is to protect validation integrity.

This agent’s goal is NOT “make CI green.” Its goal is “make the suite reflect reality and reject fake passes.”

Authority:

• May add new scenarios, invariants, and tests.
• May refactor tests to be more behavioral and less procedural.
• May improve determinism (fixed seeds, fake clocks, hermetic env).
• May only change expectations via the Spec Correction Protocol above.

Prohibitions:

• Must never weaken tests to pass.
• Must never edit holdout scenarios (if configured).
• Must never solve flakiness via retries/sleeps/loosening.

Deliverables each run:

• Scenario list (Given/When/Then)
• Proposed test diffs
• Test Integrity Note
• “Cheat vectors blocked” summary (what trivial implementations it prevents)

---

File Organization & Cleanliness (Clean Repo Standard)

Agents must keep the repository root clean. Do not create log files, temporary outputs, or artifacts in the root directory.

Standard Locations:

• Reports: reports/<category>/ (e.g., reports/performance/, reports/audit/). See Report Location Standard.
• Test Logs: test_logs/ (e.g., test_logs/test_output_full.txt).
• Task Logs: task-logs/ (e.g., task-logs/daily/).
• Artifacts: artifacts/ (for raw data, repros, or other artifacts).
• Scripts: scripts/ (e.g., scripts/test-audit/).
• General Logs: logs/ (if needed, otherwise task-logs/).

Prohibitions:

• No *.log, *.txt, *.json, or *.md files in the root directory (except standard docs like README.md, AGENTS.md, CLAUDE.md, KNOWN_ISSUES.md).
• If a tool or script defaults to writing to root, redirect its output or configure it to write to one of the standard locations.

Report Location Standard

All agents that generate reports must adhere to the following directory structure. Reports must be placed in reports/<category>/.

Categories

• Audit: reports/audit/ (e.g., audit-report-YYYY-MM-DD.md)
• Design System: reports/design-system/
• Known Issues: reports/known-issues/
• Load Test: reports/load-test/
• Performance: reports/performance/ (e.g., daily-perf-report-YYYY-MM-DD.md)
• Protocol: reports/protocol/ (e.g., protocol-report-YYYY-MM-DD.md)
• Security: reports/security/
• Telemetry: reports/telemetry/
• Test Audit: reports/test-audit/
• Weekly Synthesis: reports/weekly-synthesis/

Format

• Use Markdown (.md) for human-readable reports.
• Use JSON (.json) or Log (.log) for raw data/artifacts, but prefer placing a summary Markdown report in the corresponding folder.
• Naming convention: <type>-report-YYYY-MM-DD.md (e.g., daily-perf-report-2024-01-01.md).

Repo memory

Curated agent memory lives in .agents/ (index: .agents/MEMORY.md). Read it before substantive work. Propose additions as files in .agents/proposals/; trusted memory under .agents/memory/ changes only through reviewed commits. Code, tests, and configuration always outrank memory.