mirror of
https://github.com/PR0M3TH3AN/SeedPass.git
synced 2026-11-06 12:52:04 +00:00
Strengthen determinism gates and add TUI v2 agent testing workflow
This commit is contained in:
1 parent
feb4e0df8c
commit
3470fd6aac
25 files changed
+1495
-12
No files matched your search
@@ -0,0 +1,102 @@
|
||||
# Agent Test Format
|
||||
|
||||
This document defines the repeatable "agent-style" validation format for
|
||||
SeedPass TUI v2, CLI/CUI, and API surfaces.
|
||||
|
||||
## Goals
|
||||
|
||||
- Reproduce user and agent workflows deterministically.
|
||||
- Generate machine-readable artifacts for regressions.
|
||||
- Keep a stable command format for local runs and CI runs.
|
||||
|
||||
## Output Contract
|
||||
|
||||
Each harness run should emit:
|
||||
|
||||
- `status`: `passed` or `failed`
|
||||
- `failure`: short error detail when failed
|
||||
- `duration_sec`: run duration
|
||||
- `steps`: per-step status/timing
|
||||
- `coverage_points`: workflow checkpoints as booleans
|
||||
- optional transcripts/activity logs
|
||||
|
||||
Reports are written to timestamped artifact folders under `artifacts/`.
|
||||
|
||||
## TUI v2 Harness
|
||||
|
||||
Script:
|
||||
|
||||
```bash
|
||||
python3 scripts/ai_tui2_agent_test.py --scenario extended --verbose
|
||||
```
|
||||
|
||||
Artifacts:
|
||||
|
||||
- `artifacts/agent_tui2_test/<timestamp>/report.json`
|
||||
|
||||
Scenarios:
|
||||
|
||||
- `core`: startup/search/document edit/graph open/archive-restore
|
||||
- `extended`: `core` + palette unknown command + help overlay/navigation checks
|
||||
|
||||
## Legacy Interactive Harness
|
||||
|
||||
Script:
|
||||
|
||||
```bash
|
||||
python3 scripts/ai_tui_agent_test.py --scenario extended --python-bin .venv/bin/python --verbose
|
||||
```
|
||||
|
||||
Artifacts:
|
||||
|
||||
- `artifacts/agent_tui_test/<timestamp>/report.json`
|
||||
- transcript files (`transcript.clean.txt`, `transcript.raw.txt`)
|
||||
|
||||
Use this for legacy interactive flow regression checks.
|
||||
|
||||
## CLI/CUI Agent-Style Validation
|
||||
|
||||
Primary command suites:
|
||||
|
||||
```bash
|
||||
pytest -q src/tests/test_cli*.py src/tests/test_typer_cli.py src/tests/test_cli_agent_mode.py
|
||||
```
|
||||
|
||||
This covers user CLI commands and agent command flows.
|
||||
|
||||
## API Agent-Style Validation
|
||||
|
||||
Primary command suites:
|
||||
|
||||
```bash
|
||||
pytest -q src/tests/test_api*.py
|
||||
```
|
||||
|
||||
This covers auth, policy enforcement, entry/document operations, and vault
|
||||
endpoints from an API client perspective.
|
||||
|
||||
## TUI v2 Deep Coverage Command
|
||||
|
||||
To track TUI v2 function/path coverage:
|
||||
|
||||
```bash
|
||||
pytest -q \
|
||||
src/tests/test_tui_v2_helpers.py \
|
||||
src/tests/test_tui_v2_textual_interactions.py \
|
||||
src/tests/test_tui_v2_parity_scenarios.py \
|
||||
src/tests/test_tui_v2_large_vault_validation.py \
|
||||
src/tests/test_tui_v2_action_matrix.py \
|
||||
--cov=seedpass.tui_v2.app \
|
||||
--cov-report=term-missing
|
||||
```
|
||||
|
||||
## Determinism Gate Integration
|
||||
|
||||
Run deterministic suite:
|
||||
|
||||
```bash
|
||||
./scripts/run_determinism_tests.sh
|
||||
```
|
||||
|
||||
This includes `scripts/check_determinism_suite.py` to enforce suite shape and
|
||||
minimum determinism test count before executing tests.
|
||||
Reference in new issue
Block a user