mirror of
https://github.com/PR0M3TH3AN/SeedPass.git
synced 2026-10-27 16:02:05 +00:00
103 lines
2.4 KiB
Markdown
103 lines
2.4 KiB
Markdown
# Agent Test Format
|
|
|
|
This document defines the repeatable "agent-style" validation format for
|
|
SeedPass TUI v2, CLI/CUI, and API surfaces.
|
|
|
|
## Goals
|
|
|
|
- Reproduce user and agent workflows deterministically.
|
|
- Generate machine-readable artifacts for regressions.
|
|
- Keep a stable command format for local runs and CI runs.
|
|
|
|
## Output Contract
|
|
|
|
Each harness run should emit:
|
|
|
|
- `status`: `passed` or `failed`
|
|
- `failure`: short error detail when failed
|
|
- `duration_sec`: run duration
|
|
- `steps`: per-step status/timing
|
|
- `coverage_points`: workflow checkpoints as booleans
|
|
- optional transcripts/activity logs
|
|
|
|
Reports are written to timestamped artifact folders under `artifacts/`.
|
|
|
|
## TUI v2 Harness
|
|
|
|
Script:
|
|
|
|
```bash
|
|
python3 scripts/ai_tui2_agent_test.py --scenario extended --verbose
|
|
```
|
|
|
|
Artifacts:
|
|
|
|
- `artifacts/agent_tui2_test/<timestamp>/report.json`
|
|
|
|
Scenarios:
|
|
|
|
- `core`: startup/search/document edit/graph open/archive-restore
|
|
- `extended`: `core` + palette unknown command + help overlay/navigation checks
|
|
|
|
## Legacy Interactive Harness
|
|
|
|
Script:
|
|
|
|
```bash
|
|
python3 scripts/ai_tui_agent_test.py --scenario extended --python-bin .venv/bin/python --verbose
|
|
```
|
|
|
|
Artifacts:
|
|
|
|
- `artifacts/agent_tui_test/<timestamp>/report.json`
|
|
- transcript files (`transcript.clean.txt`, `transcript.raw.txt`)
|
|
|
|
Use this for legacy interactive flow regression checks.
|
|
|
|
## CLI/CUI Agent-Style Validation
|
|
|
|
Primary command suites:
|
|
|
|
```bash
|
|
pytest -q src/tests/test_cli*.py src/tests/test_typer_cli.py src/tests/test_cli_agent_mode.py
|
|
```
|
|
|
|
This covers user CLI commands and agent command flows.
|
|
|
|
## API Agent-Style Validation
|
|
|
|
Primary command suites:
|
|
|
|
```bash
|
|
pytest -q src/tests/test_api*.py
|
|
```
|
|
|
|
This covers auth, policy enforcement, entry/document operations, and vault
|
|
endpoints from an API client perspective.
|
|
|
|
## TUI v2 Deep Coverage Command
|
|
|
|
To track TUI v2 function/path coverage:
|
|
|
|
```bash
|
|
pytest -q \
|
|
src/tests/test_tui_v2_helpers.py \
|
|
src/tests/test_tui_v2_textual_interactions.py \
|
|
src/tests/test_tui_v2_parity_scenarios.py \
|
|
src/tests/test_tui_v2_large_vault_validation.py \
|
|
src/tests/test_tui_v2_action_matrix.py \
|
|
--cov=seedpass.tui_v2.app \
|
|
--cov-report=term-missing
|
|
```
|
|
|
|
## Determinism Gate Integration
|
|
|
|
Run deterministic suite:
|
|
|
|
```bash
|
|
./scripts/run_determinism_tests.sh
|
|
```
|
|
|
|
This includes `scripts/check_determinism_suite.py` to enforce suite shape and
|
|
minimum determinism test count before executing tests.
|