Agent Test Format

This document defines the repeatable "agent-style" validation format for SeedPass TUI v2, CLI/CUI, and API surfaces.

Goals

Reproduce user and agent workflows deterministically.
Generate machine-readable artifacts for regressions.
Keep a stable command format for local runs and CI runs.

Output Contract

Each harness run should emit:

status: passed or failed
failure: short error detail when failed
duration_sec: run duration
steps: per-step status/timing
coverage_points: workflow checkpoints as booleans
optional transcripts/activity logs

Reports are written to timestamped artifact folders under artifacts/.

TUI v2 Harness

Script:

python3 scripts/ai_tui2_agent_test.py --scenario extended --verbose

Artifacts:

artifacts/agent_tui2_test/<timestamp>/report.json

Scenarios:

core: startup/search/document edit/graph open/archive-restore
extended: core + palette unknown command + help overlay/navigation checks

Legacy Interactive Harness

Script:

python3 scripts/ai_tui_agent_test.py --scenario extended --python-bin .venv/bin/python --verbose

Artifacts:

artifacts/agent_tui_test/<timestamp>/report.json
transcript files (transcript.clean.txt, transcript.raw.txt)

Use this for legacy interactive flow regression checks.

CLI/CUI Agent-Style Validation

Primary command suites:

pytest -q src/tests/test_cli*.py src/tests/test_typer_cli.py src/tests/test_cli_agent_mode.py

This covers user CLI commands and agent command flows.

API Agent-Style Validation

Primary command suites:

pytest -q src/tests/test_api*.py

This covers auth, policy enforcement, entry/document operations, and vault endpoints from an API client perspective.

TUI v2 Deep Coverage Command

To track TUI v2 function/path coverage:

pytest -q \
  src/tests/test_tui_v2_helpers.py \
  src/tests/test_tui_v2_textual_interactions.py \
  src/tests/test_tui_v2_parity_scenarios.py \
  src/tests/test_tui_v2_large_vault_validation.py \
  src/tests/test_tui_v2_action_matrix.py \
  --cov=seedpass.tui_v2.app \
  --cov-report=term-missing

Determinism Gate Integration

Run deterministic suite:

./scripts/run_determinism_tests.sh

This includes scripts/check_determinism_suite.py to enforce suite shape and minimum determinism test count before executing tests.

2.4 KiB Raw Blame History