mirror of
https://github.com/PR0M3TH3AN/SeedPass.git
synced 2026-10-27 16:02:05 +00:00
thePR0M3TH3AN
03958106e1
This commit delivers four related improvements focused on reliability and automation while preserving existing human/TUI workflows. 1) Agent-first CLI mode (additive) - Added new `seedpass agent` command group (`src/seedpass/cli/agent.py`) and wired it into CLI root. - Implemented non-interactive profile bootstrap: `seedpass agent init` with `--seed|--seed-file|--generate-seed` and password env injection (`--password-env`). - Added policy management commands: `agent policy-show` and `agent policy-set` with allow/deny kind controls. - Added JSON-based secret retrieval: `seedpass -f <fingerprint> agent get <query> --ttl <sec>` with lease metadata and policy enforcement. - Kept regular human mode fully available; no behavioral changes to interactive TUI entrypoint. 2) TUI agent harness deepening and stabilization - Refactored harness into scenario profiles: `core`, `extended`, `stress` with `--scenario` selection. - Added deterministic seeded stress campaigns: `--stress-cycles`, `--stress-seed`. - Added scenario-specific required coverage keys and report metadata. - Added post-condition assertions in extended scenario (archive/restore consistency, lock/unlock retrieval recovery). - Hardened prompt matching for output variants and fail-fast behavior on non-zero SeedPass exit codes. - Added harness unit tests for helper logic and control flow. 3) Critical module coverage gate tooling - Added `scripts/check_critical_coverage.py` to enforce per-module thresholds from coverage JSON and report uncovered functions. - Added override controls (`--threshold`, `--no-default-thresholds`) and optional JSON result output. - Integrated coverage JSON generation into CI helper script and optional gate execution via `CRITICAL_COVERAGE_GATE=1`. - Tuned default thresholds using full-suite baseline and increased strictness: - `src/main.py >= 50%` - `src/seedpass/core/manager.py >= 60%` - `src/seedpass/core/encryption.py >= 80%` - `src/seedpass/core/migrations.py >= 90%` - Added tests for coverage gate parser/analysis logic and targeted manager worker path tests. 4) Installer validation framework for beta/main - Added cross-platform installer smoke scripts: - `scripts/installer_smoke_unix.sh` - `scripts/installer_smoke_windows.ps1` - Added GitHub Actions workflow `.github/workflows/installer-smoke.yml` with: - reduced PR matrix - full scheduled/manual matrix across OS/branch/mode - per-job log capture and artifact uploads for failure triage - Added installer testing documentation and linked from README. Additional housekeeping - Added `.venv/` to `.gitignore`. - Updated docs for new harness scenarios, seeded stress mode, post-condition metadata, coverage gate usage, and installer testing workflow.
150 lines
4.2 KiB
Markdown
150 lines
4.2 KiB
Markdown
# AI Agent TUI Testing
|
|
|
|
This document defines a repeatable, methodical TUI test workflow for AI agents.
|
|
|
|
## Goal
|
|
|
|
Run SeedPass as a real interactive user and produce deterministic artifacts:
|
|
|
|
1. Step-level pass/fail.
|
|
2. Coverage checkpoints for major TUI features.
|
|
3. Full transcript for debugging and regression comparison.
|
|
|
|
## Runner
|
|
|
|
Use:
|
|
|
|
```bash
|
|
source .venv/bin/activate
|
|
python scripts/ai_tui_agent_test.py
|
|
```
|
|
|
|
The runner:
|
|
|
|
- creates an isolated temporary `HOME`
|
|
- launches `src/main.py --no-clipboard`
|
|
- executes a scenario profile (`core`, `extended`, or `stress`)
|
|
- writes artifacts to `artifacts/agent_tui_test/<timestamp>/`
|
|
|
|
## Scenario Profiles
|
|
|
|
- `core`: fast smoke path (onboarding, invalid input checks, add/retrieve/search/list, exit)
|
|
- `extended`: full end-to-end path including all entry kinds, archive/restore, and settings flows
|
|
- `stress`: deterministic repeated negative-input and back-navigation loops
|
|
|
|
## Agent Commands Today
|
|
|
|
Use these command tiers when agents are validating changes:
|
|
|
|
1. Fast targeted checks (recommended during iteration):
|
|
|
|
```bash
|
|
source .venv/bin/activate
|
|
pytest -q src/tests/test_manager_add_password.py src/tests/test_archive_restore.py src/tests/test_stats_screen.py src/tests/test_bip85_init.py
|
|
pytest -q src/tests/test_ai_tui_agent_harness.py
|
|
python scripts/ai_tui_agent_test.py --scenario core
|
|
```
|
|
|
|
2. Full regression check (before merge):
|
|
|
|
```bash
|
|
source .venv/bin/activate
|
|
pytest -q
|
|
pytest -q --cov=src --cov-report=json:artifacts/coverage/coverage.json src/tests
|
|
python scripts/check_critical_coverage.py artifacts/coverage/coverage.json
|
|
python scripts/ai_tui_agent_test.py --scenario extended
|
|
```
|
|
|
|
3. Stability sweep for TUI prompt timing:
|
|
|
|
```bash
|
|
source .venv/bin/activate
|
|
python scripts/ai_tui_agent_test.py --scenario stress --stress-cycles 6 --stress-seed 1337
|
|
```
|
|
|
|
## Artifacts
|
|
|
|
Each run writes:
|
|
|
|
- `report.json`: status, step results, coverage points, failure info.
|
|
- `transcript.clean.txt`: ANSI-stripped terminal output.
|
|
- `transcript.raw.txt`: raw terminal output with ANSI.
|
|
|
|
`report.json` also includes scenario-specific validation metadata:
|
|
|
|
- `required_coverage_keys`
|
|
- `required_post_conditions`
|
|
- `post_conditions`
|
|
|
|
## Current Coverage Points
|
|
|
|
The harness marks these checkpoints as `true` only when observed:
|
|
|
|
- `startup_onboarding`
|
|
- `main_menu`
|
|
- `invalid_input_resilience`
|
|
- `add_password_quick`
|
|
- `invalid_length_validation`
|
|
- `add_totp`
|
|
- `add_all_entry_types`
|
|
- `retrieve_entry`
|
|
- `search_entries`
|
|
- `list_entries`
|
|
- `modify_entry`
|
|
- `archive_restore`
|
|
- `totp_codes_view`
|
|
- `settings_toggles_lock_unlock`
|
|
- `settings_stats`
|
|
- `graceful_exit`
|
|
|
|
`extended` scenario also enforces post-conditions:
|
|
|
|
- `archive_restore_consistency`: restored entries are no longer listed in archived view
|
|
- `lock_unlock_recovers_retrieval`: retrieval still works after lock/unlock
|
|
|
|
## CI/Agent Use Pattern
|
|
|
|
Recommended agent sequence:
|
|
|
|
1. `python scripts/ai_tui_agent_test.py`
|
|
2. Parse `report.json`
|
|
3. Fail task if `status != "passed"` or any checkpoint is `false`
|
|
4. If failed, inspect `transcript.clean.txt` for prompt-flow drift or regressions
|
|
5. Inspect `artifacts/coverage/coverage.json` and critical coverage gate output when module thresholds fail
|
|
|
|
## Optional Flags
|
|
|
|
```bash
|
|
python scripts/ai_tui_agent_test.py --help
|
|
```
|
|
|
|
Useful options:
|
|
|
|
- `--scenario`: `core`, `extended`, or `stress`
|
|
- `--stress-cycles`: number of loops for `stress` profile
|
|
- `--stress-seed`: fixed RNG seed for deterministic invalid-input campaigns in `stress`
|
|
- `--timeout`: per-prompt timeout
|
|
- `--password`: fixed password used during test
|
|
- `--output-dir`: custom artifact root
|
|
- `--keep-home`: preserve generated profile state for local debugging
|
|
- `--verbose`: print step progress
|
|
|
|
Coverage gate helper:
|
|
|
|
- `python scripts/check_critical_coverage.py artifacts/coverage/coverage.json`
|
|
- Use `--threshold PATH=PERCENT` to override defaults.
|
|
- Use `--no-default-thresholds` for focused local smoke checks.
|
|
|
|
## Extending Coverage
|
|
|
|
When adding new TUI flows:
|
|
|
|
1. Add deterministic steps to `scripts/ai_tui_agent_test.py`.
|
|
2. Add a new coverage checkpoint key.
|
|
3. Assert on concrete prompt/output text, not timing.
|
|
4. Keep inputs deterministic and avoid clipboard-dependent behavior.
|
|
|
|
## Roadmap
|
|
|
|
Future testing milestones are tracked in `docs/agent_testing_roadmap.md`.
|