0 Fork 0

mirror of https://github.com/PR0M3TH3AN/SeedPass.git synced 2026-10-27 07:52:03 +00:00

Files

T

thePR0M3TH3AN 03958106e1 Add agent mode, hardened TUI harness, coverage gates, and cross-platform installer smoke CI

This commit delivers four related improvements focused on reliability and automation while preserving existing human/TUI workflows.

1) Agent-first CLI mode (additive)
- Added new `seedpass agent` command group (`src/seedpass/cli/agent.py`) and wired it into CLI root.
- Implemented non-interactive profile bootstrap: `seedpass agent init` with `--seed|--seed-file|--generate-seed` and password env injection (`--password-env`).
- Added policy management commands: `agent policy-show` and `agent policy-set` with allow/deny kind controls.
- Added JSON-based secret retrieval: `seedpass -f <fingerprint> agent get <query> --ttl <sec>` with lease metadata and policy enforcement.
- Kept regular human mode fully available; no behavioral changes to interactive TUI entrypoint.

2) TUI agent harness deepening and stabilization
- Refactored harness into scenario profiles: `core`, `extended`, `stress` with `--scenario` selection.
- Added deterministic seeded stress campaigns: `--stress-cycles`, `--stress-seed`.
- Added scenario-specific required coverage keys and report metadata.
- Added post-condition assertions in extended scenario (archive/restore consistency, lock/unlock retrieval recovery).
- Hardened prompt matching for output variants and fail-fast behavior on non-zero SeedPass exit codes.
- Added harness unit tests for helper logic and control flow.

3) Critical module coverage gate tooling
- Added `scripts/check_critical_coverage.py` to enforce per-module thresholds from coverage JSON and report uncovered functions.
- Added override controls (`--threshold`, `--no-default-thresholds`) and optional JSON result output.
- Integrated coverage JSON generation into CI helper script and optional gate execution via `CRITICAL_COVERAGE_GATE=1`.
- Tuned default thresholds using full-suite baseline and increased strictness:
- `src/main.py >= 50%`
- `src/seedpass/core/manager.py >= 60%`
- `src/seedpass/core/encryption.py >= 80%`
- `src/seedpass/core/migrations.py >= 90%`
- Added tests for coverage gate parser/analysis logic and targeted manager worker path tests.

4) Installer validation framework for beta/main
- Added cross-platform installer smoke scripts:
- `scripts/installer_smoke_unix.sh`
- `scripts/installer_smoke_windows.ps1`
- Added GitHub Actions workflow `.github/workflows/installer-smoke.yml` with:
- reduced PR matrix
- full scheduled/manual matrix across OS/branch/mode
- per-job log capture and artifact uploads for failure triage
- Added installer testing documentation and linked from README.

Additional housekeeping
- Added `.venv/` to `.gitignore`.
- Updated docs for new harness scenarios, seeded stress mode, post-condition metadata, coverage gate usage, and installer testing workflow.

2026-02-23 15:47:59 -05:00

4.2 KiB

Raw Blame History

AI Agent TUI Testing

This document defines a repeatable, methodical TUI test workflow for AI agents.

Goal

Run SeedPass as a real interactive user and produce deterministic artifacts:

Step-level pass/fail.
Coverage checkpoints for major TUI features.
Full transcript for debugging and regression comparison.

Runner

Use:

source .venv/bin/activate
python scripts/ai_tui_agent_test.py

The runner:

creates an isolated temporary HOME
launches src/main.py --no-clipboard
executes a scenario profile (core, extended, or stress)
writes artifacts to artifacts/agent_tui_test/<timestamp>/

Scenario Profiles

core: fast smoke path (onboarding, invalid input checks, add/retrieve/search/list, exit)
extended: full end-to-end path including all entry kinds, archive/restore, and settings flows
stress: deterministic repeated negative-input and back-navigation loops

Agent Commands Today

Use these command tiers when agents are validating changes:

Fast targeted checks (recommended during iteration):

source .venv/bin/activate
pytest -q src/tests/test_manager_add_password.py src/tests/test_archive_restore.py src/tests/test_stats_screen.py src/tests/test_bip85_init.py
pytest -q src/tests/test_ai_tui_agent_harness.py
python scripts/ai_tui_agent_test.py --scenario core

Full regression check (before merge):

source .venv/bin/activate
pytest -q
pytest -q --cov=src --cov-report=json:artifacts/coverage/coverage.json src/tests
python scripts/check_critical_coverage.py artifacts/coverage/coverage.json
python scripts/ai_tui_agent_test.py --scenario extended

Stability sweep for TUI prompt timing:

source .venv/bin/activate
python scripts/ai_tui_agent_test.py --scenario stress --stress-cycles 6 --stress-seed 1337

Artifacts

Each run writes:

report.json: status, step results, coverage points, failure info.
transcript.clean.txt: ANSI-stripped terminal output.
transcript.raw.txt: raw terminal output with ANSI.

report.json also includes scenario-specific validation metadata:

required_coverage_keys
required_post_conditions
post_conditions

Current Coverage Points

The harness marks these checkpoints as true only when observed:

startup_onboarding
main_menu
invalid_input_resilience
add_password_quick
invalid_length_validation
add_totp
add_all_entry_types
retrieve_entry
search_entries
list_entries
modify_entry
archive_restore
totp_codes_view
settings_toggles_lock_unlock
settings_stats
graceful_exit

extended scenario also enforces post-conditions:

archive_restore_consistency: restored entries are no longer listed in archived view
lock_unlock_recovers_retrieval: retrieval still works after lock/unlock

CI/Agent Use Pattern

Recommended agent sequence:

python scripts/ai_tui_agent_test.py
Parse report.json
Fail task if status != "passed" or any checkpoint is false
If failed, inspect transcript.clean.txt for prompt-flow drift or regressions
Inspect artifacts/coverage/coverage.json and critical coverage gate output when module thresholds fail

Optional Flags

python scripts/ai_tui_agent_test.py --help

Useful options:

--scenario: core, extended, or stress
--stress-cycles: number of loops for stress profile
--stress-seed: fixed RNG seed for deterministic invalid-input campaigns in stress
--timeout: per-prompt timeout
--password: fixed password used during test
--output-dir: custom artifact root
--keep-home: preserve generated profile state for local debugging
--verbose: print step progress

Coverage gate helper:

python scripts/check_critical_coverage.py artifacts/coverage/coverage.json
Use --threshold PATH=PERCENT to override defaults.
Use --no-default-thresholds for focused local smoke checks.

Extending Coverage

When adding new TUI flows:

Add deterministic steps to scripts/ai_tui_agent_test.py.
Add a new coverage checkpoint key.
Assert on concrete prompt/output text, not timing.
Keep inputs deterministic and avoid clipboard-dependent behavior.

Roadmap

Future testing milestones are tracked in docs/agent_testing_roadmap.md.

4.2 KiB Raw Blame History