docs(test): centralize test infrastructure and add interactive agent harness

Added docs/TEST_INVENTORY.md as a central index for all tests and runners.
Created scripts/interactive_agent_tui_test.py for password-free TUI v2
exploratory testing. Integrated standalone permission tests into main
suite and fixed TUI v2 parity regression tests broken by auto-selection
removal. Updated AGENTS.md and CLAUDE.md with strategic plan references.
This commit is contained in:
thePR0M3TH3AN committed 2026-03-02 22:26:35 -05:00
1 parent 165c825d9e
commit 4eeb936e6d
9 files changed
+351 -45

No files matched your search

+19 -9
View File
@@ -56,22 +56,32 @@ cd ~/.seedpass/app && source venv/bin/activate
cd src && python main.py
```
## Planning & Strategy
Before starting work, AI agents must consult the following documents to align with the current project trajectory:
1. **[Dev Control Center](docs/dev_control_center.md)**: The single source of truth for current priorities and "what to do next."
2. **[TUI v2 Execution Plan](docs/tui_v2_integration_execution_plan_2026-03-02.md)**: The active roadmap for the TUI v2 mockup parity and hardening phase.
## Running Tests
SeedPass maintains a comprehensive test infrastructure. Agents should use the central inventory as a guide:
1. **[Test Infrastructure Inventory](docs/TEST_INVENTORY.md)**: A complete index of all unit, integration, UI, and determinism tests.
### Quick Start: Running Tests
1. Set up a virtual environment and install dependencies:
```bash
python3 -m venv venv
source venv/bin/activate
pip install --require-hashes -r requirements.lock
```
2. Run the test suite using **pytest**:
...
```bash
pytest
```
For a full CI-equivalent run (highly recommended before any PR), use:
```bash
bash scripts/run_ci_tests.sh
```
Currently the test folder is located in `src/tests/`. New tests should be placed there so `pytest` can discover them automatically.
## Style Guidelines
+12
View File
@@ -1,3 +1,15 @@
# SeedPass Agent Guide
## Planning & Strategy
Consult these first to align with current objectives:
- **[Dev Control Center](docs/dev_control_center.md)**: Current priorities.
- **[TUI v2 Execution Plan](docs/tui_v2_integration_execution_plan_2026-03-02.md)**: Roadmap for TUI v2.
## Testing
Always run the full suite before proposing changes:
- `bash scripts/run_ci_tests.sh`
- Central Index: **[Test Infrastructure Inventory](docs/TEST_INVENTORY.md)**
## TORCH Memory Integration
You have access to the TORCH memory system.
1. READ: Check `.scheduler-memory/latest/${cadence}/memories.md` for past learnings.
+38
View File
@@ -0,0 +1,38 @@
# Interactive Agent TUI Testing Protocol
This document defines how AI agents should perform exploratory and regression testing on the SeedPass TUI v2 using the `scripts/interactive_agent_tui_test.py` harness.
## 1. Overview
The testing harness uses **Dependency Injection** to replace the real SeedPass service layer with a `MockService`. This allows agents to:
1. **Bypass Encryption:** Test the UI without knowing passwords or seeds.
2. **Avoid Side Effects:** No changes are made to the real vault or filesystem.
3. **Simulate Edge Cases:** Mock specific failures (e.g., sync errors) to see how the UI recovers.
## 2. How to Run
Execute the harness using the project's virtual environment:
```bash
.venv/bin/python scripts/interactive_agent_tui_test.py
```
## 3. How to Extend
To add a new test case:
1. **Update the Mock:** If testing a new service method, add it to `AgentMockService`.
2. **Add a Step:** In `run_full_walkthrough`, use `app._run_palette_command("your-command")` or direct action calls (e.g., `app.action_toggle_archive()`).
3. **Verify State:** Use `pilot.pause()` to let the TUI update, then query widgets:
```python
status = str(app.query_one("#status").render())
assert "Expected" in status
```
## 4. Key Testing Areas
* **Palette Commands:** Ensure `_run_palette_command` correctly routes inputs to services.
* **Selection Persistence:** Verify that the "Selected Entry" remains correct after archiving or refreshing.
* **Layout Adaptability:** Test compact vs. comfortable density.
* **Reactive UI:** Verify that `_load_entries` is called automatically after mutations.
## 5. Agent Instructions for Future Runs
If you are an agent tasked with "looking for bugs":
1. **Run the script:** Confirm the baseline walkthrough passes.
2. **Introduce Chaos:** Modify the mock to throw exceptions randomly.
3. **Check Transitions:** Rapidly switch between entries and layouts.
4. **Document Failures:** If a widget isn't found or a status message is wrong, report the line number in `app.py`.
+1
View File
@@ -46,6 +46,7 @@ This directory contains SeedPass project documentation.
## QA and Testing Docs
- `TEST_INVENTORY.md` (central index of all tests and runners)
- `agent_testing_roadmap.md`
- `ai_agent_tui_testing.md`
- `agent_test_format.md`
+36
View File
@@ -0,0 +1,36 @@
# Test Infrastructure Inventory
This document provides a central index of all SeedPass tests, runners, and automation scripts.
## Core Test Runners
| Script / Command | Target | Coverage |
|-----------------|--------|----------|
| `bash scripts/run_ci_tests.sh` | **Full Suite** | Pytest (CLI, API, TUI v2), Determinism, Coverage |
| `pytest src/tests/` | **Unit/Integration** | Core logic, API, CLI, Encryption |
| `bash scripts/run_gui_tests.sh` | **GUI (BeeWare)** | Headless & Desktop GUI behavioral tests |
| `bash run_flake_tests.sh` | **Stability** | Runs suite multiple times to catch flakes |
| `.venv/bin/python scripts/interactive_agent_tui_test.py` | **Interactive TUI v2** | Mock-service based exploratory walkthrough |
## Specialized Automation
### Determinism
- `bash scripts/run_determinism_tests.sh`: Validates BIP85 compliance and cross-platform artifact consistency.
### TUI v2 Specific
- `src/tests/test_tui_v2_*`: Textual-based unit tests for the v2 interface.
- `scripts/ai_tui2_agent_test.py`: Legacy agent harness for TUI v2.
- `docs/INTERACTIVE_AGENT_TESTING.md`: Guide for manual/agent exploratory testing.
### API & CLI
- `src/tests/test_api*.py`: FastAPI-based backend endpoints.
- `src/tests/test_typer_cli.py`: Typer-based command line interface.
## Test Data & Artifacts
- `artifacts/coverage/`: Generated HTML/JSON coverage reports.
- `artifacts/ci-flakes-*.md`: Logs from flaky test detection runs.
## Guidelines for Agents
1. **Always run CI tests** before proposing changes: `bash scripts/run_ci_tests.sh`.
2. **Add regression tests** in `src/tests/` for every bug fix.
3. **Use Mock Services** for TUI testing to avoid credential requirements (see `scripts/interactive_agent_tui_test.py`).
+222
View File
@@ -0,0 +1,222 @@
#!/usr/bin/env python3
"""
SeedPass TUI v2 Interactive Agent Test Suite
============================================
This script provides a "Human-in-the-loop" style automated test for the
SeedPass TUI v2. It bypasses real encryption and filesystem requirements
by injecting a MockService layer.
Usage:
python scripts/interactive_agent_tui_test.py
Features Tested:
- Entry Creation (Password, TOTP, Doc, etc.)
- Searching and Filtering
- Document Editing
- Archiving and Restoring
- Entry Linking (Graph)
- Sensitive Data Reveal & QR Codes
- Sub-profile (Managed Account) Sessions
- Nostr Sync & Namespace Management
- Settings & Stats
"""
import asyncio
import sys
import os
from pathlib import Path
from types import SimpleNamespace
# Add src to sys.path to allow imports from seedpass
sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
from seedpass.tui_v2.app import launch_tui2
class AgentMockService:
"""
A comprehensive mock of all SeedPass core services.
Bypasses real encryption and state.
"""
def __init__(self):
self.entries = {}
self.next_id = 1
self.archived = set()
self.links = [] # list of dicts: {source, target, relation, note}
self.is_managed = False
self.nostr_index = 0
self.offline = False
# --- EntryService Interface ---
def search_entries(self, query="", kinds=None, include_archived=False, archived_only=False):
out = []
q = (query or "").lower()
for eid, entry in self.entries.items():
is_archived = eid in self.archived
if archived_only and not is_archived: continue
if not include_archived and is_archived: continue
if q and q not in entry["label"].lower(): continue
kind = entry["kind"]
if kinds and kind not in kinds: continue
out.append((
eid,
entry["label"],
entry.get("username"),
entry.get("url"),
is_archived,
SimpleNamespace(value=kind)
))
return sorted(out, key=lambda x: x[0])
def retrieve_entry(self, entry_id):
eid = int(entry_id)
entry = dict(self.entries.get(eid, {}))
if entry:
entry["archived"] = eid in self.archived
return entry
def add_entry(self, label, length, username=None, url=None, **kwargs):
eid = self.next_id
self.entries[eid] = {"id": eid, "label": label, "kind": "password", "username": username, "url": url, "length": length, **kwargs}
self.next_id += 1
return eid
def add_totp(self, label, **kwargs):
eid = self.next_id
self.entries[eid] = {"id": eid, "label": label, "kind": "totp", **kwargs}
self.next_id += 1
return eid
def add_document(self, label, content, file_type="txt", **kwargs):
eid = self.next_id
self.entries[eid] = {"id": eid, "label": label, "kind": "document", "content": content, "file_type": file_type, **kwargs}
self.next_id += 1
return eid
def modify_entry(self, entry_id, **kwargs):
eid = int(entry_id)
if eid in self.entries:
self.entries[eid].update(kwargs)
def archive_entry(self, entry_id): self.archived.add(int(entry_id))
def restore_entry(self, entry_id): self.archived.discard(int(entry_id))
def add_link(self, source_id, target_id, relation="related_to", note=""):
self.links.append({"source": int(source_id), "target": int(target_id), "relation": str(relation), "note": str(note)})
def get_links(self, entry_id):
eid = int(entry_id)
return [l for l in self.links if l["source"] == eid or l["target"] == eid]
# --- Vault/Profile/Nostr/Sync Support ---
def unlock(self, req): return SimpleNamespace(status="ok", duration=0.1)
def lock(self): pass
def stats(self):
return {"total_entries": len(self.entries), "archived": len(self.archived)}
def sync(self): return SimpleNamespace(manifest_id="agent-mock-manifest")
def start_fresh_namespace(self):
self.nostr_index += 1
return self.nostr_index
def load_managed_account(self, entry_id): self.is_managed = True
def exit_managed_account(self): self.is_managed = False
# --- Secret Data ---
def get_totp_code(self, entry_id): return "888999"
def get_totp_secret(self, entry_id): return "AGENT_MOCK_SECRET"
def get_clipboard_clear_delay(self): return 10
def copy_to_clipboard(self, val): return True
def generate_password(self, length, entry_id): return "MOCK_PASSWORD"
async def run_full_walkthrough():
service = AgentMockService()
holder = {}
def _hook(app):
holder["app"] = app
app._session_locked = False # Bypass lock
# Launch app with injected mock services
launch_tui2(
entry_service_factory=lambda: service,
vault_service_factory=lambda: service,
profile_service_factory=lambda: service,
sync_service_factory=lambda: service,
nostr_service_factory=lambda: service,
app_hook=_hook
)
app = holder["app"]
async with app.run_test() as pilot:
from textual.widgets import Static, Input, ListView
print("--- PHASE 1: Entry Management ---")
# Add various types
app._run_palette_command('add-password "Gmail" 20')
app._run_palette_command('add-totp "Bank"')
app._run_palette_command('add-document "Readme" md "Hello World"')
await pilot.pause()
print(f" [OK] Created {len(service.entries)} entries.")
# Search
app._load_entries(query="bank")
await pilot.pause()
results = app.query_one("#entry-list", ListView).children
print(f" [OK] Search 'bank' found {len(results)} results.")
# Archive
app._show_entry(1) # Gmail
await pilot.pause()
app.action_toggle_archive()
await pilot.pause()
print(f" [OK] Archive status: {app.query_one('#status').render()}")
print("\n--- PHASE 2: Advanced Interaction ---")
# Linking
app._run_palette_command("link-add 2 requires 'Auth'")
await pilot.pause()
print(f" [OK] Linked #1 to #2.")
# Reveal & QR
app._show_entry(2) # Bank TOTP
await pilot.pause()
app.action_reveal_selected()
await pilot.pause()
reveal_text = str(app.query_one("#secret-detail", Static).render())
print(f" [OK] Secret Reveal: {'REVEALED' in reveal_text}")
app.action_show_qr()
await pilot.pause()
qr_text = str(app.query_one("#secret-detail", Static).render())
# The QR code uses '##' for dark cells in the ASCII renderer
print(f" [OK] QR Code Active: {'##' in qr_text}")
if '##' not in qr_text:
print(f" [DEBUG] QR Panel Content:\n{qr_text}")
print("\n--- PHASE 3: System Operations ---")
# Managed Account
service.add_entry("Sub", 10, kind="managed_account")
app._load_entries("")
await pilot.pause()
app._run_palette_command("managed-load 4")
await pilot.pause()
print(f" [OK] Sub-profile load: {'Loaded managed' in str(app.query_one('#status').render())}")
app._run_palette_command("managed-exit")
await pilot.pause()
# Nostr & Sync
app._run_palette_command("nostr-fresh-namespace")
await pilot.pause()
app._run_palette_command("sync-now")
await pilot.pause()
print(f" [OK] Sync status: {app.query_one('#status').render()}")
print("\n--- WALKTHROUGH COMPLETE ---")
return 0
if __name__ == "__main__":
asyncio.run(run_full_walkthrough())
+16
View File
@@ -75,3 +75,19 @@ def test_atomic_write_preserves_existing_on_failure(tmp_path: Path) -> None:
# Should revert to 1 file
assert len(list(tmp_path.iterdir())) == 1
assert dest_path.read_text() == "original content"
def test_atomic_write_permissions(tmp_path: Path) -> None:
"""File permissions should be set to 0o600 (owner read/write only)."""
import stat
test_file = tmp_path / "permissions.txt"
def write_op(f):
f.write("content")
atomic_write(test_file, write_op)
st = test_file.stat()
mode = stat.S_IMODE(st.st_mode)
assert mode == 0o600
+7 -1
View File
@@ -154,8 +154,11 @@ async def test_tui2_parity_filters_cover_all_entry_kinds() -> None:
for kind in kinds:
await _run_palette(app, pilot, f"filter {kind}")
assert len(list_view.children) == 1
# Explicitly open the entry to ensure selection (parity requirement)
entry_id = entries[kinds.index(kind)]["id"]
await _run_palette(app, pilot, f"open {entry_id}")
detail_text = _widget_text(app, "#entry-detail")
assert f'"kind": "{kind}"' in detail_text
assert kind in detail_text.lower()
await _run_palette(app, pilot, "filter all")
assert len(list_view.children) == len(kinds)
@@ -170,6 +173,9 @@ async def test_tui2_parity_archive_restore_roundtrip_non_document() -> None:
async with app.run_test() as pilot:
await pilot.pause()
# Explicitly open the entry to ensure selection
await _run_palette(app, pilot, "open 1")
await _run_palette(app, pilot, "archive")
assert service.retrieve_entry(1)["archived"] is True
assert "archived" in _status_text(app)
-35
View File
@@ -1,35 +0,0 @@
import sys
from pathlib import Path
# Add src/utils to path to import atomic_write directly
sys.path.append(str(Path("src/utils").resolve()))
from atomic_write import atomic_write
import os
import stat
def test_atomic_write_permissions():
test_file = Path("test_permissions.txt")
if test_file.exists():
test_file.unlink()
def write_op(f):
f.write("content")
atomic_write(test_file, write_op)
st = test_file.stat()
mode = stat.S_IMODE(st.st_mode)
print(f"File permissions: {oct(mode)}")
if mode == 0o600:
print("SUCCESS: Permissions are 0o600")
else:
print(f"FAILURE: Permissions are {oct(mode)}")
test_file.unlink()
if __name__ == "__main__":
test_atomic_write_permissions()