mirror of
https://github.com/PR0M3TH3AN/SeedPass.git
synced 2026-11-06 04:42:35 +00:00
docs(test): centralize test infrastructure and add interactive agent harness
Added docs/TEST_INVENTORY.md as a central index for all tests and runners. Created scripts/interactive_agent_tui_test.py for password-free TUI v2 exploratory testing. Integrated standalone permission tests into main suite and fixed TUI v2 parity regression tests broken by auto-selection removal. Updated AGENTS.md and CLAUDE.md with strategic plan references.
This commit is contained in:
1 parent
165c825d9e
commit
4eeb936e6d
9 files changed
+351
-45
No files matched your search
@@ -56,22 +56,32 @@ cd ~/.seedpass/app && source venv/bin/activate
|
||||
cd src && python main.py
|
||||
```
|
||||
|
||||
## Planning & Strategy
|
||||
|
||||
Before starting work, AI agents must consult the following documents to align with the current project trajectory:
|
||||
|
||||
1. **[Dev Control Center](docs/dev_control_center.md)**: The single source of truth for current priorities and "what to do next."
|
||||
2. **[TUI v2 Execution Plan](docs/tui_v2_integration_execution_plan_2026-03-02.md)**: The active roadmap for the TUI v2 mockup parity and hardening phase.
|
||||
|
||||
## Running Tests
|
||||
|
||||
SeedPass maintains a comprehensive test infrastructure. Agents should use the central inventory as a guide:
|
||||
|
||||
1. **[Test Infrastructure Inventory](docs/TEST_INVENTORY.md)**: A complete index of all unit, integration, UI, and determinism tests.
|
||||
|
||||
### Quick Start: Running Tests
|
||||
|
||||
1. Set up a virtual environment and install dependencies:
|
||||
|
||||
```bash
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
pip install --require-hashes -r requirements.lock
|
||||
```
|
||||
|
||||
2. Run the test suite using **pytest**:
|
||||
|
||||
...
|
||||
```bash
|
||||
pytest
|
||||
```
|
||||
|
||||
For a full CI-equivalent run (highly recommended before any PR), use:
|
||||
```bash
|
||||
bash scripts/run_ci_tests.sh
|
||||
```
|
||||
|
||||
Currently the test folder is located in `src/tests/`. New tests should be placed there so `pytest` can discover them automatically.
|
||||
|
||||
## Style Guidelines
|
||||
|
||||
@@ -1,3 +1,15 @@
|
||||
# SeedPass Agent Guide
|
||||
|
||||
## Planning & Strategy
|
||||
Consult these first to align with current objectives:
|
||||
- **[Dev Control Center](docs/dev_control_center.md)**: Current priorities.
|
||||
- **[TUI v2 Execution Plan](docs/tui_v2_integration_execution_plan_2026-03-02.md)**: Roadmap for TUI v2.
|
||||
|
||||
## Testing
|
||||
Always run the full suite before proposing changes:
|
||||
- `bash scripts/run_ci_tests.sh`
|
||||
- Central Index: **[Test Infrastructure Inventory](docs/TEST_INVENTORY.md)**
|
||||
|
||||
## TORCH Memory Integration
|
||||
You have access to the TORCH memory system.
|
||||
1. READ: Check `.scheduler-memory/latest/${cadence}/memories.md` for past learnings.
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
# Interactive Agent TUI Testing Protocol
|
||||
|
||||
This document defines how AI agents should perform exploratory and regression testing on the SeedPass TUI v2 using the `scripts/interactive_agent_tui_test.py` harness.
|
||||
|
||||
## 1. Overview
|
||||
The testing harness uses **Dependency Injection** to replace the real SeedPass service layer with a `MockService`. This allows agents to:
|
||||
1. **Bypass Encryption:** Test the UI without knowing passwords or seeds.
|
||||
2. **Avoid Side Effects:** No changes are made to the real vault or filesystem.
|
||||
3. **Simulate Edge Cases:** Mock specific failures (e.g., sync errors) to see how the UI recovers.
|
||||
|
||||
## 2. How to Run
|
||||
Execute the harness using the project's virtual environment:
|
||||
```bash
|
||||
.venv/bin/python scripts/interactive_agent_tui_test.py
|
||||
```
|
||||
|
||||
## 3. How to Extend
|
||||
To add a new test case:
|
||||
1. **Update the Mock:** If testing a new service method, add it to `AgentMockService`.
|
||||
2. **Add a Step:** In `run_full_walkthrough`, use `app._run_palette_command("your-command")` or direct action calls (e.g., `app.action_toggle_archive()`).
|
||||
3. **Verify State:** Use `pilot.pause()` to let the TUI update, then query widgets:
|
||||
```python
|
||||
status = str(app.query_one("#status").render())
|
||||
assert "Expected" in status
|
||||
```
|
||||
|
||||
## 4. Key Testing Areas
|
||||
* **Palette Commands:** Ensure `_run_palette_command` correctly routes inputs to services.
|
||||
* **Selection Persistence:** Verify that the "Selected Entry" remains correct after archiving or refreshing.
|
||||
* **Layout Adaptability:** Test compact vs. comfortable density.
|
||||
* **Reactive UI:** Verify that `_load_entries` is called automatically after mutations.
|
||||
|
||||
## 5. Agent Instructions for Future Runs
|
||||
If you are an agent tasked with "looking for bugs":
|
||||
1. **Run the script:** Confirm the baseline walkthrough passes.
|
||||
2. **Introduce Chaos:** Modify the mock to throw exceptions randomly.
|
||||
3. **Check Transitions:** Rapidly switch between entries and layouts.
|
||||
4. **Document Failures:** If a widget isn't found or a status message is wrong, report the line number in `app.py`.
|
||||
@@ -46,6 +46,7 @@ This directory contains SeedPass project documentation.
|
||||
|
||||
## QA and Testing Docs
|
||||
|
||||
- `TEST_INVENTORY.md` (central index of all tests and runners)
|
||||
- `agent_testing_roadmap.md`
|
||||
- `ai_agent_tui_testing.md`
|
||||
- `agent_test_format.md`
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
# Test Infrastructure Inventory
|
||||
|
||||
This document provides a central index of all SeedPass tests, runners, and automation scripts.
|
||||
|
||||
## Core Test Runners
|
||||
|
||||
| Script / Command | Target | Coverage |
|
||||
|-----------------|--------|----------|
|
||||
| `bash scripts/run_ci_tests.sh` | **Full Suite** | Pytest (CLI, API, TUI v2), Determinism, Coverage |
|
||||
| `pytest src/tests/` | **Unit/Integration** | Core logic, API, CLI, Encryption |
|
||||
| `bash scripts/run_gui_tests.sh` | **GUI (BeeWare)** | Headless & Desktop GUI behavioral tests |
|
||||
| `bash run_flake_tests.sh` | **Stability** | Runs suite multiple times to catch flakes |
|
||||
| `.venv/bin/python scripts/interactive_agent_tui_test.py` | **Interactive TUI v2** | Mock-service based exploratory walkthrough |
|
||||
|
||||
## Specialized Automation
|
||||
|
||||
### Determinism
|
||||
- `bash scripts/run_determinism_tests.sh`: Validates BIP85 compliance and cross-platform artifact consistency.
|
||||
|
||||
### TUI v2 Specific
|
||||
- `src/tests/test_tui_v2_*`: Textual-based unit tests for the v2 interface.
|
||||
- `scripts/ai_tui2_agent_test.py`: Legacy agent harness for TUI v2.
|
||||
- `docs/INTERACTIVE_AGENT_TESTING.md`: Guide for manual/agent exploratory testing.
|
||||
|
||||
### API & CLI
|
||||
- `src/tests/test_api*.py`: FastAPI-based backend endpoints.
|
||||
- `src/tests/test_typer_cli.py`: Typer-based command line interface.
|
||||
|
||||
## Test Data & Artifacts
|
||||
- `artifacts/coverage/`: Generated HTML/JSON coverage reports.
|
||||
- `artifacts/ci-flakes-*.md`: Logs from flaky test detection runs.
|
||||
|
||||
## Guidelines for Agents
|
||||
1. **Always run CI tests** before proposing changes: `bash scripts/run_ci_tests.sh`.
|
||||
2. **Add regression tests** in `src/tests/` for every bug fix.
|
||||
3. **Use Mock Services** for TUI testing to avoid credential requirements (see `scripts/interactive_agent_tui_test.py`).
|
||||
@@ -0,0 +1,222 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
SeedPass TUI v2 Interactive Agent Test Suite
|
||||
============================================
|
||||
|
||||
This script provides a "Human-in-the-loop" style automated test for the
|
||||
SeedPass TUI v2. It bypasses real encryption and filesystem requirements
|
||||
by injecting a MockService layer.
|
||||
|
||||
Usage:
|
||||
python scripts/interactive_agent_tui_test.py
|
||||
|
||||
Features Tested:
|
||||
- Entry Creation (Password, TOTP, Doc, etc.)
|
||||
- Searching and Filtering
|
||||
- Document Editing
|
||||
- Archiving and Restoring
|
||||
- Entry Linking (Graph)
|
||||
- Sensitive Data Reveal & QR Codes
|
||||
- Sub-profile (Managed Account) Sessions
|
||||
- Nostr Sync & Namespace Management
|
||||
- Settings & Stats
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
# Add src to sys.path to allow imports from seedpass
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
|
||||
|
||||
from seedpass.tui_v2.app import launch_tui2
|
||||
|
||||
class AgentMockService:
|
||||
"""
|
||||
A comprehensive mock of all SeedPass core services.
|
||||
Bypasses real encryption and state.
|
||||
"""
|
||||
def __init__(self):
|
||||
self.entries = {}
|
||||
self.next_id = 1
|
||||
self.archived = set()
|
||||
self.links = [] # list of dicts: {source, target, relation, note}
|
||||
self.is_managed = False
|
||||
self.nostr_index = 0
|
||||
self.offline = False
|
||||
|
||||
# --- EntryService Interface ---
|
||||
def search_entries(self, query="", kinds=None, include_archived=False, archived_only=False):
|
||||
out = []
|
||||
q = (query or "").lower()
|
||||
for eid, entry in self.entries.items():
|
||||
is_archived = eid in self.archived
|
||||
if archived_only and not is_archived: continue
|
||||
if not include_archived and is_archived: continue
|
||||
if q and q not in entry["label"].lower(): continue
|
||||
kind = entry["kind"]
|
||||
if kinds and kind not in kinds: continue
|
||||
|
||||
out.append((
|
||||
eid,
|
||||
entry["label"],
|
||||
entry.get("username"),
|
||||
entry.get("url"),
|
||||
is_archived,
|
||||
SimpleNamespace(value=kind)
|
||||
))
|
||||
return sorted(out, key=lambda x: x[0])
|
||||
|
||||
def retrieve_entry(self, entry_id):
|
||||
eid = int(entry_id)
|
||||
entry = dict(self.entries.get(eid, {}))
|
||||
if entry:
|
||||
entry["archived"] = eid in self.archived
|
||||
return entry
|
||||
|
||||
def add_entry(self, label, length, username=None, url=None, **kwargs):
|
||||
eid = self.next_id
|
||||
self.entries[eid] = {"id": eid, "label": label, "kind": "password", "username": username, "url": url, "length": length, **kwargs}
|
||||
self.next_id += 1
|
||||
return eid
|
||||
|
||||
def add_totp(self, label, **kwargs):
|
||||
eid = self.next_id
|
||||
self.entries[eid] = {"id": eid, "label": label, "kind": "totp", **kwargs}
|
||||
self.next_id += 1
|
||||
return eid
|
||||
|
||||
def add_document(self, label, content, file_type="txt", **kwargs):
|
||||
eid = self.next_id
|
||||
self.entries[eid] = {"id": eid, "label": label, "kind": "document", "content": content, "file_type": file_type, **kwargs}
|
||||
self.next_id += 1
|
||||
return eid
|
||||
|
||||
def modify_entry(self, entry_id, **kwargs):
|
||||
eid = int(entry_id)
|
||||
if eid in self.entries:
|
||||
self.entries[eid].update(kwargs)
|
||||
|
||||
def archive_entry(self, entry_id): self.archived.add(int(entry_id))
|
||||
def restore_entry(self, entry_id): self.archived.discard(int(entry_id))
|
||||
|
||||
def add_link(self, source_id, target_id, relation="related_to", note=""):
|
||||
self.links.append({"source": int(source_id), "target": int(target_id), "relation": str(relation), "note": str(note)})
|
||||
|
||||
def get_links(self, entry_id):
|
||||
eid = int(entry_id)
|
||||
return [l for l in self.links if l["source"] == eid or l["target"] == eid]
|
||||
|
||||
# --- Vault/Profile/Nostr/Sync Support ---
|
||||
def unlock(self, req): return SimpleNamespace(status="ok", duration=0.1)
|
||||
def lock(self): pass
|
||||
def stats(self):
|
||||
return {"total_entries": len(self.entries), "archived": len(self.archived)}
|
||||
|
||||
def sync(self): return SimpleNamespace(manifest_id="agent-mock-manifest")
|
||||
def start_fresh_namespace(self):
|
||||
self.nostr_index += 1
|
||||
return self.nostr_index
|
||||
|
||||
def load_managed_account(self, entry_id): self.is_managed = True
|
||||
def exit_managed_account(self): self.is_managed = False
|
||||
|
||||
# --- Secret Data ---
|
||||
def get_totp_code(self, entry_id): return "888999"
|
||||
def get_totp_secret(self, entry_id): return "AGENT_MOCK_SECRET"
|
||||
def get_clipboard_clear_delay(self): return 10
|
||||
def copy_to_clipboard(self, val): return True
|
||||
def generate_password(self, length, entry_id): return "MOCK_PASSWORD"
|
||||
|
||||
async def run_full_walkthrough():
|
||||
service = AgentMockService()
|
||||
holder = {}
|
||||
|
||||
def _hook(app):
|
||||
holder["app"] = app
|
||||
app._session_locked = False # Bypass lock
|
||||
|
||||
# Launch app with injected mock services
|
||||
launch_tui2(
|
||||
entry_service_factory=lambda: service,
|
||||
vault_service_factory=lambda: service,
|
||||
profile_service_factory=lambda: service,
|
||||
sync_service_factory=lambda: service,
|
||||
nostr_service_factory=lambda: service,
|
||||
app_hook=_hook
|
||||
)
|
||||
|
||||
app = holder["app"]
|
||||
|
||||
async with app.run_test() as pilot:
|
||||
from textual.widgets import Static, Input, ListView
|
||||
|
||||
print("--- PHASE 1: Entry Management ---")
|
||||
# Add various types
|
||||
app._run_palette_command('add-password "Gmail" 20')
|
||||
app._run_palette_command('add-totp "Bank"')
|
||||
app._run_palette_command('add-document "Readme" md "Hello World"')
|
||||
await pilot.pause()
|
||||
print(f" [OK] Created {len(service.entries)} entries.")
|
||||
|
||||
# Search
|
||||
app._load_entries(query="bank")
|
||||
await pilot.pause()
|
||||
results = app.query_one("#entry-list", ListView).children
|
||||
print(f" [OK] Search 'bank' found {len(results)} results.")
|
||||
|
||||
# Archive
|
||||
app._show_entry(1) # Gmail
|
||||
await pilot.pause()
|
||||
app.action_toggle_archive()
|
||||
await pilot.pause()
|
||||
print(f" [OK] Archive status: {app.query_one('#status').render()}")
|
||||
|
||||
print("\n--- PHASE 2: Advanced Interaction ---")
|
||||
# Linking
|
||||
app._run_palette_command("link-add 2 requires 'Auth'")
|
||||
await pilot.pause()
|
||||
print(f" [OK] Linked #1 to #2.")
|
||||
|
||||
# Reveal & QR
|
||||
app._show_entry(2) # Bank TOTP
|
||||
await pilot.pause()
|
||||
app.action_reveal_selected()
|
||||
await pilot.pause()
|
||||
reveal_text = str(app.query_one("#secret-detail", Static).render())
|
||||
print(f" [OK] Secret Reveal: {'REVEALED' in reveal_text}")
|
||||
|
||||
app.action_show_qr()
|
||||
await pilot.pause()
|
||||
qr_text = str(app.query_one("#secret-detail", Static).render())
|
||||
# The QR code uses '##' for dark cells in the ASCII renderer
|
||||
print(f" [OK] QR Code Active: {'##' in qr_text}")
|
||||
if '##' not in qr_text:
|
||||
print(f" [DEBUG] QR Panel Content:\n{qr_text}")
|
||||
|
||||
print("\n--- PHASE 3: System Operations ---")
|
||||
# Managed Account
|
||||
service.add_entry("Sub", 10, kind="managed_account")
|
||||
app._load_entries("")
|
||||
await pilot.pause()
|
||||
app._run_palette_command("managed-load 4")
|
||||
await pilot.pause()
|
||||
print(f" [OK] Sub-profile load: {'Loaded managed' in str(app.query_one('#status').render())}")
|
||||
|
||||
app._run_palette_command("managed-exit")
|
||||
await pilot.pause()
|
||||
|
||||
# Nostr & Sync
|
||||
app._run_palette_command("nostr-fresh-namespace")
|
||||
await pilot.pause()
|
||||
app._run_palette_command("sync-now")
|
||||
await pilot.pause()
|
||||
print(f" [OK] Sync status: {app.query_one('#status').render()}")
|
||||
|
||||
print("\n--- WALKTHROUGH COMPLETE ---")
|
||||
return 0
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(run_full_walkthrough())
|
||||
@@ -75,3 +75,19 @@ def test_atomic_write_preserves_existing_on_failure(tmp_path: Path) -> None:
|
||||
# Should revert to 1 file
|
||||
assert len(list(tmp_path.iterdir())) == 1
|
||||
assert dest_path.read_text() == "original content"
|
||||
|
||||
|
||||
def test_atomic_write_permissions(tmp_path: Path) -> None:
|
||||
"""File permissions should be set to 0o600 (owner read/write only)."""
|
||||
import stat
|
||||
|
||||
test_file = tmp_path / "permissions.txt"
|
||||
|
||||
def write_op(f):
|
||||
f.write("content")
|
||||
|
||||
atomic_write(test_file, write_op)
|
||||
|
||||
st = test_file.stat()
|
||||
mode = stat.S_IMODE(st.st_mode)
|
||||
assert mode == 0o600
|
||||
@@ -154,8 +154,11 @@ async def test_tui2_parity_filters_cover_all_entry_kinds() -> None:
|
||||
for kind in kinds:
|
||||
await _run_palette(app, pilot, f"filter {kind}")
|
||||
assert len(list_view.children) == 1
|
||||
# Explicitly open the entry to ensure selection (parity requirement)
|
||||
entry_id = entries[kinds.index(kind)]["id"]
|
||||
await _run_palette(app, pilot, f"open {entry_id}")
|
||||
detail_text = _widget_text(app, "#entry-detail")
|
||||
assert f'"kind": "{kind}"' in detail_text
|
||||
assert kind in detail_text.lower()
|
||||
|
||||
await _run_palette(app, pilot, "filter all")
|
||||
assert len(list_view.children) == len(kinds)
|
||||
@@ -170,6 +173,9 @@ async def test_tui2_parity_archive_restore_roundtrip_non_document() -> None:
|
||||
|
||||
async with app.run_test() as pilot:
|
||||
await pilot.pause()
|
||||
# Explicitly open the entry to ensure selection
|
||||
await _run_palette(app, pilot, "open 1")
|
||||
|
||||
await _run_palette(app, pilot, "archive")
|
||||
assert service.retrieve_entry(1)["archived"] is True
|
||||
assert "archived" in _status_text(app)
|
||||
|
||||
@@ -1,35 +0,0 @@
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Add src/utils to path to import atomic_write directly
|
||||
sys.path.append(str(Path("src/utils").resolve()))
|
||||
|
||||
from atomic_write import atomic_write
|
||||
import os
|
||||
import stat
|
||||
|
||||
|
||||
def test_atomic_write_permissions():
|
||||
test_file = Path("test_permissions.txt")
|
||||
if test_file.exists():
|
||||
test_file.unlink()
|
||||
|
||||
def write_op(f):
|
||||
f.write("content")
|
||||
|
||||
atomic_write(test_file, write_op)
|
||||
|
||||
st = test_file.stat()
|
||||
mode = stat.S_IMODE(st.st_mode)
|
||||
print(f"File permissions: {oct(mode)}")
|
||||
|
||||
if mode == 0o600:
|
||||
print("SUCCESS: Permissions are 0o600")
|
||||
else:
|
||||
print(f"FAILURE: Permissions are {oct(mode)}")
|
||||
|
||||
test_file.unlink()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_atomic_write_permissions()
|
||||
Reference in new issue
Block a user