mirror of
https://github.com/PR0M3TH3AN/SeedPass.git
synced 2026-11-06 12:52:04 +00:00
ci: enforce tui2 and core api coverage thresholds
This commit is contained in:
1 parent
c1a317f9b6
commit
b77c887446
8 files changed
+102
-7
No files matched your search
@@ -132,6 +132,7 @@ Useful options:
|
||||
Coverage gate helper:
|
||||
|
||||
- `python scripts/check_critical_coverage.py artifacts/coverage/coverage.json`
|
||||
- `./scripts/tui2_coverage_gate.sh` (focused TUI v2 + service-path thresholds)
|
||||
- Use `--threshold PATH=PERCENT` to override defaults.
|
||||
- Use `--no-default-thresholds` for focused local smoke checks.
|
||||
|
||||
|
||||
@@ -64,10 +64,9 @@ Use these docs as inputs, but treat this file as the decision layer:
|
||||
|
||||
## 4) Recommended Next Steps (Priority Order)
|
||||
|
||||
1. Strengthen testing gates by adding critical coverage thresholds for TUI v2 and key service modules used by v2.
|
||||
2. Advance Nostr resilience validation with deterministic failure-mode suites (then optional real-relay soak lane).
|
||||
3. Update cutover memo to reflect current reality and remaining concrete blockers.
|
||||
4. Finish supply-chain readiness evidence and release-protection policy.
|
||||
1. Advance Nostr resilience validation with deterministic failure-mode suites (then optional real-relay soak lane).
|
||||
2. Update cutover memo to reflect current reality and remaining concrete blockers.
|
||||
3. Finish supply-chain readiness evidence and release-protection policy.
|
||||
|
||||
## 5) Next Slice Definition (Recommended Immediate Work)
|
||||
|
||||
@@ -100,6 +99,12 @@ Progress update (2026-03-02):
|
||||
- Full legacy workflow coverage closed with bug-bash evidence:
|
||||
- `python scripts/ai_tui2_agent_test.py --scenario extended --verbose` => PASS
|
||||
- `python scripts/ai_tui_agent_test.py --scenario extended --verbose` => PASS
|
||||
- Coverage gates strengthened for TUI v2 critical path:
|
||||
- `scripts/run_ci_tests.sh` now enables critical coverage gate by default.
|
||||
- `scripts/check_critical_coverage.py` default thresholds now include `src/seedpass/core/api.py=85`.
|
||||
- new focused gate `scripts/tui2_coverage_gate.sh` enforces:
|
||||
- `src/seedpass/tui_v2/app.py >= 78%`
|
||||
- `src/seedpass/core/api.py >= 85%`
|
||||
|
||||
## 6) Working Rule
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ Status legend: `Not Started`, `In Progress`, `Blocked`, `Done`
|
||||
| 4 | Backup and restore integrity validation | In Progress | Core / QA | `src/tests/test_account_roundtrip_restore.py`, `src/tests/test_portable_backup.py` |
|
||||
| 5 | Nostr sync security and resilience validation | In Progress | Core / QA | `src/tests/test_nostr_real.py`, `src/tests/test_nostr_index_size.py` |
|
||||
| 6 | Auth, lock/unlock, and access-control hardening | Done | Core | `docs/auth_lock_unlock_access_review.md`, `src/tests/test_api_new_endpoints.py`, `src/tests/test_vault_lock_flag.py`, `src/tests/test_inactivity_lock.py`, `src/tests/test_unlock_sync.py` |
|
||||
| 7 | Testing gates and quality thresholds | In Progress | QA | `scripts/check_critical_coverage.py`, CI workflows |
|
||||
| 7 | Testing gates and quality thresholds | In Progress | QA | `scripts/check_critical_coverage.py`, `scripts/tui2_coverage_gate.sh`, CI workflows |
|
||||
| 8 | Supply chain and release integrity | In Progress | DevOps | `docs/supply_chain_release_integrity.md`, `.github/workflows/release-integrity.yml`, `scripts/release_integrity.py`, `src/tests/test_release_integrity_utils.py` |
|
||||
| 9 | Operational readiness and incident runbooks | Not Started | Security / Ops | |
|
||||
| 10 | External audit and staged production rollout | Not Started | Maintainers | |
|
||||
|
||||
@@ -690,3 +690,34 @@
|
||||
- `docs/dev_control_center.md`
|
||||
- removed parity-closure pending state
|
||||
- shifted priority stack to post-parity cutover readiness (testing gates, Nostr resilience, cutover memo, supply-chain readiness).
|
||||
|
||||
## 2026-03-02 Testing-gate hardening: TUI v2 + service coverage thresholds
|
||||
- Added dedicated focused gate script: `scripts/tui2_coverage_gate.sh`.
|
||||
- Gate behavior:
|
||||
- uses `.venv/bin/python` when available (fallback `python`/`python3`)
|
||||
- checks Textual runtime availability before running focused TUI v2 coverage suite
|
||||
- runs focused tests with coverage output: `artifacts/coverage/coverage.tui2_gate.json`
|
||||
- enforces thresholds via `scripts/check_critical_coverage.py --no-default-thresholds`:
|
||||
- `src/seedpass/tui_v2/app.py >= 78%`
|
||||
- `src/seedpass/core/api.py >= 85%`
|
||||
- writes machine-readable result to `artifacts/coverage/critical_gate.tui2.json`.
|
||||
- Strengthened default critical-coverage policy:
|
||||
- `scripts/check_critical_coverage.py` default thresholds now include:
|
||||
- `src/seedpass/core/api.py = 85%`.
|
||||
- CI runner hardening:
|
||||
- `scripts/run_ci_tests.sh` now enables `CRITICAL_COVERAGE_GATE` by default (`1`).
|
||||
- adds JSON output for full-suite gate: `artifacts/coverage/critical_gate.full.json`.
|
||||
- adds focused TUI v2 gate call controlled by `TUI2_COVERAGE_GATE` (default `1`).
|
||||
- coverage-gate Python invocation now uses `python`/`python3` fallback resolution.
|
||||
- Test/doc updates:
|
||||
- `src/tests/test_check_critical_coverage.py` includes assertion for new default API threshold.
|
||||
- docs updated in:
|
||||
- `docs/ai_agent_tui_testing.md`
|
||||
- `docs/security_readiness_checklist.md`
|
||||
- `docs/dev_control_center.md`
|
||||
- Validation run (this slice):
|
||||
- `pytest -q src/tests/test_check_critical_coverage.py` -> `9 passed`
|
||||
- `./scripts/tui2_coverage_gate.sh` -> PASS
|
||||
- `src/seedpass/core/api.py: 86.64%`
|
||||
- `src/seedpass/tui_v2/app.py: 79.87%`
|
||||
- `python3 scripts/check_critical_coverage.py artifacts/coverage/coverage.json --json-output artifacts/coverage/critical_gate.full.json` -> PASS.
|
||||
@@ -19,6 +19,7 @@ from pathlib import Path
|
||||
DEFAULT_THRESHOLDS: dict[str, float] = {
|
||||
"src/main.py": 50.0,
|
||||
"src/seedpass/core/manager.py": 60.0,
|
||||
"src/seedpass/core/api.py": 85.0,
|
||||
"src/seedpass/core/encryption.py": 80.0,
|
||||
"src/seedpass/core/migrations.py": 90.0,
|
||||
}
|
||||
|
||||
+12
-2
@@ -5,6 +5,11 @@ mkdir -p artifacts/coverage
|
||||
|
||||
./scripts/tui2_check_smoke.sh
|
||||
|
||||
py_bin="python"
|
||||
if ! command -v "$py_bin" >/dev/null 2>&1; then
|
||||
py_bin="python3"
|
||||
fi
|
||||
|
||||
if [[ "${DETERMINISM_GATE:-1}" == "1" ]]; then
|
||||
./scripts/run_determinism_tests.sh
|
||||
fi
|
||||
@@ -48,7 +53,12 @@ if [[ $status -eq 124 ]]; then
|
||||
tail -n 20 pytest.log
|
||||
exit 1
|
||||
fi
|
||||
if [[ $status -eq 0 && "${CRITICAL_COVERAGE_GATE:-0}" == "1" ]]; then
|
||||
python scripts/check_critical_coverage.py artifacts/coverage/coverage.json
|
||||
if [[ $status -eq 0 && "${CRITICAL_COVERAGE_GATE:-1}" == "1" ]]; then
|
||||
"$py_bin" scripts/check_critical_coverage.py \
|
||||
artifacts/coverage/coverage.json \
|
||||
--json-output artifacts/coverage/critical_gate.full.json
|
||||
fi
|
||||
if [[ $status -eq 0 && "${TUI2_COVERAGE_GATE:-1}" == "1" ]]; then
|
||||
./scripts/tui2_coverage_gate.sh
|
||||
fi
|
||||
exit $status
|
||||
Executable
+43
@@ -0,0 +1,43 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
mkdir -p artifacts/coverage
|
||||
|
||||
if [[ -x ".venv/bin/python" ]]; then
|
||||
py_bin=".venv/bin/python"
|
||||
elif command -v python >/dev/null 2>&1; then
|
||||
py_bin="python"
|
||||
else
|
||||
py_bin="python3"
|
||||
fi
|
||||
|
||||
if ! "$py_bin" - <<'PY' >/dev/null 2>&1
|
||||
import importlib.util
|
||||
raise SystemExit(0 if importlib.util.find_spec("textual") else 1)
|
||||
PY
|
||||
then
|
||||
if [[ "${REQUIRE_TEXTUAL:-0}" == "1" ]]; then
|
||||
echo "Error: textual runtime is unavailable; cannot run TUI v2 coverage gate." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "tui2 coverage gate: skipped (textual runtime unavailable)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
"$py_bin" -m pytest -q \
|
||||
--cov=src \
|
||||
--cov-report=json:artifacts/coverage/coverage.tui2_gate.json \
|
||||
src/tests/test_tui_v2_helpers.py \
|
||||
src/tests/test_tui_v2_textual_interactions.py \
|
||||
src/tests/test_tui_v2_action_matrix.py \
|
||||
src/tests/test_tui_v2_parity_scenarios.py \
|
||||
src/tests/test_core_api_services.py
|
||||
|
||||
"$py_bin" scripts/check_critical_coverage.py \
|
||||
artifacts/coverage/coverage.tui2_gate.json \
|
||||
--no-default-thresholds \
|
||||
--threshold src/seedpass/tui_v2/app.py=78 \
|
||||
--threshold src/seedpass/core/api.py=85 \
|
||||
--json-output artifacts/coverage/critical_gate.tui2.json
|
||||
|
||||
echo "tui2 coverage gate: ok"
|
||||
@@ -23,6 +23,10 @@ def _load_module():
|
||||
coverage_gate = _load_module()
|
||||
|
||||
|
||||
def test_default_thresholds_include_tui2_service_module():
|
||||
assert coverage_gate.DEFAULT_THRESHOLDS["src/seedpass/core/api.py"] == 85.0
|
||||
|
||||
|
||||
def test_parse_threshold_overrides_valid():
|
||||
parsed = coverage_gate._parse_threshold_overrides(
|
||||
["src/main.py=40", "src/seedpass/core/manager.py=55.5"]
|
||||
|
||||
Reference in new issue
Block a user