ci: enforce tui2 and core api coverage thresholds

This commit is contained in:
thePR0M3TH3AN committed 2026-03-02 09:03:53 -05:00
1 parent c1a317f9b6
commit b77c887446
8 files changed
+102 -7

No files matched your search

+1
View File
@@ -132,6 +132,7 @@ Useful options:
Coverage gate helper:
- `python scripts/check_critical_coverage.py artifacts/coverage/coverage.json`
- `./scripts/tui2_coverage_gate.sh` (focused TUI v2 + service-path thresholds)
- Use `--threshold PATH=PERCENT` to override defaults.
- Use `--no-default-thresholds` for focused local smoke checks.
+9 -4
View File
@@ -64,10 +64,9 @@ Use these docs as inputs, but treat this file as the decision layer:
## 4) Recommended Next Steps (Priority Order)
1. Strengthen testing gates by adding critical coverage thresholds for TUI v2 and key service modules used by v2.
2. Advance Nostr resilience validation with deterministic failure-mode suites (then optional real-relay soak lane).
3. Update cutover memo to reflect current reality and remaining concrete blockers.
4. Finish supply-chain readiness evidence and release-protection policy.
1. Advance Nostr resilience validation with deterministic failure-mode suites (then optional real-relay soak lane).
2. Update cutover memo to reflect current reality and remaining concrete blockers.
3. Finish supply-chain readiness evidence and release-protection policy.
## 5) Next Slice Definition (Recommended Immediate Work)
@@ -100,6 +99,12 @@ Progress update (2026-03-02):
- Full legacy workflow coverage closed with bug-bash evidence:
- `python scripts/ai_tui2_agent_test.py --scenario extended --verbose` => PASS
- `python scripts/ai_tui_agent_test.py --scenario extended --verbose` => PASS
- Coverage gates strengthened for TUI v2 critical path:
- `scripts/run_ci_tests.sh` now enables critical coverage gate by default.
- `scripts/check_critical_coverage.py` default thresholds now include `src/seedpass/core/api.py=85`.
- new focused gate `scripts/tui2_coverage_gate.sh` enforces:
- `src/seedpass/tui_v2/app.py >= 78%`
- `src/seedpass/core/api.py >= 85%`
## 6) Working Rule
+1 -1
View File
@@ -10,7 +10,7 @@ Status legend: `Not Started`, `In Progress`, `Blocked`, `Done`
| 4 | Backup and restore integrity validation | In Progress | Core / QA | `src/tests/test_account_roundtrip_restore.py`, `src/tests/test_portable_backup.py` |
| 5 | Nostr sync security and resilience validation | In Progress | Core / QA | `src/tests/test_nostr_real.py`, `src/tests/test_nostr_index_size.py` |
| 6 | Auth, lock/unlock, and access-control hardening | Done | Core | `docs/auth_lock_unlock_access_review.md`, `src/tests/test_api_new_endpoints.py`, `src/tests/test_vault_lock_flag.py`, `src/tests/test_inactivity_lock.py`, `src/tests/test_unlock_sync.py` |
| 7 | Testing gates and quality thresholds | In Progress | QA | `scripts/check_critical_coverage.py`, CI workflows |
| 7 | Testing gates and quality thresholds | In Progress | QA | `scripts/check_critical_coverage.py`, `scripts/tui2_coverage_gate.sh`, CI workflows |
| 8 | Supply chain and release integrity | In Progress | DevOps | `docs/supply_chain_release_integrity.md`, `.github/workflows/release-integrity.yml`, `scripts/release_integrity.py`, `src/tests/test_release_integrity_utils.py` |
| 9 | Operational readiness and incident runbooks | Not Started | Security / Ops | |
| 10 | External audit and staged production rollout | Not Started | Maintainers | |
+31
View File
@@ -690,3 +690,34 @@
- `docs/dev_control_center.md`
- removed parity-closure pending state
- shifted priority stack to post-parity cutover readiness (testing gates, Nostr resilience, cutover memo, supply-chain readiness).
## 2026-03-02 Testing-gate hardening: TUI v2 + service coverage thresholds
- Added dedicated focused gate script: `scripts/tui2_coverage_gate.sh`.
- Gate behavior:
- uses `.venv/bin/python` when available (fallback `python`/`python3`)
- checks Textual runtime availability before running focused TUI v2 coverage suite
- runs focused tests with coverage output: `artifacts/coverage/coverage.tui2_gate.json`
- enforces thresholds via `scripts/check_critical_coverage.py --no-default-thresholds`:
- `src/seedpass/tui_v2/app.py >= 78%`
- `src/seedpass/core/api.py >= 85%`
- writes machine-readable result to `artifacts/coverage/critical_gate.tui2.json`.
- Strengthened default critical-coverage policy:
- `scripts/check_critical_coverage.py` default thresholds now include:
- `src/seedpass/core/api.py = 85%`.
- CI runner hardening:
- `scripts/run_ci_tests.sh` now enables `CRITICAL_COVERAGE_GATE` by default (`1`).
- adds JSON output for full-suite gate: `artifacts/coverage/critical_gate.full.json`.
- adds focused TUI v2 gate call controlled by `TUI2_COVERAGE_GATE` (default `1`).
- coverage-gate Python invocation now uses `python`/`python3` fallback resolution.
- Test/doc updates:
- `src/tests/test_check_critical_coverage.py` includes assertion for new default API threshold.
- docs updated in:
- `docs/ai_agent_tui_testing.md`
- `docs/security_readiness_checklist.md`
- `docs/dev_control_center.md`
- Validation run (this slice):
- `pytest -q src/tests/test_check_critical_coverage.py` -> `9 passed`
- `./scripts/tui2_coverage_gate.sh` -> PASS
- `src/seedpass/core/api.py: 86.64%`
- `src/seedpass/tui_v2/app.py: 79.87%`
- `python3 scripts/check_critical_coverage.py artifacts/coverage/coverage.json --json-output artifacts/coverage/critical_gate.full.json` -> PASS.
+1
View File
@@ -19,6 +19,7 @@ from pathlib import Path
DEFAULT_THRESHOLDS: dict[str, float] = {
"src/main.py": 50.0,
"src/seedpass/core/manager.py": 60.0,
"src/seedpass/core/api.py": 85.0,
"src/seedpass/core/encryption.py": 80.0,
"src/seedpass/core/migrations.py": 90.0,
}
+12 -2
View File
@@ -5,6 +5,11 @@ mkdir -p artifacts/coverage
./scripts/tui2_check_smoke.sh
py_bin="python"
if ! command -v "$py_bin" >/dev/null 2>&1; then
py_bin="python3"
fi
if [[ "${DETERMINISM_GATE:-1}" == "1" ]]; then
./scripts/run_determinism_tests.sh
fi
@@ -48,7 +53,12 @@ if [[ $status -eq 124 ]]; then
tail -n 20 pytest.log
exit 1
fi
if [[ $status -eq 0 && "${CRITICAL_COVERAGE_GATE:-0}" == "1" ]]; then
python scripts/check_critical_coverage.py artifacts/coverage/coverage.json
if [[ $status -eq 0 && "${CRITICAL_COVERAGE_GATE:-1}" == "1" ]]; then
"$py_bin" scripts/check_critical_coverage.py \
artifacts/coverage/coverage.json \
--json-output artifacts/coverage/critical_gate.full.json
fi
if [[ $status -eq 0 && "${TUI2_COVERAGE_GATE:-1}" == "1" ]]; then
./scripts/tui2_coverage_gate.sh
fi
exit $status
+43
View File
@@ -0,0 +1,43 @@
#!/usr/bin/env bash
set -euo pipefail
mkdir -p artifacts/coverage
if [[ -x ".venv/bin/python" ]]; then
py_bin=".venv/bin/python"
elif command -v python >/dev/null 2>&1; then
py_bin="python"
else
py_bin="python3"
fi
if ! "$py_bin" - <<'PY' >/dev/null 2>&1
import importlib.util
raise SystemExit(0 if importlib.util.find_spec("textual") else 1)
PY
then
if [[ "${REQUIRE_TEXTUAL:-0}" == "1" ]]; then
echo "Error: textual runtime is unavailable; cannot run TUI v2 coverage gate." >&2
exit 1
fi
echo "tui2 coverage gate: skipped (textual runtime unavailable)"
exit 0
fi
"$py_bin" -m pytest -q \
--cov=src \
--cov-report=json:artifacts/coverage/coverage.tui2_gate.json \
src/tests/test_tui_v2_helpers.py \
src/tests/test_tui_v2_textual_interactions.py \
src/tests/test_tui_v2_action_matrix.py \
src/tests/test_tui_v2_parity_scenarios.py \
src/tests/test_core_api_services.py
"$py_bin" scripts/check_critical_coverage.py \
artifacts/coverage/coverage.tui2_gate.json \
--no-default-thresholds \
--threshold src/seedpass/tui_v2/app.py=78 \
--threshold src/seedpass/core/api.py=85 \
--json-output artifacts/coverage/critical_gate.tui2.json
echo "tui2 coverage gate: ok"
@@ -23,6 +23,10 @@ def _load_module():
coverage_gate = _load_module()
def test_default_thresholds_include_tui2_service_module():
assert coverage_gate.DEFAULT_THRESHOLDS["src/seedpass/core/api.py"] == 85.0
def test_parse_threshold_overrides_valid():
parsed = coverage_gate._parse_threshold_overrides(
["src/main.py=40", "src/seedpass/core/manager.py=55.5"]