diff --git a/TODO.md b/TODO.md index d2c476e8..5168b2f8 100644 --- a/TODO.md +++ b/TODO.md @@ -6,9 +6,9 @@ This file tracks the remaining work from the latest bug search and security eval - [ ] Complete checklist item #8 (Supply chain and release integrity): - run one tagged release through `.github/workflows/release-integrity.yml` and link evidence in `docs/security_readiness_checklist.md` - - resolve or formally time-box/own the `GHSA-wj6h-64fc-37mp` exception + - resolve or formally time-box/own the `GHSA-wj6h-64fc-37mp` exception (`docs/supply_chain_release_integrity.md`) - enforce release/tag protections requiring `Release Integrity` + dependency audit checks - - publish maintainer/consumer verification runbook (checksum + cosign verification with expected issuer/identity) + - publish maintainer/consumer verification runbook (checksum + cosign verification with expected issuer/identity) ✅ `docs/release_verification_runbook.md` - [ ] Upgrade vulnerable runtime dependencies in lockfiles and packaging inputs: - `aiohttp` -> `>=3.13.3` - `cryptography` -> `>=46.0.5` diff --git a/docs/README.md b/docs/README.md index 1a75fea8..069ecddf 100644 --- a/docs/README.md +++ b/docs/README.md @@ -20,6 +20,10 @@ This directory contains SeedPass project documentation. - `secret_handling_local_exposure_review.md` - `auth_lock_unlock_access_review.md` - `supply_chain_release_integrity.md` +- `release_verification_runbook.md` +- `release_protection_policy.md` +- `operational_runbooks.md` +- `staged_rollout_runbook.md` ## Architecture and Product Docs diff --git a/docs/agent-handoffs/incidents/2026-02-15-relay-health-preflight-job.md b/docs/agent-handoffs/incidents/2026-02-15-relay-health-preflight-job.md new file mode 100644 index 00000000..78cbab59 --- /dev/null +++ b/docs/agent-handoffs/incidents/2026-02-15-relay-health-preflight-job.md @@ -0,0 +1,33 @@ +# 2026-02-15-relay-health-preflight-job + +1. Context +- Users reported intermittent Nostr publish/retrieve failures caused by weak relay availability. +- Existing behavior could fail late in sync flows without a clear preflight decision point. + +2. Observation +- Relay health was not always checked before broader sync/release operations. +- Failures were often visible only after publish/retrieve attempts. + +3. Action Taken (Runbook Procedure) +- Before sync-sensitive operations, execute relay preflight: + 1. Run relay list review (`seedpass nostr list-relays`). + 2. Ensure at least `MIN_HEALTHY_RELAYS` are expected healthy. + 3. Run a background relay health check path and inspect warnings. +- If healthy relays are below threshold: + 1. Add/rotate relays (`seedpass nostr add-relay `). + 2. Re-run preflight. + 3. Defer broad sync/release operations until threshold is met. + +4. Validation Performed +- Confirm relay-health warning path is covered by tests: + - `src/tests/test_background_relay_check.py` + - `src/tests/test_background_error_reporting.py` +- Confirm deterministic failure-mode coverage for sync error paths: + - `src/tests/test_nostr_resilience_failure_modes.py` + +5. Recommendation for Next Agents +- Keep relay preflight as a mandatory step in staged rollout checklists. +- If relay health repeatedly degrades, open a dedicated incident note with: + - failing relay set, + - retry behavior observed, + - mitigation actions and final stable relay set. diff --git a/docs/agent-handoffs/incidents/2026-03-01-missing-runbook.md b/docs/agent-handoffs/incidents/2026-03-01-missing-runbook.md index 21489930..8817c470 100644 --- a/docs/agent-handoffs/incidents/2026-03-01-missing-runbook.md +++ b/docs/agent-handoffs/incidents/2026-03-01-missing-runbook.md @@ -2,6 +2,6 @@ 1. Context: `torch/TORCH.md` documents an incident runbook (`docs/agent-handoffs/incidents/2026-02-15-relay-health-preflight-job.md`) on line 402 for resolving relay publishing failures. 2. Observation: The file `docs/agent-handoffs/incidents/2026-02-15-relay-health-preflight-job.md` does not exist in the repository. -3. Action taken: Documented the missing file in `KNOWN_ISSUES.md`. -4. Validation performed: Verified using `ls` that the file is missing from the directory. -5. Recommendation for next agents: A proper incident runbook should be created, or the reference in `torch/TORCH.md` should be removed/updated if the runbook is not needed. +3. Action taken: Documented the missing file in `KNOWN_ISSUES.md`; later created the runbook at `docs/agent-handoffs/incidents/2026-02-15-relay-health-preflight-job.md`. +4. Validation performed: Verified runbook now exists and is linked from `docs/operational_runbooks.md`. +5. Recommendation for next agents: Keep incident runbooks discoverable via `docs/operational_runbooks.md` and update references when adding new playbooks. diff --git a/docs/dev_control_center.md b/docs/dev_control_center.md index e79902a4..77bb51b8 100644 --- a/docs/dev_control_center.md +++ b/docs/dev_control_center.md @@ -52,21 +52,22 @@ Use these docs as inputs, but treat this file as the decision layer: - #5 Nostr sync security/resilience: **In Progress** - #7 Testing gates/quality thresholds: **In Progress** - #8 Supply chain/release integrity: **In Progress** - - #9 Operational runbooks: **Not Started** - - #10 External audit/rollout: **Not Started** + - #9 Operational runbooks: **In Progress** + - #10 External audit/rollout: **In Progress** ### Supply Chain Specific - `docs/supply_chain_release_integrity.md` remaining work: - execute at least one tagged production release through integrity workflow and record evidence - - resolve/document exception `GHSA-wj6h-64fc-37mp` + - resolve or renew exception `GHSA-wj6h-64fc-37mp` before expiration - enforce release protection rules - - add/validate maintainer verification runbook + - execute and record maintainer verification runbook ## 4) Recommended Next Steps (Priority Order) 1. Advance Nostr resilience validation with optional real-relay soak lane after deterministic suites. 2. Finish supply-chain readiness evidence and release-protection policy. 3. Expand operational runbooks and staged rollout evidence for security readiness closure. +4. Execute a tagged release integrity run and attach evidence URLs in readiness checklist. ## 5) Next Slice Definition (Recommended Immediate Work) @@ -115,6 +116,12 @@ Progress update (2026-03-02): - Cutover memo refreshed with current gate status and blockers: - `docs/tui_v2_cutover_decision.md` updated (March 2, 2026) - documents parity-complete status, strengthened CI gates, and remaining hardening blockers +- Supply-chain and runbook docs centralized for closure work: + - `docs/release_verification_runbook.md` + - `docs/release_protection_policy.md` + - `docs/operational_runbooks.md` + - `docs/staged_rollout_runbook.md` + - `docs/agent-handoffs/incidents/2026-02-15-relay-health-preflight-job.md` ## 6) Working Rule diff --git a/docs/operational_runbooks.md b/docs/operational_runbooks.md new file mode 100644 index 00000000..3eacc476 --- /dev/null +++ b/docs/operational_runbooks.md @@ -0,0 +1,21 @@ +# Operational Runbooks + +This index tracks operational playbooks required by security readiness item #9. + +## Incident Runbooks + +- Relay health preflight and sync degradation: + - `docs/agent-handoffs/incidents/2026-02-15-relay-health-preflight-job.md` + +## Rollout/Recovery Runbooks + +- Staged rollout and rollback drill: + - `docs/staged_rollout_runbook.md` + +## Operational Readiness Minimum + +For each release cycle: +1. Execute at least one staged rollout drill from `docs/staged_rollout_runbook.md`. +2. Validate release artifact integrity using `docs/release_verification_runbook.md`. +3. Verify relay health preflight checks for active relays before broad rollout. +4. Log drill date, owner, and outcome in `docs/security_readiness_checklist.md`. diff --git a/docs/release_protection_policy.md b/docs/release_protection_policy.md new file mode 100644 index 00000000..a6b28bd1 --- /dev/null +++ b/docs/release_protection_policy.md @@ -0,0 +1,40 @@ +# Release Protection Policy + +Purpose: define repository protection requirements so releases cannot be published without passing integrity and audit gates. + +## Branch Protection (default branch) + +Require pull requests for protected branches and require status checks to pass before merge. + +Minimum required checks: +- `Tests / test (ubuntu-latest, 3.11)` +- `Dependency Audit / audit` (scheduled/manual job should also be runnable on demand) + +## Tag/Release Protection + +Publishable release tags must be created only from protected branch tips and must be gated by successful release integrity checks. + +Required controls: +1. Restrict who can create matching release tags (maintainers only). +2. Require successful `Release Integrity / verify-and-sign` before release publication. +3. Require successful dependency audit for the same commit/tag. +4. Require signed commits/tags per org policy where available. + +## Manual Enforcement Checklist (until fully automated in platform settings) + +Before publishing any release: +1. Confirm commit is on protected branch tip. +2. Run `Release Integrity` workflow and verify success. +3. Run `Dependency Audit` workflow and verify success (or documented time-boxed exception). +4. Verify release asset signature/checksum steps using `docs/release_verification_runbook.md`. + +## Evidence Recording + +For each release, capture: +- release tag, +- protected branch commit SHA, +- workflow run URLs (`Release Integrity`, `Dependency Audit`), +- reviewer/approver identity, +- verification runbook executor. + +Record evidence in `docs/security_readiness_checklist.md`. diff --git a/docs/release_verification_runbook.md b/docs/release_verification_runbook.md new file mode 100644 index 00000000..8f055315 --- /dev/null +++ b/docs/release_verification_runbook.md @@ -0,0 +1,66 @@ +# Release Verification Runbook + +Purpose: provide maintainers and consumers a deterministic process to verify SeedPass release artifacts and signatures. + +## Inputs + +- Release assets downloaded from GitHub release: + - `SHA256SUMS` + - `SHA256SUMS.sig` + - `SHA256SUMS.pem` + - distribution artifacts (`dist/*.whl`, `dist/*.tar.gz`) +- Local checkout of this repository (for verifier script). + +## 1) Verify artifact checksums + +```bash +python scripts/release_integrity.py verify --checksum-file SHA256SUMS --base-dir . +``` + +Expected result: no mismatches and process exits `0`. + +## 2) Verify keyless signature and certificate constraints + +```bash +cosign verify-blob \ + --certificate SHA256SUMS.pem \ + --signature SHA256SUMS.sig \ + --certificate-oidc-issuer https://token.actions.githubusercontent.com \ + --certificate-identity-regexp '^https://github.com/PR0M3TH3AN/SeedPass/.github/workflows/release-integrity.yml@refs/tags/.+$' \ + SHA256SUMS +``` + +Expected result: +- signature verification passes, +- certificate issuer is GitHub OIDC, +- certificate identity matches this repository `release-integrity.yml` workflow on a tag ref. + +## 3) Verify release workflow provenance (maintainers) + +1. Open the workflow run referenced by the release. +2. Confirm workflow name: `Release Integrity`. +3. Confirm job `verify-and-sign` succeeded. +4. Confirm uploaded assets include: + - `dist/*` + - `SHA256SUMS` + - `SHA256SUMS.sig` + - `SHA256SUMS.pem` + +## 4) Failure handling + +- If checksum verification fails: + - treat artifacts as untrusted, + - re-download release assets, + - if mismatch persists, open security incident and halt rollout. +- If signature verification fails: + - treat release as untrusted, + - verify you used the exact release assets, + - if still failing, open security incident and halt rollout. + +## 5) Record evidence + +For each release, record in `docs/security_readiness_checklist.md`: +- release tag, +- workflow run URL, +- verification executor/date, +- pass/fail status. diff --git a/docs/security_readiness_checklist.md b/docs/security_readiness_checklist.md index 1c228171..a704ef51 100644 --- a/docs/security_readiness_checklist.md +++ b/docs/security_readiness_checklist.md @@ -11,9 +11,9 @@ Status legend: `Not Started`, `In Progress`, `Blocked`, `Done` | 5 | Nostr sync security and resilience validation | In Progress | Core / QA | `src/tests/test_nostr_real.py`, `src/tests/test_nostr_index_size.py`, `src/tests/test_nostr_resilience_failure_modes.py` | | 6 | Auth, lock/unlock, and access-control hardening | Done | Core | `docs/auth_lock_unlock_access_review.md`, `src/tests/test_api_new_endpoints.py`, `src/tests/test_vault_lock_flag.py`, `src/tests/test_inactivity_lock.py`, `src/tests/test_unlock_sync.py` | | 7 | Testing gates and quality thresholds | In Progress | QA | `scripts/check_critical_coverage.py`, `scripts/tui2_coverage_gate.sh`, CI workflows | -| 8 | Supply chain and release integrity | In Progress | DevOps | `docs/supply_chain_release_integrity.md`, `.github/workflows/release-integrity.yml`, `scripts/release_integrity.py`, `src/tests/test_release_integrity_utils.py` | -| 9 | Operational readiness and incident runbooks | Not Started | Security / Ops | | -| 10 | External audit and staged production rollout | Not Started | Maintainers | | +| 8 | Supply chain and release integrity | In Progress | DevOps | `docs/supply_chain_release_integrity.md`, `docs/release_verification_runbook.md`, `docs/release_protection_policy.md`, `.github/workflows/release-integrity.yml`, `scripts/release_integrity.py`, `src/tests/test_release_integrity_utils.py` | +| 9 | Operational readiness and incident runbooks | In Progress | Security / Ops | `docs/operational_runbooks.md`, `docs/agent-handoffs/incidents/2026-02-15-relay-health-preflight-job.md` | +| 10 | External audit and staged production rollout | In Progress | Maintainers | `docs/staged_rollout_runbook.md` | ## Exit Criteria Per Item diff --git a/docs/staged_rollout_runbook.md b/docs/staged_rollout_runbook.md new file mode 100644 index 00000000..2affd87d --- /dev/null +++ b/docs/staged_rollout_runbook.md @@ -0,0 +1,59 @@ +# Staged Rollout and Rollback Drill Runbook + +Purpose: provide an operational procedure for controlled release rollout and verified rollback. + +## Preconditions + +Before rollout: +1. `scripts/run_ci_tests.sh` passes. +2. `scripts/tui2_coverage_gate.sh` passes. +3. `Release Integrity` and `Dependency Audit` workflows pass for release commit/tag. +4. Artifact verification passes (`docs/release_verification_runbook.md`). + +## Stage Plan + +1. Stage 0: Internal maintainer validation +- Install from release artifacts in a clean environment. +- Run core smoke: + - launch default UI, + - launch legacy fallback, + - verify vault unlock/read/search/edit/sync. + +2. Stage 1: Limited beta audience +- Announce release to a small, known test cohort. +- Monitor incident channels for: + - startup failures, + - sync failures, + - regression in retrieve/reveal/QR paths. + +3. Stage 2: Broad availability +- Proceed only if no Sev1/Sev2 regressions are open from Stage 1. + +## Rollback Trigger Criteria + +Rollback immediately when any of the following occur: +- deterministic data-loss/corruption report is reproducible, +- critical auth/lock/unlock regression, +- release artifact integrity verification failure, +- sustained sync failure above acceptable operational threshold. + +## Rollback Procedure + +1. Stop broad rollout communications. +2. Revert default routing or publish patch with legacy fallback default. +3. Publish incident notice with: + - affected versions, + - user mitigation path, + - expected follow-up timeline. +4. Open incident handoff note under `docs/agent-handoffs/incidents/`. + +## Drill Recording Template + +For each drill or production rollout: +- date: +- owner: +- release tag/commit: +- stage completed: +- pass/fail: +- rollback invoked (yes/no): +- incident reference (if any): diff --git a/docs/supply_chain_release_integrity.md b/docs/supply_chain_release_integrity.md index b43b4029..321e8439 100644 --- a/docs/supply_chain_release_integrity.md +++ b/docs/supply_chain_release_integrity.md @@ -58,13 +58,23 @@ cosign verify-blob \ - Workflow: `.github/workflows/release-integrity.yml` - Tooling: `scripts/release_integrity.py`, `src/seedpass/release_integrity.py` - Tests: `src/tests/test_release_integrity_utils.py` +- Maintainer verification runbook: `docs/release_verification_runbook.md` +- Release protection policy: `docs/release_protection_policy.md` + +## Temporary Vulnerability Exception Register + +| Advisory | Scope | Owner | Added | Expiration | Status | Compensating Controls | +|---|---|---|---|---|---|---| +| `GHSA-wj6h-64fc-37mp` / `CVE-2025-62727` | `pip-audit` ignore in CI/release workflows | Maintainers (DevOps) | 2026-02-25 | 2026-04-15 | Temporary | locked dependencies, weekly dependency-audit workflow, release-integrity verification/signing | + +Exception policy: +- Exception must have owner + expiration date. +- Exception must be removed or renewed with explicit risk review before expiration. +- Any renewal must be documented in this table and `docs/security_readiness_checklist.md`. ## Remaining Work To Mark Checklist Item #8 Done 1. Execute at least one tagged production release through `.github/workflows/release-integrity.yml` and record the run URL/artifact links in `docs/security_readiness_checklist.md`. -2. Eliminate the temporary audit exception (`GHSA-wj6h-64fc-37mp`) or document formal risk acceptance with expiration date and owner. -3. Add branch/release protection so publishable tags require successful `Release Integrity` and `Dependency Audit` checks before release publication. -4. Add maintainer verification runbook for release consumers that includes: - - `python scripts/release_integrity.py verify --checksum-file SHA256SUMS --base-dir .` - - `cosign verify-blob --certificate SHA256SUMS.pem --signature SHA256SUMS.sig SHA256SUMS` - - expected identity/issuer constraints for keyless signing. +2. Eliminate the temporary audit exception (`GHSA-wj6h-64fc-37mp`) before expiration (`2026-04-15`) or renew with explicit risk review. +3. Enforce branch/release protection so publishable tags require successful `Release Integrity` and `Dependency Audit` checks before release publication. +4. Execute and record at least one maintainer-side verification using `docs/release_verification_runbook.md`. diff --git a/memory-update.md b/memory-update.md index c9160e0d..4a86b377 100644 --- a/memory-update.md +++ b/memory-update.md @@ -746,3 +746,25 @@ - Aligned planning docs: - `docs/dev_control_center.md` next-step priorities updated to remove completed memo-refresh item and track current hardening priorities. - `docs/tui_v2_plan.md` status updated from draft wording to implemented/reference-plan wording. + +## 2026-03-02 Supply-chain/readiness documentation closure slice +- Added maintainer/consumer release verification runbook: + - `docs/release_verification_runbook.md` + - includes checksum verification and `cosign verify-blob` with explicit issuer/identity constraints. +- Added release protection policy document: + - `docs/release_protection_policy.md` + - defines required branch/tag/release controls and evidence recording expectations. +- Added operational readiness runbook index and staged rollout drill runbook: + - `docs/operational_runbooks.md` + - `docs/staged_rollout_runbook.md` +- Closed missing incident runbook gap by adding: + - `docs/agent-handoffs/incidents/2026-02-15-relay-health-preflight-job.md` + and updated follow-up note `2026-03-01-missing-runbook.md` to reflect resolution. +- Updated supply-chain integrity doc with: + - explicit temporary exception register entry for `GHSA-wj6h-64fc-37mp` / `CVE-2025-62727` + - owner + expiration (`2026-04-15`) + compensating controls + - revised remaining-work list focused on production evidence and enforcement. +- Updated readiness/control docs: + - `docs/security_readiness_checklist.md` rows 8/9/10 now reference new runbooks and moved 9/10 to `In Progress`. + - `docs/dev_control_center.md` and `docs/README.md` now link the new runbook docs. + - `TODO.md` supply-chain item now references published verification runbook.