docs: add supply-chain verification and operational runbooks

2026-11-03 03:12:07 +00:00 · 2026-03-02 09:14:00 -05:00
1 parent 57e73f6869
commit 949d87dd4f
@@ -6,9 +6,9 @@ This file tracks the remaining work from the latest bug search and security eval

 - [ ] Complete checklist item #8 (Supply chain and release integrity):
  - run one tagged release through `.github/workflows/release-integrity.yml` and link evidence in `docs/security_readiness_checklist.md`
-  - resolve or formally time-box/own the `GHSA-wj6h-64fc-37mp` exception
+  - resolve or formally time-box/own the `GHSA-wj6h-64fc-37mp` exception (`docs/supply_chain_release_integrity.md`)
  - enforce release/tag protections requiring `Release Integrity` + dependency audit checks
-  - publish maintainer/consumer verification runbook (checksum + cosign verification with expected issuer/identity)
+  - publish maintainer/consumer verification runbook (checksum + cosign verification with expected issuer/identity) ✅ `docs/release_verification_runbook.md`
 - [ ] Upgrade vulnerable runtime dependencies in lockfiles and packaging inputs:
  - `aiohttp` -> `>=3.13.3`
  - `cryptography` -> `>=46.0.5`
@@ -20,6 +20,10 @@ This directory contains SeedPass project documentation.
 - `secret_handling_local_exposure_review.md`
 - `auth_lock_unlock_access_review.md`
 - `supply_chain_release_integrity.md`
+- `release_verification_runbook.md`
+- `release_protection_policy.md`
+- `operational_runbooks.md`
+- `staged_rollout_runbook.md`

 ## Architecture and Product Docs

@@ -0,0 +1,33 @@
+# 2026-02-15-relay-health-preflight-job
+
+1. Context
+- Users reported intermittent Nostr publish/retrieve failures caused by weak relay availability.
+- Existing behavior could fail late in sync flows without a clear preflight decision point.
+
+2. Observation
+- Relay health was not always checked before broader sync/release operations.
+- Failures were often visible only after publish/retrieve attempts.
+
+3. Action Taken (Runbook Procedure)
+- Before sync-sensitive operations, execute relay preflight:
+  1. Run relay list review (`seedpass nostr list-relays`).
+  2. Ensure at least `MIN_HEALTHY_RELAYS` are expected healthy.
+  3. Run a background relay health check path and inspect warnings.
+- If healthy relays are below threshold:
+  1. Add/rotate relays (`seedpass nostr add-relay <url>`).
+  2. Re-run preflight.
+  3. Defer broad sync/release operations until threshold is met.
+
+4. Validation Performed
+- Confirm relay-health warning path is covered by tests:
+  - `src/tests/test_background_relay_check.py`
+  - `src/tests/test_background_error_reporting.py`
+- Confirm deterministic failure-mode coverage for sync error paths:
+  - `src/tests/test_nostr_resilience_failure_modes.py`
+
+5. Recommendation for Next Agents
+- Keep relay preflight as a mandatory step in staged rollout checklists.
+- If relay health repeatedly degrades, open a dedicated incident note with:
+  - failing relay set,
+  - retry behavior observed,
+  - mitigation actions and final stable relay set.
@@ -2,6 +2,6 @@

 1. Context: `torch/TORCH.md` documents an incident runbook (`docs/agent-handoffs/incidents/2026-02-15-relay-health-preflight-job.md`) on line 402 for resolving relay publishing failures.
 2. Observation: The file `docs/agent-handoffs/incidents/2026-02-15-relay-health-preflight-job.md` does not exist in the repository.
-3. Action taken: Documented the missing file in `KNOWN_ISSUES.md`.
-4. Validation performed: Verified using `ls` that the file is missing from the directory.
-5. Recommendation for next agents: A proper incident runbook should be created, or the reference in `torch/TORCH.md` should be removed/updated if the runbook is not needed.
+3. Action taken: Documented the missing file in `KNOWN_ISSUES.md`; later created the runbook at `docs/agent-handoffs/incidents/2026-02-15-relay-health-preflight-job.md`.
+4. Validation performed: Verified runbook now exists and is linked from `docs/operational_runbooks.md`.
+5. Recommendation for next agents: Keep incident runbooks discoverable via `docs/operational_runbooks.md` and update references when adding new playbooks.
@@ -52,21 +52,22 @@ Use these docs as inputs, but treat this file as the decision layer:
  - #5 Nostr sync security/resilience: **In Progress**
  - #7 Testing gates/quality thresholds: **In Progress**
  - #8 Supply chain/release integrity: **In Progress**
-  - #9 Operational runbooks: **Not Started**
-  - #10 External audit/rollout: **Not Started**
+  - #9 Operational runbooks: **In Progress**
+  - #10 External audit/rollout: **In Progress**

 ### Supply Chain Specific
 - `docs/supply_chain_release_integrity.md` remaining work:
  - execute at least one tagged production release through integrity workflow and record evidence
-  - resolve/document exception `GHSA-wj6h-64fc-37mp`
+  - resolve or renew exception `GHSA-wj6h-64fc-37mp` before expiration
  - enforce release protection rules
-  - add/validate maintainer verification runbook
+  - execute and record maintainer verification runbook

 ## 4) Recommended Next Steps (Priority Order)

 1. Advance Nostr resilience validation with optional real-relay soak lane after deterministic suites.
 2. Finish supply-chain readiness evidence and release-protection policy.
 3. Expand operational runbooks and staged rollout evidence for security readiness closure.
+4. Execute a tagged release integrity run and attach evidence URLs in readiness checklist.

 ## 5) Next Slice Definition (Recommended Immediate Work)

@@ -115,6 +116,12 @@ Progress update (2026-03-02):
 - Cutover memo refreshed with current gate status and blockers:
  - `docs/tui_v2_cutover_decision.md` updated (March 2, 2026)
  - documents parity-complete status, strengthened CI gates, and remaining hardening blockers
+- Supply-chain and runbook docs centralized for closure work:
+  - `docs/release_verification_runbook.md`
+  - `docs/release_protection_policy.md`
+  - `docs/operational_runbooks.md`
+  - `docs/staged_rollout_runbook.md`
+  - `docs/agent-handoffs/incidents/2026-02-15-relay-health-preflight-job.md`

 ## 6) Working Rule

@@ -0,0 +1,21 @@
+# Operational Runbooks
+
+This index tracks operational playbooks required by security readiness item #9.
+
+## Incident Runbooks
+
+- Relay health preflight and sync degradation:
+  - `docs/agent-handoffs/incidents/2026-02-15-relay-health-preflight-job.md`
+
+## Rollout/Recovery Runbooks
+
+- Staged rollout and rollback drill:
+  - `docs/staged_rollout_runbook.md`
+
+## Operational Readiness Minimum
+
+For each release cycle:
+1. Execute at least one staged rollout drill from `docs/staged_rollout_runbook.md`.
+2. Validate release artifact integrity using `docs/release_verification_runbook.md`.
+3. Verify relay health preflight checks for active relays before broad rollout.
+4. Log drill date, owner, and outcome in `docs/security_readiness_checklist.md`.
@@ -0,0 +1,40 @@
+# Release Protection Policy
+
+Purpose: define repository protection requirements so releases cannot be published without passing integrity and audit gates.
+
+## Branch Protection (default branch)
+
+Require pull requests for protected branches and require status checks to pass before merge.
+
+Minimum required checks:
+- `Tests / test (ubuntu-latest, 3.11)`
+- `Dependency Audit / audit` (scheduled/manual job should also be runnable on demand)
+
+## Tag/Release Protection
+
+Publishable release tags must be created only from protected branch tips and must be gated by successful release integrity checks.
+
+Required controls:
+1. Restrict who can create matching release tags (maintainers only).
+2. Require successful `Release Integrity / verify-and-sign` before release publication.
+3. Require successful dependency audit for the same commit/tag.
+4. Require signed commits/tags per org policy where available.
+
+## Manual Enforcement Checklist (until fully automated in platform settings)
+
+Before publishing any release:
+1. Confirm commit is on protected branch tip.
+2. Run `Release Integrity` workflow and verify success.
+3. Run `Dependency Audit` workflow and verify success (or documented time-boxed exception).
+4. Verify release asset signature/checksum steps using `docs/release_verification_runbook.md`.
+
+## Evidence Recording
+
+For each release, capture:
+- release tag,
+- protected branch commit SHA,
+- workflow run URLs (`Release Integrity`, `Dependency Audit`),
+- reviewer/approver identity,
+- verification runbook executor.
+
+Record evidence in `docs/security_readiness_checklist.md`.
@@ -0,0 +1,66 @@
+# Release Verification Runbook
+
+Purpose: provide maintainers and consumers a deterministic process to verify SeedPass release artifacts and signatures.
+
+## Inputs
+
+- Release assets downloaded from GitHub release:
+  - `SHA256SUMS`
+  - `SHA256SUMS.sig`
+  - `SHA256SUMS.pem`
+  - distribution artifacts (`dist/*.whl`, `dist/*.tar.gz`)
+- Local checkout of this repository (for verifier script).
+
+## 1) Verify artifact checksums
+
+```bash
+python scripts/release_integrity.py verify --checksum-file SHA256SUMS --base-dir .
+```
+
+Expected result: no mismatches and process exits `0`.
+
+## 2) Verify keyless signature and certificate constraints
+
+```bash
+cosign verify-blob \
+  --certificate SHA256SUMS.pem \
+  --signature SHA256SUMS.sig \
+  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
+  --certificate-identity-regexp '^https://github.com/PR0M3TH3AN/SeedPass/.github/workflows/release-integrity.yml@refs/tags/.+$' \
+  SHA256SUMS
+```
+
+Expected result:
+- signature verification passes,
+- certificate issuer is GitHub OIDC,
+- certificate identity matches this repository `release-integrity.yml` workflow on a tag ref.
+
+## 3) Verify release workflow provenance (maintainers)
+
+1. Open the workflow run referenced by the release.
+2. Confirm workflow name: `Release Integrity`.
+3. Confirm job `verify-and-sign` succeeded.
+4. Confirm uploaded assets include:
+   - `dist/*`
+   - `SHA256SUMS`
+   - `SHA256SUMS.sig`
+   - `SHA256SUMS.pem`
+
+## 4) Failure handling
+
+- If checksum verification fails:
+  - treat artifacts as untrusted,
+  - re-download release assets,
+  - if mismatch persists, open security incident and halt rollout.
+- If signature verification fails:
+  - treat release as untrusted,
+  - verify you used the exact release assets,
+  - if still failing, open security incident and halt rollout.
+
+## 5) Record evidence
+
+For each release, record in `docs/security_readiness_checklist.md`:
+- release tag,
+- workflow run URL,
+- verification executor/date,
+- pass/fail status.
@@ -11,9 +11,9 @@ Status legend: `Not Started`, `In Progress`, `Blocked`, `Done`
 | 5 | Nostr sync security and resilience validation | In Progress | Core / QA | `src/tests/test_nostr_real.py`, `src/tests/test_nostr_index_size.py`, `src/tests/test_nostr_resilience_failure_modes.py` |
 | 6 | Auth, lock/unlock, and access-control hardening | Done | Core | `docs/auth_lock_unlock_access_review.md`, `src/tests/test_api_new_endpoints.py`, `src/tests/test_vault_lock_flag.py`, `src/tests/test_inactivity_lock.py`, `src/tests/test_unlock_sync.py` |
 | 7 | Testing gates and quality thresholds | In Progress | QA | `scripts/check_critical_coverage.py`, `scripts/tui2_coverage_gate.sh`, CI workflows |
-| 8 | Supply chain and release integrity | In Progress | DevOps | `docs/supply_chain_release_integrity.md`, `.github/workflows/release-integrity.yml`, `scripts/release_integrity.py`, `src/tests/test_release_integrity_utils.py` |
-| 9 | Operational readiness and incident runbooks | Not Started | Security / Ops | |
-| 10 | External audit and staged production rollout | Not Started | Maintainers | |
+| 8 | Supply chain and release integrity | In Progress | DevOps | `docs/supply_chain_release_integrity.md`, `docs/release_verification_runbook.md`, `docs/release_protection_policy.md`, `.github/workflows/release-integrity.yml`, `scripts/release_integrity.py`, `src/tests/test_release_integrity_utils.py` |
+| 9 | Operational readiness and incident runbooks | In Progress | Security / Ops | `docs/operational_runbooks.md`, `docs/agent-handoffs/incidents/2026-02-15-relay-health-preflight-job.md` |
+| 10 | External audit and staged production rollout | In Progress | Maintainers | `docs/staged_rollout_runbook.md` |

 ## Exit Criteria Per Item

@@ -0,0 +1,59 @@
+# Staged Rollout and Rollback Drill Runbook
+
+Purpose: provide an operational procedure for controlled release rollout and verified rollback.
+
+## Preconditions
+
+Before rollout:
+1. `scripts/run_ci_tests.sh` passes.
+2. `scripts/tui2_coverage_gate.sh` passes.
+3. `Release Integrity` and `Dependency Audit` workflows pass for release commit/tag.
+4. Artifact verification passes (`docs/release_verification_runbook.md`).
+
+## Stage Plan
+
+1. Stage 0: Internal maintainer validation
+- Install from release artifacts in a clean environment.
+- Run core smoke:
+  - launch default UI,
+  - launch legacy fallback,
+  - verify vault unlock/read/search/edit/sync.
+
+2. Stage 1: Limited beta audience
+- Announce release to a small, known test cohort.
+- Monitor incident channels for:
+  - startup failures,
+  - sync failures,
+  - regression in retrieve/reveal/QR paths.
+
+3. Stage 2: Broad availability
+- Proceed only if no Sev1/Sev2 regressions are open from Stage 1.
+
+## Rollback Trigger Criteria
+
+Rollback immediately when any of the following occur:
+- deterministic data-loss/corruption report is reproducible,
+- critical auth/lock/unlock regression,
+- release artifact integrity verification failure,
+- sustained sync failure above acceptable operational threshold.
+
+## Rollback Procedure
+
+1. Stop broad rollout communications.
+2. Revert default routing or publish patch with legacy fallback default.
+3. Publish incident notice with:
+   - affected versions,
+   - user mitigation path,
+   - expected follow-up timeline.
+4. Open incident handoff note under `docs/agent-handoffs/incidents/`.
+
+## Drill Recording Template
+
+For each drill or production rollout:
+- date:
+- owner:
+- release tag/commit:
+- stage completed:
+- pass/fail:
+- rollback invoked (yes/no):
+- incident reference (if any):
@@ -58,13 +58,23 @@ cosign verify-blob \
 - Workflow: `.github/workflows/release-integrity.yml`
 - Tooling: `scripts/release_integrity.py`, `src/seedpass/release_integrity.py`
 - Tests: `src/tests/test_release_integrity_utils.py`
+- Maintainer verification runbook: `docs/release_verification_runbook.md`
+- Release protection policy: `docs/release_protection_policy.md`
+
+## Temporary Vulnerability Exception Register
+
+| Advisory | Scope | Owner | Added | Expiration | Status | Compensating Controls |
+|---|---|---|---|---|---|---|
+| `GHSA-wj6h-64fc-37mp` / `CVE-2025-62727` | `pip-audit` ignore in CI/release workflows | Maintainers (DevOps) | 2026-02-25 | 2026-04-15 | Temporary | locked dependencies, weekly dependency-audit workflow, release-integrity verification/signing |
+
+Exception policy:
+- Exception must have owner + expiration date.
+- Exception must be removed or renewed with explicit risk review before expiration.
+- Any renewal must be documented in this table and `docs/security_readiness_checklist.md`.

 ## Remaining Work To Mark Checklist Item #8 Done

 1. Execute at least one tagged production release through `.github/workflows/release-integrity.yml` and record the run URL/artifact links in `docs/security_readiness_checklist.md`.
-2. Eliminate the temporary audit exception (`GHSA-wj6h-64fc-37mp`) or document formal risk acceptance with expiration date and owner.
-3. Add branch/release protection so publishable tags require successful `Release Integrity` and `Dependency Audit` checks before release publication.
-4. Add maintainer verification runbook for release consumers that includes:
-   - `python scripts/release_integrity.py verify --checksum-file SHA256SUMS --base-dir .`
-   - `cosign verify-blob --certificate SHA256SUMS.pem --signature SHA256SUMS.sig SHA256SUMS`
-   - expected identity/issuer constraints for keyless signing.
+2. Eliminate the temporary audit exception (`GHSA-wj6h-64fc-37mp`) before expiration (`2026-04-15`) or renew with explicit risk review.
+3. Enforce branch/release protection so publishable tags require successful `Release Integrity` and `Dependency Audit` checks before release publication.
+4. Execute and record at least one maintainer-side verification using `docs/release_verification_runbook.md`.
@@ -746,3 +746,25 @@
 - Aligned planning docs:
  - `docs/dev_control_center.md` next-step priorities updated to remove completed memo-refresh item and track current hardening priorities.
  - `docs/tui_v2_plan.md` status updated from draft wording to implemented/reference-plan wording.
+
+## 2026-03-02 Supply-chain/readiness documentation closure slice
+- Added maintainer/consumer release verification runbook:
+  - `docs/release_verification_runbook.md`
+  - includes checksum verification and `cosign verify-blob` with explicit issuer/identity constraints.
+- Added release protection policy document:
+  - `docs/release_protection_policy.md`
+  - defines required branch/tag/release controls and evidence recording expectations.
+- Added operational readiness runbook index and staged rollout drill runbook:
+  - `docs/operational_runbooks.md`
+  - `docs/staged_rollout_runbook.md`
+- Closed missing incident runbook gap by adding:
+  - `docs/agent-handoffs/incidents/2026-02-15-relay-health-preflight-job.md`
+  and updated follow-up note `2026-03-01-missing-runbook.md` to reflect resolution.
+- Updated supply-chain integrity doc with:
+  - explicit temporary exception register entry for `GHSA-wj6h-64fc-37mp` / `CVE-2025-62727`
+  - owner + expiration (`2026-04-15`) + compensating controls
+  - revised remaining-work list focused on production evidence and enforcement.
+- Updated readiness/control docs:
+  - `docs/security_readiness_checklist.md` rows 8/9/10 now reference new runbooks and moved 9/10 to `In Progress`.
+  - `docs/dev_control_center.md` and `docs/README.md` now link the new runbook docs.
+  - `TODO.md` supply-chain item now references published verification runbook.