mirror of
https://github.com/PR0M3TH3AN/SeedPass.git
synced 2026-10-27 07:52:03 +00:00
docs: restate the review gate around oracles rather than people
The gate asked for a third party; there isn't one. What it was actually for was catching what the author's model misses, which needs an oracle that is not the author's judgement — not necessarily another person. Records the four substitutes in order of how little they depend on already knowing what to look for, marks differential fuzzing done, and states the technique's blind spot plainly: it can only find bugs where the two implementations DIFFER, so a flaw both share by design (like the recovery-split secrecy defect, found by reading) and anything TypeScript implements alone are both invisible to it. Those need mutation testing and adversarial framing, which is what remains. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01Vtx94nXDmNgXEeThUZP4QN
This commit is contained in:
1 parent
991c71189e
commit
f0a775d83c
2 files changed
+61
-5
No files matched your search
@@ -127,7 +127,32 @@ everything still open, in the order it should be tackled.
|
||||
|
||||
### Blockers before real secrets
|
||||
|
||||
- [ ] **Independent security review of the TypeScript branch.** Review
|
||||
- [~] **Independent security review — restated, since there is no third party.**
|
||||
The point of the gate was catching what the author cannot see, which
|
||||
needs an oracle that is not the author's judgement rather than another
|
||||
person. Progress and what remains:
|
||||
- [x] **Differential fuzzing vs Python** (`scripts/differential_fuzz.py`),
|
||||
~10,000 cases over canonical JSON, password derivation, the sync
|
||||
CRDT, index0, recovery shares, the semantic index and entry
|
||||
hashing. Found a real defect: integral floats hash differently in
|
||||
the two implementations, silently, which could make two clients
|
||||
converge to different vaults. Also found — via mutation testing
|
||||
turned on the fuzzer itself — that it could not reach the
|
||||
tombstone or subject caps; fixed and re-verified.
|
||||
- [ ] **Wire the fuzzer and `cross_impl_check.py` into CI.** Neither
|
||||
runs automatically, and this branch has a documented history of
|
||||
CI going unwatched (ts-parity red for 33 runs). A check nobody
|
||||
runs is not a check.
|
||||
- [ ] **Systematic mutation testing** of the security-critical modules
|
||||
that have NO differential oracle because Python has no
|
||||
equivalent: the session agent, the API HTTP layer, high-risk
|
||||
session handling, sinks. Ad-hoc mutation was applied to every fix
|
||||
in this branch, but never measured as coverage.
|
||||
- [ ] **Adversarial pass over TS-only surfaces**, framed as "what does
|
||||
this trust?" rather than "is this correct?" — the framing that
|
||||
finds trust-boundary bugs rather than logic bugs.
|
||||
Original note follows.
|
||||
- [ ] **(superseded by the above) Independent security review of the TypeScript branch.** Review
|
||||
strategy (decided 2026-08-18): two AI families plus vectors, because
|
||||
they fail differently. (1) `/code-review ultra` — **blocked**: PR #989
|
||||
spans main→port including beta's 150 commits (6795 files/720k lines),
|
||||
|
||||
@@ -21,12 +21,43 @@ P0 gate is green, Python remains the normative reference
|
||||
| Interactive mode (TUI) | green — legacy (v1) menu tree ported |
|
||||
| Packaging + release | green |
|
||||
| Migration + rollback story | green |
|
||||
| Independent security review | **not done — the one open blocker** |
|
||||
| Independent security review | **partly substituted — see below; (3) and (4) outstanding** |
|
||||
|
||||
The gates below are green. The remaining blocker is not a gate in this list:
|
||||
no third party has reviewed the code. Two rounds of subagent review found 9
|
||||
criticals/highs between them that self-review missed entirely, which is the
|
||||
evidence for treating that as load-bearing rather than optional.
|
||||
the code has not been reviewed by anyone who did not write it. Two rounds of
|
||||
subagent review found 9 criticals/highs between them that self-review missed
|
||||
entirely, which is the evidence for treating that as load-bearing rather than
|
||||
optional.
|
||||
|
||||
**There is no third party to hand this to.** So the blocker is restated in
|
||||
terms of what it was actually for: catching what the author's mental model
|
||||
misses. That does not require another person. It requires an ORACLE THAT IS
|
||||
NOT THE AUTHOR'S JUDGEMENT, and the substitutes available are ranked by how
|
||||
little they depend on someone already knowing what to look for:
|
||||
|
||||
1. **Differential testing against a second implementation.** `scripts/
|
||||
differential_fuzz.py` feeds randomly generated inputs to both
|
||||
implementations and compares canonical output. Any disagreement is a bug
|
||||
in one of them, found without anyone knowing in advance which answer is
|
||||
right. **Done**, ~10,000 cases across seven surfaces; it found a real
|
||||
silent-divergence defect in canonical number encoding on its first
|
||||
scaled run.
|
||||
2. **Differential testing against a third-party reference.** Used where one
|
||||
exists: the official BIP-85 spec vectors, Python's `qrcode` library for
|
||||
the QR encoder, Python's `Fernet` for the partition format.
|
||||
3. **Mutation testing.** Measures whether the suite would actually catch a
|
||||
regression, rather than whether it passes. Applied ad hoc to every fix in
|
||||
this branch; not yet systematic.
|
||||
4. **Adversarial tests written from the attacker's position** — the only
|
||||
technique available for surfaces the Python side does not have (the
|
||||
session agent, the API's HTTP layer, high-risk session handling), where
|
||||
there is no second implementation to disagree with.
|
||||
|
||||
**The known blind spot of (1):** it can only find bugs where the two
|
||||
implementations differ. A flaw both share by design is invisible to it — the
|
||||
recovery-split secrecy defect was found by reading, not fuzzing — as is
|
||||
anything TypeScript implements alone. Those need (3) and (4), which is what
|
||||
remains.
|
||||
|
||||
## P0 gates (must be green to cut over)
|
||||
|
||||
|
||||
Reference in new issue
Block a user