docs: restate the review gate around oracles rather than people

The gate asked for a third party; there isn't one. What it was actually for
was catching what the author's model misses, which needs an oracle that is not
the author's judgement — not necessarily another person.

Records the four substitutes in order of how little they depend on already
knowing what to look for, marks differential fuzzing done, and states the
technique's blind spot plainly: it can only find bugs where the two
implementations DIFFER, so a flaw both share by design (like the recovery-split
secrecy defect, found by reading) and anything TypeScript implements alone are
both invisible to it. Those need mutation testing and adversarial framing,
which is what remains.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01Vtx94nXDmNgXEeThUZP4QN
This commit is contained in:
thePR0M3TH3ANandClaude Opus 5 committed 2026-08-19 11:45:01 -04:00
1 parent 991c71189e
commit f0a775d83c
2 files changed
+61 -5

No files matched your search

+26 -1
View File
@@ -127,7 +127,32 @@ everything still open, in the order it should be tackled.
### Blockers before real secrets
- [ ] **Independent security review of the TypeScript branch.** Review
- [~] **Independent security review — restated, since there is no third party.**
The point of the gate was catching what the author cannot see, which
needs an oracle that is not the author's judgement rather than another
person. Progress and what remains:
- [x] **Differential fuzzing vs Python** (`scripts/differential_fuzz.py`),
~10,000 cases over canonical JSON, password derivation, the sync
CRDT, index0, recovery shares, the semantic index and entry
hashing. Found a real defect: integral floats hash differently in
the two implementations, silently, which could make two clients
converge to different vaults. Also found — via mutation testing
turned on the fuzzer itself — that it could not reach the
tombstone or subject caps; fixed and re-verified.
- [ ] **Wire the fuzzer and `cross_impl_check.py` into CI.** Neither
runs automatically, and this branch has a documented history of
CI going unwatched (ts-parity red for 33 runs). A check nobody
runs is not a check.
- [ ] **Systematic mutation testing** of the security-critical modules
that have NO differential oracle because Python has no
equivalent: the session agent, the API HTTP layer, high-risk
session handling, sinks. Ad-hoc mutation was applied to every fix
in this branch, but never measured as coverage.
- [ ] **Adversarial pass over TS-only surfaces**, framed as "what does
this trust?" rather than "is this correct?" — the framing that
finds trust-boundary bugs rather than logic bugs.
Original note follows.
- [ ] **(superseded by the above) Independent security review of the TypeScript branch.** Review
strategy (decided 2026-08-18): two AI families plus vectors, because
they fail differently. (1) `/code-review ultra` — **blocked**: PR #989
spans main→port including beta's 150 commits (6795 files/720k lines),
+35 -4
View File
@@ -21,12 +21,43 @@ P0 gate is green, Python remains the normative reference
| Interactive mode (TUI) | green — legacy (v1) menu tree ported |
| Packaging + release | green |
| Migration + rollback story | green |
| Independent security review | **not done — the one open blocker** |
| Independent security review | **partly substituted — see below; (3) and (4) outstanding** |
The gates below are green. The remaining blocker is not a gate in this list:
no third party has reviewed the code. Two rounds of subagent review found 9
criticals/highs between them that self-review missed entirely, which is the
evidence for treating that as load-bearing rather than optional.
the code has not been reviewed by anyone who did not write it. Two rounds of
subagent review found 9 criticals/highs between them that self-review missed
entirely, which is the evidence for treating that as load-bearing rather than
optional.
**There is no third party to hand this to.** So the blocker is restated in
terms of what it was actually for: catching what the author's mental model
misses. That does not require another person. It requires an ORACLE THAT IS
NOT THE AUTHOR'S JUDGEMENT, and the substitutes available are ranked by how
little they depend on someone already knowing what to look for:
1. **Differential testing against a second implementation.** `scripts/
differential_fuzz.py` feeds randomly generated inputs to both
implementations and compares canonical output. Any disagreement is a bug
in one of them, found without anyone knowing in advance which answer is
right. **Done**, ~10,000 cases across seven surfaces; it found a real
silent-divergence defect in canonical number encoding on its first
scaled run.
2. **Differential testing against a third-party reference.** Used where one
exists: the official BIP-85 spec vectors, Python's `qrcode` library for
the QR encoder, Python's `Fernet` for the partition format.
3. **Mutation testing.** Measures whether the suite would actually catch a
regression, rather than whether it passes. Applied ad hoc to every fix in
this branch; not yet systematic.
4. **Adversarial tests written from the attacker's position** — the only
technique available for surfaces the Python side does not have (the
session agent, the API's HTTP layer, high-risk session handling), where
there is no second implementation to disagree with.
**The known blind spot of (1):** it can only find bugs where the two
implementations differ. A flaw both share by design is invisible to it — the
recovery-split secrecy defect was found by reading, not fuzzing — as is
anything TypeScript implements alone. Those need (3) and (4), which is what
remains.
## P0 gates (must be green to cut over)