feat(semantic): port the retrieval index — and stop it writing secrets to disk

The "semantic (vector search)" entry in not_yet_ported was a misnomer. It is
not vector search and needs no model: MODEL_ID is "seedpass-token-overlap-v1",
and it scores by Jaccard overlap over tokenized metadata. That is why it could
be ported at all — there is nothing to embed.

Porting it surfaced two defects in the Python implementation.

**Stored secrets were written to a plaintext, 0664 file.** `_extract_text`
appended a key_value entry's `value` — the stored secret itself — and
`build()` wrote it to semantic_index/records.json in the clear, along with a
tokenized copy that leaked it just as thoroughly. The file was created at the
process umask, so on a default install it was group- and world-readable, next
to a vault that is encrypted and 0600. Anything that could read the directory
— a backup, a sync client, another user on a shared machine — got the secret
without the master password.

Fixed in Python (index the key NAME, never the value; chmod 0600 on both
files) and never done in TypeScript. The rule is worth keeping simple: nobody
searches for a secret they do not already know, so indexing one is pure
downside. MODEL_ID is bumped to v2 so an index built by the old code is
distinguishable — a v1 index on disk still contains the secrets and should be
rebuilt.

**Entry 0 was silently unsearchable.** The check was
`int(entry.get("id", 0) or 0) <= 0`, conflating "no id" with "id 0" — and
entry ids start at 0, so the first entry any user ever created was absent from
every search result with nothing to indicate why. Found by writing a test that
queried for it. Fixed in both implementations: skip an id that is absent,
unparseable or negative, which is what the check was reaching for.

The TypeScript port is wired into the CLI (`semantic build/status/search`),
the API (the /api/v1/semantic routes are now real rather than 501), and
writes through atomicWrite for a genuine 0600 inode. Fixtures come from
scripts/generate_ts_port_fixtures.py, which asserts the leaked value is absent
from its own output, so the fixture-drift job proves provenance the way it
does for every other fixture.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01Vtx94nXDmNgXEeThUZP4QN
This commit is contained in:
thePR0M3TH3ANandClaude Opus 5 committed 2026-08-19 09:54:53 -04:00
1 parent 51f1358da4
commit 39c4e45376
13 files changed
+1189 -11

No files matched your search

+72
View File
@@ -1285,6 +1285,77 @@ def gen_qr() -> dict:
}
def gen_semantic() -> dict:
"""Reference records and rankings for the retrieval index.
Two properties are being pinned. The rankings must match so a query
answered on one implementation is answered the same way on the other. And
the records must contain no secret: the file this produces is plaintext
beside an encrypted vault, so anything indexed is readable without the
master password. The key_value case below carries a value that must not
appear anywhere in the output.
"""
import tempfile
from seedpass.core.semantic_index import SemanticIndex
entries = [
{"id": 0, "kind": "password", "label": "first-ever.example",
"username": "zero", "notes": "the very first entry a profile creates",
"tags": ["edge"]},
{"id": 1, "kind": "password", "label": "bank.example", "username": "alice",
"url": "https://bank.example", "notes": "main current account",
"tags": ["money", "daily"]},
{"id": 2, "kind": "key_value", "label": "deploy-token", "key": "DEPLOY_TOKEN",
"value": "SECRET-MUST-NOT-APPEAR", "notes": "ci pipeline", "tags": ["ops"]},
{"id": 3, "kind": "document", "label": "recovery notes",
"content": "how to recover the bank account if locked out", "tags": ["docs"]},
{"id": 4, "kind": "totp", "label": "email-2fa", "issuer": "Fastmail",
"notes": "", "tags": ["email"]},
{"id": 5, "kind": "ssh", "label": "prod-server", "fingerprint": "SHA256:abcdef",
"notes": "production access", "tags": ["ops"]},
{"id": 6, "kind": "nostr", "label": "social", "npub": "npub1example",
"notes": "", "tags": []},
{"id": 7, "kind": "seed", "label": "child seed", "notes": "not an indexed kind"},
{"id": 8, "kind": "document", "label": "", "content": "", "notes": "", "tags": []},
{"id": 9, "kind": "password", "label": "linked", "notes": "", "tags": [],
"links": [{"relation": "depends_on", "note": "see recovery notes"}]},
{"kind": "document", "label": "no id at all", "content": "skipped"},
]
queries = [
"bank",
"bank account",
"ops production",
"email",
"recover locked",
"nothing matches here",
"DEPLOY_TOKEN",
"first entry",
]
with tempfile.TemporaryDirectory() as tmpdir:
index = SemanticIndex(Path(tmpdir))
index.build(entries)
records = json.loads(
(Path(tmpdir) / "semantic_index" / "records.json").read_text(
encoding="utf-8"
)
)
results = {q: index.search(q, k=10) for q in queries}
kind_filtered = index.search("ops", k=10, kind="ssh")
blob = json.dumps(records)
assert "SECRET-MUST-NOT-APPEAR" not in blob, "the index leaked a stored secret"
return {
"description": "semantic index records and rankings",
"entries": entries,
"records": records,
"queries": results,
"kind_filtered": kind_filtered,
}
def main() -> None:
FIXTURES_DIR.mkdir(parents=True, exist_ok=True)
@@ -1309,6 +1380,7 @@ def main() -> None:
"password_kdf.json": gen_password_kdf(),
"legacy_payloads.json": gen_legacy_payloads(),
"qr.json": gen_qr(),
"semantic.json": gen_semantic(),
}
entries_fixture, vault_fixture = gen_entries_and_vault()
files["entries_index.json"] = entries_fixture