feat(semantic): port the retrieval index — and stop it writing secrets to disk

The "semantic (vector search)" entry in not_yet_ported was a misnomer. It is
not vector search and needs no model: MODEL_ID is "seedpass-token-overlap-v1",
and it scores by Jaccard overlap over tokenized metadata. That is why it could
be ported at all — there is nothing to embed.

Porting it surfaced two defects in the Python implementation.

**Stored secrets were written to a plaintext, 0664 file.** `_extract_text`
appended a key_value entry's `value` — the stored secret itself — and
`build()` wrote it to semantic_index/records.json in the clear, along with a
tokenized copy that leaked it just as thoroughly. The file was created at the
process umask, so on a default install it was group- and world-readable, next
to a vault that is encrypted and 0600. Anything that could read the directory
— a backup, a sync client, another user on a shared machine — got the secret
without the master password.

Fixed in Python (index the key NAME, never the value; chmod 0600 on both
files) and never done in TypeScript. The rule is worth keeping simple: nobody
searches for a secret they do not already know, so indexing one is pure
downside. MODEL_ID is bumped to v2 so an index built by the old code is
distinguishable — a v1 index on disk still contains the secrets and should be
rebuilt.

**Entry 0 was silently unsearchable.** The check was
`int(entry.get("id", 0) or 0) <= 0`, conflating "no id" with "id 0" — and
entry ids start at 0, so the first entry any user ever created was absent from
every search result with nothing to indicate why. Found by writing a test that
queried for it. Fixed in both implementations: skip an id that is absent,
unparseable or negative, which is what the check was reaching for.

The TypeScript port is wired into the CLI (`semantic build/status/search`),
the API (the /api/v1/semantic routes are now real rather than 501), and
writes through atomicWrite for a genuine 0600 inode. Fixtures come from
scripts/generate_ts_port_fixtures.py, which asserts the leaked value is absent
from its own output, so the fixture-drift job proves provenance the way it
does for every other fixture.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01Vtx94nXDmNgXEeThUZP4QN
This commit is contained in:
thePR0M3TH3ANandClaude Opus 5 committed 2026-08-19 09:54:53 -04:00
1 parent 51f1358da4
commit 39c4e45376
13 files changed
+1189 -11

No files matched your search

+66 -2
View File
@@ -10,7 +10,7 @@
*/
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { mkdtemp, mkdir, readFile, writeFile } from "node:fs/promises";
import { mkdtemp, mkdir, readFile, writeFile, stat } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import process from "node:process";
@@ -430,13 +430,77 @@ describe("vault export and import", () => {
});
});
describe("semantic index", () => {
it("builds, reports status, and ranks by metadata", async () => {
const built = await call("POST", "/api/v1/semantic/build");
expect(built.status).toBe(200);
expect(built.json.records).toBeGreaterThan(0);
const status = await call("GET", "/api/v1/semantic/status");
expect(status.json.built).toBe(true);
expect(status.json.model_id).toBe("seedpass-token-overlap-v2");
const hits = await call("POST", "/api/v1/semantic/search", { body: { query: "bank" } });
expect(hits.json.results.length).toBeGreaterThan(0);
expect(hits.json.results[0].ref).toMatch(/^sp:\/\/entry\//);
});
it("never writes a stored secret into the plaintext index", async () => {
await call("POST", "/api/v1/semantic/build");
// The vault holds a key_value entry whose value is this string. The
// index file sits unencrypted next to the vault, so the value must not
// reach it — nor a tokenized form of it.
const raw = await readFile(
join(app.profileDir(FINGERPRINT), "semantic_index", "records.json"),
"utf8",
);
expect(raw).not.toContain("kv-secret-value");
expect(raw.toLowerCase()).not.toContain("kv-secret-value".toLowerCase());
// But the entry is still findable by its name.
const hits = await call("POST", "/api/v1/semantic/search", { body: { query: "api-token" } });
expect(hits.json.results.map((r: any) => r.label)).toContain("api-token");
});
it("writes the index at 0600, not the process umask", async () => {
await call("POST", "/api/v1/semantic/build");
const path = join(app.profileDir(FINGERPRINT), "semantic_index", "records.json");
expect((await stat(path)).mode & 0o777).toBe(0o600);
});
it("says so rather than returning nothing when no index exists", async () => {
const fresh = new ApiServer({ host: "127.0.0.1", port: 0, token: TOKEN });
const other = await mkdtemp(join(tmpdir(), "seedpass-nosem-"));
const otherApp = new AppDir(other);
await otherApp.mutateFingerprints((d) => {
d.fingerprints.push(FINGERPRINT);
d.last_used = FINGERPRINT;
});
await mkdir(otherApp.profileDir(FINGERPRINT), { recursive: true });
await writeFile(
join(otherApp.profileDir(FINGERPRINT), INDEX_FILENAME),
await encryptV3(deriveIndexKeyBytes(MNEMONIC), utf8(JSON.stringify({ schema_version: 4, entries: {} }))),
);
registerRoutes(fresh, buildContext({ app: otherApp, fingerprint: FINGERPRINT, mnemonic: MNEMONIC }));
const bound = await fresh.listen();
try {
const res = await fetch(`http://${bound.host}:${bound.port}/api/v1/semantic/search`, {
method: "POST",
headers: { authorization: `Bearer ${TOKEN}`, "content-type": "application/json" },
body: JSON.stringify({ query: "anything" }),
});
expect(res.status).toBe(409);
} finally {
await fresh.close();
}
});
});
describe("the unported surface answers honestly", () => {
it("returns 501 with the reason, not 404", async () => {
for (const [method, path, feature] of [
["GET", "/api/v1/high-risk/status", "high-risk partitions"],
["GET", "/api/v1/agent/job-profiles", "agent job profiles"],
["POST", "/api/v1/agent/recovery/split", "agent recovery split"],
["POST", "/api/v1/semantic/search", "semantic"],
] as const) {
const res = await call(method, path);
// 404 would read as "you typed the path wrong" and send an integrator