storage-cors-3: prune tombstoned service snapshots and persisted feed cache

decisions/DECISIONS_2026-10-22_14-21-00_storage-cors-3.md

@@ -0,0 +1,4 @@
+- Dispatch independent audit confirmed7=3pass/4fail and routed buffer/service only; merged routing076ec802 after evidence67140d89, merged HEAD89c5494a.
+- Buffer defers deletion of active versions to existing client.recordTombstone; preserve guarded-event same-ID cleanup. No new deletion timestamp/key policy.
+- Service rejects deleted/invalid/client-tombstone-covered objects before author/access bypass and direct caching; prune affected cached copies using authoritative guard plus existing explicit deletion IDs. Persist cleaned optimistic snapshots and empty active results.
+- Separate buffer commit ed5fb649; service commit follows. No app/feed coordinator/client.js writes. Coordinator race remains open pending faithful registered-pipeline/view/disposal evidence and new routing.

docs/testing/storage-cors-3-integrity.md

@@ -38,3 +38,44 @@ Restored mutants: unconditional deletion fails all3 ordering scenarios; omitted
 An initial focused command named nonexistent `tests/video-event-buffer.test.mjs` and failed before tests. Corrected to existing `tests/nostr/videoEventBuffer.test.mjs`; no registration/assertion changes.
 
 No browser/full/build/live signer/cloud/relay qualification. Coordinator stale-render remains open, read-only and outside this change.
+
+## Service snapshot/persistence boundary
+
+```yaml
+test_integrity_note:
+  change_type: ["new_tests"]
+  scenarios:
+    - id: SCN-storage3-service-prune
+      given: "multiple cached versions and unrelated content"
+      when: "different-ID deletion marker and late older version arrive through actual subscription"
+      then: "service map, callback, emitter and persisted snapshot contain only unrelated content"
+    - id: SCN-storage3-service-empty
+      given: "single live version or an all-ineligible persisted cache"
+      when: "deletion empties feed or empty active cache loads"
+      then: "empty persistence replaces stale cache and replacement consumer sees no resurrected entry"
+    - id: SCN-storage3-service-persistent
+      given: "deleted and invalid persisted objects for public or author viewer"
+      when: "optimistic load filters the cache"
+      then: "only live content is notified, cached and persisted before author bypass"
+    - id: SCN-storage3-service-known-tombstone
+      given: "stale nondeleted persisted copy covered by authoritative client tombstone"
+      when: "reload or direct cache API encounters stale copy"
+      then: "stale copy stays absent while newer valid and unrelated entries survive"
+    - id: SCN-storage3-service-author
+      given: "legacy synthesized roots sharing d-tag across distinct authors"
+      when: "one author deletion then newer live content and replayed older deletion arrive"
+      then: "unrelated author and strictly newer content remain in every service snapshot"
+  observable_outcomes:
+    - "actual service map, callback, videos:updated and filtered localStorage snapshots"
+  determinism_controls:
+    - "same actual subscription/client fixture as buffer cases, explicit EOSE and zero real network attempts"
+  anti_cheat_rationale:
+    prevents: ["ID-only removal", "flag-only filtering", "author bypass of deleted state", "empty-persistence skip", "overbroad author cleanup", "over-mocking ingest"]
+  relaxation:
+    did_relax_any_assertion: false
+    if_true_explain_spec_basis: ""
+```
+
+New service tests: old service baseline 0pass/8fail; fixed8/8. Eight restored mutants all fail: old-service8; allow-deleted4; allow-invalid4; ignore-known-tombstone2; skip-map-prune5; skip-empty-persist1; skip-direct-cache-guard1; skip-optimistic-persist2. Every retained scenario rejects the old service and at least one selective mutant. Mutator `/tmp/storage3-service-mutants.py` restores product source in finally; logs `/tmp/storage3-service-mutant-results.log` and uniquely named `/tmp/storage3-<mutant>.log`.
+
+Existing live-feed regression exposed that an update may carry the original event ID and a separate deleted object rather than a shared client object. Preserve explicit deletion-ID removal alongside authoritative client tombstone cleanup; the regression was not modified. No new root/address matching implementation: service calls client `isOlderThanTombstone`, including its existing legacy/author key derivation. Existing access/privacy/NSFW/consumer/subscription behavior is covered by unchanged focused suites.

js/services/nostrService.js

@@ -1449,7 +1449,7 @@ export class NostrService {
 
     const map = this.ensureVideosMap();
     for (const video of videos) {
-      if (video && typeof video.id === "string" && video.id) {
+      if (video && typeof video.id === "string" && video.id && !this.isVideoUnavailable(video)) {
         map.set(video.id, video);
       }
     }
@@ -1516,11 +1516,17 @@ export class NostrService {
     return this.videosByAuthorIndex;
   }
 
+  isVideoUnavailable(video) {
+    return video?.deleted === true || video?.invalid === true ||
+      (typeof this.nostrClient?.isOlderThanTombstone === "function" &&
+        this.nostrClient.isOlderThanTombstone(video));
+  }
+
   shouldIncludeVideo(video, {
     blacklistedEventIds = new Set(),
     isAuthorBlocked = () => false,
   } = {}) {
-    if (!video || typeof video !== "object") {
+    if (!video || typeof video !== "object" || this.isVideoUnavailable(video)) {
       return false;
     }
 
@@ -1737,9 +1743,7 @@ export class NostrService {
         this.emit("videos:updated", { videos: filtered, reason });
 
         // Persist the fresh filtered result for next time
-        if (filtered.length > 0) {
-          persistFilteredVideos(filtered);
-        }
+        persistFilteredVideos(filtered);
 
         return filtered;
       };
@@ -1758,6 +1762,7 @@ export class NostrService {
 
           if (optimisticFiltered.length > 0) {
             this.cacheVideos(optimisticFiltered);
+            persistFilteredVideos(optimisticFiltered);
             hasOptimisticData = true;
             optimisticResult = optimisticFiltered;
 
@@ -1821,9 +1826,12 @@ export class NostrService {
 
             if (deletedUpdates.length > 0) {
               const map = this.ensureVideosMap();
-              for (const v of deletedUpdates) {
-                if (v.id) {
-                  map.delete(v.id);
+              const deletedIds = new Set(deletedUpdates.map((video) => video.id));
+              // Deletion markers have their own event IDs. Consult the
+              // client's authoritative timestamp/key guard for cached copies.
+              for (const [id, video] of map) {
+                if (deletedIds.has(id) || this.isVideoUnavailable(video)) {
+                  map.delete(id);
                 }
               }
               this.markAuthorIndexDirty();

test_logs/TEST_LOG_2026-10-22_14-21-00_storage-cors-3-service.md

@@ -0,0 +1,10 @@
+- Product buffer commit ed5fb64991ae4aaf5df4ac76f4bd13ca9633aea0; service edits not yet committed when checked.
+- `env -u NODE_TEST_CONTEXT node tests/storage-tombstone-service.test.mjs`: old service natural exit1,0pass/8fail; fixed exit0,8pass.
+- `python3 /tmp/storage3-service-mutants.py`: exit0,8 deliberately broken variants each exited1, product restored; failing scenario counts8/4/4/2/5/1/1/2.
+- Focused command: `env -u NODE_TEST_CONTEXT node scripts/run-targeted-tests.mjs tests/storage-tombstone-buffer.test.mjs tests/storage-tombstone-service.test.mjs tests/nostr/videoEventBuffer.test.mjs tests/nostr-delete-flow.test.mjs tests/nostr-service-live-feed-notification.test.mjs tests/nostr-service-access-control.test.mjs tests/nostr-service-access-control-ready.test.mjs tests/services/nostr-service.test.mjs tests/author-moderation-override.test.mjs tests/moderation-stage.test.mjs tests/app/feedCoordinator.test.mjs tests/app/feedCoordinatorWatchHistoryRefresh.test.mjs`.
+- Final focused exit0 across12 files:11 files summarize59 passing tests; existing access-control suite outputs5 named passing checks and exit0 but no final TAP summary. That output limit is disclosed; not called a complete footer. New tests13/13 across two files.
+- Initial focused run stopped at existing live-feed test4 (6pass/1fail); explicit same-event-ID removal restored alongside new cleanup; no test changes.
+- `env -u NODE_TEST_CONTEXT npm run lint`: initial sandbox exit2 at git-grep EPERM; approved unchanged command outside sandbox exits0,all11 stages.
+- `git diff --check`: exit0. Service net growth8 lines; grandfather limit passes.
+- Raw logs `/tmp/storage3-focused.log`, `/tmp/storage3-lint.log`, `/tmp/storage3-service-baseline.log`, `/tmp/storage3-service-mutant-results.log`.
+- No browser/full/build/queue. Coordinator stale-render remains unresolved/read-only. Live signer/relay/cloud/IndexedDB restart and DOM/full registered pipeline unverified.

tests/storage-tombstone-service.test.mjs

@@ -0,0 +1,85 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import { tombstoneFixture, videoEvent, ids, author, filteredStorageKey } from './test-helpers/storage-tombstone-fixture.mjs';
+
+const cachedVideo = (id, extra = {}) => ({ id, pubkey: author, videoRootId: id, created_at: 100, title: id, url: 'https://example.invalid/video.mp4', ...extra });
+const assertSnapshot = (f, expected) => {
+  assert.deepEqual(ids([...f.service.getVideosMap().values()]), expected);
+  assert.deepEqual(ids(f.notifications.at(-1).videos), expected);
+  assert.deepEqual(ids(f.updates.at(-1).videos), expected);
+  assert.deepEqual(ids(f.persisted()), expected);
+};
+
+test('different-ID tombstone removes every obsolete cached version and preserves unrelated live content', async t => {
+  const f = tombstoneFixture(t); await f.load();
+  await f.ingest(videoEvent('first', 100));
+  await f.ingest(videoEvent('edit', 150), videoEvent('other', 100, { root: 'other', pubkey: 'b'.repeat(64) }));
+  await f.ingest(videoEvent('delete', 200, { deleted: true }));
+  assertSnapshot(f, ['other']);
+  assert.deepEqual(ids(f.updates.at(-1).deleted), ['delete']);
+  await f.ingest(videoEvent('late', 120));
+  assertSnapshot(f, ['other']);
+});
+
+test('deletion-only empty result persists and second consumer receives no optimistic resurrection', async t => {
+  const f = tombstoneFixture(t); await f.load(); await f.ingest(videoEvent('first', 100));
+  await f.ingest(videoEvent('delete', 200, { deleted: true }));
+  assertSnapshot(f, []);
+  const count = f.notifications.length;
+  await f.load();
+  assert.deepEqual(f.notifications.slice(count).map(entry => ids(entry.videos)), [[]]);
+  assert.deepEqual(f.persisted(), []);
+});
+
+for (const viewer of ['public', 'author']) {
+  test(`persistent deleted and invalid entries are rejected before ${viewer} access bypass`, async t => {
+    const f = tombstoneFixture(t, { persisted: [cachedVideo('deleted', { deleted: true }), cachedVideo('invalid', { invalid: true }), cachedVideo('live')] });
+    if (viewer === 'author') f.client.pubkey = author;
+    const initial = await f.load();
+    assert.deepEqual(ids(initial), ['live']);
+    assert.equal(f.notifications.some(entry => entry.videos.some(video => video.deleted || video.invalid)), false);
+    assert.deepEqual(ids([...f.service.getVideosMap().values()]), ['live']);
+    assert.deepEqual(ids(f.persisted()), ['live']);
+  });
+}
+
+test('all-ineligible persistent cache is cleared even when active snapshot is empty', async t => {
+  const f = tombstoneFixture(t, { persisted: [cachedVideo('deleted', { deleted: true }), cachedVideo('invalid', { invalid: true })] });
+  assert.deepEqual(await f.load(), []);
+  assert.deepEqual(f.persisted(), []);
+  assert.deepEqual(f.notifications.map(entry => ids(entry.videos)), [[]]);
+});
+
+test('stale nondeleted persistent copies honor known tombstone while strictly newer and unrelated entries survive', async t => {
+  const f = tombstoneFixture(t); await f.load();
+  await f.ingest(videoEvent('old', 100));
+  const stale = f.persisted()[0];
+  await f.ingest(videoEvent('delete', 200, { deleted: true }));
+  await f.ingest(videoEvent('newer', 300), videoEvent('other', 150, { root: 'other' }));
+  // A stale storage snapshot contains an independent, still-nondeleted copy.
+  localStorage.setItem(filteredStorageKey, JSON.stringify([stale, ...f.persisted()]));
+  const start = f.notifications.length;
+  await f.load();
+  assert.equal(f.notifications.slice(start).some(entry => entry.videos.some(video => video.id === 'old')), false);
+  assertSnapshot(f, ['newer', 'other']);
+});
+
+test('direct cache API never reintroduces deleted invalid or tombstone-covered copies', async t => {
+  const f = tombstoneFixture(t); await f.load(); await f.ingest(videoEvent('old', 100));
+  const stale = f.persisted()[0];
+  await f.ingest(videoEvent('delete', 200, { deleted: true }));
+  f.service.cacheVideos([stale, cachedVideo('deleted', { deleted: true }), cachedVideo('invalid', { invalid: true }), cachedVideo('live')]);
+  assert.deepEqual(ids([...f.service.getVideosMap().values()]), ['live']);
+});
+
+test('legacy tombstone cleanup retains other author with same d-tag and strictly newer content', async t => {
+  const f = tombstoneFixture(t); await f.load();
+  const legacy = { root: `LEGACY:${author}:same`, dTag: 'same' };
+  const other = { root: `LEGACY:${'b'.repeat(64)}:same`, dTag: 'same', pubkey: 'b'.repeat(64) };
+  await f.ingest(videoEvent('old', 100, legacy), videoEvent('other-author', 100, other));
+  await f.ingest(videoEvent('delete', 200, { ...legacy, deleted: true }));
+  assertSnapshot(f, ['other-author']);
+  await f.ingest(videoEvent('newer', 300, legacy));
+  await f.ingest(videoEvent('delete-replayed', 200, { ...legacy, deleted: true }));
+  assertSnapshot(f, ['newer', 'other-author']);
+});