From 2a979c2eca6d8de7cfebf99de4c62b54c8d6e908 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Sun, 21 Jun 2026 22:23:32 -0400 Subject: [PATCH 01/43] =?UTF-8?q?docs(deploy):=20record=20release-channel?= =?UTF-8?q?=20=E2=86=92=20Vercel=20project=20promotion=20+=20on-demand=20d?= =?UTF-8?q?eploy?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Captures the 3-project mapping (bitvid-unstable/beta/bitvid → unstable.bitvid.network / bitvid-beta.vercel.app / bitvid.network), the vercel.json knobs (npm install pin, deploymentEnabled.unstable), the promotion git steps (merge unstable→beta taking unstable's tree, ff main), per-project on-demand deploy commands, and the version-marker verification + soak-skip caveat. Co-Authored-By: Claude Opus 4.8 --- docs/deployment-notes.md | 69 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) diff --git a/docs/deployment-notes.md b/docs/deployment-notes.md index 5a1dc95b..7579d804 100644 --- a/docs/deployment-notes.md +++ b/docs/deployment-notes.md @@ -12,3 +12,72 @@ - Keep source HTML entry points (`index.html`, `embed.html`) pointing at logical asset paths (for example, `js/index.js` and `css/tailwind.generated.css`). The build pipeline rewrites those references to content-hashed filenames in `dist/`. - Deploy pipelines should always purge HTML entrypoints (`/` + `/index.html` and `/embed.html`) at the edge after publish so clients do not keep serving stale source shells. - App routing query parameters (such as `?v=` for video pointers) are unchanged and remain part of runtime URL behavior. + +## Release channels → Vercel projects (promotion + on-demand deploy) + +The three release branches each map to a **separate Vercel project** under the +`pr0m3th3ans-projects` team: + +| Branch | Vercel project | Live URL | +|------------|-------------------|---------------------------| +| `unstable` | `bitvid-unstable` | `unstable.bitvid.network` | +| `beta` | `bitvid-beta` | `bitvid-beta.vercel.app` | +| `main` | `bitvid` | `bitvid.network` (prod) | + +The repo's local `.vercel/project.json` is normally linked to **`bitvid-unstable`** +so day-to-day `vercel deploy --prod` targets the unstable site. + +### vercel.json knobs +- `installCommand: "npm install"` is pinned because both `package-lock.json` and + `pnpm-lock.yaml` exist; without the pin Vercel auto-selects pnpm and the build + fails on the unstable/beta projects. +- `git.deploymentEnabled.unstable: false` disables auto-deploy for a branch named + `unstable`. It only affects that branch key, so it's harmless on the beta/main + projects (they deploy `beta`/`main`). The bitvid-beta/bitvid projects ARE + git-connected, so a push to `beta`/`main` also auto-triggers a build — explicit + CLI deploys (below) just make the result deterministic. + +### Promotion (unstable → beta → main) +`unstable` is the integration branch. `main` is usually a clean ancestor +(fast-forward). `beta` can carry an extra superseded commit (e.g. an old +`vercel.json` pin), so prefer a merge that takes unstable's tree: + +```bash +# beta: merge unstable, preferring unstable on conflict; verify trees match +git checkout beta && git reset --hard origin/beta +git merge origin/unstable -X theirs --no-edit -m "Promote unstable → beta: " +git diff --quiet origin/unstable HEAD && echo "beta tree == unstable" # expect 0 +git push origin beta + +# main: fast-forward to unstable +git checkout main && git reset --hard origin/main +git merge --ff-only origin/unstable +git push origin main +``` + +### Build/deploy each project (on-demand) +```bash +# beta +vercel link --yes --project bitvid-beta && vercel deploy --prod --yes +# production +vercel link --yes --project bitvid && vercel deploy --prod --yes # aliases to bitvid.network +# restore the day-to-day link +vercel link --yes --project bitvid-unstable +``` + +### Verify +The footer version marker (`v: <8hex> • `) is content-hash based; all three +projects build from the same source in the same Vercel env, so the markers should +**match each other** after a promotion. Verify per domain and confirm a known new +module is served, e.g.: + +```bash +for host in unstable.bitvid.network bitvid-beta.vercel.app bitvid.network; do + echo -n "$host : "; curl -s "https://$host/index.html?cb=$(date +%s)" \ + | grep -oE 'v: [0-9a-f]{8} • [0-9-]+' | head -1 +done +``` + +> Caveat: promoting a large batch straight to `main` skips the `beta` soak. Do it +> only deliberately; otherwise let changes soak on `beta` first. Rollback = redeploy +> the previous `bitvid` production deployment (`vercel rollback` or redeploy by URL). From bd056b2e3b03b6229e159fc2989fb099875d4abf Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Sun, 21 Jun 2026 22:40:57 -0400 Subject: [PATCH 02/43] docs(nip71): interop migration plan + refresh stale vendored NIP-71 spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds docs/nip71-migration-plan.md — the agreed spec to build from: - Dual-event, opt-in (off by default): keep canonical kind 30078; add addressable 34235/34236 mirror (d=videoRootId, edits in lockstep). - Research correction: 34235/36 are NOT deprecated — current NIP-71 designates the addressable pair for content that needs post-publication updates (exactly bitvid's edit-in-place model). - WebTorrent rides STANDARD NIP-94 imeta fields (magnet, i) rather than a hack; private videos never mirrored; HTTPS url required to mirror. - Phases: 0 retarget builders → 1 outbound publish/edit/delete → 1.5 NIP-89 "Open in bitvid" handler + NIP-51 kind 30005 portable playlists → 2 inbound ingest. Also refreshes docs/nips/71.md (was stale: only 21/22; now the current 233-line spec with 34235/36 addressable kinds, waveform, origin, multi-audio). Updates TODO #17 to point at the plan. Co-Authored-By: Claude Opus 4.8 --- docs/nip71-migration-plan.md | 154 +++++++++++++++++++++++++++++ docs/nips/71.md | 102 ++++++++++++++++++- todo/TODO_2026-06-20_pre-launch.md | 24 +++-- 3 files changed, 268 insertions(+), 12 deletions(-) create mode 100644 docs/nip71-migration-plan.md diff --git a/docs/nip71-migration-plan.md b/docs/nip71-migration-plan.md new file mode 100644 index 00000000..241dacbf --- /dev/null +++ b/docs/nip71-migration-plan.md @@ -0,0 +1,154 @@ +# NIP-71 Interop Plan for bitvid + +Status: **planning / spec** (no code yet). Last updated 2026-06-22. + +Goal: let a publisher **opt in** to making a bitvid video fully NIP-71 compliant so +it (a) shows up on other Nostr video clients (Nostube, Amethyst, Flare, zap.stream +recordings, …) and (b) still carries bitvid's extra data (WebTorrent, flags) for a +richer experience inside bitvid. Eventually bitvid also **ingests** other clients' +NIP-71 videos. Decisions below were made with the maintainer; see "Decisions". + +--- + +## Decisions (locked) + +1. **Kind: addressable `34235` (normal) / `34236` (short).** +2. **Opt-in, off by default** — per-video toggle; private videos never mirrored. +3. **Scope: outbound first, inbound later** (Phase 1 vs Phase 2). + +### Why 34235/34236 (research-validated) + +The earlier assumption that 34235/36 were "deprecated legacy" was **wrong**. The +current NIP-71 (master) defines **all four** kinds and explicitly designates the +addressable pair for editable content: + +> "For content that may need updates after publication (such as correcting +> metadata, descriptions, or handling URL migrations), addressable versions are +> available: Kind `34235` … Kind `34236` … include a `d` tag as a unique +> identifier and can be updated while maintaining the same addressable reference." + +That is **exactly** bitvid's model (replaceable kind 30078, edit-in-place via +`d` = `videoRootId`). So 34235/36 is not a compromise — it's the spec-recommended +kind for what bitvid does, and it's what Nostube/Goblinbox consume today. + +> Hedge: keep the builder **kind-parameterized** so we can also emit / switch to +> `21`/`22` (regular) later without a rewrite if the ecosystem leans that way. + +--- + +## Architecture: dual-event, additive + +- **Canonical stays kind 30078** (NIP-78 app-data), unchanged. Source of truth for + WebTorrent, private videos, moderation, view counts — none of which NIP-71 models. +- **Mirror = kind 34235/34236**, addressable with **`d` = `videoRootId`** (same + d-tag as the 30078), so the mirror edits/deletes in **lockstep** with the canonical. +- Correlation is automatic via `(pubkey, videoRootId)`. We also set: + - `["origin", "bitvid", "", "https://bitvid.network/?v="]` + so other clients can attribute + deep-link back ("watch on bitvid"). + - Optionally `["a", "30078::"]` back-pointer (harmless elsewhere). + +--- + +## Field mapping: 30078 JSON → 34235/36 tags + +Sourced against NIP-71 + NIP-92 (`imeta`) + NIP-94 (file metadata fields). + +| bitvid (30078 content) | NIP-71 / imeta | notes | +|---|---|---| +| — | `["d", videoRootId]` | required for addressable; lockstep with 30078 | +| `title` | `["title", …]` | required | +| `description` | `.content` and `["alt", …]` | | +| first publish time | `["published_at", …]` | **stable across edits** (no reordering elsewhere) | +| `url` (HTTPS) | `imeta url <…>` | primary playable source for foreign clients | +| derived MIME | `imeta m <…>` | not stored today → derive from extension, default `video/mp4` | +| `thumbnail` | `imeta image <…>` | | +| `fileSha256` | `imeta x <…>` | | +| `originalFileSha256` | `imeta ox <…>` | | +| `magnet` | `imeta magnet ` | **standard NIP-94 field** — torrent-aware clients can use it | +| `infoHash` | `imeta i ` | **standard NIP-94 field** | +| `ws` / `xs` | encoded into the `magnet` URI (`&ws=`/`&xs=`) | also kept on 30078 canonical | +| `isNsfw` | `["content-warning", ""]` | standard | +| hashtags (if/when added) | `["t", …]` (repeated) | bitvid has no hashtags field yet — future | +| `isForKids`, `mode`, etc. | bitvid-namespaced extra tags | ignored by other clients | +| — | `["client", "bitvid"]` | | + +**Two hard rules that fall out of this:** + +1. **Private videos are never mirrored.** NIP-71 has no encryption; mirroring would + leak them. The toggle is disabled (with explanation) for private videos. +2. **A hosted HTTPS `url` is required to mirror.** Foreign clients can't play a + magnet-only video; bitvid's magnet still rides along (`magnet`/`i`) for + torrent-aware clients, but `url` must be present or the toggle is disabled. + +**Timestamp hygiene:** `created_at` = actual signing time; `published_at` = display +time. (Avoids the future-`created_at` weirdness seen in some published events.) + +--- + +## Phases + +### Phase 0 — Retarget builders (no UX; flag stays off) +- Point existing `buildNip71VideoEvent` at **34235/36 + `d`=`videoRootId`** (it + currently targets 21/22). Implement the field mapping incl. `magnet`/`i`/`ox`. +- Normal-vs-short selection: default 34235; choose 34236 when known-portrait + (dimensions when available; otherwise a manual "short" toggle). +- Cheat-resistant tests: bitvid video → spec-valid 34235 event → round-trips back + via `convertEventToVideo`; magnet/infohash present; private refused; url-less refused. + +### Phase 1 — Outbound publish/edit/delete (the headline feature) +- Edit form toggle: **"Also publish to other Nostr video apps (NIP-71)"** — off by + default, gated to public + HTTPS-url videos. +- Publish: 30078 (as today) **+** the 34235/36 mirror to the write-relay set + (reuse `getDeletePublishRelays` write set + the publish-outcome toast). +- Lifecycle parity: **edit** updates both (same d-tag); **delete** NIP-09s both + addresses; **toggle-off** NIP-09s the mirror (removes it from other clients). +- Mutation-verified lifecycle tests, then flip `FEATURE_PUBLISH_NIP71` on. + +### Phase 1.5 — Discoverability glue (cheap, high interop value) +- **NIP-89 handler registration** (kind `31990`): publish once that bitvid handles + video kinds `21/22/34235/34236` with a `web` URL template. Result: other clients + (Amethyst, etc.) can show **"Open in bitvid"** when they encounter a video event. +- **NIP-51 video curation sets** (kind `30005`): publish/read bitvid playlists as + portable video sets so playlists interop across clients. (Sets reference videos + by `e`/`a` tags.) + +### Phase 2 — Inbound ingest (separate, bigger) +- Feed source querying `34235/36` (and `21/22`) from relays; map via + `convertEventToVideo`. +- Dedup vs bitvid-native (by `a`-address / `x` hash / `url`), moderation + trust + gating (reuse blacklist / NIP-51 mute), and a **distinct discovery surface** so + foreign videos read as "from the wider Nostrverse," not bitvid-native. +- Honor `origin` for attribution; respect the no-URL-allowlist rule but apply + existing moderation. +- (Cross-client comments/zaps via NIP-22 `kind:1111` addressed to the 34235 event + is a further follow-on — bitvid comments currently target 30078.) + +--- + +## Enrichment backlog (nice-to-have, not v1) +`dim`, `duration`, `bitrate`, `waveform`, multiple `imeta` variants + multi-audio +(`ov`/language) tracks, `text-track` (WebVTT captions/subtitles/chapters), +`segment` (chapters) — all need more captured at upload time. v1 ships the minimal +spec-valid imeta (`url`/`m`/`image` + `magnet`/`i`) and grows from there. + +--- + +## Risks / watch-items +- **Editing addressable events across foreign clients**: some cache aggressively; + edits may lag elsewhere. Acceptable; document it. +- **Magnet-only catalogs**: those videos simply won't appear on non-torrent clients + (by design — they still work in bitvid). +- **Kind drift**: if the ecosystem consolidates on 21/22, add/dual-emit later + (builder stays kind-parameterized). +- **Ingest safety (Phase 2)**: arbitrary foreign media + metadata → keep moderation, + no autoplay of untrusted sources, sanitize before render. + +--- + +## Sources +- NIP-71 Video Events (current): https://github.com/nostr-protocol/nips/blob/master/71.md (vendored at `docs/nips/71.md`) +- NIP-92 inline metadata (`imeta`): https://github.com/nostr-protocol/nips/blob/master/92.md +- NIP-94 file metadata (`magnet`, `i`, `x`, `ox`, `fallback`, …): https://github.com/nostr-protocol/nips/blob/master/94.md +- NIP-89 recommended application handlers (`31989`/`31990`): https://github.com/nostr-protocol/nips/blob/master/89.md +- NIP-51 lists/sets incl. kind `30005` video curation: https://github.com/nostr-protocol/nips/blob/master/51.md +- Existing bitvid code: `js/nostr/nip71.js` (builders/parser), `js/nostrEventSchemas.js` (`NIP71_VIDEO`/`NIP71_SHORT_VIDEO`), `js/constants.js` (`FEATURE_PUBLISH_NIP71`). diff --git a/docs/nips/71.md b/docs/nips/71.md index 303ec5a3..e55e470d 100644 --- a/docs/nips/71.md +++ b/docs/nips/71.md @@ -20,16 +20,34 @@ Nothing except cavaliership and common sense prevents a _short_ video from being The format uses a _regular event_ kind `21` for _normal_ videos and `22` for _short_ videos. +## Addressable Video Events + +For content that may need updates after publication (such as correcting metadata, descriptions, or handling URL migrations), addressable versions are available: + +- Kind `34235` for _addressable normal videos_ +- Kind `34236` for _addressable short videos_ + +These addressable events follow the same format as their regular counterparts but include a `d` tag as a unique identifier and can be updated while maintaining the same addressable reference. This is particularly useful for: + +- Metadata corrections (descriptions, titles, tags) without republishing +- Preservation of imported content IDs from legacy platforms +- URL migration when hosting changes +- Platform migration tracking + The `.content` of these events is a summary or description on the video content. The primary source of video information is the `imeta` tags which is defined in [NIP-92](92.md) -Each `imeta` tag can be used to specify a variant of the video by the `dim` & `m` properties. +Each `imeta` tag can be used to specify a variant of the video by the `dim` & `m` properties, as well as multiple audio tracks (languages). -This NIP defines the following additional `imeta` properties aside form those listen in [NIP-92](92.md) & [NIP-94](94.md): +Clients should check if audio tracks are available and prefer them over in-video audio +to allow smooth resolution switching without interrupting audio. The `ov` (original version) flags the original language track. + +This NIP defines the following additional `imeta` properties aside from those listed in [NIP-92](92.md) & [NIP-94](94.md): * `duration` (recommended) the duration of the video/audio in seconds (floating point number) * `bitrate` (recommended) the average bitrate of the video/audio in bits/sec +* `waveform` (optional, audio only) amplitude values over time, space separated full integers, less than 100 values should be enough to render a nice visual Example: ```json @@ -72,6 +90,18 @@ Example: "service nip96", "duration 29.21" ], + ["imeta", + "url https://myaudio.com/audio/en/12345.mp3", + "x b2e0a7a82ac9f3f3a71f1d9a78c381d5be9d1cf19dce258765c17c8a76287c93", + "m audio/mp3", + "waveform 0 7 35 8 100 100 49 8 4 16 8 10 7 2 20 10 100 100 100 100 100 100 15 100 100 100 25 60 5 4 3 1 0 100 100 15 100 29 88 0 33 11 39 100 100 19 4 100 42 35 5 0 1 5 0 0 11 38 100 94 17 11 44 58 5 100 100 100 55 14 72 100 100 57 6 1 14 2 16 100 100 40 16 100 100 6 32 14 13 41 36 16 14 6 3 0 1 2 1 6 0", + "l en ISO-639-1 ov", + "fallback https://myotherserver.com/audio/en/12345.mp3", + "fallback https://andanotherserver.com/audio/en/12345.mp3", + "service nip96", + "bitrate 320000", + "duration 29.24" + ], ] ``` @@ -81,6 +111,9 @@ The `image` tag contains a preview image (at the same resolution). Multiple `ima Additionally `service nip96` may be included to allow clients to search the authors NIP-96 server list to find the file using the hash. +### Required tags for addressable events: +* `d` - Unique identifier for this video (user-chosen string, required for kinds 34235, 34236) + ### Other tags: * `title` (required) title of the video * `published_at`, for the timestamp in unix seconds (stringified) of the first time the video was published @@ -92,7 +125,10 @@ Additionally `service nip96` may be included to allow clients to search the auth * `p` (optional, repeated) 32-bytes hex pubkey of a participant in the video, optional recommended relay URL * `r` (optional, repeated) references / links to web pages -```jsonc +### Optional tags for imported content: +* `origin` - Track original platform and ID: `["origin", "", "", "", ""]` + +```yaml { "id": "<32-bytes lowercase hex-encoded SHA-256 of the the serialized event data>", "pubkey": "<32-bytes lowercase hex-encoded public key of the event creator>", @@ -135,3 +171,63 @@ Additionally `service nip96` may be included to allow clients to search the auth ] } ``` + +## Addressable Event Example + +```yaml +{ + "id": <32-bytes lowercase hex-encoded SHA-256 of the the serialized event data>, + "pubkey": <32-bytes lowercase hex-encoded public key of the event creator>, + "created_at": , + "kind": 34235 | 34236, + "content": "", + "tags": [ + ["d", ""], + ["title", ""], + ["published_at", "<unix timestamp>"], + ["alt", "<description for accessibility>"], + + // video data + ["imeta", + "url https://example.com/media.mp4", + "m video/mp4", + "dim 480x480", + "blurhash eVF$^OI:${M{%LRjWBoLoLaeR*", + "image https://example.com/thumb.jpg", + "x 3093509d1e0bc604ff60cb9286f4cd7c781553bc8991937befaacfdc28ec5cdc" + ], + // audio data + ["imeta", + "url https://example.com/audio.mp3", + "x b2e0a7a82ac9f3f3a71f1d9a78c381d5be9d1cf19dce258765c17c8a76287c93", + "m audio/mp3", + "l en ISO-639-1 ov" + ], + + ["duration", <duration in seconds>], + ["content-warning", "<reason>"], + + // origin tracking for imported content + ["origin", "<platform>", "<external-id>", "<original-url>", "<optional-metadata>"], + + // participants + ["p", "<32-bytes hex of a pubkey>", "<optional recommended relay URL>"], + + // hashtags + ["t", "<tag>"], + ["t", "<tag>"], + + // reference links + ["r", "<url>"] + ] +} +``` + +## Referencing Addressable Events + +To reference an addressable video: + +``` +["a", "34235:<pubkey>:<d-tag-value>", "<relay-url>"] // for normal videos +["a", "34236:<pubkey>:<d-tag-value>", "<relay-url>"] // for short videos +``` diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index ae55b88c..ce964288 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -248,12 +248,18 @@ Threat model / cautions (write these into the feature + docs): - [ ] Research: confirm NIP-53 kinds + tag shapes, how zap.stream vs shosho.live populate `streaming`/`recording`, and HLS playback support in the player. -### 17. Short-form / external NIP-71 video note support -- [ ] Likely **NIP-71 Video Events** — the format Amethyst & co. use for "short - form" vertical video: kind **22** (short video) and kind **21** (normal - video). (Confirm current kinds; NIP-71 churned from the older 34235/34236.) -- [ ] bitvid currently publishes its OWN kind 30078 + attaches NIP-71 metadata. - This item is about CONSUMING external NIP-71 notes (render them in the feed / - a short-form view) so bitvid can show video posted by other Nostr apps. -- [ ] Research: confirm whether short-form is purely NIP-71 kind 22, how it maps - onto bitvid's video model (url/magnet/thumbnail), and the vertical/feed UX. +### 17. NIP-71 interop (full plan in `docs/nip71-migration-plan.md`) +Research done; decisions locked. **See the plan doc** — it supersedes the rough +notes below. Summary: +- Opt-in, off by default. Dual-event: keep canonical kind 30078; add an addressable + **34235/34236** mirror (`d`=`videoRootId`, edits in lockstep). 34235/36 are NOT + deprecated — current NIP-71 designates them for editable content. +- WebTorrent rides standard NIP-94 `imeta` fields (`magnet`, `i`); private videos + never mirrored; HTTPS `url` required to mirror. +- [ ] Phase 0: retarget `buildNip71VideoEvent` to 34235/36 + field mapping + tests. +- [ ] Phase 1: opt-in toggle, publish/edit/delete lifecycle parity, flip + `FEATURE_PUBLISH_NIP71` on. +- [ ] Phase 1.5: NIP-89 handler reg (kind 31990 → "Open in bitvid" elsewhere); + NIP-51 kind 30005 portable playlists. +- [ ] Phase 2: inbound ingest of external NIP-71 videos (dedup, moderation, trust, + `origin` attribution, distinct discovery surface). From 5a71016388620df3a130619131227394943a905b Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Sun, 21 Jun 2026 22:44:41 -0400 Subject: [PATCH 03/43] docs(nip71): hashtags + existing-note conversion in the plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Document that the explore/kids feed ALREADY consumes NIP-71 hashtags (both raw ["t",…] tags and video.nip71.hashtags) — feed algo needs zero changes; only capture is missing (no per-video hashtags field on 30078 today). - Add a "Converting existing videos" section: opting in is an edit that maps every available 30078 field into its NIP-71 home (incl. hashtags → t) and writes the 34235/36 mirror with d=videoRootId; non-destructive, lockstep edits. - Pull hashtag capture + field conversion explicitly into Phase 1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- docs/nip71-migration-plan.md | 42 +++++++++++++++++++++++++++++++++++- 1 file changed, 41 insertions(+), 1 deletion(-) diff --git a/docs/nip71-migration-plan.md b/docs/nip71-migration-plan.md index 241dacbf..30fb2388 100644 --- a/docs/nip71-migration-plan.md +++ b/docs/nip71-migration-plan.md @@ -68,7 +68,7 @@ Sourced against NIP-71 + NIP-92 (`imeta`) + NIP-94 (file metadata fields). | `infoHash` | `imeta i <infohash>` | **standard NIP-94 field** | | `ws` / `xs` | encoded into the `magnet` URI (`&ws=`/`&xs=`) | also kept on 30078 canonical | | `isNsfw` | `["content-warning", "<reason>"]` | standard | -| hashtags (if/when added) | `["t", …]` (repeated) | bitvid has no hashtags field yet — future | +| hashtags | `["t", <tag>]` (repeated) | written to **both** 30078 and the mirror; see "Hashtags & the feed" | | `isForKids`, `mode`, etc. | bitvid-namespaced extra tags | ignored by other clients | | — | `["client", "bitvid"]` | | @@ -85,6 +85,42 @@ time. (Avoids the future-`created_at` weirdness seen in some published events.) --- +## Hashtags & the feed (no algo changes) + +The explore/kids feed scoring **already** consumes NIP-71 hashtags: +`js/feedEngine/exploreScoring.js` builds each video's tag vector from **both** the +raw `["t", …]` tags on the event **and** `video.nip71.hashtags`, then matches it +against the user's interests/disinterests vector. `nip71.js` already parses `t` +tags into `metadata.hashtags` and merges NIP-71 metadata onto videos by +`videoRootId` (`nip71Cache`). So a native 30078 video, a 34235/36 mirror, and an +ingested foreign NIP-71 video all slot into the **same** hashtag-driven scoring — +**the feed algorithm needs zero changes.** + +The only gap is **capture**: bitvid's current 30078 content schema has no per-video +hashtags field (videos only get the topic tag `["t","video"]`). So the work is to +add a hashtags input and emit `["t", tag]` on **both** events. This is part of +Phase 1. + +## Converting existing videos (the edit/upgrade flow) + +Opting an existing video into NIP-71 is just an **edit**. When the publisher toggles +the mirror on (or edits an already-mirrored video), bitvid maps every available +30078 field into its NIP-71 home per the table above — title→`title`, +description→`.content`/`alt`, url→`imeta url`, thumbnail→`imeta image`, +magnet/infoHash→`imeta magnet`/`i`, nsfw→`content-warning`, hashtags→`t`, etc. — +and writes the 34235/36 mirror with `d` = `videoRootId` so it stays addressable and +lockstep-editable. + +Practical notes for conversion: +- Existing videos have **no user hashtags yet**, so the edit form surfaces the new + hashtags input; tags the publisher adds are written to **both** the 30078 and the + mirror (keeping the feed unified, native + mirrored). +- Conversion is non-destructive: the 30078 stays canonical; the mirror is additive. +- Re-editing later updates both in lockstep (same `d`-tag); toggling off NIP-09s the + mirror only. + +--- + ## Phases ### Phase 0 — Retarget builders (no UX; flag stays off) @@ -98,6 +134,10 @@ time. (Avoids the future-`created_at` weirdness seen in some published events.) ### Phase 1 — Outbound publish/edit/delete (the headline feature) - Edit form toggle: **"Also publish to other Nostr video apps (NIP-71)"** — off by default, gated to public + HTTPS-url videos. +- **Add a hashtags input** to the publish/edit form; write `["t", tag]` on **both** + the 30078 and the 34235/36 mirror (the feed already scores these — no algo change). +- **Field conversion on edit/upgrade**: map all available 30078 fields into their + NIP-71 homes per the mapping table (see "Converting existing videos"). - Publish: 30078 (as today) **+** the 34235/36 mirror to the write-relay set (reuse `getDeletePublishRelays` write set + the publish-outcome toast). - Lifecycle parity: **edit** updates both (same d-tag); **delete** NIP-09s both From 80dce871954a64935a98cc6b2e21fabb352e8a65 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Sun, 21 Jun 2026 22:49:17 -0400 Subject: [PATCH 04/43] =?UTF-8?q?docs(nip71):=20plan=20review=20pass=20?= =?UTF-8?q?=E2=80=94=20discovery/relays,=20dedup,=20capture,=20delete=20nu?= =?UTF-8?q?ance?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Improvements from a second pass over the plan + code: - Add "Discovery & relays" section: the mirror must publish to the user's NIP-65 OUTBOX write relays (publish kind 10002 first if missing) or outbox clients (Amethyst) can't find it — discovery is a relay problem, not just format. bitvid already has publishRelayList/outbox; reuse it. - Architecture: bitvid never double-renders its OWN mirror (merged onto 30078 by videoRootId via nip71Cache) — dedup applies from Phase 1, not just Phase 2. - 3rd hard rule: url must be publicly reachable (gate on verifyPublicAccess). - Capture: wire existing upload `duration` into imeta; add a loadedmetadata probe for `dim` + 34235-vs-34236 selection (pulled into v1; trimmed from backlog). - Mapping: split `.content` (full) vs `alt` (short a11y text); note published_at is read from the original 30078 so it's stable across edits. - Surface the mirror toggle in the My Videos tab (existing management surface). - Delete nuance: NIP-09 (a/k tags) AND empty-replace the addressable event so NIP-09-ignoring clients still see it cleared. - Phase 0: add an interop golden-shape test to guard against silent drift. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- docs/nip71-migration-plan.md | 74 +++++++++++++++++++++++++++++------- 1 file changed, 60 insertions(+), 14 deletions(-) diff --git a/docs/nip71-migration-plan.md b/docs/nip71-migration-plan.md index 30fb2388..5f0d610f 100644 --- a/docs/nip71-migration-plan.md +++ b/docs/nip71-migration-plan.md @@ -46,6 +46,10 @@ kind for what bitvid does, and it's what Nostube/Goblinbox consume today. - `["origin", "bitvid", "<videoRootId>", "https://bitvid.network/?v=<nevent|naddr>"]` so other clients can attribute + deep-link back ("watch on bitvid"). - Optionally `["a", "30078:<pubkey>:<videoRootId>"]` back-pointer (harmless elsewhere). +- **bitvid never double-renders its own mirror.** Inside bitvid the 34235/36 is + merged onto its 30078 by `videoRootId` (`nip71Cache` already keys on it) — it is + metadata enrichment, never a second feed card. This dedup applies from Phase 1 + (our own mirror), not just Phase 2 (foreign videos). --- @@ -57,11 +61,14 @@ Sourced against NIP-71 + NIP-92 (`imeta`) + NIP-94 (file metadata fields). |---|---|---| | — | `["d", videoRootId]` | required for addressable; lockstep with 30078 | | `title` | `["title", …]` | required | -| `description` | `.content` and `["alt", …]` | | -| first publish time | `["published_at", …]` | **stable across edits** (no reordering elsewhere) | +| `description` | `.content` (full description) | | +| short text | `["alt", …]` | accessibility text — title / first line, NOT a dump of the full description | +| first publish time | `["published_at", …]` | read from the ORIGINAL 30078 (root `created_at`/`published_at`); **stable across edits** so foreign clients don't reorder | | `url` (HTTPS) | `imeta url <…>` | primary playable source for foreign clients | | derived MIME | `imeta m <…>` | not stored today → derive from extension, default `video/mp4` | | `thumbnail` | `imeta image <…>` | | +| `duration` (upload `#input-duration`) | `imeta duration <sec>` | already collected at upload — wire it through | +| dimensions (probe at upload) | `imeta dim <w>x<h>` **+ 34235-vs-34236 selection** | not captured today → add a `loadedmetadata` probe | | `fileSha256` | `imeta x <…>` | | | `originalFileSha256` | `imeta ox <…>` | | | `magnet` | `imeta magnet <magnet:?xt=…&ws=…&xs=…>` | **standard NIP-94 field** — torrent-aware clients can use it | @@ -79,6 +86,10 @@ Sourced against NIP-71 + NIP-92 (`imeta`) + NIP-94 (file metadata fields). 2. **A hosted HTTPS `url` is required to mirror.** Foreign clients can't play a magnet-only video; bitvid's magnet still rides along (`magnet`/`i`) for torrent-aware clients, but `url` must be present or the toggle is disabled. +3. **The `url` must be publicly reachable** (CORS + range requests) so other + clients can actually play it. Gate the toggle on bitvid's existing public-access + verification (`verifyPublicAccess`) — don't advertise a video elsewhere that + only bitvid can fetch. **Timestamp hygiene:** `created_at` = actual signing time; `published_at` = display time. (Avoids the future-`created_at` weirdness seen in some published events.) @@ -121,27 +132,61 @@ Practical notes for conversion: --- +## Discovery & relays (the part that decides if anyone sees it) + +A format-perfect mirror is **invisible** if it isn't on the relays other clients +read. This is the #1 thing that makes or breaks interop, and it's a *relay* problem, +not a format problem. + +- bitvid already has NIP-65 (`relayManager.publishRelayList`, kind 10002, + outbox + reads), and `publishVideo` targets `this.writeRelays`. The mirror must publish to + the user's **NIP-65 write relays (outbox model)** — that's where outbox-aware + clients (Amethyst, etc.) look for that author's content. +- **Ensure the user has a published, accurate NIP-65 list** when they first enable + the mirror (publish kind 10002 if missing). Without it, outbox clients can't find + the video regardless of correctness. +- Consider also fanning the mirror out to a small set of **well-known general/video + relays** so non-outbox clients (and search) can discover it. Keep this list short + and configurable; respect the relay-fan-out cap lessons (don't storm). +- Relay reads for any future ingest go through the subscription manager + (lint:pool-access), never `pool.list` directly. + +--- + ## Phases ### Phase 0 — Retarget builders (no UX; flag stays off) - Point existing `buildNip71VideoEvent` at **34235/36 + `d`=`videoRootId`** (it - currently targets 21/22). Implement the field mapping incl. `magnet`/`i`/`ox`. + currently targets 21/22). Implement the field mapping incl. `magnet`/`i`/`ox`/ + `duration` (already collected at upload). - Normal-vs-short selection: default 34235; choose 34236 when known-portrait - (dimensions when available; otherwise a manual "short" toggle). + (from captured dimensions; otherwise a manual "short" toggle). - Cheat-resistant tests: bitvid video → spec-valid 34235 event → round-trips back via `convertEventToVideo`; magnet/infohash present; private refused; url-less refused. +- **Interop golden test**: assert the produced event matches the spec shape + (`d`/`title`/`published_at`/`imeta url`+`image`) so a foreign parser (Nostube-style) + can read it — guards against silent shape drift. ### Phase 1 — Outbound publish/edit/delete (the headline feature) - Edit form toggle: **"Also publish to other Nostr video apps (NIP-71)"** — off by - default, gated to public + HTTPS-url videos. -- **Add a hashtags input** to the publish/edit form; write `["t", tag]` on **both** - the 30078 and the 34235/36 mirror (the feed already scores these — no algo change). + default, gated to **public + HTTPS-url + public-access-verified** videos. +- Surface the same control in the **My Videos tab** (per-video mirror status + + toggle; it already lists owned videos with action buttons). Optional bulk + "make discoverable" there later. +- **Capture dimensions at upload** (`loadedmetadata` probe) for 34235-vs-34236 + selection + `imeta dim`; wire the existing `duration` through. +- **Add a hashtags input**; write `["t", tag]` on **both** the 30078 and the mirror + (the feed already scores these — no algo change). - **Field conversion on edit/upgrade**: map all available 30078 fields into their NIP-71 homes per the mapping table (see "Converting existing videos"). -- Publish: 30078 (as today) **+** the 34235/36 mirror to the write-relay set - (reuse `getDeletePublishRelays` write set + the publish-outcome toast). -- Lifecycle parity: **edit** updates both (same d-tag); **delete** NIP-09s both - addresses; **toggle-off** NIP-09s the mirror (removes it from other clients). +- Publish: 30078 (as today) **+** the 34235/36 mirror to the user's **outbox + (NIP-65 write) relays**; publish a kind-10002 list first if the user has none + (see "Discovery & relays"). Reuse the publish-outcome toast. +- Lifecycle parity: **edit** updates both (same d-tag); **delete** publishes a + NIP-09 (`kind 5`) referencing the mirror's `["a","34235:<pubkey>:<d>"]` (+ `k`), + AND empties the addressable event (empty-replace, the watch-history/encrypted-sync + lesson) so clients that ignore NIP-09 still see it cleared; **toggle-off** does the + same for the mirror only. - Mutation-verified lifecycle tests, then flip `FEATURE_PUBLISH_NIP71` on. ### Phase 1.5 — Discoverability glue (cheap, high interop value) @@ -166,10 +211,11 @@ Practical notes for conversion: --- ## Enrichment backlog (nice-to-have, not v1) -`dim`, `duration`, `bitrate`, `waveform`, multiple `imeta` variants + multi-audio +`bitrate`, `waveform`, multiple resolution `imeta` variants + multi-audio (`ov`/language) tracks, `text-track` (WebVTT captions/subtitles/chapters), -`segment` (chapters) — all need more captured at upload time. v1 ships the minimal -spec-valid imeta (`url`/`m`/`image` + `magnet`/`i`) and grows from there. +`segment` (chapters) — all need more captured/transcoded at upload time. v1 ships +spec-valid imeta with `url`/`m`/`image`/`dim`/`duration`/`x` + `magnet`/`i`, and +grows from there. (`dim`/`duration` are pulled into v1 — see Phase 1.) --- From 0e2f683b62348c7b70c601a2f1914999bbb15f61 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Sun, 21 Jun 2026 23:05:12 -0400 Subject: [PATCH 05/43] =?UTF-8?q?feat(nip71):=20Phase=200=20=E2=80=94=20ad?= =?UTF-8?q?dressable=2034235/36=20mirror=20builder=20+=20magnet/i/ox=20ime?= =?UTF-8?q?ta?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First build step of the NIP-71 interop plan (no UX; FEATURE_PUBLISH_NIP71 stays off). Pure mapping from a bitvid video (kind-30078 shape) to an opt-in addressable NIP-71 mirror event. - js/nostr/nip71.js (foundational): - normalizeNip71Kind now recognizes the addressable kinds 34235/34236 (so the parser preserves them), alongside 21/22. - imeta builder + parser gained standard NIP-94 fields magnet, i (infohash), ox — bitvid's WebTorrent now rides the standard, round-trippable. - js/nostr/nip71Mirror.js (new): buildNip71MirrorEvent(video, options) → { ok:true, event } | { ok:false, reason }. Builds kind 34235 (normal) / 34236 (short, by dims or explicit flag), d=videoRootId, title/published_at(stable)/alt, imeta (url/m/image/x/ox/magnet/i/dim/duration), content-warning (NSFW), t hashtags, origin + a back-pointer to the canonical 30078, client=bitvid. Hard rules enforced: private videos refused; non-HTTPS-url refused. tests/nip71-mirror.test.mjs: spec-shape mapping, round-trip via extractNip71MetadataFromTags (magnet/i/ox survive), private + url-less refusals, portrait→34236 + content-warning, explicit short override, MIME derivation. Mutation-verified (private refusal, magnet imeta, d-tag all load-bearing). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/nostr/nip71.js | 34 ++++++++ js/nostr/nip71Mirror.js | 169 ++++++++++++++++++++++++++++++++++++ tests/nip71-mirror.test.mjs | 146 +++++++++++++++++++++++++++++++ 3 files changed, 349 insertions(+) create mode 100644 js/nostr/nip71Mirror.js create mode 100644 tests/nip71-mirror.test.mjs diff --git a/js/nostr/nip71.js b/js/nostr/nip71.js index 0fb18a06..cea83e9c 100644 --- a/js/nostr/nip71.js +++ b/js/nostr/nip71.js @@ -176,6 +176,12 @@ function normalizeNip71Kind(value) { if (numeric === 22) { return 22; } + if (numeric === 34235) { + return 34235; + } + if (numeric === 34236) { + return 34236; + } return 21; } @@ -199,6 +205,9 @@ function buildImetaTags(variants) { const dim = stringFromInput(variant.dim); const url = stringFromInput(variant.url); const x = stringFromInput(variant.x); + const ox = stringFromInput(variant.ox); + const magnet = stringFromInput(variant.magnet); + const infohash = stringFromInput(variant.i); const rawMime = stringFromInput(variant.m); const mime = rawMime ? rawMime.toLowerCase() : ""; const duration = formatNonNegativeNumber(variant.duration, { @@ -221,6 +230,9 @@ function buildImetaTags(variants) { !dim && !url && !x && + !ox && + !magnet && + !infohash && !mime && !duration && !bitrate && @@ -241,9 +253,19 @@ function buildImetaTags(variants) { if (x) { entries.push(`x ${x}`); } + if (ox) { + entries.push(`ox ${ox}`); + } if (mime) { entries.push(`m ${mime}`); } + // WebTorrent rides standard NIP-94 imeta fields so torrent-aware clients can use it. + if (magnet) { + entries.push(`magnet ${magnet}`); + } + if (infohash) { + entries.push(`i ${infohash}`); + } if (duration) { entries.push(`duration ${duration}`); } @@ -456,6 +478,15 @@ function parseImetaTag(tag) { case "x": variant.x = parsed.value; break; + case "ox": + variant.ox = parsed.value; + break; + case "magnet": + variant.magnet = parsed.value; + break; + case "i": + variant.i = parsed.value; + break; case "m": variant.m = parsed.value; break; @@ -489,6 +520,9 @@ function parseImetaTag(tag) { Boolean(variant.dim) || Boolean(variant.url) || Boolean(variant.x) || + Boolean(variant.ox) || + Boolean(variant.magnet) || + Boolean(variant.i) || Boolean(variant.m) || (variant.duration !== undefined && variant.duration !== null && variant.duration !== "") || (variant.bitrate !== undefined && variant.bitrate !== null && variant.bitrate !== "") || diff --git a/js/nostr/nip71Mirror.js b/js/nostr/nip71Mirror.js new file mode 100644 index 00000000..ff1c4a5a --- /dev/null +++ b/js/nostr/nip71Mirror.js @@ -0,0 +1,169 @@ +// Phase 0 of the NIP-71 interop plan (docs/nip71-migration-plan.md). +// +// Maps a bitvid video (the kind-30078 content shape) to an opt-in, addressable +// NIP-71 *mirror* event (kind 34235 normal / 34236 short). The 30078 stays the +// canonical record; this mirror is additive and lets the video show up on other +// Nostr video clients (Nostube, Amethyst, …). +// +// Pure / no UX wiring (gated by FEATURE_PUBLISH_NIP71 at the call sites later). +// Hard rules baked in: private videos are NEVER mirrored, and a publicly-playable +// HTTPS url is required (foreign clients can't play magnet-only). bitvid's +// WebTorrent rides standard NIP-94 imeta fields (magnet, i). + +import { buildNip71MetadataTags, buildVideoPointerValue } from "./nip71.js"; +import { extractBtihFromMagnet } from "../magnetShared.js"; + +export const NIP71_NORMAL_VIDEO_KIND = 34235; +export const NIP71_SHORT_VIDEO_KIND = 34236; + +const VIDEO_MIME_BY_EXT = { + mp4: "video/mp4", + m4v: "video/mp4", + webm: "video/webm", + ogv: "video/ogg", + ogg: "video/ogg", + mov: "video/quicktime", + mkv: "video/x-matroska", + m3u8: "application/x-mpegURL", +}; + +function str(value) { + return typeof value === "string" ? value.trim() : ""; +} + +function isHttpsUrl(value) { + return /^https:\/\//i.test(str(value)); +} + +// Best-effort MIME from the url extension; bitvid doesn't persist a MIME today. +export function deriveVideoMime(url) { + const value = str(url).toLowerCase(); + if (!value) { + return "video/mp4"; + } + const path = value.split(/[?#]/)[0]; + const ext = path.includes(".") ? path.slice(path.lastIndexOf(".") + 1) : ""; + return VIDEO_MIME_BY_EXT[ext] || "video/mp4"; +} + +function normalizePublishedAt(value) { + const numeric = Number(value); + if (!Number.isFinite(numeric) || numeric <= 0) { + return Math.floor(Date.now() / 1000); + } + // Tolerate millisecond timestamps. + return numeric > 10_000_000_000 ? Math.floor(numeric / 1000) : Math.floor(numeric); +} + +/** + * Build an addressable NIP-71 mirror event from a bitvid video object. + * + * @param {object} video - bitvid video (30078 content shape) + `pubkey`. + * @param {object} [options] + * @param {boolean} [options.short] - Force 34236 (short) / 34235 (normal). When + * omitted, inferred from dimensions (portrait => short). + * @param {number} [options.createdAt] - Signing time (unix seconds). Default now. + * @param {number|string} [options.publishedAt] - First-publish time for the + * `published_at` tag; kept stable across edits. Defaults to the video's + * published_at/created_at, else now. + * @param {string} [options.watchUrl] - Canonical bitvid watch URL for `origin`. + * @returns {{ok:true, event:object} | {ok:false, reason:string}} + */ +export function buildNip71MirrorEvent(video, options = {}) { + if (!video || typeof video !== "object") { + return { ok: false, reason: "invalid" }; + } + + const videoRootId = str(video.videoRootId); + const title = str(video.title); + const pubkey = str(video.pubkey).toLowerCase(); + if (!videoRootId || !title || !pubkey) { + return { ok: false, reason: "invalid" }; + } + + // Hard rule 1: private videos are never mirrored (no NIP-71 encryption). + if (video.isPrivate === true) { + return { ok: false, reason: "private" }; + } + + // Hard rule 2: foreign clients need a hosted, publicly-playable HTTPS url. + const url = str(video.url); + if (!isHttpsUrl(url)) { + return { ok: false, reason: "no-url" }; + } + + const magnet = str(video.magnet); + const infohash = + str(video.infoHash) || (magnet ? str(extractBtihFromMagnet(magnet)) : ""); + + const width = Number(video.width); + const height = Number(video.height); + const hasDims = + Number.isFinite(width) && Number.isFinite(height) && width > 0 && height > 0; + const duration = Number(video.duration); + + const variant = { + url, + m: deriveVideoMime(url), + image: str(video.thumbnail) ? [str(video.thumbnail)] : [], + x: str(video.fileSha256) || undefined, + ox: str(video.originalFileSha256) || undefined, + magnet: magnet || undefined, + i: infohash || undefined, + dim: hasDims ? `${width}x${height}` : undefined, + duration: Number.isFinite(duration) && duration > 0 ? duration : undefined, + }; + + const hashtags = Array.isArray(video.hashtags) + ? video.hashtags.map(str).filter(Boolean) + : []; + + const metadata = { + title, + // Short accessibility text — NOT a dump of the full description. + alt: title, + publishedAt: normalizePublishedAt( + options.publishedAt ?? video.published_at ?? video.created_at, + ), + imeta: [variant], + hashtags, + contentWarning: + video.isNsfw === true ? str(video.nsfwReason) || "nsfw" : undefined, + }; + + const tags = buildNip71MetadataTags(metadata); + + // Addressable identity + bitvid linkage (ignored by foreign clients). + tags.push(["d", videoRootId]); + const aValue = buildVideoPointerValue(pubkey, videoRootId); + if (aValue) { + tags.push(["a", aValue]); // back-pointer to the canonical kind 30078 + } + const origin = ["origin", "bitvid", videoRootId]; + const watchUrl = str(options.watchUrl); + if (watchUrl) { + origin.push(watchUrl); + } + tags.push(origin); + tags.push(["client", "bitvid"]); + + const short = + options.short === true || + (options.short !== false && hasDims && height > width); + + const createdAt = Number.isFinite(options.createdAt) + ? Math.floor(options.createdAt) + : Math.floor(Date.now() / 1000); + + const event = { + kind: short ? NIP71_SHORT_VIDEO_KIND : NIP71_NORMAL_VIDEO_KIND, + pubkey, + created_at: createdAt, + tags, + content: str(video.description), + }; + + return { ok: true, event }; +} + +export default buildNip71MirrorEvent; diff --git a/tests/nip71-mirror.test.mjs b/tests/nip71-mirror.test.mjs new file mode 100644 index 00000000..fb93a57b --- /dev/null +++ b/tests/nip71-mirror.test.mjs @@ -0,0 +1,146 @@ +// Phase 0 of the NIP-71 interop plan: bitvid video (kind 30078 shape) -> addressable +// NIP-71 mirror event (kind 34235/34236). Scenarios assert the observable event +// shape AND that bitvid's own NIP-71 parser reads it back (round-trip), plus the +// two hard rules (private never mirrored, HTTPS url required) and short selection. + +import "./test-helpers/setup-localstorage.mjs"; +import assert from "node:assert/strict"; +import test from "node:test"; +import { + buildNip71MirrorEvent, + deriveVideoMime, + NIP71_NORMAL_VIDEO_KIND, + NIP71_SHORT_VIDEO_KIND, +} from "../js/nostr/nip71Mirror.js"; +import { extractNip71MetadataFromTags } from "../js/nostr/nip71.js"; + +const PUBKEY = "a".repeat(64); +const INFOHASH = "b".repeat(40); + +function baseVideo(overrides = {}) { + return { + videoRootId: "root-123", + pubkey: PUBKEY, + title: "Crystalline Bliss", + description: "A lo-fi voxel puzzle journey.", + url: "https://cdn.example.com/u/abc/video.mp4", + thumbnail: "https://cdn.example.com/u/abc/thumb.jpg", + magnet: `magnet:?xt=urn:btih:${INFOHASH}&ws=https://cdn.example.com/u/abc/video.mp4`, + infoHash: INFOHASH, + fileSha256: "c".repeat(64), + originalFileSha256: "d".repeat(64), + isPrivate: false, + isNsfw: false, + hashtags: ["gamestr", "voxel"], + duration: 123, + ...overrides, + }; +} + +// Find the imeta tag and parse "key value" entries into an object (last wins, +// images/magnet collected). +function imetaFields(event) { + const tag = event.tags.find((t) => t[0] === "imeta"); + assert.ok(tag, "event must have an imeta tag"); + const fields = {}; + for (let i = 1; i < tag.length; i += 1) { + const sp = tag[i].indexOf(" "); + const k = tag[i].slice(0, sp); + const v = tag[i].slice(sp + 1); + fields[k] = v; + } + return fields; +} + +function tagValue(event, name) { + const tag = event.tags.find((t) => t[0] === name); + return tag ? tag[1] : undefined; +} + +test("maps a public bitvid video to a spec-shaped addressable 34235 event", async () => { + const res = buildNip71MirrorEvent(baseVideo(), { createdAt: 1000, publishedAt: 900 }); + assert.equal(res.ok, true); + const ev = res.event; + + assert.equal(ev.kind, NIP71_NORMAL_VIDEO_KIND, "landscape/default => 34235"); + assert.equal(ev.pubkey, PUBKEY); + assert.equal(ev.content, "A lo-fi voxel puzzle journey."); + + // Required + addressable identity + assert.equal(tagValue(ev, "d"), "root-123", "addressable d-tag = videoRootId"); + assert.equal(tagValue(ev, "title"), "Crystalline Bliss"); + assert.equal(tagValue(ev, "published_at"), "900", "stable first-publish time"); + + // Back-pointer + attribution + hashtags + assert.equal(tagValue(ev, "a"), `30078:${PUBKEY}:root-123`, "links to canonical 30078"); + const origin = ev.tags.find((t) => t[0] === "origin"); + assert.deepEqual(origin.slice(0, 3), ["origin", "bitvid", "root-123"]); + const tTags = ev.tags.filter((t) => t[0] === "t").map((t) => t[1]).sort(); + assert.deepEqual(tTags, ["gamestr", "voxel"]); + + // imeta: HTTPS url + standard NIP-94 magnet/infohash/hashes + const f = imetaFields(ev); + assert.equal(f.url, "https://cdn.example.com/u/abc/video.mp4"); + assert.equal(f.m, "video/mp4"); + assert.equal(f.image, "https://cdn.example.com/u/abc/thumb.jpg"); + assert.equal(f.x, "c".repeat(64)); + assert.equal(f.ox, "d".repeat(64)); + assert.equal(f.i, INFOHASH, "infohash rides standard NIP-94 'i'"); + assert.ok(f.magnet.startsWith("magnet:?xt=urn:btih:"), "magnet rides standard NIP-94 'magnet'"); + assert.equal(f.duration, "123"); +}); + +test("round-trips: the produced event parses back via bitvid's NIP-71 parser", async () => { + const res = buildNip71MirrorEvent(baseVideo()); + const parsed = extractNip71MetadataFromTags(res.event); + assert.ok(parsed, "parser must read the mirror event"); + const meta = parsed.metadata; + assert.equal(meta.kind, NIP71_NORMAL_VIDEO_KIND, "addressable kind preserved"); + assert.equal(parsed.source.kind, NIP71_NORMAL_VIDEO_KIND, "source kind preserved"); + assert.equal(meta.title, "Crystalline Bliss"); + assert.deepEqual(meta.hashtags.sort(), ["gamestr", "voxel"]); + const v = meta.imeta[0]; + assert.equal(v.url, "https://cdn.example.com/u/abc/video.mp4"); + assert.equal(v.magnet, baseVideo().magnet, "magnet survives the round-trip"); + assert.equal(v.i, INFOHASH, "infohash survives the round-trip"); + assert.equal(v.ox, "d".repeat(64), "original-file hash survives the round-trip"); +}); + +test("private videos are NEVER mirrored", async () => { + const res = buildNip71MirrorEvent(baseVideo({ isPrivate: true })); + assert.equal(res.ok, false); + assert.equal(res.reason, "private"); +}); + +test("a video without an HTTPS url cannot be mirrored", async () => { + // magnet-only: bitvid plays it, foreign clients can't. + const res = buildNip71MirrorEvent(baseVideo({ url: "" })); + assert.equal(res.ok, false); + assert.equal(res.reason, "no-url"); + + const httpRes = buildNip71MirrorEvent(baseVideo({ url: "http://insecure.example/v.mp4" })); + assert.equal(httpRes.ok, false, "non-HTTPS url is refused"); +}); + +test("portrait dimensions select the short kind 34236; content-warning for NSFW", async () => { + const res = buildNip71MirrorEvent( + baseVideo({ width: 1080, height: 1920, isNsfw: true, nsfwReason: "mature" }), + ); + assert.equal(res.event.kind, NIP71_SHORT_VIDEO_KIND, "portrait => 34236 short"); + assert.equal(tagValue(res.event, "content-warning"), "mature"); + assert.equal(imetaFields(res.event).dim, "1080x1920"); +}); + +test("explicit short flag overrides dimension inference", async () => { + const landscapeButShort = buildNip71MirrorEvent( + baseVideo({ width: 1920, height: 1080 }), + { short: true }, + ); + assert.equal(landscapeButShort.event.kind, NIP71_SHORT_VIDEO_KIND); +}); + +test("deriveVideoMime maps common extensions and defaults to mp4", async () => { + assert.equal(deriveVideoMime("https://x/y.webm"), "video/webm"); + assert.equal(deriveVideoMime("https://x/y.m3u8?token=1"), "application/x-mpegURL"); + assert.equal(deriveVideoMime("https://x/y"), "video/mp4"); +}); From a004ee0b111bdb336222ff14eb612ae51f94730c Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Sun, 21 Jun 2026 23:05:31 -0400 Subject: [PATCH 06/43] docs(todo): mark NIP-71 Phase 0 done Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- todo/TODO_2026-06-20_pre-launch.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index ce964288..8d15e37b 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -256,7 +256,9 @@ notes below. Summary: deprecated — current NIP-71 designates them for editable content. - WebTorrent rides standard NIP-94 `imeta` fields (`magnet`, `i`); private videos never mirrored; HTTPS `url` required to mirror. -- [ ] Phase 0: retarget `buildNip71VideoEvent` to 34235/36 + field mapping + tests. +- [x] Phase 0 (commit 0e2f683b): `buildNip71MirrorEvent` (js/nostr/nip71Mirror.js) + maps a bitvid video → addressable 34235/36 event; magnet/i/ox added to the + imeta builder+parser; mutation-verified tests. Flag still off, no UX. - [ ] Phase 1: opt-in toggle, publish/edit/delete lifecycle parity, flip `FEATURE_PUBLISH_NIP71` on. - [ ] Phase 1.5: NIP-89 handler reg (kind 31990 → "Open in bitvid" elsewhere); From 5c93b7496e228e4949688c858d1254d95b880e0b Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Sun, 21 Jun 2026 23:10:19 -0400 Subject: [PATCH 07/43] docs(nip71): audited moderation/WoT/mute/NSFW interop requirements MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Audit (verified in code) of how moderation, web-of-trust, mute/block lists, and the site ALLOW_NSFW_CONTENT config interact with NIP-71 interop: - Controls key on author pubkey + event id, so they apply to NIP-71 videos automatically IF ingested videos share the feed pipeline (blacklist-filter, moderation/WoT, suppression, kids, nsfw). - Phase 1 (outbound): gate the mirror on ALLOW_NSFW_CONTENT (don't mirror NSFW from an instance that forbids it); builder stays pure + always sets content-warning. - Phase 2 (inbound) gap: mergeNip71MetadataIntoVideo does NOT map content-warning → isNsfw, so foreign NSFW would bypass the site filter — must add that mapping; plus route ingest through the full pipeline and trust-gate unknown authors. - Kids feed already safe (allowlist on isForKids). No code changes yet (both items land in their phases); captured as checklist items. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- docs/nip71-migration-plan.md | 39 +++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/docs/nip71-migration-plan.md b/docs/nip71-migration-plan.md index 5f0d610f..05509fc7 100644 --- a/docs/nip71-migration-plan.md +++ b/docs/nip71-migration-plan.md @@ -169,7 +169,9 @@ not a format problem. ### Phase 1 — Outbound publish/edit/delete (the headline feature) - Edit form toggle: **"Also publish to other Nostr video apps (NIP-71)"** — off by - default, gated to **public + HTTPS-url + public-access-verified** videos. + default, gated to **public + HTTPS-url + public-access-verified** videos, and + **not offered for `isNsfw` videos when `ALLOW_NSFW_CONTENT=false`** (see + "Moderation, web-of-trust, mute/block & NSFW"). - Surface the same control in the **My Videos tab** (per-video mirror status + toggle; it already lists owned videos with action buttons). Optional bulk "make discoverable" there later. @@ -210,6 +212,41 @@ not a format problem. --- +## Moderation, web-of-trust, mute/block & NSFW (audited 2026-06-22) + +The existing controls key on **author pubkey** and **event id**, so they apply to +NIP-71 videos automatically — *as long as ingested videos flow through the same feed +pipeline*. One real gap (inbound NSFW) + one config-respect (outbound). + +**Outbound (Phase 1):** +- [ ] Respect the site `ALLOW_NSFW_CONTENT` config (`config/instance-config.js`, + default `false`): when false, **do not offer/allow the mirror for `isNsfw` + videos** — an instance that won't surface NSFW shouldn't publish it outward. + The pure builder stays config-agnostic and always sets `content-warning` when + NSFW; gate at the call site. +- Mute/block/WoT/admin-blacklist don't newly apply outbound — it's the user's own + content (a blacklisted user can't publish anyway). + +**Inbound (Phase 2) — the real moderation work:** +- [ ] **`content-warning` → `isNsfw`**: `mergeNip71MetadataIntoVideo` does NOT + currently map a NIP-71 `content-warning` to `video.isNsfw`. The site NSFW + filter keys on `isNsfw`, so foreign NSFW would bypass it. Map it in the ingest + path so `ALLOW_NSFW_CONTENT=false` filters foreign NSFW too. +- [ ] **Same pipeline, no bypass**: route every ingested video through the existing + stages — `blacklist-filter` (`shouldIncludeVideo`: admin/event-id blacklist + + blocked authors), `moderation` (admin whitelist/blacklist + trusted-mute / + trusted-report thresholds = web-of-trust), watch-history suppression, + kids-audience, NSFW. These key on pubkey/id, so mute lists (NIP-51 kind 10000), + blocks, admin lists, and WoT thresholds catch foreign authors automatically. +- [ ] **Trust gate for strangers**: don't surface unknown/untrusted foreign authors + by default — apply the same trust-seed / web-of-trust gating native videos get + (consider requiring author ∈ WoT, or a stricter default threshold for ingest). +- **Kids feed is already safe**: `kidsAudienceFilterStage` is allowlist-based + (`isForKids !== true` ⇒ excluded) and also drops `isNsfw` + disallowed + content-warnings, so foreign videos (no `isForKids`) never enter the kids feed. + +--- + ## Enrichment backlog (nice-to-have, not v1) `bitrate`, `waveform`, multiple resolution `imeta` variants + multi-audio (`ov`/language) tracks, `text-track` (WebVTT captions/subtitles/chapters), From e20191d6bc2854e33537e15a777a464511ddbf40 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 09:03:41 -0400 Subject: [PATCH 08/43] =?UTF-8?q?feat(nip71):=20Phase=201a=20=E2=80=94=20m?= =?UTF-8?q?irror=20publish=20service=20with=20site=20NSFW=20gate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Publish service for the opt-in NIP-71 mirror (kind 34235/36) to the user's write/outbox relays, built on the Phase 0 builder. Flag still off; not yet wired into the publish path (next increment). - js/services/nip71MirrorService.js: createNip71MirrorService (DI) → isAvailable, canMirror(video), publish(video, options). Signs with the active signer and publishes to the write set (getDeletePublishRelays). Resolves the moderation audit item: the site ALLOW_NSFW_CONTENT gate — an instance that won't surface NSFW must not publish it outward (enforced in both publish() and canMirror(), defense-in-depth). Private/url rules come from the builder. tests/nip71-mirror-service.test.mjs: publishes a public video; NSFW blocked when ALLOW_NSFW_CONTENT=false (and allowed when true); private/url-less refused; unavailable without signer. Mutation-verified (NSFW gate load-bearing). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/services/nip71MirrorService.js | 122 ++++++++++++++++++++++++++++ tests/nip71-mirror-service.test.mjs | 107 ++++++++++++++++++++++++ 2 files changed, 229 insertions(+) create mode 100644 js/services/nip71MirrorService.js create mode 100644 tests/nip71-mirror-service.test.mjs diff --git a/js/services/nip71MirrorService.js b/js/services/nip71MirrorService.js new file mode 100644 index 00000000..dcb92e5d --- /dev/null +++ b/js/services/nip71MirrorService.js @@ -0,0 +1,122 @@ +// Phase 1 of the NIP-71 interop plan (docs/nip71-migration-plan.md): publish the +// opt-in addressable NIP-71 mirror (kind 34235/34236) for a bitvid video to the +// user's write/outbox relays. +// +// Builds on the Phase 0 pure builder (buildNip71MirrorEvent) which already enforces +// the "private never mirrored" and "HTTPS url required" rules. This service adds: +// - the site-wide NSFW gate: when ALLOW_NSFW_CONTENT is false, an instance that +// won't surface NSFW must not publish it outward either (moderation audit item). +// - signing with the active signer + publishing to the write (outbox) relay set. +// +// Dependency-injected so it unit-tests without a live signer/relay pool. + +import { buildNip71MirrorEvent } from "../nostr/nip71Mirror.js"; +import { nostrClient } from "../nostrClientFacade.js"; +import { getActiveSigner } from "../nostr/index.js"; +import { + publishEventToRelays as defaultPublishEventToRelays, + summarizePublishResults as defaultSummarize, +} from "../nostrPublish.js"; +import { ALLOW_NSFW_CONTENT } from "../config.js"; +import { userLogger } from "../utils/logger.js"; + +function str(value) { + return typeof value === "string" ? value.trim() : ""; +} + +export function createNip71MirrorService({ + buildMirrorEvent = buildNip71MirrorEvent, + getActivePubkey = () => str(nostrClient?.pubkey), + getSigner = () => getActiveSigner(), + // Outbox/write set (mirrors getDeletePublishRelays from the delete work). + getWriteRelays = () => + typeof nostrClient?.getDeletePublishRelays === "function" + ? nostrClient.getDeletePublishRelays() + : [], + getPool = () => nostrClient?.pool || null, + publishEventToRelays = defaultPublishEventToRelays, + summarizePublishResults = defaultSummarize, + signEvent = async (template) => { + const signer = getActiveSigner(); + if (!signer || typeof signer.signEvent !== "function") { + const err = new Error("Active signer missing signEvent support."); + err.code = "sign-event-missing"; + throw err; + } + return signer.signEvent(template); + }, + // Instance policy: only mirror NSFW outward when the site permits NSFW. + allowNsfw = () => ALLOW_NSFW_CONTENT === true, + now = () => Date.now(), +} = {}) { + function isAvailable() { + const pubkey = getActivePubkey(); + const signer = getSigner(); + return Boolean(pubkey) && Boolean(signer) && typeof signer.signEvent === "function"; + } + + // Whether a video is even eligible for the mirror (drives UI gating too). + function canMirror(video) { + if (!video || typeof video !== "object") { + return { ok: false, reason: "invalid" }; + } + if (video.isPrivate === true) { + return { ok: false, reason: "private" }; + } + if (video.isNsfw === true && !allowNsfw()) { + return { ok: false, reason: "nsfw-blocked" }; + } + if (!/^https:\/\//i.test(str(video.url))) { + return { ok: false, reason: "no-url" }; + } + return { ok: true }; + } + + async function publish(video, options = {}) { + if (!isAvailable()) { + return { ok: false, error: "unavailable" }; + } + // Site NSFW policy gate (defense in depth alongside the UI not offering it). + if (video?.isNsfw === true && !allowNsfw()) { + return { ok: false, reason: "nsfw-blocked" }; + } + + const pubkey = str(video?.pubkey) || getActivePubkey(); + const built = buildMirrorEvent( + { ...video, pubkey }, + { ...options, createdAt: Math.floor(now() / 1000) }, + ); + if (!built.ok) { + return { ok: false, reason: built.reason }; + } + + let signed; + try { + signed = await signEvent(built.event); + } catch (error) { + userLogger.warn("[nip71Mirror] Failed to sign mirror event:", error); + return { ok: false, error: "sign-failed", cause: error }; + } + + const relays = getWriteRelays() || []; + const pool = getPool(); + if (!pool || !relays.length) { + return { ok: false, error: "no-relays" }; + } + + const results = await publishEventToRelays(pool, relays, signed); + const { accepted, failed } = summarizePublishResults(results); + return { + ok: accepted.length > 0, + accepted: accepted.length, + total: accepted.length + failed.length, + event: signed, + }; + } + + return { isAvailable, canMirror, publish }; +} + +export const nip71MirrorService = createNip71MirrorService(); + +export default nip71MirrorService; diff --git a/tests/nip71-mirror-service.test.mjs b/tests/nip71-mirror-service.test.mjs new file mode 100644 index 00000000..010cc242 --- /dev/null +++ b/tests/nip71-mirror-service.test.mjs @@ -0,0 +1,107 @@ +// Phase 1: the NIP-71 mirror publish service. Scenarios assert the observable +// outcomes at the publish boundary, especially the site-wide NSFW moderation gate +// (ALLOW_NSFW_CONTENT) — an instance that won't surface NSFW must not publish it +// outward — plus the private / url / availability gates and a real publish. + +import "./test-helpers/setup-localstorage.mjs"; +import assert from "node:assert/strict"; +import test from "node:test"; +import { createNip71MirrorService } from "../js/services/nip71MirrorService.js"; +import { + publishEventToRelays, + summarizePublishResults, +} from "../js/nostrPublish.js"; + +const PUBKEY = "a".repeat(64); + +function baseVideo(overrides = {}) { + return { + videoRootId: "root-1", + pubkey: PUBKEY, + title: "Hello", + description: "desc", + url: "https://cdn.example.com/v.mp4", + thumbnail: "https://cdn.example.com/t.jpg", + isPrivate: false, + isNsfw: false, + ...overrides, + }; +} + +// Faithful relay twin: stores published events; thenable publish => success path. +function makeRelay() { + const events = []; + return { + events, + publish(_urls, event) { + events.push(event); + return Promise.resolve(); + }, + }; +} + +function makeService({ allowNsfw = false, available = true } = {}) { + const relay = makeRelay(); + const signer = available + ? { signEvent: async (tpl) => ({ ...tpl, id: "id", sig: "sig" }) } + : null; + const service = createNip71MirrorService({ + getActivePubkey: () => (available ? PUBKEY : ""), + getSigner: () => signer, + getWriteRelays: () => ["wss://write.relay"], + getPool: () => relay, + publishEventToRelays, + summarizePublishResults, + signEvent: signer ? signer.signEvent : async () => { throw new Error("no signer"); }, + allowNsfw: () => allowNsfw === true, + }); + return { service, relay }; +} + +test("publishes a public video as an addressable 34235 mirror to the write relays", async () => { + const { service, relay } = makeService(); + const result = await service.publish(baseVideo()); + assert.equal(result.ok, true); + assert.equal(result.accepted, 1); + assert.equal(relay.events.length, 1); + assert.equal(relay.events[0].kind, 34235); + assert.equal(relay.events[0].tags.find((t) => t[0] === "d")[1], "root-1"); +}); + +test("NSFW video is NOT mirrored when the instance forbids NSFW (ALLOW_NSFW_CONTENT=false)", async () => { + const { service, relay } = makeService({ allowNsfw: false }); + const result = await service.publish(baseVideo({ isNsfw: true })); + assert.equal(result.ok, false); + assert.equal(result.reason, "nsfw-blocked"); + assert.equal(relay.events.length, 0, "must not publish NSFW outward"); + assert.deepEqual(service.canMirror(baseVideo({ isNsfw: true })), { + ok: false, + reason: "nsfw-blocked", + }); +}); + +test("NSFW video IS mirrored when the instance allows NSFW", async () => { + const { service, relay } = makeService({ allowNsfw: true }); + const result = await service.publish(baseVideo({ isNsfw: true, nsfwReason: "mature" })); + assert.equal(result.ok, true); + assert.equal(relay.events.length, 1); + assert.equal( + relay.events[0].tags.find((t) => t[0] === "content-warning")[1], + "mature", + ); +}); + +test("private and url-less videos are refused (builder rules)", async () => { + const { service, relay } = makeService(); + assert.equal((await service.publish(baseVideo({ isPrivate: true }))).reason, "private"); + assert.equal((await service.publish(baseVideo({ url: "" }))).reason, "no-url"); + assert.equal(relay.events.length, 0); +}); + +test("publish is unavailable without a signer / pubkey", async () => { + const { service } = makeService({ available: false }); + assert.equal(service.isAvailable(), false); + const result = await service.publish(baseVideo()); + assert.equal(result.ok, false); + assert.equal(result.error, "unavailable"); +}); From 58f1909570f03013817d1d8d03f34763879c6b18 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 09:18:31 -0400 Subject: [PATCH 09/43] =?UTF-8?q?feat(nip71):=20Phase=201b=20=E2=80=94=20m?= =?UTF-8?q?irror=20teardown=20(NIP-09=20delete=20+=20empty-replace=20tombs?= =?UTF-8?q?tone)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds remove(video, options) to the mirror service for delete / toggle-off: - NIP-09 kind-5 delete referencing BOTH addressable kinds (34235:pub:root + 34236:pub:root, with k tags) so it's robust to a since-changed orientation. - empty-replace tombstone (same d-tag, title, NO imeta) so clients that cache addressable events but ignore NIP-09 still see the entry cleared; uses the short kind for portrait videos. Published to the write/outbox relay set; unavailable without a signer. tests/nip71-mirror-service.test.mjs: remove publishes delete (both kinds) + tombstone; portrait => 34236 tombstone; unavailable without signer. Mutation-verified (tombstone emission + both-kind a-tags load-bearing). Note: full unit suite green apart from the documented order-dependent flakes (uploadModal-reset; a transient login-modal clipboard global-state flake) — both pass in isolation and are unrelated to this change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/services/nip71MirrorService.js | 74 ++++++++++++++++++++++++++++- tests/nip71-mirror-service.test.mjs | 40 ++++++++++++++++ 2 files changed, 113 insertions(+), 1 deletion(-) diff --git a/js/services/nip71MirrorService.js b/js/services/nip71MirrorService.js index dcb92e5d..25bbec5c 100644 --- a/js/services/nip71MirrorService.js +++ b/js/services/nip71MirrorService.js @@ -114,7 +114,79 @@ export function createNip71MirrorService({ }; } - return { isAvailable, canMirror, publish }; + // Tear down the mirror (on delete, or when the user toggles sync off). Belt-and- + // suspenders per the plan: a NIP-09 delete referencing BOTH addressable kinds + // (robust to a since-changed orientation), AND an empty-replace tombstone so + // clients that cache addressable events but ignore NIP-09 still see it cleared. + async function remove(video, options = {}) { + if (!isAvailable()) { + return { ok: false, error: "unavailable" }; + } + const pubkey = str(video?.pubkey) || getActivePubkey(); + const root = str(video?.videoRootId); + if (!pubkey || !root) { + return { ok: false, error: "invalid" }; + } + + const createdAt = Math.floor(now() / 1000); + const reason = str(options.reason) || "Removed NIP-71 mirror"; + + const deleteEvent = { + kind: 5, + pubkey, + created_at: createdAt, + content: reason, + tags: [ + ["a", `34235:${pubkey}:${root}`], + ["a", `34236:${pubkey}:${root}`], + ["k", "34235"], + ["k", "34236"], + ], + }; + + const width = Number(video?.width); + const height = Number(video?.height); + const portrait = + Number.isFinite(width) && Number.isFinite(height) && width > 0 && height > 0 && height > width; + const tombstoneKind = + options.short === true || (options.short !== false && portrait) ? 34236 : 34235; + const tombstone = { + kind: tombstoneKind, + pubkey, + created_at: createdAt, + // No imeta → nothing plays; title kept so the entry reads as removed. + tags: [["d", root], ["title", str(video?.title) || "[removed]"], ["client", "bitvid"]], + content: "", + }; + + const relays = getWriteRelays() || []; + const pool = getPool(); + if (!pool || !relays.length) { + return { ok: false, error: "no-relays" }; + } + + let publishedOk = 0; + let total = 0; + for (const template of [deleteEvent, tombstone]) { + total += 1; + let signed; + try { + signed = await signEvent(template); + } catch (error) { + userLogger.warn("[nip71Mirror] Failed to sign teardown event:", error); + continue; + } + const results = await publishEventToRelays(pool, relays, signed); + const { accepted } = summarizePublishResults(results); + if (accepted.length > 0) { + publishedOk += 1; + } + } + + return { ok: publishedOk > 0, published: publishedOk, total }; + } + + return { isAvailable, canMirror, publish, remove }; } export const nip71MirrorService = createNip71MirrorService(); diff --git a/tests/nip71-mirror-service.test.mjs b/tests/nip71-mirror-service.test.mjs index 010cc242..ab9718cd 100644 --- a/tests/nip71-mirror-service.test.mjs +++ b/tests/nip71-mirror-service.test.mjs @@ -105,3 +105,43 @@ test("publish is unavailable without a signer / pubkey", async () => { assert.equal(result.ok, false); assert.equal(result.error, "unavailable"); }); + +test("remove() publishes a NIP-09 delete (both addressable kinds) AND a tombstone", async () => { + const { service, relay } = makeService(); + const result = await service.remove(baseVideo()); + assert.equal(result.ok, true); + assert.equal(relay.events.length, 2, "delete + empty-replace tombstone"); + + const del = relay.events.find((e) => e.kind === 5); + assert.ok(del, "must publish a NIP-09 kind-5 delete"); + const aTags = del.tags.filter((t) => t[0] === "a").map((t) => t[1]).sort(); + assert.deepEqual(aTags, [ + `34235:${PUBKEY}:root-1`, + `34236:${PUBKEY}:root-1`, + ], "delete references both addressable kinds (orientation-robust)"); + const kTags = del.tags.filter((t) => t[0] === "k").map((t) => t[1]).sort(); + assert.deepEqual(kTags, ["34235", "34236"]); + + const tomb = relay.events.find((e) => e.kind === 34235 || e.kind === 34236); + assert.ok(tomb, "must publish an empty-replace tombstone"); + assert.equal(tomb.tags.find((t) => t[0] === "d")[1], "root-1", "same d-tag"); + assert.equal( + tomb.tags.some((t) => t[0] === "imeta"), + false, + "tombstone has no playable imeta", + ); +}); + +test("remove() tombstone uses the short kind for portrait videos", async () => { + const { service, relay } = makeService(); + await service.remove(baseVideo({ width: 1080, height: 1920 })); + const tomb = relay.events.find((e) => e.kind === 34235 || e.kind === 34236); + assert.equal(tomb.kind, 34236, "portrait => 34236 tombstone"); +}); + +test("remove() is unavailable without a signer", async () => { + const { service } = makeService({ available: false }); + const result = await service.remove(baseVideo()); + assert.equal(result.ok, false); + assert.equal(result.error, "unavailable"); +}); From 0c92a259c5750392b37dbd6f9c7a1b4a99f8444e Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 09:19:07 -0400 Subject: [PATCH 10/43] docs(todo): NIP-71 Phase 1 progress (1a/1b done; 1c-1e remain) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- todo/TODO_2026-06-20_pre-launch.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index 8d15e37b..a3583bfb 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -259,8 +259,15 @@ notes below. Summary: - [x] Phase 0 (commit 0e2f683b): `buildNip71MirrorEvent` (js/nostr/nip71Mirror.js) maps a bitvid video → addressable 34235/36 event; magnet/i/ox added to the imeta builder+parser; mutation-verified tests. Flag still off, no UX. -- [ ] Phase 1: opt-in toggle, publish/edit/delete lifecycle parity, flip - `FEATURE_PUBLISH_NIP71` on. +- [ ] Phase 1 (in progress): + - [x] 1a (e20191d6): `nip71MirrorService.publish` + the ALLOW_NSFW_CONTENT + gate (instance that forbids NSFW won't mirror it outward). + - [x] 1b (58f19095): `remove` teardown — NIP-09 delete (both addressable + kinds) + empty-replace tombstone. + - [ ] 1c: wire into the publish/edit/delete path + per-video opt-in flag. + - [ ] 1d: UI — toggle (edit modal + My Videos), dimension-probe capture, + hashtags input. + - [ ] 1e: flip `FEATURE_PUBLISH_NIP71` on. - [ ] Phase 1.5: NIP-89 handler reg (kind 31990 → "Open in bitvid" elsewhere); NIP-51 kind 30005 portable playlists. - [ ] Phase 2: inbound ingest of external NIP-71 videos (dedup, moderation, trust, From 783c7df0b78fcca72f4bce28011cffd0f4a8718e Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 09:40:09 -0400 Subject: [PATCH 11/43] feat(nip71): My Videos opt-in "Share to apps" mirror toggle (1c/1d) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Makes the NIP-71 mirror user-facing via the My Videos tab — the natural manage-my-videos surface — without touching the single-writer publish path. - js/constants.js: new FEATURE_NIP71_MIRROR flag (default on). Independent of FEATURE_PUBLISH_NIP71, which is already wired to the LEGACY 21/22 auto-publish in videoPublisher.js — flipping that would activate the wrong (non-opt-in) model, so the opt-in mirror gets its own flag and that legacy path stays dormant. - js/services/nip71MirrorFlags.js: per-video opt-in persistence (pubkey+videoRootId, off by default) + pure resolveMirrorToggle decision. - MyVideosController: per-row "Share to apps" / "Shared ✓" toggle (gated by the flag), wired to nip71MirrorService.publish/remove. Ineligible videos show a disabled "Share off" with the reason (private / NSFW-not-allowed-here / needs hosted URL) — the ALLOW_NSFW_CONTENT moderation gate surfaces here too. tests/nip71-mirror-flags.test.mjs: persistence scoping + toggle decision (enabled→remove, eligible→publish, ineligible→blocked). Mutation-verified. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/constants.js | 9 +++ js/services/nip71MirrorFlags.js | 77 ++++++++++++++++++++++ js/ui/profileModal/MyVideosController.js | 84 ++++++++++++++++++++++++ tests/nip71-mirror-flags.test.mjs | 56 ++++++++++++++++ 4 files changed, 226 insertions(+) create mode 100644 js/services/nip71MirrorFlags.js create mode 100644 tests/nip71-mirror-flags.test.mjs diff --git a/js/constants.js b/js/constants.js index b664ae45..f6a9c202 100644 --- a/js/constants.js +++ b/js/constants.js @@ -182,6 +182,9 @@ const DEFAULT_FLAGS = Object.freeze({ URL_FIRST_ENABLED: DEFAULT_PLAYBACK_SOURCE !== "torrent", // try URL before magnet in the player FEATURE_WATCH_HISTORY_V2: true, FEATURE_PUBLISH_NIP71: false, + // Opt-in NIP-71 *mirror* (addressable 34235/36) managed per-video from the + // My Videos tab. Independent of FEATURE_PUBLISH_NIP71 (legacy 21/22 auto-publish). + FEATURE_NIP71_MIRROR: true, FEATURE_HASHTAG_PREFERENCES: false, FEATURE_SEARCH_FILTERS: Boolean(IS_DEV_MODE), FEATURE_TRUST_SEEDS: true, // Rollback: disable to drop baseline trust seeds without code changes. @@ -202,6 +205,7 @@ const runtimeFlags = (() => { URL_FIRST_ENABLED: DEFAULT_FLAGS.URL_FIRST_ENABLED, FEATURE_WATCH_HISTORY_V2: DEFAULT_FLAGS.FEATURE_WATCH_HISTORY_V2, FEATURE_PUBLISH_NIP71: DEFAULT_FLAGS.FEATURE_PUBLISH_NIP71, + FEATURE_NIP71_MIRROR: DEFAULT_FLAGS.FEATURE_NIP71_MIRROR, FEATURE_HASHTAG_PREFERENCES: DEFAULT_FLAGS.FEATURE_HASHTAG_PREFERENCES, FEATURE_SEARCH_FILTERS: DEFAULT_FLAGS.FEATURE_SEARCH_FILTERS, FEATURE_TRUST_SEEDS: DEFAULT_FLAGS.FEATURE_TRUST_SEEDS, @@ -233,6 +237,11 @@ export let FEATURE_PUBLISH_NIP71 = coerceBoolean( DEFAULT_FLAGS.FEATURE_PUBLISH_NIP71 ); +export let FEATURE_NIP71_MIRROR = coerceBoolean( + runtimeFlags.FEATURE_NIP71_MIRROR, + DEFAULT_FLAGS.FEATURE_NIP71_MIRROR +); + export let FEATURE_HASHTAG_PREFERENCES = coerceBoolean( runtimeFlags.FEATURE_HASHTAG_PREFERENCES, DEFAULT_FLAGS.FEATURE_HASHTAG_PREFERENCES diff --git a/js/services/nip71MirrorFlags.js b/js/services/nip71MirrorFlags.js new file mode 100644 index 00000000..7c733f6b --- /dev/null +++ b/js/services/nip71MirrorFlags.js @@ -0,0 +1,77 @@ +// Per-video opt-in state for the NIP-71 mirror, persisted locally per +// pubkey+videoRootId. Off by default — a publisher explicitly opts a video in +// (from the My Videos tab). Local-only is fine: it's a convenience for the +// management UI; the authoritative state is whether the mirror event exists on +// relays. + +const FLAG_KEY = "bitvid:nip71-mirror:v1"; + +function readFlags() { + try { + if (typeof localStorage === "undefined") { + return {}; + } + const raw = localStorage.getItem(FLAG_KEY); + const parsed = raw ? JSON.parse(raw) : {}; + return parsed && typeof parsed === "object" ? parsed : {}; + } catch (error) { + return {}; + } +} + +function writeFlags(flags) { + try { + if (typeof localStorage !== "undefined") { + localStorage.setItem(FLAG_KEY, JSON.stringify(flags || {})); + } + } catch (error) { + // Best-effort. + } +} + +function norm(value) { + return typeof value === "string" ? value.trim() : ""; +} + +export function isMirrorEnabled(pubkey, videoRootId) { + const pk = norm(pubkey); + const root = norm(videoRootId); + if (!pk || !root) { + return false; + } + return readFlags()[pk]?.[root] === true; +} + +export function setMirrorEnabled(pubkey, videoRootId, enabled) { + const pk = norm(pubkey); + const root = norm(videoRootId); + if (!pk || !root) { + return; + } + const flags = readFlags(); + const entry = flags[pk] && typeof flags[pk] === "object" ? flags[pk] : {}; + if (enabled === true) { + entry[root] = true; + } else { + delete entry[root]; + } + flags[pk] = entry; + writeFlags(flags); +} + +// Pure decision for a toggle click: given the current opt-in state and the +// service's eligibility check (canMirror), what should happen? +// { action: "remove" } — currently on → tear down +// { action: "publish" } — eligible + off → publish +// { action: "blocked", reason: "<why>" } — off but ineligible (private / nsfw / no-url) +export function resolveMirrorToggle({ enabled, eligibility } = {}) { + if (enabled === true) { + return { action: "remove" }; + } + if (!eligibility || eligibility.ok !== true) { + return { action: "blocked", reason: eligibility?.reason || "ineligible" }; + } + return { action: "publish" }; +} + +export const NIP71_MIRROR_FLAG_KEY = FLAG_KEY; diff --git a/js/ui/profileModal/MyVideosController.js b/js/ui/profileModal/MyVideosController.js index 45035de5..da22c043 100644 --- a/js/ui/profileModal/MyVideosController.js +++ b/js/ui/profileModal/MyVideosController.js @@ -4,6 +4,20 @@ import { getApplication } from "../../applicationContext.js"; import { collapseUserVideos } from "./myVideosData.js"; import { classifyVideoHealth, isUrlUnderBase } from "./myVideosHealth.js"; import { reconcileStorage } from "./myVideosReconcile.js"; +import { FEATURE_NIP71_MIRROR } from "../../constants.js"; +import { nip71MirrorService } from "../../services/nip71MirrorService.js"; +import { + isMirrorEnabled, + setMirrorEnabled, + resolveMirrorToggle, +} from "../../services/nip71MirrorFlags.js"; + +const MIRROR_REASON_TEXT = { + private: "Private videos can't be shared to other apps.", + "nsfw-blocked": "NSFW videos aren't shared from this instance.", + "no-url": "Needs a hosted (HTTPS) URL to appear on other apps.", + invalid: "This video can't be mirrored.", +}; // Maps a health severity to a status-pill style, mirroring // ProfileRelayController.applyStatusPill so the look stays consistent. @@ -369,6 +383,10 @@ export class MyVideosController { actions.appendChild( this.buildActionButton("Edit", () => this.handleEdit(video)), ); + const mirrorBtn = this.buildMirrorButton(video); + if (mirrorBtn) { + actions.appendChild(mirrorBtn); + } actions.appendChild( this.buildActionButton("Delete", () => this.handleDelete(video), true), ); @@ -376,6 +394,72 @@ export class MyVideosController { return actions; } + // Opt-in "publish to other Nostr video apps (NIP-71)" toggle. Reflects the + // per-video opt-in flag; disabled (with a reason) for ineligible videos. + buildMirrorButton(video) { + if (!FEATURE_NIP71_MIRROR) { + return null; + } + const enabled = isMirrorEnabled(this.pubkey, video?.videoRootId); + const eligibility = nip71MirrorService.canMirror(video); + if (!enabled && eligibility.ok !== true) { + const reason = MIRROR_REASON_TEXT[eligibility.reason] || "Can't be shared."; + const btn = this.buildActionButton("Share off", () => { + this.mainController.showStatus?.(reason); + }); + btn.disabled = true; + btn.title = reason; + btn.classList.add("opacity-50"); + return btn; + } + return this.buildActionButton( + enabled ? "Shared ✓" : "Share to apps", + (event) => this.handleToggleMirror(video, event?.currentTarget || null), + ); + } + + async handleToggleMirror(video, btn) { + const enabled = isMirrorEnabled(this.pubkey, video?.videoRootId); + const eligibility = nip71MirrorService.canMirror(video); + const decision = resolveMirrorToggle({ enabled, eligibility }); + + if (decision.action === "blocked") { + this.mainController.showError?.( + MIRROR_REASON_TEXT[decision.reason] || "This video can't be shared.", + ); + return; + } + if (btn) { + btn.disabled = true; + } + try { + if (decision.action === "publish") { + const result = await nip71MirrorService.publish(video); + if (result?.ok) { + setMirrorEnabled(this.pubkey, video.videoRootId, true); + this.mainController.showSuccess?.( + `Shared to other Nostr apps (${result.accepted}/${result.total} relays).`, + ); + } else { + this.mainController.showError?.( + "Couldn't share this video. Please try again.", + ); + } + } else { + const result = await nip71MirrorService.remove(video); + setMirrorEnabled(this.pubkey, video.videoRootId, false); + this.mainController.showSuccess?.( + result?.ok ? "Removed from other Nostr apps." : "Stopped sharing locally.", + ); + } + } catch (error) { + devLogger.warn("[myVideos] mirror toggle failed:", error); + this.mainController.showError?.("Sharing action failed. Please try again."); + } finally { + void this.populate({ forceFetch: true }); + } + } + buildActionButton(label, onClick, danger = false) { const btn = document.createElement("button"); btn.type = "button"; diff --git a/tests/nip71-mirror-flags.test.mjs b/tests/nip71-mirror-flags.test.mjs new file mode 100644 index 00000000..84558e64 --- /dev/null +++ b/tests/nip71-mirror-flags.test.mjs @@ -0,0 +1,56 @@ +// Per-video NIP-71 mirror opt-in: persistence (per pubkey+videoRootId) and the +// pure toggle-decision used by the My Videos tab. + +import "./test-helpers/setup-localstorage.mjs"; +import assert from "node:assert/strict"; +import test from "node:test"; +import { + isMirrorEnabled, + setMirrorEnabled, + resolveMirrorToggle, +} from "../js/services/nip71MirrorFlags.js"; + +const PK = "a".repeat(64); + +test("opt-in flag persists per pubkey+videoRootId and is off by default", () => { + localStorage.clear(); + assert.equal(isMirrorEnabled(PK, "root-1"), false, "off by default"); + + setMirrorEnabled(PK, "root-1", true); + assert.equal(isMirrorEnabled(PK, "root-1"), true); + assert.equal(isMirrorEnabled(PK, "root-2"), false, "scoped per videoRootId"); + assert.equal(isMirrorEnabled("b".repeat(64), "root-1"), false, "scoped per pubkey"); + + setMirrorEnabled(PK, "root-1", false); + assert.equal(isMirrorEnabled(PK, "root-1"), false, "can be turned back off"); +}); + +test("resolveMirrorToggle: enabled => remove", () => { + assert.deepEqual( + resolveMirrorToggle({ enabled: true, eligibility: { ok: true } }), + { action: "remove" }, + ); + // remove even if currently ineligible (so a now-private video can be pulled). + assert.deepEqual( + resolveMirrorToggle({ enabled: true, eligibility: { ok: false, reason: "private" } }), + { action: "remove" }, + ); +}); + +test("resolveMirrorToggle: off + eligible => publish", () => { + assert.deepEqual( + resolveMirrorToggle({ enabled: false, eligibility: { ok: true } }), + { action: "publish" }, + ); +}); + +test("resolveMirrorToggle: off + ineligible => blocked with reason", () => { + assert.deepEqual( + resolveMirrorToggle({ enabled: false, eligibility: { ok: false, reason: "no-url" } }), + { action: "blocked", reason: "no-url" }, + ); + assert.deepEqual( + resolveMirrorToggle({ enabled: false }), + { action: "blocked", reason: "ineligible" }, + ); +}); From 6b97626da8cd187257e20d49ea1520097da1210d Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 09:41:15 -0400 Subject: [PATCH 12/43] docs(todo): NIP-71 mirror toggle live; note remaining polish Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- todo/TODO_2026-06-20_pre-launch.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index a3583bfb..daad6071 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -264,10 +264,16 @@ notes below. Summary: gate (instance that forbids NSFW won't mirror it outward). - [x] 1b (58f19095): `remove` teardown — NIP-09 delete (both addressable kinds) + empty-replace tombstone. - - [ ] 1c: wire into the publish/edit/delete path + per-video opt-in flag. - - [ ] 1d: UI — toggle (edit modal + My Videos), dimension-probe capture, - hashtags input. - - [ ] 1e: flip `FEATURE_PUBLISH_NIP71` on. + - [x] 1c/1d (783c7df0): My Videos "Share to apps" opt-in toggle wired to the + mirror service via new FEATURE_NIP71_MIRROR flag (on) + + nip71MirrorFlags (per-video opt-in). Ineligible videos show the reason + (private / NSFW-blocked / no-url). LIVE on unstable. + - [ ] Remaining polish: auto-update the mirror when a video is edited in the + edit modal (currently the user re-toggles from My Videos); capture + dimensions at upload (short 34236 + imeta dim — defaults to 34235 now); + hashtags input → t tags (bitvid has no hashtags field yet). + - NOTE: legacy FEATURE_PUBLISH_NIP71 (videoPublisher.js auto-21/22) stays + OFF/dormant — superseded by the opt-in 34235/36 mirror; clean up later. - [ ] Phase 1.5: NIP-89 handler reg (kind 31990 → "Open in bitvid" elsewhere); NIP-51 kind 30005 portable playlists. - [ ] Phase 2: inbound ingest of external NIP-71 videos (dedup, moderation, trust, From dff1a27c0c670e4bedf9a3b509bacda3984d77f2 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 09:57:59 -0400 Subject: [PATCH 13/43] docs(nip71): record dual-event vs single-canonical decision Decided to keep the dual-event model (kind 30078 canonical + additive 34235/36 mirror) rather than collapse to a single canonical 34235 note. Captures the rationale: private videos can't be public NIP-71 (no encryption), and kind 30078 is the addressing root (watch-history/comments/reactions/views/playlists/naddr/ moderation + hardcoded video filters) so switching it is a large forward-only migration not worth the interop win right now. Accepted trade-off: keep the mirror in sync on edit/delete (Phase 1 polish). Phase 0/1 builder/parser remain reusable if single-canonical is ever revisited. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- docs/nip71-migration-plan.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/nip71-migration-plan.md b/docs/nip71-migration-plan.md index 05509fc7..3b3d7d0e 100644 --- a/docs/nip71-migration-plan.md +++ b/docs/nip71-migration-plan.md @@ -15,6 +15,24 @@ NIP-71 videos. Decisions below were made with the maintainer; see "Decisions". 1. **Kind: addressable `34235` (normal) / `34236` (short).** 2. **Opt-in, off by default** — per-video toggle; private videos never mirrored. 3. **Scope: outbound first, inbound later** (Phase 1 vs Phase 2). +4. **Dual-event model (decided 2026-06-22), NOT single-canonical.** Considered + collapsing to one note type by making bitvid's canonical video event itself + kind 34235/36. Verdict: keep kind 30078 canonical + an additive mirror. + - *Why a single note is appealing:* one event, no drift, fully discoverable. + - *Why we're not doing it (yet):* (a) **private videos can't be public NIP-71** + (no encryption) so they'd stay 30078 regardless — "one note for everything" + is impossible, only "one note for public"; (b) **kind 30078 is the addressing + root** — `30078:pubkey:d` is referenced by watch-history pointers, comments, + reactions, view counts, playlists, naddr deep-links, and moderation, plus + hardcoded video subscription filters (`nostrService.VIDEO_KIND`, + `videoPointer.DEFAULT_VIDEO_KIND`, `kinds:[30078]` across client/channel/ + search). Switching the canonical kind ripples through all of them and needs a + forward-only dual-read migration. Not worth the risk for the interop win. + - *Accepted trade-off:* two events per shared video; the mirror can drift from + the canonical, so the lifecycle must keep them in sync (auto re-publish on + edit / remove on delete) — see Phase 1 polish. + - The Phase 0/1 builder+parser are reusable if we ever revisit single-canonical + (they'd become bitvid's native 34235 read/write layer). ### Why 34235/34236 (research-validated) From 89d9a62b87d0c648a12dec46631365ab05680457 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 10:22:27 -0400 Subject: [PATCH 14/43] feat(nip71): auto-sync the mirror on edit/delete (event-driven, no nostrService growth) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the dual-event drift the user flagged: an opted-in 34235 mirror now stays in lockstep with the canonical 30078 automatically — no per-video babysitting. - js/services/nip71MirrorSync.js: syncNip71MirrorAfterEdit (re-publish if still eligible; unshare + clear flag if it became ineligible, e.g. flipped private), syncNip71MirrorAfterDelete (remove + clear flag). Plus initNip71MirrorSync() which subscribes to nostrService's EXISTING videos:edited/videos:deleted events — so the size-capped nostrService is untouched. - js/services/nip71MirrorFlags.js: pure resolveEditSync/resolveDeleteSync decisions. - applicationBootstrap: one-line initNip71MirrorSync(app.nostrService) wiring. - All best-effort: a mirror failure never breaks the core edit/delete. tests: resolver decisions (lockstep/unshare/none) + delete clears the flag + event wiring is idempotent. Mutation-verified (eligibility branch load-bearing). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/services/nip71MirrorFlags.js | 20 +++++++ js/services/nip71MirrorSync.js | 95 +++++++++++++++++++++++++++++++ js/ui/applicationBootstrap.js | 4 ++ tests/nip71-mirror-flags.test.mjs | 18 ++++++ tests/nip71-mirror-sync.test.mjs | 45 +++++++++++++++ 5 files changed, 182 insertions(+) create mode 100644 js/services/nip71MirrorSync.js create mode 100644 tests/nip71-mirror-sync.test.mjs diff --git a/js/services/nip71MirrorFlags.js b/js/services/nip71MirrorFlags.js index 7c733f6b..ebe7f95e 100644 --- a/js/services/nip71MirrorFlags.js +++ b/js/services/nip71MirrorFlags.js @@ -74,4 +74,24 @@ export function resolveMirrorToggle({ enabled, eligibility } = {}) { return { action: "publish" }; } +// Lifecycle decisions for keeping an opted-in mirror in sync (used by the +// nostrService edit/delete hooks). Pure so they're cheat-resistant to test. +// +// On EDIT of a mirrored video: re-publish if still eligible; if it became +// ineligible (e.g. flipped private), pull the mirror down and clear the flag. +export function resolveEditSync({ featureOn, enabled, eligible } = {}) { + if (featureOn !== true || enabled !== true) { + return { action: "none" }; + } + return eligible === true ? { action: "publish" } : { action: "unshare" }; +} + +// On DELETE of a mirrored video: remove the mirror and clear the flag. +export function resolveDeleteSync({ featureOn, enabled } = {}) { + if (featureOn !== true || enabled !== true) { + return { action: "none" }; + } + return { action: "unshare" }; +} + export const NIP71_MIRROR_FLAG_KEY = FLAG_KEY; diff --git a/js/services/nip71MirrorSync.js b/js/services/nip71MirrorSync.js new file mode 100644 index 00000000..853e0ae2 --- /dev/null +++ b/js/services/nip71MirrorSync.js @@ -0,0 +1,95 @@ +// Keeps an opted-in NIP-71 mirror (kind 34235/36) in lockstep with the canonical +// kind-30078 video on edit/delete — the "no UX fuss" glue so the two events never +// drift. Extracted from nostrService (which is size-capped); called from its +// edit/delete handlers. All best-effort: a mirror failure must never break the +// core edit/delete. + +import { FEATURE_NIP71_MIRROR } from "../constants.js"; +import { nip71MirrorService } from "./nip71MirrorService.js"; +import { + isMirrorEnabled, + setMirrorEnabled, + resolveEditSync, + resolveDeleteSync, +} from "./nip71MirrorFlags.js"; +import { devLogger } from "../utils/logger.js"; + +export async function syncNip71MirrorAfterEdit({ updatedData, pubkey } = {}) { + if (!updatedData || typeof updatedData !== "object") { + return; + } + const videoRootId = + typeof updatedData.videoRootId === "string" ? updatedData.videoRootId : ""; + const authorPubkey = typeof pubkey === "string" ? pubkey : ""; + if (!videoRootId || !authorPubkey) { + return; + } + const video = { ...updatedData, pubkey: authorPubkey, videoRootId }; + const decision = resolveEditSync({ + featureOn: FEATURE_NIP71_MIRROR, + enabled: isMirrorEnabled(authorPubkey, videoRootId), + eligible: nip71MirrorService.canMirror(video).ok, + }); + if (decision.action === "none") { + return; + } + try { + if (decision.action === "publish") { + await nip71MirrorService.publish(video); + } else { + // "unshare": no longer eligible (e.g. flipped private) — pull it down. + await nip71MirrorService.remove(video); + setMirrorEnabled(authorPubkey, videoRootId, false); + } + } catch (error) { + devLogger.warn("[nip71MirrorSync] edit sync failed", error); + } +} + +// Drive the lifecycle off nostrService's existing "videos:edited"/"videos:deleted" +// events — so the size-capped nostrService needs no changes. Idempotent: a guard +// prevents double-registration. Returns an unsubscribe fn. +let registered = false; +export function initNip71MirrorSync(nostrService) { + if (!nostrService || typeof nostrService.on !== "function" || registered) { + return () => {}; + } + registered = true; + const offEdit = nostrService.on("videos:edited", ({ updatedData, pubkey } = {}) => { + void syncNip71MirrorAfterEdit({ updatedData, pubkey }); + }); + const offDelete = nostrService.on("videos:deleted", ({ videoRootId, video, pubkey } = {}) => { + void syncNip71MirrorAfterDelete({ videoRootId, video, pubkey }); + }); + return () => { + registered = false; + if (typeof offEdit === "function") offEdit(); + if (typeof offDelete === "function") offDelete(); + }; +} + +export async function syncNip71MirrorAfterDelete({ videoRootId, video, pubkey } = {}) { + const root = typeof videoRootId === "string" ? videoRootId : ""; + const authorPubkey = typeof pubkey === "string" ? pubkey : ""; + if (!root || !authorPubkey) { + return; + } + const decision = resolveDeleteSync({ + featureOn: FEATURE_NIP71_MIRROR, + enabled: isMirrorEnabled(authorPubkey, root), + }); + if (decision.action === "none") { + return; + } + try { + await nip71MirrorService.remove({ + ...(video || {}), + videoRootId: root, + pubkey: authorPubkey, + }); + } catch (error) { + devLogger.warn("[nip71MirrorSync] delete sync failed", error); + } finally { + setMirrorEnabled(authorPubkey, root, false); + } +} diff --git a/js/ui/applicationBootstrap.js b/js/ui/applicationBootstrap.js index 63c44fb1..fc79375e 100644 --- a/js/ui/applicationBootstrap.js +++ b/js/ui/applicationBootstrap.js @@ -17,6 +17,7 @@ import hashtagPreferences, { import NwcSettingsService from "../services/nwcSettingsService.js"; import nostrService from "../services/nostrService.js"; import storageService from "../services/storageService.js"; +import { initNip71MirrorSync } from "../services/nip71MirrorSync.js"; import watchHistoryService from "../watchHistoryService.js"; import r2Service from "../services/r2Service.js"; import s3UploadService from "../services/s3UploadService.js"; @@ -849,6 +850,9 @@ export default class ApplicationBootstrap { app.videoSubscription = subscription || null; }), ); + // Keep an opted-in NIP-71 mirror in lockstep on edit/delete (event-driven so + // nostrService stays untouched). + nostrUnsubscribes.push(initNip71MirrorSync(app.nostrService)); nostrUnsubscribes.push( app.nostrService.on("directMessages:notification", () => { void app.refreshUnreadDmIndicator({ reason: "dm-notification" }); diff --git a/tests/nip71-mirror-flags.test.mjs b/tests/nip71-mirror-flags.test.mjs index 84558e64..f3d36245 100644 --- a/tests/nip71-mirror-flags.test.mjs +++ b/tests/nip71-mirror-flags.test.mjs @@ -8,6 +8,8 @@ import { isMirrorEnabled, setMirrorEnabled, resolveMirrorToggle, + resolveEditSync, + resolveDeleteSync, } from "../js/services/nip71MirrorFlags.js"; const PK = "a".repeat(64); @@ -54,3 +56,19 @@ test("resolveMirrorToggle: off + ineligible => blocked with reason", () => { { action: "blocked", reason: "ineligible" }, ); }); + +test("resolveEditSync: keeps a mirrored video in lockstep, unshares when ineligible", () => { + // not mirrored, or feature off → do nothing + assert.deepEqual(resolveEditSync({ featureOn: true, enabled: false, eligible: true }), { action: "none" }); + assert.deepEqual(resolveEditSync({ featureOn: false, enabled: true, eligible: true }), { action: "none" }); + // mirrored + still eligible → re-publish (no drift) + assert.deepEqual(resolveEditSync({ featureOn: true, enabled: true, eligible: true }), { action: "publish" }); + // mirrored but became ineligible (e.g. now private) → pull it down + assert.deepEqual(resolveEditSync({ featureOn: true, enabled: true, eligible: false }), { action: "unshare" }); +}); + +test("resolveDeleteSync: removes the mirror only when it was shared", () => { + assert.deepEqual(resolveDeleteSync({ featureOn: true, enabled: true }), { action: "unshare" }); + assert.deepEqual(resolveDeleteSync({ featureOn: true, enabled: false }), { action: "none" }); + assert.deepEqual(resolveDeleteSync({ featureOn: false, enabled: true }), { action: "none" }); +}); diff --git a/tests/nip71-mirror-sync.test.mjs b/tests/nip71-mirror-sync.test.mjs new file mode 100644 index 00000000..9bca3c29 --- /dev/null +++ b/tests/nip71-mirror-sync.test.mjs @@ -0,0 +1,45 @@ +// Auto-sync glue: keeps an opted-in NIP-71 mirror in lockstep on edit/delete, +// driven by nostrService's existing videos:edited/videos:deleted events (so the +// size-capped nostrService is untouched). + +import "./test-helpers/setup-localstorage.mjs"; +import assert from "node:assert/strict"; +import test from "node:test"; +import { + syncNip71MirrorAfterDelete, + initNip71MirrorSync, +} from "../js/services/nip71MirrorSync.js"; +import { isMirrorEnabled, setMirrorEnabled } from "../js/services/nip71MirrorFlags.js"; + +const PK = "a".repeat(64); + +test("deleting a SHARED video clears its opt-in flag (mirror torn down)", async () => { + localStorage.clear(); + setMirrorEnabled(PK, "root-x", true); + await syncNip71MirrorAfterDelete({ videoRootId: "root-x", video: {}, pubkey: PK }); + assert.equal(isMirrorEnabled(PK, "root-x"), false, "flag cleared after delete"); +}); + +test("deleting a NON-shared video is a no-op (and never throws)", async () => { + localStorage.clear(); + await syncNip71MirrorAfterDelete({ videoRootId: "root-y", video: {}, pubkey: PK }); + assert.equal(isMirrorEnabled(PK, "root-y"), false); +}); + +test("initNip71MirrorSync subscribes to edit+delete events, idempotently", () => { + const events = {}; + const svc = { + on(name, handler) { + (events[name] = events[name] || []).push(handler); + return () => {}; + }, + }; + const off = initNip71MirrorSync(svc); + assert.equal(events["videos:edited"]?.length, 1, "subscribes to videos:edited"); + assert.equal(events["videos:deleted"]?.length, 1, "subscribes to videos:deleted"); + + initNip71MirrorSync(svc); // guard prevents double-registration + assert.equal(events["videos:edited"].length, 1, "no double-register"); + + off(); // resets the guard for other tests +}); From 618525b4124a274cfdaaa76077bddc84ad5fb72f Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 10:23:23 -0400 Subject: [PATCH 15/43] docs(todo): NIP-71 mirror auto-sync done; note remaining polish Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- todo/TODO_2026-06-20_pre-launch.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index daad6071..82a864c1 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -268,10 +268,13 @@ notes below. Summary: mirror service via new FEATURE_NIP71_MIRROR flag (on) + nip71MirrorFlags (per-video opt-in). Ineligible videos show the reason (private / NSFW-blocked / no-url). LIVE on unstable. - - [ ] Remaining polish: auto-update the mirror when a video is edited in the - edit modal (currently the user re-toggles from My Videos); capture - dimensions at upload (short 34236 + imeta dim — defaults to 34235 now); - hashtags input → t tags (bitvid has no hashtags field yet). + - [x] Auto-sync (89d9a62b): mirror stays in lockstep on edit/delete, + event-driven via nostrService videos:edited/deleted (no nostrService + growth). Edit re-publishes (or unshares if now-private); delete tears + down + clears the flag. + - [ ] Remaining polish: capture dimensions at upload (short 34236 + imeta + dim — defaults to 34235 now); hashtags input → t tags (no hashtags + field yet); optional account-level "auto-share new public videos". - NOTE: legacy FEATURE_PUBLISH_NIP71 (videoPublisher.js auto-21/22) stays OFF/dormant — superseded by the opt-in 34235/36 mirror; clean up later. - [ ] Phase 1.5: NIP-89 handler reg (kind 31990 → "Open in bitvid" elsewhere); From a1bf7b151f3b4f236d9990804f2529ca169a17a9 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 10:41:46 -0400 Subject: [PATCH 16/43] feat(video): hashtags data layer (content + t tags + parser) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Polish toward NIP-71 parity + better native discovery. Hashtags now flow through the video data layer (the input UI in the upload/edit modals is the remaining step): - videoPayloadBuilder: sanitizeVideoHashtags (normalize/dedupe, drop reserved "video"); persists hashtags in 30078 content AND emits ["t", tag] so bitvid's own feed scoring (exploreScoring/kidsScoring already read t tags) and relay queries pick them up. - convertEventToVideo: exposes video.hashtags from content → the NIP-71 mirror builder already carries them as t tags, so a shared video is categorized on other apps too. tests/video-hashtags.test.mjs: sanitize rules, content+t-tag emission, round-trip to video.hashtags, empty case. Mutation-verified (t-tag emission load-bearing). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/nostr/nip71.js | 6 ++++ js/nostr/videoPayloadBuilder.js | 42 +++++++++++++++++++++-- tests/video-hashtags.test.mjs | 59 +++++++++++++++++++++++++++++++++ 3 files changed, 104 insertions(+), 3 deletions(-) create mode 100644 tests/video-hashtags.test.mjs diff --git a/js/nostr/nip71.js b/js/nostr/nip71.js index cea83e9c..069e8b43 100644 --- a/js/nostr/nip71.js +++ b/js/nostr/nip71.js @@ -1515,6 +1515,11 @@ export function convertEventToVideo(event = {}) { const isPrivate = parsedContent.isPrivate === true; const isNsfw = parsedContent.isNsfw === true; const isForKids = parsedContent.isForKids === true && !isNsfw; + const hashtags = Array.isArray(parsedContent.hashtags) + ? parsedContent.hashtags + .filter((value) => typeof value === "string" && value.trim()) + .map((value) => value.trim()) + : []; const videoRootId = safeTrim(parsedContent.videoRootId) || event.id; const wsField = safeTrim(parsedContent.ws); const xsField = safeTrim(parsedContent.xs); @@ -1618,6 +1623,7 @@ export function convertEventToVideo(event = {}) { description, mode, deleted, + hashtags, ws, xs, storagePointer, diff --git a/js/nostr/videoPayloadBuilder.js b/js/nostr/videoPayloadBuilder.js index 82361bca..430dcb52 100644 --- a/js/nostr/videoPayloadBuilder.js +++ b/js/nostr/videoPayloadBuilder.js @@ -5,8 +5,33 @@ import { buildNip71MetadataTags, extractNip71MetadataFromTags } from "./nip71.js import { buildVideoPostEvent } from "../nostrEventSchemas.js"; import { computeSha256HexFromValue } from "../utils/cryptoUtils.js"; import { inferMimeTypeFromUrl } from "../utils/mime.js"; +import { normalizeHashtag } from "../utils/hashtagNormalization.js"; import { devLogger } from "../utils/logger.js"; +// Normalize free-form hashtag input (array or comma/space string) to a deduped, +// lowercased list. Drops the reserved "video" topic tag and caps the count. +export function sanitizeVideoHashtags(input) { + const raw = Array.isArray(input) + ? input + : typeof input === "string" + ? input.split(/[\s,]+/) + : []; + const out = []; + const seen = new Set(); + for (const item of raw) { + const tag = normalizeHashtag(item); + if (!tag || tag === "video" || seen.has(tag)) { + continue; + } + seen.add(tag); + out.push(tag); + if (out.length >= 20) { + break; + } + } + return out; +} + export function extractVideoPublishPayload(rawPayload) { let videoData = rawPayload; let nip71Metadata = null; @@ -161,13 +186,24 @@ export async function prepareVideoPublishPayload(videoPayload, pubkey, { timesta contentObject.xs = finalXs; } + const hashtags = sanitizeVideoHashtags(videoData.hashtags); + if (hashtags.length) { + contentObject.hashtags = hashtags; + } + const nip71Tags = buildNip71MetadataTags( nip71Metadata && typeof nip71Metadata === "object" ? nip71Metadata : null ); - const additionalTags = storagePointer - ? [["s", storagePointer], ...nip71Tags] - : nip71Tags; + // Emit hashtags as `t` tags on the 30078 so bitvid's own feed scoring (and + // relay queries) pick them up, in addition to persisting them in content. + const hashtagTags = hashtags.map((tag) => ["t", tag]); + + const additionalTags = [ + ...(storagePointer ? [["s", storagePointer]] : []), + ...hashtagTags, + ...nip71Tags, + ]; const event = buildVideoPostEvent({ pubkey: normalizedPubkey, diff --git a/tests/video-hashtags.test.mjs b/tests/video-hashtags.test.mjs new file mode 100644 index 00000000..49fe3241 --- /dev/null +++ b/tests/video-hashtags.test.mjs @@ -0,0 +1,59 @@ +// Hashtags polish: a video's hashtags are persisted in the 30078 content AND +// emitted as `t` tags (so bitvid's feed scoring + relay queries see them), and +// round-trip back through convertEventToVideo onto `video.hashtags` (which the +// NIP-71 mirror builder then carries as `t` tags too). + +import "./test-helpers/setup-localstorage.mjs"; +import assert from "node:assert/strict"; +import test from "node:test"; +import { + sanitizeVideoHashtags, + prepareVideoPublishPayload, +} from "../js/nostr/videoPayloadBuilder.js"; +import { convertEventToVideo } from "../js/nostr/nip71.js"; + +const PUBKEY = "a".repeat(64); + +function publish(extra) { + return prepareVideoPublishPayload( + { title: "Hello", url: "https://cdn.example.com/v.mp4", ...extra }, + PUBKEY, + { timestamp: 1000 }, + ); +} + +test("sanitizeVideoHashtags: normalizes, dedupes, drops the reserved 'video' tag", () => { + assert.deepEqual( + sanitizeVideoHashtags(["#GameStr", "gamestr", " Voxel ", "video", ""]), + ["gamestr", "voxel"], + ); + assert.deepEqual(sanitizeVideoHashtags("#ai, blender cartoon"), ["ai", "blender", "cartoon"]); + assert.deepEqual(sanitizeVideoHashtags(null), []); +}); + +test("publish payload persists hashtags in content AND emits matching `t` tags", async () => { + const { event } = await publish({ + videoRootId: "root-1", + hashtags: ["#GameStr", "voxel", "voxel"], + }); + + const content = JSON.parse(event.content); + assert.deepEqual(content.hashtags, ["gamestr", "voxel"], "stored in content (normalized/deduped)"); + + const tTags = event.tags.filter((t) => t[0] === "t").map((t) => t[1]); + assert.ok(tTags.includes("gamestr")); + assert.ok(tTags.includes("voxel")); +}); + +test("round-trip: convertEventToVideo exposes video.hashtags from content", async () => { + const { event } = await publish({ videoRootId: "root-2", hashtags: ["nostr", "p2p"] }); + const video = convertEventToVideo(event); + assert.equal(video.invalid, false); + assert.deepEqual(video.hashtags.sort(), ["nostr", "p2p"]); +}); + +test("no hashtags → no content.hashtags", async () => { + const { event } = await publish({ videoRootId: "root-3" }); + const content = JSON.parse(event.content); + assert.equal(content.hashtags, undefined); +}); From e793b359c3501d02bc72a1e631cd2bfe85641a7a Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 11:30:12 -0400 Subject: [PATCH 17/43] feat(nip71): mirror carries the video's hashtags (from t tags / nip71 metadata) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the real hashtag gap. Verified the upload modal's existing NIP-71 hashtag editor (Nip71FormManager → nip71.hashtags → buildNip71MetadataTags) ALREADY emits `t` tags on the 30078, which bitvid's feed scoring already reads — so publish+feed hashtags work today. The gap was the 34235 mirror reading only `video.hashtags` (empty for form-authored videos). buildNip71MirrorEvent now resolves hashtags from whichever source is present — video.hashtags, video.nip71.hashtags, or the raw `t` tags on video.tags — deduped, lowercased, with bitvid's internal "video" topic tag dropped. So a shared video stays categorized on Nostube/Amethyst regardless of how its tags were authored. (The earlier content.hashtags builder path from a1bf7b15 stays as a harmless secondary input; the form/t-tag path is the one users actually exercise.) tests/nip71-mirror.test.mjs: hashtags sourced from nip71.hashtags; from raw t tags with the 'video' topic dropped + deduped. Mutation-verified. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/nostr/nip71Mirror.js | 33 ++++++++++++++++++++++++++++++--- js/nostr/videoPayloadBuilder.js | 18 +++++++++++++++--- tests/nip71-mirror.test.mjs | 18 ++++++++++++++++++ tests/video-hashtags.test.mjs | 29 +++++++++++++++++++++++++++++ 4 files changed, 92 insertions(+), 6 deletions(-) diff --git a/js/nostr/nip71Mirror.js b/js/nostr/nip71Mirror.js index ff1c4a5a..7a7faf74 100644 --- a/js/nostr/nip71Mirror.js +++ b/js/nostr/nip71Mirror.js @@ -46,6 +46,35 @@ export function deriveVideoMime(url) { return VIDEO_MIME_BY_EXT[ext] || "video/mp4"; } +// Hashtags can live in several places depending on how the video was authored: +// bitvid's own `hashtags`, the parsed NIP-71 metadata (`nip71.hashtags`), or the +// raw `t` tags on the 30078 (produced by the upload form's NIP-71 hashtag editor). +// Source from whichever is present so a shared video stays categorized elsewhere. +// Drops bitvid's internal "video" topic tag and dedupes (case-insensitive). +function resolveHashtags(video) { + let raw = []; + if (Array.isArray(video.hashtags) && video.hashtags.length) { + raw = video.hashtags; + } else if (Array.isArray(video.nip71?.hashtags) && video.nip71.hashtags.length) { + raw = video.nip71.hashtags; + } else if (Array.isArray(video.tags)) { + raw = video.tags + .filter((t) => Array.isArray(t) && t[0] === "t" && typeof t[1] === "string") + .map((t) => t[1]); + } + const out = []; + const seen = new Set(); + for (const value of raw) { + const tag = str(value).toLowerCase(); + if (!tag || tag === "video" || seen.has(tag)) { + continue; + } + seen.add(tag); + out.push(tag); + } + return out; +} + function normalizePublishedAt(value) { const numeric = Number(value); if (!Number.isFinite(numeric) || numeric <= 0) { @@ -114,9 +143,7 @@ export function buildNip71MirrorEvent(video, options = {}) { duration: Number.isFinite(duration) && duration > 0 ? duration : undefined, }; - const hashtags = Array.isArray(video.hashtags) - ? video.hashtags.map(str).filter(Boolean) - : []; + const hashtags = resolveHashtags(video); const metadata = { title, diff --git a/js/nostr/videoPayloadBuilder.js b/js/nostr/videoPayloadBuilder.js index 430dcb52..db3da3af 100644 --- a/js/nostr/videoPayloadBuilder.js +++ b/js/nostr/videoPayloadBuilder.js @@ -474,6 +474,16 @@ export function prepareVideoEditPayload(params) { contentObject.xs = finalXs; } + // Prefer edited hashtags, else preserve the base event's so an edit that omits + // the field never strips them. + const hashtags = sanitizeVideoHashtags( + updatedData?.hashtags !== undefined ? updatedData.hashtags : baseEvent.hashtags, + ); + if (hashtags.length) { + contentObject.hashtags = hashtags; + } + const hashtagTags = hashtags.map((tag) => ["t", tag]); + let metadataForTags = nip71Metadata && typeof nip71Metadata === "object" ? nip71Metadata : null; if (!metadataForTags) { @@ -489,9 +499,11 @@ export function prepareVideoEditPayload(params) { const nip71Tags = buildNip71MetadataTags(metadataForTags); - const additionalTags = storagePointer - ? [["s", storagePointer], ...nip71Tags] - : nip71Tags; + const additionalTags = [ + ...(storagePointer ? [["s", storagePointer]] : []), + ...hashtagTags, + ...nip71Tags, + ]; // A replaceable event only wins if its created_at is strictly greater than // the version it replaces (NIP-01 keeps the highest created_at; ties go to diff --git a/tests/nip71-mirror.test.mjs b/tests/nip71-mirror.test.mjs index fb93a57b..2acaea27 100644 --- a/tests/nip71-mirror.test.mjs +++ b/tests/nip71-mirror.test.mjs @@ -139,6 +139,24 @@ test("explicit short flag overrides dimension inference", async () => { assert.equal(landscapeButShort.event.kind, NIP71_SHORT_VIDEO_KIND); }); +test("mirror sources hashtags from nip71.hashtags when video.hashtags is absent", async () => { + const v = baseVideo({ hashtags: undefined, nip71: { hashtags: ["alpha", "beta"] } }); + const res = buildNip71MirrorEvent(v); + const tTags = res.event.tags.filter((t) => t[0] === "t").map((t) => t[1]).sort(); + assert.deepEqual(tTags, ["alpha", "beta"]); +}); + +test("mirror sources hashtags from raw `t` tags, dropping the internal 'video' topic", async () => { + const v = baseVideo({ + hashtags: undefined, + tags: [["t", "video"], ["t", "Gamestr"], ["t", "gamestr"], ["d", "root-123"]], + }); + const res = buildNip71MirrorEvent(v); + const tTags = res.event.tags.filter((t) => t[0] === "t").map((t) => t[1]); + assert.ok(!tTags.includes("video"), "must not leak the bitvid 'video' topic tag"); + assert.deepEqual(tTags, ["gamestr"], "deduped + lowercased"); +}); + test("deriveVideoMime maps common extensions and defaults to mp4", async () => { assert.equal(deriveVideoMime("https://x/y.webm"), "video/webm"); assert.equal(deriveVideoMime("https://x/y.m3u8?token=1"), "application/x-mpegURL"); diff --git a/tests/video-hashtags.test.mjs b/tests/video-hashtags.test.mjs index 49fe3241..a28bf1ed 100644 --- a/tests/video-hashtags.test.mjs +++ b/tests/video-hashtags.test.mjs @@ -9,6 +9,7 @@ import test from "node:test"; import { sanitizeVideoHashtags, prepareVideoPublishPayload, + prepareVideoEditPayload, } from "../js/nostr/videoPayloadBuilder.js"; import { convertEventToVideo } from "../js/nostr/nip71.js"; @@ -57,3 +58,31 @@ test("no hashtags → no content.hashtags", async () => { const content = JSON.parse(event.content); assert.equal(content.hashtags, undefined); }); + +function edit(baseHashtags, updatedData) { + return prepareVideoEditPayload({ + baseEvent: { + id: "e1", + videoRootId: "root-9", + title: "Old", + url: "https://cdn.example.com/v.mp4", + hashtags: baseHashtags, + created_at: 1000, + tags: [], + }, + originalEventStub: { id: "e1" }, + updatedData, + userPubkey: PUBKEY, + resolveEventDTag: () => "root-9", + }); +} + +test("editing WITHOUT a hashtags field preserves the existing hashtags", () => { + const { contentObject } = edit(["alpha", "beta"], { title: "New title" }); + assert.deepEqual(contentObject.hashtags, ["alpha", "beta"], "not stripped on edit"); +}); + +test("editing WITH hashtags overrides them", () => { + const { contentObject } = edit(["alpha"], { hashtags: ["gamma", "delta"] }); + assert.deepEqual(contentObject.hashtags, ["gamma", "delta"]); +}); From 382da3085b598dfc61529b25569995346dc5294e Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 11:31:17 -0400 Subject: [PATCH 18/43] docs(todo): hashtags polish done (publish+feed+mirror) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- todo/TODO_2026-06-20_pre-launch.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index 82a864c1..707c1eeb 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -272,9 +272,13 @@ notes below. Summary: event-driven via nostrService videos:edited/deleted (no nostrService growth). Edit re-publishes (or unshares if now-private); delete tears down + clears the flag. - - [ ] Remaining polish: capture dimensions at upload (short 34236 + imeta - dim — defaults to 34235 now); hashtags input → t tags (no hashtags - field yet); optional account-level "auto-share new public videos". + - [x] Hashtags (a1bf7b15, e793b359): the upload modal's existing NIP-71 + hashtag editor already emits `t` tags (feed scoring reads them); the + mirror now carries them too (from t tags / nip71.hashtags). So + publish + feed + mirror hashtags all work. + - [ ] Remaining polish (low value): capture dimensions at upload (short 34236 + + imeta dim — defaults to 34235 now); optional account-level + "auto-share new public videos". - NOTE: legacy FEATURE_PUBLISH_NIP71 (videoPublisher.js auto-21/22) stays OFF/dormant — superseded by the opt-in 34235/36 mirror; clean up later. - [ ] Phase 1.5: NIP-89 handler reg (kind 31990 → "Open in bitvid" elsewhere); From 115521928c2a8ba373515883840dfa18c680eed0 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 11:44:02 -0400 Subject: [PATCH 19/43] feat(video): capture & persist dimensions/duration at upload (enables shorts) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Auto-captures a video's width/height/duration at file-select and persists them on the 30078 — the durable foundation for a future "shorts" vertical feed (filter on height > width) and for the NIP-71 mirror's short-kind selection. - js/utils/videoProbe.js: probeVideoMetadata(src) loads metadata into a hidden <video> (element injectable for tests); resolves {width,height,duration} or null. - UploadModal: probes the selected file (objectURL) in the background and threads width/height/duration into the publish payload. - videoPayloadBuilder: persists width/height/duration in content (publish) and preserves them across edits; normalizers exported. - convertEventToVideo: exposes video.width/height/duration → the mirror builder already uses them (portrait ⇒ 34236 short + imeta dim). tests: probe resolution (incl portrait + error/timeout → null); content persist + parse; portrait⇒34236 / landscape⇒34235 mirror selection; edit preserves dims. Mutation-verified earlier on the mirror short-selection. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/nostr/nip71.js | 12 +++++ js/nostr/videoPayloadBuilder.js | 41 ++++++++++++++ js/ui/components/UploadModal.js | 32 +++++++++++ js/utils/videoProbe.js | 80 ++++++++++++++++++++++++++++ tests/video-dimensions.test.mjs | 94 +++++++++++++++++++++++++++++++++ tests/video-probe.test.mjs | 59 +++++++++++++++++++++ 6 files changed, 318 insertions(+) create mode 100644 js/utils/videoProbe.js create mode 100644 tests/video-dimensions.test.mjs create mode 100644 tests/video-probe.test.mjs diff --git a/js/nostr/nip71.js b/js/nostr/nip71.js index 069e8b43..44d74579 100644 --- a/js/nostr/nip71.js +++ b/js/nostr/nip71.js @@ -1520,6 +1520,15 @@ export function convertEventToVideo(event = {}) { .filter((value) => typeof value === "string" && value.trim()) .map((value) => value.trim()) : []; + const toPositiveInt = (value) => { + const n = Number(value); + return Number.isFinite(n) && n > 0 ? Math.floor(n) : undefined; + }; + const width = toPositiveInt(parsedContent.width); + const height = toPositiveInt(parsedContent.height); + const durationValue = Number(parsedContent.duration); + const duration = + Number.isFinite(durationValue) && durationValue > 0 ? durationValue : undefined; const videoRootId = safeTrim(parsedContent.videoRootId) || event.id; const wsField = safeTrim(parsedContent.ws); const xsField = safeTrim(parsedContent.xs); @@ -1624,6 +1633,9 @@ export function convertEventToVideo(event = {}) { mode, deleted, hashtags, + width, + height, + duration, ws, xs, storagePointer, diff --git a/js/nostr/videoPayloadBuilder.js b/js/nostr/videoPayloadBuilder.js index db3da3af..9410feed 100644 --- a/js/nostr/videoPayloadBuilder.js +++ b/js/nostr/videoPayloadBuilder.js @@ -8,6 +8,18 @@ import { inferMimeTypeFromUrl } from "../utils/mime.js"; import { normalizeHashtag } from "../utils/hashtagNormalization.js"; import { devLogger } from "../utils/logger.js"; +// Pixel dimensions: positive integer or 0 (= "unknown / omit"). +export function normalizeVideoDimension(value) { + const n = Number(value); + return Number.isFinite(n) && n > 0 ? Math.floor(n) : 0; +} + +// Duration in seconds: positive (float ok) or 0 (= omit). +export function normalizeVideoDuration(value) { + const n = Number(value); + return Number.isFinite(n) && n > 0 ? n : 0; +} + // Normalize free-form hashtag input (array or comma/space string) to a deduped, // lowercased list. Drops the reserved "video" topic tag and caps the count. export function sanitizeVideoHashtags(input) { @@ -191,6 +203,21 @@ export async function prepareVideoPublishPayload(videoPayload, pubkey, { timesta contentObject.hashtags = hashtags; } + // Captured-at-upload dimensions/duration: persist so a video's orientation is + // known durably (drives 34236 short selection + a future "shorts" feed filter). + const width = normalizeVideoDimension(videoData.width); + const height = normalizeVideoDimension(videoData.height); + const duration = normalizeVideoDuration(videoData.duration); + if (width) { + contentObject.width = width; + } + if (height) { + contentObject.height = height; + } + if (duration) { + contentObject.duration = duration; + } + const nip71Tags = buildNip71MetadataTags( nip71Metadata && typeof nip71Metadata === "object" ? nip71Metadata : null ); @@ -484,6 +511,20 @@ export function prepareVideoEditPayload(params) { } const hashtagTags = hashtags.map((tag) => ["t", tag]); + // Preserve captured dimensions/duration across edits (prefer edited values). + const width = normalizeVideoDimension(updatedData?.width ?? baseEvent.width); + const height = normalizeVideoDimension(updatedData?.height ?? baseEvent.height); + const duration = normalizeVideoDuration(updatedData?.duration ?? baseEvent.duration); + if (width) { + contentObject.width = width; + } + if (height) { + contentObject.height = height; + } + if (duration) { + contentObject.duration = duration; + } + let metadataForTags = nip71Metadata && typeof nip71Metadata === "object" ? nip71Metadata : null; if (!metadataForTags) { diff --git a/js/ui/components/UploadModal.js b/js/ui/components/UploadModal.js index f4344088..fa618431 100644 --- a/js/ui/components/UploadModal.js +++ b/js/ui/components/UploadModal.js @@ -1,6 +1,7 @@ // components/UploadModal.js import { createModalAccessibility } from "./modalAccessibility.js"; import { Nip71FormManager } from "./nip71FormManager.js"; +import { probeVideoMetadata } from "../../utils/videoProbe.js"; import { MediaUploader } from "./mediaUploader.js"; import { devLogger, userLogger } from "../../utils/logger.js"; import { @@ -483,10 +484,35 @@ export class UploadModal { // --- Immediate Upload Handlers --- + // Best-effort: read the file's dimensions/duration so we can persist them + // (orientation → 34236 short selection + a future "shorts" feed). Never blocks + // upload; failures just leave the metadata unset. + async captureVideoMetadata(file) { + this.capturedMetadata = null; + if (!file || typeof URL === "undefined" || typeof URL.createObjectURL !== "function") { + return; + } + const objectUrl = URL.createObjectURL(file); + try { + this.capturedMetadata = await probeVideoMetadata(objectUrl); + } catch (error) { + this.capturedMetadata = null; + } finally { + try { + URL.revokeObjectURL(objectUrl); + } catch (revokeError) { + // ignore + } + } + } + async handleVideoSelection(e) { const file = e.target.files?.[0]; if (!file) return; + // Probe dimensions/duration in the background while the upload proceeds. + void this.captureVideoMetadata(file); + if (!this.storageConfigured) { alert("Please configure storage before selecting a file."); e.target.value = ""; // Clear selection @@ -900,6 +926,12 @@ export class UploadModal { enableComments: this.toggles.comments?.checked || true, ...audienceFlags, + // Auto-captured at file-select (best-effort; absent ⇒ omitted). Drives + // 34236 short selection + a future "shorts" feed (height > width). + width: this.capturedMetadata?.width || 0, + height: this.capturedMetadata?.height || 0, + duration: this.capturedMetadata?.duration || 0, + // These might be empty depending on mode, filled below url: "", magnet: "", diff --git a/js/utils/videoProbe.js b/js/utils/videoProbe.js new file mode 100644 index 00000000..0fdc6bbc --- /dev/null +++ b/js/utils/videoProbe.js @@ -0,0 +1,80 @@ +// Auto-capture a video's dimensions + duration by loading just its metadata into +// a hidden <video> element. Used at upload time so bitvid can persist width/height +// (orientation → 34236 short selection + a future "shorts" feed) and duration. +// +// Element creation is injectable so the resolution logic is unit-testable without +// a DOM or a real media file. Always resolves (never rejects): null on failure. + +export function probeVideoMetadata(src, { createVideoEl, timeoutMs = 8000 } = {}) { + return new Promise((resolve) => { + const source = typeof src === "string" ? src.trim() : ""; + if (!source) { + resolve(null); + return; + } + + let el = null; + try { + el = + typeof createVideoEl === "function" + ? createVideoEl() + : typeof document !== "undefined" && document.createElement + ? document.createElement("video") + : null; + } catch (error) { + el = null; + } + if (!el || typeof el.addEventListener !== "function") { + resolve(null); + return; + } + + let settled = false; + const cleanup = () => { + try { + el.removeAttribute?.("src"); + el.load?.(); + } catch (error) { + // ignore + } + }; + const finish = (result) => { + if (settled) { + return; + } + settled = true; + clearTimeout(timer); + cleanup(); + resolve(result); + }; + + const timer = setTimeout(() => finish(null), timeoutMs); + + try { + el.preload = "metadata"; + el.muted = true; + } catch (error) { + // ignore (fakes may not allow these) + } + + el.addEventListener("loadedmetadata", () => { + const width = Number(el.videoWidth); + const height = Number(el.videoHeight); + const durationRaw = Number(el.duration); + finish({ + width: Number.isFinite(width) && width > 0 ? Math.floor(width) : 0, + height: Number.isFinite(height) && height > 0 ? Math.floor(height) : 0, + duration: Number.isFinite(durationRaw) && durationRaw > 0 ? durationRaw : 0, + }); + }); + el.addEventListener("error", () => finish(null)); + + try { + el.src = source; + } catch (error) { + finish(null); + } + }); +} + +export default probeVideoMetadata; diff --git a/tests/video-dimensions.test.mjs b/tests/video-dimensions.test.mjs new file mode 100644 index 00000000..30bf0e5d --- /dev/null +++ b/tests/video-dimensions.test.mjs @@ -0,0 +1,94 @@ +// Dimension capture: width/height/duration are persisted in the 30078 content, +// survive edits, parse back onto the video, and drive the NIP-71 mirror's +// normal(34235)-vs-short(34236) selection + imeta dim. (Foundation for a future +// "shorts" vertical feed, which filters on height > width.) + +import "./test-helpers/setup-localstorage.mjs"; +import assert from "node:assert/strict"; +import test from "node:test"; +import { + prepareVideoPublishPayload, + prepareVideoEditPayload, + normalizeVideoDimension, +} from "../js/nostr/videoPayloadBuilder.js"; +import { convertEventToVideo } from "../js/nostr/nip71.js"; +import { + buildNip71MirrorEvent, + NIP71_NORMAL_VIDEO_KIND, + NIP71_SHORT_VIDEO_KIND, +} from "../js/nostr/nip71Mirror.js"; + +const PUBKEY = "a".repeat(64); + +function publish(extra) { + return prepareVideoPublishPayload( + { title: "Hi", url: "https://cdn.example.com/v.mp4", ...extra }, + PUBKEY, + { timestamp: 1000 }, + ); +} + +test("normalizeVideoDimension: positive int or 0", () => { + assert.equal(normalizeVideoDimension(1920), 1920); + assert.equal(normalizeVideoDimension("1080"), 1080); + assert.equal(normalizeVideoDimension(0), 0); + assert.equal(normalizeVideoDimension(-5), 0); + assert.equal(normalizeVideoDimension("abc"), 0); +}); + +test("publish persists width/height/duration in content; parser exposes them", async () => { + const { event } = await publish({ + videoRootId: "root-1", + width: 1080, + height: 1920, + duration: 42.5, + }); + const content = JSON.parse(event.content); + assert.equal(content.width, 1080); + assert.equal(content.height, 1920); + assert.equal(content.duration, 42.5); + + const video = convertEventToVideo(event); + assert.equal(video.width, 1080); + assert.equal(video.height, 1920); + assert.equal(video.duration, 42.5); +}); + +test("a portrait (height>width) video mirrors as 34236 short with imeta dim", async () => { + const { event } = await publish({ videoRootId: "root-2", width: 1080, height: 1920 }); + const video = convertEventToVideo(event); + const res = buildNip71MirrorEvent({ ...video, pubkey: PUBKEY }); + assert.equal(res.event.kind, NIP71_SHORT_VIDEO_KIND, "portrait => 34236"); + const imeta = res.event.tags.find((t) => t[0] === "imeta"); + assert.ok(imeta.some((e) => e === "dim 1080x1920"), "carries imeta dim"); +}); + +test("a landscape video mirrors as 34235 normal", async () => { + const { event } = await publish({ videoRootId: "root-3", width: 1920, height: 1080 }); + const video = convertEventToVideo(event); + const res = buildNip71MirrorEvent({ ...video, pubkey: PUBKEY }); + assert.equal(res.event.kind, NIP71_NORMAL_VIDEO_KIND, "landscape => 34235"); +}); + +test("editing preserves dimensions when the field is omitted", () => { + const { contentObject } = prepareVideoEditPayload({ + baseEvent: { + id: "e1", + videoRootId: "root-9", + title: "Old", + url: "https://cdn.example.com/v.mp4", + width: 1280, + height: 720, + duration: 30, + created_at: 1000, + tags: [], + }, + originalEventStub: { id: "e1" }, + updatedData: { title: "New" }, + userPubkey: PUBKEY, + resolveEventDTag: () => "root-9", + }); + assert.equal(contentObject.width, 1280); + assert.equal(contentObject.height, 720); + assert.equal(contentObject.duration, 30); +}); diff --git a/tests/video-probe.test.mjs b/tests/video-probe.test.mjs new file mode 100644 index 00000000..e0768cc0 --- /dev/null +++ b/tests/video-probe.test.mjs @@ -0,0 +1,59 @@ +// probeVideoMetadata resolves a video's dimensions/duration from a hidden element, +// or null on error/timeout. Uses an injected fake element so it's deterministic. + +import assert from "node:assert/strict"; +import test from "node:test"; +import { probeVideoMetadata } from "../js/utils/videoProbe.js"; + +// Minimal fake <video>: fires the named event on the next tick when src is set. +function fakeVideo({ fire = "loadedmetadata", videoWidth = 0, videoHeight = 0, duration = 0 } = {}) { + const handlers = {}; + return { + videoWidth, + videoHeight, + duration, + addEventListener(name, h) { + handlers[name] = h; + }, + removeAttribute() {}, + load() {}, + set src(_v) { + setTimeout(() => handlers[fire]?.(), 0); + }, + }; +} + +test("resolves width/height/duration from loadedmetadata", async () => { + const result = await probeVideoMetadata("blob:abc", { + createVideoEl: () => fakeVideo({ videoWidth: 1080, videoHeight: 1920, duration: 12.3 }), + }); + assert.deepEqual(result, { width: 1080, height: 1920, duration: 12.3 }); +}); + +test("a portrait probe yields height > width (drives short selection)", async () => { + const r = await probeVideoMetadata("blob:abc", { + createVideoEl: () => fakeVideo({ videoWidth: 720, videoHeight: 1280 }), + }); + assert.ok(r.height > r.width); + assert.equal(r.duration, 0, "missing duration normalizes to 0"); +}); + +test("resolves null on a media error", async () => { + const r = await probeVideoMetadata("blob:bad", { + createVideoEl: () => fakeVideo({ fire: "error" }), + }); + assert.equal(r, null); +}); + +test("resolves null for an empty source and times out gracefully", async () => { + assert.equal(await probeVideoMetadata("", { createVideoEl: () => fakeVideo() }), null); + // never fires any event -> timeout path + const neverFires = { + addEventListener() {}, + removeAttribute() {}, + load() {}, + set src(_v) {}, + }; + const r = await probeVideoMetadata("blob:x", { createVideoEl: () => neverFires, timeoutMs: 20 }); + assert.equal(r, null); +}); From 685e52656b62675ff423e6c41c923a986e3ddd68 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 14:06:19 -0400 Subject: [PATCH 20/43] nip71: account-level "auto-share new public videos" toggle MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds an opt-in account preference (off by default, per pubkey) that auto-mirrors newly published eligible public videos to NIP-71 (kinds 34235/34236) so they surface on other Nostr video clients — no per-video opt-in needed. - nip71MirrorFlags: isAutoShareEnabled/setAutoShareEnabled (persisted per pubkey) + pure resolvePublishSync({featureOn,autoShare,eligible}). - nip71MirrorSync: syncNip71MirrorAfterPublish + a videos:published subscription in initNip71MirrorSync; converts the canonical 30078 event, checks eligibility (public + hosted URL + ALLOW_NSFW gate), publishes the mirror, and records the per-video flag so later edits/deletes stay in lockstep. All best-effort. - My Videos pane: an "Auto-share new public videos to other Nostr apps" checkbox wired to setAutoShareEnabled, gated by FEATURE_NIP71_MIRROR. Tests (mutation-verified): auto-share persistence, resolvePublishSync (eligible+on=>publish; off/ineligible=>none), and videos:published subscription wiring. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- components/profile-modal.html | 4 ++ js/services/nip71MirrorFlags.js | 51 ++++++++++++++++++++++++ js/services/nip71MirrorSync.js | 44 ++++++++++++++++++++ js/ui/profileModal/MyVideosController.js | 29 ++++++++++++++ tests/nip71-mirror-flags.test.mjs | 20 ++++++++++ tests/nip71-mirror-sync.test.mjs | 1 + 6 files changed, 149 insertions(+) diff --git a/components/profile-modal.html b/components/profile-modal.html index 51421cdf..1f19b708 100644 --- a/components/profile-modal.html +++ b/components/profile-modal.html @@ -1028,6 +1028,10 @@ role="status" aria-live="polite" ></p> + <label id="profileMyVideosAutoShareRow" class="hidden flex items-start gap-2 text-sm text-muted-strong"> + <input id="profileMyVideosAutoShare" type="checkbox" class="checkbox mt-0.5" /> + <span>Auto-share new public videos to other Nostr apps (NIP-71)</span> + </label> <p id="profileMyVideosLoading" class="text-sm text-muted hidden" diff --git a/js/services/nip71MirrorFlags.js b/js/services/nip71MirrorFlags.js index ebe7f95e..899e2c84 100644 --- a/js/services/nip71MirrorFlags.js +++ b/js/services/nip71MirrorFlags.js @@ -5,6 +5,7 @@ // relays. const FLAG_KEY = "bitvid:nip71-mirror:v1"; +const AUTO_SHARE_KEY = "bitvid:nip71-autoshare:v1"; function readFlags() { try { @@ -74,6 +75,56 @@ export function resolveMirrorToggle({ enabled, eligibility } = {}) { return { action: "publish" }; } +// Account-level "auto-share new public videos" preference (per pubkey, off by +// default). When on, newly published eligible public videos are mirrored +// automatically — no per-video opt-in needed. +function readAutoShare() { + try { + if (typeof localStorage === "undefined") { + return {}; + } + const raw = localStorage.getItem(AUTO_SHARE_KEY); + const parsed = raw ? JSON.parse(raw) : {}; + return parsed && typeof parsed === "object" ? parsed : {}; + } catch (error) { + return {}; + } +} + +export function isAutoShareEnabled(pubkey) { + const pk = norm(pubkey); + return pk ? readAutoShare()[pk] === true : false; +} + +export function setAutoShareEnabled(pubkey, enabled) { + const pk = norm(pubkey); + if (!pk) { + return; + } + const map = readAutoShare(); + if (enabled === true) { + map[pk] = true; + } else { + delete map[pk]; + } + try { + if (typeof localStorage !== "undefined") { + localStorage.setItem(AUTO_SHARE_KEY, JSON.stringify(map)); + } + } catch (error) { + // best-effort + } +} + +// On publish of a NEW video: mirror it only when auto-share is on AND it's +// eligible (public + hosted URL + allowed). Otherwise do nothing. +export function resolvePublishSync({ featureOn, autoShare, eligible } = {}) { + if (featureOn !== true || autoShare !== true) { + return { action: "none" }; + } + return eligible === true ? { action: "publish" } : { action: "none" }; +} + // Lifecycle decisions for keeping an opted-in mirror in sync (used by the // nostrService edit/delete hooks). Pure so they're cheat-resistant to test. // diff --git a/js/services/nip71MirrorSync.js b/js/services/nip71MirrorSync.js index 853e0ae2..9091afa0 100644 --- a/js/services/nip71MirrorSync.js +++ b/js/services/nip71MirrorSync.js @@ -6,14 +6,54 @@ import { FEATURE_NIP71_MIRROR } from "../constants.js"; import { nip71MirrorService } from "./nip71MirrorService.js"; +import { convertEventToVideo } from "../nostr/index.js"; import { isMirrorEnabled, setMirrorEnabled, + isAutoShareEnabled, resolveEditSync, resolveDeleteSync, + resolvePublishSync, } from "./nip71MirrorFlags.js"; import { devLogger } from "../utils/logger.js"; +// On publish of a NEW video, auto-mirror it when the account opted into +// "auto-share new public videos" and it's eligible. Sets the per-video flag so +// later edits/deletes keep it in sync. Best-effort. +export async function syncNip71MirrorAfterPublish(detail = {}) { + const pubkey = typeof detail?.pubkey === "string" ? detail.pubkey : ""; + const legacyEvent = detail?.result?.legacy; + if (!pubkey || !legacyEvent || typeof legacyEvent !== "object") { + return; + } + let video; + try { + video = convertEventToVideo(legacyEvent); + } catch (error) { + return; + } + if (!video || video.invalid || !video.videoRootId) { + return; + } + const withPubkey = { ...video, pubkey }; + const decision = resolvePublishSync({ + featureOn: FEATURE_NIP71_MIRROR, + autoShare: isAutoShareEnabled(pubkey), + eligible: nip71MirrorService.canMirror(withPubkey).ok, + }); + if (decision.action !== "publish") { + return; + } + try { + const result = await nip71MirrorService.publish(withPubkey); + if (result?.ok) { + setMirrorEnabled(pubkey, video.videoRootId, true); + } + } catch (error) { + devLogger.warn("[nip71MirrorSync] auto-share publish failed", error); + } +} + export async function syncNip71MirrorAfterEdit({ updatedData, pubkey } = {}) { if (!updatedData || typeof updatedData !== "object") { return; @@ -61,10 +101,14 @@ export function initNip71MirrorSync(nostrService) { const offDelete = nostrService.on("videos:deleted", ({ videoRootId, video, pubkey } = {}) => { void syncNip71MirrorAfterDelete({ videoRootId, video, pubkey }); }); + const offPublish = nostrService.on("videos:published", (detail) => { + void syncNip71MirrorAfterPublish(detail); + }); return () => { registered = false; if (typeof offEdit === "function") offEdit(); if (typeof offDelete === "function") offDelete(); + if (typeof offPublish === "function") offPublish(); }; } diff --git a/js/ui/profileModal/MyVideosController.js b/js/ui/profileModal/MyVideosController.js index da22c043..2ef0992f 100644 --- a/js/ui/profileModal/MyVideosController.js +++ b/js/ui/profileModal/MyVideosController.js @@ -10,6 +10,8 @@ import { isMirrorEnabled, setMirrorEnabled, resolveMirrorToggle, + isAutoShareEnabled, + setAutoShareEnabled, } from "../../services/nip71MirrorFlags.js"; const MIRROR_REASON_TEXT = { @@ -48,6 +50,8 @@ export class MyVideosController { this.refreshBtn = null; this.orphansSection = null; this.orphanListEl = null; + this.autoShareRow = null; + this.autoShareInput = null; this.loading = false; this.publicBaseUrl = ""; this.pubkey = ""; @@ -63,6 +67,10 @@ export class MyVideosController { document.getElementById("profileMyVideosOrphans") || null; this.orphanListEl = document.getElementById("profileMyVideosOrphanList") || null; + this.autoShareRow = + document.getElementById("profileMyVideosAutoShareRow") || null; + this.autoShareInput = + document.getElementById("profileMyVideosAutoShare") || null; } registerEventListeners() { @@ -71,6 +79,26 @@ export class MyVideosController { void this.populate({ forceFetch: true }); }); } + if (this.autoShareInput instanceof HTMLInputElement) { + this.autoShareInput.addEventListener("change", () => { + const pubkey = this.getPubkey(); + if (!pubkey) { + return; + } + setAutoShareEnabled(pubkey, this.autoShareInput.checked === true); + }); + } + } + + refreshAutoShareToggle(pubkey) { + if (!(this.autoShareRow instanceof HTMLElement)) { + return; + } + const eligible = FEATURE_NIP71_MIRROR && !!pubkey; + this.autoShareRow.classList.toggle("hidden", !eligible); + if (eligible && this.autoShareInput instanceof HTMLInputElement) { + this.autoShareInput.checked = isAutoShareEnabled(pubkey); + } } getPubkey() { @@ -114,6 +142,7 @@ export class MyVideosController { return; } const pubkey = this.getPubkey(); + this.refreshAutoShareToggle(pubkey); if (!pubkey) { this.renderRows([]); this.setSummary(""); diff --git a/tests/nip71-mirror-flags.test.mjs b/tests/nip71-mirror-flags.test.mjs index f3d36245..62318e16 100644 --- a/tests/nip71-mirror-flags.test.mjs +++ b/tests/nip71-mirror-flags.test.mjs @@ -10,6 +10,9 @@ import { resolveMirrorToggle, resolveEditSync, resolveDeleteSync, + isAutoShareEnabled, + setAutoShareEnabled, + resolvePublishSync, } from "../js/services/nip71MirrorFlags.js"; const PK = "a".repeat(64); @@ -72,3 +75,20 @@ test("resolveDeleteSync: removes the mirror only when it was shared", () => { assert.deepEqual(resolveDeleteSync({ featureOn: true, enabled: false }), { action: "none" }); assert.deepEqual(resolveDeleteSync({ featureOn: false, enabled: true }), { action: "none" }); }); + +test("auto-share preference persists per pubkey, off by default", () => { + localStorage.clear(); + assert.equal(isAutoShareEnabled(PK), false); + setAutoShareEnabled(PK, true); + assert.equal(isAutoShareEnabled(PK), true); + assert.equal(isAutoShareEnabled("c".repeat(64)), false, "scoped per pubkey"); + setAutoShareEnabled(PK, false); + assert.equal(isAutoShareEnabled(PK), false); +}); + +test("resolvePublishSync: auto-mirror a new video only when auto-share on AND eligible", () => { + assert.deepEqual(resolvePublishSync({ featureOn: true, autoShare: true, eligible: true }), { action: "publish" }); + assert.deepEqual(resolvePublishSync({ featureOn: true, autoShare: true, eligible: false }), { action: "none" }); + assert.deepEqual(resolvePublishSync({ featureOn: true, autoShare: false, eligible: true }), { action: "none" }); + assert.deepEqual(resolvePublishSync({ featureOn: false, autoShare: true, eligible: true }), { action: "none" }); +}); diff --git a/tests/nip71-mirror-sync.test.mjs b/tests/nip71-mirror-sync.test.mjs index 9bca3c29..f76d210e 100644 --- a/tests/nip71-mirror-sync.test.mjs +++ b/tests/nip71-mirror-sync.test.mjs @@ -37,6 +37,7 @@ test("initNip71MirrorSync subscribes to edit+delete events, idempotently", () => const off = initNip71MirrorSync(svc); assert.equal(events["videos:edited"]?.length, 1, "subscribes to videos:edited"); assert.equal(events["videos:deleted"]?.length, 1, "subscribes to videos:deleted"); + assert.equal(events["videos:published"]?.length, 1, "subscribes to videos:published"); initNip71MirrorSync(svc); // guard prevents double-registration assert.equal(events["videos:edited"].length, 1, "no double-register"); From e418e729d646e195f721bf23119b9554b92ddc3c Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 15:00:01 -0400 Subject: [PATCH 21/43] nip71: add FEATURE_NIP71_INGEST admin toggle (default on) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Foundation for inbound NIP-71 ingest. Adds a single admin switch in the instance config (config/instance-config.js, default true), re-exported through config.js and surfaced as a runtime-overridable flag in constants.js — same pattern as FEATURE_NIP71_MIRROR / ALLOW_NSFW_CONTENT. A deployer flips one constant to disable all inbound ingest if it ever causes problems (spam, low-quality content, performance). The ingest adapter + service (gated on this flag) land in follow-up commits; its behavioral test will assert the service is inert when the flag is off. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- config/instance-config.js | 15 +++++++++++++++ js/config.js | 2 ++ js/constants.js | 10 ++++++++++ 3 files changed, 27 insertions(+) diff --git a/config/instance-config.js b/config/instance-config.js index 49e36ea6..6a22e0ef 100644 --- a/config/instance-config.js +++ b/config/instance-config.js @@ -234,6 +234,21 @@ export const DEFAULT_WHITELIST_MODE_ENABLED = true; */ export const ALLOW_NSFW_CONTENT = false; +/** + * Whether the client ingests NIP-71 video events published by *other* Nostr + * apps (kinds 21/22 and addressable 34235/34236) and surfaces them in bitvid's + * feeds. + * + * Defaults to `true` so bitvid discovers videos from the wider Nostr ecosystem + * (Nostube, Flotilla, etc.). bitvid's own outbound mirrors are skipped to avoid + * duplicate listings, and ingested NSFW content still respects + * `ALLOW_NSFW_CONTENT` plus the standard trust/blacklist/mute gating. + * + * Set to `false` to disable all inbound NIP-71 ingest with a single switch if it + * ever causes problems (spam, low-quality content, performance). + */ +export const FEATURE_NIP71_INGEST = true; + /** * Emergency fallback accounts for seeding the moderation graph. * diff --git a/js/config.js b/js/config.js index 4f9e6b98..d4e20286 100644 --- a/js/config.js +++ b/js/config.js @@ -24,6 +24,7 @@ import { ADMIN_WHITELIST_MODE_STORAGE_KEY, DEFAULT_WHITELIST_MODE_ENABLED, ALLOW_NSFW_CONTENT, + FEATURE_NIP71_INGEST, DEFAULT_TRUST_SEED_NPUBS, DEFAULT_BLUR_THRESHOLD, DEFAULT_AUTOPLAY_BLOCK_THRESHOLD, @@ -117,6 +118,7 @@ export const ADMIN_LIST_NAMESPACE = "bitvid:admin"; // Reserved for Nostr lists export const ADMIN_COMMUNITY_BLACKLIST_SOURCES = "community:blacklist:sources"; export const ADMIN_COMMUNITY_BLACKLIST_PREFIX = "community:blacklist"; export { ALLOW_NSFW_CONTENT }; +export { FEATURE_NIP71_INGEST }; export { DEFAULT_TRUST_SEED_NPUBS }; export { DEFAULT_BLUR_THRESHOLD }; export { DEFAULT_AUTOPLAY_BLOCK_THRESHOLD }; diff --git a/js/constants.js b/js/constants.js index f6a9c202..63d2b534 100644 --- a/js/constants.js +++ b/js/constants.js @@ -6,6 +6,7 @@ import { DEFAULT_TRUSTED_SPAM_HIDE_THRESHOLD as CONFIG_DEFAULT_TRUSTED_SPAM_HIDE_THRESHOLD, DEFAULT_PLAYBACK_SOURCE, DEFAULT_PLAYBACK_START_TIMEOUT, + FEATURE_NIP71_INGEST as CONFIG_FEATURE_NIP71_INGEST, IS_DEV_MODE, } from "./config.js"; @@ -185,6 +186,9 @@ const DEFAULT_FLAGS = Object.freeze({ // Opt-in NIP-71 *mirror* (addressable 34235/36) managed per-video from the // My Videos tab. Independent of FEATURE_PUBLISH_NIP71 (legacy 21/22 auto-publish). FEATURE_NIP71_MIRROR: true, + // Inbound ingest of NIP-71 videos published by other Nostr apps. Sourced from + // the admin instance config so a deployer flips it in one place. + FEATURE_NIP71_INGEST: CONFIG_FEATURE_NIP71_INGEST !== false, FEATURE_HASHTAG_PREFERENCES: false, FEATURE_SEARCH_FILTERS: Boolean(IS_DEV_MODE), FEATURE_TRUST_SEEDS: true, // Rollback: disable to drop baseline trust seeds without code changes. @@ -206,6 +210,7 @@ const runtimeFlags = (() => { FEATURE_WATCH_HISTORY_V2: DEFAULT_FLAGS.FEATURE_WATCH_HISTORY_V2, FEATURE_PUBLISH_NIP71: DEFAULT_FLAGS.FEATURE_PUBLISH_NIP71, FEATURE_NIP71_MIRROR: DEFAULT_FLAGS.FEATURE_NIP71_MIRROR, + FEATURE_NIP71_INGEST: DEFAULT_FLAGS.FEATURE_NIP71_INGEST, FEATURE_HASHTAG_PREFERENCES: DEFAULT_FLAGS.FEATURE_HASHTAG_PREFERENCES, FEATURE_SEARCH_FILTERS: DEFAULT_FLAGS.FEATURE_SEARCH_FILTERS, FEATURE_TRUST_SEEDS: DEFAULT_FLAGS.FEATURE_TRUST_SEEDS, @@ -242,6 +247,11 @@ export let FEATURE_NIP71_MIRROR = coerceBoolean( DEFAULT_FLAGS.FEATURE_NIP71_MIRROR ); +export let FEATURE_NIP71_INGEST = coerceBoolean( + runtimeFlags.FEATURE_NIP71_INGEST, + DEFAULT_FLAGS.FEATURE_NIP71_INGEST +); + export let FEATURE_HASHTAG_PREFERENCES = coerceBoolean( runtimeFlags.FEATURE_HASHTAG_PREFERENCES, DEFAULT_FLAGS.FEATURE_HASHTAG_PREFERENCES From 286621dd4ad2cb0f0ae816cb535ada6e1291967d Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 15:02:11 -0400 Subject: [PATCH 22/43] nip71: foreign event -> bitvid video ingest adapter buildVideoFromNip71Event() converts a NIP-71 video event published by another Nostr app (kinds 21/22/34235/34236) into a bitvid video object shaped like convertEventToVideo's output, so ingested videos flow through the existing feed pipeline (moderation, trust, NSFW gating, rendering). Reads the imeta variant + standard NIP-71 tags (not bitvid's JSON content): url/magnet, dim->width/height, image->thumbnail, x/ox hashes, i/magnet->infoHash, title, summary->description, t->hashtags, duration. Safety: - content-warning -> isNsfw so ALLOW_NSFW_CONTENT + blur thresholds apply. - skips bitvid's own outbound mirrors (["client","bitvid"]) to avoid double-listing content already present as canonical kind-30078. - never marks foreign content private (no encryption in NIP-71). - stamps source:"nip71-ingest"/foreign:true provenance markers. Pure + dependency-free; mutation-verified (mirror-skip, nsfw mapping). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/nostr/nip71IngestAdapter.js | 195 ++++++++++++++++++++++++++++ tests/nip71-ingest-adapter.test.mjs | 142 ++++++++++++++++++++ 2 files changed, 337 insertions(+) create mode 100644 js/nostr/nip71IngestAdapter.js create mode 100644 tests/nip71-ingest-adapter.test.mjs diff --git a/js/nostr/nip71IngestAdapter.js b/js/nostr/nip71IngestAdapter.js new file mode 100644 index 00000000..fe308d79 --- /dev/null +++ b/js/nostr/nip71IngestAdapter.js @@ -0,0 +1,195 @@ +// Adapter: converts a *foreign* NIP-71 video event (published by another Nostr +// app — kinds 21/22 regular, 34235/34236 addressable) into a bitvid video +// object shaped like the output of convertEventToVideo, so it can flow through +// the existing feed pipeline (moderation, trust, NSFW gating, rendering). +// +// This is the inverse of nip71Mirror.js (which builds bitvid → NIP-71). It reads +// the imeta variant + standard NIP-71 tags rather than bitvid's JSON content, +// and maps the `content-warning` tag to bitvid's `isNsfw` flag. +// +// Pure + dependency-free so it is cheat-resistant to test. The ingest *service* +// (nip71IngestService.js) handles subscriptions, dedup, and own-mirror skipping. + +import { extractNip71MetadataFromTags } from "./nip71.js"; +import { extractBtihFromMagnet, extractMagnetHints } from "../magnetShared.js"; + +const NIP71_KINDS = new Set([21, 22, 34235, 34236]); + +function trimString(value) { + return typeof value === "string" ? value.trim() : ""; +} + +function parseDim(dim) { + const raw = trimString(dim); + const match = /^(\d+)x(\d+)$/i.exec(raw); + if (!match) { + return { width: undefined, height: undefined }; + } + const width = Number(match[1]); + const height = Number(match[2]); + return { + width: Number.isFinite(width) && width > 0 ? width : undefined, + height: Number.isFinite(height) && height > 0 ? height : undefined, + }; +} + +function dTagFromEvent(event) { + const tags = Array.isArray(event?.tags) ? event.tags : []; + for (const tag of tags) { + if (Array.isArray(tag) && tag[0] === "d") { + const value = trimString(tag[1]); + if (value) { + return value; + } + } + } + return ""; +} + +// bitvid stamps its own outbound mirrors with ["client","bitvid"]. The ingest +// service uses this to avoid re-listing our own content (which already exists as +// a canonical kind-30078 video). +export function isBitvidMirrorEvent(event) { + const tags = Array.isArray(event?.tags) ? event.tags : []; + return tags.some( + (tag) => + Array.isArray(tag) && + tag[0] === "client" && + trimString(tag[1]).toLowerCase() === "bitvid", + ); +} + +function pickPrimaryVariant(imeta) { + if (!Array.isArray(imeta) || !imeta.length) { + return null; + } + // Prefer a variant that advertises a video mimetype and a url; fall back to + // the first variant that has any playable source. + const withVideoUrl = imeta.find( + (variant) => + trimString(variant?.url) && + trimString(variant?.m).toLowerCase().startsWith("video/"), + ); + if (withVideoUrl) { + return withVideoUrl; + } + const withUrl = imeta.find((variant) => trimString(variant?.url)); + if (withUrl) { + return withUrl; + } + return imeta.find((variant) => trimString(variant?.magnet)) || null; +} + +/** + * Convert a foreign NIP-71 event into a bitvid video object. + * + * @param {object} event - The raw Nostr event (kind 21/22/34235/34236). + * @returns {object} A video object (invalid:false) or { invalid:true, reason }. + */ +export function buildVideoFromNip71Event(event = {}) { + if (!event || typeof event !== "object") { + return { invalid: true, reason: "missing event" }; + } + if (!NIP71_KINDS.has(Number(event.kind))) { + return { id: event.id, invalid: true, reason: "not a NIP-71 video kind" }; + } + if (isBitvidMirrorEvent(event)) { + // Our own outbound mirror — the canonical 30078 is already in the feed. + return { id: event.id, invalid: true, reason: "bitvid-mirror" }; + } + + const parsed = extractNip71MetadataFromTags(event); + const metadata = parsed?.metadata || {}; + const variant = pickPrimaryVariant(metadata.imeta); + + const url = trimString(variant?.url); + const rawMagnet = trimString(variant?.magnet); + const magnet = rawMagnet.toLowerCase().startsWith("magnet:?") ? rawMagnet : ""; + + if (!url && !magnet) { + return { id: event.id, invalid: true, reason: "missing playable source" }; + } + + const title = trimString(metadata.title); + if (!title) { + return { id: event.id, invalid: true, reason: "missing title" }; + } + + const { width, height } = parseDim(variant?.dim); + + const durationCandidate = Number(metadata.duration); + const duration = + Number.isFinite(durationCandidate) && durationCandidate > 0 + ? durationCandidate + : undefined; + + const thumbnail = + Array.isArray(variant?.image) && variant.image.length + ? trimString(variant.image[0]) + : ""; + + // content-warning → isNsfw so the standard NSFW gate (ALLOW_NSFW_CONTENT) and + // blur thresholds apply to ingested content too. + const isNsfw = Boolean(trimString(metadata.contentWarning)); + + const hashtags = Array.isArray(metadata.hashtags) + ? metadata.hashtags + .map((value) => trimString(value)) + .filter((value) => value) + : []; + + let infoHash = ""; + const iField = trimString(variant?.i).toLowerCase(); + if (/^[0-9a-f]{40}$/.test(iField)) { + infoHash = iField; + } else if (magnet) { + infoHash = extractBtihFromMagnet(magnet) || ""; + } + + const hints = magnet ? extractMagnetHints(magnet) : { ws: "", xs: "" }; + + // Addressable kinds (34235/36) are keyed by d-tag; regular kinds (21/22) use + // the event id as the stable root. + const dTag = dTagFromEvent(event); + const videoRootId = dTag || trimString(event.id); + + return { + id: event.id, + videoRootId, + version: 3, + isPrivate: false, + isNsfw, + isForKids: false, + title, + url, + magnet, + rawMagnet: magnet ? rawMagnet : "", + infoHash, + fileSha256: trimString(variant?.x), + originalFileSha256: trimString(variant?.ox), + thumbnail, + description: trimString(metadata.summary), + mode: "live", + deleted: false, + hashtags, + width, + height, + duration, + ws: hints.ws || "", + xs: hints.xs || "", + storagePointer: null, + infoJsonUrl: "", + enableComments: true, + pubkey: trimString(event.pubkey), + created_at: Number.isFinite(event.created_at) ? event.created_at : 0, + tags: Array.isArray(event.tags) ? event.tags : [], + // Provenance markers so the rest of the app can distinguish ingested videos + // from native bitvid uploads (e.g. to hide WebTorrent-only affordances). + source: "nip71-ingest", + foreign: true, + nip71Kind: Number(event.kind), + invalid: false, + }; +} + +export { NIP71_KINDS }; diff --git a/tests/nip71-ingest-adapter.test.mjs b/tests/nip71-ingest-adapter.test.mjs new file mode 100644 index 00000000..c1aaa68f --- /dev/null +++ b/tests/nip71-ingest-adapter.test.mjs @@ -0,0 +1,142 @@ +// Foreign NIP-71 video event -> bitvid video object adapter. Scenario-style: +// asserts the externally observable shape a feed item needs, and the safety +// behaviors (own-mirror skip, content-warning -> isNsfw, source provenance). + +import assert from "node:assert/strict"; +import test from "node:test"; +import { + buildVideoFromNip71Event, + isBitvidMirrorEvent, +} from "../js/nostr/nip71IngestAdapter.js"; + +const SHA = "a".repeat(64); + +function foreignEvent(overrides = {}) { + return { + id: "evt-1", + pubkey: "pk-foreign", + kind: 21, + created_at: 1_700_000_000, + content: "A description from another client", + tags: [ + ["title", "Foreign Video"], + [ + "imeta", + "url https://example.com/v.mp4", + "m video/mp4", + "dim 1920x1080", + "image https://example.com/thumb.jpg", + `x ${SHA}`, + ], + ["t", "music"], + ["duration", "240"], + ], + ...overrides, + }; +} + +test("maps a foreign NIP-71 event to a playable bitvid video", () => { + const v = buildVideoFromNip71Event(foreignEvent()); + assert.equal(v.invalid, false); + assert.equal(v.title, "Foreign Video"); + assert.equal(v.url, "https://example.com/v.mp4"); + assert.equal(v.thumbnail, "https://example.com/thumb.jpg"); + assert.equal(v.description, "A description from another client"); + assert.equal(v.width, 1920); + assert.equal(v.height, 1080); + assert.equal(v.duration, 240); + assert.deepEqual(v.hashtags, ["music"]); + assert.equal(v.fileSha256, SHA); + assert.equal(v.pubkey, "pk-foreign"); + // Provenance markers so the app can distinguish ingested content. + assert.equal(v.source, "nip71-ingest"); + assert.equal(v.foreign, true); + assert.equal(v.nip71Kind, 21); + // Never treat foreign content as private (no encryption in NIP-71). + assert.equal(v.isPrivate, false); +}); + +test("content-warning maps to isNsfw so the NSFW gate applies", () => { + const clean = buildVideoFromNip71Event(foreignEvent()); + assert.equal(clean.isNsfw, false); + + const flagged = buildVideoFromNip71Event( + foreignEvent({ + tags: [ + ["title", "Spicy"], + ["imeta", "url https://example.com/v.mp4", "m video/mp4"], + ["content-warning", "explicit"], + ], + }), + ); + assert.equal(flagged.isNsfw, true); +}); + +test("addressable kinds key the root on the d-tag; regular kinds on the event id", () => { + const regular = buildVideoFromNip71Event(foreignEvent({ id: "regular-id", kind: 22 })); + assert.equal(regular.videoRootId, "regular-id"); + + const addressable = buildVideoFromNip71Event( + foreignEvent({ + id: "addr-id", + kind: 34235, + tags: [ + ["d", "stable-root"], + ["title", "Addr"], + ["imeta", "url https://example.com/v.mp4", "m video/mp4"], + ], + }), + ); + assert.equal(addressable.videoRootId, "stable-root"); +}); + +test("skips bitvid's own outbound mirrors (avoids duplicate listings)", () => { + const mirror = foreignEvent({ + kind: 34235, + tags: [ + ["d", "root"], + ["client", "bitvid"], + ["title", "Our mirror"], + ["imeta", "url https://example.com/v.mp4", "m video/mp4"], + ], + }); + assert.equal(isBitvidMirrorEvent(mirror), true); + const v = buildVideoFromNip71Event(mirror); + assert.equal(v.invalid, true); + assert.equal(v.reason, "bitvid-mirror"); +}); + +test("rejects events with no playable source or no title", () => { + const noSource = buildVideoFromNip71Event( + foreignEvent({ tags: [["title", "No media"]] }), + ); + assert.equal(noSource.invalid, true); + assert.equal(noSource.reason, "missing playable source"); + + const noTitle = buildVideoFromNip71Event( + foreignEvent({ tags: [["imeta", "url https://example.com/v.mp4", "m video/mp4"]] }), + ); + assert.equal(noTitle.invalid, true); + assert.equal(noTitle.reason, "missing title"); +}); + +test("rejects non-NIP-71 kinds", () => { + const v = buildVideoFromNip71Event(foreignEvent({ kind: 30078 })); + assert.equal(v.invalid, true); + assert.equal(v.reason, "not a NIP-71 video kind"); +}); + +test("extracts a webtorrent magnet + infohash when present", () => { + const magnet = `magnet:?xt=urn:btih:${"b".repeat(40)}&dn=clip`; + const v = buildVideoFromNip71Event( + foreignEvent({ + tags: [ + ["title", "Torrent clip"], + ["imeta", `magnet ${magnet}`, "m video/mp4", `i ${"c".repeat(40)}`], + ], + }), + ); + assert.equal(v.invalid, false); + assert.equal(v.magnet, magnet); + assert.equal(v.infoHash, "c".repeat(40), "prefers the imeta `i` infohash field"); +}); From c8499693124485d131840e73c94c30c6cd7e9e08 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 15:26:06 -0400 Subject: [PATCH 23/43] nip71: inbound ingest service (whitelist-scoped, default on) Pulls NIP-71 video events published by other Nostr apps (kinds 21/22/34235/34236), converts them via the ingest adapter, and injects them into nostrClient.activeMap/allEvents so they flow through the existing feed pipeline. Gated by FEATURE_NIP71_INGEST. Mirrors bitvid's whitelist model: since the whitelist is "authors allowed to publish/render here," ingest scopes its relay subscription to whitelisted pubkeys while whitelist mode is on (bounded, no firehose) and re-subscribes when the whitelist changes. When a deployer disables whitelist mode, ingest opens to all authors (capped by a limit). Foreign content is still gated by the same render-time filter (shouldIncludeVideo -> accessControl.canAccess): whitelist, blacklist, NSFW (content-warning -> isNsfw), private-skip, plus trust/mute/blur from the feed engine. Native bitvid videos always win a shared root; among ingested videos newest wins. Refreshes via the existing videos:updated reactive path. - accessControl.getWhitelistPubkeys(): hex whitelist for subscription scope. - Wired in applicationBootstrap (start on init, stop on teardown). - nostrService untouched (at its size cap); logic is a standalone service. Mutation-verified: native-wins, whitelist scoping, feature gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/accessControl.js | 6 + js/services/nip71IngestService.js | 254 ++++++++++++++++++++++++++++ js/ui/applicationBootstrap.js | 19 +++ tests/nip71-ingest-service.test.mjs | 219 ++++++++++++++++++++++++ 4 files changed, 498 insertions(+) create mode 100644 js/services/nip71IngestService.js create mode 100644 tests/nip71-ingest-service.test.mjs diff --git a/js/accessControl.js b/js/accessControl.js index 67cc834b..0d31e511 100644 --- a/js/accessControl.js +++ b/js/accessControl.js @@ -492,6 +492,12 @@ class AccessControl { return Array.from(this.whitelist); } + // Hex-encoded whitelist pubkeys (resolved from npubs). Used by NIP-71 ingest + // to scope its relay subscription to authors this instance already allows. + getWhitelistPubkeys() { + return Array.from(this.whitelistPubkeys); + } + getBlacklist() { return Array.from(this.blacklist); } diff --git a/js/services/nip71IngestService.js b/js/services/nip71IngestService.js new file mode 100644 index 00000000..2fe701ff --- /dev/null +++ b/js/services/nip71IngestService.js @@ -0,0 +1,254 @@ +// Inbound NIP-71 ingest: subscribes to video events published by *other* Nostr +// apps (kinds 21/22/34235/34236), converts them to bitvid video objects, and +// injects them into the active-video store so they flow through the existing +// feed pipeline (whitelist, blacklist, NSFW gate, trust/mute/blur, rendering). +// +// Design — ingest mirrors the whitelist model: bitvid's whitelist is "authors +// allowed to publish/render here," so ingest pulls those same authors' NIP-71 +// content from the wider ecosystem. When whitelist mode is ON, the relay +// subscription is scoped to whitelisted pubkeys (bounded, no firehose); when a +// deployer disables whitelist mode, ingest opens to all authors (capped). +// +// All gating is reused, not re-implemented: injected videos carry +// source:"nip71-ingest" + isNsfw (mapped from content-warning), and the +// render-time filter (nostrService.shouldIncludeVideo -> accessControl.canAccess) +// remains the authority. Gated by FEATURE_NIP71_INGEST so a single admin switch +// disables it entirely. + +import { FEATURE_NIP71_INGEST } from "../constants.js"; +import { + buildVideoFromNip71Event, + NIP71_KINDS, +} from "../nostr/nip71IngestAdapter.js"; +import { devLogger } from "../utils/logger.js"; + +const FLUSH_DEBOUNCE_MS = 200; +const DEFAULT_LIMIT = 200; +const SUBSCRIPTION_KEY = "nip71-ingest"; + +export function createNip71IngestService({ + nostrClient, + nostrService, + accessControl, + featureEnabled = FEATURE_NIP71_INGEST, + logger = devLogger, + flushDelayMs = FLUSH_DEBOUNCE_MS, + limit = DEFAULT_LIMIT, +} = {}) { + let subscription = null; + let buffer = []; + let flushTimer = null; + let offWhitelistChange = null; + let started = false; + + function isAvailable() { + return ( + featureEnabled === true && + !!nostrClient && + typeof nostrClient.getSubscriptionManager === "function" + ); + } + + // Build the relay filter set. Returns null when there is nothing to ingest + // (whitelist mode on but no whitelisted authors yet). + function resolveFilters() { + const kinds = Array.from(NIP71_KINDS); + const whitelistOn = accessControl?.whitelistMode?.() === true; + if (whitelistOn) { + const authors = + typeof accessControl?.getWhitelistPubkeys === "function" + ? accessControl.getWhitelistPubkeys().filter(Boolean) + : []; + if (!authors.length) { + return null; + } + return [{ kinds, authors, limit }]; + } + return [{ kinds, limit }]; + } + + function scheduleFlush() { + if (flushTimer) { + return; + } + flushTimer = setTimeout(() => { + flushTimer = null; + flush(); + }, flushDelayMs); + } + + // Inject a converted foreign video into the active store. Native bitvid videos + // always win for a given root; among ingested videos, newest wins. + function injectVideo(video) { + if (!video || video.invalid) { + return false; + } + const activeMap = nostrClient?.activeMap; + const allEvents = nostrClient?.allEvents; + if (!(activeMap instanceof Map) || !(allEvents instanceof Map)) { + return false; + } + const key = + typeof nostrClient.getActiveKey === "function" + ? nostrClient.getActiveKey(video) + : ""; + if (!key) { + return false; + } + const existing = activeMap.get(key); + if (existing) { + const existingForeign = existing.source === "nip71-ingest"; + if (!existingForeign) { + return false; // native bitvid video wins + } + if (Number(existing.created_at) >= Number(video.created_at)) { + return false; // older or equal ingest + } + } + activeMap.set(key, video); + allEvents.set(video.id, video); + return true; + } + + function flush() { + if (!buffer.length) { + return 0; + } + const events = buffer; + buffer = []; + + // Collapse to newest-per-root within the batch before injecting. + const newestByRoot = new Map(); + for (const event of events) { + const video = buildVideoFromNip71Event(event); + if (video.invalid || !video.videoRootId) { + continue; + } + const prev = newestByRoot.get(video.videoRootId); + if (!prev || Number(video.created_at) > Number(prev.created_at)) { + newestByRoot.set(video.videoRootId, video); + } + } + + let injected = 0; + for (const video of newestByRoot.values()) { + if (injectVideo(video)) { + injected += 1; + } + } + + if (injected > 0) { + // Re-render through the existing reactive path. The payload is the + // post-filter active list (whitelist/NSFW/etc. already applied); the feed + // re-reads getFilteredActiveVideos itself, so this just signals "refresh". + let videos = []; + try { + videos = + typeof nostrService?.getFilteredActiveVideos === "function" + ? nostrService.getFilteredActiveVideos() + : []; + } catch (error) { + logger?.warn?.("[nip71Ingest] getFilteredActiveVideos failed", error); + } + if (typeof nostrService?.emit === "function") { + nostrService.emit("videos:updated", { + videos, + deleted: [], + reason: "nip71-ingest", + }); + } + } + + return injected; + } + + function start() { + if (started || !isAvailable()) { + return false; + } + started = true; + + // Re-subscribe when the whitelist changes so the author scope stays current. + if (typeof accessControl?.onWhitelistChange === "function") { + offWhitelistChange = accessControl.onWhitelistChange(() => { + restart(); + }); + } + + return openSubscription(); + } + + function openSubscription() { + const filters = resolveFilters(); + if (!filters) { + return false; // nothing to ingest (e.g. empty whitelist) — stay armed for re-subscribe + } + try { + const manager = nostrClient.getSubscriptionManager(); + subscription = manager.subscribe({ + key: SUBSCRIPTION_KEY, + filters, + label: "nip71-ingest", + onEvent: (event) => { + if (event && NIP71_KINDS.has(Number(event.kind))) { + buffer.push(event); + scheduleFlush(); + } + }, + }); + return true; + } catch (error) { + logger?.warn?.("[nip71Ingest] subscribe failed", error); + subscription = null; + return false; + } + } + + function closeSubscription() { + if (flushTimer) { + clearTimeout(flushTimer); + flushTimer = null; + } + buffer = []; + if (subscription && typeof subscription.close === "function") { + try { + subscription.close(); + } catch (error) { + logger?.warn?.("[nip71Ingest] close failed", error); + } + } + subscription = null; + } + + function restart() { + if (!started) { + return false; + } + closeSubscription(); + return openSubscription(); + } + + function stop() { + closeSubscription(); + if (typeof offWhitelistChange === "function") { + try { + offWhitelistChange(); + } catch (error) { + // best-effort + } + offWhitelistChange = null; + } + started = false; + } + + return { + isAvailable, + start, + stop, + restart, + // Exposed for tests: deterministic flush + single-video injection. + flush, + injectVideo, + resolveFilters, + }; +} diff --git a/js/ui/applicationBootstrap.js b/js/ui/applicationBootstrap.js index fc79375e..5749f7d3 100644 --- a/js/ui/applicationBootstrap.js +++ b/js/ui/applicationBootstrap.js @@ -18,6 +18,7 @@ import NwcSettingsService from "../services/nwcSettingsService.js"; import nostrService from "../services/nostrService.js"; import storageService from "../services/storageService.js"; import { initNip71MirrorSync } from "../services/nip71MirrorSync.js"; +import { createNip71IngestService } from "../services/nip71IngestService.js"; import watchHistoryService from "../watchHistoryService.js"; import r2Service from "../services/r2Service.js"; import s3UploadService from "../services/s3UploadService.js"; @@ -853,6 +854,24 @@ export default class ApplicationBootstrap { // Keep an opted-in NIP-71 mirror in lockstep on edit/delete (event-driven so // nostrService stays untouched). nostrUnsubscribes.push(initNip71MirrorSync(app.nostrService)); + + // Inbound NIP-71 ingest: pull video events published by other Nostr apps + // (scoped to whitelisted authors while whitelist mode is on) and surface + // them in the feed. Gated by FEATURE_NIP71_INGEST. Best-effort. + const nip71IngestService = createNip71IngestService({ + nostrClient, + nostrService: app.nostrService, + accessControl, + }); + app.nip71IngestService = nip71IngestService; + if (nip71IngestService.isAvailable()) { + try { + nip71IngestService.start(); + } catch (error) { + devLogger.warn("[applicationBootstrap] NIP-71 ingest start failed", error); + } + nostrUnsubscribes.push(() => nip71IngestService.stop()); + } nostrUnsubscribes.push( app.nostrService.on("directMessages:notification", () => { void app.refreshUnreadDmIndicator({ reason: "dm-notification" }); diff --git a/tests/nip71-ingest-service.test.mjs b/tests/nip71-ingest-service.test.mjs new file mode 100644 index 00000000..0eb404b6 --- /dev/null +++ b/tests/nip71-ingest-service.test.mjs @@ -0,0 +1,219 @@ +// Inbound NIP-71 ingest service. Scenario-style with a virtualized nostrClient +// (in-memory activeMap/allEvents + a fake subscription manager) and a fake +// accessControl. Asserts observable outcomes: foreign videos land in the active +// store and trigger a refresh; native videos are never clobbered; the relay +// subscription is scoped to the whitelist; and FEATURE_NIP71_INGEST off makes +// the service inert. + +import assert from "node:assert/strict"; +import test from "node:test"; +import { createNip71IngestService } from "../js/services/nip71IngestService.js"; + +const KINDS = [21, 22, 34235, 34236]; + +function activeKey(video) { + return video.videoRootId ? `ROOT:${video.videoRootId}` : `LEGACY:${video.id}`; +} + +function makeHarness({ whitelistOn = false, whitelist = [] } = {}) { + const activeMap = new Map(); + const allEvents = new Map(); + let capturedFilters = null; + let onEvent = null; + let closed = false; + let whitelistListener = null; + + const manager = { + subscribe(opts) { + capturedFilters = opts.filters; + onEvent = opts.onEvent; + return { + close() { + closed = true; + }, + update() {}, + }; + }, + }; + + const emitted = []; + const nostrClient = { + activeMap, + allEvents, + getActiveKey: activeKey, + getSubscriptionManager: () => manager, + }; + const nostrService = { + emit(name, detail) { + emitted.push({ name, detail }); + }, + getFilteredActiveVideos: () => Array.from(activeMap.values()), + }; + const accessControl = { + whitelistMode: () => whitelistOn, + getWhitelistPubkeys: () => whitelist, + onWhitelistChange: (fn) => { + whitelistListener = fn; + return () => { + whitelistListener = null; + }; + }, + }; + + return { + activeMap, + allEvents, + emitted, + get capturedFilters() { + return capturedFilters; + }, + get closed() { + return closed; + }, + fireEvent: (e) => onEvent?.(e), + triggerWhitelistChange: () => whitelistListener?.(), + nostrClient, + nostrService, + accessControl, + }; +} + +function foreignEvent(overrides = {}) { + return { + id: overrides.id || "evt-1", + pubkey: overrides.pubkey || "author-1", + kind: overrides.kind || 21, + created_at: overrides.created_at ?? 1000, + content: "desc", + tags: overrides.tags || [ + ["title", overrides.title || "Foreign"], + ["imeta", `url ${overrides.url || "https://e/v.mp4"}`, "m video/mp4"], + ], + }; +} + +function makeService(harness, extra = {}) { + return createNip71IngestService({ + nostrClient: harness.nostrClient, + nostrService: harness.nostrService, + accessControl: harness.accessControl, + featureEnabled: true, + flushDelayMs: 0, + ...extra, + }); +} + +test("ingests a foreign NIP-71 video into the active store and signals a refresh", () => { + const h = makeHarness(); + const svc = makeService(h); + assert.equal(svc.start(), true); + + h.fireEvent(foreignEvent({ id: "e1", title: "Hello" })); + const injected = svc.flush(); + + assert.equal(injected, 1); + const stored = h.activeMap.get("ROOT:e1"); + assert.ok(stored, "video present in activeMap"); + assert.equal(stored.title, "Hello"); + assert.equal(stored.source, "nip71-ingest"); + assert.ok(h.allEvents.has("e1"), "raw video cached in allEvents"); + + const refresh = h.emitted.find((e) => e.name === "videos:updated"); + assert.ok(refresh, "emits videos:updated"); + assert.equal(refresh.detail.reason, "nip71-ingest"); +}); + +test("never clobbers a native bitvid video sharing the same root", () => { + const h = makeHarness(); + const native = { id: "native", videoRootId: "shared", source: "nostr", created_at: 500, title: "Native" }; + h.activeMap.set("ROOT:shared", native); + + const svc = makeService(h); + svc.start(); + h.fireEvent(foreignEvent({ id: "foreign", created_at: 9999, tags: [["d", "shared"], ["title", "Foreign"], ["imeta", "url https://e/v.mp4", "m video/mp4"], ], kind: 34235 })); + const injected = svc.flush(); + + assert.equal(injected, 0, "native video is not overwritten even by a newer foreign event"); + assert.equal(h.activeMap.get("ROOT:shared"), native); +}); + +test("among ingested videos for one root, newest wins", () => { + const h = makeHarness(); + const svc = makeService(h); + svc.start(); + + h.fireEvent(foreignEvent({ id: "old", kind: 34235, created_at: 100, tags: [["d", "r"], ["title", "Old"], ["imeta", "url https://e/v1.mp4", "m video/mp4"]] })); + svc.flush(); + h.fireEvent(foreignEvent({ id: "new", kind: 34235, created_at: 200, tags: [["d", "r"], ["title", "New"], ["imeta", "url https://e/v2.mp4", "m video/mp4"]] })); + svc.flush(); + + assert.equal(h.activeMap.get("ROOT:r").title, "New"); + + // An older event for the same root must not resurrect. + h.fireEvent(foreignEvent({ id: "older", kind: 34235, created_at: 50, tags: [["d", "r"], ["title", "Older"], ["imeta", "url https://e/v3.mp4", "m video/mp4"]] })); + svc.flush(); + assert.equal(h.activeMap.get("ROOT:r").title, "New", "older event does not override newer"); +}); + +test("scopes the relay subscription to whitelisted authors when whitelist mode is on", () => { + const h = makeHarness({ whitelistOn: true, whitelist: ["aa", "bb"] }); + const svc = makeService(h); + assert.equal(svc.start(), true); + + const filters = h.capturedFilters; + assert.ok(Array.isArray(filters) && filters.length === 1); + assert.deepEqual(filters[0].kinds.slice().sort((a, b) => a - b), KINDS); + assert.deepEqual(filters[0].authors, ["aa", "bb"], "subscribes only to whitelisted authors"); +}); + +test("with whitelist on but no authors, does not open a subscription (nothing to ingest)", () => { + const h = makeHarness({ whitelistOn: true, whitelist: [] }); + const svc = makeService(h); + assert.equal(svc.start(), false, "no subscription opened"); + assert.equal(h.capturedFilters, null); +}); + +test("opens an unscoped (capped) subscription when whitelist mode is off", () => { + const h = makeHarness({ whitelistOn: false }); + const svc = makeService(h); + svc.start(); + const filters = h.capturedFilters; + assert.ok(filters[0].authors === undefined, "no author scope when whitelist disabled"); + assert.ok(Number.isFinite(filters[0].limit), "still capped by a limit"); +}); + +test("re-subscribes when the whitelist changes", () => { + const h = makeHarness({ whitelistOn: true, whitelist: ["aa"] }); + const svc = makeService(h); + svc.start(); + assert.deepEqual(h.capturedFilters[0].authors, ["aa"]); + + // Whitelist grows; the access-control change fires. + h.accessControl.getWhitelistPubkeys = () => ["aa", "cc"]; + h.triggerWhitelistChange(); + assert.deepEqual(h.capturedFilters[0].authors, ["aa", "cc"], "subscription re-scoped after change"); +}); + +test("FEATURE_NIP71_INGEST off makes the service completely inert", () => { + const h = makeHarness(); + const svc = createNip71IngestService({ + nostrClient: h.nostrClient, + nostrService: h.nostrService, + accessControl: h.accessControl, + featureEnabled: false, + flushDelayMs: 0, + }); + assert.equal(svc.isAvailable(), false); + assert.equal(svc.start(), false); + assert.equal(h.capturedFilters, null, "no relay subscription when ingest is disabled"); +}); + +test("stop() closes the subscription and unhooks the whitelist listener", () => { + const h = makeHarness({ whitelistOn: true, whitelist: ["aa"] }); + const svc = makeService(h); + svc.start(); + svc.stop(); + assert.equal(h.closed, true, "subscription closed"); + // After stop, a whitelist change must not reopen anything. + h.triggerWhitelistChange(); +}); From dcce28cc9766b73ef08198c70bf23836b167b465 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 15:46:28 -0400 Subject: [PATCH 24/43] nip71: fix ingest never subscribing due to whitelist-hydration race MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The admin whitelist hydrates asynchronously (remote fetch, then a second hex-pubkey rebuild once NostrTools loads) and does not always emit a change we can hook — e.g. loaded-from-cache-unchanged, or the async hex rebuild that emits nothing. So at startup getWhitelistPubkeys() was empty, resolveFilters() returned null, no subscription opened, and the onWhitelistChange restart never fired: ingest stayed silent even when a whitelisted author had foreign NIP-71 videos on the relays. Fix: bounded retry (every 2s, ~60s) that re-attempts opening the subscription until the author scope hydrates, in addition to the onWhitelistChange listener. Author-scoping is kept deliberately — a broad kinds:[21,22,...] subscription would be a REQ firehose that violates bitvid's relay fan-out invariants. Also adds dev-mode logging (subscribe author/kind count; per-flush events in / injected) so ingest is observable in the console. Tests: new race test (empty whitelist at start -> authors hydrate without an emit -> retry subscribes); mutation-verified. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/services/nip71IngestService.js | 54 +++++++++++++++++++++++++++-- tests/nip71-ingest-service.test.mjs | 22 ++++++++++++ 2 files changed, 73 insertions(+), 3 deletions(-) diff --git a/js/services/nip71IngestService.js b/js/services/nip71IngestService.js index 2fe701ff..d767f17b 100644 --- a/js/services/nip71IngestService.js +++ b/js/services/nip71IngestService.js @@ -25,6 +25,13 @@ import { devLogger } from "../utils/logger.js"; const FLUSH_DEBOUNCE_MS = 200; const DEFAULT_LIMIT = 200; const SUBSCRIPTION_KEY = "nip71-ingest"; +// The admin whitelist hydrates asynchronously (remote fetch, then a second hex +// rebuild once NostrTools loads) and does not always emit a change we can hook +// (e.g. loaded-from-cache unchanged). So if the author scope isn't ready at +// start, poll a bounded number of times until it is, instead of silently never +// subscribing. +const OPEN_RETRY_DELAY_MS = 2000; +const MAX_OPEN_ATTEMPTS = 30; // ~60s of retries export function createNip71IngestService({ nostrClient, @@ -34,10 +41,14 @@ export function createNip71IngestService({ logger = devLogger, flushDelayMs = FLUSH_DEBOUNCE_MS, limit = DEFAULT_LIMIT, + openRetryDelayMs = OPEN_RETRY_DELAY_MS, + maxOpenAttempts = MAX_OPEN_ATTEMPTS, } = {}) { let subscription = null; let buffer = []; let flushTimer = null; + let retryTimer = null; + let attemptsLeft = maxOpenAttempts; let offWhitelistChange = null; let started = false; @@ -137,6 +148,10 @@ export function createNip71IngestService({ } } + logger?.log?.( + `[nip71Ingest] flush: ${events.length} event(s) in, ${injected} injected`, + ); + if (injected > 0) { // Re-render through the existing reactive path. The payload is the // post-filter active list (whitelist/NSFW/etc. already applied); the feed @@ -167,6 +182,7 @@ export function createNip71IngestService({ return false; } started = true; + attemptsLeft = maxOpenAttempts; // Re-subscribe when the whitelist changes so the author scope stays current. if (typeof accessControl?.onWhitelistChange === "function") { @@ -175,13 +191,34 @@ export function createNip71IngestService({ }); } - return openSubscription(); + return attemptOpen(); + } + + // Open the subscription; if the author scope isn't ready yet (whitelist still + // hydrating), retry on a timer rather than giving up silently. + function attemptOpen() { + if (retryTimer) { + clearTimeout(retryTimer); + retryTimer = null; + } + if (subscription) { + return true; + } + const opened = openSubscription(); + if (opened) { + return true; + } + if (started && attemptsLeft > 0) { + attemptsLeft -= 1; + retryTimer = setTimeout(attemptOpen, openRetryDelayMs); + } + return false; } function openSubscription() { const filters = resolveFilters(); if (!filters) { - return false; // nothing to ingest (e.g. empty whitelist) — stay armed for re-subscribe + return false; // nothing to ingest yet (e.g. whitelist not hydrated) — retry } try { const manager = nostrClient.getSubscriptionManager(); @@ -196,6 +233,12 @@ export function createNip71IngestService({ } }, }); + const authorCount = Array.isArray(filters[0]?.authors) + ? filters[0].authors.length + : "all"; + logger?.log?.( + `[nip71Ingest] subscribed (authors: ${authorCount}, kinds: ${filters[0]?.kinds?.join(",")})`, + ); return true; } catch (error) { logger?.warn?.("[nip71Ingest] subscribe failed", error); @@ -209,6 +252,10 @@ export function createNip71IngestService({ clearTimeout(flushTimer); flushTimer = null; } + if (retryTimer) { + clearTimeout(retryTimer); + retryTimer = null; + } buffer = []; if (subscription && typeof subscription.close === "function") { try { @@ -225,7 +272,8 @@ export function createNip71IngestService({ return false; } closeSubscription(); - return openSubscription(); + attemptsLeft = maxOpenAttempts; + return attemptOpen(); } function stop() { diff --git a/tests/nip71-ingest-service.test.mjs b/tests/nip71-ingest-service.test.mjs index 0eb404b6..a94cfcec 100644 --- a/tests/nip71-ingest-service.test.mjs +++ b/tests/nip71-ingest-service.test.mjs @@ -99,6 +99,9 @@ function makeService(harness, extra = {}) { accessControl: harness.accessControl, featureEnabled: true, flushDelayMs: 0, + // Disable the hydration retry by default so each test is deterministic and + // leaves no dangling timers; the retry behavior is covered explicitly below. + maxOpenAttempts: 0, ...extra, }); } @@ -182,6 +185,25 @@ test("opens an unscoped (capped) subscription when whitelist mode is off", () => assert.ok(Number.isFinite(filters[0].limit), "still capped by a limit"); }); +test("retries until the whitelist hydrates, then subscribes (no missed-emit race)", async () => { + // Simulates the real startup race: whitelist mode is on but the hex author + // set isn't populated yet at start(), and no change event is emitted. + const h = makeHarness({ whitelistOn: true, whitelist: [] }); + const svc = makeService(h, { maxOpenAttempts: 5, openRetryDelayMs: 5 }); + assert.equal(svc.start(), false, "cannot open yet — authors not hydrated"); + assert.equal(h.capturedFilters, null); + + // Authors hydrate a moment later, WITHOUT firing onWhitelistChange. + h.accessControl.getWhitelistPubkeys = () => ["aa", "bb"]; + + // Wait past a retry tick. + await new Promise((resolve) => setTimeout(resolve, 20)); + + assert.ok(h.capturedFilters, "retry opened the subscription after hydration"); + assert.deepEqual(h.capturedFilters[0].authors, ["aa", "bb"]); + svc.stop(); +}); + test("re-subscribes when the whitelist changes", () => { const h = makeHarness({ whitelistOn: true, whitelist: ["aa"] }); const svc = makeService(h); From c4df08d5a18374fa83294dbd4512bdf89812643d Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 16:05:01 -0400 Subject: [PATCH 25/43] nip71: make ingest observable + throttle its feed refreshes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two stabilization fixes after live testing showed ingest was invisible and could amplify feed load: 1. Diagnostics were using devLogger.log, which is gated by IS_DEV_MODE (false in all current builds) — so the [nip71Ingest] subscribe/flush lines never printed and the feature was impossible to observe. Switched the subscribe + per-flush lines to userLogger.info so we can confirm in the console whether it subscribed (author count) and whether events are arriving/injecting. (Temporary while stabilizing.) 2. Coalesce ingest-driven "videos:updated" signals behind an 8s throttle. A burst of incoming events previously emitted a refresh per flush; each refresh re-runs the feed and re-subscribes moderation/social context for every video, which can contribute to a REQ storm. Injected videos are already in the active store, so throttling only delays the visual refresh, not the data. Tests: throttle coalescing test (one immediate refresh per window, both videos still stored); mutation-verified. 1234/1235 unit (known flake). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/services/nip71IngestService.js | 79 +++++++++++++++++++++-------- tests/nip71-ingest-service.test.mjs | 19 +++++++ 2 files changed, 76 insertions(+), 22 deletions(-) diff --git a/js/services/nip71IngestService.js b/js/services/nip71IngestService.js index d767f17b..51181f60 100644 --- a/js/services/nip71IngestService.js +++ b/js/services/nip71IngestService.js @@ -20,9 +20,13 @@ import { buildVideoFromNip71Event, NIP71_KINDS, } from "../nostr/nip71IngestAdapter.js"; -import { devLogger } from "../utils/logger.js"; +import { devLogger, userLogger } from "../utils/logger.js"; const FLUSH_DEBOUNCE_MS = 200; +// Coalesce ingest-driven feed refreshes so a burst of incoming events can't +// trigger a re-render/re-subscription storm. The injected videos are already in +// the active store; this only controls how often we *signal* the feed. +const REFRESH_THROTTLE_MS = 8000; const DEFAULT_LIMIT = 200; const SUBSCRIPTION_KEY = "nip71-ingest"; // The admin whitelist hydrates asynchronously (remote fetch, then a second hex @@ -43,6 +47,7 @@ export function createNip71IngestService({ limit = DEFAULT_LIMIT, openRetryDelayMs = OPEN_RETRY_DELAY_MS, maxOpenAttempts = MAX_OPEN_ATTEMPTS, + refreshThrottleMs = REFRESH_THROTTLE_MS, } = {}) { let subscription = null; let buffer = []; @@ -51,6 +56,8 @@ export function createNip71IngestService({ let attemptsLeft = maxOpenAttempts; let offWhitelistChange = null; let started = false; + let lastRefreshAt = 0; + let pendingRefreshTimer = null; function isAvailable() { return ( @@ -148,35 +155,59 @@ export function createNip71IngestService({ } } - logger?.log?.( + // Visible (not dev-gated) so ingest is observable in any build while we + // stabilize the feature. + userLogger.info( `[nip71Ingest] flush: ${events.length} event(s) in, ${injected} injected`, ); if (injected > 0) { - // Re-render through the existing reactive path. The payload is the - // post-filter active list (whitelist/NSFW/etc. already applied); the feed - // re-reads getFilteredActiveVideos itself, so this just signals "refresh". - let videos = []; - try { - videos = - typeof nostrService?.getFilteredActiveVideos === "function" - ? nostrService.getFilteredActiveVideos() - : []; - } catch (error) { - logger?.warn?.("[nip71Ingest] getFilteredActiveVideos failed", error); - } - if (typeof nostrService?.emit === "function") { - nostrService.emit("videos:updated", { - videos, - deleted: [], - reason: "nip71-ingest", - }); - } + requestRefresh(); } return injected; } + // Signal the feed to re-render, throttled so a burst of ingested events can't + // drive a re-render/re-subscription storm. The injected videos already live in + // the active store; the feed re-reads getFilteredActiveVideos itself, so this + // is purely a "refresh now" nudge. + function requestRefresh() { + if (pendingRefreshTimer) { + return; + } + const now = Date.now(); + const elapsed = now - lastRefreshAt; + if (elapsed >= refreshThrottleMs) { + emitRefresh(); + return; + } + pendingRefreshTimer = setTimeout(() => { + pendingRefreshTimer = null; + emitRefresh(); + }, refreshThrottleMs - elapsed); + } + + function emitRefresh() { + lastRefreshAt = Date.now(); + let videos = []; + try { + videos = + typeof nostrService?.getFilteredActiveVideos === "function" + ? nostrService.getFilteredActiveVideos() + : []; + } catch (error) { + logger?.warn?.("[nip71Ingest] getFilteredActiveVideos failed", error); + } + if (typeof nostrService?.emit === "function") { + nostrService.emit("videos:updated", { + videos, + deleted: [], + reason: "nip71-ingest", + }); + } + } + function start() { if (started || !isAvailable()) { return false; @@ -236,7 +267,7 @@ export function createNip71IngestService({ const authorCount = Array.isArray(filters[0]?.authors) ? filters[0].authors.length : "all"; - logger?.log?.( + userLogger.info( `[nip71Ingest] subscribed (authors: ${authorCount}, kinds: ${filters[0]?.kinds?.join(",")})`, ); return true; @@ -256,6 +287,10 @@ export function createNip71IngestService({ clearTimeout(retryTimer); retryTimer = null; } + if (pendingRefreshTimer) { + clearTimeout(pendingRefreshTimer); + pendingRefreshTimer = null; + } buffer = []; if (subscription && typeof subscription.close === "function") { try { diff --git a/tests/nip71-ingest-service.test.mjs b/tests/nip71-ingest-service.test.mjs index a94cfcec..f7664fea 100644 --- a/tests/nip71-ingest-service.test.mjs +++ b/tests/nip71-ingest-service.test.mjs @@ -102,6 +102,8 @@ function makeService(harness, extra = {}) { // Disable the hydration retry by default so each test is deterministic and // leaves no dangling timers; the retry behavior is covered explicitly below. maxOpenAttempts: 0, + // Emit refreshes synchronously (no throttle timer) for deterministic tests. + refreshThrottleMs: 0, ...extra, }); } @@ -216,6 +218,23 @@ test("re-subscribes when the whitelist changes", () => { assert.deepEqual(h.capturedFilters[0].authors, ["aa", "cc"], "subscription re-scoped after change"); }); +test("coalesces refresh signals so an event burst can't storm the feed", () => { + const h = makeHarness(); + const svc = makeService(h, { refreshThrottleMs: 10000 }); + svc.start(); + + h.fireEvent(foreignEvent({ id: "a", kind: 34235, tags: [["d", "ra"], ["title", "A"], ["imeta", "url https://e/a.mp4", "m video/mp4"]] })); + svc.flush(); + h.fireEvent(foreignEvent({ id: "b", kind: 34235, tags: [["d", "rb"], ["title", "B"], ["imeta", "url https://e/b.mp4", "m video/mp4"]] })); + svc.flush(); + + const refreshes = h.emitted.filter((e) => e.name === "videos:updated"); + assert.equal(refreshes.length, 1, "only one immediate refresh within the throttle window"); + // Both videos still landed in the store regardless of the throttled signal. + assert.ok(h.activeMap.has("ROOT:ra") && h.activeMap.has("ROOT:rb")); + svc.stop(); +}); + test("FEATURE_NIP71_INGEST off makes the service completely inert", () => { const h = makeHarness(); const svc = createNip71IngestService({ From d1246dc11344e4bd34f39007d7ef9c881d1118bf Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 16:23:45 -0400 Subject: [PATCH 26/43] nip71: defer ingest until feed renders; diagnose display path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Live logs proved ingest works (subscribed to 17 whitelisted authors, received 61 events, injected 39) — but injecting foreign authors during cold-start triggers the moderation/WoT stage to hydrate their social graph (kind 30000), which competes with the initial feed load and stalls it ("Fetching recent videos…" hanging, needs a refresh). - startWhenFeedReady(): defer the first ingest subscription until the native feed's first videos:updated (with a 12s fallback for empty feeds), keeping ingest off the cold-start critical path. Wired in applicationBootstrap (kept thin so the size cap holds). - emitRefresh now logs how many ingested videos survive the feed filter ("N ingested of M in filtered feed") so we can tell whether foreign videos are filtered out vs. injected-but-not-rendered. Tests: defer-until-feed-ready + fallback-timer; mutation-verified. 1236/1237 unit (known UploadModal flake). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/services/nip71IngestService.js | 59 +++++++++++++++++++++++++++++ js/ui/applicationBootstrap.js | 12 +++--- tests/nip71-ingest-service.test.mjs | 32 ++++++++++++++++ 3 files changed, 98 insertions(+), 5 deletions(-) diff --git a/js/services/nip71IngestService.js b/js/services/nip71IngestService.js index 51181f60..f28e3cf7 100644 --- a/js/services/nip71IngestService.js +++ b/js/services/nip71IngestService.js @@ -36,6 +36,11 @@ const SUBSCRIPTION_KEY = "nip71-ingest"; // subscribing. const OPEN_RETRY_DELAY_MS = 2000; const MAX_OPEN_ATTEMPTS = 30; // ~60s of retries +// Defer the first subscription until the native feed has rendered (signalled by +// the first videos:updated). Injecting foreign authors hydrates their WoT graph +// (kind 30000), which during cold-start competes with the initial feed load; a +// fallback covers the empty-feed case where no videos:updated ever fires. +const FEED_READY_FALLBACK_MS = 12000; export function createNip71IngestService({ nostrClient, @@ -48,6 +53,7 @@ export function createNip71IngestService({ openRetryDelayMs = OPEN_RETRY_DELAY_MS, maxOpenAttempts = MAX_OPEN_ATTEMPTS, refreshThrottleMs = REFRESH_THROTTLE_MS, + feedReadyFallbackMs = FEED_READY_FALLBACK_MS, } = {}) { let subscription = null; let buffer = []; @@ -58,6 +64,9 @@ export function createNip71IngestService({ let started = false; let lastRefreshAt = 0; let pendingRefreshTimer = null; + let deferArmed = false; + let deferTimer = null; + let offFeedReady = null; function isAvailable() { return ( @@ -199,6 +208,13 @@ export function createNip71IngestService({ } catch (error) { logger?.warn?.("[nip71Ingest] getFilteredActiveVideos failed", error); } + // Confirm ingested videos survive the feed filter (whitelist/NSFW/etc.). + const foreignCount = Array.isArray(videos) + ? videos.filter((v) => v?.source === "nip71-ingest").length + : 0; + userLogger.info( + `[nip71Ingest] refresh: ${foreignCount} ingested of ${videos.length} in filtered feed`, + ); if (typeof nostrService?.emit === "function") { nostrService.emit("videos:updated", { videos, @@ -208,6 +224,35 @@ export function createNip71IngestService({ } } + // Defer the first start until the native feed has rendered (first + // videos:updated), with a fallback timer for the empty-feed case. Keeps ingest + // off the cold-start critical path. + function startWhenFeedReady() { + if (started || deferArmed || !isAvailable()) { + return false; + } + deferArmed = true; + + const begin = () => { + if (typeof offFeedReady === "function") { + offFeedReady(); + offFeedReady = null; + } + if (deferTimer) { + clearTimeout(deferTimer); + deferTimer = null; + } + deferArmed = false; + start(); + }; + + if (typeof nostrService?.on === "function") { + offFeedReady = nostrService.on("videos:updated", begin); + } + deferTimer = setTimeout(begin, feedReadyFallbackMs); + return true; + } + function start() { if (started || !isAvailable()) { return false; @@ -313,6 +358,19 @@ export function createNip71IngestService({ function stop() { closeSubscription(); + if (typeof offFeedReady === "function") { + try { + offFeedReady(); + } catch (error) { + // best-effort + } + offFeedReady = null; + } + if (deferTimer) { + clearTimeout(deferTimer); + deferTimer = null; + } + deferArmed = false; if (typeof offWhitelistChange === "function") { try { offWhitelistChange(); @@ -326,6 +384,7 @@ export function createNip71IngestService({ return { isAvailable, + startWhenFeedReady, start, stop, restart, diff --git a/js/ui/applicationBootstrap.js b/js/ui/applicationBootstrap.js index 5749f7d3..2f04c9f1 100644 --- a/js/ui/applicationBootstrap.js +++ b/js/ui/applicationBootstrap.js @@ -858,6 +858,12 @@ export default class ApplicationBootstrap { // Inbound NIP-71 ingest: pull video events published by other Nostr apps // (scoped to whitelisted authors while whitelist mode is on) and surface // them in the feed. Gated by FEATURE_NIP71_INGEST. Best-effort. + // + // Deferred until the native feed has rendered once: injecting foreign + // authors triggers the moderation/WoT stage to hydrate their social graph + // (kind 30000), which during cold-start competes with the initial feed load + // and can stall it. Starting after the first videos:updated keeps ingest off + // the critical path; a fallback timer covers the empty-feed case. const nip71IngestService = createNip71IngestService({ nostrClient, nostrService: app.nostrService, @@ -865,11 +871,7 @@ export default class ApplicationBootstrap { }); app.nip71IngestService = nip71IngestService; if (nip71IngestService.isAvailable()) { - try { - nip71IngestService.start(); - } catch (error) { - devLogger.warn("[applicationBootstrap] NIP-71 ingest start failed", error); - } + nip71IngestService.startWhenFeedReady(); nostrUnsubscribes.push(() => nip71IngestService.stop()); } nostrUnsubscribes.push( diff --git a/tests/nip71-ingest-service.test.mjs b/tests/nip71-ingest-service.test.mjs index f7664fea..41b7918b 100644 --- a/tests/nip71-ingest-service.test.mjs +++ b/tests/nip71-ingest-service.test.mjs @@ -43,10 +43,17 @@ function makeHarness({ whitelistOn = false, whitelist = [] } = {}) { getActiveKey: activeKey, getSubscriptionManager: () => manager, }; + const feedListeners = new Set(); const nostrService = { emit(name, detail) { emitted.push({ name, detail }); }, + on(name, fn) { + if (name === "videos:updated") { + feedListeners.add(fn); + } + return () => feedListeners.delete(fn); + }, getFilteredActiveVideos: () => Array.from(activeMap.values()), }; const accessControl = { @@ -72,6 +79,7 @@ function makeHarness({ whitelistOn = false, whitelist = [] } = {}) { }, fireEvent: (e) => onEvent?.(e), triggerWhitelistChange: () => whitelistListener?.(), + triggerFeedReady: () => feedListeners.forEach((fn) => fn({})), nostrClient, nostrService, accessControl, @@ -218,6 +226,30 @@ test("re-subscribes when the whitelist changes", () => { assert.deepEqual(h.capturedFilters[0].authors, ["aa", "cc"], "subscription re-scoped after change"); }); +test("startWhenFeedReady defers subscription until the native feed renders", () => { + const h = makeHarness({ whitelistOn: true, whitelist: ["aa"] }); + const svc = makeService(h, { feedReadyFallbackMs: 100000 }); + + assert.equal(svc.startWhenFeedReady(), true); + assert.equal(h.capturedFilters, null, "does not subscribe during cold-start"); + + // Native feed renders -> first videos:updated -> ingest starts now. + h.triggerFeedReady(); + assert.ok(h.capturedFilters, "subscribes once the feed is ready"); + assert.deepEqual(h.capturedFilters[0].authors, ["aa"]); + svc.stop(); +}); + +test("startWhenFeedReady falls back to starting if the feed never emits", async () => { + const h = makeHarness({ whitelistOn: true, whitelist: ["aa"] }); + const svc = makeService(h, { feedReadyFallbackMs: 5 }); + svc.startWhenFeedReady(); + assert.equal(h.capturedFilters, null); + await new Promise((resolve) => setTimeout(resolve, 20)); + assert.ok(h.capturedFilters, "fallback timer started ingest"); + svc.stop(); +}); + test("coalesces refresh signals so an event burst can't storm the feed", () => { const h = makeHarness(); const svc = makeService(h, { refreshThrottleMs: 10000 }); From 639d045e280320f8dfbef2224ee7b3be32b87240 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 16:31:31 -0400 Subject: [PATCH 27/43] nip71: default ingest OFF until feed pipeline can absorb it Ingest's data path is verified working (subscribe -> fetch -> adapt -> inject -> survives the whitelist filter), but surfacing a burst of ingested authors pushes bitvid's moderation/WoT stage to hydrate trust + report data (kind 30000 / 1984) for many cold authors at once, storming relays and stalling the feed render ("nothing loads"). The native feed avoids this only because it trickles authors in gradually. Flip FEATURE_NIP71_INGEST default to false to restore a stable feed for launch. All ingest code stays in place behind the admin toggle; re-enable once moderation/WoT hydration is bounded/non-blocking for an author influx, or whitelisted authors are treated as pre-trusted (skipping the WoT fetch). Tracked as the next NIP-71 increment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- config/instance-config.js | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/config/instance-config.js b/config/instance-config.js index 6a22e0ef..ba450e93 100644 --- a/config/instance-config.js +++ b/config/instance-config.js @@ -246,8 +246,14 @@ export const ALLOW_NSFW_CONTENT = false; * * Set to `false` to disable all inbound NIP-71 ingest with a single switch if it * ever causes problems (spam, low-quality content, performance). + * + * NOTE: currently OFF by default. Ingest's data pipeline works, but surfacing a + * burst of ingested authors pushes bitvid's moderation/WoT stage to hydrate + * trust + report data for many cold authors at once, which storms relays and + * stalls the feed render. Re-enable once that hydration is bounded/non-blocking + * (or whitelisted authors are treated as pre-trusted). See the NIP-71 plan doc. */ -export const FEATURE_NIP71_INGEST = true; +export const FEATURE_NIP71_INGEST = false; /** * Emergency fallback accounts for seeding the moderation graph. From 96a985ce89bda5f8e9491207da12ab2e5ee7bb42 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 16:54:41 -0400 Subject: [PATCH 28/43] =?UTF-8?q?nip71:=20fix=20feed=20hang=20=E2=80=94=20?= =?UTF-8?q?surface=20published=5Fat=20so=20ingest=20skips=20history=20fetc?= =?UTF-8?q?h?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Deep dive into the "nothing loads / Fetching recent videos… hangs" symptom found the real blocker: it was NOT the moderation/WoT stage (that uses cached reads and passes unknown authors through clean). It was the feed's resolve-posted-at stage. resolveVideoPostedAt() short-circuits when a video has nip71.publishedAt, otherwise it does a per-video `nostrClient.hydrateVideoHistory()` network fetch (looking up kind-30078 history). Foreign NIP-71 videos have no such history, and the ingest adapter wasn't setting nip71.publishedAt — so a burst of ~32 ingested videos triggered ~32 blocking history fetches that stalled the render and flooded relays. Fix: the ingest adapter now surfaces nip71.publishedAt from the NIP-71 published_at tag (falling back to created_at). The feed short-circuits and never fetches history for ingested videos. Re-enables FEATURE_NIP71_INGEST by default now that the hang is addressed. Tests: adapter published_at extraction + created_at fallback; mutation-verified. 1237/1238 unit (known UploadModal flake). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- config/instance-config.js | 8 +------- js/nostr/nip71IngestAdapter.js | 17 ++++++++++++++++- tests/nip71-ingest-adapter.test.mjs | 19 +++++++++++++++++++ 3 files changed, 36 insertions(+), 8 deletions(-) diff --git a/config/instance-config.js b/config/instance-config.js index ba450e93..6a22e0ef 100644 --- a/config/instance-config.js +++ b/config/instance-config.js @@ -246,14 +246,8 @@ export const ALLOW_NSFW_CONTENT = false; * * Set to `false` to disable all inbound NIP-71 ingest with a single switch if it * ever causes problems (spam, low-quality content, performance). - * - * NOTE: currently OFF by default. Ingest's data pipeline works, but surfacing a - * burst of ingested authors pushes bitvid's moderation/WoT stage to hydrate - * trust + report data for many cold authors at once, which storms relays and - * stalls the feed render. Re-enable once that hydration is bounded/non-blocking - * (or whitelisted authors are treated as pre-trusted). See the NIP-71 plan doc. */ -export const FEATURE_NIP71_INGEST = false; +export const FEATURE_NIP71_INGEST = true; /** * Emergency fallback accounts for seeding the moderation graph. diff --git a/js/nostr/nip71IngestAdapter.js b/js/nostr/nip71IngestAdapter.js index fe308d79..d5e1c337 100644 --- a/js/nostr/nip71IngestAdapter.js +++ b/js/nostr/nip71IngestAdapter.js @@ -153,6 +153,18 @@ export function buildVideoFromNip71Event(event = {}) { const dTag = dTagFromEvent(event); const videoRootId = dTag || trimString(event.id); + // The feed's resolve-posted-at stage fetches per-video kind-30078 history when + // a timestamp isn't already known. Foreign NIP-71 videos have no such history, + // so without a posted-at the feed would fire a blocking history fetch per + // ingested video (a relay storm that stalls the render). Surface the NIP-71 + // published_at (falling back to created_at) so the feed short-circuits. + const createdAt = Number.isFinite(event.created_at) ? Math.floor(event.created_at) : 0; + const publishedAtNum = Number(metadata.publishedAt); + const publishedAt = + Number.isFinite(publishedAtNum) && publishedAtNum > 0 + ? Math.floor(publishedAtNum) + : createdAt; + return { id: event.id, videoRootId, @@ -181,7 +193,10 @@ export function buildVideoFromNip71Event(event = {}) { infoJsonUrl: "", enableComments: true, pubkey: trimString(event.pubkey), - created_at: Number.isFinite(event.created_at) ? event.created_at : 0, + created_at: createdAt, + // Lets the feed's resolve-posted-at stage short-circuit instead of fetching + // non-existent kind-30078 history for foreign videos. + nip71: { publishedAt }, tags: Array.isArray(event.tags) ? event.tags : [], // Provenance markers so the rest of the app can distinguish ingested videos // from native bitvid uploads (e.g. to hide WebTorrent-only affordances). diff --git a/tests/nip71-ingest-adapter.test.mjs b/tests/nip71-ingest-adapter.test.mjs index c1aaa68f..de9cde86 100644 --- a/tests/nip71-ingest-adapter.test.mjs +++ b/tests/nip71-ingest-adapter.test.mjs @@ -56,6 +56,25 @@ test("maps a foreign NIP-71 event to a playable bitvid video", () => { assert.equal(v.isPrivate, false); }); +test("surfaces nip71.publishedAt so the feed skips per-video history fetches", () => { + // The published_at tag becomes nip71.publishedAt (feed resolve-posted-at uses + // it to avoid a blocking kind-30078 history fetch per ingested video). + const withPublished = buildVideoFromNip71Event( + foreignEvent({ + tags: [ + ["title", "Has published_at"], + ["published_at", "1700000123"], + ["imeta", "url https://example.com/v.mp4", "m video/mp4"], + ], + }), + ); + assert.equal(withPublished.nip71.publishedAt, 1700000123); + + // Falls back to created_at when there's no published_at tag. + const noPublished = buildVideoFromNip71Event(foreignEvent({ created_at: 1699999999 })); + assert.equal(noPublished.nip71.publishedAt, 1699999999); +}); + test("content-warning maps to isNsfw so the NSFW gate applies", () => { const clean = buildVideoFromNip71Event(foreignEvent()); assert.equal(clean.isNsfw, false); From 733cb5937b2d84984b24f203eb8d982bc70bfc5f Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 18:24:05 -0400 Subject: [PATCH 29/43] nostr: sanitize hex-only filter fields (fix relay "from_hex" REQ rejects) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Strict relays (primal) reject a whole REQ with "uneven size input to from_hex" when any ids/authors/#e/#p/#q entry isn't 64-char hex — silently dropping that subscription's results. One malformed value anywhere in a filter poisoned the request. Sanitize at the universal pool choke point (normalizeFilterList, used by both the shimmed pool.sub and pool.list): lowercase/trim and drop non- 64-hex entries from the hex-only fields; if a once-populated hex field ends up empty, drop that filter rather than silently broadening the query. Non-hex fields (#a/#d/#t/kinds/…) are untouched. Dev-logs what it strips (with the filter's kinds) so the malformed source can be traced. Also records pre-launch TODO items: NIP-71 inbound ingest is now live (Phase 2, incl. the resolve-posted-at history-fetch fix), and a new launch-blocker — channel-profile walls fetch only kind-30078 so a creator's NIP-71 videos don't show on their profile. Tests: filter-hex-sanitize (strip/keep/lowercase/drop-empty/leave-non-hex); mutation-verified. 1244/1245 unit (known UploadModal flake). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/nostr/toolkit.js | 63 +++++++++++++++++++++++++++-- tests/filter-hex-sanitize.test.mjs | 64 ++++++++++++++++++++++++++++++ todo/TODO_2026-06-20_pre-launch.md | 28 ++++++++++++- 3 files changed, 150 insertions(+), 5 deletions(-) create mode 100644 tests/filter-hex-sanitize.test.mjs diff --git a/js/nostr/toolkit.js b/js/nostr/toolkit.js index 96361159..94f46e5a 100644 --- a/js/nostr/toolkit.js +++ b/js/nostr/toolkit.js @@ -147,17 +147,74 @@ function normalizeRelayList(relays) { .filter((relay) => relay); } +// Filter fields that relays hex-decode (from_hex) and therefore MUST contain +// 64-char lowercase hex. A single malformed/odd-length entry makes strict relays +// (e.g. primal) reject the whole REQ with "uneven size input to from_hex", +// silently dropping that subscription's results. +const HEX64_FILTER_FIELDS = ["ids", "authors", "#e", "#p", "#q"]; +const HEX64_PATTERN = /^[0-9a-f]{64}$/; + +// Strip invalid hex from the hex-only filter fields so one bad value can't get a +// whole REQ rejected. Returns the sanitized filter, or null if a hex field that +// had entries ends up empty (the query would be meaningless/over-broad without +// them, so the filter is dropped rather than silently widened). +export function sanitizeHexFilterFields(filter) { + if (!filter || typeof filter !== "object") { + return filter; + } + let sanitized = filter; + for (const field of HEX64_FILTER_FIELDS) { + const values = filter[field]; + if (!Array.isArray(values) || values.length === 0) { + continue; + } + const kept = []; + for (const value of values) { + const normalized = typeof value === "string" ? value.trim().toLowerCase() : ""; + if (HEX64_PATTERN.test(normalized)) { + kept.push(normalized); + } + } + const changed = + kept.length !== values.length || kept.some((value, i) => value !== values[i]); + if (!changed) { + continue; + } + const droppedCount = values.length - kept.length; + if (droppedCount > 0) { + devLogger.warn( + `[toolkit] dropped ${droppedCount} invalid hex value(s) from filter "${field}"` + + ` (kinds: ${Array.isArray(filter.kinds) ? filter.kinds.join(",") : "?"})`, + ); + } + if (kept.length === 0) { + return null; // a once-populated hex field is now empty — drop the filter + } + if (sanitized === filter) { + sanitized = { ...filter }; + } + sanitized[field] = kept; + } + return sanitized; +} + function normalizeFilterList(filters) { if (!filters) { return []; } + let list; if (!Array.isArray(filters)) { if (typeof filters === "object" && filters !== null) { - return [filters]; + list = [filters]; + } else { + return []; } - return []; + } else { + list = filters.filter((candidate) => candidate && typeof candidate === "object"); } - return filters.filter((candidate) => candidate && typeof candidate === "object"); + return list + .map((filter) => sanitizeHexFilterFields(filter)) + .filter((filter) => filter && typeof filter === "object"); } function safeInvoke(handler, args, contextLabel) { diff --git a/tests/filter-hex-sanitize.test.mjs b/tests/filter-hex-sanitize.test.mjs new file mode 100644 index 00000000..a3701af7 --- /dev/null +++ b/tests/filter-hex-sanitize.test.mjs @@ -0,0 +1,64 @@ +// Sanitizing hex-only filter fields (ids/authors/#e/#p/#q) before they reach a +// relay. One odd-length/non-hex value otherwise makes strict relays reject the +// whole REQ ("uneven size input to from_hex"), silently dropping results. + +import assert from "node:assert/strict"; +import test from "node:test"; +import { sanitizeHexFilterFields } from "../js/nostr/toolkit.js"; + +const HEX_A = "a".repeat(64); +const HEX_B = "b".repeat(64); +const ODD = "abc"; // odd length -> from_hex failure +const SHORT = "a".repeat(63); // wrong length + +test("strips invalid hex from authors but keeps valid entries", () => { + const out = sanitizeHexFilterFields({ kinds: [0], authors: [HEX_A, ODD, HEX_B] }); + assert.deepEqual(out.authors, [HEX_A, HEX_B]); + assert.deepEqual(out.kinds, [0], "non-hex fields untouched"); +}); + +test("lowercases and trims valid hex", () => { + const out = sanitizeHexFilterFields({ ids: [` ${HEX_A.toUpperCase()} `] }); + assert.deepEqual(out.ids, [HEX_A]); +}); + +test("drops the whole filter when a populated hex field becomes empty", () => { + // If every author was invalid, removing the key would broaden the query to + // "all authors" — drop the filter instead. + const out = sanitizeHexFilterFields({ kinds: [1], authors: [ODD, SHORT] }); + assert.equal(out, null); +}); + +test("sanitizes event/pubkey tag filters (#e/#p/#q)", () => { + const out = sanitizeHexFilterFields({ + kinds: [1984], + "#e": [HEX_A, ODD], + "#p": [HEX_B], + }); + assert.deepEqual(out["#e"], [HEX_A]); + assert.deepEqual(out["#p"], [HEX_B]); +}); + +test("leaves non-hex tag fields (#a/#d/#t) alone", () => { + const filter = { + kinds: [21, 22], + "#a": ["30078:" + HEX_A + ":some-d-tag"], + "#d": ["subscriptions"], + "#t": ["music"], + }; + const out = sanitizeHexFilterFields(filter); + assert.deepEqual(out["#a"], filter["#a"]); + assert.deepEqual(out["#d"], filter["#d"]); + assert.deepEqual(out["#t"], filter["#t"]); +}); + +test("returns the same object when nothing needs sanitizing (no needless copy)", () => { + const filter = { kinds: [0], authors: [HEX_A] }; + const out = sanitizeHexFilterFields(filter); + assert.equal(out, filter); +}); + +test("ignores empty/absent hex fields", () => { + const out = sanitizeHexFilterFields({ kinds: [1], authors: [] }); + assert.deepEqual(out, { kinds: [1], authors: [] }); +}); diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index 707c1eeb..ce7d4949 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -283,5 +283,29 @@ notes below. Summary: OFF/dormant — superseded by the opt-in 34235/36 mirror; clean up later. - [ ] Phase 1.5: NIP-89 handler reg (kind 31990 → "Open in bitvid" elsewhere); NIP-51 kind 30005 portable playlists. -- [ ] Phase 2: inbound ingest of external NIP-71 videos (dedup, moderation, trust, - `origin` attribution, distinct discovery surface). +- [x] Phase 2: inbound ingest of external NIP-71 videos — LIVE on unstable. + - `js/nostr/nip71IngestAdapter.js` (foreign 21/22/34235/34236 → bitvid video; + content-warning→isNsfw; skips bitvid mirrors; surfaces nip71.publishedAt), + `js/services/nip71IngestService.js` (whitelist-scoped subscription, + hydration retry, deferred-until-feed-ready, throttled refresh), + wired in applicationBootstrap. Admin toggle `FEATURE_NIP71_INGEST` + (instance-config) default ON. + - Whitelist model: ingest is scoped to whitelisted authors and reuses the + existing render-time filter (whitelist/NSFW/blacklist/private). When the + admin disables whitelist mode, ingest opens to all authors (capped). + - KEY FIX: the feed's resolve-posted-at stage was firing a per-video + hydrateVideoHistory() network fetch for each ingested video (no kind-30078 + history exists) → relay storm + feed hang. Fixed by surfacing + nip71.publishedAt so the stage short-circuits. + - Verified live: Goblinbox (Nostube) content renders; liveness check works. + +### 17b. Channel Profile wall shows only bitvid videos, not NIP-71 (LAUNCH-BLOCKER) +- [ ] The channel-profile grid (`js/channelProfile.js`, ~line 5387) fetches ONLY + `{ kinds:[30078], authors:[pubkey], "#t":["video"] }`, so a creator's + NIP-71 videos (kinds 21/22/34235/34236 — incl. their Nostube uploads) never + appear on their profile wall even though they show in the main feed via + ingest. Fix: also fetch the author's NIP-71 video kinds and merge them + (convert via `buildVideoFromNip71Event`, skip bitvid mirrors, dedupe by + videoRootId, native-30078 wins a shared root — same precedence as ingest). + Consider reusing the ingested videos already in `nostrClient.allEvents` for + that author rather than a fresh fetch. From a8e9f5202a054b8da90b86cc27c176e0f553d2b8 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 18:47:00 -0400 Subject: [PATCH 30/43] adminLists: batch community-blacklist fetch (kill cold-start kind-30000 storm) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cold-start fired one kind-30000 REQ per community-blacklist curator (loadCommunityBlacklistEntries did Promise.all over references, each a fetchLatestListEvent) — ~28-32 parallel REQs that dominated the login relay-REQ spike and starved the feed. A Nostr filter ORs `authors` and `#d`, so all curators collapse into ONE batched REQ. New js/adminListBatch.js (pure, mutation-tested): buildBatchedReferenceFilter + selectNewestEventsForReferences (newest per author:d-tag, discards author×d-tag cross-products) + fetchBatchedReferenceEvents (routes through the subscription manager's list — health-gated/timeout-bounded, not raw pool). Helpers extracted to their own module so adminListStore stays under its size cap. Best-effort: on batch failure we return nothing and the cached admin state carries over (readCachedAdminState). Addresses pre-launch TODO #9 (cold-login REQ storm). Tests + mutation- verified (newest selection, cross-product discard). 1249/1250 unit (known UploadModal flake). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/adminListBatch.js | 88 +++++++++++++++++++++++++ js/adminListStore.js | 38 +++++------ tests/admin-list-batched-fetch.test.mjs | 73 ++++++++++++++++++++ 3 files changed, 180 insertions(+), 19 deletions(-) create mode 100644 js/adminListBatch.js create mode 100644 tests/admin-list-batched-fetch.test.mjs diff --git a/js/adminListBatch.js b/js/adminListBatch.js new file mode 100644 index 00000000..83f7a385 --- /dev/null +++ b/js/adminListBatch.js @@ -0,0 +1,88 @@ +// Pure helpers for batching the community-blacklist curator fetch into a single +// multi-author REQ instead of one kind-30000 REQ per curator (the cold-start +// relay storm). Kept dependency-free so they're cheat-resistant to test. + +// Build a single batched filter for many (authorHex, dTag) references. A Nostr +// filter ORs `authors` and ORs `#d`, so one REQ replaces N per-curator REQs. +// Over-fetches author×dtag cross-products, which selectNewestEventsForReferences +// discards when matching back to exact references. +export function buildBatchedReferenceFilter(references, { kind = 30000 } = {}) { + const authors = new Set(); + const dTags = new Set(); + for (const ref of Array.isArray(references) ? references : []) { + const author = + typeof ref?.authorHex === "string" ? ref.authorHex.trim().toLowerCase() : ""; + const dTag = typeof ref?.dTag === "string" ? ref.dTag.trim() : ""; + if (author) authors.add(author); + if (dTag) dTags.add(dTag); + } + if (!authors.size || !dTags.size) { + return null; + } + return { + kinds: [kind], + authors: Array.from(authors), + "#d": Array.from(dTags), + }; +} + +function dTagOf(event) { + const tags = Array.isArray(event?.tags) ? event.tags : []; + for (const tag of tags) { + if (Array.isArray(tag) && tag[0] === "d") { + return typeof tag[1] === "string" ? tag[1] : ""; + } + } + return ""; +} + +// Group batched events to the newest per (author, dTag), then return the events +// matching the requested references (in reference order). Cross-product events +// (author×dtag combinations that weren't requested) are discarded. +export function selectNewestEventsForReferences(events, references) { + const newest = new Map(); // key `${pubkey}:${dtag}` -> event + for (const event of Array.isArray(events) ? events : []) { + const pubkey = typeof event?.pubkey === "string" ? event.pubkey.toLowerCase() : ""; + const dTag = dTagOf(event); + if (!pubkey || !dTag) { + continue; + } + const key = `${pubkey}:${dTag}`; + const prev = newest.get(key); + if ( + !prev || + event.created_at > prev.created_at || + (event.created_at === prev.created_at && event.id > prev.id) + ) { + newest.set(key, event); + } + } + const matched = []; + for (const ref of Array.isArray(references) ? references : []) { + const author = typeof ref?.authorHex === "string" ? ref.authorHex.toLowerCase() : ""; + const dTag = typeof ref?.dTag === "string" ? ref.dTag : ""; + const event = newest.get(`${author}:${dTag}`); + if (event) { + matched.push(event); + } + } + return matched; +} + +// Fetch all curator blacklists in ONE batched REQ and return the newest event +// matched to each reference. `listEvents` is injected (the subscription +// manager's health-gated, timeout-bounded list) so this stays off the raw pool. +// Returns [] when there's nothing to batch; rejects on fetch failure. +export async function fetchBatchedReferenceEvents({ + references, + listEvents, + relays, + timeoutMs, +}) { + const filter = buildBatchedReferenceFilter(references); + if (!filter || typeof listEvents !== "function") { + return []; + } + const events = await listEvents({ filters: [filter], relays, timeoutMs }); + return selectNewestEventsForReferences(events, references); +} diff --git a/js/adminListStore.js b/js/adminListStore.js index 83eddb5e..140c1696 100644 --- a/js/adminListStore.js +++ b/js/adminListStore.js @@ -22,6 +22,7 @@ import { } from "./nostrEventSchemas.js"; import { CACHE_POLICIES, STORAGE_TIERS } from "./nostr/cachePolicies.js"; import { publishEventToRelay } from "./nostrPublish.js"; +import { fetchBatchedReferenceEvents } from "./adminListBatch.js"; const ADMIN_STATE_CACHE_VERSION = 1; const ADMIN_STATE_CACHE_KEY = `bitvid_admin_state_v${ADMIN_STATE_CACHE_VERSION}`; @@ -688,25 +689,24 @@ async function loadCommunityBlacklistEntries() { return []; } - const results = await Promise.all( - references.map(async (reference) => { - try { - const event = await fetchLatestListEvent( - { "#d": [reference.dTag], authors: [reference.authorHex] }, - reference.dTag, - ); - return event ? extractNpubsFromEvent(event) : []; - } catch (error) { - devLogger.warn( - `[adminListStore] Failed to load community blacklist ${reference.dTag}:`, - error, - ); - return []; - } - }) - ); - - return results.flat(); + // Batch all curator blacklists into ONE REQ instead of one-per-curator — this + // was the cold-start kind-30000 relay storm. Best-effort: on failure we return + // nothing and the cached admin state (readCachedAdminState) carries over. + try { + const matched = await fetchBatchedReferenceEvents({ + references, + listEvents: (args) => requireNostrClient().getSubscriptionManager().list(args), + relays: ensureNostrReady(), + timeoutMs: SHORT_TIMEOUT_MS, + }); + return matched.flatMap((event) => extractNpubsFromEvent(event)); + } catch (error) { + devLogger.warn( + "[adminListStore] Batched community blacklist fetch failed:", + error, + ); + return []; + } } function buildListEvent(listKey, npubs, actorHex) { diff --git a/tests/admin-list-batched-fetch.test.mjs b/tests/admin-list-batched-fetch.test.mjs new file mode 100644 index 00000000..8da47e4f --- /dev/null +++ b/tests/admin-list-batched-fetch.test.mjs @@ -0,0 +1,73 @@ +// Batching the community-blacklist curator fetch into a single multi-author REQ +// (instead of one kind-30000 REQ per curator — the cold-start relay storm). +// Tests the pure filter-builder + result-selector. + +import assert from "node:assert/strict"; +import test from "node:test"; +import { + buildBatchedReferenceFilter, + selectNewestEventsForReferences, +} from "../js/adminListBatch.js"; + +const A = "a".repeat(64); +const B = "b".repeat(64); + +function evt({ id, pubkey, d, created_at }) { + return { id, pubkey, created_at, tags: [["d", d]] }; +} + +test("builds one filter with all unique authors + d-tags", () => { + const filter = buildBatchedReferenceFilter([ + { authorHex: A, dTag: "community:blacklist" }, + { authorHex: B, dTag: "community:blacklist" }, + { authorHex: A, dTag: "community:blacklist" }, // dup + ]); + assert.deepEqual(filter.kinds, [30000]); + assert.deepEqual(filter.authors.sort(), [A, B]); + assert.deepEqual(filter["#d"], ["community:blacklist"]); +}); + +test("returns null when there are no usable references", () => { + assert.equal(buildBatchedReferenceFilter([]), null); + assert.equal(buildBatchedReferenceFilter([{ authorHex: "", dTag: "" }]), null); +}); + +test("selects the newest event per (author, d-tag) reference", () => { + const references = [ + { authorHex: A, dTag: "list-1" }, + { authorHex: B, dTag: "list-1" }, + ]; + const events = [ + evt({ id: "a-old", pubkey: A, d: "list-1", created_at: 100 }), + evt({ id: "a-new", pubkey: A, d: "list-1", created_at: 200 }), + evt({ id: "b-only", pubkey: B, d: "list-1", created_at: 50 }), + ]; + const matched = selectNewestEventsForReferences(events, references); + assert.deepEqual(matched.map((e) => e.id), ["a-new", "b-only"]); +}); + +test("discards cross-product events that aren't real references", () => { + // The batched filter (authors A,B × d-tags d1,d2) can return (A,d2) even + // though only (A,d1) and (B,d2) were requested. Those must be dropped. + const references = [ + { authorHex: A, dTag: "d1" }, + { authorHex: B, dTag: "d2" }, + ]; + const events = [ + evt({ id: "a-d1", pubkey: A, d: "d1", created_at: 10 }), + evt({ id: "a-d2-crossproduct", pubkey: A, d: "d2", created_at: 99 }), + evt({ id: "b-d2", pubkey: B, d: "d2", created_at: 10 }), + ]; + const matched = selectNewestEventsForReferences(events, references); + assert.deepEqual(matched.map((e) => e.id).sort(), ["a-d1", "b-d2"]); +}); + +test("tie-break on created_at uses higher event id (deterministic)", () => { + const references = [{ authorHex: A, dTag: "d1" }]; + const events = [ + evt({ id: "id-aaa", pubkey: A, d: "d1", created_at: 100 }), + evt({ id: "id-zzz", pubkey: A, d: "d1", created_at: 100 }), + ]; + const matched = selectNewestEventsForReferences(events, references); + assert.deepEqual(matched.map((e) => e.id), ["id-zzz"]); +}); From 3fb3c6a07e0375639efc048176f74b186329038e Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 18:47:52 -0400 Subject: [PATCH 31/43] docs(todo): mark cold-start REQ-storm + from_hex fixes done Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- todo/TODO_2026-06-20_pre-launch.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index ce7d4949..55a3b4a1 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -131,8 +131,16 @@ tests → `npm run build` + `npm run test:unit` green → commit + push. surfacing the same control from the Storage settings pane. ### 9. Cap the cold-login relay-REQ storm further -- [ ] Investigate the ~117 REQ/s spike at login. Respect the `capReadRelays` invariant - everywhere fan-out happens; consider staggering subsystem subscriptions at login. +- [x] **Biggest source fixed** (`a8e9f520`): the community-blacklist load fired one + kind-30000 REQ *per curator* (~28-32 parallel) at cold start. Collapsed into a + single batched multi-author REQ (`js/adminListBatch.js`), routed through the + subscription manager. This was the dominant `kind 30000` storm. +- [x] **Relay `from_hex` REQ rejects fixed** (`733cb593`): one odd-length/non-hex + value in `ids/authors/#e/#p/#q` made strict relays reject a whole REQ; now + sanitized at the pool choke point (`normalizeFilterList`). +- [ ] Re-measure the login REQ/s spike after these. If still high, look at remaining + fan-out (profiles kind 0, reports kind 1984, mute lists kind 10000) and consider + staggering subsystem subscriptions at login. Respect `capReadRelays` everywhere. ## Open — lower priority / infra From 07b212943d0ab884e8af03fb32d51b205e138716 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 19:07:22 -0400 Subject: [PATCH 32/43] docs(todo): scope remaining logged-in cold-start WoT fan-out Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- todo/TODO_2026-06-20_pre-launch.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index 55a3b4a1..85c817cf 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -138,9 +138,18 @@ tests → `npm run build` + `npm run test:unit` green → commit + push. - [x] **Relay `from_hex` REQ rejects fixed** (`733cb593`): one odd-length/non-hex value in `ids/authors/#e/#p/#q` made strict relays reject a whole REQ; now sanitized at the pool choke point (`normalizeFilterList`). -- [ ] Re-measure the login REQ/s spike after these. If still high, look at remaining - fan-out (profiles kind 0, reports kind 1984, mute lists kind 10000) and consider - staggering subsystem subscriptions at login. Respect `capReadRelays` everywhere. +- [ ] **Remaining: logged-IN cold-start WoT/social-graph fan-out** (~94-104 REQ/s, + bracketed by `[lists-sync-start]`…`[lists-sync-complete]`). Logged-OUT is now + clean. The spike is per-contact hydration over the user's ~25 follows/trusted + contacts: `kind 30000` (follow sets), `30002` (block lists), `30005/30015` + (interest sets), `10050` (DM relay hints), plus `0` (profiles). NOTE: mutes + (`kind 10000`) are ALREADY batched (moderationService.refreshTrustedMuteSubscriptions, + one multi-author filter). Fix = apply the same multi-author batching to the + other per-contact fetches (userBlocks 30002, hashtagPreferences 30005/30015, + follow-set/contacts hydration 30000, DM 10050) and/or DEFER non-feed-critical + ones (DM relays, interest sets) until after first render. Multi-subsystem + + sensitive (trust) → do as a dedicated perf pass, not a rushed change. + Source still to pin: the loop issuing 30000/30002/30005/30015 per contact. ## Open — lower priority / infra From 7236496a16a1525f1d84f6580e33b67dd2eb2e8d Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 19:20:48 -0400 Subject: [PATCH 33/43] nip71: show creators' NIP-71 videos on their channel-profile wall (17b) The channel-profile grid fetched only kind-30078, so a creator's NIP-71 videos (incl. their Nostube cross-posts) appeared in the main feed via ingest but were absent from their profile wall. New js/channelProfileVideos.js: - buildChannelVideoFilters(pubkey): native (kind 30078, #t=video) + NIP-71 (kinds 21/22/34235/34236) filters for the author. - convertChannelEvent(entry): routes NIP-71 kinds through buildVideoFromNip71Event (which rejects bitvid's own outbound mirrors so they don't duplicate the canonical note); kind-30078 uses the standard converter. channelProfile.js now uses these (net fewer lines, stays under size cap). Existing buildRenderableChannelVideos dedupe-by-root + canAccess handle precedence and whitelist/blacklist gating, so foreign videos inherit the same gating as native ones. Closes pre-launch TODO 17b. Tests + mutation-verified (NIP-71 routing, mirror skip, dual-filter build). 1253/1254 unit (known UploadModal flake). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/channelProfile.js | 17 ++---- js/channelProfileVideos.js | 29 ++++++++++ tests/channel-profile-videos.test.mjs | 81 +++++++++++++++++++++++++++ todo/TODO_2026-06-20_pre-launch.md | 16 +++--- 4 files changed, 123 insertions(+), 20 deletions(-) create mode 100644 js/channelProfileVideos.js create mode 100644 tests/channel-profile-videos.test.mjs diff --git a/js/channelProfile.js b/js/channelProfile.js index c4449fd9..a348d47b 100644 --- a/js/channelProfile.js +++ b/js/channelProfile.js @@ -1,7 +1,7 @@ // js/channelProfile.js import { nostrClient } from "./nostrClientFacade.js"; -import { convertEventToVideo as sharedConvertEventToVideo } from "./nostr/index.js"; +import { convertChannelEvent, buildChannelVideoFilters } from "./channelProfileVideos.js"; import { DEFAULT_RELAY_URLS } from "./nostr/toolkit.js"; import { subscriptions } from "./subscriptions.js"; import { attachHealthBadges } from "./gridHealth.js"; @@ -4461,7 +4461,7 @@ function normalizeRenderableChannelVideo(entry) { typeof entry.kind === "number"; if (looksLikeRawEvent) { - const converted = sharedConvertEventToVideo(entry); + const converted = convertChannelEvent(entry); if (converted && !converted.invalid) { return converted; } @@ -5382,13 +5382,8 @@ async function loadUserVideos(pubkey) { }); try { - // 1) Build filter for videos from this pubkey - const filter = { - kinds: [30078], - authors: [pubkey], - "#t": ["video"], - limit: 200 - }; + // 1) Build filters for this pubkey's videos: native kind-30078 + NIP-71. + const filters = buildChannelVideoFilters(pubkey); // 2) Collect raw events from all relays const events = []; @@ -5401,7 +5396,7 @@ async function loadUserVideos(pubkey) { try { const fallbackEvents = await nostrClient.pool.list( Array.from(DEFAULT_RELAY_URLS), - [filter] + filters ); if (Array.isArray(fallbackEvents)) { events.push(...fallbackEvents); @@ -5412,7 +5407,7 @@ async function loadUserVideos(pubkey) { } else { const settled = await pMapSettled( relayList, - (url) => nostrClient.pool.list([url], [filter]), + (url) => nostrClient.pool.list([url], filters), { concurrency: RELAY_BACKGROUND_CONCURRENCY } ); settled.forEach((result, index) => { diff --git a/js/channelProfileVideos.js b/js/channelProfileVideos.js new file mode 100644 index 00000000..474d3de7 --- /dev/null +++ b/js/channelProfileVideos.js @@ -0,0 +1,29 @@ +// Channel-profile video sourcing: a creator's wall should show BOTH their +// bitvid-native kind-30078 videos AND the NIP-71 videos they published via other +// apps (Nostube etc.) — matching what the main feed shows via ingest. Previously +// the profile fetched only kind-30078, so cross-posted NIP-71 videos were absent. + +import { convertEventToVideo as sharedConvertEventToVideo } from "./nostr/index.js"; +import { buildVideoFromNip71Event, NIP71_KINDS } from "./nostr/nip71IngestAdapter.js"; + +// Two filters for one author: native video notes (kind 30078, tagged "video") +// and their NIP-71 videos (kinds 21/22/34235/34236). +export function buildChannelVideoFilters(pubkey, { limit = 200 } = {}) { + return [ + { kinds: [30078], authors: [pubkey], "#t": ["video"], limit }, + { kinds: Array.from(NIP71_KINDS), authors: [pubkey], limit }, + ]; +} + +// Convert a raw channel event to a bitvid video object. Foreign NIP-71 kinds go +// through the ingest adapter (which also rejects bitvid's own outbound mirrors so +// they don't duplicate the canonical kind-30078 note); everything else uses the +// standard kind-30078 converter. May return an { invalid: true } object — the +// caller filters those out (same contract as convertEventToVideo). +export function convertChannelEvent(entry) { + const kind = Number(entry?.kind); + if (NIP71_KINDS.has(kind)) { + return buildVideoFromNip71Event(entry); + } + return sharedConvertEventToVideo(entry); +} diff --git a/tests/channel-profile-videos.test.mjs b/tests/channel-profile-videos.test.mjs new file mode 100644 index 00000000..36c73809 --- /dev/null +++ b/tests/channel-profile-videos.test.mjs @@ -0,0 +1,81 @@ +// Channel-profile wall must source BOTH native kind-30078 videos and the +// creator's NIP-71 videos (cross-posted via other apps), while skipping bitvid's +// own outbound mirrors so they don't duplicate the canonical note. + +import assert from "node:assert/strict"; +import test from "node:test"; +import { + buildChannelVideoFilters, + convertChannelEvent, +} from "../js/channelProfileVideos.js"; + +const PK = "a".repeat(64); + +test("builds a native (30078) filter AND a NIP-71 filter for the author", () => { + const filters = buildChannelVideoFilters(PK); + assert.equal(filters.length, 2); + + const native = filters.find((f) => f.kinds.includes(30078)); + assert.deepEqual(native.authors, [PK]); + assert.deepEqual(native["#t"], ["video"]); + + const nip71 = filters.find((f) => !f.kinds.includes(30078)); + assert.deepEqual(nip71.kinds.slice().sort((a, b) => a - b), [21, 22, 34235, 34236]); + assert.deepEqual(nip71.authors, [PK]); +}); + +test("converts a foreign NIP-71 event into a renderable video", () => { + const v = convertChannelEvent({ + id: "evt", + pubkey: PK, + kind: 34235, + created_at: 1000, + content: "desc", + tags: [ + ["d", "root"], + ["title", "Nostube clip"], + ["imeta", "url https://example.com/v.mp4", "m video/mp4"], + ], + }); + assert.equal(v.invalid, false); + assert.equal(v.title, "Nostube clip"); + assert.equal(v.source, "nip71-ingest"); +}); + +test("skips bitvid's own mirror (returns invalid so the caller drops it)", () => { + const v = convertChannelEvent({ + id: "mirror", + pubkey: PK, + kind: 34235, + created_at: 1000, + content: "", + tags: [ + ["d", "root"], + ["client", "bitvid"], + ["title", "Our mirror"], + ["imeta", "url https://example.com/v.mp4", "m video/mp4"], + ], + }); + assert.equal(v.invalid, true); + assert.equal(v.reason, "bitvid-mirror"); +}); + +test("routes kind-30078 events through the standard converter", () => { + const v = convertChannelEvent({ + id: "native", + pubkey: PK, + kind: 30078, + created_at: 2000, + content: JSON.stringify({ + version: 3, + title: "Native bitvid video", + videoRootId: "vr", + url: "https://example.com/native.mp4", + }), + tags: [["d", "vr"], ["t", "video"]], + }); + assert.equal(v.invalid, false); + assert.equal(v.title, "Native bitvid video"); + // Not stamped as ingested — it's a native note. + assert.notEqual(v.source, "nip71-ingest"); +}); diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index 85c817cf..a20e5b91 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -317,12 +317,10 @@ notes below. Summary: - Verified live: Goblinbox (Nostube) content renders; liveness check works. ### 17b. Channel Profile wall shows only bitvid videos, not NIP-71 (LAUNCH-BLOCKER) -- [ ] The channel-profile grid (`js/channelProfile.js`, ~line 5387) fetches ONLY - `{ kinds:[30078], authors:[pubkey], "#t":["video"] }`, so a creator's - NIP-71 videos (kinds 21/22/34235/34236 — incl. their Nostube uploads) never - appear on their profile wall even though they show in the main feed via - ingest. Fix: also fetch the author's NIP-71 video kinds and merge them - (convert via `buildVideoFromNip71Event`, skip bitvid mirrors, dedupe by - videoRootId, native-30078 wins a shared root — same precedence as ingest). - Consider reusing the ingested videos already in `nostrClient.allEvents` for - that author rather than a fresh fetch. +- [x] **Fixed** (`js/channelProfileVideos.js`): the channel grid now fetches the + author's NIP-71 videos (kinds 21/22/34235/34236) alongside kind-30078 via + `buildChannelVideoFilters`, and `convertChannelEvent` routes NIP-71 events + through `buildVideoFromNip71Event` (skips bitvid mirrors). Existing + dedupe-by-root + canAccess in `buildRenderableChannelVideos` handle + precedence/gating. Logic extracted to a helper so channelProfile.js stayed + under its size cap. Tests + mutation-verified. From 0112903a6d746f13b67111418ff219859b110b66 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 19:28:15 -0400 Subject: [PATCH 34/43] docs: NIP-71 on-boarding + cross-ecosystem dedup dev plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds docs/nip71-onboarding-plan.md and pre-launch TODO 17c for the agreed two-way NIP-71 interop: - Phase 1 (first): cross-ecosystem dedup — prefer the bitvid (30078) version, show exactly one in every grid; identity = author + import-link/hash/infohash. - Phase 2: creator-initiated self-import (model A) — re-host to the creator's storage (CDN) + WebTorrent, publish a signed 30078 with an imported-from provenance tag. No forging others' notes; no instance auto-rehost. Planning only; no code changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- docs/nip71-onboarding-plan.md | 126 +++++++++++++++++++++++++++++ todo/TODO_2026-06-20_pre-launch.md | 18 +++++ 2 files changed, 144 insertions(+) create mode 100644 docs/nip71-onboarding-plan.md diff --git a/docs/nip71-onboarding-plan.md b/docs/nip71-onboarding-plan.md new file mode 100644 index 00000000..0abe1874 --- /dev/null +++ b/docs/nip71-onboarding-plan.md @@ -0,0 +1,126 @@ +# NIP-71 On-Boarding + Cross-Ecosystem Dedup — Dev Plan + +Status: **planned** (not started). Owner: TBD. Branch: `unstable`. +Companion to `docs/nip71-migration-plan.md` (off-boarding / mirror / ingest, all shipped). + +## Goal + +Make NIP-71 interop a **two-way street**: + +1. **Off-boarding (DONE):** a bitvid video (kind 30078) is mirrored *out* to NIP-71 + (addressable 34235/36) so other apps can discover it. +2. **On-boarding (THIS PLAN):** a creator's NIP-71 video published via another app + can be pulled *into* bitvid so it gains bitvid's advantages — CDN-hosted URL, + a WebTorrent source, and (optionally) bitvid extras. + +And the invariant that makes both safe to coexist: + +3. **One video, never two.** When the same video exists in both ecosystems, always + prefer the **bitvid** (kind-30078) version, show **exactly one**, in **every** + grid/list/scenario. Never both. + +## The hard constraint (why on-boarding is creator-initiated) + +Off-boarding works because the bitvid creator owns the video and signs BOTH events +with their own key. On-boarding is **not symmetric**: a foreign NIP-71 video is +signed by the foreign creator's key, and bitvid cannot forge a kind-30078 note +authored by someone else (no private key). Therefore: + +- **On-boarding = creator-initiated self-import only.** A creator who published + NIP-71 videos elsewhere logs into bitvid **with the same npub**, and bitvid + offers to import them: re-host the file to **their own** connected storage (CDN), + optionally seed WebTorrent, and publish a bitvid kind-30078 version **signed by + their key**. No auto-rehosting of other people's files (consent / cost / + liability). This mirrors off-boarding exactly: both directions author-owned and + consensual. + +## Locked decisions (from design discussion) + +- On-boarding model: **A — creator-initiated self-import.** +- Dedup identity: **explicit import-link + content hash** (`fileSha256`/`ox`) + + infohash (`i`/magnet btih). NO fuzzy url/title matching (avoids false merges). + Note: hash-less foreign videos (e.g. url+image only) won't dedup unless they + were on-boarded (which adds an explicit link) — acceptable. +- Advantages on import: **CDN url + WebTorrent source** (baseline). bitvid extras + (comments, view counts, edit-in-place, private toggle) come free once it's a + real 30078 note; no extra work. +- Cadence: **one-shot import with a re-run button** (not continuous lockstep). + Re-running re-imports anything new. + +## Phasing — dedup FIRST (it's needed regardless of on-boarding) + +### Phase 1 — Cross-ecosystem dedup ("prefer bitvid, show one, everywhere") + +Why first: even today, a creator who independently posts to bitvid AND Nostube can +produce two cards. And the moment on-boarding exists, the original + imported +versions both exist on relays. Dedup must land before/with on-boarding. + +- **Identity key:** `author + (importLink ‖ fileSha256 ‖ ox ‖ infoHash)`. + - `importLink`: pointer an on-boarded 30078 carries to the original NIP-71 + event/address (added in Phase 2). Most reliable. + - Hashes/infohash already extracted by `convertEventToVideo` (30078) and + `buildVideoFromNip71Event` (foreign) — confirm both populate `fileSha256` / + `originalFileSha256` / `infoHash`. +- **Preference:** within an identity group, keep the bitvid version + (`source !== "nip71-ingest"` / kind 30078); else keep the NIP-71 one. +- **Placement (one chokepoint → all grids):** the feed AND channel profile already + call `app.dedupeVideosByRoot(...)`. Add a pure `collapseCrossEcosystem(videos)` + pass and either chain it after root-dedup inside that shared helper, or wrap + both into one `dedupeVideos(videos)` used everywhere (feed engine + `createDedupeByRootStage`, `buildRenderableChannelVideos`, My Videos, search, + playlists, watch history). Audit every list builder for the shared call. +- **Bitvid's own outbound mirrors** are already skipped at ingest + (`client=bitvid` + `a` back-pointer) — keep that; this pass handles the rest. +- **Tests (cheat-resistant, pure):** same-hash bitvid+foreign → only bitvid kept; + foreign-only (no bitvid) → foreign kept; distinct videos not merged; hash-less + pair NOT merged (documented limitation); import-link match → original hidden. + Mutation-verify the prefer-bitvid selection and the identity grouping. + +### Phase 2 — Creator-initiated on-board import + +- **Discovery:** on login, find the logged-in creator's own NIP-71 videos (reuse + the ingest subscription/adapter, scoped to `authors:[viewerPubkey]`) that have + **no** corresponding bitvid 30078 yet (via the Phase-1 identity key). +- **UI:** a "Import your videos from other apps" surface (My Videos pane or a + one-time prompt): list importable items, let the creator pick, show what import + does. Requires their **storage connected** (re-host target) and signer. +- **Import action (per video), reusing existing plumbing:** + 1. Fetch the source file from the foreign `url` (or resolve via magnet). + 2. Re-host to the creator's connected storage (reuse `storageService` / + `r2Service` upload path used by normal uploads) → bitvid CDN url. + 3. Optionally create a `.torrent` + WebTorrent magnet (reuse upload's torrent + path) so the video gets a P2P source. + 4. Build + publish a kind-30078 note **signed by the creator** (reuse + `videoPublisher` / `buildVideoPostEvent`), carrying: title, description, + hashtags, dims/duration (from the foreign imeta via the adapter), the new + CDN url + magnet, AND an explicit `["imported-from", "<orig event id / a-addr>"]` + provenance tag for dedup + attribution. + 5. (Optional, symmetric) since it's now a normal bitvid video, the existing + outbound mirror/auto-share can keep the NIP-71 side in lockstep going forward. +- **Result:** the bitvid 30078 is now canonical; Phase-1 dedup hides the original + foreign event everywhere; the creator gets CDN + WebTorrent + bitvid features. +- **Guards:** + - Only the **same-pubkey** creator can import their own videos (signer == author). + - Never import NSFW outward against `ALLOW_NSFW_CONTENT` (mirror the existing + gate); carry `content-warning → isNsfw`. + - Re-host uses the creator's storage only; surface cost/ownership in the copy. + - One-shot; a "re-scan / import new" button re-runs discovery. + +## Non-goals / future + +- No instance-level auto-rehosting of arbitrary creators' files (model B) — consent/ + cost/liability. Revisit only with explicit per-instance policy + creator opt-in. +- No fuzzy (url/title) dedup — revisit only if hash-less dual-posts become a real + problem. +- Continuous lockstep on the inbound side (re-import on the creator editing the + original elsewhere) — deferred; one-shot + re-run is enough for launch. + +## Reuse map (existing code to lean on) + +- `js/nostr/nip71IngestAdapter.js` — foreign event → video (hashes, dims, imeta). +- `js/nostr/nip71Mirror.js` — outbound mirror (keeps NIP-71 side in lockstep after import). +- `js/services/nip71IngestService.js` — author-scoped subscription pattern for discovery. +- `js/channelProfileVideos.js`, feed `createDedupeByRootStage`, `app.dedupeVideosByRoot` + — the dedup chokepoints to extend. +- `storageService` / `r2Service` / upload + torrent paths — re-host + seed. +- `videoPublisher` / `buildVideoPostEvent` — publish the signed 30078. diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index a20e5b91..b1b93f80 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -316,6 +316,24 @@ notes below. Summary: nip71.publishedAt so the stage short-circuits. - Verified live: Goblinbox (Nostube) content renders; liveness check works. +### 17c. NIP-71 on-boarding + cross-ecosystem dedup (IMPORTANT — full plan in docs/nip71-onboarding-plan.md) +Make NIP-71 interop two-way and guarantee a video is never shown twice. +**See `docs/nip71-onboarding-plan.md` for the full dev plan.** Locked decisions: +creator-initiated self-import (model A — can't forge others' notes), dedup on +explicit import-link + content hash/infohash (no fuzzy url/title), import grants +CDN url + WebTorrent (bitvid extras come free), one-shot import + re-run button. +- [ ] **Phase 1 (do first): cross-ecosystem dedup** — `collapseCrossEcosystem` + keyed on `author + (importLink ‖ fileSha256 ‖ ox ‖ infoHash)`, prefer the + bitvid (kind-30078) version, applied at the shared dedupe chokepoint so + EVERY grid (feed, channel profile, My Videos, search, playlists, watch + history) collapses dupes. Pure + mutation-tested. Needed regardless of + on-boarding (independent dual-posts already double up). +- [ ] **Phase 2: creator-initiated import** — discover the logged-in creator's own + NIP-71 videos lacking a bitvid version; re-host the file to THEIR storage + (CDN) + optional WebTorrent seed; publish a kind-30078 signed by them with an + `["imported-from", <orig>]` provenance tag. Same-pubkey-only guard; + ALLOW_NSFW gate; reuses ingest adapter + storage/upload + videoPublisher. + ### 17b. Channel Profile wall shows only bitvid videos, not NIP-71 (LAUNCH-BLOCKER) - [x] **Fixed** (`js/channelProfileVideos.js`): the channel grid now fetches the author's NIP-71 videos (kinds 21/22/34235/34236) alongside kind-30078 via From c404618137484419a0cf36390555ee68ba5b2970 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 19:30:42 -0400 Subject: [PATCH 35/43] docs: flag externally-managed storage for NIP-71 reference imports Imports not produced by bitvid's storage system (reference imports / external URLs) must be explicitly flagged externally-managed so storage tooling never treats them as bitvid-owned: external badge (never "missing from bucket"), excluded from orphan reconciliation, delete "file left behind" warning suppressed, liveness unverifiable. Adds import modes (full re-host vs reference) + Phase 2b to the plan and TODO 17c. Notes the existing isUrlUnderBase heuristic is the baseline but a durable flag is needed. Planning only; no code changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- docs/nip71-onboarding-plan.md | 36 +++++++++++++++++++++++++++++- todo/TODO_2026-06-20_pre-launch.md | 16 +++++++++---- 2 files changed, 47 insertions(+), 5 deletions(-) diff --git a/docs/nip71-onboarding-plan.md b/docs/nip71-onboarding-plan.md index 0abe1874..6098f300 100644 --- a/docs/nip71-onboarding-plan.md +++ b/docs/nip71-onboarding-plan.md @@ -97,8 +97,42 @@ versions both exist on relays. Dedup must land before/with on-boarding. provenance tag for dedup + attribution. 5. (Optional, symmetric) since it's now a normal bitvid video, the existing outbound mirror/auto-share can keep the NIP-71 side in lockstep going forward. +- **Import modes (decision):** offer both + - **Full import (re-host):** file copied to the creator's storage → bitvid-managed + (CDN url under their bucket), normal storage tooling applies. + - **Reference import (no re-host):** publish a 30078 that keeps the **external** + URL (lighter, no storage cost). The video is now a bitvid note but its file is + **externally managed** — see storage-provenance below. + - **Result:** the bitvid 30078 is now canonical; Phase-1 dedup hides the original - foreign event everywhere; the creator gets CDN + WebTorrent + bitvid features. + foreign event everywhere; the creator gets bitvid features + (full import) CDN + + WebTorrent, or (reference import) discovery without re-hosting. + +### Storage provenance — flag externally-managed imports + +Any imported (or otherwise bitvid-converted) video whose file was **not** produced +by our storage system must be explicitly flagged as **externally managed** so the +storage tooling never treats it as a bitvid-owned object. + +- **Today:** `isUrlUnderBase(url, publicBaseUrl)` (myVideosHealth.js) already + classifies bucket vs external ("Hosted URL" vs "External URL"), and + `reconcileStorage` only matches files under the base. So a reference-imported + external URL is *mostly* handled by the heuristic. +- **Add (durable + explicit):** stamp the on-boarded note as externally managed + rather than relying solely on the URL-vs-base heuristic (which breaks if the + creator changes their storage base, or an external host coincidentally matches). + Carry it on the video object (e.g. `externalStorage: true`, derived from the + `imported-from`/reference-import path) and, where appropriate, on the event. +- **All storage surfaces must honor it:** + - **My Videos health:** show an "External URL / externally managed" badge; never + flag as missing-from-bucket. + - **Orphan reconciliation:** exclude — never list as an orphan, never offer + bucket delete for a file we don't own. + - **Delete flow:** suppress the "a hosted file is left behind" storage-cleanup + warning for externally-managed videos (there's nothing in our bucket to clean). + - **Liveness:** external URL stays "unverifiable" (existing behavior). +- **Full import** sets `externalStorage: false` (it IS in the creator's bucket) and + behaves like any native upload. - **Guards:** - Only the **same-pubkey** creator can import their own videos (signer == author). - Never import NSFW outward against `ALLOW_NSFW_CONTENT` (mirror the existing diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index b1b93f80..1a24eb57 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -329,10 +329,18 @@ CDN url + WebTorrent (bitvid extras come free), one-shot import + re-run button. history) collapses dupes. Pure + mutation-tested. Needed regardless of on-boarding (independent dual-posts already double up). - [ ] **Phase 2: creator-initiated import** — discover the logged-in creator's own - NIP-71 videos lacking a bitvid version; re-host the file to THEIR storage - (CDN) + optional WebTorrent seed; publish a kind-30078 signed by them with an - `["imported-from", <orig>]` provenance tag. Same-pubkey-only guard; - ALLOW_NSFW gate; reuses ingest adapter + storage/upload + videoPublisher. + NIP-71 videos lacking a bitvid version; offer FULL import (re-host to THEIR + storage + optional WebTorrent) or REFERENCE import (keep external URL, no + re-host); publish a kind-30078 signed by them with an `["imported-from", <orig>]` + provenance tag. Same-pubkey-only guard; ALLOW_NSFW gate; reuses ingest adapter + + storage/upload + videoPublisher. +- [ ] **Phase 2b: externally-managed storage flag** — imports NOT produced by our + storage system (reference imports / external URLs) must be explicitly flagged + (`externalStorage: true`, durable — not just the `isUrlUnderBase` heuristic) so + storage tooling never treats them as bitvid-owned: My Videos shows an external + badge (never "missing from bucket"), orphan reconciliation excludes them, the + delete "hosted file left behind" warning is suppressed, liveness stays + unverifiable. Full (re-hosted) imports set the flag false. ### 17b. Channel Profile wall shows only bitvid videos, not NIP-71 (LAUNCH-BLOCKER) - [x] **Fixed** (`js/channelProfileVideos.js`): the channel grid now fetches the From 065706a3cfcfa46c078a8bbd644af94535caf81b Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 19:51:43 -0400 Subject: [PATCH 36/43] videos: cross-ecosystem dedup (prefer bitvid, show one) across all grids MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 1 of the on-boarding plan, and the shared layer for unified grid loading. A video that exists as both a bitvid kind-30078 note and a NIP-71 mirror/original previously could appear twice (e.g. independent dual-posts). js/utils/videoDeduper.js: collapseCrossEcosystem() groups events by a per-author identity (fileSha256 / originalFileSha256 / infoHash, plus an `eid` namespace that links an on-boarded note's `importedFrom` to the original foreign event) via union-find, and keeps the bitvid-native version (else newest). dedupeVideos() = root-dedup THEN cross-ecosystem collapse. Routed through the shared chokepoint app.dedupeVideosByRoot (playback coordinator), so the feed stage, channel profile, and search all collapse duplicates identically — "show one, prefer bitvid" everywhere. Channel fallback path updated too. Limitation (documented): hash-less independent dual-posts can't be matched until the bitvid side carries on-board provenance (Phase 2). Tests + mutation-verified (prefer-bitvid, per-author identity, infohash, provenance link, root+cross composition). 1262/1263 unit (known flake). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/app.js | 3 +- js/app/playbackCoordinator.js | 7 +- js/channelProfile.js | 3 +- js/utils/videoDeduper.js | 121 ++++++++++++++++++++++++++ tests/cross-ecosystem-dedupe.test.mjs | 111 +++++++++++++++++++++++ todo/TODO_2026-06-20_pre-launch.md | 14 +-- 6 files changed, 250 insertions(+), 9 deletions(-) create mode 100644 tests/cross-ecosystem-dedupe.test.mjs diff --git a/js/app.js b/js/app.js index ad5ce4fc..59db1b8a 100644 --- a/js/app.js +++ b/js/app.js @@ -135,7 +135,7 @@ import resolveVideoPointer, { buildVideoAddressPointer, } from "./utils/videoPointer.js"; import { isValidMagnetUri } from "./utils/magnetValidators.js"; -import { dedupeToNewestByRoot } from "./utils/videoDeduper.js"; +import { dedupeToNewestByRoot, dedupeVideos } from "./utils/videoDeduper.js"; import { buildServiceWorkerFallbackStatus } from "./utils/serviceWorkerFallbackMessages.js"; import { batchFetchProfilesFromRelays } from "./utils/profileBatchFetcher.js"; import { @@ -633,6 +633,7 @@ class Application { fetchProfileMetadata, ensureProfileMetadataSubscription, dedupeToNewestByRoot, + dedupeVideos, buildServiceWorkerFallbackStatus, sanitizeProfileMediaUrl, UNSUPPORTED_BTITH_MESSAGE, diff --git a/js/app/playbackCoordinator.js b/js/app/playbackCoordinator.js index 78dbedab..e9f00649 100644 --- a/js/app/playbackCoordinator.js +++ b/js/app/playbackCoordinator.js @@ -56,6 +56,7 @@ export function createPlaybackCoordinator(deps) { fetchProfileMetadata, ensureProfileMetadataSubscription, dedupeToNewestByRoot, + dedupeVideos, buildServiceWorkerFallbackStatus, sanitizeProfileMediaUrl, UNSUPPORTED_BTITH_MESSAGE, @@ -216,7 +217,11 @@ export function createPlaybackCoordinator(deps) { if (!Array.isArray(videos) || videos.length === 0) { return []; } - return dedupeToNewestByRoot(videos); + // Root-dedupe AND collapse cross-ecosystem duplicates (prefer bitvid). This + // is the shared chokepoint for the feed stage + channel/search grids. + return typeof dedupeVideos === "function" + ? dedupeVideos(videos) + : dedupeToNewestByRoot(videos); }, autoplayModalVideo() { diff --git a/js/channelProfile.js b/js/channelProfile.js index a348d47b..c670a116 100644 --- a/js/channelProfile.js +++ b/js/channelProfile.js @@ -2,6 +2,7 @@ import { nostrClient } from "./nostrClientFacade.js"; import { convertChannelEvent, buildChannelVideoFilters } from "./channelProfileVideos.js"; +import { dedupeVideos } from "./utils/videoDeduper.js"; import { DEFAULT_RELAY_URLS } from "./nostr/toolkit.js"; import { subscriptions } from "./subscriptions.js"; import { attachHealthBadges } from "./gridHealth.js"; @@ -4512,7 +4513,7 @@ function buildRenderableChannelVideos({ events = [], app } = {}) { const newestByRoot = app?.dedupeVideosByRoot?.(uniqueVideos) ?? - dedupeToNewestByRoot(uniqueVideos); + dedupeVideos(uniqueVideos); let videos = newestByRoot.filter((video) => !video.deleted); videos = videos.filter((video) => { diff --git a/js/utils/videoDeduper.js b/js/utils/videoDeduper.js index bfbe8c05..928d820d 100644 --- a/js/utils/videoDeduper.js +++ b/js/utils/videoDeduper.js @@ -34,4 +34,125 @@ export function dedupeToNewestByRoot(videos) { return Array.from(map.values()); } +// Identity keys that mark two events as "the same video" across ecosystems +// (bitvid kind-30078 vs a NIP-71 mirror/original). Scoped per author. Content +// hashes + infohash are the reliable signals; `importedFrom` (Phase 2 on-board +// provenance) links an on-boarded bitvid note to the original foreign event via a +// shared "eid" namespace. Videos with none of these only ever match themselves. +function crossEcosystemIdentityKeys(video) { + const author = + typeof video?.pubkey === "string" ? video.pubkey.trim().toLowerCase() : ""; + if (!author) { + return []; + } + const keys = []; + const push = (ns, value) => { + const s = typeof value === "string" ? value.trim().toLowerCase() : ""; + if (s) { + keys.push(`${author}|${ns}|${s}`); + } + }; + push("sha", video.fileSha256); + push("ox", video.originalFileSha256); + push("ih", video.infoHash); + push("eid", video.id); + push("eid", video.importedFrom); + return keys; +} + +function isBitvidNative(video) { + return video?.source !== "nip71-ingest" && video?.foreign !== true; +} + +// Within a group of same-video events, prefer the bitvid-native (kind-30078) +// version; among the same kind, the newest (tie: higher id) for determinism. +function pickPreferredVideo(group) { + let best = null; + for (const video of group) { + if (!best) { + best = video; + continue; + } + const nativeV = isBitvidNative(video); + const nativeBest = isBitvidNative(best); + if (nativeV !== nativeBest) { + if (nativeV) best = video; + continue; + } + const tV = Number(video.created_at); + const tBest = Number(best.created_at); + const cV = Number.isFinite(tV) ? tV : -Infinity; + const cBest = Number.isFinite(tBest) ? tBest : -Infinity; + if (cV > cBest || (cV === cBest && String(video.id) > String(best.id))) { + best = video; + } + } + return best; +} + +/** + * Collapse duplicates of the same video that appear in BOTH ecosystems (a bitvid + * kind-30078 note and a NIP-71 mirror/original) so a grid never shows both. + * Always prefers the bitvid version. Same-ecosystem, distinct videos are + * untouched. Pure. + */ +export function collapseCrossEcosystem(videos) { + const list = Array.isArray(videos) ? videos.filter(Boolean) : []; + if (list.length < 2) { + return list.slice(); + } + + // Union-find: merge any videos that share an identity key. + const parent = list.map((_, i) => i); + const find = (i) => { + let root = i; + while (parent[root] !== root) root = parent[root]; + while (parent[i] !== root) { + const next = parent[i]; + parent[i] = root; + i = next; + } + return root; + }; + const union = (a, b) => { + const ra = find(a); + const rb = find(b); + if (ra !== rb) parent[ra] = rb; + }; + + const keyToIndex = new Map(); + list.forEach((video, i) => { + for (const key of crossEcosystemIdentityKeys(video)) { + if (keyToIndex.has(key)) { + union(keyToIndex.get(key), i); + } else { + keyToIndex.set(key, i); + } + } + }); + + const groups = new Map(); + list.forEach((video, i) => { + const root = find(i); + const group = groups.get(root); + if (group) group.push(video); + else groups.set(root, [video]); + }); + + const result = []; + for (const group of groups.values()) { + result.push(group.length === 1 ? group[0] : pickPreferredVideo(group)); + } + return result; +} + +/** + * The shared grid dedupe: collapse to newest-per-root, THEN collapse + * cross-ecosystem duplicates (prefer bitvid). Use this everywhere a video list is + * built so every grid behaves identically. + */ +export function dedupeVideos(videos) { + return collapseCrossEcosystem(dedupeToNewestByRoot(videos)); +} + export default dedupeToNewestByRoot; diff --git a/tests/cross-ecosystem-dedupe.test.mjs b/tests/cross-ecosystem-dedupe.test.mjs new file mode 100644 index 00000000..25df7501 --- /dev/null +++ b/tests/cross-ecosystem-dedupe.test.mjs @@ -0,0 +1,111 @@ +// Cross-ecosystem dedupe: when the same video exists as a bitvid kind-30078 note +// AND a NIP-71 mirror/original, show exactly one — always the bitvid version. +// This is the shared pass behind every grid (feed, channel, search, ...). + +import assert from "node:assert/strict"; +import test from "node:test"; +import { + collapseCrossEcosystem, + dedupeVideos, +} from "../js/utils/videoDeduper.js"; + +const PK = "a".repeat(64); +const SHA = "f".repeat(64); + +const bitvid = (over = {}) => ({ + id: "bitvid-id", + pubkey: PK, + videoRootId: "bitvid-root", + created_at: 100, + // native: no nip71-ingest source + ...over, +}); +const foreign = (over = {}) => ({ + id: "foreign-id", + pubkey: PK, + videoRootId: "foreign-root", + created_at: 200, + source: "nip71-ingest", + foreign: true, + ...over, +}); + +test("same file hash → keeps the bitvid version, drops the foreign one", () => { + const out = collapseCrossEcosystem([ + foreign({ fileSha256: SHA }), + bitvid({ fileSha256: SHA }), + ]); + assert.equal(out.length, 1); + assert.equal(out[0].id, "bitvid-id"); +}); + +test("prefers bitvid even when the foreign version is NEWER", () => { + const out = collapseCrossEcosystem([ + bitvid({ fileSha256: SHA, created_at: 1 }), + foreign({ fileSha256: SHA, created_at: 9999 }), + ]); + assert.equal(out.length, 1); + assert.equal(out[0].id, "bitvid-id"); +}); + +test("matches on infohash too", () => { + const ih = "1".repeat(40); + const out = collapseCrossEcosystem([ + foreign({ infoHash: ih }), + bitvid({ infoHash: ih }), + ]); + assert.deepEqual(out.map((v) => v.id), ["bitvid-id"]); +}); + +test("matches via on-board provenance link (importedFrom -> foreign event id)", () => { + const out = collapseCrossEcosystem([ + foreign({ id: "orig-evt" }), + bitvid({ id: "imported-note", importedFrom: "orig-evt" }), + ]); + assert.equal(out.length, 1); + assert.equal(out[0].id, "imported-note"); +}); + +test("distinct videos (no shared identity) are NOT merged", () => { + const out = collapseCrossEcosystem([ + bitvid({ id: "v1", fileSha256: "1".repeat(64) }), + foreign({ id: "v2", fileSha256: "2".repeat(64) }), + ]); + assert.equal(out.length, 2); +}); + +test("hash-less videos are not merged (documented limitation)", () => { + const out = collapseCrossEcosystem([ + bitvid({ id: "v1" }), + foreign({ id: "v2" }), + ]); + assert.equal(out.length, 2); +}); + +test("two foreign versions of one file collapse to the newest (no bitvid present)", () => { + const out = collapseCrossEcosystem([ + foreign({ id: "old", fileSha256: SHA, created_at: 100 }), + foreign({ id: "new", fileSha256: SHA, created_at: 200 }), + ]); + assert.equal(out.length, 1); + assert.equal(out[0].id, "new"); +}); + +test("dedupeVideos collapses by root THEN across ecosystems", () => { + // Two bitvid events same root (older + newer) + a foreign mirror by hash. + const out = dedupeVideos([ + bitvid({ id: "old", videoRootId: "r", created_at: 1, fileSha256: SHA }), + bitvid({ id: "new", videoRootId: "r", created_at: 2, fileSha256: SHA }), + foreign({ id: "mirror", fileSha256: SHA, created_at: 3 }), + ]); + assert.equal(out.length, 1); + assert.equal(out[0].id, "new", "newest bitvid per root wins, foreign mirror dropped"); +}); + +test("different authors with the same hash are NOT merged (identity is per-author)", () => { + const out = collapseCrossEcosystem([ + bitvid({ id: "mine", pubkey: PK, fileSha256: SHA }), + foreign({ id: "theirs", pubkey: "b".repeat(64), fileSha256: SHA }), + ]); + assert.equal(out.length, 2); +}); diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index 1a24eb57..5dbff124 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -322,12 +322,14 @@ Make NIP-71 interop two-way and guarantee a video is never shown twice. creator-initiated self-import (model A — can't forge others' notes), dedup on explicit import-link + content hash/infohash (no fuzzy url/title), import grants CDN url + WebTorrent (bitvid extras come free), one-shot import + re-run button. -- [ ] **Phase 1 (do first): cross-ecosystem dedup** — `collapseCrossEcosystem` - keyed on `author + (importLink ‖ fileSha256 ‖ ox ‖ infoHash)`, prefer the - bitvid (kind-30078) version, applied at the shared dedupe chokepoint so - EVERY grid (feed, channel profile, My Videos, search, playlists, watch - history) collapses dupes. Pure + mutation-tested. Needed regardless of - on-boarding (independent dual-posts already double up). +- [x] **Phase 1: cross-ecosystem dedup** (`js/utils/videoDeduper.js`): + `collapseCrossEcosystem` keyed per-author on `fileSha256/ox/infoHash` + an + `eid` namespace linking on-board provenance (`importedFrom`); prefers the + bitvid (kind-30078) version. Composed into `dedupeVideos` (root-dedup THEN + cross-ecosystem) and routed through the shared chokepoint + `app.dedupeVideosByRoot` → covers the feed stage + channel + search at once. + Pure + mutation-verified. (Hash-less dual-posts still can't dedup — documented + limitation; covered once on-boarded videos carry the provenance link.) - [ ] **Phase 2: creator-initiated import** — discover the logged-in creator's own NIP-71 videos lacking a bitvid version; offer FULL import (re-host to THEIR storage + optional WebTorrent) or REFERENCE import (keep external URL, no From 1075d3f822748be2de031865dc470ce22ecdac55 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 20:00:36 -0400 Subject: [PATCH 37/43] channel: optimistic render includes ingested NIP-71 videos MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit getCachedChannelVideoEvents early-returned kind-30078-only rawEvents and skipped allEvents, so a creator's ingested NIP-71 videos didn't show in the optimistic (pre-fetch) render. Now it merges raw 30078 (first, so they win the downstream id-dedupe) with processed allEvents (which holds ingested foreign videos) → the wall paints instantly on revisit / when the feed already loaded that author. Cross-ecosystem dedupe collapses any overlap. Also records TODO 17d (unified grid loading follow-ups): streaming channel/search fetch, per-channel persisted cache, My Videos NIP-71 inclusion, and consolidating bespoke loaders behind one shared streaming grid source. 1262/1263 unit (known UploadModal flake). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/channelProfile.js | 10 +++++----- todo/TODO_2026-06-20_pre-launch.md | 20 ++++++++++++++++++++ 2 files changed, 25 insertions(+), 5 deletions(-) diff --git a/js/channelProfile.js b/js/channelProfile.js index c670a116..b1ec7893 100644 --- a/js/channelProfile.js +++ b/js/channelProfile.js @@ -4428,10 +4428,10 @@ function getCachedChannelVideoEvents(pubkey) { } }); - if (rawMatches.length) { - return rawMatches; - } - + // Include processed videos from allEvents too (raw 30078 first so they win the + // id-dedupe downstream): this surfaces ingested NIP-71 videos for the author so + // their wall renders optimistically on revisit / when the feed already loaded + // them. Cross-ecosystem dedupe collapses any overlap. const processedMatches = collectMatches( nostrClient?.allEvents, (callback) => { @@ -4448,7 +4448,7 @@ function getCachedChannelVideoEvents(pubkey) { } ); - return processedMatches; + return [...rawMatches, ...processedMatches]; } function normalizeRenderableChannelVideo(entry) { diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index 5dbff124..8279ef57 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -344,6 +344,26 @@ CDN url + WebTorrent (bitvid extras come free), one-shot import + re-run button. delete "hosted file left behind" warning is suppressed, liveness stays unverifiable. Full (re-hosted) imports set the flag false. +### 17d. Unified video-grid loading (streaming + optimistic) — follow-ups +Goal: every grid loads with the same fast UX. Status: cross-ecosystem dedup +(17c Phase 1) is live at the shared chokepoint; main feed already streams + +has a persisted cache; channel/search now render optimistically from +`allEvents` (incl. ingested NIP-71). Remaining: +- [ ] **Streaming channel/search fetch.** `loadUserVideos` does a one-shot + `pMapSettled(relays, pool.list)` and renders only after ALL relays settle, + so a slow relay holds up the grid (~2-3s). Switch to the subscription + manager's streaming + render incrementally (first-relay-wins), like the main + feed's buffered subscription. Extract the fetch into `channelProfileVideos.js` + (channelProfile.js is at its size cap). +- [ ] **Per-channel persisted cache** for instant revisits (mirror the main feed's + `bitvid:filtered-videos` optimistic cache, keyed per author). +- [ ] **My Videos: include NIP-71** in its `allEvents` read so a creator's own + cross-posted videos appear in management too (reuse the cross-ecosystem + dedupe so the bitvid version wins). +- [ ] **Consolidate** the bespoke loaders (channel/search/My Videos) behind one + shared streaming grid source so future grids inherit streaming + optimistic + + dedupe for free. + ### 17b. Channel Profile wall shows only bitvid videos, not NIP-71 (LAUNCH-BLOCKER) - [x] **Fixed** (`js/channelProfileVideos.js`): the channel grid now fetches the author's NIP-71 videos (kinds 21/22/34235/34236) alongside kind-30078 via From 80f058f67c01b503331692d2fa0281a0b5c1b895 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 20:23:51 -0400 Subject: [PATCH 38/43] channel: per-channel persisted cache for instant cold-load paint MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A cold hard-refresh on a profile previously sat on "Loading videos…" for ~5s — the optimistic in-memory cache is empty on cold start (ingest hasn't injected yet) and the grid waited on relay connection. Now channelProfileVideos.js persists the last-rendered wall per author (localStorage bitvid:channel-videos:v1) and loadUserVideos paints it immediately on cold load, before relays connect; the live fetch then replaces it. Bounded (60 videos/channel, 20 channels LRU-evicted) and strips the heavy raw `tags` array. Same proven pattern as the main feed's optimistic cache. Closes TODO 17d per-channel cache. Tests + mutation-verified (round-trip, tag-strip, per-channel + channel-count caps). 1267/1268 unit (known flake). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/channelProfile.js | 10 +++-- js/channelProfileVideos.js | 56 ++++++++++++++++++++++++++ tests/channel-video-cache.test.mjs | 64 ++++++++++++++++++++++++++++++ todo/TODO_2026-06-20_pre-launch.md | 7 +++- 4 files changed, 132 insertions(+), 5 deletions(-) create mode 100644 tests/channel-video-cache.test.mjs diff --git a/js/channelProfile.js b/js/channelProfile.js index b1ec7893..e8e4935d 100644 --- a/js/channelProfile.js +++ b/js/channelProfile.js @@ -1,7 +1,7 @@ // js/channelProfile.js import { nostrClient } from "./nostrClientFacade.js"; -import { convertChannelEvent, buildChannelVideoFilters } from "./channelProfileVideos.js"; +import { convertChannelEvent, buildChannelVideoFilters, loadCachedChannelVideos, saveCachedChannelVideos } from "./channelProfileVideos.js"; import { dedupeVideos } from "./utils/videoDeduper.js"; import { DEFAULT_RELAY_URLS } from "./nostr/toolkit.js"; import { subscriptions } from "./subscriptions.js"; @@ -5353,10 +5353,13 @@ async function loadUserVideos(pubkey) { let renderedFromCache = false; if (container) { + // In-memory cache (warm) first; else the persisted per-channel cache so a + // COLD hard-refresh paints last-seen videos before relays connect. const cachedEvents = getCachedChannelVideoEvents(pubkey); - if (cachedEvents.length) { + const cacheSource = cachedEvents.length ? cachedEvents : loadCachedChannelVideos(pubkey); + if (cacheSource.length) { const cachedVideos = buildRenderableChannelVideos({ - events: cachedEvents, + events: cacheSource, app }); if (cachedVideos.length) { @@ -5438,6 +5441,7 @@ async function loadUserVideos(pubkey) { } const videos = buildRenderableChannelVideos({ events, app }); + saveCachedChannelVideos(pubkey, videos); const rendered = await renderChannelVideosFromList({ videos, container, diff --git a/js/channelProfileVideos.js b/js/channelProfileVideos.js index 474d3de7..766d8f3b 100644 --- a/js/channelProfileVideos.js +++ b/js/channelProfileVideos.js @@ -27,3 +27,59 @@ export function convertChannelEvent(entry) { } return sharedConvertEventToVideo(entry); } + +// ---- Per-channel persisted cache (instant cold-load paint) ------------------- +// Mirrors the main feed's optimistic cache: on a cold hard-refresh of a profile, +// render the last-seen videos from localStorage BEFORE relays connect, then the +// live fetch replaces them. Keyed per author; bounded so storage can't grow +// without limit. + +const CHANNEL_CACHE_KEY = "bitvid:channel-videos:v1"; +const MAX_CACHED_CHANNELS = 20; +const MAX_VIDEOS_PER_CHANNEL = 60; + +function normPubkey(pubkey) { + return typeof pubkey === "string" ? pubkey.trim().toLowerCase() : ""; +} + +function readChannelCacheMap() { + try { + if (typeof localStorage === "undefined") return {}; + const raw = localStorage.getItem(CHANNEL_CACHE_KEY); + const parsed = raw ? JSON.parse(raw) : {}; + return parsed && typeof parsed === "object" ? parsed : {}; + } catch (error) { + return {}; + } +} + +export function loadCachedChannelVideos(pubkey) { + const pk = normPubkey(pubkey); + if (!pk) return []; + const entry = readChannelCacheMap()[pk]; + return entry && Array.isArray(entry.videos) ? entry.videos : []; +} + +export function saveCachedChannelVideos(pubkey, videos) { + const pk = normPubkey(pubkey); + if (!pk || !Array.isArray(videos) || typeof localStorage === "undefined") { + return; + } + try { + const map = readChannelCacheMap(); + // Drop the heavy raw `tags` array — the render + dedupe paths use parsed + // top-level fields; the live fetch restores full objects moments later. + const slim = videos.slice(0, MAX_VIDEOS_PER_CHANNEL).map(({ tags, ...rest }) => rest); + map[pk] = { videos: slim, ts: Date.now() }; + const keys = Object.keys(map); + if (keys.length > MAX_CACHED_CHANNELS) { + keys + .sort((a, b) => (map[a]?.ts || 0) - (map[b]?.ts || 0)) + .slice(0, keys.length - MAX_CACHED_CHANNELS) + .forEach((k) => delete map[k]); + } + localStorage.setItem(CHANNEL_CACHE_KEY, JSON.stringify(map)); + } catch (error) { + // best-effort; cache is non-critical + } +} diff --git a/tests/channel-video-cache.test.mjs b/tests/channel-video-cache.test.mjs new file mode 100644 index 00000000..b270cece --- /dev/null +++ b/tests/channel-video-cache.test.mjs @@ -0,0 +1,64 @@ +// Per-channel persisted cache: instant cold-load paint of a profile wall. +// Round-trips last-seen videos via localStorage, strips heavy tags, and bounds +// both per-channel count and number of cached channels. + +import "./test-helpers/setup-localstorage.mjs"; +import assert from "node:assert/strict"; +import test from "node:test"; +import { + loadCachedChannelVideos, + saveCachedChannelVideos, +} from "../js/channelProfileVideos.js"; + +const PK = "a".repeat(64); + +test("round-trips videos for a channel, stripping raw tags", () => { + localStorage.clear(); + saveCachedChannelVideos(PK, [ + { id: "v1", title: "One", pubkey: PK, created_at: 1, url: "https://e/1.mp4", tags: [["e", "x"]] }, + ]); + const out = loadCachedChannelVideos(PK); + assert.equal(out.length, 1); + assert.equal(out[0].id, "v1"); + assert.equal(out[0].url, "https://e/1.mp4"); + assert.equal(out[0].tags, undefined, "heavy tags array not persisted"); +}); + +test("scoped per pubkey", () => { + localStorage.clear(); + saveCachedChannelVideos(PK, [{ id: "v1", title: "x", pubkey: PK, created_at: 1 }]); + assert.equal(loadCachedChannelVideos("b".repeat(64)).length, 0); +}); + +test("caps videos per channel (60)", () => { + localStorage.clear(); + const many = Array.from({ length: 100 }, (_, i) => ({ + id: `v${i}`, + title: `t${i}`, + pubkey: PK, + created_at: i, + })); + saveCachedChannelVideos(PK, many); + assert.equal(loadCachedChannelVideos(PK).length, 60); +}); + +test("evicts the least-recently-saved channel past the cap (20)", () => { + localStorage.clear(); + for (let i = 0; i < 22; i += 1) { + const pk = i.toString(16).padStart(64, "0"); + saveCachedChannelVideos(pk, [{ id: `v${i}`, title: "t", pubkey: pk, created_at: i }]); + } + // The two oldest channels (0,1) should have been evicted; the newest remains. + const oldest = "0".padStart(64, "0"); + const newest = (21).toString(16).padStart(64, "0"); + assert.equal(loadCachedChannelVideos(oldest).length, 0, "oldest evicted"); + assert.equal(loadCachedChannelVideos(newest).length, 1, "newest kept"); +}); + +test("missing / empty inputs are safe no-ops", () => { + localStorage.clear(); + saveCachedChannelVideos("", [{ id: "v" }]); // no pubkey + saveCachedChannelVideos(PK, null); // bad videos + assert.deepEqual(loadCachedChannelVideos(PK), []); + assert.deepEqual(loadCachedChannelVideos(""), []); +}); diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index 8279ef57..747031bc 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -355,8 +355,11 @@ has a persisted cache; channel/search now render optimistically from manager's streaming + render incrementally (first-relay-wins), like the main feed's buffered subscription. Extract the fetch into `channelProfileVideos.js` (channelProfile.js is at its size cap). -- [ ] **Per-channel persisted cache** for instant revisits (mirror the main feed's - `bitvid:filtered-videos` optimistic cache, keyed per author). +- [x] **Per-channel persisted cache** (`channelProfileVideos.js`, + `bitvid:channel-videos:v1`): cold hard-refresh of a profile now paints the + last-seen videos from localStorage before relays connect, then the live fetch + replaces them. Bounded (60 videos/channel, 20 channels LRU), strips raw tags. + Tests + mutation-verified. - [ ] **My Videos: include NIP-71** in its `allEvents` read so a creator's own cross-posted videos appear in management too (reuse the cross-ecosystem dedupe so the bitvid version wins). From f760abde3d546c5d3e990d2de841f0cdbc07e80d Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 21:00:30 -0400 Subject: [PATCH 39/43] nip71 ingest: cache into videosMap + faster refresh (consistent feed presence) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two symptoms: (1) on warm refresh, NIP-71 videos appear a few seconds after bitvid ones; (2) navigating profile -> home sometimes showed only bitvid. Root cause: ingest wrote only nostrClient.activeMap (the live feed-engine source), not nostrService.videosMap (the persisted/author-feed cache), and the refresh nudge was throttled at 8s — so NIP-71 entered the feed late and inconsistently across render paths. - Ingest now also nostrService.cacheVideos(injected) so the videos persist and appear in author-scoped feeds + optimistic/persisted render paths, not just the live activeMap engine run. - Refresh throttle 8s -> 2.5s so the post-inject feed re-render fires promptly. Tests + mutation-verified (injected videos forwarded to cacheVideos). 1268/1269 unit (known UploadModal flake). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/services/nip71IngestService.js | 18 +++++++++++++++--- tests/nip71-ingest-service.test.mjs | 13 +++++++++++++ 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/js/services/nip71IngestService.js b/js/services/nip71IngestService.js index f28e3cf7..16cd4958 100644 --- a/js/services/nip71IngestService.js +++ b/js/services/nip71IngestService.js @@ -26,7 +26,7 @@ const FLUSH_DEBOUNCE_MS = 200; // Coalesce ingest-driven feed refreshes so a burst of incoming events can't // trigger a re-render/re-subscription storm. The injected videos are already in // the active store; this only controls how often we *signal* the feed. -const REFRESH_THROTTLE_MS = 8000; +const REFRESH_THROTTLE_MS = 2500; const DEFAULT_LIMIT = 200; const SUBSCRIPTION_KEY = "nip71-ingest"; // The admin whitelist hydrates asynchronously (remote fetch, then a second hex @@ -157,10 +157,22 @@ export function createNip71IngestService({ } } - let injected = 0; + const injectedVideos = []; for (const video of newestByRoot.values()) { if (injectVideo(video)) { - injected += 1; + injectedVideos.push(video); + } + } + const injected = injectedVideos.length; + + // Also cache into nostrService.videosMap so ingested videos persist and show + // up in the author-scoped feeds + the optimistic/persisted render paths — not + // just the live activeMap engine run. Keeps NIP-71 present across navigation. + if (injected > 0 && typeof nostrService?.cacheVideos === "function") { + try { + nostrService.cacheVideos(injectedVideos); + } catch (error) { + logger?.warn?.("[nip71Ingest] cacheVideos failed", error); } } diff --git a/tests/nip71-ingest-service.test.mjs b/tests/nip71-ingest-service.test.mjs index 41b7918b..5db3c472 100644 --- a/tests/nip71-ingest-service.test.mjs +++ b/tests/nip71-ingest-service.test.mjs @@ -44,6 +44,7 @@ function makeHarness({ whitelistOn = false, whitelist = [] } = {}) { getSubscriptionManager: () => manager, }; const feedListeners = new Set(); + const cached = []; const nostrService = { emit(name, detail) { emitted.push({ name, detail }); @@ -55,6 +56,7 @@ function makeHarness({ whitelistOn = false, whitelist = [] } = {}) { return () => feedListeners.delete(fn); }, getFilteredActiveVideos: () => Array.from(activeMap.values()), + cacheVideos: (videos) => cached.push(...videos), }; const accessControl = { whitelistMode: () => whitelistOn, @@ -71,6 +73,7 @@ function makeHarness({ whitelistOn = false, whitelist = [] } = {}) { activeMap, allEvents, emitted, + cached, get capturedFilters() { return capturedFilters; }, @@ -136,6 +139,16 @@ test("ingests a foreign NIP-71 video into the active store and signals a refresh assert.equal(refresh.detail.reason, "nip71-ingest"); }); +test("caches injected videos into nostrService (persisted/author-feed paths)", () => { + const h = makeHarness(); + const svc = makeService(h); + svc.start(); + h.fireEvent(foreignEvent({ id: "e1", title: "Hello" })); + svc.flush(); + assert.equal(h.cached.length, 1, "injected video forwarded to cacheVideos"); + assert.equal(h.cached[0].id, "e1"); +}); + test("never clobbers a native bitvid video sharing the same root", () => { const h = makeHarness(); const native = { id: "native", videoRootId: "shared", source: "nostr", created_at: 500, title: "Native" }; From 5e2b22156fcce9e03411f5921b4a8111235a5d40 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 21:26:20 -0400 Subject: [PATCH 40/43] docs: video-liveness plan (speed, playability probe, multi-source) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Audited the card liveness/health system. Policy (cardSourceVisibility) is already correct — hides only when neither CDN nor WebTorrent is healthy, un-hides when WebTorrent flips green. Documents the real gaps (slow probes; opaque HEAD no-cors inaccuracy for cross-origin NIP-71 CDNs; single-source adapter dropping extra imeta) and a 3-phase plan: multi-source capture, video-element playability probe across sources, and speed tuning. Adds docs/video-liveness-plan.md + expands pre-launch TODO #5. Planning only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- docs/video-liveness-plan.md | 73 ++++++++++++++++++++++++++++++ todo/TODO_2026-06-20_pre-launch.md | 22 +++++++-- 2 files changed, 90 insertions(+), 5 deletions(-) create mode 100644 docs/video-liveness-plan.md diff --git a/docs/video-liveness-plan.md b/docs/video-liveness-plan.md new file mode 100644 index 00000000..22b28092 --- /dev/null +++ b/docs/video-liveness-plan.md @@ -0,0 +1,73 @@ +# Video Liveness / Card Health — Dev Plan + +Status: **planned** (investigated; not started). Branch: `unstable`. + +## Goal +Cards should only render videos that can actually play, decide fast, support +multiple source links (fall back if one is down), and never show two of the same +video (handled separately by cross-ecosystem dedup). Prefer CDN; if CDN is dead +but a WebTorrent source is live, keep the card. + +## What already works (don't rebuild) +- **Hide/show policy is correct** (`js/utils/cardSourceVisibility.js`): a card + hides only when **neither** CDN (`urlHealthState`) **nor** WebTorrent + (`streamHealthState`) is healthy (and neither is pending); it un-hides the + moment WebTorrent flips `healthy`. This already matches "no CDN → block; if a + webtorrent flips green → bring it back." +- Owner's own videos bypass hiding. + +## The real problems (diagnosis) +1. **Too slow.** `URL_PROBE_TIMEOUT_MS = 8s` (+ a retry), WebTorrent + `PROBE_TIMEOUT_MS = 20s`. While a probe is *pending*, the card is **shown** + (pending ≠ hidden). So unplayable videos sit visible for many seconds before + being hidden. → tune timeouts + concurrency for a faster verdict. +2. **Probe accuracy.** The CDN check races a `HEAD {mode:"no-cors"}` which is + **opaque** (status unreadable) for cross-origin hosts (most NIP-71 CDNs like + `relay.towardsliberty.com`). Opaque without `confirmPlayable` → `unknown` (not + healthy). Whether a card shows then hinges on timing/streamState. Net: foreign + CDNs are judged unreliably. → use the **video-element probe** (`confirmPlayable`, + actually loads the media) as the source of truth for the card verdict. +3. **Single source only.** The ingest adapter keeps ONE `url` (the primary imeta + variant) and drops the rest. NIP-71 events can list **multiple imeta** (mirrors + / alternate media). Example: Walker/THE Bitcoin Podcast event has a `video/mp4` + imeta AND an `audio/mpeg` imeta; other events may list multiple video mirrors. + → capture all playable sources and fall back across them. + +## Plan + +### Phase 1 — Multi-source capture (adapter + video model) +- `buildVideoFromNip71Event` (and the 30078 path where applicable) should expose a + `sources` list: every imeta `url` whose `m` is a video type (and the magnet), + in order, each with `{ url, mimeType, sha256, dim, duration }`. Keep the primary + `url`/`magnet` for back-compat; add `sources` for fallback. +- Skip non-video media (e.g. `audio/mpeg`) for the *video* player, but we may keep + audio as a separate affordance later (out of scope). + +### Phase 2 — Liveness uses real playability across sources +- The card URL probe should try sources **in order** via the video-element probe + (`confirmPlayable`); `healthy` if ANY source loads; `offline` only if ALL fail. +- Cache per-source results (already have a probe cache) so scrolling doesn't + re-probe. +- Player playback should consume the same ordered `sources` so the one that + probed healthy is what plays (and can fail over at play time too). + +### Phase 3 — Speed +- Lower `URL_PROBE_TIMEOUT_MS` (8s → ~3–4s) and the video-element probe timeout; + keep one retry only for `timeout` (not `error`). Consider bumping URL-probe + concurrency (it's cheap vs WebTorrent swarm joins). +- WebTorrent `PROBE_TIMEOUT_MS` (20s): keep generous (swarm joins are slow) but + ensure it runs in parallel and never blocks the CDN verdict. +- DECISION NEEDED: **show-pending (current) vs hide-until-verified.** Showing + pending means a brief flash of soon-to-be-hidden dead cards; hide-until-verified + means good cards pop in after the probe. Recommend: keep show-pending for cards + with a *bitvid-native/whitelisted* source (fast/trusted), but **hide-until-verified + for foreign/ingested** videos (so unplayable strangers never flash in). + +## Reuse map +- `js/ui/urlHealthController.js` (probeUrl / probeUrlWithVideoElement) — make + it iterate sources + prefer the video-element verdict. +- `js/gridHealth.js` (WebTorrent swarm probe) — timing only. +- `js/utils/cardSourceVisibility.js` — policy already correct; no change expected. +- `js/nostr/nip71IngestAdapter.js` — add `sources`. +- `js/constants.js` — `URL_PROBE_TIMEOUT_MS`, probe concurrency. +- Player/playback service — consume `sources` for fail-over. diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index 747031bc..b40498d4 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -106,11 +106,23 @@ tests → `npm run build` + `npm run test:unit` green → commit + push. double-counting, and resilience to dead relays (now that feed defaults are reserved, ensure view reads/writes use a sane relay set too). -### 5. Card-hide / video-liveness check -- [ ] Audit the card-hide system that gates a card on a video-liveness probe - (`gridHealth.js`, probe concurrency now 4). Confirm: dead URL/magnet hides the - card correctly, a live video is never falsely hidden, and the probe can't - stampede connections (regression risk from the freeze fix). +### 5. Card-hide / video-liveness check — full plan in docs/video-liveness-plan.md +Audited (2026-06-23). Finding: the hide/show **policy is already correct** +(`cardSourceVisibility.js` hides only when neither CDN nor WebTorrent is healthy; +un-hides when WebTorrent flips green). The real gaps: +- [ ] **Speed**: `URL_PROBE_TIMEOUT_MS=8s` (+retry) and WebTorrent probe 20s mean + unplayable cards stay *visible* (pending isn't hidden) for seconds. Tune + timeouts + concurrency for a faster verdict. +- [ ] **Probe accuracy**: the CDN check races an opaque `HEAD no-cors` (status + unreadable for cross-origin NIP-71 CDNs). Use the **video-element probe** + (`confirmPlayable`, actually loads media) as the card verdict. +- [ ] **Multi-source**: ingest adapter keeps ONE url; NIP-71 events can list + multiple imeta (mirrors / alternate media). Capture all video sources + + magnet; probe/play them in order; healthy if ANY plays, offline only if ALL + fail. (Example: Walker/THE Bitcoin Podcast event — mp4 + mp3 imeta.) +- [ ] **Decision**: show-pending (current) vs hide-until-verified for foreign + ingested videos (so unplayable strangers never flash in). Recommend + hide-until-verified for foreign, show-pending for bitvid-native. ## Open — medium priority From 9dbe1286c008a6c54aa47c9560896ecadc89724c Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 21:48:51 -0400 Subject: [PATCH 41/43] liveness: multi-source video support + faster CDN verdict (#5 phases 1-3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses "some NIP-71 videos don't play / he has multiple links" + "speed the liveness check + don't render unplayable". - Phase 1 — multi-source capture: nip71IngestAdapter collectVideoSources() exposes a `sources` list (all video imeta urls, in order; audio/image excluded; deduped) so mirrors are preserved (the adapter previously kept only one url). - Phase 2 — multi-source liveness: urlHealthController.probeUrlList() probes every candidate (primary + sources) via the existing video-element playability probe — healthy as soon as one source plays, offline only if ALL fail. So a dead primary host no longer hides a video with a live mirror. (Card hide policy in cardSourceVisibility was already correct.) - Phase 3 — speed: URL_PROBE_TIMEOUT_MS 8s -> 4s for a fast verdict so dead cards hide quickly; WebTorrent swarm probe stays generous. Remaining (TODO #5): player-side fail-over at play time; hide-until-verified for foreign ingested videos. Tests + mutation-verified (collectVideoSources video-only filter, probeUrlList first-ok/offline/timeout). 1271/1272 unit (known UploadModal flake). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/constants.js | 5 +- js/nostr/nip71IngestAdapter.js | 41 +++++++++++++++++ js/ui/urlHealthController.js | 47 ++++++++++++++++++- tests/nip71-ingest-adapter.test.mjs | 66 +++++++++++++++++++++++++++ tests/url-health-multisource.test.mjs | 57 +++++++++++++++++++++++ todo/TODO_2026-06-20_pre-launch.md | 28 ++++++------ 6 files changed, 229 insertions(+), 15 deletions(-) create mode 100644 tests/url-health-multisource.test.mjs diff --git a/js/constants.js b/js/constants.js index 63d2b534..5cb050cc 100644 --- a/js/constants.js +++ b/js/constants.js @@ -36,7 +36,10 @@ export const DEFAULT_CACHE_TTL_MS = FIVE_MINUTES_MS; export const PROFILE_CACHE_TTL_MS = TEN_MINUTES_MS; export const URL_HEALTH_TTL_MS = FORTY_FIVE_MINUTES_MS; export const URL_HEALTH_RETRY_MS = FIVE_MINUTES_MS; -export const URL_PROBE_TIMEOUT_MS = 8 * ONE_SECOND_MS; +// Per-source playability probe budget. Kept tight so a dead host yields a fast +// verdict (and the card hides quickly) — a slow-but-valid CDN still gets a retry +// via URL_HEALTH/RETRY paths. Lowered from 8s. +export const URL_PROBE_TIMEOUT_MS = 4 * ONE_SECOND_MS; function coerceNonNegativeInteger(value, fallback) { const numeric = Number(value); diff --git a/js/nostr/nip71IngestAdapter.js b/js/nostr/nip71IngestAdapter.js index d5e1c337..00ae4e65 100644 --- a/js/nostr/nip71IngestAdapter.js +++ b/js/nostr/nip71IngestAdapter.js @@ -80,6 +80,43 @@ function pickPrimaryVariant(imeta) { return imeta.find((variant) => trimString(variant?.magnet)) || null; } +function isVideoVariant(variant) { + if (!trimString(variant?.url)) { + return false; + } + const m = trimString(variant?.m).toLowerCase(); + // Accept explicit video/* and untyped variants; exclude audio/image/etc. + return m === "" || m.startsWith("video/"); +} + +// All playable video URLs from a NIP-71 event's imeta variants, in order, so the +// player + liveness probe can fail over if one host is down. Audio/image +// variants are excluded. De-duplicated by url. +export function collectVideoSources(imeta) { + const out = []; + const seen = new Set(); + for (const variant of Array.isArray(imeta) ? imeta : []) { + if (!isVideoVariant(variant)) { + continue; + } + const url = trimString(variant.url); + if (!url || seen.has(url)) { + continue; + } + seen.add(url); + const durationNum = Number(variant.duration); + out.push({ + url, + mimeType: trimString(variant.m), + sha256: trimString(variant.x), + dim: trimString(variant.dim), + duration: + Number.isFinite(durationNum) && durationNum > 0 ? durationNum : undefined, + }); + } + return out; +} + /** * Convert a foreign NIP-71 event into a bitvid video object. * @@ -153,6 +190,9 @@ export function buildVideoFromNip71Event(event = {}) { const dTag = dTagFromEvent(event); const videoRootId = dTag || trimString(event.id); + // All playable video URLs (mirrors) for fail-over at probe + play time. + const sources = collectVideoSources(metadata.imeta); + // The feed's resolve-posted-at stage fetches per-video kind-30078 history when // a timestamp isn't already known. Foreign NIP-71 videos have no such history, // so without a posted-at the feed would fire a blocking history fetch per @@ -174,6 +214,7 @@ export function buildVideoFromNip71Event(event = {}) { isForKids: false, title, url, + sources, magnet, rawMagnet: magnet ? rawMagnet : "", infoHash, diff --git a/js/ui/urlHealthController.js b/js/ui/urlHealthController.js index a4503692..36c5b730 100644 --- a/js/ui/urlHealthController.js +++ b/js/ui/urlHealthController.js @@ -152,6 +152,21 @@ export default class UrlHealthController { this.updateUrlHealthBadge(badgeEl, { status: "checking" }, eventId); + // Try the primary url plus any additional NIP-71 source mirrors, so one dead + // host doesn't mark a video offline when an alternate plays. + const sources = Array.isArray(video?.sources) ? video.sources : []; + const candidateUrls = []; + const seenUrls = new Set(); + for (const candidate of [ + trimmedUrl, + ...sources.map((s) => (s && typeof s.url === "string" ? s.url.trim() : "")), + ]) { + if (candidate && !seenUrls.has(candidate)) { + seenUrls.add(candidate); + candidateUrls.push(candidate); + } + } + const probeOptions = { confirmPlayable: true }; const getInFlight = this.state.getInFlightUrlProbe; const existingProbe = @@ -175,7 +190,7 @@ export default class UrlHealthController { return; } - const probePromise = this.probeUrl(trimmedUrl, probeOptions) + const probePromise = this.probeUrlList(candidateUrls, probeOptions) .then((result) => { const outcome = result?.outcome || "error"; let entry; @@ -341,6 +356,36 @@ export default class UrlHealthController { } } + // Probe several candidate URLs (mirrors) in order; "ok" as soon as one is + // playable, "timeout" if none played but at least one timed out, else + // "offline". Lets a video with a dead primary host survive on a live mirror. + async probeUrlList(urls, options = {}) { + const list = Array.isArray(urls) + ? urls.filter((u) => typeof u === "string" && u.trim()) + : []; + if (!list.length) { + return { outcome: "error" }; + } + let sawTimeout = false; + for (const url of list) { + let result; + try { + result = await this.probeUrl(url, options); + } catch (err) { + this.logger.warn(`[probeUrlList] probe threw for ${url}:`, err); + continue; + } + const outcome = result?.outcome; + if (outcome === "ok") { + return { outcome: "ok", url }; + } + if (outcome === "timeout") { + sawTimeout = true; + } + } + return { outcome: sawTimeout ? "timeout" : "offline" }; + } + async probeUrl(url, options = {}) { const trimmed = typeof url === "string" ? url.trim() : ""; if (!trimmed) { diff --git a/tests/nip71-ingest-adapter.test.mjs b/tests/nip71-ingest-adapter.test.mjs index de9cde86..88f8fa8f 100644 --- a/tests/nip71-ingest-adapter.test.mjs +++ b/tests/nip71-ingest-adapter.test.mjs @@ -7,6 +7,7 @@ import test from "node:test"; import { buildVideoFromNip71Event, isBitvidMirrorEvent, + collectVideoSources, } from "../js/nostr/nip71IngestAdapter.js"; const SHA = "a".repeat(64); @@ -145,6 +146,71 @@ test("rejects non-NIP-71 kinds", () => { assert.equal(v.reason, "not a NIP-71 video kind"); }); +test("collectVideoSources keeps video urls in order, excludes audio/image, dedupes", () => { + const sources = collectVideoSources([ + { url: "https://a/v.mp4", m: "video/mp4", x: "a".repeat(64), dim: "1920x1080", duration: "120" }, + { url: "https://a/v.mp3", m: "audio/mpeg" }, + { url: "https://b/v.webm", m: "video/webm" }, + { url: "https://a/v.mp4", m: "video/mp4" }, // dup + { url: "https://c/untyped" }, // untyped accepted + { url: "https://a/thumb.jpg", m: "image/jpeg" }, + ]); + assert.deepEqual( + sources.map((s) => s.url), + ["https://a/v.mp4", "https://b/v.webm", "https://c/untyped"], + ); + assert.equal(sources[0].sha256, "a".repeat(64)); + assert.equal(sources[0].dim, "1920x1080"); + assert.equal(sources[0].duration, 120); +}); + +test("real Walker/podcast event (mp4 + mp3 imeta): one video source, mp3 excluded", () => { + const event = { + id: "a425040161016e64245282037c179bcb4aaaf1ce98024c9d7c59b32aabe1c3d2", + pubkey: "b7ed68b062de6b4a12e51fd5285c1e1e0ed0e5128cda93ab11b4150b55ed32fc", + kind: 21, + created_at: 1781815633, + content: "Privacy isn't about having something to hide…", + tags: [ + ["title", "The Praxeology of Privacy"], + ["published_at", "1781815633"], + [ + "imeta", + "url https://relay.towardsliberty.com/84751f62.mp4", + "x 84751f62034f7396a75c0ba5096964cee2a4d65a076f655b43b6324b0fbefecd", + "m video/mp4", + "duration 5556", + "image https://relay.towardsliberty.com/a2e8159.jpg", + ], + [ + "imeta", + "url https://relay.towardsliberty.com/2885921.mp3", + "m audio/mpeg", + "duration 5556", + ], + ["t", "bitcoin"], + ], + }; + const v = buildVideoFromNip71Event(event); + assert.equal(v.invalid, false); + assert.equal(v.url, "https://relay.towardsliberty.com/84751f62.mp4"); + assert.equal(v.sources.length, 1, "only the video imeta becomes a source"); + assert.equal(v.sources[0].url, "https://relay.towardsliberty.com/84751f62.mp4"); +}); + +test("multiple video mirrors all become sources (fail-over)", () => { + const v = buildVideoFromNip71Event({ + id: "e", pubkey: "p", kind: 22, created_at: 1, + content: "", + tags: [ + ["title", "Mirrored"], + ["imeta", "url https://m1/v.mp4", "m video/mp4"], + ["imeta", "url https://m2/v.mp4", "m video/mp4"], + ], + }); + assert.deepEqual(v.sources.map((s) => s.url), ["https://m1/v.mp4", "https://m2/v.mp4"]); +}); + test("extracts a webtorrent magnet + infohash when present", () => { const magnet = `magnet:?xt=urn:btih:${"b".repeat(40)}&dn=clip`; const v = buildVideoFromNip71Event( diff --git a/tests/url-health-multisource.test.mjs b/tests/url-health-multisource.test.mjs new file mode 100644 index 00000000..8d82cb93 --- /dev/null +++ b/tests/url-health-multisource.test.mjs @@ -0,0 +1,57 @@ +// Multi-source URL liveness: probe candidate mirrors in order — healthy as soon +// as one plays, offline only if all fail (so a dead primary host doesn't hide a +// video that has a working mirror). + +import assert from "node:assert/strict"; +import test from "node:test"; +import UrlHealthController from "../js/ui/urlHealthController.js"; + +function controllerWith(outcomesByUrl) { + const ctrl = new UrlHealthController({ logger: { warn() {} } }); + const probed = []; + ctrl.probeUrl = async (url) => { + probed.push(url); + return { outcome: outcomesByUrl[url] || "error" }; + }; + return { ctrl, probed }; +} + +test("returns ok on the first playable source and stops probing further", async () => { + const { ctrl, probed } = controllerWith({ + "https://dead/v.mp4": "error", + "https://live/v.mp4": "ok", + "https://other/v.mp4": "ok", + }); + const res = await ctrl.probeUrlList([ + "https://dead/v.mp4", + "https://live/v.mp4", + "https://other/v.mp4", + ]); + assert.equal(res.outcome, "ok"); + assert.equal(res.url, "https://live/v.mp4"); + assert.deepEqual(probed, ["https://dead/v.mp4", "https://live/v.mp4"], "stops at first ok"); +}); + +test("offline only when every source fails", async () => { + const { ctrl } = controllerWith({ + "https://a/v.mp4": "error", + "https://b/v.mp4": "error", + }); + const res = await ctrl.probeUrlList(["https://a/v.mp4", "https://b/v.mp4"]); + assert.equal(res.outcome, "offline"); +}); + +test("reports timeout if none played but at least one timed out", async () => { + const { ctrl } = controllerWith({ + "https://a/v.mp4": "error", + "https://b/v.mp4": "timeout", + }); + const res = await ctrl.probeUrlList(["https://a/v.mp4", "https://b/v.mp4"]); + assert.equal(res.outcome, "timeout"); +}); + +test("empty / invalid list returns error", async () => { + const { ctrl } = controllerWith({}); + assert.equal((await ctrl.probeUrlList([])).outcome, "error"); + assert.equal((await ctrl.probeUrlList(["", " "])).outcome, "error"); +}); diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index b40498d4..eeb86891 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -110,19 +110,21 @@ tests → `npm run build` + `npm run test:unit` green → commit + push. Audited (2026-06-23). Finding: the hide/show **policy is already correct** (`cardSourceVisibility.js` hides only when neither CDN nor WebTorrent is healthy; un-hides when WebTorrent flips green). The real gaps: -- [ ] **Speed**: `URL_PROBE_TIMEOUT_MS=8s` (+retry) and WebTorrent probe 20s mean - unplayable cards stay *visible* (pending isn't hidden) for seconds. Tune - timeouts + concurrency for a faster verdict. -- [ ] **Probe accuracy**: the CDN check races an opaque `HEAD no-cors` (status - unreadable for cross-origin NIP-71 CDNs). Use the **video-element probe** - (`confirmPlayable`, actually loads media) as the card verdict. -- [ ] **Multi-source**: ingest adapter keeps ONE url; NIP-71 events can list - multiple imeta (mirrors / alternate media). Capture all video sources + - magnet; probe/play them in order; healthy if ANY plays, offline only if ALL - fail. (Example: Walker/THE Bitcoin Podcast event — mp4 + mp3 imeta.) -- [ ] **Decision**: show-pending (current) vs hide-until-verified for foreign - ingested videos (so unplayable strangers never flash in). Recommend - hide-until-verified for foreign, show-pending for bitvid-native. +- [x] **Speed** (`URL_PROBE_TIMEOUT_MS` 8s → 4s): dead hosts yield a fast verdict + so unplayable cards hide quickly. WebTorrent probe stays generous (swarm). +- [x] **Probe accuracy**: confirmed the card probe already uses the video-element + probe (`confirmPlayable:true`, actually loads media) — accurate, not the + opaque HEAD. No change needed. +- [x] **Multi-source**: ingest adapter exposes `sources` (all video imeta urls; + `collectVideoSources`); `urlHealthController.probeUrlList` probes them in + order — healthy as soon as one plays, offline only if all fail. So a dead + primary host no longer hides a video with a working mirror. +- [ ] **Remaining — player fail-over at play time**: the *liveness* now tries all + sources, but actual playback should also fail over to the next source if the + chosen one dies mid-load (playbackService consumes `video.sources`). +- [ ] **Remaining — hide-until-verified for foreign/ingested** (decision): so + unplayable strangers never briefly flash before the probe hides them + (bitvid-native stays show-pending). UX policy change in cardSourceVisibility. ## Open — medium priority From dd1ce113c18744054299d0251b97f503027e018f Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 22:22:18 -0400 Subject: [PATCH 42/43] zap: graceful error when a recipient's Lightning endpoint is unreachable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sending a zap to a creator whose LNURL endpoint is down or doesn't send CORS headers (e.g. unitypay.cash/.well-known/lnurlp/<name>) made the browser fetch reject with a raw "Failed to fetch", which bubbled up as the zap's error toast — the "clunky errors" reported during the zap audit. fetchPayServiceData now catches the network/CORS rejection and throws a clear, coded error ("Couldn't reach this recipient's Lightning address. It may be offline or blocking browser requests."), preserving the original as `cause`. Non-OK HTTP and valid responses are unchanged. Part of the zap audit (TODO #3). Also recorded: the video-modal popover mis-positioning (3a) and that the deeper CORS issue (static client can't fetch CORS-less LNURL hosts) may need a proxy — tracked separately. Tests + mutation-verified. 1274 pass (known UploadModal flake). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- js/payments/lnurl.js | 21 ++++++++++--- tests/lnurl-fetch-error.test.mjs | 47 ++++++++++++++++++++++++++++++ todo/TODO_2026-06-20_pre-launch.md | 21 +++++++++++++ 3 files changed, 85 insertions(+), 4 deletions(-) create mode 100644 tests/lnurl-fetch-error.test.mjs diff --git a/js/payments/lnurl.js b/js/payments/lnurl.js index 36cb8fb5..1329be1f 100644 --- a/js/payments/lnurl.js +++ b/js/payments/lnurl.js @@ -232,10 +232,23 @@ export async function fetchPayServiceData(url, { fetcher } = {}) { const targetUrl = sanitizeUrl(url); const fetchFn = ensureFetchFunction(fetcher); - const response = await fetchFn(targetUrl, { - method: "GET", - headers: { Accept: "application/json" }, - }); + let response; + try { + response = await fetchFn(targetUrl, { + method: "GET", + headers: { Accept: "application/json" }, + }); + } catch (error) { + // Browser fetch rejects (TypeError "Failed to fetch") when the recipient's + // Lightning endpoint is unreachable or — common for LNURL — doesn't send CORS + // headers. Surface a clear, actionable message instead of the raw error. + const friendly = new Error( + "Couldn't reach this recipient's Lightning address. It may be offline or blocking browser requests.", + ); + friendly.cause = error; + friendly.code = "lnurl-unreachable"; + throw friendly; + } if (!response.ok) { throw new Error(`Failed to load LNURL metadata (${response.status}).`); diff --git a/tests/lnurl-fetch-error.test.mjs b/tests/lnurl-fetch-error.test.mjs new file mode 100644 index 00000000..e3e96b53 --- /dev/null +++ b/tests/lnurl-fetch-error.test.mjs @@ -0,0 +1,47 @@ +// A recipient's Lightning endpoint being unreachable (down, or — common for +// LNURL — missing CORS headers) makes the browser fetch reject with +// "Failed to fetch". The zap flow should surface a clear, actionable message, +// not the raw error. + +import assert from "node:assert/strict"; +import test from "node:test"; +import { fetchPayServiceData } from "../js/payments/lnurl.js"; + +test("network/CORS failure throws a friendly, coded error", async () => { + const fetcher = async () => { + throw new TypeError("Failed to fetch"); + }; + await assert.rejects( + () => fetchPayServiceData("https://example.com/.well-known/lnurlp/x", { fetcher }), + (err) => { + assert.match(err.message, /reach this recipient's Lightning address/i); + assert.equal(err.code, "lnurl-unreachable"); + assert.ok(err.cause, "preserves the original error as cause"); + return true; + }, + ); +}); + +test("a non-OK HTTP response still reports a status error (unchanged)", async () => { + const fetcher = async () => ({ ok: false, status: 502 }); + await assert.rejects( + () => fetchPayServiceData("https://example.com/.well-known/lnurlp/x", { fetcher }), + /Failed to load LNURL metadata \(502\)/, + ); +}); + +test("a valid LNURL pay response parses (no false friendly error)", async () => { + const fetcher = async () => ({ + ok: true, + json: async () => ({ + callback: "https://example.com/cb", + minSendable: 1000, + maxSendable: 100000, + tag: "payRequest", + }), + }); + const meta = await fetchPayServiceData("https://example.com/.well-known/lnurlp/x", { + fetcher, + }); + assert.equal(meta.callback, "https://example.com/cb"); +}); diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index eeb86891..276edc0c 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -94,6 +94,27 @@ tests → `npm run build` + `npm run test:unit` green → commit + push. wrapper, so add carefully — plus a feed/view-level seed→delete→assert test. (Local + publish/read layers are already unit+mutation tested.) +### 3a. Video-modal popover mis-positioning (zap + embed) — BUG +Confirmed via screenshots: the zap/embed popovers OPEN and are fully functional +(comment box, amount, send all render) but appear anchored to the far LEFT of the +content column instead of `bottom-end` under the button (which is in the right- +side action bar). Both share `js/ui/overlay/popoverEngine.js`. +- Leading cause: the engine only portals a panel into the body-level overlay root + `if (!panel.isConnected)`. The zap/embed panels are pre-existing connected + elements INSIDE the modal, so they're NOT portaled out, and `position:fixed` + resolves against an in-modal containing block (something in the modal subtree + has transform/filter/contain) → horizontal offset. +- Ruled out: positioning CSS present in built css; floating-ui local bundle loads; + panel class correct (`.popover__panel`/`.popover-panel` share the rule). +- Fix (do carefully — shared engine drives every popover): either portal connected + in-modal panels out to `#uiOverlay` on open (and restore/clean on close without + losing the element for re-open), OR remove the containing-block ancestor in the + modal subtree. VERIFY non-modal popovers (card "⋯" menu, feed-settings gear, + FeedInfoPopover) still position correctly after the change. +- To pin exactly: inspect `#modalZapDialog` while open — computed `position`, + `left`, `top`, and its DOM parent (is it under `#uiOverlay` at body level or + still nested in the modal?). + ### 3. Zaps system + platform-fee zap split - [ ] Audit the zap flow (NWC / `nwcClient.js`, `zapController.js`, `zapReceiptValidator.js`). - [ ] Verify the platform-fee split: correct recipients, correct percentages, rounding, From 07b42788d5ca0d61fad02fc96685de8aa291fc68 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 22:26:07 -0400 Subject: [PATCH 43/43] docs: zap-audit plan + capture cross-session learnings in TODO - docs/zap-audit-plan.md: architecture map, findings (LNURL CORS send-error, comment box confirmed working, popover mis-position), and prioritized next steps (CORS/LNURL proxy decision first, then popover, fee-split correctness, receipt validation, NWC UX). Notes the injectable-fetcher/deps test seams. - TODO #3 points to the doc; #9 adds two unverified learnings: confirm the community-blacklist batching actually dropped the kind-30000 storm (still saw =32 in a later load), and trace the `from_hex` malformed-hex source via the new sanitizer dev-log. Planning/docs only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- docs/zap-audit-plan.md | 62 ++++++++++++++++++++++++++++++ todo/TODO_2026-06-20_pre-launch.md | 25 +++++++++--- 2 files changed, 82 insertions(+), 5 deletions(-) create mode 100644 docs/zap-audit-plan.md diff --git a/docs/zap-audit-plan.md b/docs/zap-audit-plan.md new file mode 100644 index 00000000..b8c51cfa --- /dev/null +++ b/docs/zap-audit-plan.md @@ -0,0 +1,62 @@ +# Zap System Audit — Plan & Findings + +Status: **in progress** (audit started 2026-06-23). Branch: `unstable`. +Pre-launch priority #3. The zap flow feels clunky and errors on send. + +## Architecture (map for the next session) +- **Two controllers**: `js/ui/components/video-modal/zapController.js` (the in-modal + UI — button, popover, comment/amount inputs; dispatches `video:zap` / `zap:open`) + and `js/ui/zapController.js` (the logic — `open()`, `sendZap()`, retry, receipts). + `ModalManager` wires `video:zap → zapController.sendZap`. +- **Payment flow**: `sendZap` → `splitAndZap` (`js/payments/zapSplit.js`) → per share + (`creator`, `platform`) `processShare` → `lnurl.fetchPayServiceData` → + `buildZapRequest` (kind 9734, comment in `content`) → `lnurl.requestInvoice` + (`nostr=` zapRequest + optional `comment=` param) → `wallet.sendPayment` (NWC, + `nwcClient.js`) → `zapReceiptValidator` (kind 9735). +- **Fee split**: `PLATFORM_FEE_PERCENT` (instance-config, currently 30) → + `platformShare = floor(amount*fee/100)`, `creatorShare = amount - platformShare`; + platform address via `platformAddress.js` / `PLATFORM_LUD16_OVERRIDE`. + +## Findings so far +- **Send error = recipient LNURL unreachable (CORS/offline).** [PARTLY FIXED] + Sending to a creator whose LNURL host is down or lacks CORS headers (observed: + `unitypay.cash/.well-known/lnurlp/goblinbox`) makes the browser `fetch` reject + with "Failed to fetch". Shipped (`dd1ce113`): `fetchPayServiceData` now throws a + clear, coded (`lnurl-unreachable`) message. **Root issue remains**: a static + client CANNOT fetch CORS-less LNURL hosts — see "CORS proxy" below. +- **Comment box works.** Confirmed via screenshot — the textarea renders and binds; + the comment is carried in the zap-request `content` + the LNURL `comment` param. + The earlier "message doesn't work" was the popover being mis-positioned/hard to + use (see 3a), not a comment bug. +- **Popover mis-positioning (zap + embed)** — see TODO 3a; shared `popoverEngine`. + +## To do (next session, in priority order) +1. **CORS / LNURL proxy decision (the real reliability fix).** A static client + can't fetch LNURL endpoints that omit CORS headers, so zaps to many creators + fail. Options: (a) a small server-side/edge proxy (Vercel function) that fetches + the LNURL pay-service data + invoice and returns it with CORS — what most web + wallets do; (b) document the limitation and only support CORS-enabled hosts; + (c) route via the connected NWC wallet if it can resolve LN addresses. Decide, + then implement. Confirm whether `unitypay.cash` is also just offline. +2. **Popover positioning (3a).** Fix the shared engine safely (portal in-modal + panels out to `#uiOverlay`, or remove the modal-subtree containing block). + Needs the live computed `position/left/top` + DOM parent of `#modalZapDialog`. + VERIFY non-modal popovers still position right after. +3. **Platform-fee split correctness.** Verify recipients/percentages/rounding; + ensure the fee can't swallow the whole zap, be bypassed, or (earlier report) + land the platform fee in the *sender's* own wallet when zapping certain targets. + Check the self-zap / creator==platform edge case. Add cheat-resistant tests. +4. **Receipt validation (kind 9735).** Confirm accuracy before crediting; the + `unvalidatedReceipts`/`failureReceipts` paths and the "Awaiting validated zap + receipt" state. (Earlier fix: `zapReceiptValidator` SimplePool `.list` shim.) +5. **NWC error UX.** "Budget exceeded" and partial-share retry flow — make the + messaging + retry obviously usable (it's currently terse). +6. **General clunkiness.** Step back on the whole flow: amount presets, the + split summary, success/partial/failure states, focus, and mobile. + +## Cheat-resistant testing notes +- `fetchPayServiceData` / `requestInvoice` accept an injectable `fetcher` — test + failure modes by injecting rejecting/erroring fetchers (see + `tests/lnurl-fetch-error.test.mjs`). +- `splitAndZap` takes injectable `dependencies` (see `tests/zap-split.test.mjs`) — + use it to assert fee math, rounding, and recipient routing without real LN. diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index 276edc0c..20f82d3c 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -115,11 +115,17 @@ side action bar). Both share `js/ui/overlay/popoverEngine.js`. `left`, `top`, and its DOM parent (is it under `#uiOverlay` at body level or still nested in the modal?). -### 3. Zaps system + platform-fee zap split -- [ ] Audit the zap flow (NWC / `nwcClient.js`, `zapController.js`, `zapReceiptValidator.js`). -- [ ] Verify the platform-fee split: correct recipients, correct percentages, rounding, - and that the fee can't silently swallow the whole zap or be bypassed. -- [ ] Confirm zap-receipt validation (kind 9735) is accurate before crediting. +### 3. Zaps system + platform-fee zap split — full plan in docs/zap-audit-plan.md +Audit started 2026-06-23. See the doc for architecture map + findings. Summary: +- [x] **Send-error softened** (`dd1ce113`): recipient LNURL unreachable/CORS now + shows a clear message, not raw "Failed to fetch". +- [ ] **CORS / LNURL proxy decision** (the real reliability fix — static client + can't fetch CORS-less LNURL hosts; many zaps fail). Top of the next session. +- [ ] **Popover positioning (3a)**, **platform-fee split correctness** (incl. the + earlier "fee landed in my own wallet" report + self-zap edge), **receipt + validation (9735)**, **NWC budget/retry UX**, **general clunkiness**. +- [x] Comment box confirmed WORKING (the "message doesn't work" was the popover + mis-position making it hard to use — see 3a). ### 4. View counter accuracy & reliability - [ ] Audit the view-counter system (`viewEvents.js`, `reactionCounter.js`) for @@ -170,6 +176,15 @@ un-hides when WebTorrent flips green). The real gaps: kind-30000 REQ *per curator* (~28-32 parallel) at cold start. Collapsed into a single batched multi-author REQ (`js/adminListBatch.js`), routed through the subscription manager. This was the dominant `kind 30000` storm. +- [ ] **VERIFY the batching actually reduced it.** A later logged-OUT load + (2026-06-23 00:45) STILL showed `kind 30000=32` — either that build wasn't + loaded, or there's ANOTHER kind-30000 source besides + `loadCommunityBlacklistEntries` (admin editor/whitelist/blacklist lists? a + per-author follow-set fetch?). Re-check on a fresh hard-refresh; if still ~32, + trace the actual emitter. +- [ ] **`from_hex` source not root-caused.** The sanitizer (`733cb593`) defends + + dev-logs `[toolkit] dropped N invalid hex value(s) from filter "<field>"` — + watch that log to find which subscription ships the odd-length hex, fix at source. - [x] **Relay `from_hex` REQ rejects fixed** (`733cb593`): one odd-length/non-hex value in `ids/authors/#e/#p/#q` made strict relays reject a whole REQ; now sanitized at the pool choke point (`normalizeFilterList`).