From 0112903a6d746f13b67111418ff219859b110b66 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Mon, 22 Jun 2026 19:28:15 -0400 Subject: [PATCH] docs: NIP-71 on-boarding + cross-ecosystem dedup dev plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds docs/nip71-onboarding-plan.md and pre-launch TODO 17c for the agreed two-way NIP-71 interop: - Phase 1 (first): cross-ecosystem dedup — prefer the bitvid (30078) version, show exactly one in every grid; identity = author + import-link/hash/infohash. - Phase 2: creator-initiated self-import (model A) — re-host to the creator's storage (CDN) + WebTorrent, publish a signed 30078 with an imported-from provenance tag. No forging others' notes; no instance auto-rehost. Planning only; no code changes. Co-Authored-By: Claude Opus 4.8 --- docs/nip71-onboarding-plan.md | 126 +++++++++++++++++++++++++++++ todo/TODO_2026-06-20_pre-launch.md | 18 +++++ 2 files changed, 144 insertions(+) create mode 100644 docs/nip71-onboarding-plan.md diff --git a/docs/nip71-onboarding-plan.md b/docs/nip71-onboarding-plan.md new file mode 100644 index 00000000..0abe1874 --- /dev/null +++ b/docs/nip71-onboarding-plan.md @@ -0,0 +1,126 @@ +# NIP-71 On-Boarding + Cross-Ecosystem Dedup — Dev Plan + +Status: **planned** (not started). Owner: TBD. Branch: `unstable`. +Companion to `docs/nip71-migration-plan.md` (off-boarding / mirror / ingest, all shipped). + +## Goal + +Make NIP-71 interop a **two-way street**: + +1. **Off-boarding (DONE):** a bitvid video (kind 30078) is mirrored *out* to NIP-71 + (addressable 34235/36) so other apps can discover it. +2. **On-boarding (THIS PLAN):** a creator's NIP-71 video published via another app + can be pulled *into* bitvid so it gains bitvid's advantages — CDN-hosted URL, + a WebTorrent source, and (optionally) bitvid extras. + +And the invariant that makes both safe to coexist: + +3. **One video, never two.** When the same video exists in both ecosystems, always + prefer the **bitvid** (kind-30078) version, show **exactly one**, in **every** + grid/list/scenario. Never both. + +## The hard constraint (why on-boarding is creator-initiated) + +Off-boarding works because the bitvid creator owns the video and signs BOTH events +with their own key. On-boarding is **not symmetric**: a foreign NIP-71 video is +signed by the foreign creator's key, and bitvid cannot forge a kind-30078 note +authored by someone else (no private key). Therefore: + +- **On-boarding = creator-initiated self-import only.** A creator who published + NIP-71 videos elsewhere logs into bitvid **with the same npub**, and bitvid + offers to import them: re-host the file to **their own** connected storage (CDN), + optionally seed WebTorrent, and publish a bitvid kind-30078 version **signed by + their key**. No auto-rehosting of other people's files (consent / cost / + liability). This mirrors off-boarding exactly: both directions author-owned and + consensual. + +## Locked decisions (from design discussion) + +- On-boarding model: **A — creator-initiated self-import.** +- Dedup identity: **explicit import-link + content hash** (`fileSha256`/`ox`) + + infohash (`i`/magnet btih). NO fuzzy url/title matching (avoids false merges). + Note: hash-less foreign videos (e.g. url+image only) won't dedup unless they + were on-boarded (which adds an explicit link) — acceptable. +- Advantages on import: **CDN url + WebTorrent source** (baseline). bitvid extras + (comments, view counts, edit-in-place, private toggle) come free once it's a + real 30078 note; no extra work. +- Cadence: **one-shot import with a re-run button** (not continuous lockstep). + Re-running re-imports anything new. + +## Phasing — dedup FIRST (it's needed regardless of on-boarding) + +### Phase 1 — Cross-ecosystem dedup ("prefer bitvid, show one, everywhere") + +Why first: even today, a creator who independently posts to bitvid AND Nostube can +produce two cards. And the moment on-boarding exists, the original + imported +versions both exist on relays. Dedup must land before/with on-boarding. + +- **Identity key:** `author + (importLink ‖ fileSha256 ‖ ox ‖ infoHash)`. + - `importLink`: pointer an on-boarded 30078 carries to the original NIP-71 + event/address (added in Phase 2). Most reliable. + - Hashes/infohash already extracted by `convertEventToVideo` (30078) and + `buildVideoFromNip71Event` (foreign) — confirm both populate `fileSha256` / + `originalFileSha256` / `infoHash`. +- **Preference:** within an identity group, keep the bitvid version + (`source !== "nip71-ingest"` / kind 30078); else keep the NIP-71 one. +- **Placement (one chokepoint → all grids):** the feed AND channel profile already + call `app.dedupeVideosByRoot(...)`. Add a pure `collapseCrossEcosystem(videos)` + pass and either chain it after root-dedup inside that shared helper, or wrap + both into one `dedupeVideos(videos)` used everywhere (feed engine + `createDedupeByRootStage`, `buildRenderableChannelVideos`, My Videos, search, + playlists, watch history). Audit every list builder for the shared call. +- **Bitvid's own outbound mirrors** are already skipped at ingest + (`client=bitvid` + `a` back-pointer) — keep that; this pass handles the rest. +- **Tests (cheat-resistant, pure):** same-hash bitvid+foreign → only bitvid kept; + foreign-only (no bitvid) → foreign kept; distinct videos not merged; hash-less + pair NOT merged (documented limitation); import-link match → original hidden. + Mutation-verify the prefer-bitvid selection and the identity grouping. + +### Phase 2 — Creator-initiated on-board import + +- **Discovery:** on login, find the logged-in creator's own NIP-71 videos (reuse + the ingest subscription/adapter, scoped to `authors:[viewerPubkey]`) that have + **no** corresponding bitvid 30078 yet (via the Phase-1 identity key). +- **UI:** a "Import your videos from other apps" surface (My Videos pane or a + one-time prompt): list importable items, let the creator pick, show what import + does. Requires their **storage connected** (re-host target) and signer. +- **Import action (per video), reusing existing plumbing:** + 1. Fetch the source file from the foreign `url` (or resolve via magnet). + 2. Re-host to the creator's connected storage (reuse `storageService` / + `r2Service` upload path used by normal uploads) → bitvid CDN url. + 3. Optionally create a `.torrent` + WebTorrent magnet (reuse upload's torrent + path) so the video gets a P2P source. + 4. Build + publish a kind-30078 note **signed by the creator** (reuse + `videoPublisher` / `buildVideoPostEvent`), carrying: title, description, + hashtags, dims/duration (from the foreign imeta via the adapter), the new + CDN url + magnet, AND an explicit `["imported-from", ""]` + provenance tag for dedup + attribution. + 5. (Optional, symmetric) since it's now a normal bitvid video, the existing + outbound mirror/auto-share can keep the NIP-71 side in lockstep going forward. +- **Result:** the bitvid 30078 is now canonical; Phase-1 dedup hides the original + foreign event everywhere; the creator gets CDN + WebTorrent + bitvid features. +- **Guards:** + - Only the **same-pubkey** creator can import their own videos (signer == author). + - Never import NSFW outward against `ALLOW_NSFW_CONTENT` (mirror the existing + gate); carry `content-warning → isNsfw`. + - Re-host uses the creator's storage only; surface cost/ownership in the copy. + - One-shot; a "re-scan / import new" button re-runs discovery. + +## Non-goals / future + +- No instance-level auto-rehosting of arbitrary creators' files (model B) — consent/ + cost/liability. Revisit only with explicit per-instance policy + creator opt-in. +- No fuzzy (url/title) dedup — revisit only if hash-less dual-posts become a real + problem. +- Continuous lockstep on the inbound side (re-import on the creator editing the + original elsewhere) — deferred; one-shot + re-run is enough for launch. + +## Reuse map (existing code to lean on) + +- `js/nostr/nip71IngestAdapter.js` — foreign event → video (hashes, dims, imeta). +- `js/nostr/nip71Mirror.js` — outbound mirror (keeps NIP-71 side in lockstep after import). +- `js/services/nip71IngestService.js` — author-scoped subscription pattern for discovery. +- `js/channelProfileVideos.js`, feed `createDedupeByRootStage`, `app.dedupeVideosByRoot` + — the dedup chokepoints to extend. +- `storageService` / `r2Service` / upload + torrent paths — re-host + seed. +- `videoPublisher` / `buildVideoPostEvent` — publish the signed 30078. diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md index a20e5b91..b1b93f80 100644 --- a/todo/TODO_2026-06-20_pre-launch.md +++ b/todo/TODO_2026-06-20_pre-launch.md @@ -316,6 +316,24 @@ notes below. Summary: nip71.publishedAt so the stage short-circuits. - Verified live: Goblinbox (Nostube) content renders; liveness check works. +### 17c. NIP-71 on-boarding + cross-ecosystem dedup (IMPORTANT — full plan in docs/nip71-onboarding-plan.md) +Make NIP-71 interop two-way and guarantee a video is never shown twice. +**See `docs/nip71-onboarding-plan.md` for the full dev plan.** Locked decisions: +creator-initiated self-import (model A — can't forge others' notes), dedup on +explicit import-link + content hash/infohash (no fuzzy url/title), import grants +CDN url + WebTorrent (bitvid extras come free), one-shot import + re-run button. +- [ ] **Phase 1 (do first): cross-ecosystem dedup** — `collapseCrossEcosystem` + keyed on `author + (importLink ‖ fileSha256 ‖ ox ‖ infoHash)`, prefer the + bitvid (kind-30078) version, applied at the shared dedupe chokepoint so + EVERY grid (feed, channel profile, My Videos, search, playlists, watch + history) collapses dupes. Pure + mutation-tested. Needed regardless of + on-boarding (independent dual-posts already double up). +- [ ] **Phase 2: creator-initiated import** — discover the logged-in creator's own + NIP-71 videos lacking a bitvid version; re-host the file to THEIR storage + (CDN) + optional WebTorrent seed; publish a kind-30078 signed by them with an + `["imported-from", ]` provenance tag. Same-pubkey-only guard; + ALLOW_NSFW gate; reuses ingest adapter + storage/upload + videoPublisher. + ### 17b. Channel Profile wall shows only bitvid videos, not NIP-71 (LAUNCH-BLOCKER) - [x] **Fixed** (`js/channelProfileVideos.js`): the channel grid now fetches the author's NIP-71 videos (kinds 21/22/34235/34236) alongside kind-30078 via