bitvid/docs/nip71-onboarding-plan.md

NIP-71 On-Boarding + Cross-Ecosystem Dedup — Dev Plan

Status: planned (not started). Owner: TBD. Branch: unstable. Companion to docs/nip71-migration-plan.md (off-boarding / mirror / ingest, all shipped).

Goal

Make NIP-71 interop a two-way street:

1. Off-boarding (DONE): a bitvid video (kind 30078) is mirrored out to NIP-71 (addressable 34235/36) so other apps can discover it.
2. On-boarding (THIS PLAN): a creator's NIP-71 video published via another app can be pulled into bitvid so it gains bitvid's advantages — CDN-hosted URL, a WebTorrent source, and (optionally) bitvid extras.

And the invariant that makes both safe to coexist:

3. One video, never two. When the same video exists in both ecosystems, always prefer the bitvid (kind-30078) version, show exactly one, in every grid/list/scenario. Never both.

The hard constraint (why on-boarding is creator-initiated)

Off-boarding works because the bitvid creator owns the video and signs BOTH events with their own key. On-boarding is not symmetric: a foreign NIP-71 video is signed by the foreign creator's key, and bitvid cannot forge a kind-30078 note authored by someone else (no private key). Therefore:

• On-boarding = creator-initiated self-import only. A creator who published NIP-71 videos elsewhere logs into bitvid with the same npub, and bitvid offers to import them: re-host the file to their own connected storage (CDN), optionally seed WebTorrent, and publish a bitvid kind-30078 version signed by their key. No auto-rehosting of other people's files (consent / cost / liability). This mirrors off-boarding exactly: both directions author-owned and consensual.

Locked decisions (from design discussion)

• On-boarding model: A — creator-initiated self-import.
• Dedup identity: explicit import-link + content hash (fileSha256/ox) + infohash (i/magnet btih). NO fuzzy url/title matching (avoids false merges). Note: hash-less foreign videos (e.g. url+image only) won't dedup unless they were on-boarded (which adds an explicit link) — acceptable.
• Advantages on import: CDN url + WebTorrent source (baseline). bitvid extras (comments, view counts, edit-in-place, private toggle) come free once it's a real 30078 note; no extra work.
• Cadence: one-shot import with a re-run button (not continuous lockstep). Re-running re-imports anything new.

Phasing — dedup FIRST (it's needed regardless of on-boarding)

Phase 1 — Cross-ecosystem dedup ("prefer bitvid, show one, everywhere")

Why first: even today, a creator who independently posts to bitvid AND Nostube can produce two cards. And the moment on-boarding exists, the original + imported versions both exist on relays. Dedup must land before/with on-boarding.

• Identity key: author + (importLink ‖ fileSha256 ‖ ox ‖ infoHash).
  • importLink: pointer an on-boarded 30078 carries to the original NIP-71 event/address (added in Phase 2). Most reliable.
  • Hashes/infohash already extracted by convertEventToVideo (30078) and buildVideoFromNip71Event (foreign) — confirm both populate fileSha256 / originalFileSha256 / infoHash.
• Preference: within an identity group, keep the bitvid version (source !== "nip71-ingest" / kind 30078); else keep the NIP-71 one.
• Placement (one chokepoint → all grids): the feed AND channel profile already call app.dedupeVideosByRoot(...). Add a pure collapseCrossEcosystem(videos) pass and either chain it after root-dedup inside that shared helper, or wrap both into one dedupeVideos(videos) used everywhere (feed engine createDedupeByRootStage, buildRenderableChannelVideos, My Videos, search, playlists, watch history). Audit every list builder for the shared call.
• Bitvid's own outbound mirrors are already skipped at ingest (client=bitvid + a back-pointer) — keep that; this pass handles the rest.
• Tests (cheat-resistant, pure): same-hash bitvid+foreign → only bitvid kept; foreign-only (no bitvid) → foreign kept; distinct videos not merged; hash-less pair NOT merged (documented limitation); import-link match → original hidden. Mutation-verify the prefer-bitvid selection and the identity grouping.

Phase 2 — Creator-initiated on-board import

• Discovery: on login, find the logged-in creator's own NIP-71 videos (reuse the ingest subscription/adapter, scoped to authors:[viewerPubkey]) that have no corresponding bitvid 30078 yet (via the Phase-1 identity key).

• UI: a "Import your videos from other apps" surface (My Videos pane or a one-time prompt): list importable items, let the creator pick, show what import does. Requires their storage connected (re-host target) and signer.

• Import action (per video), reusing existing plumbing:

  1. Fetch the source file from the foreign url (or resolve via magnet).
  2. Re-host to the creator's connected storage (reuse storageService / r2Service upload path used by normal uploads) → bitvid CDN url.
  3. Optionally create a .torrent + WebTorrent magnet (reuse upload's torrent path) so the video gets a P2P source.
  4. Build + publish a kind-30078 note signed by the creator (reuse videoPublisher / buildVideoPostEvent), carrying: title, description, hashtags, dims/duration (from the foreign imeta via the adapter), the new CDN url + magnet, AND an explicit ["imported-from", "<orig event id / a-addr>"] provenance tag for dedup + attribution.
  5. (Optional, symmetric) since it's now a normal bitvid video, the existing outbound mirror/auto-share can keep the NIP-71 side in lockstep going forward.

• Import modes (decision): offer both

  • Full import (re-host): file copied to the creator's storage → bitvid-managed (CDN url under their bucket), normal storage tooling applies.
  • Reference import (no re-host): publish a 30078 that keeps the external URL (lighter, no storage cost). The video is now a bitvid note but its file is externally managed — see storage-provenance below.

• Result: the bitvid 30078 is now canonical; Phase-1 dedup hides the original foreign event everywhere; the creator gets bitvid features + (full import) CDN + WebTorrent, or (reference import) discovery without re-hosting.

Storage provenance — flag externally-managed imports

Any imported (or otherwise bitvid-converted) video whose file was not produced by our storage system must be explicitly flagged as externally managed so the storage tooling never treats it as a bitvid-owned object.

• Today: isUrlUnderBase(url, publicBaseUrl) (myVideosHealth.js) already classifies bucket vs external ("Hosted URL" vs "External URL"), and reconcileStorage only matches files under the base. So a reference-imported external URL is mostly handled by the heuristic.
• Add (durable + explicit): stamp the on-boarded note as externally managed rather than relying solely on the URL-vs-base heuristic (which breaks if the creator changes their storage base, or an external host coincidentally matches). Carry it on the video object (e.g. externalStorage: true, derived from the imported-from/reference-import path) and, where appropriate, on the event.
• All storage surfaces must honor it:
  • My Videos health: show an "External URL / externally managed" badge; never flag as missing-from-bucket.
  • Orphan reconciliation: exclude — never list as an orphan, never offer bucket delete for a file we don't own.
  • Delete flow: suppress the "a hosted file is left behind" storage-cleanup warning for externally-managed videos (there's nothing in our bucket to clean).
  • Liveness: external URL stays "unverifiable" (existing behavior).
• Full import sets externalStorage: false (it IS in the creator's bucket) and behaves like any native upload.
• Guards:
  • Only the same-pubkey creator can import their own videos (signer == author).
  • Never import NSFW outward against ALLOW_NSFW_CONTENT (mirror the existing gate); carry content-warning → isNsfw.
  • Re-host uses the creator's storage only; surface cost/ownership in the copy.
  • One-shot; a "re-scan / import new" button re-runs discovery.

Non-goals / future

• No instance-level auto-rehosting of arbitrary creators' files (model B) — consent/ cost/liability. Revisit only with explicit per-instance policy + creator opt-in.
• No fuzzy (url/title) dedup — revisit only if hash-less dual-posts become a real problem.
• Continuous lockstep on the inbound side (re-import on the creator editing the original elsewhere) — deferred; one-shot + re-run is enough for launch.

Reuse map (existing code to lean on)

• js/nostr/nip71IngestAdapter.js — foreign event → video (hashes, dims, imeta).
• js/nostr/nip71Mirror.js — outbound mirror (keeps NIP-71 side in lockstep after import).
• js/services/nip71IngestService.js — author-scoped subscription pattern for discovery.
• js/channelProfileVideos.js, feed createDedupeByRootStage, app.dedupeVideosByRoot — the dedup chokepoints to extend.
• storageService / r2Service / upload + torrent paths — re-host + seed.
• videoPublisher / buildVideoPostEvent — publish the signed 30078.