From 163f7e11d5488680108dd9c41be70fc8111e0208 Mon Sep 17 00:00:00 2001
From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com>
Date: Wed, 1 Jul 2026 11:10:38 -0400
Subject: [PATCH] [docs] Dev plans for live-stream ingest (#16) and shorts
 (#16b)

Two flag-gated, watch-only content types, each spec'd following the repo's
docs/*-plan.md convention with open decisions embedded as DECISION callouts.

- docs/shorts-plan.md (#16b): NIP-71 kind 22. Ingest already exists
  (nip71IngestAdapter handles kind 22), so this is an isShort marker + a Shorts
  sidebar tab + vertical UI + a shorts feed source. Behind FEATURE_SHORTS.
- docs/live-ingest-plan.md (#16): NIP-53 kind 30311 (+1311 chat), HLS playback.
  Leads with a Phase 0 research spike (confirm tag shapes from zap.stream/
  shosho.live, de-risk hls.js + CORS). Behind FEATURE_LIVE_INGEST; publishing
  (#16c) is explicitly out of scope with its own future flag.

Both docs specify the "off = no trace" wiring (no tab, no feed registration, no
subscriptions, no route, no lazy assets) per the locked shared requirement.
TODO #16/#16b now link their plans.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 docs/live-ingest-plan.md           | 173 +++++++++++++++++++++++++++++
 docs/shorts-plan.md                | 146 ++++++++++++++++++++++++
 todo/TODO_2026-06-20_pre-launch.md |   2 +
 3 files changed, 321 insertions(+)
 create mode 100644 docs/live-ingest-plan.md
 create mode 100644 docs/shorts-plan.md

diff --git a/docs/live-ingest-plan.md b/docs/live-ingest-plan.md
new file mode 100644
index 00000000..51cec142
--- /dev/null
+++ b/docs/live-ingest-plan.md
@@ -0,0 +1,173 @@
+# Live Streams — INGEST / watch-only — Dev Plan
+
+TODO ref: **#16** in `todo/TODO_2026-06-20_pre-launch.md` (watch-only).
+Publishing / "Go Live" is **#16c** — a separate, larger effort with its **own**
+config flag; it is explicitly **out of scope** for this doc.
+Status: **PLANNING → needs a research spike (Phase 0)**. Not started.
+
+Live streams are **NIP-53 Live Activities** — a *different* NIP from short-form
+video (#16b / NIP-71). Kind **30311** is the live event (addressable, carries the
+`streaming` URL — usually HLS `.m3u8` — plus `status`, title, host `p` tags);
+kind **1311** is live chat. zap.stream and shosho.live publish these.
+
+Scope of this feature: **discover live events, play the HLS stream, optionally
+show read-only chat.** No broadcasting, no media hosting. Everything ships behind
+a config flag that leaves **no trace** when off.
+
+---
+
+## Decisions needed
+
+> **DECISION 1 — Listing scope.** List only **whitelisted hosts** (reuse the
+> existing whitelist/WoT filter on the host `p`/author), or **all** `status=live`
+> events discovered on the read relays (capped)?
+> *Recommendation: whitelisted hosts for v1 — consistent with the rest of bitvid's
+> trust model and avoids surfacing arbitrary/abusive streams.*
+
+> **DECISION 2 — Live chat in v1?** Read-only kind-1311 chat alongside the player,
+> or playback-only for v1 (chat later)?
+> *Recommendation: playback-only v1; add read-only chat in a later phase. Chat is
+> a live subscription with its own moderation questions.*
+
+> **DECISION 3 — HLS playback dependency.** OK to lazy-load a small HLS library
+> (**hls.js**, ~cdn/vendored) ONLY when the flag is on, or must playback stay
+> dependency-free (native `<video>` HLS, which only Safari supports reliably)?
+> *Recommendation: allow hls.js, lazy-imported behind the flag (native HLS alone
+> won't play `.m3u8` in Chrome/Firefox). It never loads when the flag is off.*
+
+> **DECISION 4 — Ended streams / VOD.** When `status=ended`, some events carry a
+> `recording` URL (a VOD). Show ended streams with playback of the recording,
+> hide them, or list-but-mark-ended? *Recommendation: v1 shows only `status=live`;
+> treat recordings as a later enhancement.*
+
+> **DECISION 5 — Discovery relays.** Use the user's normal read relays, or add
+> stream-heavy relays (zap.stream's relay etc.) for discovery? *Recommendation:
+> user read relays first; make an optional extra-relay list configurable if
+> coverage is thin.*
+
+---
+
+## Config flags (off = no trace)
+
+Add **`FEATURE_LIVE_INGEST`** to `config/instance-config.js` (default `false`),
+threaded through `js/config.js` → `js/constants.js` like `FEATURE_NIP71_INGEST`.
+(The separate `FEATURE_LIVE_PUBLISH` for #16c is **not** part of this doc.)
+
+When `FEATURE_LIVE_INGEST` is off, all of the following must be absent:
+1. **Sidebar tab** — the "Live" link is not rendered in `components/sidebar.html`.
+2. **Feed/view registration** — `FEED_TYPES.LIVE` is not registered.
+3. **Subscriptions** — no kind-30311 discovery subscription and no kind-1311 chat
+   subscription are ever created.
+4. **Route** — `#view=live` falls back to the default view.
+5. **HLS library** — hls.js is dynamically imported only inside the live player,
+   so it never loads when the flag is off (keeps the bundle/network clean).
+6. **Config surface** — the two flags are the only footprint.
+
+---
+
+## Field mapping — kind 30311 (to VERIFY in Phase 0)
+
+Expected tag shape (confirm against captured real events before building):
+
+| Tag | Purpose |
+|-----|---------|
+| `d` | unique identifier (addressable key with pubkey+kind) |
+| `title` / `summary` | stream title / description |
+| `streaming` | **HLS `.m3u8` playback URL** (may appear more than once) |
+| `recording` | VOD URL when ended |
+| `status` | `planned` / `live` / `ended` |
+| `starts` / `ends` | unix timestamps |
+| `image` | thumbnail/poster |
+| `p` (role `host`/`participant`) | host + participants (pubkeys) |
+| `service` | provider hint (zap.stream etc.) |
+| `current_participants` / `total_participants` | viewer counts (optional) |
+
+Kind 1311 (live chat): a note tagged with the 30311 `a`-address; content is the
+chat message. Read-only ingest = subscribe by `#a`, render newest.
+
+> These are the *documented* shapes; **Phase 0 captures real events** from
+> zap.stream and shosho.live to confirm exact keys, multiplicity of `streaming`,
+> and how each provider marks live vs ended.
+
+---
+
+## Architecture
+
+- **Flag:** `FEATURE_LIVE_INGEST` (instance-config → config → constants).
+- **Feed type:** add `LIVE: "live"` to `FEED_TYPES` (`js/constants.js`).
+- **Discovery source:** a new live service (e.g. `js/services/liveIngestService.js`)
+  subscribing to kind 30311 (`status=live`, scoped per DECISION 1), normalizing
+  events into a lightweight "live card" model (title, host, image, streaming URL,
+  status, viewer count). Mirrors the deferred/throttled pattern of
+  `nip71IngestService` so it stays off the cold-start critical path.
+- **Sidebar tab + view:** "Live" link in `components/sidebar.html` (flag-gated) +
+  `views/live.html` (grid of live cards).
+- **Player:** a live player path. The current player
+  (`js/services/playbackService.js`) sets `videoEl.src` for URL/WebTorrent; HLS
+  needs hls.js attached to the `<video>` element when the source is `.m3u8`
+  (native only on Safari). Lazy-load hls.js on first live play (DECISION 3).
+- **Chat (later, DECISION 2):** read-only kind-1311 subscription by `#a`, rendered
+  beside the player; teardown on close.
+- **Liveness/staleness:** an event with `status=live` but no recent update / dead
+  `.m3u8` should be de-emphasized or dropped (see Risks).
+
+---
+
+## Phases (each flag-gated from day one)
+
+- **Phase 0 — Research spike (do FIRST, small but essential).**
+  - Capture real kind-30311 events from zap.stream + shosho.live; confirm tags,
+    `streaming` multiplicity, live/ended marking, host `p` roles.
+  - Prototype HLS playback in a throwaway page: hls.js attaching to `<video>`
+    with a real `.m3u8`; confirm CORS behavior from the static origin.
+  - Output: lock DECISIONS 1–5 and the field mapping; update this doc.
+- **Phase 1 — Discovery + Live tab (medium).** Flag, `FEED_TYPES.LIVE`, live
+  service (30311 discovery, scoped), sidebar link, `views/live.html`, grid of
+  live cards, route gated by flag. Clicking a card opens… (Phase 2).
+- **Phase 2 — HLS playback (medium).** Live player path + lazy hls.js; play the
+  `streaming` URL; graceful error when the stream is down.
+- **Phase 3 — Read-only chat (medium, DECISION 2).** kind-1311 subscription +
+  render + teardown.
+- **Phase 4 — Polish.** Viewer counts, ended/recording handling (DECISION 4),
+  optional extra discovery relays (DECISION 5), poster/skeleton states.
+
+---
+
+## Moderation, whitelist & NSFW
+
+Per DECISION 1 (recommended: whitelisted hosts): the live list is scoped to
+whitelisted authors and reuses the existing trust filter, so the admin
+whitelist/blacklist governs who can appear. Open question for chat (Phase 3):
+kind-1311 messages come from arbitrary viewers — decide whether chat inherits the
+WoT mute list or is simply unmoderated/read-only-with-hide. **Flag this before
+Phase 3.**
+
+---
+
+## Risks / watch-items
+
+- **HLS is new to the player.** Native `.m3u8` works only in Safari; Chrome/
+  Firefox need hls.js. This is the main build risk — de-risk in Phase 0.
+- **CORS on external streaming servers.** The `.m3u8` + segments are served by a
+  third-party origin; playback in a static site depends on their CORS. Verify in
+  Phase 0 (same class of constraint as the storage/edge CORS work).
+- **Stale "live" events.** Streams end without always updating `status`; a dead
+  `.m3u8` must fail gracefully and ideally be filtered. Needs a liveness signal
+  (recent update time / probe).
+- **Relay fan-out.** A live-discovery subscription adds REQ load — respect the
+  cold-login relay-cap / circuit-breaker invariants (AGENTS.md §17); keep it
+  deferred and capped like nip71 ingest.
+- **Scope creep into #16c.** Keep this strictly watch-only; broadcasting is a
+  separate flag + plan.
+
+---
+
+## Sources
+
+- NIP-53 (Live Activities; kind 30311 live event, kind 1311 live chat).
+- Reference implementations: zap.stream, shosho.live.
+- Existing code: `js/services/nip71IngestService.js` (deferred/throttled ingest
+  pattern to mirror), `js/services/playbackService.js` (player entry for HLS),
+  `js/app/feedCoordinator.js`, `components/sidebar.html`, `js/constants.js`
+  (`FEED_TYPES`).
+- hls.js (candidate HLS playback dependency, DECISION 3).
diff --git a/docs/shorts-plan.md b/docs/shorts-plan.md
new file mode 100644
index 00000000..5777a605
--- /dev/null
+++ b/docs/shorts-plan.md
@@ -0,0 +1,146 @@
+# Shorts (short-form vertical video) — Dev Plan
+
+TODO ref: **#16b** in `todo/TODO_2026-06-20_pre-launch.md`.
+Status: **PLANNING** — decisions open (see below). Not started.
+
+Short-form vertical video is **NIP-71 kind 22** (the short counterpart to kind 21
+normal video), plus the addressable short kind **34236**. It is a *different*
+feature from live streams (#16, NIP-53) — do not conflate them.
+
+The good news: **bitvid already ingests kind 22.** The inbound NIP-71 adapter
+(`js/nostr/nip71IngestAdapter.js`, `NIP71_KINDS = {21, 22, 34235, 34236}`) parses
+short-form notes today and they already flow into the main feed. So this feature
+is **not new ingest code** — it is (a) a way to distinguish "short" videos,
+(b) a dedicated **Shorts sidebar tab** with a vertical/swipe UI, and (c) a feed
+source that lists only shorts. Everything ships behind a config flag that leaves
+**no trace** when off.
+
+---
+
+## Decisions needed
+
+> **DECISION 1 — Discovery scope.** Does Shorts pull from the **same whitelisted
+> authors** as the main feed (reuse the existing render-time whitelist/WoT
+> filter), or does it have its **own discovery scope** (e.g. all authors, capped)?
+> *Recommendation: same whitelist scope as the main feed for v1 — least surprise,
+> reuses the existing moderation filter, no new relay fan-out.*
+
+> **DECISION 2 — Moderation / NSFW.** Inherit the existing filters
+> (blacklist / WoT mute / NSFW / private), or add short-specific handling?
+> *Recommendation: inherit — shorts are just kind-22 videos; the render-time
+> `filterVideos` path already applies.*
+
+> **DECISION 3 — Do shorts also appear in the main feed, or only under Shorts?**
+> Today ingested kind-22 videos land in the main grids. Options:
+> (a) leave them in the main feed AND add a Shorts tab (Shorts = a filtered view);
+> (b) exclude shorts from the main/Recent/For-You feeds and surface them ONLY in
+> the Shorts tab.
+> *Recommendation: (a) for v1 — simplest, no changes to existing feeds; revisit
+> exclusion later if the main feed feels diluted.*
+
+> **DECISION 4 — What counts as a "short"?** See Field mapping below; confirm the
+> detection rule (kind 22/34236 vs aspect-ratio) once we've eyeballed real events.
+
+> **DECISION 5 — UI ambition for v1.** Minimal (a normal grid that opens shorts in
+> a portrait player) vs full (TikTok-style full-screen vertical swipe/next feed)?
+> *Recommendation: ship the grid + portrait player first (small), then layer the
+> swipe feed as a phase 2 — keeps the first release cheap.*
+
+---
+
+## Config flag (off = no trace)
+
+Add `FEATURE_SHORTS` to `config/instance-config.js` (default `false`), threaded
+through `js/config.js` → `js/constants.js` exactly like `FEATURE_NIP71_INGEST`.
+
+When the flag is off, **all** of the following must be absent (the "as if it were
+never added" requirement):
+1. **Sidebar tab** — the Shorts link is not rendered in `components/sidebar.html`
+   (not merely `hidden`).
+2. **Feed registration** — `FEED_TYPES.SHORTS` is not registered in
+   `js/app/feedCoordinator.js` (no pipeline, no sorter).
+3. **Subscription/source** — no shorts-specific relay subscription is created.
+4. **Route** — `#view=shorts` falls back to the default view (a stale bookmark
+   can't resurrect the tab).
+5. **Config surface** — the flag is the only footprint.
+
+---
+
+## Field mapping — detecting a "short"
+
+A video object needs a reliable `isShort` marker so a feed source can filter.
+Candidate signals (to confirm against real events in Phase 0):
+
+| Signal | Meaning | Notes |
+|--------|---------|-------|
+| `kind === 22` | NIP-71 regular short | primary signal for ingested foreign shorts |
+| `kind === 34236` | NIP-71 **addressable** short | bitvid's own mirror uses 34235/36 (`nip71Mirror.js`) |
+| `imeta dim` height > width | portrait aspect | fallback when kind is ambiguous (matches the mirror's `short = hasDims && height > width` heuristic) |
+
+Plan: surface `isShort` on the bitvid video object in `nip71IngestAdapter.js`
+(it already tracks the source kind) and in the local mirror path, so both foreign
+and native shorts are detectable without re-fetching.
+
+---
+
+## Architecture
+
+Reuses the existing feed engine + ingest; new pieces are thin.
+
+- **Flag:** `FEATURE_SHORTS` (instance-config → config → constants).
+- **Feed type:** add `SHORTS: "shorts"` to `FEED_TYPES` (`js/constants.js`).
+- **Feed source/registration:** register `FEED_TYPES.SHORTS` in
+  `js/app/feedCoordinator.js`, sourced from the already-populated active-video
+  cache filtered to `isShort` (per DECISION 1, scoped to the same authors as the
+  main feed). A dedicated sorter can be chronological or reuse the For-You scorer
+  — TBD, likely chronological for v1.
+- **Sidebar tab:** add a Shorts link in `components/sidebar.html`, rendered only
+  when the flag is on.
+- **View:** `views/shorts.html` (a portrait-oriented grid container).
+- **Player UI:** v1 opens a short in the existing player modal with a portrait
+  layout; v2 adds a full-screen vertical swipe/next feed.
+- **Diversity:** apply the existing `spreadAuthors` pass (feed sorters) so the
+  Shorts tab doesn't cluster by creator, consistent with For You / Trending.
+
+---
+
+## Phases (each flag-gated from day one)
+
+- **Phase 0 — Detection + flag (small).** Add `FEATURE_SHORTS` (off), `isShort`
+  derivation in the ingest adapter + mirror, unit tests for detection. No UI.
+- **Phase 1 — Tab + feed (small–medium).** `FEED_TYPES.SHORTS`, sidebar link,
+  `views/shorts.html`, feed registration filtered to `isShort`, route gated by
+  flag. Grid renders; opens in the standard player.
+- **Phase 2 — Vertical UX (medium).** Portrait player layout + swipe/next feel.
+- **Phase 3 — Polish.** Autoplay-on-scroll, mute toggle, per-tab empty states,
+  optional exclusion from the main feed (DECISION 3 revisit).
+
+---
+
+## Moderation, whitelist & NSFW
+
+Per DECISION 2 (recommended: inherit): shorts pass through the same render-time
+`filterVideos` path (blacklist event ids, author blacklist, WoT mute, NSFW gate,
+private exclusion) as every other grid. No new moderation surface. The admin
+per-event block list (#25) already applies to any event id, shorts included.
+
+---
+
+## Risks / watch-items
+
+- **Aspect-ratio detection is soft** — not every short carries `imeta dim`; kind
+  is the reliable signal, dims are the fallback. Confirm in Phase 0.
+- **Double-listing** — if shorts stay in the main feed (DECISION 3a), make sure
+  the cross-ecosystem dedup (import-link + infohash, see nip71 onboarding) still
+  collapses a short that also exists as a 30078/34235 video.
+- **Empty tab** — with a whitelist-scoped source, a small instance may have zero
+  shorts; ship a clear empty state so the tab doesn't look broken.
+
+---
+
+## Sources
+
+- NIP-71 (video events; kinds 21/22 regular, 34235/34236 addressable).
+- Existing code: `js/nostr/nip71IngestAdapter.js`, `js/nostr/nip71Mirror.js`,
+  `js/app/feedCoordinator.js`, `js/feedEngine/sorters.js`, `components/sidebar.html`.
+- `docs/nip71-migration-plan.md` (dual-event model, short=34236 heuristic).
diff --git a/todo/TODO_2026-06-20_pre-launch.md b/todo/TODO_2026-06-20_pre-launch.md
index 82477719..e55dec7a 100644
--- a/todo/TODO_2026-06-20_pre-launch.md
+++ b/todo/TODO_2026-06-20_pre-launch.md
@@ -1337,6 +1337,7 @@ Threat model / cautions (write these into the feature + docs):
 > config flags from the start.
 
 ### 16. Nostr live streams — INGEST / watch-only (zap.stream, shosho.live)
+> **Dev plan: `docs/live-ingest-plan.md`** (decisions open + Phase 0 research spike).
 > Ingesting live streams (watch others' streams) and PUBLISHING a stream
 > ("go live", #16c) are TWO separate functions and must each have their OWN config
 > flag — the maintainer wants to enable/disable ingest and publish independently.
@@ -1366,6 +1367,7 @@ Threat model / cautions (write these into the feature + docs):
       streaming-server dependency early (relates to the CORS/edge constraints).
 
 ### 16b. Nostr short-form video notes — watch-only (new sidebar tab)
+> **Dev plan: `docs/shorts-plan.md`** (decisions open; ingest already exists).
 - [ ] Short-form (vertical/portrait) video is **NIP-71 kind 22** (the short-form
       counterpart to kind 21 normal video) — distinct from live streams. bitvid
       already ingests NIP-71 (see #17), so much of the parsing exists; this is about