mirror of
https://github.com/PR0M3TH3AN/bitvid.git
synced 2026-10-27 07:52:24 +00:00
views: deterministic d-tag so relays dedupe a viewer's repeat views (#4)
View counts inflated. viewCounter.js dedupes locally by (viewer, time-window) correctly, but the published events couldn't be deduped by relays: generateViewEventDedupeTag built the kind-30079 `d` tag with random entropy + exact timestamp, so every view was a UNIQUE event. And exactCountForPointer runs a NIP-45 COUNT (raw event count, no dedupe) and bumped the displayed total up to it (max(deduped, raw)). Net: reloads / multi-device / re-watches each added a view. Make the `d` tag deterministic and window-bucketed (scope:viewer:bucket, no entropy) so kind-30079 view events are genuinely parameterized-replaceable: relays keep one event per (viewer, video, window), so the cheap NIP-45 COUNT becomes accurate AND scalable and the raw count stops inflating. buildViewEvent no longer invents a random `d` tag (the only production caller, publishViewEvent, always supplies the deterministic one). Pre-existing entropy events age out over the backfill horizon. Tests: tests/view-event-dedupe.test.mjs. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
1 parent
926652156d
commit
a9cb8a94b4
4 files changed
+110
-16
No files matched your search
@@ -145,11 +145,14 @@ function generateViewEventDedupeTag(actorPubkey, pointer, createdAtSeconds) {
|
||||
typeof actorPubkey === "string" && actorPubkey.trim()
|
||||
? actorPubkey.trim().toLowerCase()
|
||||
: "anon";
|
||||
const timestamp = Number.isFinite(createdAtSeconds)
|
||||
? Math.max(0, Math.floor(createdAtSeconds))
|
||||
: Math.floor(Date.now() / 1000);
|
||||
const entropy = generateViewEventEntropy();
|
||||
return `${scope}:${normalizedActor}:${timestamp}:${entropy}`;
|
||||
// DETERMINISTIC + window-bucketed. Kind-30079 view events are parameterized-
|
||||
// replaceable (deduped by kind+pubkey+d-tag), so a stable d-tag per
|
||||
// (viewer, video, dedupe-window) makes the relay collapse a viewer's repeat
|
||||
// views in the same window into ONE event. That keeps both the NIP-45 COUNT and
|
||||
// a list() accurate AND scalable. The old entropy+exact-timestamp tag made every
|
||||
// view a unique event, defeating dedupe and inflating counts (item #4 audit).
|
||||
const bucketIndex = deriveViewEventBucketIndex(createdAtSeconds);
|
||||
return `${scope}:${normalizedActor}:${bucketIndex}`;
|
||||
}
|
||||
|
||||
export function hasRecentViewPublish(scope, bucketIndex, actorPubkey) {
|
||||
|
||||
@@ -2199,11 +2199,11 @@ export function buildViewEvent(params) {
|
||||
tags.push(...sanitizedAdditionalTags.map((tag) => tag.slice()));
|
||||
}
|
||||
|
||||
const resolvedDedupeTag =
|
||||
dedupeTag ||
|
||||
(schema?.identifierTag?.name
|
||||
? Math.random().toString(36).slice(2) + Math.random().toString(36).slice(2)
|
||||
: "");
|
||||
// No random fallback: view events must be parameterized-replaceable on the
|
||||
// exact d-tag the caller supplies (publishViewEvent computes a deterministic,
|
||||
// window-bucketed one). A random d-tag here would make every view a unique
|
||||
// event and defeat relay-side dedupe, inflating view counts (item #4 audit).
|
||||
const resolvedDedupeTag = typeof dedupeTag === "string" ? dedupeTag : "";
|
||||
|
||||
if (resolvedDedupeTag && schema?.identifierTag?.name) {
|
||||
const identifierName = schema.identifierTag.name;
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
// Item #4 (view-count accuracy): a view event's `d` tag must be DETERMINISTIC and
|
||||
// window-bucketed so kind-30079 view events are parameterized-replaceable —
|
||||
// relays then collapse a viewer's repeat views in the same window into one event,
|
||||
// making both the NIP-45 COUNT and a list() accurate and scalable. The old
|
||||
// entropy+timestamp tag made every view unique and inflated counts.
|
||||
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import { buildViewEvent } from "../js/nostrEventSchemas.js";
|
||||
import { __testExports } from "../js/nostr/viewEvents.js";
|
||||
import { VIEW_COUNT_DEDUPE_WINDOW_SECONDS } from "../js/config.js";
|
||||
|
||||
const { generateViewEventDedupeTag } = __testExports;
|
||||
|
||||
const VIEWER = "f".repeat(64);
|
||||
const OTHER = "0".repeat(64);
|
||||
const VIDEO = { type: "e", value: "video-abc" };
|
||||
const VIDEO2 = { type: "e", value: "video-xyz" };
|
||||
const W = VIEW_COUNT_DEDUPE_WINDOW_SECONDS;
|
||||
// A timestamp comfortably inside one window bucket, and another in the same one.
|
||||
const T0 = 100 * W + 10;
|
||||
const T0_SAME = 100 * W + (W - 1); // same bucket as T0
|
||||
const T_NEXT = 101 * W + 5; // next bucket
|
||||
|
||||
test("same viewer + video within one window → identical d-tag (replaceable)", () => {
|
||||
const a = generateViewEventDedupeTag(VIEWER, VIDEO, T0);
|
||||
const b = generateViewEventDedupeTag(VIEWER, VIDEO, T0_SAME);
|
||||
assert.equal(a, b, "repeat views in the same window must dedupe to one event");
|
||||
});
|
||||
|
||||
test("d-tag has no entropy: stable across calls", () => {
|
||||
const a = generateViewEventDedupeTag(VIEWER, VIDEO, T0);
|
||||
const b = generateViewEventDedupeTag(VIEWER, VIDEO, T0);
|
||||
assert.equal(a, b);
|
||||
});
|
||||
|
||||
test("next window → different d-tag (counts as a new view)", () => {
|
||||
const a = generateViewEventDedupeTag(VIEWER, VIDEO, T0);
|
||||
const c = generateViewEventDedupeTag(VIEWER, VIDEO, T_NEXT);
|
||||
assert.notEqual(a, c);
|
||||
});
|
||||
|
||||
test("different viewer or different video → different d-tag", () => {
|
||||
const base = generateViewEventDedupeTag(VIEWER, VIDEO, T0);
|
||||
assert.notEqual(base, generateViewEventDedupeTag(OTHER, VIDEO, T0));
|
||||
assert.notEqual(base, generateViewEventDedupeTag(VIEWER, VIDEO2, T0));
|
||||
});
|
||||
|
||||
test("buildViewEvent does NOT invent a random d-tag when none is supplied", () => {
|
||||
const event = buildViewEvent({
|
||||
pubkey: VIEWER,
|
||||
created_at: T0,
|
||||
pointerValue: VIDEO.value,
|
||||
pointerTag: ["e", VIDEO.value],
|
||||
});
|
||||
const dTags = event.tags.filter((t) => Array.isArray(t) && t[0] === "d");
|
||||
assert.equal(dTags.length, 0, "no d tag should be auto-generated (no entropy)");
|
||||
});
|
||||
|
||||
test("buildViewEvent uses the exact deterministic d-tag it is given", () => {
|
||||
const dedupeTag = generateViewEventDedupeTag(VIEWER, VIDEO, T0);
|
||||
const event = buildViewEvent({
|
||||
pubkey: VIEWER,
|
||||
created_at: T0,
|
||||
pointerValue: VIDEO.value,
|
||||
pointerTag: ["e", VIDEO.value],
|
||||
dedupeTag,
|
||||
});
|
||||
assert.ok(
|
||||
event.tags.some((t) => Array.isArray(t) && t[0] === "d" && t[1] === dedupeTag),
|
||||
"the supplied deterministic d-tag must be on the event",
|
||||
);
|
||||
});
|
||||
@@ -198,12 +198,29 @@ Audit started 2026-06-23. See the doc for architecture map + findings. Summary:
|
||||
mis-position making it hard to use — see 3a).
|
||||
|
||||
### 4. View counter accuracy & reliability — DEEP AUDIT
|
||||
- [ ] **Deep audit of the whole view-counting system** (`viewEvents.js`,
|
||||
`reactionCounter.js`, the quarantined `tests/view-counter.test.mjs`): accuracy
|
||||
and reliability — dedup per viewer/session, relay aggregation, double-counting,
|
||||
and resilience to dead relays (now that feed defaults are reserved, ensure view
|
||||
reads/writes use a sane relay set too). This audit underpins the popularity
|
||||
chart (#26) and the Trending tab (#27), so do it first / alongside them.
|
||||
Audit finding (2026-06-24): counts INFLATE. `viewCounter.js` dedupes locally by
|
||||
(viewer, time-window) correctly, but two things defeated it: (a)
|
||||
`generateViewEventDedupeTag` built the kind-30079 `d` tag with random entropy +
|
||||
exact timestamp, so every view was a UNIQUE event and relays could not dedupe the
|
||||
parameterized-replaceable event; (b) `exactCountForPointer` runs a NIP-45 COUNT
|
||||
(raw event count, no per-viewer dedupe) and bumped the displayed total UP to it
|
||||
(`max(deduped, raw)`). Net: reloads / multi-device / re-watches each added a view.
|
||||
- [x] **Root fix — deterministic, window-bucketed `d` tag** (`viewEvents.js`):
|
||||
`scope:viewer:bucket`, no entropy. Kind-30079 view events are now genuinely
|
||||
replaceable, so relays collapse a viewer's repeat views in a window into ONE
|
||||
event → the NIP-45 COUNT is accurate AND scalable, and `max(deduped, raw)`
|
||||
stops inflating. `buildViewEvent` no longer invents a random `d` tag
|
||||
(`nostrEventSchemas.js`). Transition: pre-existing entropy events age out over
|
||||
`VIEW_COUNT_BACKFILL_MAX_DAYS`. Tests: `tests/view-event-dedupe.test.mjs`
|
||||
(same-window dedupe, next-window separate, per-viewer/per-video, no auto-d).
|
||||
- [ ] **Remaining triage of the quarantined `tests/view-counter.test.mjs`** (still
|
||||
quarantined): the `d`-tag/session assertions now pass; it next fails on
|
||||
`viewCounter.js` cache-TTL/hydration behavior (stale cache should re-fetch
|
||||
fresh from relays). Separate from the accuracy root cause — triage + un-
|
||||
quarantine as its own pass.
|
||||
- [ ] **`reactionCounter.js`** (likes/kind-7) not yet audited; same dedupe-by-
|
||||
replaceable question may apply. Underpins the popularity chart (#26) /
|
||||
Trending tab (#27).
|
||||
|
||||
### 5. Card-hide / video-liveness check — full plan in docs/video-liveness-plan.md
|
||||
Audited (2026-06-23). Finding: the hide/show **policy is already correct**
|
||||
|
||||
Reference in new issue
Block a user