From db4f11ba6d3f6b0d2989c14ce1805ce1d089b90e Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Sun, 15 Mar 2026 17:12:48 -0400 Subject: [PATCH] update --- AGENTS.md | 370 +++++++++++++++++ SPEC.md | 1138 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 1508 insertions(+) create mode 100644 AGENTS.md create mode 100644 SPEC.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..82b5de4 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,370 @@ +# AGENTS.md + +## Purpose + +This repository contains a Python tool that incrementally syncs downloadable file claims from configured LBRY/Odysee channels into a structured local archive. + +Primary goals: + +1. Resolve configured channels to stable LBRY identities. +2. Enumerate downloadable claims published by those channels. +3. Download only new claims or new versions. +4. Preserve old versions instead of overwriting them. +5. Maintain deterministic local structure and persistent state. + +This project prioritizes correctness, repeatability, and safe incremental behavior over cleverness. + +--- + +## Non-negotiable rules + +- Do not scrape Odysee HTML pages when the local LBRY daemon can provide the same metadata. +- Prefer stable identifiers over presentation-layer names: + - channel identity = `channel_claim_id` + - claim identity = `claim_id` + - version identity = `sd_hash` if available, otherwise a deterministic fallback +- Never use title or filename alone as identity. +- Never overwrite an existing downloaded version in place. +- Never mark a download as complete in state unless the file exists and metadata is written. +- Keep changes small and targeted. Do not do unrelated refactors. +- Do not add new dependencies unless they clearly reduce complexity or improve reliability. +- Do not add cloud services, databases, or background workers unless explicitly requested. + +--- + +## Expected architecture + +Agents should preserve this separation of concerns: + +- `main.py` + - entry point + - orchestration + - summary logging +- `config_loader.py` + - load and validate YAML config +- `lbry_client.py` + - all JSON-RPC communication with local `lbrynet` daemon + - isolate daemon/API compatibility here +- `planner.py` + - compare discovered claims against local state + - decide: skip / new / new_version / redownload_missing +- `downloader.py` + - execute downloads + - write metadata files + - finalize state updates +- `state_db.py` + - load/save state + - atomic writes only +- `models.py` + - dataclasses / typed structures +- `utils.py` + - slugifying, hashing, time helpers, filesystem helpers + +If the implementation diverges from this shape, preserve the same boundaries. + +--- + +## Filesystem contract + +Default base directory is expected to be: + +`~/Documents/lbry-downloads` + +Expected structure: + +```text +lbry-downloads/ + config.yaml + state/ + database.json + run-history.jsonl + channels/ + __/ + channel.json + claims/ + __/ + claim.json + versions/ + / + + metadata.json + download.json + checksums.txt +```` + +Rules: + +* Sanitize filesystem names. +* Preserve extensions when possible. +* Keep folder names deterministic. +* Store metadata beside each downloaded file version. +* Do not flatten version folders. + +--- + +## State management rules + +State is a core feature, not a convenience. + +Requirements: + +* Persistent state must exist across runs. +* State writes must be atomic. +* A claim may have multiple versions. +* A rerun must skip already-downloaded versions. +* If state says a version exists but the local file is missing, treat it as `redownload_missing`. +* Keep prior versions on disk unless the user explicitly asks for pruning. + +Preferred initial format: + +* JSON state database for v1 +* append-only `run-history.jsonl` for auditability + +Do not silently delete historical records. + +--- + +## Channel handling rules + +Accepted user inputs may include: + +* Odysee channel URLs +* LBRY URIs +* channel claim IDs + +Normalization rules: + +* Accept convenient input forms. +* Resolve to canonical LBRY metadata as early as possible. +* Store and operate on `channel_claim_id` internally. +* If a channel display name changes, keep continuity through the claim ID. + +Do not build logic around URL string shape beyond initial parsing. + +--- + +## Download rules + +A claim should be downloaded only if it is actually downloadable through the daemon. + +Include by default: + +* downloadable stream/file claims + +Exclude by default: + +* reposts +* channels +* collections +* unsupported claim types + +If repost support is added, make it config-driven and default-off. + +For every successful download: + +1. create channel / claim / version folders +2. download into the version folder +3. write normalized metadata +4. write raw or near-raw daemon response +5. optionally write checksum file +6. update state only after success + +Do not reorder that sequence. + +--- + +## Version detection rules + +Use this priority order for version identity: + +1. `sd_hash` +2. `stream_hash` +3. `txid:nout` +4. deterministic hash of normalized metadata JSON + +Interpretation: + +* same `claim_id` + same version token => skip +* same `claim_id` + different version token => new version +* unseen `claim_id` => new claim + +Do not use title changes, filename changes, or folder names as version signals. + +--- + +## Logging and reporting + +Logging should be useful during long sync runs. + +Always log: + +* daemon connectivity result +* channel resolution result +* scan progress +* per-claim action decision +* download success/failure +* final summary counts + +Preferred summary fields: + +* channels scanned +* claims examined +* new downloads +* new versions +* skipped existing +* redownloaded missing +* failures + +Make logs concise and grep-friendly. + +--- + +## Error handling + +Prefer graceful degradation over hard failure. + +Rules: + +* If daemon is unavailable, fail fast with a clear message. +* If one channel fails to resolve, log it and continue with the rest. +* If one claim fails, log it and continue. +* Do not write partial success to state. +* Do not swallow exceptions silently. +* If adding retries, keep them bounded and explicit. + +Avoid “best guess” behavior when metadata is incomplete. Prefer a logged skip. + +--- + +## Configuration rules + +Expected config file is YAML. + +Config should cover: + +* daemon API URL +* timeout +* base output directory +* state file path +* max workers +* dry-run +* checksum writing +* verification of existing files +* repost inclusion +* channel list + +Agents may extend config, but must follow these rules: + +* add defaults conservatively +* preserve backwards compatibility where practical +* validate new fields explicitly +* do not remove existing config keys without migration notes + +--- + +## Development workflow + +Before finishing changes: + +1. run formatting +2. run linting if configured +3. run tests if present +4. run a dry-run against sample config if practical +5. verify no obvious regression in path/state/version logic + +If the repo has a `Makefile`, `pyproject.toml`, `justfile`, or documented task runner, use that instead of inventing commands. + +Preferred commands when available: + +```bash +python -m pytest +python -m ruff check . +python -m ruff format . +python main.py --dry-run +``` + +If these tools are not configured in the repo, do not add them casually. + +--- + +## Code style + +* Target readable Python over clever Python. +* Prefer small pure functions for planning and normalization logic. +* Keep daemon I/O isolated from business logic. +* Add types where they materially improve clarity. +* Use dataclasses or clearly documented dict schemas. +* Avoid deeply nested control flow. +* Keep path logic deterministic and testable. + +When changing behavior, update: + +* config comments/examples +* state schema notes if relevant +* any sample output or CLI docs + +--- + +## Testing priorities + +If tests exist or are added, prioritize these cases: + +1. channel input normalization +2. slug/path sanitization +3. version token generation +4. skip vs new vs new-version decisions +5. missing-file redownload behavior +6. atomic state write behavior +7. metadata normalization stability + +The most valuable tests are the ones that prevent duplicate downloads or broken state. + +--- + +## Safe change policy + +Safe changes: + +* bug fixes in normalization, planning, downloading, or state handling +* clearer logging +* improved config validation +* additional tests +* daemon compatibility fixes isolated to client wrapper + +Changes that need extra caution: + +* state schema changes +* path layout changes +* version token logic changes +* concurrency changes +* anything that could duplicate downloads or orphan files + +For high-impact changes, preserve backwards compatibility or document a migration path. + +--- + +## Things agents should not do + +* Do not replace the daemon-based approach with scraping. +* Do not switch persistence to SQLite unless explicitly requested. +* Do not add async/concurrency complexity unless needed. +* Do not rename major folders or config keys casually. +* Do not remove version history. +* Do not “clean up” files automatically without an explicit retention policy. +* Do not assume Odysee URL formats are permanent. +* Do not assume filenames are unique. +* Do not commit secrets, auth tokens, or machine-specific absolute paths. + +--- + +## When uncertain + +When faced with an ambiguous implementation choice, prefer: + +1. stable identifiers over names +2. deterministic behavior over convenience +3. append-only/history-preserving behavior over destructive updates +4. explicit logging over silent magic +5. compatibility in `lbry_client.py` over leaking API quirks through the codebase + +This project is an incremental archive/sync tool. Protect the archive first. \ No newline at end of file diff --git a/SPEC.md b/SPEC.md new file mode 100644 index 0000000..00f78cd --- /dev/null +++ b/SPEC.md @@ -0,0 +1,1138 @@ +# LBRY / Odysee Channel Downloader — Detailed Implementation Spec + +## Goal + +Build a local Python tool that: + +1. Accepts a list of Odysee/LBRY channels in a config file. +2. Resolves each channel to stable LBRY identifiers. +3. Enumerates all downloadable file claims published by those channels. +4. Downloads each claim into a structured folder layout. +5. Skips anything already downloaded. +6. Detects new versions of previously seen content and downloads only the new version. +7. Maintains persistent state so repeated runs behave like an incremental sync. + +This should be designed for **robustness**, not page scraping. + +--- + +## Recommended Architecture + +### Use the LBRY SDK daemon, not Odysee HTML scraping + +The tool should talk to a locally running `lbrynet` daemon through its JSON-RPC API, typically at: + +```text +http://127.0.0.1:5279 +``` + +Why this is the right design: + +* Odysee web pages are presentation-layer HTML and can change. +* The daemon exposes blockchain-backed claim metadata. +* The daemon handles resolution and downloading. +* You get stable IDs like `claim_id`, `channel_claim_id`, `txid`, `nout`, `sd_hash`, and file info. +* You avoid reverse-engineering Odysee’s frontend behavior. + +### Input format recommendation + +Let users enter channels in either of these forms: + +* Odysee URL, for convenience +* LBRY URI, for correctness +* Channel claim ID, for maximum stability + +Accepted examples: + +```text +https://odysee.com/@SomeChannel:1 +lbry://@SomeChannel#1 +@SomeChannel#1 +f3b9b2b7c1c3d8e9a1234567890abcdef123456 +``` + +### Normalization strategy + +On the first successful resolve of a channel, store these canonical values in state: + +* original input +* normalized LBRY URI +* `channel_claim_id` +* current channel name + +From then on, use `channel_claim_id` internally whenever possible. + +This protects the sync process from: + +* display name changes +* URL changes +* case differences +* Odysee frontend URL changes + +--- + +## What counts as a “downloadable file” + +The tool should only process claims that actually resolve to downloadable content. + +In practice, that usually means stream/file claims that the daemon can download with `get`. + +### Include + +* stream claims with downloadable content +* claims with valid source blobs / `sd_hash` +* files returned by channel claim search that are downloadable through the daemon + +### Exclude + +* repost-only entries unless explicitly enabled +* channels +* collections/playlists +* unsupported claim types +* claims that resolve but are not downloadable files + +Config should allow optional inclusion of reposts: + +```yaml +include_reposts: false +``` + +Default should be `false`. + +--- + +## Folder Layout + +Base folder: + +```text +~/Documents/lbry-downloads/ +``` + +Within that: + +```text +lbry-downloads/ + config.yaml + state/ + database.json + run-history.jsonl + locks/ + channels/ + __/ + channel.json + claims/ + __/ + claim.json + versions/ + / + file.ext + metadata.json + download.json + checksums.txt +``` + +### Directory naming rules + +#### Channel folder name + +Use: + +```text +__ +``` + +Example: + +```text +TheModernInvestor__f3b9b2b7c1c3d8e9a1234567890abcdef123456 +``` + +#### Claim folder name + +Use: + +```text +__ +``` + +Example: + +```text +bitcoin-market-update__8ab12cd34ef56aa90bb11223344556677889900 +``` + +#### Version folder name + +Use a deterministic version token based on the best available content identity: + +Priority order: + +1. `sd_hash` +2. `stream_hash` +3. `txid:nout` +4. fallback hash of resolved metadata JSON + +Example: + +```text +sd_4f8c5d7e9a... +``` + +This gives a clean per-version subfolder and makes version detection robust. + +--- + +## State Model + +Use a persistent JSON database initially. + +File: + +```text +state/database.json +``` + +Later this can be upgraded to SQLite, but JSON is fine for v1. + +### Top-level shape + +```json +{ + "schema_version": 1, + "channels": {}, + "claims": {}, + "downloads": {}, + "last_run": null +} +``` + +### Channel record + +```json +{ + "input": "https://odysee.com/@SomeChannel:1", + "normalized_uri": "lbry://@SomeChannel#1", + "channel_claim_id": "f3b9...", + "channel_name": "@SomeChannel", + "folder": "channels/SomeChannel__f3b9...", + "last_scan": "2026-03-15T12:34:56Z" +} +``` + +### Claim record + +Key by claim ID. + +```json +{ + "claim_id": "8ab12...", + "channel_claim_id": "f3b9...", + "name": "bitcoin-market-update", + "permanent_url": "lbry://@SomeChannel#1/bitcoin-market-update#8", + "claim_folder": "channels/SomeChannel__f3b9.../claims/bitcoin-market-update__8ab12...", + "first_seen": "2026-03-15T12:34:56Z", + "last_seen": "2026-03-15T12:34:56Z", + "latest_version_token": "sd_4f8c5d7e...", + "versions": { + "sd_4f8c5d7e...": { + "txid": "...", + "nout": 0, + "sd_hash": "4f8c5d7e...", + "stream_hash": "...", + "published_at": "...", + "downloaded": true, + "file_relpath": "versions/sd_4f8c5d7e.../video.mp4" + } + } +} +``` + +### Why key by claim ID + +Because filenames and titles can change. + +Claim ID is the stable identity for the published item. + +### How versioning should work + +A claim may be updated over time. The same `claim_id` can point to newer content. + +The script should treat a claim as “new version available” when the current version token differs from the latest stored version token. + +That means: + +* same claim ID + same version token → skip +* same claim ID + different version token → download into a new version subfolder +* unseen claim ID → new download + +--- + +## Config File Spec + +Use YAML for readability. + +Path: + +```text +~/Documents/lbry-downloads/config.yaml +``` + +Example: + +```yaml +lbrynet: + api_url: "http://127.0.0.1:5279" + timeout_seconds: 60 + +general: + base_dir: "~/Documents/lbry-downloads" + state_file: "~/Documents/lbry-downloads/state/database.json" + max_workers: 2 + log_level: "INFO" + dry_run: false + verify_existing_files: true + write_checksums: true + filename_mode: "original" + include_reposts: false + channel_page_size: 50 + keep_missing_claim_records: true + +channels: + - input: "https://odysee.com/@SomeChannel:1" + enabled: true + tags_include: [] + tags_exclude: [] + - input: "lbry://@AnotherChannel#5" + enabled: true + tags_include: [] + tags_exclude: [] +``` + +### Field behavior + +#### `api_url` + +Location of the local daemon. + +#### `base_dir` + +Root output directory. + +#### `max_workers` + +Number of concurrent downloads. Keep low by default because daemon/resource behavior may vary. + +#### `dry_run` + +If true, the script resolves and plans actions but does not download. + +#### `verify_existing_files` + +If true, check that previously downloaded files still exist where expected. + +#### `write_checksums` + +If true, compute SHA256 for downloaded files and write `checksums.txt`. + +#### `filename_mode` + +Allowed values: + +* `original` → preserve daemon/original filename when possible +* `safe` → sanitize aggressively + +#### `channel_page_size` + +Batch size for paginated claim searches. + +#### `keep_missing_claim_records` + +If a previously seen claim no longer appears in a later channel scan, keep it in state instead of deleting it. + +--- + +## Functional Requirements + +## 1. Startup and validation + +On launch, the script must: + +1. Load config. +2. Expand `~` paths. +3. Create base directories if missing. +4. Load or initialize state DB. +5. Verify that `lbrynet` API is reachable. +6. Fail with a helpful message if the daemon is unavailable. + +Validation checks: + +* config file exists +* YAML parses +* channels list is not empty +* at least one channel is enabled +* daemon health check succeeds + +--- + +## 2. Channel resolution + +For each enabled channel: + +1. Parse the input string. +2. If input is an Odysee URL, extract the channel slug/claim suffix if possible. +3. Convert to a resolvable LBRY-style reference. +4. Call resolve through the daemon. +5. Confirm the resolved claim is a channel claim. +6. Store normalized channel metadata in state. + +Expected stored channel metadata: + +* display name +* normalized URI +* claim ID +* permanent URL if available +* short URL if available + +If resolution fails: + +* log error +* continue to next channel +* do not abort whole run unless `fail_fast` is enabled later + +--- + +## 3. Enumerating claims in a channel + +For each resolved channel, the script must enumerate all claims published by that channel. + +The enumeration should: + +* paginate until exhausted +* collect claim metadata records +* ignore duplicates +* filter non-downloadable items +* optionally exclude reposts + +Each candidate claim should be evaluated for: + +* `claim_id` +* `name` +* `title` +* `value_type` +* `sd_hash` +* file/source metadata +* release/publish timestamp +* claim sequence or tx info + +The exact daemon methods can vary slightly by SDK version, so implementation should isolate API calls in a client wrapper. + +### Client wrapper design + +Create a module like: + +```text +lbry_client.py +``` + +With functions such as: + +```python +def resolve(url_or_uri) -> dict: ... +def list_channel_claims(channel_claim_id: str, page: int, page_size: int) -> dict: ... +def get_claim(claim_id: str) -> dict: ... +def download(uri: str, download_directory: str) -> dict: ... +def file_list(claim_id: str | None = None) -> dict: ... +``` + +This keeps the rest of the app independent from API quirks. + +--- + +## 4. Deciding whether a claim should be downloaded + +For every claim found in the channel listing: + +### Step A — determine if downloadable + +A claim is eligible if: + +* it resolves to a stream/file +* daemon can provide download metadata +* it is not channel/collection-only content + +### Step B — compute version token + +Build a deterministic version token using: + +1. `sd_hash` +2. else `stream_hash` +3. else `txid:nout` +4. else hash of normalized metadata JSON + +### Step C — compare to state + +Cases: + +#### New claim + +`claim_id` not in state: + +* create claim record +* mark action = `download_new` + +#### Existing claim, same version + +`claim_id` exists and version token already recorded: + +* action = `skip_existing` + +#### Existing claim, new version + +`claim_id` exists but version token not recorded: + +* action = `download_new_version` + +--- + +## 5. Download behavior + +For any claim that needs downloading: + +1. Create channel folder if missing. +2. Create claim folder if missing. +3. Create version folder. +4. Ask daemon to download into that version folder. +5. Capture returned metadata. +6. Find resulting file path. +7. Write metadata files. +8. Update state only after download succeeds. + +### Files written per version + +Inside each version folder: + +```text +file.ext +metadata.json +download.json +checksums.txt +``` + +#### `metadata.json` + +Normalized metadata assembled by our tool: + +* claim ID +* channel claim ID +* title +* name +* version token +* permanent URL +* timestamps +* hashes +* original filename if known +* local file path + +#### `download.json` + +Raw or near-raw daemon response from the download call. + +#### `checksums.txt` + +SHA256 checksum of the downloaded file. + +--- + +## 6. Skip logic + +The skip logic should be conservative and deterministic. + +A file should be skipped only if: + +* claim ID exists in state +* version token exists in that claim’s `versions` +* expected local file exists +* optionally checksum still matches if verification is enabled later + +If state says a version is downloaded but the file is missing: + +* mark as `redownload_missing` +* download again into the same version folder if empty, or recreate safely + +### Avoid duplicate downloads caused by title changes + +Never rely on title or filename as identity. + +Identity must be: + +* claim ID for item identity +* version token for content version identity + +--- + +## 7. Updating state safely + +State writes must be atomic. + +Recommended pattern: + +1. Write updated JSON to `database.json.tmp` +2. fsync if possible +3. replace `database.json` + +Also append one line per action to: + +```text +state/run-history.jsonl +``` + +Each line example: + +```json +{"ts":"2026-03-15T12:45:00Z","channel_claim_id":"f3...","claim_id":"8ab...","action":"download_new_version","version_token":"sd_4f8...","status":"success"} +``` + +This gives easy auditing without parsing the whole DB. + +--- + +## 8. Logging + +The script should log: + +* daemon connectivity +* channel resolution results +* pagination progress +* skipped claims +* new claims +* new versions +* failures +* summary counts + +Recommended outputs: + +* console logging +* optional file logging later + +Example summary: + +```text +Run complete. +Channels scanned: 3 +Claims examined: 842 +New downloads: 7 +New versions: 2 +Skipped existing: 833 +Failures: 0 +``` + +--- + +## 9. Error handling + +The script should continue past individual failures where possible. + +Handle these separately: + +### Daemon unavailable + +Abort run with clear message. + +### Channel resolve failure + +Log and continue. + +### Claim metadata incomplete + +Log warning and skip. + +### Download failure + +Log failure and continue. +Do not mark as downloaded in state. + +### Partial/incomplete local file + +Either: + +* remove partial file and retry once, or +* mark failed and continue + +For v1, keep it simple: + +* no automatic retry beyond one retry per claim + +--- + +## 10. Suggested Python package structure + +```text +lbry-downloads/ + config.yaml + main.py + lbry_client.py + config_loader.py + state_db.py + planner.py + downloader.py + models.py + utils.py + requirements.txt +``` + +### Module responsibilities + +#### `main.py` + +Program entry point. +Coordinates the run. + +#### `lbry_client.py` + +JSON-RPC wrapper for the daemon. + +#### `config_loader.py` + +Loads and validates YAML config. + +#### `state_db.py` + +Loads, updates, and atomically saves state. + +#### `planner.py` + +Compares channel claims with state and decides actions. + +#### `downloader.py` + +Performs downloads and writes metadata. + +#### `models.py` + +Typed dataclasses / pydantic models if desired. + +#### `utils.py` + +Slugify, hashing, time helpers, path helpers. + +--- + +## 11. Suggested core algorithm + +```text +load config +ensure directories exist +load state +check daemon + +for each enabled channel: + resolve channel + save/update channel record + page through channel claims + for each claim: + if not downloadable: + continue + compute version token + compare against state + if new or new version or missing local file: + queue download action + else: + queue skip action + +execute queued downloads +write metadata +update state after each successful download +write run summary +``` + +--- + +## 12. Recommended metadata normalization + +Build a normalized metadata object for each claim version with fields like: + +```json +{ + "claim_id": "...", + "channel_claim_id": "...", + "name": "...", + "title": "...", + "claim_type": "stream", + "value_type": "stream", + "version_token": "sd_...", + "sd_hash": "...", + "stream_hash": "...", + "txid": "...", + "nout": 0, + "permanent_url": "...", + "short_url": "...", + "canonical_url": "...", + "timestamp": 0, + "release_time": 0, + "fee": null, + "tags": [], + "languages": [], + "source_name": "...", + "source_media_type": "...", + "file_name": "...", + "download_path": "..." +} +``` + +This makes future migration much easier. + +--- + +## 13. Filename and path sanitization + +All filesystem names should be sanitized. + +Rules: + +* replace `/`, `\\`, `:`, `*`, `?`, `"`, `<`, `>`, `|` +* collapse repeated whitespace +* trim trailing periods/spaces +* limit path component length, e.g. 120 chars +* preserve extension where possible + +Recommended slug helper behavior: + +```text +"Bitcoin Market Update!!!" -> "Bitcoin Market Update" +``` + +For stricter mode: + +```text +"Bitcoin Market Update!!!" -> "bitcoin-market-update" +``` + +--- + +## 14. Concurrency + +For v1, keep concurrency low. + +Recommendation: + +* enumerate sequentially +* download with `max_workers` of 1 or 2 + +Reason: + +* easier logging +* fewer daemon surprises +* less chance of file/path contention + +--- + +## 15. CLI behavior + +The script should support: + +```bash +python main.py +``` + +Optional future flags: + +```bash +python main.py --config ~/Documents/lbry-downloads/config.yaml +python main.py --dry-run +python main.py --channel @SomeChannel#1 +python main.py --repair-missing +``` + +For v1, minimum supported: + +* optional `--config` +* optional `--dry-run` + +--- + +## 16. Minimum external dependencies + +Recommended `requirements.txt`: + +```text +PyYAML>=6.0 +requests>=2.31.0 +``` + +Optional: + +```text +python-slugify>=8.0.0 +``` + +But a custom sanitizer is fine, so this can stay minimal. + +--- + +## 17. Daemon assumptions + +The spec assumes the user has installed and started the LBRY SDK daemon locally. + +The script should not try to install or manage the daemon itself. + +It should simply detect and report whether the daemon is reachable. + +Helpful startup error example: + +```text +Could not connect to LBRY daemon at http://127.0.0.1:5279. +Make sure lbrynet is installed and running before starting the downloader. +``` + +--- + +## 18. Why Odysee links in config are okay but not ideal as primary identity + +They are fine for user convenience. + +But the script should convert them internally to stable identifiers because: + +* Odysee is a frontend +* URLs can change format +* channel names can change +* claim IDs are more stable than presentation-layer URLs + +Best practice: + +* accept Odysee links +* resolve once +* store `channel_claim_id` +* use `channel_claim_id` and canonical LBRY URI from then on + +--- + +## 19. Practical recommendations for v1 + +### Use this identity model + +* Channel identity: `channel_claim_id` +* Claim identity: `claim_id` +* Version identity: `sd_hash` or fallback version token + +### Use this storage model + +* JSON state DB +* per-channel folders +* per-claim folders +* per-version subfolders + +### Use this behavior model + +* skip identical versions +* download only new claims or changed versions +* keep prior versions on disk + +That gives you: + +* reproducible organization +* incremental sync +* protection against duplicate downloads +* visible version history + +--- + +## 20. Nice-to-have future features + +Not required for v1, but useful later: + +* SQLite state DB +* daemon method compatibility layer for multiple SDK versions +* retry queue +* checksum verification of existing files on every run +* tag filters +* date filters +* delete/archival policy for removed claims +* export report CSV +* webhook or email notifications +* parallel downloads with backoff +* channel alias cache + +--- + +## 21. Explicit implementation decisions + +These are the recommended final choices for the first working version. + +### Decision 1 + +Use local `lbrynet` JSON-RPC API. + +### Decision 2 + +Accept Odysee URLs in config, but normalize to LBRY URI and `channel_claim_id`. + +### Decision 3 + +Organize downloads as: + +```text +channel -> claim -> version -> file +``` + +### Decision 4 + +Use `claim_id` as stable item identity. + +### Decision 5 + +Use `sd_hash` as primary version identity. + +### Decision 6 + +Store persistent state in JSON. + +### Decision 7 + +Skip only when both state and local file presence confirm the version already exists. + +### Decision 8 + +Write metadata files beside every downloaded file. + +--- + +## 22. Example sample config + +```yaml +lbrynet: + api_url: "http://127.0.0.1:5279" + timeout_seconds: 60 + +general: + base_dir: "~/Documents/lbry-downloads" + state_file: "~/Documents/lbry-downloads/state/database.json" + max_workers: 2 + log_level: "INFO" + dry_run: false + verify_existing_files: true + write_checksums: true + filename_mode: "original" + include_reposts: false + channel_page_size: 50 + keep_missing_claim_records: true + +channels: + - input: "https://odysee.com/@SomeChannel:1" + enabled: true + tags_include: [] + tags_exclude: [] + + - input: "lbry://@AnotherChannel#5" + enabled: true + tags_include: [] + tags_exclude: [] +``` + +--- + +## 23. Example first-run behavior + +Given a config with 2 channels: + +1. Script starts. +2. Connects to local daemon. +3. Resolves both channels. +4. Enumerates all downloadable claims. +5. Downloads every eligible file. +6. Writes state and metadata. + +Example resulting structure: + +```text +~/Documents/lbry-downloads/ + config.yaml + state/ + database.json + run-history.jsonl + channels/ + SomeChannel__f3b9.../ + channel.json + claims/ + bitcoin-market-update__8ab12.../ + claim.json + versions/ + sd_4f8c5d7e.../ + update.mp4 + metadata.json + download.json + checksums.txt +``` + +--- + +## 24. Example repeat-run behavior + +Second run: + +* same channels resolve successfully +* 500 existing claims unchanged +* 2 new claims published +* 1 old claim updated with new `sd_hash` + +Result: + +* 500 skipped +* 2 new downloads +* 1 new version folder added under an existing claim folder + +--- + +## 25. Bottom-line recommendation + +The most robust solution is: + +* **Use LBRY SDK locally** +* **Treat Odysee links as user input only** +* **Normalize channels to claim IDs** +* **Track claims by claim ID** +* **Track versions by `sd_hash`** +* **Download into channel/claim/version folders** +* **Store persistent state and metadata** + +That is the best balance of: + +* robustness +* maintainability +* incremental sync +* resistance to frontend/API changes + +--- + +## 26. Copy-paste starter config block + +```yaml +lbrynet: + api_url: "http://127.0.0.1:5279" + timeout_seconds: 60 + +general: + base_dir: "~/Documents/lbry-downloads" + state_file: "~/Documents/lbry-downloads/state/database.json" + max_workers: 2 + log_level: "INFO" + dry_run: false + verify_existing_files: true + write_checksums: true + filename_mode: "original" + include_reposts: false + channel_page_size: 50 + keep_missing_claim_records: true + +channels: + - input: "https://odysee.com/@SomeChannel:1" + enabled: true + tags_include: [] + tags_exclude: [] +``` + +--- + +## 27. Implementation note for the next step + +When turning this into code, the most important design choice is to isolate all daemon API interactions in one client module. That way, if the LBRY SDK method signatures differ slightly across versions, only one part of the code has to change.