LBRY-Downloader/SPEC.md

LBRY / Odysee Channel Downloader — Detailed Implementation Spec

Goal

Build a local Python tool that:

1. Accepts a list of Odysee/LBRY channels in a config file.
2. Resolves each channel to stable LBRY identifiers.
3. Enumerates all downloadable file claims published by those channels.
4. Downloads each claim into a structured folder layout.
5. Skips anything already downloaded.
6. Detects new versions of previously seen content and downloads only the new version.
7. Maintains persistent state so repeated runs behave like an incremental sync.

This should be designed for robustness, not page scraping.

---

Recommended Architecture

Use the LBRY SDK daemon, not Odysee HTML scraping

The tool should talk to a locally running lbrynet daemon through its JSON-RPC API, typically at:

http://127.0.0.1:5279

Why this is the right design:

• Odysee web pages are presentation-layer HTML and can change.
• The daemon exposes blockchain-backed claim metadata.
• The daemon handles resolution and downloading.
• You get stable IDs like claim_id, channel_claim_id, txid, nout, sd_hash, and file info.
• You avoid reverse-engineering Odysee’s frontend behavior.

Input format recommendation

Let users enter channels in either of these forms:

• Odysee URL, for convenience
• LBRY URI, for correctness
• Channel claim ID, for maximum stability

Accepted examples:

https://odysee.com/@SomeChannel:1
lbry://@SomeChannel#1
@SomeChannel#1
f3b9b2b7c1c3d8e9a1234567890abcdef123456

Normalization strategy

On the first successful resolve of a channel, store these canonical values in state:

• original input
• normalized LBRY URI
• channel_claim_id
• current channel name

From then on, use channel_claim_id internally whenever possible.

This protects the sync process from:

• display name changes
• URL changes
• case differences
• Odysee frontend URL changes

---

What counts as a “downloadable file”

The tool should only process claims that actually resolve to downloadable content.

In practice, that usually means stream/file claims that the daemon can download with get.

Include

• stream claims with downloadable content
• claims with valid source blobs / sd_hash
• files returned by channel claim search that are downloadable through the daemon

Exclude

• repost-only entries unless explicitly enabled
• channels
• collections/playlists
• unsupported claim types
• claims that resolve but are not downloadable files

Config should allow optional inclusion of reposts:

include_reposts: false

Default should be false.

---

Folder Layout

Base folder:

~/Documents/lbry-downloads/

Within that:

lbry-downloads/
  config.yaml
  state/
    database.json
    run-history.jsonl
    locks/
  channels/
    <channel_slug>__<channel_claim_id>/
      channel.json
      claims/
        <claim_slug>__<claim_id>/
          claim.json
          versions/
            <version_token>/
              file.ext
              metadata.json
              download.json
              checksums.txt

Directory naming rules

Channel folder name

Use:

<sanitized_channel_name>__<channel_claim_id>

Example:

TheModernInvestor__f3b9b2b7c1c3d8e9a1234567890abcdef123456

Claim folder name

Use:

<sanitized_claim_name>__<claim_id>

Example:

bitcoin-market-update__8ab12cd34ef56aa90bb11223344556677889900

Version folder name

Use a deterministic version token based on the best available content identity:

Priority order:

1. sd_hash
2. stream_hash
3. txid:nout
4. fallback hash of resolved metadata JSON

Example:

sd_4f8c5d7e9a...

This gives a clean per-version subfolder and makes version detection robust.

---

State Model

Use a persistent JSON database initially.

File:

state/database.json

Later this can be upgraded to SQLite, but JSON is fine for v1.

Top-level shape

{
  "schema_version": 1,
  "channels": {},
  "claims": {},
  "downloads": {},
  "last_run": null
}

Channel record

{
  "input": "https://odysee.com/@SomeChannel:1",
  "normalized_uri": "lbry://@SomeChannel#1",
  "channel_claim_id": "f3b9...",
  "channel_name": "@SomeChannel",
  "folder": "channels/SomeChannel__f3b9...",
  "last_scan": "2026-03-15T12:34:56Z"
}

Claim record

Key by claim ID.

{
  "claim_id": "8ab12...",
  "channel_claim_id": "f3b9...",
  "name": "bitcoin-market-update",
  "permanent_url": "lbry://@SomeChannel#1/bitcoin-market-update#8",
  "claim_folder": "channels/SomeChannel__f3b9.../claims/bitcoin-market-update__8ab12...",
  "first_seen": "2026-03-15T12:34:56Z",
  "last_seen": "2026-03-15T12:34:56Z",
  "latest_version_token": "sd_4f8c5d7e...",
  "versions": {
    "sd_4f8c5d7e...": {
      "txid": "...",
      "nout": 0,
      "sd_hash": "4f8c5d7e...",
      "stream_hash": "...",
      "published_at": "...",
      "downloaded": true,
      "file_relpath": "versions/sd_4f8c5d7e.../video.mp4"
    }
  }
}

Why key by claim ID

Because filenames and titles can change.

Claim ID is the stable identity for the published item.

How versioning should work

A claim may be updated over time. The same claim_id can point to newer content.

The script should treat a claim as “new version available” when the current version token differs from the latest stored version token.

That means:

• same claim ID + same version token → skip
• same claim ID + different version token → download into a new version subfolder
• unseen claim ID → new download

---

Config File Spec

Use YAML for readability.

Path:

~/Documents/lbry-downloads/config.yaml

Example:

lbrynet:
  api_url: "http://127.0.0.1:5279"
  timeout_seconds: 60

general:
  base_dir: "~/Documents/lbry-downloads"
  state_file: "~/Documents/lbry-downloads/state/database.json"
  max_workers: 2
  log_level: "INFO"
  dry_run: false
  verify_existing_files: true
  write_checksums: true
  filename_mode: "original"
  include_reposts: false
  channel_page_size: 50
  keep_missing_claim_records: true

channels:
  - input: "https://odysee.com/@SomeChannel:1"
    enabled: true
    tags_include: []
    tags_exclude: []
  - input: "lbry://@AnotherChannel#5"
    enabled: true
    tags_include: []
    tags_exclude: []

Field behavior

api_url

Location of the local daemon.

base_dir

Root output directory.

max_workers

Number of concurrent downloads. Keep low by default because daemon/resource behavior may vary.

dry_run

If true, the script resolves and plans actions but does not download.

verify_existing_files

If true, check that previously downloaded files still exist where expected.

write_checksums

If true, compute SHA256 for downloaded files and write checksums.txt.

filename_mode

Allowed values:

• original → preserve daemon/original filename when possible
• safe → sanitize aggressively

channel_page_size

Batch size for paginated claim searches.

keep_missing_claim_records

If a previously seen claim no longer appears in a later channel scan, keep it in state instead of deleting it.

---

Functional Requirements

1. Startup and validation

On launch, the script must:

1. Load config.
2. Expand ~ paths.
3. Create base directories if missing.
4. Load or initialize state DB.
5. Verify that lbrynet API is reachable.
6. Fail with a helpful message if the daemon is unavailable.

Validation checks:

• config file exists
• YAML parses
• channels list is not empty
• at least one channel is enabled
• daemon health check succeeds

---

2. Channel resolution

For each enabled channel:

1. Parse the input string.
2. If input is an Odysee URL, extract the channel slug/claim suffix if possible.
3. Convert to a resolvable LBRY-style reference.
4. Call resolve through the daemon.
5. Confirm the resolved claim is a channel claim.
6. Store normalized channel metadata in state.

Expected stored channel metadata:

• display name
• normalized URI
• claim ID
• permanent URL if available
• short URL if available

If resolution fails:

• log error
• continue to next channel
• do not abort whole run unless fail_fast is enabled later

---

3. Enumerating claims in a channel

For each resolved channel, the script must enumerate all claims published by that channel.

The enumeration should:

• paginate until exhausted
• collect claim metadata records
• ignore duplicates
• filter non-downloadable items
• optionally exclude reposts

Each candidate claim should be evaluated for:

• claim_id
• name
• title
• value_type
• sd_hash
• file/source metadata
• release/publish timestamp
• claim sequence or tx info

The exact daemon methods can vary slightly by SDK version, so implementation should isolate API calls in a client wrapper.

Client wrapper design

Create a module like:

lbry_client.py

With functions such as:

def resolve(url_or_uri) -> dict: ...
def list_channel_claims(channel_claim_id: str, page: int, page_size: int) -> dict: ...
def get_claim(claim_id: str) -> dict: ...
def download(uri: str, download_directory: str) -> dict: ...
def file_list(claim_id: str | None = None) -> dict: ...

This keeps the rest of the app independent from API quirks.

---

4. Deciding whether a claim should be downloaded

For every claim found in the channel listing:

Step A — determine if downloadable

A claim is eligible if:

• it resolves to a stream/file
• daemon can provide download metadata
• it is not channel/collection-only content

Step B — compute version token

Build a deterministic version token using:

1. sd_hash
2. else stream_hash
3. else txid:nout
4. else hash of normalized metadata JSON

Step C — compare to state

Cases:

New claim

claim_id not in state:

• create claim record
• mark action = download_new

Existing claim, same version

claim_id exists and version token already recorded:

• action = skip_existing

Existing claim, new version

claim_id exists but version token not recorded:

• action = download_new_version

---

5. Download behavior

For any claim that needs downloading:

1. Create channel folder if missing.
2. Create claim folder if missing.
3. Create version folder.
4. Ask daemon to download into that version folder.
5. Capture returned metadata.
6. Find resulting file path.
7. Write metadata files.
8. Update state only after download succeeds.

Files written per version

Inside each version folder:

file.ext
metadata.json
download.json
checksums.txt

metadata.json

Normalized metadata assembled by our tool:

• claim ID
• channel claim ID
• title
• name
• version token
• permanent URL
• timestamps
• hashes
• original filename if known
• local file path

download.json

Raw or near-raw daemon response from the download call.

checksums.txt

SHA256 checksum of the downloaded file.

---

6. Skip logic

The skip logic should be conservative and deterministic.

A file should be skipped only if:

• claim ID exists in state
• version token exists in that claim’s versions
• expected local file exists
• optionally checksum still matches if verification is enabled later

If state says a version is downloaded but the file is missing:

• mark as redownload_missing
• download again into the same version folder if empty, or recreate safely

Avoid duplicate downloads caused by title changes

Never rely on title or filename as identity.

Identity must be:

• claim ID for item identity
• version token for content version identity

---

7. Updating state safely

State writes must be atomic.

Recommended pattern:

1. Write updated JSON to database.json.tmp
2. fsync if possible
3. replace database.json

Also append one line per action to:

state/run-history.jsonl

Each line example:

{"ts":"2026-03-15T12:45:00Z","channel_claim_id":"f3...","claim_id":"8ab...","action":"download_new_version","version_token":"sd_4f8...","status":"success"}

This gives easy auditing without parsing the whole DB.

---

8. Logging

The script should log:

• daemon connectivity
• channel resolution results
• pagination progress
• skipped claims
• new claims
• new versions
• failures
• summary counts

Recommended outputs:

• console logging
• optional file logging later

Example summary:

Run complete.
Channels scanned: 3
Claims examined: 842
New downloads: 7
New versions: 2
Skipped existing: 833
Failures: 0

---

9. Error handling

The script should continue past individual failures where possible.

Handle these separately:

Daemon unavailable

Abort run with clear message.

Channel resolve failure

Log and continue.

Claim metadata incomplete

Log warning and skip.

Download failure

Log failure and continue. Do not mark as downloaded in state.

Partial/incomplete local file

Either:

• remove partial file and retry once, or
• mark failed and continue

For v1, keep it simple:

• no automatic retry beyond one retry per claim

---

10. Suggested Python package structure

lbry-downloads/
  config.yaml
  main.py
  lbry_client.py
  config_loader.py
  state_db.py
  planner.py
  downloader.py
  models.py
  utils.py
  requirements.txt

Module responsibilities

main.py

Program entry point. Coordinates the run.

lbry_client.py

JSON-RPC wrapper for the daemon.

config_loader.py

Loads and validates YAML config.

state_db.py

Loads, updates, and atomically saves state.

planner.py

Compares channel claims with state and decides actions.

downloader.py

Performs downloads and writes metadata.

models.py

Typed dataclasses / pydantic models if desired.

utils.py

Slugify, hashing, time helpers, path helpers.

---

11. Suggested core algorithm

load config
ensure directories exist
load state
check daemon

for each enabled channel:
    resolve channel
    save/update channel record
    page through channel claims
    for each claim:
        if not downloadable:
            continue
        compute version token
        compare against state
        if new or new version or missing local file:
            queue download action
        else:
            queue skip action

execute queued downloads
write metadata
update state after each successful download
write run summary

---

12. Recommended metadata normalization

Build a normalized metadata object for each claim version with fields like:

{
  "claim_id": "...",
  "channel_claim_id": "...",
  "name": "...",
  "title": "...",
  "claim_type": "stream",
  "value_type": "stream",
  "version_token": "sd_...",
  "sd_hash": "...",
  "stream_hash": "...",
  "txid": "...",
  "nout": 0,
  "permanent_url": "...",
  "short_url": "...",
  "canonical_url": "...",
  "timestamp": 0,
  "release_time": 0,
  "fee": null,
  "tags": [],
  "languages": [],
  "source_name": "...",
  "source_media_type": "...",
  "file_name": "...",
  "download_path": "..."
}

This makes future migration much easier.

---

13. Filename and path sanitization

All filesystem names should be sanitized.

Rules:

• replace /, \\, :, *, ?, ", <, >, |
• collapse repeated whitespace
• trim trailing periods/spaces
• limit path component length, e.g. 120 chars
• preserve extension where possible

Recommended slug helper behavior:

"Bitcoin Market Update!!!" -> "Bitcoin Market Update"

For stricter mode:

"Bitcoin Market Update!!!" -> "bitcoin-market-update"

---

14. Concurrency

For v1, keep concurrency low.

Recommendation:

• enumerate sequentially
• download with max_workers of 1 or 2

Reason:

• easier logging
• fewer daemon surprises
• less chance of file/path contention

---

15. CLI behavior

The script should support:

python main.py

Optional future flags:

python main.py --config ~/Documents/lbry-downloads/config.yaml
python main.py --dry-run
python main.py --channel @SomeChannel#1
python main.py --repair-missing

For v1, minimum supported:

• optional --config
• optional --dry-run

---

16. Minimum external dependencies

Recommended requirements.txt:

PyYAML>=6.0
requests>=2.31.0

Optional:

python-slugify>=8.0.0

But a custom sanitizer is fine, so this can stay minimal.

---

17. Daemon assumptions

The spec assumes the user has installed and started the LBRY SDK daemon locally.

The script should not try to install or manage the daemon itself.

It should simply detect and report whether the daemon is reachable.

Helpful startup error example:

Could not connect to LBRY daemon at http://127.0.0.1:5279.
Make sure lbrynet is installed and running before starting the downloader.

---

18. Why Odysee links in config are okay but not ideal as primary identity

They are fine for user convenience.

But the script should convert them internally to stable identifiers because:

• Odysee is a frontend
• URLs can change format
• channel names can change
• claim IDs are more stable than presentation-layer URLs

Best practice:

• accept Odysee links
• resolve once
• store channel_claim_id
• use channel_claim_id and canonical LBRY URI from then on

---

19. Practical recommendations for v1

Use this identity model

• Channel identity: channel_claim_id
• Claim identity: claim_id
• Version identity: sd_hash or fallback version token

Use this storage model

• JSON state DB
• per-channel folders
• per-claim folders
• per-version subfolders

Use this behavior model

• skip identical versions
• download only new claims or changed versions
• keep prior versions on disk

That gives you:

• reproducible organization
• incremental sync
• protection against duplicate downloads
• visible version history

---

20. Nice-to-have future features

Not required for v1, but useful later:

• SQLite state DB
• daemon method compatibility layer for multiple SDK versions
• retry queue
• checksum verification of existing files on every run
• tag filters
• date filters
• delete/archival policy for removed claims
• export report CSV
• webhook or email notifications
• parallel downloads with backoff
• channel alias cache

---

21. Explicit implementation decisions

These are the recommended final choices for the first working version.

Decision 1

Use local lbrynet JSON-RPC API.

Decision 2

Accept Odysee URLs in config, but normalize to LBRY URI and channel_claim_id.

Decision 3

Organize downloads as:

channel -> claim -> version -> file

Decision 4

Use claim_id as stable item identity.

Decision 5

Use sd_hash as primary version identity.

Decision 6

Store persistent state in JSON.

Decision 7

Skip only when both state and local file presence confirm the version already exists.

Decision 8

Write metadata files beside every downloaded file.

---

22. Example sample config

lbrynet:
  api_url: "http://127.0.0.1:5279"
  timeout_seconds: 60

general:
  base_dir: "~/Documents/lbry-downloads"
  state_file: "~/Documents/lbry-downloads/state/database.json"
  max_workers: 2
  log_level: "INFO"
  dry_run: false
  verify_existing_files: true
  write_checksums: true
  filename_mode: "original"
  include_reposts: false
  channel_page_size: 50
  keep_missing_claim_records: true

channels:
  - input: "https://odysee.com/@SomeChannel:1"
    enabled: true
    tags_include: []
    tags_exclude: []

  - input: "lbry://@AnotherChannel#5"
    enabled: true
    tags_include: []
    tags_exclude: []

---

23. Example first-run behavior

Given a config with 2 channels:

1. Script starts.
2. Connects to local daemon.
3. Resolves both channels.
4. Enumerates all downloadable claims.
5. Downloads every eligible file.
6. Writes state and metadata.

Example resulting structure:

~/Documents/lbry-downloads/
  config.yaml
  state/
    database.json
    run-history.jsonl
  channels/
    SomeChannel__f3b9.../
      channel.json
      claims/
        bitcoin-market-update__8ab12.../
          claim.json
          versions/
            sd_4f8c5d7e.../
              update.mp4
              metadata.json
              download.json
              checksums.txt

---

24. Example repeat-run behavior

Second run:

• same channels resolve successfully
• 500 existing claims unchanged
• 2 new claims published
• 1 old claim updated with new sd_hash

Result:

• 500 skipped
• 2 new downloads
• 1 new version folder added under an existing claim folder

---

25. Bottom-line recommendation

The most robust solution is:

• Use LBRY SDK locally
• Treat Odysee links as user input only
• Normalize channels to claim IDs
• Track claims by claim ID
• Track versions by sd_hash
• Download into channel/claim/version folders
• Store persistent state and metadata

That is the best balance of:

• robustness
• maintainability
• incremental sync
• resistance to frontend/API changes

---

26. Copy-paste starter config block

lbrynet:
  api_url: "http://127.0.0.1:5279"
  timeout_seconds: 60

general:
  base_dir: "~/Documents/lbry-downloads"
  state_file: "~/Documents/lbry-downloads/state/database.json"
  max_workers: 2
  log_level: "INFO"
  dry_run: false
  verify_existing_files: true
  write_checksums: true
  filename_mode: "original"
  include_reposts: false
  channel_page_size: 50
  keep_missing_claim_records: true

channels:
  - input: "https://odysee.com/@SomeChannel:1"
    enabled: true
    tags_include: []
    tags_exclude: []

---

27. Implementation note for the next step

When turning this into code, the most important design choice is to isolate all daemon API interactions in one client module. That way, if the LBRY SDK method signatures differ slightly across versions, only one part of the code has to change.