Add direct-first download modes and resilience controls

2026-10-27 08:22:24 +00:00 · 2026-04-01 20:36:35 -04:00
1 parent 66d7a02458
commit 7b50110f2a
@@ -2,13 +2,23 @@

 A Python tool that incrementally syncs downloadable file claims from configured LBRY/Odysee channels into a structured local archive.

+## Recommended Defaults
+
+For most users:
+
+- Files only: `lbry-downloader --non-video-only`
+- Videos only: `lbry-downloader --video-only`
+- Mixed per-channel behavior: set `content_mode` in `config.yaml` and run `lbry-downloader`
+- Optional local-node mode: `lbry-downloader --p2p`
+
 ## Features

 - **Incremental Sync**: Only downloads new claims and updated versions
 - **Version Preservation**: Keeps old versions instead of overwriting them
 - **Robust Identity**: Uses stable LBRY claim IDs, not display names or titles
 - **Atomic State**: Database writes are atomic to prevent corruption
- **Daemon-Based**: Talks directly to local `lbrynet` daemon via JSON-RPC
+- **Direct Mode by Default**: Uses Odysee's public proxy + CDN without requiring a local node
+- **Optional P2P Mode**: Can use a local `lbrynet` daemon when explicitly requested
 - **Streaming Fallback**: Automatically uses daemon's streaming endpoint when P2P peers are unavailable
 - **Flexible Input**: Accepts Odysee URLs, LBRY URIs, or claim IDs
 - **Audit Trail**: Maintains run history in JSONL format
@@ -41,21 +51,22 @@ cd LBRY-Downloader

 The setup script will:
 - ✅ Check and install system dependencies
- ✅ Download and install LBRY SDK (lbrynet) - **No blockchain sync required!**
 - ✅ Set up a Python virtual environment
 - ✅ Install Python packages
 - ✅ Ask for your download location
 - ✅ Configure channels to download from
 - ✅ Set download limits
 - ✅ Create launcher scripts
+- ✅ Prepare a direct-first downloader setup that works without a local node
+- ✅ Leave `lbrynet` as an optional install only for users who want `--p2p`

-**Important:** The LBRY SDK (lbrynet) requires syncing blockchain **headers** (not full blocks) to function. On first run:
+**Important:** Direct mode is now the default and does not require `lbrynet`.
+
+If you choose to use `--p2p`, then `lbrynet` requires syncing blockchain **headers** (not full blocks) on first use:
 - Downloads ~1.7 million block headers (takes 5-10 minutes)
 - Uses minimal storage (headers only, not full blockchain)
- Subsequent starts are instant
- This is required to verify content claims on the network
-
-This is lighter than Bitcoin's full blockchain sync, but still necessary for the downloader to work.
+- Subsequent starts are much faster
+- This is only needed for the optional node-backed mode

 ### Manual Install

@@ -71,7 +82,9 @@ To remove LBRY Downloader:

 ## Prerequisites

-You need to have the LBRY SDK daemon (`lbrynet`) installed and running:
+Direct mode is the default, so a local LBRY daemon is not required for normal use.
+
+If you want to use P2P mode explicitly with `--p2p`, then you need to have the LBRY SDK daemon (`lbrynet`) installed and running:

 ```bash
 # The daemon should be accessible at http://127.0.0.1:5279
@@ -104,12 +117,19 @@ general:
  max_workers: 2
  dry_run: false
  include_reposts: false
+  direct_base_urls:
+    - "https://odysee.com"
+  direct_max_retries_per_url: 2
+  direct_retry_backoff_seconds: 2.0
+  direct_auto_fallback_to_p2p: false

 channels:
  - input: "https://odysee.com/@SomeChannel:1"
    enabled: true
+    content_mode: "all"
  - input: "lbry://@AnotherChannel#5"
    enabled: true
+    content_mode: "non_video_only"
 ```

 ### Finding Channel URLs
@@ -183,6 +203,36 @@ channels:
    # No download_path - uses general.base_dir/channels/
 ```

+### Per-Channel Content Mode
+
+You can control the download type for each channel independently.
+
+```yaml
+channels:
+  - input: "https://odysee.com/@VideoChannel:1"
+    enabled: true
+    content_mode: "video_only"
+
+  - input: "https://odysee.com/@FileArchive:1"
+    enabled: true
+    content_mode: "non_video_only"
+
+  - input: "https://odysee.com/@MixedChannel:1"
+    enabled: true
+    content_mode: "all"
+```
+
+Available values:
+
+- `all`: Download both video and non-video files for that channel.
+- `video_only`: Download only video files such as `.mp4`, `.webm`, and `.mkv`.
+- `non_video_only`: Download only non-video files such as `.zip`, PDFs, and similar file claims.
+
+Precedence:
+
+- Per-channel `content_mode` is the normal default for that channel.
+- CLI flags `--video-only` and `--non-video-only` override all channels for a single run.
+
 ### Download Limit

 Limit how many new downloads per channel (downloads most recent first):
@@ -201,6 +251,33 @@ This is useful for:
 - **Large channels**: Gradually sync channels with thousands of videos
 - **Bandwidth**: Control how much data is downloaded per run

+### Direct-Mode Rate Limit Fallbacks
+
+If Odysee starts returning `429 Too Many Requests`, you can tune the direct downloader instead of hardcoding random public mirrors.
+
+```yaml
+general:
+  direct_base_urls:
+    - "https://odysee.com"
+    # - "https://your-mirror.example"
+  direct_max_retries_per_url: 2
+  direct_retry_backoff_seconds: 2.0
+  direct_auto_fallback_to_p2p: false
+```
+
+What these do:
+
+- `direct_base_urls`: direct download bases to try in order. Odysee should normally stay first.
+- `direct_max_retries_per_url`: how many retries to allow after HTTP `429` for the same direct URL.
+- `direct_retry_backoff_seconds`: base delay for exponential backoff after HTTP `429`.
+- `direct_auto_fallback_to_p2p`: if `true`, a direct-mode run will try the local `lbrynet` daemon after repeated rate-limit failures.
+
+Recommended pattern:
+
+- Keep `https://odysee.com` first.
+- Add your own mirror only if you actually control or trust it.
+- Turn on `direct_auto_fallback_to_p2p: true` only if you also maintain a working local node.
+
 ## Launching the Application

 After installation, you can launch LBRY Downloader in several ways:
@@ -248,7 +325,7 @@ Now you can run `lbry-downloader` from any directory!

 ### Prerequisites Before Running

-**Important:** Make sure the LBRY daemon is running first:
+**Important:** If you are using `--p2p`, make sure the LBRY daemon is running first:

 ```bash
 # Start the daemon
@@ -260,15 +337,70 @@ lbrynet status

 ## Usage

+### Recommended Default: Direct + Non-Video Downloads Only
+
+If your main goal is to archive file attachments such as `.zip` bundles and skip video files such as `.mp4`, this should be the primary command you use:
+
 ```bash
-# Run with default config
-python main.py
+python3 main.py --non-video-only
+```

-# Use custom config
-python main.py --config ./my-config.yaml
+This mode keeps non-video claim downloads and skips video claims before the per-channel download limit is applied.

-# Dry run - see what would happen
-python main.py --dry-run
+### Common Commands
+
+```bash
+# Recommended default: download only non-video files such as zip archives
+python3 main.py --non-video-only
+
+# Download only video files such as mp4/webm/mkv
+python3 main.py --video-only
+
+# Download everything supported by the tool
+python3 main.py
+
+# Preview what would be downloaded without downloading anything
+python3 main.py --dry-run
+
+# Preview only non-video downloads
+python3 main.py --dry-run --non-video-only
+
+# Use a custom config file
+python3 main.py --config ./my-config.yaml
+
+# Run explicitly in direct mode (normally unnecessary because this is the default)
+python3 main.py --direct
+
+# Use the local LBRY daemon and P2P network instead of direct mode
+python3 main.py --p2p
+
+# Use P2P mode but keep only non-video files
+python3 main.py --p2p --non-video-only
+
+# Use P2P mode but keep only video files
+python3 main.py --p2p --video-only
+```
+
+### Flag Reference
+
+- `--non-video-only`: Download only non-video claims such as `.zip`, PDFs, and other non-video files.
+- `--video-only`: Download only video claims such as `.mp4`, `.webm`, and `.mkv`.
+- `--dry-run`: Show what would be downloaded without changing anything.
+- `--direct`: Use direct mode explicitly. This is already the default.
+- `--p2p`: Use the local LBRY daemon and peer-to-peer network instead of the default direct mode.
+- `--config PATH`: Use a config file other than `~/Documents/lbry-downloads/config.yaml`.
+
+If you configure `content_mode` per channel, you usually only need the CLI content flags when you want a temporary run-wide override.
+
+### Running Through the Launcher
+
+If you installed the launcher script, the same commands work there too:
+
+```bash
+lbry-downloader --non-video-only
+lbry-downloader --video-only
+lbry-downloader --dry-run --non-video-only
+lbry-downloader --p2p --non-video-only
 ```

 ## Directory Structure
@@ -319,15 +451,27 @@ Make sure `lbrynet` is running:
 lbrynet start
 ```

+This only applies when you are using `--p2p`. Direct mode does not require a local daemon.
+
 ### "Config file not found"
 Create a config at the default location or specify a path:
 ```bash
-python main.py --config ./config.yaml
+python3 main.py --config ./config.yaml
 ```

 ### Dry run shows no downloads
 Check that your channels are enabled and have downloadable content (streams, not just reposts).

+### HTTP 429 / Too Many Requests
+
+If direct mode is being rate-limited:
+
+1. Reduce how aggressively you rerun the downloader.
+2. Increase `direct_retry_backoff_seconds`.
+3. Keep `direct_max_retries_per_url` modest, such as `2` or `3`.
+4. Add a trusted mirror to `direct_base_urls` if you actually have one.
+5. Enable `direct_auto_fallback_to_p2p: true` if you want automatic fallback to the local node when direct mode keeps hitting `429`.
+
 ## Architecture

 - `main.py` - Entry point and orchestration
@@ -341,12 +485,12 @@ Check that your channels are enabled and have downloadable content (streams, not

 ## How Downloads Work

-The downloader uses a two-stage approach for maximum reliability:
+The downloader supports two transport modes:

-1. **P2P Attempt**: First tries to download from LBRY's peer-to-peer network
-2. **Streaming Fallback**: If P2P peers are unavailable (common for older content), automatically falls back to the daemon's local streaming endpoint
+1. **Direct mode (default)**: Uses Odysee's public proxy for metadata and Odysee CDN for the actual download
+2. **P2P mode (`--p2p`)**: Uses the local LBRY daemon and peer-to-peer network, with daemon streaming fallback when peers are unavailable

-This ensures content downloads successfully even when the P2P network has no seeders.
+Direct mode is the intended default for most users. P2P mode remains available when you specifically want node-backed downloading.

 ## License