mirror of
https://github.com/PR0M3TH3AN/LBRY-Downloader.git
synced 2026-10-27 08:22:24 +00:00
Complete LBRY Downloader implementation v1.0.0
Features: - Incremental sync with version tracking (sd_hash) - Per-channel download paths - Download limit (default: 10 most recent per channel) - Interactive setup script with user prompts - Uninstall script for clean removal - Comprehensive test suite (42 tests) - Full documentation suite Files: - 9 Python core modules (main, models, utils, config, client, state, planner, downloader, init) - 3 Setup scripts (setup.py, uninstall.py, install.sh) - 5 Test files (42 tests, all passing) - 9 Documentation files - 4 Build/config files Total: 31 files, ~4,500 lines
This commit is contained in:
thePR0M3TH3AN committed
1 parent
db4f11ba6d
commit
e707400ffd
31 files changed
+5825
No files matched your search
+52
@@ -0,0 +1,52 @@
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
*.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
|
||||
# Virtual environments
|
||||
venv/
|
||||
ENV/
|
||||
env/
|
||||
.venv
|
||||
|
||||
# IDEs
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
|
||||
# Testing
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
.tox/
|
||||
|
||||
# Application data (local testing)
|
||||
~/Documents/lbry-downloads/
|
||||
state/
|
||||
channels/
|
||||
*.json
|
||||
*.jsonl
|
||||
*.log
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
@@ -0,0 +1,49 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to this project will be documented in this file.
|
||||
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
- Initial implementation of LBRY Downloader
|
||||
- Support for incremental sync of LBRY/Odysee channels
|
||||
- Version tracking using sd_hash
|
||||
- Atomic state database writes
|
||||
- Configurable YAML configuration
|
||||
- Dry-run mode for testing
|
||||
- Run history logging in JSONL format
|
||||
- Comprehensive test suite
|
||||
- Support for Odysee URLs, LBRY URIs, and claim IDs
|
||||
- Automatic path sanitization
|
||||
- Checksum generation for downloaded files
|
||||
- **Per-channel download paths** - Each channel can have its own download location
|
||||
- **Download limit** - Limit downloads to most recent N items per channel (default: 10)
|
||||
- **Interactive setup script** - Automated installation with user prompts
|
||||
- **Uninstall script** - Easy removal of the application
|
||||
|
||||
### Technical Details
|
||||
- JSON-RPC client for lbrynet daemon
|
||||
- Modular architecture with clear separation of concerns
|
||||
- State management with claim and version tracking
|
||||
- Download planning and execution pipeline
|
||||
- Comprehensive error handling
|
||||
|
||||
## Future Ideas
|
||||
|
||||
### Planned
|
||||
- SQLite state database option for large archives
|
||||
- Tag-based filtering
|
||||
- Date range filtering
|
||||
- Parallel download support with rate limiting
|
||||
- Resume interrupted downloads
|
||||
- Export reports (CSV, JSON)
|
||||
- Webhook notifications
|
||||
|
||||
### Under Consideration
|
||||
- GUI interface
|
||||
- Docker containerization
|
||||
- Automatic thumbnail/metadata download
|
||||
- Video format selection
|
||||
- Bandwidth limiting
|
||||
@@ -0,0 +1,189 @@
|
||||
# Documentation Index
|
||||
|
||||
Not sure where to start? This guide helps you find the right documentation.
|
||||
|
||||
## 🚀 Getting Started
|
||||
|
||||
**New user? Start here:**
|
||||
|
||||
1. **[LINUX_SETUP.md](LINUX_SETUP.md)** - Complete Linux installation guide
|
||||
- System requirements
|
||||
- Step-by-step installation
|
||||
- Configuration examples
|
||||
- Troubleshooting common issues
|
||||
- Setting up automation
|
||||
|
||||
2. **[QUICKSTART.md](QUICKSTART.md)** - Get running in 5 minutes
|
||||
- Minimal setup instructions
|
||||
- Basic usage
|
||||
- Common commands
|
||||
|
||||
3. **[README.md](README.md)** - Project overview
|
||||
- Features
|
||||
- Basic usage
|
||||
- Architecture
|
||||
|
||||
## 📖 Detailed Documentation
|
||||
|
||||
### For Users
|
||||
|
||||
- **[README.md](README.md)** - Main project documentation
|
||||
- **[LINUX_SETUP.md](LINUX_SETUP.md)** - Linux-specific setup (Debian/Ubuntu/Mint)
|
||||
- **[QUICKSTART.md](QUICKSTART.md)** - Quick start guide
|
||||
- **[config.yaml.example](config.yaml.example)** - Example configuration
|
||||
- **[PER_CHANNEL_PATHS.md](docs/PER_CHANNEL_PATHS.md)** - Per-channel download paths guide
|
||||
|
||||
### Setup Scripts
|
||||
|
||||
- **[setup.py](setup.py)** - Interactive installation script (recommended)
|
||||
- Checks dependencies
|
||||
- Sets up virtual environment
|
||||
- Configures download location
|
||||
- Sets up channels
|
||||
- Creates launcher scripts
|
||||
|
||||
- **[uninstall.py](uninstall.py)** - Uninstallation script
|
||||
- Finds installation automatically
|
||||
- Shows what will be removed
|
||||
- Asks for confirmation
|
||||
- Cleans up PATH entries
|
||||
|
||||
### For Developers
|
||||
|
||||
- **[SPEC.md](SPEC.md)** - Detailed implementation specification
|
||||
- Architecture decisions
|
||||
- Data models
|
||||
- API design
|
||||
- File structure
|
||||
- State management
|
||||
|
||||
- **[AGENTS.md](AGENTS.md)** - Development guidelines
|
||||
- Coding standards
|
||||
- Architecture rules
|
||||
- Testing requirements
|
||||
- Change policies
|
||||
|
||||
- **[CHANGELOG.md](CHANGELOG.md)** - Version history and changes
|
||||
|
||||
## 🔧 Setup Guides by Platform
|
||||
|
||||
### Linux (Debian/Ubuntu/Mint)
|
||||
|
||||
**Quick Install:**
|
||||
```bash
|
||||
./install.sh
|
||||
```
|
||||
|
||||
**Manual Install:**
|
||||
See [LINUX_SETUP.md](LINUX_SETUP.md) for:
|
||||
- Installing Python and dependencies
|
||||
- Installing lbrynet daemon
|
||||
- Configuration setup
|
||||
- Systemd service setup
|
||||
- Cron job automation
|
||||
|
||||
### Other Platforms
|
||||
|
||||
The tool should work on any platform with Python 3.8+, but you'll need to:
|
||||
1. Install Python 3 and pip
|
||||
2. Install the LBRY SDK (`lbrynet`)
|
||||
3. Install Python dependencies from `requirements.txt`
|
||||
4. Run `python init.py`
|
||||
|
||||
## 📋 Configuration
|
||||
|
||||
- **Example:** [config.yaml.example](config.yaml.example)
|
||||
- **Location:** `~/Documents/lbry-downloads/config.yaml`
|
||||
- **Format:** YAML
|
||||
|
||||
## 🧪 Development
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
make test
|
||||
# OR
|
||||
python -m pytest tests/ -v
|
||||
```
|
||||
|
||||
### Code Structure
|
||||
|
||||
```
|
||||
main.py - Entry point
|
||||
models.py - Data structures
|
||||
utils.py - Helper functions
|
||||
config_loader.py - Configuration parsing
|
||||
lbry_client.py - Daemon communication
|
||||
state_db.py - State management
|
||||
planner.py - Download decisions
|
||||
downloader.py - Download execution
|
||||
```
|
||||
|
||||
See [SPEC.md](SPEC.md) for detailed architecture.
|
||||
|
||||
## 🐛 Troubleshooting
|
||||
|
||||
**Common issues are covered in:**
|
||||
- [LINUX_SETUP.md](LINUX_SETUP.md) - Platform-specific issues
|
||||
- [README.md](README.md) - General troubleshooting
|
||||
- [AGENTS.md](AGENTS.md) - Development/debugging
|
||||
|
||||
## 📦 Additional Files
|
||||
|
||||
- **[install.sh](install.sh)** - Automated Linux installer
|
||||
- **[init.py](init.py)** - Configuration initialization
|
||||
- **[Makefile](Makefile)** - Common development tasks
|
||||
- **[pyproject.toml](pyproject.toml)** - Python package configuration
|
||||
- **[requirements.txt](requirements.txt)** - Python dependencies
|
||||
|
||||
## 📝 Contributing
|
||||
|
||||
1. Read [AGENTS.md](AGENTS.md) for development guidelines
|
||||
2. Review [SPEC.md](SPEC.md) for architecture details
|
||||
3. Follow existing code style
|
||||
4. Add tests for new features
|
||||
5. Update documentation
|
||||
|
||||
## 🔗 External Resources
|
||||
|
||||
- **LBRY SDK:** https://github.com/lbryio/lbry-sdk
|
||||
- **LBRY Website:** https://lbry.com
|
||||
- **Odysee:** https://odysee.com
|
||||
|
||||
## 🆘 Getting Help
|
||||
|
||||
1. Check the logs: `~/Documents/lbry-downloads/state/run-history.jsonl`
|
||||
2. Run with debug mode: Set `log_level: "DEBUG"` in config
|
||||
3. Test daemon: `curl http://127.0.0.1:5279`
|
||||
4. Review [LINUX_SETUP.md](LINUX_SETUP.md) troubleshooting section
|
||||
|
||||
## 📚 Documentation Roadmap
|
||||
|
||||
- [x] README - Project overview
|
||||
- [x] QUICKSTART - Quick start guide
|
||||
- [x] LINUX_SETUP - Linux installation
|
||||
- [x] SPEC - Implementation details
|
||||
- [x] AGENTS - Development guidelines
|
||||
- [x] CHANGELOG - Version history
|
||||
- [ ] API documentation (generated)
|
||||
- [ ] Video tutorial (future)
|
||||
- [ ] FAQ (future)
|
||||
|
||||
## 🎯 Quick Reference
|
||||
|
||||
**Most users need:**
|
||||
1. [LINUX_SETUP.md](LINUX_SETUP.md) - To install
|
||||
2. [config.yaml.example](config.yaml.example) - To configure
|
||||
3. [README.md](README.md) - To understand features
|
||||
|
||||
**Developers need:**
|
||||
1. [AGENTS.md](AGENTS.md) - To understand code standards
|
||||
2. [SPEC.md](SPEC.md) - To understand architecture
|
||||
3. Source code - Well-commented Python files
|
||||
|
||||
---
|
||||
|
||||
**Can't find what you need?**
|
||||
- Check the source code - it's well-documented
|
||||
- Look at the tests in `tests/` for usage examples
|
||||
- Review [SPEC.md](SPEC.md) for design decisions
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 LBRY Downloader Contributors
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
+647
@@ -0,0 +1,647 @@
|
||||
# Linux Setup Guide (Debian/Ubuntu/Mint)
|
||||
|
||||
Complete instructions for setting up LBRY Downloader on Debian-based Linux distributions.
|
||||
|
||||
## Quick Start (Automated Setup)
|
||||
|
||||
The easiest way to install is using the interactive setup script:
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/yourusername/LBRY-Downloader.git
|
||||
cd LBRY-Downloader
|
||||
|
||||
# Run the interactive installer
|
||||
./setup.py
|
||||
```
|
||||
|
||||
This script will:
|
||||
1. Check and install system dependencies
|
||||
2. Set up a Python virtual environment
|
||||
3. Install Python packages
|
||||
4. Ask for your download location
|
||||
5. Configure channels to download from
|
||||
6. Set download limits
|
||||
7. Create launcher scripts
|
||||
|
||||
**Want to uninstall later?** Just run `./uninstall.py`
|
||||
|
||||
---
|
||||
|
||||
## Manual Installation
|
||||
|
||||
If you prefer to set things up manually, follow the steps below.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### 1. System Packages
|
||||
|
||||
```bash
|
||||
# Update system
|
||||
sudo apt update && sudo apt upgrade -y
|
||||
|
||||
# Install Python and required system packages
|
||||
sudo apt install -y \
|
||||
python3 \
|
||||
python3-pip \
|
||||
python3-venv \
|
||||
git \
|
||||
curl \
|
||||
wget
|
||||
|
||||
# Verify Python installation
|
||||
python3 --version # Should be 3.8 or higher
|
||||
pip3 --version
|
||||
```
|
||||
|
||||
### 2. Install LBRY SDK (lbrynet daemon)
|
||||
|
||||
The LBRY SDK is required to communicate with the LBRY network.
|
||||
|
||||
#### Option A: Download Pre-built Binary (Recommended)
|
||||
|
||||
```bash
|
||||
# Create directory for lbrynet
|
||||
mkdir -p ~/.local/bin
|
||||
cd ~/.local/bin
|
||||
|
||||
# Download latest release (check https://github.com/lbryio/lbry-sdk/releases for latest)
|
||||
# For x86_64 systems:
|
||||
LBRY_VERSION="0.113.0" # Check for latest version
|
||||
wget "https://github.com/lbryio/lbry-sdk/releases/download/v${LBRY_VERSION}/lbry-sdk-linux.zip"
|
||||
|
||||
# Extract
|
||||
unzip lbry-sdk-linux.zip
|
||||
rm lbry-sdk-linux.zip
|
||||
|
||||
# Make executable
|
||||
chmod +x lbrynet
|
||||
|
||||
# Add to PATH if not already there
|
||||
if ! echo $PATH | grep -q "$HOME/.local/bin"; then
|
||||
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
|
||||
source ~/.bashrc
|
||||
fi
|
||||
|
||||
# Verify installation
|
||||
lbrynet --version
|
||||
```
|
||||
|
||||
#### Option B: Build from Source (Advanced)
|
||||
|
||||
```bash
|
||||
# Install build dependencies
|
||||
sudo apt install -y \
|
||||
build-essential \
|
||||
libssl-dev \
|
||||
libffi-dev \
|
||||
python3-dev \
|
||||
python3-venv
|
||||
|
||||
# Clone and build (not recommended for most users)
|
||||
git clone https://github.com/lbryio/lbry-sdk.git
|
||||
cd lbry-sdk
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
### 1. Clone the Repository
|
||||
|
||||
```bash
|
||||
# Navigate to where you want to install
|
||||
cd ~/Documents # or wherever you prefer
|
||||
|
||||
# Clone (if you have git access) or extract the zip
|
||||
git clone https://github.com/yourusername/LBRY-Downloader.git
|
||||
# OR if you downloaded a zip:
|
||||
# unzip LBRY-Downloader.zip -d LBRY-Downloader
|
||||
|
||||
cd LBRY-Downloader
|
||||
```
|
||||
|
||||
### 2. Set Up Python Environment
|
||||
|
||||
```bash
|
||||
# Option A: Using system Python (simpler)
|
||||
pip3 install --user -r requirements.txt
|
||||
|
||||
# Option B: Using virtual environment (recommended for developers)
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 3. Verify Installation
|
||||
|
||||
```bash
|
||||
# Test Python imports
|
||||
python3 -c "from main import run_sync; print('✓ Imports OK')"
|
||||
|
||||
# Test CLI
|
||||
python3 main.py --help
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### 1. Initialize Configuration
|
||||
|
||||
```bash
|
||||
# Run initialization script
|
||||
python3 init.py
|
||||
|
||||
# This creates:
|
||||
# ~/Documents/lbry-downloads/config.yaml
|
||||
# ~/Documents/lbry-downloads/state/
|
||||
# ~/Documents/lbry-downloads/channels/
|
||||
```
|
||||
|
||||
### 2. Edit Configuration
|
||||
|
||||
```bash
|
||||
# Open config in your preferred editor
|
||||
nano ~/Documents/lbry-downloads/config.yaml
|
||||
# OR
|
||||
gedit ~/Documents/lbry-downloads/config.yaml
|
||||
# OR
|
||||
vim ~/Documents/lbry-downloads/config.yaml
|
||||
```
|
||||
|
||||
### 3. Add Your Channels
|
||||
|
||||
Edit the `channels:` section:
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
- input: "https://odysee.com/@SomeChannel:1"
|
||||
enabled: true
|
||||
tags_include: []
|
||||
tags_exclude: []
|
||||
|
||||
- input: "lbry://@AnotherChannel#5"
|
||||
enabled: true
|
||||
tags_include: []
|
||||
tags_exclude: []
|
||||
```
|
||||
|
||||
**Finding Channel URLs:**
|
||||
- Go to Odysee.com and navigate to a channel
|
||||
- Copy the URL from your browser
|
||||
- Paste it into the config
|
||||
|
||||
### 4. Configure Per-Channel Download Locations (Optional)
|
||||
|
||||
By default, all downloads go to `~/Documents/lbry-downloads/channels/`. You can specify custom locations for each channel:
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
# Downloads to default location
|
||||
- input: "https://odysee.com/@RegularChannel:1"
|
||||
enabled: true
|
||||
|
||||
# Downloads to custom location (external drive)
|
||||
- input: "https://odysee.com/@LargeVideos:1"
|
||||
enabled: true
|
||||
download_path: "/mnt/media/videos"
|
||||
|
||||
# Downloads to custom location (home directory)
|
||||
- input: "https://odysee.com/@Music:1"
|
||||
enabled: true
|
||||
download_path: "~/Music/LBRY"
|
||||
```
|
||||
|
||||
**Why use custom locations?**
|
||||
- Store large channels on external drives
|
||||
- Organize content by type (videos, music, etc.)
|
||||
- Manage disk space across multiple drives
|
||||
- Keep important channels on fast SSD, archive on slow HDD
|
||||
|
||||
**Note:** Make sure the custom path exists and you have write permissions:
|
||||
```bash
|
||||
mkdir -p /mnt/media/videos
|
||||
# Check permissions
|
||||
ls -la /mnt/media/
|
||||
```
|
||||
|
||||
### 5. Configure Download Limits (Optional)
|
||||
|
||||
Limit how many new items to download per channel. This is especially useful for:
|
||||
- **Testing** - Download just a few files first
|
||||
- **Large channels** - Channels with thousands of videos
|
||||
- **Bandwidth control** - Limit data usage per run
|
||||
|
||||
```yaml
|
||||
general:
|
||||
# Default: download 10 most recent items per channel
|
||||
download_limit: 10
|
||||
|
||||
# To download everything (use with caution!):
|
||||
# download_limit: 0
|
||||
|
||||
# For testing, download only 2-3 items:
|
||||
# download_limit: 3
|
||||
```
|
||||
|
||||
The tool downloads **most recent items first** (sorted by publish date), so:
|
||||
- `download_limit: 10` = Download the 10 newest videos
|
||||
- Older items remain in the channel but won't be downloaded yet
|
||||
- On next run, it will download the next 10 (if new items were published)
|
||||
|
||||
**Recommended approach for large channels:**
|
||||
```yaml
|
||||
general:
|
||||
download_limit: 10 # Start with a small number
|
||||
|
||||
channels:
|
||||
- input: "https://odysee.com/@HugeArchive:1"
|
||||
enabled: true
|
||||
# Downloads only 10 most recent to test
|
||||
```
|
||||
|
||||
Once you're happy with how it works, increase the limit or set to 0.
|
||||
|
||||
## Running the Daemon
|
||||
|
||||
The LBRY daemon (`lbrynet`) must be running before using the downloader.
|
||||
|
||||
### Start the Daemon
|
||||
|
||||
```bash
|
||||
# Start lbrynet
|
||||
lbrynet start
|
||||
|
||||
# Check status
|
||||
lbrynet status
|
||||
|
||||
# You should see JSON output with daemon information
|
||||
```
|
||||
|
||||
### Running as a Systemd Service (Optional)
|
||||
|
||||
Create a systemd service to auto-start lbrynet:
|
||||
|
||||
```bash
|
||||
# Create service file
|
||||
sudo tee /etc/systemd/system/lbrynet.service > /dev/null <<EOF
|
||||
[Unit]
|
||||
Description=LBRY SDK Daemon
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=$USER
|
||||
ExecStart=$HOME/.local/bin/lbrynet start
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
|
||||
# Reload systemd
|
||||
sudo systemctl daemon-reload
|
||||
|
||||
# Enable and start service
|
||||
sudo systemctl enable lbrynet
|
||||
sudo systemctl start lbrynet
|
||||
|
||||
# Check status
|
||||
sudo systemctl status lbrynet
|
||||
|
||||
# View logs
|
||||
sudo journalctl -u lbrynet -f
|
||||
```
|
||||
|
||||
### Stop the Daemon
|
||||
|
||||
```bash
|
||||
# If running manually
|
||||
lbrynet stop
|
||||
|
||||
# If running as systemd service
|
||||
sudo systemctl stop lbrynet
|
||||
```
|
||||
|
||||
## Using the Downloader
|
||||
|
||||
### First Run (Dry Run)
|
||||
|
||||
Always test with dry-run first:
|
||||
|
||||
```bash
|
||||
cd ~/Documents/LBRY-Downloader # or wherever you installed it
|
||||
|
||||
python3 main.py --dry-run
|
||||
```
|
||||
|
||||
This shows what would be downloaded without actually downloading anything.
|
||||
|
||||
### Normal Run
|
||||
|
||||
```bash
|
||||
# Make sure lbrynet is running
|
||||
lbrynet status
|
||||
|
||||
# Run the downloader
|
||||
python3 main.py
|
||||
```
|
||||
|
||||
### Custom Config Location
|
||||
|
||||
```bash
|
||||
python3 main.py --config /path/to/custom/config.yaml
|
||||
```
|
||||
|
||||
### Viewing Progress
|
||||
|
||||
The tool outputs progress to the console:
|
||||
|
||||
```
|
||||
2024-03-15 14:30:00 - INFO - LBRY Downloader starting
|
||||
2024-03-15 14:30:01 - INFO - Daemon is healthy
|
||||
2024-03-15 14:30:02 - INFO - Resolved channel: @SomeChannel (abc123...)
|
||||
2024-03-15 14:30:05 - INFO - Total claims found: 150
|
||||
2024-03-15 14:30:06 - INFO - New claim: interesting-video (download_new)
|
||||
...
|
||||
|
||||
Run complete.
|
||||
Channels scanned: 2
|
||||
Claims examined: 150
|
||||
New downloads: 3
|
||||
New versions: 1
|
||||
Skipped existing: 146
|
||||
Failures: 0
|
||||
```
|
||||
|
||||
## Directory Structure
|
||||
|
||||
After running, your downloads will be organized as:
|
||||
|
||||
```
|
||||
~/Documents/lbry-downloads/
|
||||
├── config.yaml # Your configuration
|
||||
├── state/
|
||||
│ ├── database.json # State database (don't edit)
|
||||
│ └── run-history.jsonl # Audit log
|
||||
└── channels/
|
||||
└── SomeChannel__abc123/ # Channel folder
|
||||
├── channel.json # Channel metadata
|
||||
└── claims/
|
||||
└── interesting-video__def456/
|
||||
├── claim.json # Claim metadata
|
||||
└── versions/
|
||||
└── sd_abc789/
|
||||
├── interesting-video.mp4
|
||||
├── metadata.json
|
||||
├── download.json
|
||||
└── checksums.txt
|
||||
```
|
||||
|
||||
## Automation (Optional)
|
||||
|
||||
### Create a Wrapper Script
|
||||
|
||||
```bash
|
||||
# Create convenient wrapper
|
||||
tee ~/bin/lbry-sync > /dev/null <<'EOF'
|
||||
#!/bin/bash
|
||||
cd ~/Documents/LBRY-Downloader
|
||||
source venv/bin/activate 2>/dev/null || true
|
||||
|
||||
# Check if lbrynet is running
|
||||
if ! curl -s http://127.0.0.1:5279 > /dev/null 2>&1; then
|
||||
echo "Starting lbrynet daemon..."
|
||||
lbrynet start &
|
||||
sleep 5
|
||||
fi
|
||||
|
||||
# Run downloader
|
||||
python3 main.py "$@"
|
||||
EOF
|
||||
|
||||
chmod +x ~/bin/lbry-sync
|
||||
|
||||
# Now you can just run:
|
||||
# lbry-sync
|
||||
# lbry-sync --dry-run
|
||||
```
|
||||
|
||||
### Cron Job for Regular Syncs
|
||||
|
||||
```bash
|
||||
# Edit crontab
|
||||
crontab -e
|
||||
|
||||
# Add line to run daily at 3 AM:
|
||||
0 3 * * * cd ~/Documents/LBRY-Downloader && python3 main.py >> ~/lbry-sync.log 2>&1
|
||||
|
||||
# Or weekly on Sundays at 2 AM:
|
||||
0 2 * * 0 cd ~/Documents/LBRY-Downloader && python3 main.py >> ~/lbry-sync.log 2>&1
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Could not connect to LBRY daemon"
|
||||
|
||||
```bash
|
||||
# Check if lbrynet is running
|
||||
lbrynet status
|
||||
|
||||
# If not running, start it
|
||||
lbrynet start
|
||||
|
||||
# Check if it's listening on the expected port
|
||||
netstat -tlnp | grep 5279
|
||||
# OR
|
||||
ss -tlnp | grep 5279
|
||||
|
||||
# Check daemon logs (if running as systemd)
|
||||
sudo journalctl -u lbrynet -n 50
|
||||
```
|
||||
|
||||
### "Permission denied" when running lbrynet
|
||||
|
||||
```bash
|
||||
# Ensure lbrynet is executable
|
||||
chmod +x ~/.local/bin/lbrynet
|
||||
|
||||
# Check PATH
|
||||
echo $PATH
|
||||
# Should include /home/YOURUSERNAME/.local/bin
|
||||
```
|
||||
|
||||
### "ModuleNotFoundError" for Python packages
|
||||
|
||||
```bash
|
||||
# Reinstall dependencies
|
||||
cd ~/Documents/LBRY-Downloader
|
||||
|
||||
# If using system Python:
|
||||
pip3 install --user --force-reinstall -r requirements.txt
|
||||
|
||||
# If using venv:
|
||||
source venv/bin/activate
|
||||
pip install --force-reinstall -r requirements.txt
|
||||
```
|
||||
|
||||
### "Config file not found"
|
||||
|
||||
```bash
|
||||
# Run initialization again
|
||||
python3 init.py
|
||||
|
||||
# Or manually create the directory
|
||||
mkdir -p ~/Documents/lbry-downloads/state
|
||||
mkdir -p ~/Documents/lbry-downloads/channels
|
||||
|
||||
# And create a config file from the example
|
||||
cp config.yaml.example ~/Documents/lbry-downloads/config.yaml
|
||||
```
|
||||
|
||||
### Downloads are slow
|
||||
|
||||
LBRY downloads can be slow depending on network conditions. This is normal:
|
||||
|
||||
```yaml
|
||||
# In config.yaml, reduce max_workers if needed
|
||||
general:
|
||||
max_workers: 1 # Download one at a time
|
||||
timeout_seconds: 300 # Increase timeout
|
||||
```
|
||||
|
||||
### Disk space issues
|
||||
|
||||
Check the download directory size:
|
||||
|
||||
```bash
|
||||
du -sh ~/Documents/lbry-downloads/channels/
|
||||
du -sh ~/Documents/lbry-downloads/channels/* | sort -hr | head -20
|
||||
```
|
||||
|
||||
## Updating
|
||||
|
||||
### Update the Downloader
|
||||
|
||||
```bash
|
||||
cd ~/Documents/LBRY-Downloader
|
||||
|
||||
# If using git:
|
||||
git pull
|
||||
|
||||
# If manual download, re-download and replace files
|
||||
|
||||
# Update Python dependencies
|
||||
pip3 install --user --upgrade -r requirements.txt
|
||||
# OR if using venv:
|
||||
source venv/bin/activate
|
||||
pip install --upgrade -r requirements.txt
|
||||
```
|
||||
|
||||
### Update lbrynet
|
||||
|
||||
```bash
|
||||
# Stop current daemon
|
||||
lbrynet stop
|
||||
|
||||
# Download new version (see installation section)
|
||||
cd ~/.local/bin
|
||||
# ... download and extract new version ...
|
||||
|
||||
# Restart
|
||||
lbrynet start
|
||||
```
|
||||
|
||||
## Uninstallation
|
||||
|
||||
```bash
|
||||
# Stop daemon
|
||||
lbrynet stop
|
||||
|
||||
# Remove downloader
|
||||
rm -rf ~/Documents/LBRY-Downloader
|
||||
rm -rf ~/Documents/lbry-downloads
|
||||
|
||||
# Remove lbrynet (if installed locally)
|
||||
rm ~/.local/bin/lbrynet
|
||||
|
||||
# Remove systemd service (if created)
|
||||
sudo systemctl stop lbrynet
|
||||
sudo systemctl disable lbrynet
|
||||
sudo rm /etc/systemd/system/lbrynet.service
|
||||
sudo systemctl daemon-reload
|
||||
|
||||
# Remove cron job
|
||||
crontab -e
|
||||
# Delete the line with lbry-downloader
|
||||
```
|
||||
|
||||
## Getting Help
|
||||
|
||||
1. Check the logs: `~/Documents/lbry-downloads/state/run-history.jsonl`
|
||||
2. Run with debug logging: Set `log_level: "DEBUG"` in config.yaml
|
||||
3. Test daemon connectivity:
|
||||
```bash
|
||||
curl -X POST http://127.0.0.1:5279 \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"jsonrpc":"2.0","method":"status","params":{},"id":1}'
|
||||
```
|
||||
|
||||
## Uninstallation
|
||||
|
||||
To remove LBRY Downloader:
|
||||
|
||||
```bash
|
||||
# Run the uninstall script
|
||||
./uninstall.py
|
||||
```
|
||||
|
||||
This will:
|
||||
- Find your installation and download directories
|
||||
- Ask for confirmation before removing
|
||||
- Show you what's being deleted
|
||||
- Optionally help clean up PATH entries
|
||||
|
||||
**Manual uninstall:**
|
||||
|
||||
```bash
|
||||
# Stop the daemon
|
||||
lbrynet stop
|
||||
|
||||
# Remove installation directory
|
||||
rm -rf ~/Documents/LBRY-Downloader
|
||||
|
||||
# Remove downloads (optional - backup first!)
|
||||
rm -rf ~/Documents/lbry-downloads
|
||||
|
||||
# Remove lbrynet (optional)
|
||||
rm ~/.local/bin/lbrynet
|
||||
|
||||
# Remove from PATH if you added it
|
||||
# Edit ~/.bashrc and remove the lbry-downloader PATH line
|
||||
```
|
||||
|
||||
## Quick Reference
|
||||
|
||||
```bash
|
||||
# Automated Setup (one-time)
|
||||
./setup.py
|
||||
|
||||
# Manual Setup (one-time)
|
||||
sudo apt install python3 python3-pip
|
||||
pip3 install --user -r requirements.txt
|
||||
python3 init.py
|
||||
|
||||
# Daily use
|
||||
lbrynet start # Start daemon
|
||||
python3 main.py --dry-run # Test first
|
||||
python3 main.py # Run sync
|
||||
lbrynet stop # Stop daemon (optional)
|
||||
|
||||
# Locations
|
||||
Config: ~/Documents/lbry-downloads/config.yaml
|
||||
Downloads: ~/Documents/lbry-downloads/channels/
|
||||
State: ~/Documents/lbry-downloads/state/
|
||||
Logs: ~/Documents/lbry-downloads/state/run-history.jsonl
|
||||
```
|
||||
@@ -0,0 +1,40 @@
|
||||
.PHONY: help install test lint format clean run dry-run
|
||||
|
||||
help:
|
||||
@echo "LBRY Downloader - Available commands:"
|
||||
@echo " make install - Install dependencies"
|
||||
@echo " make test - Run tests"
|
||||
@echo " make lint - Run linting (if configured)"
|
||||
@echo " make format - Format code (if configured)"
|
||||
@echo " make clean - Clean generated files"
|
||||
@echo " make run - Run the downloader"
|
||||
@echo " make dry-run - Run in dry-run mode"
|
||||
|
||||
install:
|
||||
pip install -r requirements.txt
|
||||
@echo "Dependencies installed"
|
||||
|
||||
test:
|
||||
python3 -m pytest tests/ -v
|
||||
|
||||
lint:
|
||||
@echo "Linting not configured. Add ruff or flake8 to requirements.txt"
|
||||
@echo "and configure this target."
|
||||
|
||||
format:
|
||||
@echo "Formatting not configured. Add ruff or black to requirements.txt"
|
||||
@echo "and configure this target."
|
||||
|
||||
clean:
|
||||
find . -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true
|
||||
find . -type f -name "*.pyc" -delete
|
||||
find . -type f -name "*.pyo" -delete
|
||||
find . -type d -name "*.egg-info" -exec rm -rf {} + 2>/dev/null || true
|
||||
find . -type d -name .pytest_cache -exec rm -rf {} + 2>/dev/null || true
|
||||
rm -rf build/ dist/ .coverage htmlcov/
|
||||
|
||||
run:
|
||||
python3 main.py
|
||||
|
||||
dry-run:
|
||||
python3 main.py --dry-run
|
||||
@@ -0,0 +1,249 @@
|
||||
# Project Summary
|
||||
|
||||
## LBRY Downloader v1.0.0
|
||||
|
||||
A complete Python tool for incrementally syncing LBRY/Odysee channel content.
|
||||
|
||||
---
|
||||
|
||||
## What Was Built
|
||||
|
||||
### Core Application (2,165 lines of Python)
|
||||
|
||||
1. **main.py** (312 lines)
|
||||
- CLI entry point with argparse
|
||||
- Orchestrates the entire sync process
|
||||
- Channel resolution and claim enumeration
|
||||
- Statistics tracking and summary reporting
|
||||
|
||||
2. **models.py** (244 lines)
|
||||
- Dataclasses for all data structures
|
||||
- Config, Channel, Claim, ClaimVersion, DownloadAction
|
||||
- StateDatabase with serialization
|
||||
- NormalizedMetadata structure
|
||||
|
||||
3. **utils.py** (303 lines)
|
||||
- Path sanitization and slugification
|
||||
- Version token generation (sd_hash priority)
|
||||
- File type detection (downloadable claims)
|
||||
- URL normalization (Odysee to LBRY)
|
||||
- Metadata extraction from daemon responses
|
||||
- SHA256 checksums
|
||||
|
||||
4. **config_loader.py** (185 lines)
|
||||
- YAML configuration parsing
|
||||
- Path expansion and validation
|
||||
- Config error handling
|
||||
- Directory initialization
|
||||
|
||||
5. **lbry_client.py** (264 lines)
|
||||
- JSON-RPC client for lbrynet daemon
|
||||
- All daemon methods: resolve, claim_search, get, file_list
|
||||
- Error handling and timeout management
|
||||
- Health check functionality
|
||||
|
||||
6. **state_db.py** (175 lines)
|
||||
- Atomic JSON state database
|
||||
- Channel and claim CRUD operations
|
||||
- Run history logging (JSONL format)
|
||||
- Statistics generation
|
||||
|
||||
7. **planner.py** (203 lines)
|
||||
- Download decision logic
|
||||
- New claim detection
|
||||
- Version change detection
|
||||
- Skip/redownload logic
|
||||
- State updates from actions
|
||||
|
||||
8. **downloader.py** (208 lines)
|
||||
- Download execution
|
||||
- File management and moving
|
||||
- Metadata writing (metadata.json, download.json)
|
||||
- Checksum generation
|
||||
|
||||
9. **init.py** (62 lines)
|
||||
- Configuration initialization script
|
||||
- Directory structure setup
|
||||
- User guidance
|
||||
|
||||
### Documentation
|
||||
|
||||
- **README.md** - Comprehensive user documentation
|
||||
- **QUICKSTART.md** - Getting started guide
|
||||
- **SPEC.md** - Detailed implementation specification
|
||||
- **AGENTS.md** - Developer guidelines and rules
|
||||
- **CHANGELOG.md** - Version history
|
||||
- **LICENSE** - MIT License
|
||||
|
||||
### Configuration & Build
|
||||
|
||||
- **config.yaml.example** - Example configuration
|
||||
- **pyproject.toml** - Modern Python packaging
|
||||
- **requirements.txt** - Dependencies (PyYAML, requests)
|
||||
- **Makefile** - Common tasks (test, clean, run)
|
||||
- **.gitignore** - Git ignore patterns
|
||||
|
||||
### Tests (38 tests, all passing)
|
||||
|
||||
- **test_utils.py** - 16 tests for utility functions
|
||||
- **test_config.py** - 7 tests for configuration loading
|
||||
- **test_planner.py** - 4 tests for download planning
|
||||
- **test_state_db.py** - 7 tests for state management
|
||||
|
||||
---
|
||||
|
||||
## Features Implemented
|
||||
|
||||
### Core Functionality
|
||||
✅ Channel resolution (Odysee URL, LBRY URI, claim ID)
|
||||
✅ Incremental sync (only new/changed content)
|
||||
✅ Version tracking (sd_hash, stream_hash, txid fallback)
|
||||
✅ Atomic state writes (prevents corruption)
|
||||
✅ Run history logging (JSONL format)
|
||||
✅ Dry-run mode (test without downloading)
|
||||
✅ Configurable YAML configuration
|
||||
✅ Comprehensive logging
|
||||
✅ Error handling and recovery
|
||||
|
||||
### Download Management
|
||||
✅ Structured folder layout (channel/claim/version)
|
||||
✅ **Per-channel download paths** - Custom location per channel
|
||||
✅ Metadata preservation (metadata.json, download.json)
|
||||
✅ Checksum generation (SHA256)
|
||||
✅ File verification
|
||||
✅ Missing file redownload
|
||||
✅ Repost filtering (optional)
|
||||
|
||||
### Quality & Testing
|
||||
✅ 42 unit tests (all passing)
|
||||
✅ Type hints throughout
|
||||
✅ Clear error messages
|
||||
✅ Input validation
|
||||
✅ Path sanitization
|
||||
✅ Atomic operations
|
||||
|
||||
---
|
||||
|
||||
## Architecture Highlights
|
||||
|
||||
### Design Principles
|
||||
- **Daemon-first**: Uses local lbrynet, not HTML scraping
|
||||
- **Identity-stable**: claim_id + sd_hash, not titles/names
|
||||
- **Version-preserving**: Keeps old versions, doesn't overwrite
|
||||
- **State-driven**: Persistent database for incremental sync
|
||||
- **Atomic writes**: State updates are crash-safe
|
||||
|
||||
### Code Organization
|
||||
```
|
||||
┌─────────────┐
|
||||
│ main.py │ Entry point, orchestration
|
||||
└──────┬──────┘
|
||||
│
|
||||
┌───┴───┐
|
||||
│ │
|
||||
┌──▼───┐ ┌▼────────────┐
|
||||
│Config│ │State Database│
|
||||
└──────┘ └─────────────┘
|
||||
│
|
||||
┌───┴──────────────┐
|
||||
│ │
|
||||
┌──▼─────┐ ┌────▼────┐
|
||||
│Planner │ │Downloader│
|
||||
└──┬─────┘ └────┬────┘
|
||||
│ │
|
||||
└────────┬────────┘
|
||||
│
|
||||
┌──────▼──────┐
|
||||
│ LbryClient │ JSON-RPC to daemon
|
||||
└─────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Install
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Initialize
|
||||
python init.py
|
||||
|
||||
# Configure (edit ~/Documents/lbry-downloads/config.yaml)
|
||||
|
||||
# Test
|
||||
python main.py --dry-run
|
||||
|
||||
# Run
|
||||
python main.py
|
||||
|
||||
# Test suite
|
||||
make test
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Technical Specifications
|
||||
|
||||
- **Language**: Python 3.8+
|
||||
- **Dependencies**: PyYAML>=6.0, requests>=2.31.0
|
||||
- **Test Framework**: pytest
|
||||
- **License**: MIT
|
||||
- **Lines of Code**: 2,165 (Python)
|
||||
- **Test Coverage**: 38 unit tests
|
||||
- **Documentation**: 8 markdown files
|
||||
|
||||
---
|
||||
|
||||
## Files Created
|
||||
|
||||
### Python Modules (9 files)
|
||||
1. main.py
|
||||
2. models.py
|
||||
3. utils.py
|
||||
4. config_loader.py
|
||||
5. lbry_client.py
|
||||
6. state_db.py
|
||||
7. planner.py
|
||||
8. downloader.py
|
||||
9. init.py
|
||||
|
||||
### Setup Scripts (3 files)
|
||||
1. setup.py - Interactive installation wizard
|
||||
2. uninstall.py - Clean removal tool
|
||||
3. install.sh - Wrapper script
|
||||
|
||||
### Tests (5 files)
|
||||
1. tests/__init__.py
|
||||
2. tests/test_utils.py
|
||||
3. tests/test_config.py
|
||||
4. tests/test_planner.py
|
||||
5. tests/test_state_db.py
|
||||
6. tests/test_download_limit.py
|
||||
|
||||
### Documentation (9 files)
|
||||
1. README.md
|
||||
2. QUICKSTART.md
|
||||
3. SPEC.md
|
||||
4. AGENTS.md
|
||||
5. CHANGELOG.md
|
||||
6. LICENSE
|
||||
7. config.yaml.example
|
||||
8. PROJECT_SUMMARY.md (this file)
|
||||
9. DOCUMENTATION_INDEX.md
|
||||
|
||||
### Build & Config (4 files)
|
||||
1. requirements.txt
|
||||
2. pyproject.toml
|
||||
3. Makefile
|
||||
4. .gitignore
|
||||
|
||||
**Total: 31 files, ~4,500 lines**
|
||||
|
||||
---
|
||||
|
||||
## Status
|
||||
|
||||
✅ **COMPLETE** - Ready for use
|
||||
|
||||
All core functionality implemented and tested.
|
||||
@@ -0,0 +1,99 @@
|
||||
# Quick Start Guide
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
# Clone or download the repository
|
||||
cd LBRY-Downloader
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Initialize configuration
|
||||
python init.py
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Edit `~/Documents/lbry-downloads/config.yaml`:
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
- input: "https://odysee.com/@YourFavoriteChannel:1"
|
||||
enabled: true
|
||||
- input: "lbry://@AnotherChannel#5"
|
||||
enabled: true
|
||||
```
|
||||
|
||||
## Running
|
||||
|
||||
```bash
|
||||
# Test what would be downloaded
|
||||
python main.py --dry-run
|
||||
|
||||
# Actually download
|
||||
python main.py
|
||||
|
||||
# Use custom config
|
||||
python main.py --config ./custom-config.yaml
|
||||
```
|
||||
|
||||
## Understanding Output
|
||||
|
||||
```
|
||||
Run complete.
|
||||
Channels scanned: 2
|
||||
Claims examined: 150
|
||||
New downloads: 3
|
||||
New versions: 1
|
||||
Skipped existing: 146
|
||||
Failures: 0
|
||||
```
|
||||
|
||||
## Directory Structure
|
||||
|
||||
Downloads are saved to:
|
||||
```
|
||||
~/Documents/lbry-downloads/
|
||||
channels/
|
||||
ChannelName__abc123/
|
||||
channel.json
|
||||
claims/
|
||||
video-name__def456/
|
||||
claim.json
|
||||
versions/
|
||||
sd_hash789/
|
||||
video.mp4
|
||||
metadata.json
|
||||
download.json
|
||||
checksums.txt
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**"Could not connect to LBRY daemon"**
|
||||
```bash
|
||||
lbrynet start
|
||||
```
|
||||
|
||||
**"Config file not found"**
|
||||
```bash
|
||||
python init.py
|
||||
```
|
||||
|
||||
**"No channels enabled"**
|
||||
Check that at least one channel has `enabled: true`
|
||||
|
||||
**Want to see more details?**
|
||||
Edit config.yaml and set `log_level: "DEBUG"`
|
||||
|
||||
## Updates
|
||||
|
||||
The tool is designed to be run repeatedly. It will:
|
||||
- Skip already downloaded files
|
||||
- Detect new versions of existing claims
|
||||
- Download only what's new or changed
|
||||
|
||||
## Support
|
||||
|
||||
See README.md for detailed documentation.
|
||||
@@ -0,0 +1,215 @@
|
||||
# LBRY Downloader
|
||||
|
||||
A Python tool that incrementally syncs downloadable file claims from configured LBRY/Odysee channels into a structured local archive.
|
||||
|
||||
## Features
|
||||
|
||||
- **Incremental Sync**: Only downloads new claims and updated versions
|
||||
- **Version Preservation**: Keeps old versions instead of overwriting them
|
||||
- **Robust Identity**: Uses stable LBRY claim IDs, not display names or titles
|
||||
- **Atomic State**: Database writes are atomic to prevent corruption
|
||||
- **Daemon-Based**: Talks directly to local `lbrynet` daemon via JSON-RPC
|
||||
- **Flexible Input**: Accepts Odysee URLs, LBRY URIs, or claim IDs
|
||||
- **Audit Trail**: Maintains run history in JSONL format
|
||||
- **Dry-Run Mode**: Test what would happen without downloading
|
||||
|
||||
## Installation
|
||||
|
||||
### Easy Install (Recommended)
|
||||
|
||||
Run the interactive setup script that handles everything:
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/yourusername/LBRY-Downloader.git
|
||||
cd LBRY-Downloader
|
||||
|
||||
# Run the interactive installer
|
||||
./setup.py
|
||||
```
|
||||
|
||||
The setup script will:
|
||||
- ✅ Check and install system dependencies
|
||||
- ✅ Set up a Python virtual environment
|
||||
- ✅ Install Python packages
|
||||
- ✅ Ask for your download location
|
||||
- ✅ Configure channels to download from
|
||||
- ✅ Set download limits
|
||||
- ✅ Create launcher scripts
|
||||
|
||||
### Manual Install
|
||||
|
||||
See [LINUX_SETUP.md](LINUX_SETUP.md) for detailed manual installation instructions.
|
||||
|
||||
### Uninstall
|
||||
|
||||
To remove LBRY Downloader:
|
||||
|
||||
```bash
|
||||
./uninstall.py
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
You need to have the LBRY SDK daemon (`lbrynet`) installed and running:
|
||||
|
||||
```bash
|
||||
# The daemon should be accessible at http://127.0.0.1:5279
|
||||
# Check if it's running:
|
||||
curl -X POST http://127.0.0.1:5279 \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"jsonrpc":"2.0","method":"status","params":{},"id":1}'
|
||||
```
|
||||
|
||||
To install `lbrynet`, see: https://lbry.com/get
|
||||
|
||||
## Configuration
|
||||
|
||||
Copy the example config and edit it:
|
||||
|
||||
```bash
|
||||
cp config.yaml.example ~/Documents/lbry-downloads/config.yaml
|
||||
# Edit the file to add your channels
|
||||
```
|
||||
|
||||
Example configuration:
|
||||
|
||||
```yaml
|
||||
lbrynet:
|
||||
api_url: "http://127.0.0.1:5279"
|
||||
timeout_seconds: 60
|
||||
|
||||
general:
|
||||
base_dir: "~/Documents/lbry-downloads"
|
||||
max_workers: 2
|
||||
dry_run: false
|
||||
include_reposts: false
|
||||
|
||||
channels:
|
||||
- input: "https://odysee.com/@SomeChannel:1"
|
||||
enabled: true
|
||||
- input: "lbry://@AnotherChannel#5"
|
||||
enabled: true
|
||||
```
|
||||
|
||||
### Per-Channel Download Locations
|
||||
|
||||
You can specify a custom download location for each channel:
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
- input: "https://odysee.com/@TechVideos:1"
|
||||
enabled: true
|
||||
download_path: "/mnt/media/tech" # Downloads go here instead of base_dir
|
||||
|
||||
- input: "https://odysee.com/@Music:1"
|
||||
enabled: true
|
||||
download_path: "~/Music/LBRY"
|
||||
|
||||
- input: "https://odysee.com/@DefaultLocation:1"
|
||||
enabled: true
|
||||
# No download_path - uses general.base_dir/channels/
|
||||
```
|
||||
|
||||
### Download Limit
|
||||
|
||||
Limit how many new downloads per channel (downloads most recent first):
|
||||
|
||||
```yaml
|
||||
general:
|
||||
# Download only 10 most recent items per channel (default)
|
||||
download_limit: 10
|
||||
|
||||
# Or set to 0 to download everything
|
||||
# download_limit: 0
|
||||
```
|
||||
|
||||
This is useful for:
|
||||
- **Testing**: Download just a few files to verify everything works
|
||||
- **Large channels**: Gradually sync channels with thousands of videos
|
||||
- **Bandwidth**: Control how much data is downloaded per run
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Run with default config
|
||||
python main.py
|
||||
|
||||
# Use custom config
|
||||
python main.py --config ./my-config.yaml
|
||||
|
||||
# Dry run - see what would happen
|
||||
python main.py --dry-run
|
||||
```
|
||||
|
||||
## Directory Structure
|
||||
|
||||
Downloads are organized as:
|
||||
|
||||
```
|
||||
~/Documents/lbry-downloads/
|
||||
config.yaml
|
||||
state/
|
||||
database.json
|
||||
run-history.jsonl
|
||||
channels/
|
||||
<channel_name>__<channel_claim_id>/
|
||||
channel.json
|
||||
claims/
|
||||
<claim_name>__<claim_id>/
|
||||
claim.json
|
||||
versions/
|
||||
<version_token>/
|
||||
<downloaded_file>
|
||||
metadata.json
|
||||
download.json
|
||||
checksums.txt
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
1. **Channel Resolution**: Converts URLs/URIs to stable channel claim IDs
|
||||
2. **Claim Enumeration**: Paginates through all claims in each channel
|
||||
3. **Version Detection**: Uses `sd_hash` to detect updated content
|
||||
4. **Smart Skipping**: Only downloads new claims or changed versions
|
||||
5. **Metadata Preservation**: Stores normalized metadata beside each file
|
||||
|
||||
## State Management
|
||||
|
||||
The tool maintains state in `state/database.json`:
|
||||
- Tracks which channels have been scanned
|
||||
- Records claim metadata and version history
|
||||
- Prevents duplicate downloads
|
||||
- Detects missing files for re-download
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Could not connect to LBRY daemon"
|
||||
Make sure `lbrynet` is running:
|
||||
```bash
|
||||
lbrynet start
|
||||
```
|
||||
|
||||
### "Config file not found"
|
||||
Create a config at the default location or specify a path:
|
||||
```bash
|
||||
python main.py --config ./config.yaml
|
||||
```
|
||||
|
||||
### Dry run shows no downloads
|
||||
Check that your channels are enabled and have downloadable content (streams, not just reposts).
|
||||
|
||||
## Architecture
|
||||
|
||||
- `main.py` - Entry point and orchestration
|
||||
- `lbry_client.py` - JSON-RPC daemon communication
|
||||
- `config_loader.py` - Configuration parsing
|
||||
- `state_db.py` - Persistent state management
|
||||
- `planner.py` - Download decision logic
|
||||
- `downloader.py` - Download execution
|
||||
- `models.py` - Data structures
|
||||
- `utils.py` - Helper functions
|
||||
|
||||
## License
|
||||
|
||||
MIT License
|
||||
@@ -0,0 +1,86 @@
|
||||
# LBRY Downloader Configuration
|
||||
# Place this file at ~/Documents/lbry-downloads/config.yaml
|
||||
# or specify a different path with --config
|
||||
|
||||
lbrynet:
|
||||
# URL of the local LBRY daemon (lbrynet)
|
||||
api_url: "http://127.0.0.1:5279"
|
||||
# Timeout for daemon requests in seconds
|
||||
timeout_seconds: 60
|
||||
|
||||
general:
|
||||
# Base directory for all downloads and state
|
||||
base_dir: "~/Documents/lbry-downloads"
|
||||
# Path to the state database
|
||||
state_file: "~/Documents/lbry-downloads/state/database.json"
|
||||
# Number of concurrent downloads (keep low to avoid daemon issues)
|
||||
max_workers: 2
|
||||
# Logging level: DEBUG, INFO, WARNING, ERROR, CRITICAL
|
||||
log_level: "INFO"
|
||||
# If true, show what would be downloaded without actually downloading
|
||||
dry_run: false
|
||||
# Verify that previously downloaded files still exist
|
||||
verify_existing_files: true
|
||||
# Write SHA256 checksums for downloaded files
|
||||
write_checksums: true
|
||||
# Filename handling: 'original' or 'safe' (aggressively sanitized)
|
||||
filename_mode: "original"
|
||||
# Include reposts in downloads (default: false)
|
||||
include_reposts: false
|
||||
# Number of claims to fetch per page from the daemon
|
||||
channel_page_size: 50
|
||||
# Keep records of claims even if they disappear from the channel
|
||||
keep_missing_claim_records: true
|
||||
|
||||
# Download limit: Maximum number of downloads per channel per run
|
||||
#
|
||||
# This limits how many NEW downloads the tool will perform for each channel
|
||||
# in a single run. Useful for:
|
||||
# - Testing (download just a few files to verify everything works)
|
||||
# - Bandwidth management (don't overwhelm your connection)
|
||||
# - Gradual syncing (download a bit at a time for large channels)
|
||||
#
|
||||
# The tool will download the MOST RECENT items first (sorted by publish date).
|
||||
#
|
||||
# Values:
|
||||
# - 0 or "all": No limit, download everything new (use with caution on large channels!)
|
||||
# - 1, 2, 3, etc.: Download only this many most recent items per channel
|
||||
#
|
||||
# Examples:
|
||||
# download_limit: 10 # Download only 10 most recent items (default)
|
||||
# download_limit: 0 # Download all new items
|
||||
# download_limit: 5 # Download only 5 most recent items (good for testing)
|
||||
download_limit: 10
|
||||
|
||||
channels:
|
||||
# Add your channels here. The tool will download all downloadable content
|
||||
# from each enabled channel.
|
||||
#
|
||||
# Supported input formats:
|
||||
# - Odysee URL: https://odysee.com/@ChannelName:1
|
||||
# - LBRY URI: lbry://@ChannelName#1
|
||||
# - Short form: @ChannelName:1
|
||||
# - Claim ID: f3b9b2b7c1c3d8e9a1234567890abcdef123456
|
||||
#
|
||||
# Per-channel options:
|
||||
# - input: (required) Channel URL/URI/ID
|
||||
# - enabled: (optional) true/false - default: true
|
||||
# - download_path: (optional) Custom download location for this channel
|
||||
# If not set, uses general.base_dir/channels/
|
||||
# - tags_include: (optional) List of tags to include (empty = all)
|
||||
# - tags_exclude: (optional) List of tags to exclude
|
||||
#
|
||||
# Example 1: Basic channel
|
||||
# - input: "https://odysee.com/@SomeChannel:1"
|
||||
# enabled: true
|
||||
#
|
||||
# Example 2: Channel with custom download location
|
||||
# - input: "https://odysee.com/@TechChannel:1"
|
||||
# enabled: true
|
||||
# download_path: "/mnt/media/videos/tech"
|
||||
#
|
||||
# Example 3: Channel with tag filtering
|
||||
# - input: "lbry://@AnotherChannel#5"
|
||||
# enabled: true
|
||||
# tags_include: ["documentary", "science"]
|
||||
# tags_exclude: ["shorts"]
|
||||
@@ -0,0 +1,236 @@
|
||||
"""Configuration loading and validation."""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError:
|
||||
print("Error: PyYAML is required. Install with: pip install PyYAML")
|
||||
sys.exit(1)
|
||||
|
||||
from models import Config, ChannelConfig, GeneralConfig, LbrynetConfig
|
||||
from utils import expand_path
|
||||
|
||||
|
||||
class ConfigError(Exception):
|
||||
"""Raised when configuration is invalid."""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
def load_config(config_path: Optional[str] = None) -> Config:
|
||||
"""
|
||||
Load and validate the configuration file.
|
||||
|
||||
Args:
|
||||
config_path: Path to config file. If None, uses default location.
|
||||
|
||||
Returns:
|
||||
Validated Config object.
|
||||
|
||||
Raises:
|
||||
ConfigError: If config is missing, invalid, or has errors.
|
||||
"""
|
||||
if config_path is None:
|
||||
config_path = "~/Documents/lbry-downloads/config.yaml"
|
||||
|
||||
config_file = expand_path(config_path)
|
||||
|
||||
if not config_file.exists():
|
||||
raise ConfigError(
|
||||
f"Config file not found: {config_file}\n"
|
||||
f"Create one at this location or specify a different path with --config"
|
||||
)
|
||||
|
||||
try:
|
||||
with open(config_file, "r") as f:
|
||||
raw_config = yaml.safe_load(f)
|
||||
except yaml.YAMLError as e:
|
||||
raise ConfigError(f"Failed to parse YAML config: {e}")
|
||||
except Exception as e:
|
||||
raise ConfigError(f"Failed to read config file: {e}")
|
||||
|
||||
if raw_config is None:
|
||||
raw_config = {}
|
||||
|
||||
return _parse_config(raw_config)
|
||||
|
||||
|
||||
def _parse_config(raw: Dict[str, Any]) -> Config:
|
||||
"""Parse raw config dict into Config object."""
|
||||
config = Config()
|
||||
|
||||
# Parse lbrynet section
|
||||
if "lbrynet" in raw:
|
||||
lbrynet_raw = raw["lbrynet"]
|
||||
config.lbrynet = LbrynetConfig(
|
||||
api_url=lbrynet_raw.get("api_url", config.lbrynet.api_url),
|
||||
timeout_seconds=lbrynet_raw.get(
|
||||
"timeout_seconds", config.lbrynet.timeout_seconds
|
||||
),
|
||||
)
|
||||
|
||||
# Parse general section
|
||||
if "general" in raw:
|
||||
general_raw = raw["general"]
|
||||
config.general = GeneralConfig(
|
||||
base_dir=general_raw.get("base_dir", config.general.base_dir),
|
||||
state_file=general_raw.get("state_file", config.general.state_file),
|
||||
max_workers=general_raw.get("max_workers", config.general.max_workers),
|
||||
log_level=general_raw.get("log_level", config.general.log_level),
|
||||
dry_run=general_raw.get("dry_run", config.general.dry_run),
|
||||
verify_existing_files=general_raw.get(
|
||||
"verify_existing_files", config.general.verify_existing_files
|
||||
),
|
||||
write_checksums=general_raw.get(
|
||||
"write_checksums", config.general.write_checksums
|
||||
),
|
||||
filename_mode=general_raw.get(
|
||||
"filename_mode", config.general.filename_mode
|
||||
),
|
||||
include_reposts=general_raw.get(
|
||||
"include_reposts", config.general.include_reposts
|
||||
),
|
||||
channel_page_size=general_raw.get(
|
||||
"channel_page_size", config.general.channel_page_size
|
||||
),
|
||||
keep_missing_claim_records=general_raw.get(
|
||||
"keep_missing_claim_records", config.general.keep_missing_claim_records
|
||||
),
|
||||
download_limit=general_raw.get(
|
||||
"download_limit", config.general.download_limit
|
||||
),
|
||||
)
|
||||
|
||||
# Validate and expand paths
|
||||
config.general.base_dir = str(expand_path(config.general.base_dir))
|
||||
config.general.state_file = str(expand_path(config.general.state_file))
|
||||
|
||||
# Validate max_workers
|
||||
if config.general.max_workers < 1:
|
||||
raise ConfigError("max_workers must be at least 1")
|
||||
if config.general.max_workers > 10:
|
||||
# Warn but don't fail - user might know what they're doing
|
||||
print(
|
||||
f"Warning: max_workers is set to {config.general.max_workers}, which is high"
|
||||
)
|
||||
|
||||
# Validate download_limit
|
||||
# 0 means "all" (no limit), negative values are not allowed
|
||||
if config.general.download_limit < 0:
|
||||
raise ConfigError("download_limit must be 0 (for all) or a positive number")
|
||||
|
||||
# Validate log_level
|
||||
valid_levels = ["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]
|
||||
if config.general.log_level.upper() not in valid_levels:
|
||||
raise ConfigError(f"log_level must be one of: {', '.join(valid_levels)}")
|
||||
config.general.log_level = config.general.log_level.upper()
|
||||
|
||||
# Validate filename_mode
|
||||
valid_modes = ["original", "safe"]
|
||||
if config.general.filename_mode not in valid_modes:
|
||||
raise ConfigError(f"filename_mode must be one of: {', '.join(valid_modes)}")
|
||||
|
||||
# Parse channels section
|
||||
if "channels" not in raw or not raw["channels"]:
|
||||
raise ConfigError("Config must contain at least one channel in 'channels' list")
|
||||
|
||||
channels_raw = raw["channels"]
|
||||
if not isinstance(channels_raw, list):
|
||||
raise ConfigError("'channels' must be a list")
|
||||
|
||||
config.channels = _parse_channels(channels_raw)
|
||||
|
||||
# Check that at least one channel is enabled
|
||||
enabled_channels = [c for c in config.channels if c.enabled]
|
||||
if not enabled_channels:
|
||||
raise ConfigError("At least one channel must be enabled")
|
||||
|
||||
return config
|
||||
|
||||
|
||||
def _parse_channels(channels_raw: List[Dict[str, Any]]) -> List[ChannelConfig]:
|
||||
"""Parse channel configurations."""
|
||||
channels = []
|
||||
|
||||
for i, channel_raw in enumerate(channels_raw):
|
||||
if not isinstance(channel_raw, dict):
|
||||
raise ConfigError(f"Channel {i + 1} must be a dictionary")
|
||||
|
||||
if "input" not in channel_raw:
|
||||
raise ConfigError(f"Channel {i + 1} is missing required 'input' field")
|
||||
|
||||
input_val = channel_raw["input"]
|
||||
if not input_val or not isinstance(input_val, str):
|
||||
raise ConfigError(f"Channel {i + 1} has invalid 'input' value")
|
||||
|
||||
# Parse download_path if provided
|
||||
download_path = channel_raw.get("download_path")
|
||||
if download_path:
|
||||
download_path = str(expand_path(download_path))
|
||||
|
||||
channel = ChannelConfig(
|
||||
input=input_val,
|
||||
enabled=channel_raw.get("enabled", True),
|
||||
download_path=download_path,
|
||||
tags_include=channel_raw.get("tags_include", []),
|
||||
tags_exclude=channel_raw.get("tags_exclude", []),
|
||||
)
|
||||
|
||||
channels.append(channel)
|
||||
|
||||
return channels
|
||||
|
||||
|
||||
def create_default_config() -> str:
|
||||
"""Generate a default configuration file content."""
|
||||
return """# LBRY Downloader Configuration
|
||||
# Place this file at ~/Documents/lbry-downloads/config.yaml
|
||||
|
||||
lbrynet:
|
||||
api_url: "http://127.0.0.1:5279"
|
||||
timeout_seconds: 60
|
||||
|
||||
general:
|
||||
base_dir: "~/Documents/lbry-downloads"
|
||||
state_file: "~/Documents/lbry-downloads/state/database.json"
|
||||
max_workers: 2
|
||||
log_level: "INFO"
|
||||
dry_run: false
|
||||
verify_existing_files: true
|
||||
write_checksums: true
|
||||
filename_mode: "original"
|
||||
include_reposts: false
|
||||
channel_page_size: 50
|
||||
keep_missing_claim_records: true
|
||||
|
||||
channels:
|
||||
# Add your channels here. Examples:
|
||||
# - input: "https://odysee.com/@SomeChannel:1"
|
||||
# enabled: true
|
||||
# tags_include: []
|
||||
# tags_exclude: []
|
||||
# - input: "lbry://@AnotherChannel#5"
|
||||
# enabled: true
|
||||
"""
|
||||
|
||||
|
||||
def ensure_directories(config: Config) -> None:
|
||||
"""
|
||||
Create necessary base directories if they don't exist.
|
||||
|
||||
Args:
|
||||
config: Validated configuration object.
|
||||
"""
|
||||
base_path = Path(config.general.base_dir)
|
||||
base_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Create state directory
|
||||
state_path = base_path / "state"
|
||||
state_path.mkdir(exist_ok=True)
|
||||
|
||||
# Create channels directory
|
||||
channels_path = base_path / "channels"
|
||||
channels_path.mkdir(exist_ok=True)
|
||||
@@ -0,0 +1,142 @@
|
||||
# Per-Channel Download Paths Example
|
||||
|
||||
This example shows how to configure different download locations for each channel.
|
||||
|
||||
## Use Case: Organize by Content Type
|
||||
|
||||
```yaml
|
||||
lbrynet:
|
||||
api_url: "http://127.0.0.1:5279"
|
||||
|
||||
general:
|
||||
base_dir: "~/Documents/lbry-downloads"
|
||||
max_workers: 2
|
||||
|
||||
channels:
|
||||
# Tech videos go to external drive
|
||||
- input: "https://odysee.com/@TechChannel:1"
|
||||
enabled: true
|
||||
download_path: "/mnt/media/videos/tech"
|
||||
|
||||
# Music goes to Music folder
|
||||
- input: "https://odysee.com/@MusicChannel:1"
|
||||
enabled: true
|
||||
download_path: "~/Music/LBRY"
|
||||
|
||||
# Podcasts go to Podcasts folder
|
||||
- input: "https://odysee.com/@PodcastChannel:1"
|
||||
enabled: true
|
||||
download_path: "~/Podcasts"
|
||||
|
||||
# Everything else goes to default location
|
||||
- input: "https://odysee.com/@MiscChannel:1"
|
||||
enabled: true
|
||||
# No download_path - uses general.base_dir/channels/
|
||||
```
|
||||
|
||||
## Use Case: Manage Disk Space
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
# Large archive channel goes to big external drive
|
||||
- input: "https://odysee.com/@Archive:1"
|
||||
enabled: true
|
||||
download_path: "/mnt/8tb-drive/archive"
|
||||
|
||||
# Frequently accessed channel goes to fast SSD
|
||||
- input: "https://odysee.com/@DailyUpdates:1"
|
||||
enabled: true
|
||||
download_path: "~/Videos/daily"
|
||||
```
|
||||
|
||||
## Use Case: Separate Personal and Work
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
# Work-related content
|
||||
- input: "https://odysee.com/@TechTalks:1"
|
||||
enabled: true
|
||||
download_path: "~/Work/Videos"
|
||||
|
||||
# Personal entertainment
|
||||
- input: "https://odysee.com/@Entertainment:1"
|
||||
enabled: true
|
||||
download_path: "~/Videos/entertainment"
|
||||
```
|
||||
|
||||
## Setting Up Custom Paths
|
||||
|
||||
### 1. Create the directories
|
||||
|
||||
```bash
|
||||
# External drive example
|
||||
sudo mkdir -p /mnt/media/videos/tech
|
||||
sudo chown $USER:$USER /mnt/media/videos/tech
|
||||
|
||||
# Home directory example
|
||||
mkdir -p ~/Music/LBRY
|
||||
mkdir -p ~/Podcasts
|
||||
```
|
||||
|
||||
### 2. Edit config.yaml
|
||||
|
||||
```bash
|
||||
nano ~/Documents/lbry-downloads/config.yaml
|
||||
```
|
||||
|
||||
### 3. Test with dry-run
|
||||
|
||||
```bash
|
||||
python main.py --dry-run
|
||||
```
|
||||
|
||||
Check the output to see where files would be downloaded.
|
||||
|
||||
### 4. Run the download
|
||||
|
||||
```bash
|
||||
python main.py
|
||||
```
|
||||
|
||||
## Resulting Directory Structure
|
||||
|
||||
With the example above, your files will be organized as:
|
||||
|
||||
```
|
||||
/mnt/media/videos/tech/
|
||||
TechChannel__abc123/
|
||||
channel.json
|
||||
claims/
|
||||
video1__def456/
|
||||
versions/
|
||||
sd_xyz789/
|
||||
video.mp4
|
||||
|
||||
~/Music/LBRY/
|
||||
MusicChannel__ghi789/
|
||||
channel.json
|
||||
claims/
|
||||
song1__jkl012/
|
||||
versions/
|
||||
sd_mno345/
|
||||
song.mp3
|
||||
|
||||
~/Podcasts/
|
||||
PodcastChannel__pqr678/
|
||||
...
|
||||
|
||||
~/Documents/lbry-downloads/channels/
|
||||
MiscChannel__stu901/
|
||||
...
|
||||
```
|
||||
|
||||
## Tips
|
||||
|
||||
1. **Path expansion**: Use `~` for home directory - it will be expanded automatically
|
||||
2. **Absolute paths**: Always use absolute paths for external drives
|
||||
3. **Permissions**: Ensure you have write permissions to custom paths
|
||||
4. **Existing downloads**: If you move a channel's download location after files have been downloaded, the tool will redownload them to the new location
|
||||
|
||||
## State Management
|
||||
|
||||
The state database still lives in `~/Documents/lbry-downloads/state/` regardless of where you download files. This keeps all metadata and tracking in one place.
|
||||
+209
@@ -0,0 +1,209 @@
|
||||
"""Downloader module - executes downloads and writes metadata."""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import Dict, Optional
|
||||
|
||||
from lbry_client import LbryClient
|
||||
from models import Config, DownloadAction
|
||||
from utils import compute_sha256
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DownloadError(Exception):
|
||||
"""Raised when a download fails."""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class Downloader:
|
||||
"""Handles downloading claims and writing metadata."""
|
||||
|
||||
def __init__(self, client: LbryClient, config: Config, base_dir: Path):
|
||||
self.client = client
|
||||
self.config = config
|
||||
self.base_dir = base_dir
|
||||
|
||||
def download(self, action: DownloadAction, dry_run: bool = False) -> bool:
|
||||
"""
|
||||
Execute a download action.
|
||||
|
||||
Args:
|
||||
action: The download action to execute.
|
||||
dry_run: If True, don't actually download.
|
||||
|
||||
Returns:
|
||||
True if download succeeded or was skipped in dry-run.
|
||||
|
||||
Raises:
|
||||
DownloadError: If the download fails.
|
||||
"""
|
||||
if action.action == "skip_existing":
|
||||
logger.debug(f"Skipping already downloaded: {action.claim_name}")
|
||||
return True
|
||||
|
||||
if dry_run:
|
||||
logger.info(f"[DRY-RUN] Would download: {action.claim_name}")
|
||||
return True
|
||||
|
||||
# Create directories
|
||||
version_dir = Path(action.target_dir)
|
||||
version_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
logger.info(f"Downloading: {action.claim_name}")
|
||||
|
||||
try:
|
||||
# Call daemon to download
|
||||
result = self.client.get(
|
||||
uri=action.uri,
|
||||
download_directory=str(version_dir),
|
||||
)
|
||||
|
||||
# Find the downloaded file
|
||||
file_info = self._extract_file_info(result)
|
||||
if not file_info:
|
||||
raise DownloadError("No file info in download result")
|
||||
|
||||
downloaded_path = Path(file_info["download_path"])
|
||||
|
||||
# If file was downloaded to a different location, move it
|
||||
if downloaded_path.parent != version_dir:
|
||||
target_path = version_dir / downloaded_path.name
|
||||
shutil.move(str(downloaded_path), str(target_path))
|
||||
downloaded_path = target_path
|
||||
|
||||
# Write metadata files
|
||||
self._write_metadata(version_dir, action, file_info)
|
||||
self._write_download_json(version_dir, result)
|
||||
|
||||
# Write checksum if enabled
|
||||
if self.config.general.write_checksums:
|
||||
self._write_checksum(version_dir, downloaded_path)
|
||||
|
||||
# Update action metadata with file path
|
||||
rel_path = downloaded_path.relative_to(self.base_dir)
|
||||
action.metadata["local_file_path"] = str(rel_path)
|
||||
|
||||
logger.info(f"Downloaded: {action.claim_name} -> {downloaded_path.name}")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Download failed for {action.claim_name}: {e}")
|
||||
raise DownloadError(f"Download failed: {e}")
|
||||
|
||||
def _extract_file_info(self, result: Dict) -> Optional[Dict]:
|
||||
"""
|
||||
Extract file information from download result.
|
||||
|
||||
The result structure varies depending on daemon version and
|
||||
whether the file was already downloaded.
|
||||
"""
|
||||
# Direct file result
|
||||
if "download_path" in result:
|
||||
return result
|
||||
|
||||
# Nested in outputs
|
||||
if "outputs" in result and result["outputs"]:
|
||||
output = result["outputs"][0]
|
||||
if "value" in output:
|
||||
value = output["value"]
|
||||
if isinstance(value, dict) and "source" in value:
|
||||
source = value["source"]
|
||||
return {
|
||||
"download_path": source.get("name"),
|
||||
"file_name": source.get("name"),
|
||||
"media_type": source.get("media_type"),
|
||||
}
|
||||
|
||||
# Try to get from file_list
|
||||
if "claim_id" in result:
|
||||
file_list = self.client.file_list(claim_id=result["claim_id"])
|
||||
if "items" in file_list and file_list["items"]:
|
||||
return file_list["items"][0]
|
||||
|
||||
return None
|
||||
|
||||
def _write_metadata(
|
||||
self,
|
||||
version_dir: Path,
|
||||
action: DownloadAction,
|
||||
file_info: Dict,
|
||||
) -> None:
|
||||
"""
|
||||
Write normalized metadata.json for the download.
|
||||
|
||||
Args:
|
||||
version_dir: Directory to write metadata to.
|
||||
action: The download action.
|
||||
file_info: File information from daemon.
|
||||
"""
|
||||
metadata = {
|
||||
"claim_id": action.claim_id,
|
||||
"channel_claim_id": action.channel_claim_id,
|
||||
"name": action.claim_name,
|
||||
"title": action.metadata.get("title"),
|
||||
"version_token": action.version_token,
|
||||
"permanent_url": action.uri,
|
||||
"downloaded_at": file_info.get("added_on"),
|
||||
"file_name": file_info.get("file_name")
|
||||
or file_info.get("download_path", "").split("/")[-1],
|
||||
"media_type": file_info.get("media_type"),
|
||||
"file_size": file_info.get("written_bytes"),
|
||||
"sd_hash": action.metadata.get("sd_hash"),
|
||||
"stream_hash": action.metadata.get("stream_hash"),
|
||||
"txid": action.metadata.get("txid"),
|
||||
"nout": action.metadata.get("nout"),
|
||||
"tags": action.metadata.get("tags", []),
|
||||
"languages": action.metadata.get("languages", []),
|
||||
}
|
||||
|
||||
metadata_path = version_dir / "metadata.json"
|
||||
with open(metadata_path, "w") as f:
|
||||
json.dump(metadata, f, indent=2, default=str)
|
||||
|
||||
logger.debug(f"Wrote metadata: {metadata_path}")
|
||||
|
||||
def _write_download_json(
|
||||
self,
|
||||
version_dir: Path,
|
||||
result: Dict,
|
||||
) -> None:
|
||||
"""
|
||||
Write raw daemon response to download.json.
|
||||
|
||||
Args:
|
||||
version_dir: Directory to write to.
|
||||
result: Raw daemon response.
|
||||
"""
|
||||
download_path = version_dir / "download.json"
|
||||
with open(download_path, "w") as f:
|
||||
json.dump(result, f, indent=2, default=str)
|
||||
|
||||
logger.debug(f"Wrote download metadata: {download_path}")
|
||||
|
||||
def _write_checksum(self, version_dir: Path, file_path: Path) -> None:
|
||||
"""
|
||||
Write SHA256 checksum for the downloaded file.
|
||||
|
||||
Args:
|
||||
version_dir: Directory to write checksum to.
|
||||
file_path: Path to the downloaded file.
|
||||
"""
|
||||
if not file_path.exists():
|
||||
logger.warning(f"Cannot write checksum - file not found: {file_path}")
|
||||
return
|
||||
|
||||
try:
|
||||
checksum = compute_sha256(file_path)
|
||||
checksums_path = version_dir / "checksums.txt"
|
||||
|
||||
with open(checksums_path, "w") as f:
|
||||
f.write(f"SHA256({file_path.name})= {checksum}\n")
|
||||
|
||||
logger.debug(f"Wrote checksum: {checksums_path}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to compute checksum: {e}")
|
||||
@@ -0,0 +1,64 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Initialize LBRY Downloader configuration.
|
||||
|
||||
This script creates the initial configuration file and directory structure.
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from config_loader import create_default_config, ensure_directories
|
||||
from models import Config, GeneralConfig, LbrynetConfig
|
||||
from utils import expand_path
|
||||
|
||||
|
||||
def main():
|
||||
print("LBRY Downloader - Initialization")
|
||||
print("=" * 40)
|
||||
|
||||
# Default paths
|
||||
base_dir = expand_path("~/Documents/lbry-downloads")
|
||||
config_file = base_dir / "config.yaml"
|
||||
|
||||
# Check if already exists
|
||||
if config_file.exists():
|
||||
print(f"\nConfiguration already exists at: {config_file}")
|
||||
response = input("Overwrite? (y/N): ").strip().lower()
|
||||
if response != "y":
|
||||
print("Aborted.")
|
||||
sys.exit(0)
|
||||
|
||||
# Create default config
|
||||
config_content = create_default_config()
|
||||
|
||||
# Create directories
|
||||
base_dir.mkdir(parents=True, exist_ok=True)
|
||||
(base_dir / "state").mkdir(exist_ok=True)
|
||||
(base_dir / "channels").mkdir(exist_ok=True)
|
||||
|
||||
# Write config
|
||||
with open(config_file, "w") as f:
|
||||
f.write(config_content)
|
||||
|
||||
print(f"\n✓ Created configuration: {config_file}")
|
||||
print(f"✓ Created directories:")
|
||||
print(f" - {base_dir}")
|
||||
print(f" - {base_dir / 'state'}")
|
||||
print(f" - {base_dir / 'channels'}")
|
||||
|
||||
print("\n" + "=" * 40)
|
||||
print("Next steps:")
|
||||
print("1. Edit the configuration file to add your channels")
|
||||
print(f" nano {config_file}")
|
||||
print("\n2. Ensure lbrynet daemon is running:")
|
||||
print(" lbrynet start")
|
||||
print("\n3. Run the downloader:")
|
||||
print(" python main.py")
|
||||
print("\n4. Or test first with dry-run:")
|
||||
print(" python main.py --dry-run")
|
||||
print("=" * 40)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+252
@@ -0,0 +1,252 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# LBRY Downloader - Quick Installation Script
|
||||
#
|
||||
# This script runs the interactive setup.py which handles:
|
||||
# - Dependency checking and installation
|
||||
# - Virtual environment setup
|
||||
# - User configuration (download location, channels, etc.)
|
||||
# - Launcher script creation
|
||||
#
|
||||
|
||||
set -e
|
||||
|
||||
echo "================================================"
|
||||
echo " LBRY Downloader - Installation"
|
||||
echo "================================================"
|
||||
echo ""
|
||||
|
||||
# Check for Python
|
||||
if ! command -v python3 &> /dev/null; then
|
||||
echo "Error: Python 3 is required but not installed"
|
||||
echo "Please install Python 3.8 or higher"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Get script directory
|
||||
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
|
||||
cd "$SCRIPT_DIR"
|
||||
|
||||
# Check if setup.py exists
|
||||
if [ ! -f "setup.py" ]; then
|
||||
echo "Error: setup.py not found"
|
||||
echo "Please ensure you're running this from the LBRY-Downloader directory"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Run the interactive setup
|
||||
echo "Starting interactive setup..."
|
||||
echo "This will guide you through installation and configuration"
|
||||
echo ""
|
||||
python3 setup.py
|
||||
|
||||
echo ""
|
||||
echo "================================================"
|
||||
echo "Installation script completed!"
|
||||
echo "================================================"
|
||||
echo " LBRY Downloader - Linux Installation"
|
||||
echo "================================================"
|
||||
echo ""
|
||||
|
||||
# Colors for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Function to print colored output
|
||||
print_status() {
|
||||
echo -e "${GREEN}✓${NC} $1"
|
||||
}
|
||||
|
||||
print_warning() {
|
||||
echo -e "${YELLOW}⚠${NC} $1"
|
||||
}
|
||||
|
||||
print_error() {
|
||||
echo -e "${RED}✗${NC} $1"
|
||||
}
|
||||
|
||||
# Check if running on Debian-based system
|
||||
if ! command -v apt-get &> /dev/null; then
|
||||
print_error "This script is designed for Debian-based systems (Debian, Ubuntu, Mint)"
|
||||
echo "Please install manually following the instructions in LINUX_SETUP.md"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Get script directory
|
||||
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
|
||||
cd "$SCRIPT_DIR"
|
||||
|
||||
# Check Python version
|
||||
echo "Checking Python installation..."
|
||||
if command -v python3 &> /dev/null; then
|
||||
PYTHON_VERSION=$(python3 --version | cut -d' ' -f2 | cut -d'.' -f1,2)
|
||||
REQUIRED_VERSION="3.8"
|
||||
|
||||
if [ "$(printf '%s\n' "$REQUIRED_VERSION" "$PYTHON_VERSION" | sort -V | head -n1)" = "$REQUIRED_VERSION" ]; then
|
||||
print_status "Python $PYTHON_VERSION found (3.8+ required)"
|
||||
else
|
||||
print_error "Python 3.8 or higher required, found $PYTHON_VERSION"
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
print_error "Python 3 not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Install system packages
|
||||
echo ""
|
||||
echo "Installing system packages..."
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y python3-pip python3-venv git curl wget unzip
|
||||
print_status "System packages installed"
|
||||
|
||||
# Install Python dependencies
|
||||
echo ""
|
||||
echo "Installing Python dependencies..."
|
||||
if [ -d "venv" ]; then
|
||||
print_warning "Virtual environment already exists"
|
||||
else
|
||||
python3 -m venv venv
|
||||
print_status "Created virtual environment"
|
||||
fi
|
||||
|
||||
source venv/bin/activate
|
||||
pip install --upgrade pip
|
||||
pip install -r requirements.txt
|
||||
print_status "Python dependencies installed"
|
||||
|
||||
# Check for lbrynet
|
||||
echo ""
|
||||
echo "Checking for LBRY SDK (lbrynet)..."
|
||||
if command -v lbrynet &> /dev/null; then
|
||||
print_status "lbrynet found at: $(which lbrynet)"
|
||||
lbrynet --version
|
||||
else
|
||||
print_warning "lbrynet not found in PATH"
|
||||
echo ""
|
||||
echo "The LBRY SDK (lbrynet) is required to download content."
|
||||
echo ""
|
||||
read -p "Would you like to download and install lbrynet now? (y/N): " -n 1 -r
|
||||
echo
|
||||
|
||||
if [[ $REPLY =~ ^[Yy]$ ]]; then
|
||||
echo "Downloading lbrynet..."
|
||||
mkdir -p ~/.local/bin
|
||||
|
||||
# Get latest release
|
||||
LBRY_VERSION="0.113.0"
|
||||
DOWNLOAD_URL="https://github.com/lbryio/lbry-sdk/releases/download/v${LBRY_VERSION}/lbry-sdk-linux.zip"
|
||||
|
||||
cd ~/.local/bin
|
||||
if wget -q "$DOWNLOAD_URL"; then
|
||||
unzip -q lbry-sdk-linux.zip
|
||||
rm lbry-sdk-linux.zip
|
||||
chmod +x lbrynet
|
||||
print_status "lbrynet installed to ~/.local/bin"
|
||||
|
||||
# Check if ~/.local/bin is in PATH
|
||||
if [[ ":$PATH:" != *":$HOME/.local/bin:"* ]]; then
|
||||
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
|
||||
print_status "Added ~/.local/bin to PATH in ~/.bashrc"
|
||||
print_warning "Please run: source ~/.bashrc"
|
||||
fi
|
||||
|
||||
cd "$SCRIPT_DIR"
|
||||
else
|
||||
print_error "Failed to download lbrynet"
|
||||
echo "Please install manually from: https://github.com/lbryio/lbry-sdk/releases"
|
||||
fi
|
||||
else
|
||||
print_warning "Skipping lbrynet installation"
|
||||
echo "You'll need to install it manually before using the downloader"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Initialize configuration
|
||||
echo ""
|
||||
echo "Initializing configuration..."
|
||||
if [ ! -f "$HOME/Documents/lbry-downloads/config.yaml" ]; then
|
||||
source venv/bin/activate
|
||||
python init.py
|
||||
else
|
||||
print_warning "Configuration already exists at ~/Documents/lbry-downloads/config.yaml"
|
||||
fi
|
||||
|
||||
# Create convenient wrapper script
|
||||
echo ""
|
||||
echo "Creating wrapper script..."
|
||||
mkdir -p ~/bin
|
||||
cat > ~/bin/lbry-sync << 'EOF'
|
||||
#!/bin/bash
|
||||
# LBRY Downloader wrapper script
|
||||
|
||||
INSTALL_DIR="$( cd "$( dirname "$(readlink -f "$0")" )" && cd .. && pwd )"
|
||||
DOWNLOADER_DIR="$INSTALL_DIR/Documents/LBRY-Downloader"
|
||||
|
||||
# Check if we're in a different location
|
||||
if [ ! -f "$DOWNLOADER_DIR/main.py" ]; then
|
||||
# Try to find the installation
|
||||
if [ -f "$HOME/Documents/LBRY-Downloader/main.py" ]; then
|
||||
DOWNLOADER_DIR="$HOME/Documents/LBRY-Downloader"
|
||||
elif [ -f "$PWD/main.py" ]; then
|
||||
DOWNLOADER_DIR="$PWD"
|
||||
else
|
||||
echo "Error: Cannot find LBRY-Downloader installation"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
cd "$DOWNLOADER_DIR"
|
||||
source venv/bin/activate 2>/dev/null || true
|
||||
|
||||
# Check if lbrynet is running
|
||||
if ! curl -s http://127.0.0.1:5279 > /dev/null 2>&1; then
|
||||
echo "Starting lbrynet daemon..."
|
||||
lbrynet start &
|
||||
sleep 5
|
||||
fi
|
||||
|
||||
python main.py "$@"
|
||||
EOF
|
||||
|
||||
chmod +x ~/bin/lbry-sync
|
||||
|
||||
if [[ ":$PATH:" == *":$HOME/bin:"* ]]; then
|
||||
print_status "Created wrapper script: ~/bin/lbry-sync"
|
||||
else
|
||||
print_warning "~/bin is not in your PATH"
|
||||
echo "Add this to your ~/.bashrc:"
|
||||
echo 'export PATH="$HOME/bin:$PATH"'
|
||||
fi
|
||||
|
||||
# Final instructions
|
||||
echo ""
|
||||
echo "================================================"
|
||||
echo -e "${GREEN}Installation Complete!${NC}"
|
||||
echo "================================================"
|
||||
echo ""
|
||||
echo "Next steps:"
|
||||
echo ""
|
||||
echo "1. Edit configuration to add your channels:"
|
||||
echo " nano ~/Documents/lbry-downloads/config.yaml"
|
||||
echo ""
|
||||
echo "2. Start the LBRY daemon:"
|
||||
echo " lbrynet start"
|
||||
echo ""
|
||||
echo "3. Test with dry-run:"
|
||||
echo " python main.py --dry-run"
|
||||
echo " # OR:"
|
||||
echo " ~/bin/lbry-sync --dry-run"
|
||||
echo ""
|
||||
echo "4. Run the downloader:"
|
||||
echo " python main.py"
|
||||
echo " # OR:"
|
||||
echo " ~/bin/lbry-sync"
|
||||
echo ""
|
||||
echo "For detailed instructions, see:"
|
||||
echo " - LINUX_SETUP.md"
|
||||
echo " - QUICKSTART.md"
|
||||
echo ""
|
||||
echo "================================================"
|
||||
+263
@@ -0,0 +1,263 @@
|
||||
"""LBRY daemon JSON-RPC client."""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
try:
|
||||
import requests
|
||||
except ImportError:
|
||||
raise ImportError("requests is required. Install with: pip install requests")
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class LbryClientError(Exception):
|
||||
"""Raised when LBRY daemon communication fails."""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class LbryClient:
|
||||
"""JSON-RPC client for lbrynet daemon."""
|
||||
|
||||
def __init__(self, api_url: str = "http://127.0.0.1:5279", timeout: int = 60):
|
||||
self.api_url = api_url
|
||||
self.timeout = timeout
|
||||
self.session = requests.Session()
|
||||
|
||||
def _call(self, method: str, **params) -> Dict[str, Any]:
|
||||
"""
|
||||
Make a JSON-RPC call to the daemon.
|
||||
|
||||
Args:
|
||||
method: The daemon method to call.
|
||||
**params: Parameters for the method.
|
||||
|
||||
Returns:
|
||||
The result field from the JSON-RPC response.
|
||||
|
||||
Raises:
|
||||
LbryClientError: If the call fails or returns an error.
|
||||
"""
|
||||
payload = {"jsonrpc": "2.0", "method": method, "params": params, "id": 1}
|
||||
|
||||
try:
|
||||
response = self.session.post(
|
||||
self.api_url, json=payload, timeout=self.timeout
|
||||
)
|
||||
response.raise_for_status()
|
||||
except requests.exceptions.ConnectionError as e:
|
||||
raise LbryClientError(
|
||||
f"Could not connect to LBRY daemon at {self.api_url}. "
|
||||
f"Make sure lbrynet is installed and running. Error: {e}"
|
||||
)
|
||||
except requests.exceptions.Timeout:
|
||||
raise LbryClientError(
|
||||
f"Request to LBRY daemon timed out after {self.timeout} seconds"
|
||||
)
|
||||
except requests.exceptions.RequestException as e:
|
||||
raise LbryClientError(f"Request to LBRY daemon failed: {e}")
|
||||
|
||||
try:
|
||||
data = response.json()
|
||||
except json.JSONDecodeError as e:
|
||||
raise LbryClientError(f"Invalid JSON response from daemon: {e}")
|
||||
|
||||
if "error" in data and data["error"]:
|
||||
error = data["error"]
|
||||
raise LbryClientError(
|
||||
f"Daemon returned error: {error.get('message', error)}"
|
||||
)
|
||||
|
||||
return data.get("result", {})
|
||||
|
||||
def status(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Check daemon status.
|
||||
|
||||
Returns:
|
||||
Status information from the daemon.
|
||||
|
||||
Raises:
|
||||
LbryClientError: If the daemon is unreachable.
|
||||
"""
|
||||
return self._call("status")
|
||||
|
||||
def resolve(self, urls: List[str]) -> Dict[str, Any]:
|
||||
"""
|
||||
Resolve URLs to claim metadata.
|
||||
|
||||
Args:
|
||||
urls: List of URLs or URIs to resolve.
|
||||
|
||||
Returns:
|
||||
Dictionary mapping URLs to their resolved claim data.
|
||||
"""
|
||||
if isinstance(urls, str):
|
||||
urls = [urls]
|
||||
|
||||
result = self._call("resolve", urls=urls)
|
||||
return result
|
||||
|
||||
def claim_search(
|
||||
self,
|
||||
channel_claim_id: Optional[str] = None,
|
||||
claim_type: Optional[str] = None,
|
||||
page: int = 1,
|
||||
page_size: int = 50,
|
||||
**kwargs,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Search for claims.
|
||||
|
||||
Args:
|
||||
channel_claim_id: Filter by channel claim ID.
|
||||
claim_type: Filter by claim type (e.g., 'stream').
|
||||
page: Page number (1-indexed).
|
||||
page_size: Number of results per page.
|
||||
**kwargs: Additional search parameters.
|
||||
|
||||
Returns:
|
||||
Search results containing claims and pagination info.
|
||||
"""
|
||||
params: Dict[str, Any] = {
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
}
|
||||
|
||||
if channel_claim_id:
|
||||
params["channel_ids"] = [channel_claim_id]
|
||||
|
||||
if claim_type:
|
||||
params["claim_type"] = claim_type
|
||||
|
||||
params.update(kwargs)
|
||||
|
||||
return self._call("claim_search", **params)
|
||||
|
||||
def get_channel_claims(
|
||||
self,
|
||||
channel_claim_id: str,
|
||||
page: int = 1,
|
||||
page_size: int = 50,
|
||||
include_reposts: bool = False,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Get all claims published by a channel.
|
||||
|
||||
Args:
|
||||
channel_claim_id: The channel's claim ID.
|
||||
page: Page number.
|
||||
page_size: Number of results per page.
|
||||
include_reposts: Whether to include reposts.
|
||||
|
||||
Returns:
|
||||
Claims published by the channel.
|
||||
"""
|
||||
params = {
|
||||
"channel_ids": [channel_claim_id],
|
||||
"claim_type": ["stream", "repost"],
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
"order_by": ["release_time"],
|
||||
}
|
||||
|
||||
if not include_reposts:
|
||||
params["claim_type"] = ["stream"]
|
||||
|
||||
return self._call("claim_search", **params)
|
||||
|
||||
def get(
|
||||
self, uri: str, download_directory: Optional[str] = None, **kwargs
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Download a claim.
|
||||
|
||||
Args:
|
||||
uri: The LBRY URI to download.
|
||||
download_directory: Directory to save the file.
|
||||
**kwargs: Additional download parameters.
|
||||
|
||||
Returns:
|
||||
Download result containing file info.
|
||||
"""
|
||||
params = {"uri": uri}
|
||||
|
||||
if download_directory:
|
||||
params["download_directory"] = download_directory
|
||||
|
||||
params.update(kwargs)
|
||||
|
||||
return self._call("get", **params)
|
||||
|
||||
def file_list(self, claim_id: Optional[str] = None, **kwargs) -> Dict[str, Any]:
|
||||
"""
|
||||
List files managed by the daemon.
|
||||
|
||||
Args:
|
||||
claim_id: Filter by claim ID.
|
||||
**kwargs: Additional filter parameters.
|
||||
|
||||
Returns:
|
||||
List of file information.
|
||||
"""
|
||||
params = {}
|
||||
|
||||
if claim_id:
|
||||
params["claim_id"] = claim_id
|
||||
|
||||
params.update(kwargs)
|
||||
|
||||
return self._call("file_list", **params)
|
||||
|
||||
def file_delete(self, claim_id: Optional[str] = None, **kwargs) -> Dict[str, Any]:
|
||||
"""
|
||||
Delete a file managed by the daemon.
|
||||
|
||||
Args:
|
||||
claim_id: The claim ID of the file to delete.
|
||||
**kwargs: Additional parameters.
|
||||
|
||||
Returns:
|
||||
Result from the daemon.
|
||||
"""
|
||||
params: Dict[str, Any] = {}
|
||||
|
||||
if claim_id:
|
||||
params["claim_id"] = claim_id
|
||||
|
||||
params.update(kwargs)
|
||||
|
||||
return self._call("file_delete", **params)
|
||||
|
||||
def version(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Get daemon version info.
|
||||
|
||||
Returns:
|
||||
Version information.
|
||||
"""
|
||||
return self._call("version")
|
||||
|
||||
|
||||
def check_daemon_health(client: LbryClient) -> bool:
|
||||
"""
|
||||
Check if the daemon is running and accessible.
|
||||
|
||||
Args:
|
||||
client: LbryClient instance.
|
||||
|
||||
Returns:
|
||||
True if daemon is healthy.
|
||||
|
||||
Raises:
|
||||
LbryClientError: If daemon is not accessible.
|
||||
"""
|
||||
try:
|
||||
status = client.status()
|
||||
logger.debug(f"Daemon status: {status}")
|
||||
return True
|
||||
except LbryClientError:
|
||||
raise
|
||||
@@ -0,0 +1,408 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
LBRY Downloader - Incrementally sync LBRY/Odysee channel content.
|
||||
|
||||
This tool downloads all downloadable file claims from configured channels,
|
||||
maintaining persistent state to avoid re-downloading unchanged content.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import List, Optional
|
||||
|
||||
from config_loader import ConfigError, ensure_directories, load_config
|
||||
from downloader import DownloadError, Downloader
|
||||
from lbry_client import LbryClient, LbryClientError, check_daemon_health
|
||||
from models import Channel, Config, DownloadAction
|
||||
from planner import Planner
|
||||
from state_db import StateDb
|
||||
from utils import (
|
||||
create_channel_folder_name,
|
||||
expand_path,
|
||||
format_summary,
|
||||
normalize_odysee_url,
|
||||
)
|
||||
|
||||
|
||||
def setup_logging(log_level: str) -> None:
|
||||
"""Configure logging."""
|
||||
logging.basicConfig(
|
||||
level=getattr(logging, log_level),
|
||||
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
|
||||
datefmt="%Y-%m-%d %H:%M:%S",
|
||||
)
|
||||
|
||||
|
||||
def resolve_channel(
|
||||
client: LbryClient,
|
||||
channel_input: str,
|
||||
) -> Optional[Channel]:
|
||||
"""
|
||||
Resolve a channel input to a Channel object.
|
||||
|
||||
Args:
|
||||
client: Lbry client instance.
|
||||
channel_input: URL, URI, or claim ID.
|
||||
|
||||
Returns:
|
||||
Resolved Channel or None if resolution failed.
|
||||
"""
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Normalize input
|
||||
if channel_input.startswith("http"):
|
||||
uri = normalize_odysee_url(channel_input)
|
||||
elif not channel_input.startswith("lbry://"):
|
||||
uri = f"lbry://{channel_input}"
|
||||
else:
|
||||
uri = channel_input
|
||||
|
||||
try:
|
||||
logger.debug(f"Resolving: {uri}")
|
||||
result = client.resolve([uri])
|
||||
|
||||
if not result or uri not in result:
|
||||
logger.error(f"Could not resolve channel: {channel_input}")
|
||||
return None
|
||||
|
||||
resolved = result[uri]
|
||||
|
||||
# Check if it's a channel
|
||||
if "claim" in resolved:
|
||||
claim = resolved["claim"]
|
||||
else:
|
||||
claim = resolved
|
||||
|
||||
value_type = claim.get("value_type", "").lower()
|
||||
if value_type != "channel":
|
||||
logger.error(f"Resolved claim is not a channel: {channel_input}")
|
||||
return None
|
||||
|
||||
channel_claim_id = claim.get("claim_id")
|
||||
channel_name = claim.get("name")
|
||||
|
||||
if not channel_claim_id:
|
||||
logger.error(f"No claim_id in resolved channel: {channel_input}")
|
||||
return None
|
||||
|
||||
# Create folder name
|
||||
folder = create_channel_folder_name(channel_name, channel_claim_id)
|
||||
|
||||
value = claim.get("value", {})
|
||||
|
||||
channel = Channel(
|
||||
input=channel_input,
|
||||
normalized_uri=uri,
|
||||
channel_claim_id=channel_claim_id,
|
||||
channel_name=channel_name,
|
||||
folder=folder,
|
||||
display_name=value.get("title"),
|
||||
permanent_url=claim.get("permanent_url"),
|
||||
short_url=claim.get("short_url"),
|
||||
)
|
||||
|
||||
logger.info(f"Resolved channel: {channel_name} ({channel_claim_id})")
|
||||
return channel
|
||||
|
||||
except LbryClientError as e:
|
||||
logger.error(f"Failed to resolve channel {channel_input}: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def enumerate_channel_claims(
|
||||
client: LbryClient,
|
||||
channel: Channel,
|
||||
config: Config,
|
||||
) -> List[dict]:
|
||||
"""
|
||||
Enumerate all downloadable claims from a channel.
|
||||
|
||||
Args:
|
||||
client: Lbry client.
|
||||
channel: Channel to enumerate.
|
||||
config: Application config.
|
||||
|
||||
Returns:
|
||||
List of claim metadata.
|
||||
"""
|
||||
logger = logging.getLogger(__name__)
|
||||
all_claims = []
|
||||
page = 1
|
||||
|
||||
logger.info(f"Scanning channel: {channel.channel_name}")
|
||||
|
||||
while True:
|
||||
try:
|
||||
result = client.get_channel_claims(
|
||||
channel_claim_id=channel.channel_claim_id,
|
||||
page=page,
|
||||
page_size=config.general.channel_page_size,
|
||||
include_reposts=config.general.include_reposts,
|
||||
)
|
||||
|
||||
items = result.get("items", [])
|
||||
if not items:
|
||||
break
|
||||
|
||||
all_claims.extend(items)
|
||||
logger.debug(f" Page {page}: {len(items)} claims")
|
||||
|
||||
# Check if we've reached the end
|
||||
total_pages = result.get("total_pages", 1)
|
||||
if page >= total_pages:
|
||||
break
|
||||
|
||||
page += 1
|
||||
|
||||
except LbryClientError as e:
|
||||
logger.error(f"Failed to enumerate page {page}: {e}")
|
||||
break
|
||||
|
||||
logger.info(f" Total claims found: {len(all_claims)}")
|
||||
return all_claims
|
||||
|
||||
|
||||
def run_sync(config: Config, dry_run: bool = False) -> dict:
|
||||
"""
|
||||
Run the main synchronization process.
|
||||
|
||||
Args:
|
||||
config: Application configuration.
|
||||
dry_run: If True, don't actually download.
|
||||
|
||||
Returns:
|
||||
Statistics dictionary.
|
||||
"""
|
||||
logger = logging.getLogger(__name__)
|
||||
stats = {
|
||||
"channels_scanned": 0,
|
||||
"claims_examined": 0,
|
||||
"new_downloads": 0,
|
||||
"new_versions": 0,
|
||||
"skipped_existing": 0,
|
||||
"redownloaded_missing": 0,
|
||||
"failures": 0,
|
||||
}
|
||||
|
||||
# Initialize components
|
||||
client = LbryClient(
|
||||
api_url=config.lbrynet.api_url,
|
||||
timeout=config.lbrynet.timeout_seconds,
|
||||
)
|
||||
|
||||
# Check daemon health
|
||||
logger.info(f"Checking daemon at {config.lbrynet.api_url}")
|
||||
try:
|
||||
check_daemon_health(client)
|
||||
logger.info("Daemon is healthy")
|
||||
except LbryClientError as e:
|
||||
logger.error(str(e))
|
||||
return stats
|
||||
|
||||
# Load state
|
||||
state_db = StateDb(config.general.state_file)
|
||||
state_db.load()
|
||||
|
||||
# Initialize planner and downloader
|
||||
base_path = Path(config.general.base_dir)
|
||||
planner = Planner(config, state_db.data, str(base_path))
|
||||
downloader = Downloader(client, config, base_path)
|
||||
|
||||
# Process each enabled channel
|
||||
enabled_channels = [c for c in config.channels if c.enabled]
|
||||
|
||||
for channel_config in enabled_channels:
|
||||
stats["channels_scanned"] += 1
|
||||
|
||||
# Resolve channel
|
||||
channel = resolve_channel(client, channel_config.input)
|
||||
if not channel:
|
||||
stats["failures"] += 1
|
||||
continue
|
||||
|
||||
# Set custom download path if provided in config
|
||||
if channel_config.download_path:
|
||||
channel.download_path = channel_config.download_path
|
||||
|
||||
# Check if we already have this channel in state
|
||||
existing = state_db.get_channel(channel.channel_claim_id)
|
||||
if existing:
|
||||
# Update with new info but keep folder and download_path
|
||||
channel.folder = existing.folder
|
||||
if existing.download_path:
|
||||
channel.download_path = existing.download_path
|
||||
|
||||
# Enumerate claims
|
||||
claims = enumerate_channel_claims(client, channel, config)
|
||||
|
||||
# Plan downloads
|
||||
actions = planner.process_channel_claims(channel, claims)
|
||||
|
||||
# Apply download limit if configured
|
||||
# Separate skip actions from download actions
|
||||
skip_actions = [a for a in actions if a.action == "skip_existing"]
|
||||
download_actions = [a for a in actions if a.action != "skip_existing"]
|
||||
|
||||
# Sort download actions by release_time (most recent first)
|
||||
# Claims without release_time will be at the end
|
||||
download_actions.sort(
|
||||
key=lambda a: a.metadata.get("release_time") or 0, reverse=True
|
||||
)
|
||||
|
||||
# Apply limit: 0 means "all" (no limit), otherwise limit to download_limit
|
||||
if (
|
||||
config.general.download_limit > 0
|
||||
and len(download_actions) > config.general.download_limit
|
||||
):
|
||||
skipped_due_to_limit = download_actions[config.general.download_limit :]
|
||||
download_actions = download_actions[: config.general.download_limit]
|
||||
|
||||
logger.info(
|
||||
f"Download limit ({config.general.download_limit}) reached. "
|
||||
f"Skipping {len(skipped_due_to_limit)} older items."
|
||||
)
|
||||
|
||||
# Mark skipped items in stats
|
||||
for action in skipped_due_to_limit:
|
||||
stats["skipped_existing"] += 1
|
||||
|
||||
# Recombine actions
|
||||
actions = skip_actions + download_actions
|
||||
stats["claims_examined"] += len(actions)
|
||||
|
||||
# Execute downloads
|
||||
for action in actions:
|
||||
if action.action == "skip_existing":
|
||||
stats["skipped_existing"] += 1
|
||||
elif action.action == "redownload_missing":
|
||||
stats["redownloaded_missing"] += 1
|
||||
elif action.action in ("download_new", "download_new_version"):
|
||||
if action.action == "download_new":
|
||||
stats["new_downloads"] += 1
|
||||
else:
|
||||
stats["new_versions"] += 1
|
||||
|
||||
try:
|
||||
success = downloader.download(action, dry_run=dry_run)
|
||||
|
||||
if success and not dry_run and action.action not in ("skip_existing",):
|
||||
# Update state
|
||||
planner.update_state_from_action(action, success=True)
|
||||
|
||||
# Log to history
|
||||
history_file = base_path / "state" / "run-history.jsonl"
|
||||
state_db.log_run_history(
|
||||
str(history_file),
|
||||
channel.channel_claim_id,
|
||||
action.claim_id,
|
||||
action.action,
|
||||
action.version_token,
|
||||
"success" if success else "failure",
|
||||
)
|
||||
|
||||
except DownloadError as e:
|
||||
logger.error(f"Download failed: {e}")
|
||||
stats["failures"] += 1
|
||||
|
||||
# Log failure
|
||||
history_file = base_path / "state" / "run-history.jsonl"
|
||||
state_db.log_run_history(
|
||||
str(history_file),
|
||||
channel.channel_claim_id,
|
||||
action.claim_id,
|
||||
action.action,
|
||||
action.version_token,
|
||||
"failure",
|
||||
str(e),
|
||||
)
|
||||
|
||||
# Update channel last scan time
|
||||
from datetime import datetime, timezone
|
||||
|
||||
channel.last_scan = datetime.now(timezone.utc).isoformat()
|
||||
state_db.set_channel(channel)
|
||||
|
||||
# Save state
|
||||
if not dry_run:
|
||||
state_db.update_last_run()
|
||||
state_db.save()
|
||||
logger.info("State saved")
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
def main():
|
||||
"""Main entry point."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="LBRY/Odysee Channel Downloader",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
python main.py # Run with default config
|
||||
python main.py --config ./config.yaml # Use custom config
|
||||
python main.py --dry-run # Plan actions but don't download
|
||||
""",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--config",
|
||||
"-c",
|
||||
help="Path to configuration file (default: ~/Documents/lbry-downloads/config.yaml)",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--dry-run",
|
||||
"-n",
|
||||
action="store_true",
|
||||
help="Show what would be downloaded without actually downloading",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
try:
|
||||
# Load config
|
||||
config = load_config(args.config)
|
||||
|
||||
# Setup logging
|
||||
setup_logging(config.general.log_level)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
logger.info("LBRY Downloader starting")
|
||||
|
||||
# Ensure directories exist
|
||||
ensure_directories(config)
|
||||
|
||||
# Run sync
|
||||
stats = run_sync(config, dry_run=args.dry_run)
|
||||
|
||||
# Print summary
|
||||
summary = format_summary(
|
||||
channels_scanned=stats["channels_scanned"],
|
||||
claims_examined=stats["claims_examined"],
|
||||
new_downloads=stats["new_downloads"],
|
||||
new_versions=stats["new_versions"],
|
||||
skipped_existing=stats["skipped_existing"],
|
||||
redownloaded_missing=stats["redownloaded_missing"],
|
||||
failures=stats["failures"],
|
||||
)
|
||||
print("\n" + summary)
|
||||
|
||||
# Exit with error code if there were failures
|
||||
if stats["failures"] > 0:
|
||||
sys.exit(1)
|
||||
|
||||
except ConfigError as e:
|
||||
print(f"Configuration error: {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
except KeyboardInterrupt:
|
||||
print("\nInterrupted by user", file=sys.stderr)
|
||||
sys.exit(130)
|
||||
except Exception as e:
|
||||
print(f"Unexpected error: {e}", file=sys.stderr)
|
||||
raise
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,289 @@
|
||||
"""Data models for LBRY Downloader."""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChannelConfig:
|
||||
"""Configuration for a channel to sync."""
|
||||
|
||||
input: str
|
||||
enabled: bool = True
|
||||
download_path: Optional[str] = None
|
||||
tags_include: List[str] = field(default_factory=list)
|
||||
tags_exclude: List[str] = field(default_factory=list)
|
||||
|
||||
|
||||
@dataclass
|
||||
class GeneralConfig:
|
||||
"""General application configuration."""
|
||||
|
||||
base_dir: str = "~/Documents/lbry-downloads"
|
||||
state_file: str = "~/Documents/lbry-downloads/state/database.json"
|
||||
max_workers: int = 2
|
||||
log_level: str = "INFO"
|
||||
dry_run: bool = False
|
||||
verify_existing_files: bool = True
|
||||
write_checksums: bool = True
|
||||
filename_mode: str = "original"
|
||||
include_reposts: bool = False
|
||||
channel_page_size: int = 50
|
||||
keep_missing_claim_records: bool = True
|
||||
download_limit: int = (
|
||||
10 # Number of most recent downloads per channel, or "all" (0)
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class LbrynetConfig:
|
||||
"""LBRY daemon configuration."""
|
||||
|
||||
api_url: str = "http://127.0.0.1:5279"
|
||||
timeout_seconds: int = 60
|
||||
|
||||
|
||||
@dataclass
|
||||
class Config:
|
||||
"""Complete application configuration."""
|
||||
|
||||
lbrynet: LbrynetConfig = field(default_factory=LbrynetConfig)
|
||||
general: GeneralConfig = field(default_factory=GeneralConfig)
|
||||
channels: List[ChannelConfig] = field(default_factory=list)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Channel:
|
||||
"""A resolved LBRY channel."""
|
||||
|
||||
input: str
|
||||
normalized_uri: str
|
||||
channel_claim_id: str
|
||||
channel_name: str
|
||||
folder: str
|
||||
download_path: Optional[str] = None
|
||||
display_name: Optional[str] = None
|
||||
permanent_url: Optional[str] = None
|
||||
short_url: Optional[str] = None
|
||||
last_scan: Optional[str] = None
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"input": self.input,
|
||||
"normalized_uri": self.normalized_uri,
|
||||
"channel_claim_id": self.channel_claim_id,
|
||||
"channel_name": self.channel_name,
|
||||
"folder": self.folder,
|
||||
"download_path": self.download_path,
|
||||
"display_name": self.display_name,
|
||||
"permanent_url": self.permanent_url,
|
||||
"short_url": self.short_url,
|
||||
"last_scan": self.last_scan,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, Any]) -> "Channel":
|
||||
return cls(
|
||||
input=data["input"],
|
||||
normalized_uri=data["normalized_uri"],
|
||||
channel_claim_id=data["channel_claim_id"],
|
||||
channel_name=data["channel_name"],
|
||||
folder=data["folder"],
|
||||
download_path=data.get("download_path"),
|
||||
display_name=data.get("display_name"),
|
||||
permanent_url=data.get("permanent_url"),
|
||||
short_url=data.get("short_url"),
|
||||
last_scan=data.get("last_scan"),
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ClaimVersion:
|
||||
"""A specific version of a claim."""
|
||||
|
||||
version_token: str
|
||||
txid: str
|
||||
nout: int
|
||||
sd_hash: Optional[str] = None
|
||||
stream_hash: Optional[str] = None
|
||||
published_at: Optional[str] = None
|
||||
downloaded: bool = False
|
||||
file_relpath: Optional[str] = None
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"version_token": self.version_token,
|
||||
"txid": self.txid,
|
||||
"nout": self.nout,
|
||||
"sd_hash": self.sd_hash,
|
||||
"stream_hash": self.stream_hash,
|
||||
"published_at": self.published_at,
|
||||
"downloaded": self.downloaded,
|
||||
"file_relpath": self.file_relpath,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, Any]) -> "ClaimVersion":
|
||||
return cls(
|
||||
version_token=data["version_token"],
|
||||
txid=data["txid"],
|
||||
nout=data["nout"],
|
||||
sd_hash=data.get("sd_hash"),
|
||||
stream_hash=data.get("stream_hash"),
|
||||
published_at=data.get("published_at"),
|
||||
downloaded=data.get("downloaded", False),
|
||||
file_relpath=data.get("file_relpath"),
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Claim:
|
||||
"""A LBRY claim (downloadable content)."""
|
||||
|
||||
claim_id: str
|
||||
channel_claim_id: str
|
||||
name: str
|
||||
permanent_url: str
|
||||
claim_folder: str
|
||||
title: Optional[str] = None
|
||||
first_seen: Optional[str] = None
|
||||
last_seen: Optional[str] = None
|
||||
latest_version_token: Optional[str] = None
|
||||
versions: Dict[str, ClaimVersion] = field(default_factory=dict)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"claim_id": self.claim_id,
|
||||
"channel_claim_id": self.channel_claim_id,
|
||||
"name": self.name,
|
||||
"title": self.title,
|
||||
"permanent_url": self.permanent_url,
|
||||
"claim_folder": self.claim_folder,
|
||||
"first_seen": self.first_seen,
|
||||
"last_seen": self.last_seen,
|
||||
"latest_version_token": self.latest_version_token,
|
||||
"versions": {k: v.to_dict() for k, v in self.versions.items()},
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, Any]) -> "Claim":
|
||||
versions = {
|
||||
k: ClaimVersion.from_dict(v) for k, v in data.get("versions", {}).items()
|
||||
}
|
||||
return cls(
|
||||
claim_id=data["claim_id"],
|
||||
channel_claim_id=data["channel_claim_id"],
|
||||
name=data["name"],
|
||||
title=data.get("title"),
|
||||
permanent_url=data["permanent_url"],
|
||||
claim_folder=data["claim_folder"],
|
||||
first_seen=data.get("first_seen"),
|
||||
last_seen=data.get("last_seen"),
|
||||
latest_version_token=data.get("latest_version_token"),
|
||||
versions=versions,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class DownloadAction:
|
||||
"""Represents a planned download action."""
|
||||
|
||||
claim_id: str
|
||||
claim_name: str
|
||||
channel_claim_id: str
|
||||
version_token: str
|
||||
action: str # 'download_new', 'download_new_version', 'redownload_missing', 'skip_existing'
|
||||
uri: str
|
||||
target_dir: str
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
class StateDatabase:
|
||||
"""In-memory representation of the state database."""
|
||||
|
||||
schema_version: int = 1
|
||||
channels: Dict[str, Channel] = field(default_factory=dict)
|
||||
claims: Dict[str, Claim] = field(default_factory=dict)
|
||||
downloads: Dict[str, Any] = field(default_factory=dict)
|
||||
last_run: Optional[str] = None
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"schema_version": self.schema_version,
|
||||
"channels": {k: v.to_dict() for k, v in self.channels.items()},
|
||||
"claims": {k: v.to_dict() for k, v in self.claims.items()},
|
||||
"downloads": self.downloads,
|
||||
"last_run": self.last_run,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, Any]) -> "StateDatabase":
|
||||
channels = {
|
||||
k: Channel.from_dict(v) for k, v in data.get("channels", {}).items()
|
||||
}
|
||||
claims = {k: Claim.from_dict(v) for k, v in data.get("claims", {}).items()}
|
||||
return cls(
|
||||
schema_version=data.get("schema_version", 1),
|
||||
channels=channels,
|
||||
claims=claims,
|
||||
downloads=data.get("downloads", {}),
|
||||
last_run=data.get("last_run"),
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class NormalizedMetadata:
|
||||
"""Normalized claim metadata for storage."""
|
||||
|
||||
claim_id: str
|
||||
channel_claim_id: str
|
||||
name: str
|
||||
title: Optional[str]
|
||||
claim_type: str
|
||||
value_type: str
|
||||
version_token: str
|
||||
sd_hash: Optional[str]
|
||||
stream_hash: Optional[str]
|
||||
txid: str
|
||||
nout: int
|
||||
permanent_url: str
|
||||
short_url: Optional[str]
|
||||
canonical_url: Optional[str]
|
||||
timestamp: int
|
||||
release_time: Optional[int]
|
||||
fee: Optional[Dict]
|
||||
tags: List[str]
|
||||
languages: List[str]
|
||||
source_name: Optional[str]
|
||||
source_media_type: Optional[str]
|
||||
file_name: Optional[str]
|
||||
download_path: Optional[str]
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"claim_id": self.claim_id,
|
||||
"channel_claim_id": self.channel_claim_id,
|
||||
"name": self.name,
|
||||
"title": self.title,
|
||||
"claim_type": self.claim_type,
|
||||
"value_type": self.value_type,
|
||||
"version_token": self.version_token,
|
||||
"sd_hash": self.sd_hash,
|
||||
"stream_hash": self.stream_hash,
|
||||
"txid": self.txid,
|
||||
"nout": self.nout,
|
||||
"permanent_url": self.permanent_url,
|
||||
"short_url": self.short_url,
|
||||
"canonical_url": self.canonical_url,
|
||||
"timestamp": self.timestamp,
|
||||
"release_time": self.release_time,
|
||||
"fee": self.fee,
|
||||
"tags": self.tags,
|
||||
"languages": self.languages,
|
||||
"source_name": self.source_name,
|
||||
"source_media_type": self.source_media_type,
|
||||
"file_name": self.file_name,
|
||||
"download_path": self.download_path,
|
||||
}
|
||||
+271
@@ -0,0 +1,271 @@
|
||||
"""Planner module - decides what to download based on state."""
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional, Tuple
|
||||
|
||||
from models import (
|
||||
Channel,
|
||||
Claim,
|
||||
ClaimVersion,
|
||||
Config,
|
||||
DownloadAction,
|
||||
StateDatabase,
|
||||
)
|
||||
from utils import (
|
||||
create_channel_folder_name,
|
||||
create_claim_folder_name,
|
||||
extract_claim_metadata,
|
||||
generate_version_token,
|
||||
is_downloadable_claim,
|
||||
)
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Planner:
|
||||
"""Plans download actions by comparing discovered claims with state."""
|
||||
|
||||
def __init__(self, config: Config, state: StateDatabase, base_dir: str):
|
||||
self.config = config
|
||||
self.state = state
|
||||
self.base_dir = Path(base_dir)
|
||||
|
||||
def process_channel_claims(
|
||||
self,
|
||||
channel: Channel,
|
||||
claims_data: List[Dict],
|
||||
) -> List[DownloadAction]:
|
||||
"""
|
||||
Process claims from a channel and decide what to do with each.
|
||||
|
||||
Args:
|
||||
channel: The channel these claims belong to.
|
||||
claims_data: List of claim metadata from daemon.
|
||||
|
||||
Returns:
|
||||
List of planned download actions.
|
||||
"""
|
||||
actions = []
|
||||
|
||||
for claim_data in claims_data:
|
||||
action = self._process_single_claim(channel, claim_data)
|
||||
if action:
|
||||
actions.append(action)
|
||||
|
||||
return actions
|
||||
|
||||
def _process_single_claim(
|
||||
self,
|
||||
channel: Channel,
|
||||
claim_data: Dict,
|
||||
) -> Optional[DownloadAction]:
|
||||
"""
|
||||
Process a single claim and decide what action to take.
|
||||
|
||||
Args:
|
||||
channel: The channel this claim belongs to.
|
||||
claim_data: Claim metadata from daemon.
|
||||
|
||||
Returns:
|
||||
DownloadAction or None if claim should be skipped.
|
||||
"""
|
||||
# Handle wrapped response format
|
||||
if "claim" in claim_data:
|
||||
claim_value = claim_data["claim"]
|
||||
else:
|
||||
claim_value = claim_data
|
||||
|
||||
# Check if this is a downloadable claim
|
||||
if not is_downloadable_claim(claim_value):
|
||||
logger.debug(f"Skipping non-downloadable claim")
|
||||
return None
|
||||
|
||||
# Extract metadata
|
||||
try:
|
||||
metadata = extract_claim_metadata(claim_value)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to extract metadata for claim: {e}")
|
||||
return None
|
||||
|
||||
claim_id = metadata["claim_id"]
|
||||
claim_name = metadata["name"]
|
||||
|
||||
# Skip reposts if not enabled
|
||||
value_type = claim_value.get("value_type", "").lower()
|
||||
if value_type == "repost" and not self.config.general.include_reposts:
|
||||
logger.debug(f"Skipping repost claim {claim_name}")
|
||||
return None
|
||||
|
||||
# Generate version token
|
||||
version_token = generate_version_token(
|
||||
sd_hash=metadata.get("sd_hash"),
|
||||
stream_hash=metadata.get("stream_hash"),
|
||||
txid=metadata.get("txid"),
|
||||
nout=metadata.get("nout"),
|
||||
metadata=metadata,
|
||||
)
|
||||
|
||||
# Build paths
|
||||
channel_folder = create_channel_folder_name(
|
||||
channel.channel_name, channel.channel_claim_id
|
||||
)
|
||||
claim_folder = create_claim_folder_name(claim_name, claim_id)
|
||||
|
||||
# Use custom download path if provided, otherwise use base_dir
|
||||
if channel.download_path:
|
||||
download_base = Path(channel.download_path)
|
||||
else:
|
||||
download_base = self.base_dir / "channels"
|
||||
|
||||
version_dir = (
|
||||
download_base
|
||||
/ channel_folder
|
||||
/ "claims"
|
||||
/ claim_folder
|
||||
/ "versions"
|
||||
/ version_token
|
||||
)
|
||||
|
||||
# Check state for this claim
|
||||
existing_claim = self.state.claims.get(claim_id)
|
||||
|
||||
if existing_claim is None:
|
||||
# New claim
|
||||
action = DownloadAction(
|
||||
claim_id=claim_id,
|
||||
claim_name=claim_name,
|
||||
channel_claim_id=channel.channel_claim_id,
|
||||
version_token=version_token,
|
||||
action="download_new",
|
||||
uri=metadata["permanent_url"],
|
||||
target_dir=str(version_dir),
|
||||
metadata=metadata,
|
||||
)
|
||||
logger.info(f"New claim: {claim_name} ({action.action})")
|
||||
return action
|
||||
|
||||
# Check if this is a new version
|
||||
if version_token in existing_claim.versions:
|
||||
# We know about this version
|
||||
version = existing_claim.versions[version_token]
|
||||
|
||||
if version.downloaded:
|
||||
# Check if file still exists
|
||||
if self.config.general.verify_existing_files:
|
||||
if version.file_relpath:
|
||||
expected_path = self.base_dir / version.file_relpath
|
||||
if not expected_path.exists():
|
||||
# File is missing - need to redownload
|
||||
action = DownloadAction(
|
||||
claim_id=claim_id,
|
||||
claim_name=claim_name,
|
||||
channel_claim_id=channel.channel_claim_id,
|
||||
version_token=version_token,
|
||||
action="redownload_missing",
|
||||
uri=metadata["permanent_url"],
|
||||
target_dir=str(version_dir),
|
||||
metadata=metadata,
|
||||
)
|
||||
logger.info(f"Redownload missing: {claim_name}")
|
||||
return action
|
||||
|
||||
# File exists - skip
|
||||
action = DownloadAction(
|
||||
claim_id=claim_id,
|
||||
claim_name=claim_name,
|
||||
channel_claim_id=channel.channel_claim_id,
|
||||
version_token=version_token,
|
||||
action="skip_existing",
|
||||
uri=metadata["permanent_url"],
|
||||
target_dir=str(version_dir),
|
||||
metadata=metadata,
|
||||
)
|
||||
logger.debug(f"Skipping existing: {claim_name}")
|
||||
return action
|
||||
else:
|
||||
# Version exists but not downloaded - download it
|
||||
action = DownloadAction(
|
||||
claim_id=claim_id,
|
||||
claim_name=claim_name,
|
||||
channel_claim_id=channel.channel_claim_id,
|
||||
version_token=version_token,
|
||||
action="download_new_version",
|
||||
uri=metadata["permanent_url"],
|
||||
target_dir=str(version_dir),
|
||||
metadata=metadata,
|
||||
)
|
||||
logger.info(f"Download incomplete: {claim_name}")
|
||||
return action
|
||||
else:
|
||||
# New version of existing claim
|
||||
action = DownloadAction(
|
||||
claim_id=claim_id,
|
||||
claim_name=claim_name,
|
||||
channel_claim_id=channel.channel_claim_id,
|
||||
version_token=version_token,
|
||||
action="download_new_version",
|
||||
uri=metadata["permanent_url"],
|
||||
target_dir=str(version_dir),
|
||||
metadata=metadata,
|
||||
)
|
||||
logger.info(f"New version: {claim_name}")
|
||||
return action
|
||||
|
||||
def update_state_from_action(self, action: DownloadAction, success: bool) -> None:
|
||||
"""
|
||||
Update the state database based on a completed action.
|
||||
|
||||
Args:
|
||||
action: The action that was executed.
|
||||
success: Whether the action succeeded.
|
||||
"""
|
||||
if not success:
|
||||
return
|
||||
|
||||
now = datetime.now(timezone.utc).isoformat()
|
||||
|
||||
# Get or create claim record
|
||||
claim = self.state.claims.get(action.claim_id)
|
||||
if claim is None:
|
||||
channel_folder = create_channel_folder_name(
|
||||
action.channel_claim_id[:20], # We don't have channel name here
|
||||
action.channel_claim_id,
|
||||
)
|
||||
claim_folder = create_claim_folder_name(action.claim_name, action.claim_id)
|
||||
|
||||
claim = Claim(
|
||||
claim_id=action.claim_id,
|
||||
channel_claim_id=action.channel_claim_id,
|
||||
name=action.claim_name,
|
||||
permanent_url=action.uri,
|
||||
claim_folder=f"channels/{channel_folder}/claims/{claim_folder}",
|
||||
first_seen=now,
|
||||
last_seen=now,
|
||||
versions={},
|
||||
)
|
||||
self.state.claims[action.claim_id] = claim
|
||||
|
||||
# Update claim metadata
|
||||
claim.last_seen = now
|
||||
claim.latest_version_token = action.version_token
|
||||
|
||||
# Get or create version record
|
||||
if action.version_token not in claim.versions:
|
||||
metadata = action.metadata
|
||||
claim.versions[action.version_token] = ClaimVersion(
|
||||
version_token=action.version_token,
|
||||
txid=metadata.get("txid", ""),
|
||||
nout=metadata.get("nout", 0),
|
||||
sd_hash=metadata.get("sd_hash"),
|
||||
stream_hash=metadata.get("stream_hash"),
|
||||
published_at=metadata.get("release_time"),
|
||||
)
|
||||
|
||||
# Update version record
|
||||
version = claim.versions[action.version_token]
|
||||
if success:
|
||||
version.downloaded = True
|
||||
# File path will be set by downloader
|
||||
@@ -0,0 +1,58 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=45", "wheel"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "lbry-downloader"
|
||||
version = "1.0.0"
|
||||
description = "Incremental sync tool for LBRY/Odysee channels"
|
||||
readme = "README.md"
|
||||
license = {text = "MIT"}
|
||||
requires-python = ">=3.8"
|
||||
authors = [
|
||||
{name = "LBRY Downloader Contributors"}
|
||||
]
|
||||
keywords = ["lbry", "odysee", "download", "sync", "archive"]
|
||||
classifiers = [
|
||||
"Development Status :: 4 - Beta",
|
||||
"Intended Audience :: End Users/Desktop",
|
||||
"License :: OSI Approved :: MIT License",
|
||||
"Programming Language :: Python :: 3",
|
||||
"Programming Language :: Python :: 3.8",
|
||||
"Programming Language :: Python :: 3.9",
|
||||
"Programming Language :: Python :: 3.10",
|
||||
"Programming Language :: Python :: 3.11",
|
||||
"Programming Language :: Python :: 3.12",
|
||||
"Topic :: Internet :: File Transfer Protocol (FTP)",
|
||||
"Topic :: System :: Archiving",
|
||||
]
|
||||
dependencies = [
|
||||
"PyYAML>=6.0",
|
||||
"requests>=2.31.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=7.0.0",
|
||||
"pytest-cov",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
lbry-downloader = "main:main"
|
||||
|
||||
[project.urls]
|
||||
Homepage = "https://github.com/yourusername/LBRY-Downloader"
|
||||
Repository = "https://github.com/yourusername/LBRY-Downloader"
|
||||
Issues = "https://github.com/yourusername/LBRY-Downloader/issues"
|
||||
|
||||
[tool.setuptools]
|
||||
py-modules = [
|
||||
"main",
|
||||
"models",
|
||||
"utils",
|
||||
"config_loader",
|
||||
"lbry_client",
|
||||
"state_db",
|
||||
"planner",
|
||||
"downloader",
|
||||
]
|
||||
@@ -0,0 +1,2 @@
|
||||
PyYAML>=6.0
|
||||
requests>=2.31.0
|
||||
@@ -0,0 +1,593 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
LBRY Downloader - Interactive Setup Script
|
||||
|
||||
This script handles:
|
||||
- Dependency checking and installation
|
||||
- Virtual environment setup
|
||||
- Configuration with user prompts
|
||||
- Download location selection
|
||||
- Channel configuration
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import subprocess
|
||||
import platform
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import Optional, List, Tuple
|
||||
|
||||
|
||||
# Colors for terminal output
|
||||
class Colors:
|
||||
GREEN = "\033[92m"
|
||||
YELLOW = "\033[93m"
|
||||
RED = "\033[91m"
|
||||
BLUE = "\033[94m"
|
||||
BOLD = "\033[1m"
|
||||
END = "\033[0m"
|
||||
|
||||
|
||||
def print_success(msg: str):
|
||||
print(f"{Colors.GREEN}✓{Colors.END} {msg}")
|
||||
|
||||
|
||||
def print_warning(msg: str):
|
||||
print(f"{Colors.YELLOW}⚠{Colors.END} {msg}")
|
||||
|
||||
|
||||
def print_error(msg: str):
|
||||
print(f"{Colors.RED}✗{Colors.END} {msg}")
|
||||
|
||||
|
||||
def print_info(msg: str):
|
||||
print(f"{Colors.BLUE}ℹ{Colors.END} {msg}")
|
||||
|
||||
|
||||
def print_header(msg: str):
|
||||
print(f"\n{Colors.BOLD}{msg}{Colors.END}")
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
def run_command(
|
||||
cmd: List[str], check: bool = True, capture: bool = False
|
||||
) -> Tuple[int, str, str]:
|
||||
"""Run a shell command and return result."""
|
||||
try:
|
||||
if capture:
|
||||
result = subprocess.run(cmd, check=check, capture_output=True, text=True)
|
||||
return result.returncode, result.stdout, result.stderr
|
||||
else:
|
||||
result = subprocess.run(cmd, check=check)
|
||||
return result.returncode, "", ""
|
||||
except subprocess.CalledProcessError as e:
|
||||
if check:
|
||||
raise
|
||||
return e.returncode, "", str(e)
|
||||
except FileNotFoundError:
|
||||
if check:
|
||||
raise
|
||||
return 1, "", f"Command not found: {cmd[0]}"
|
||||
|
||||
|
||||
def check_python_version() -> bool:
|
||||
"""Check if Python version is 3.8 or higher."""
|
||||
version = sys.version_info
|
||||
if version.major < 3 or (version.major == 3 and version.minor < 8):
|
||||
print_error(f"Python 3.8+ required, found {version.major}.{version.minor}")
|
||||
return False
|
||||
print_success(f"Python {version.major}.{version.minor}.{version.micro} found")
|
||||
return True
|
||||
|
||||
|
||||
def check_system_dependencies() -> bool:
|
||||
"""Check if required system packages are available."""
|
||||
print_header("Checking System Dependencies")
|
||||
|
||||
missing = []
|
||||
|
||||
# Check for apt (Debian-based)
|
||||
if shutil.which("apt-get"):
|
||||
print_success("Package manager (apt-get) found")
|
||||
elif shutil.which("dnf"):
|
||||
print_success("Package manager (dnf) found")
|
||||
elif shutil.which("pacman"):
|
||||
print_success("Package manager (pacman) found")
|
||||
else:
|
||||
print_warning("No recognized package manager found")
|
||||
|
||||
# Check for git
|
||||
if shutil.which("git"):
|
||||
print_success("Git found")
|
||||
else:
|
||||
print_warning("Git not found (optional)")
|
||||
|
||||
# Check for curl
|
||||
if shutil.which("curl"):
|
||||
print_success("curl found")
|
||||
else:
|
||||
missing.append("curl")
|
||||
|
||||
return len(missing) == 0
|
||||
|
||||
|
||||
def install_system_dependencies() -> bool:
|
||||
"""Install required system packages."""
|
||||
print_header("Installing System Dependencies")
|
||||
|
||||
packages = ["python3-pip", "python3-venv"]
|
||||
|
||||
if shutil.which("apt-get"):
|
||||
# Debian/Ubuntu/Mint
|
||||
cmd = ["sudo", "apt-get", "update"]
|
||||
print_info("Updating package lists...")
|
||||
try:
|
||||
run_command(cmd)
|
||||
except:
|
||||
print_warning("Failed to update package lists, continuing anyway...")
|
||||
|
||||
cmd = ["sudo", "apt-get", "install", "-y"] + packages
|
||||
print_info("Installing packages: " + ", ".join(packages))
|
||||
try:
|
||||
run_command(cmd)
|
||||
print_success("System dependencies installed")
|
||||
return True
|
||||
except Exception as e:
|
||||
print_error(f"Failed to install packages: {e}")
|
||||
return False
|
||||
else:
|
||||
print_warning("Automatic installation only supported on apt-based systems")
|
||||
print_info("Please manually install: " + ", ".join(packages))
|
||||
return True # Continue anyway
|
||||
|
||||
|
||||
def setup_virtual_environment(install_dir: Path) -> Path:
|
||||
"""Create and setup virtual environment."""
|
||||
print_header("Setting up Virtual Environment")
|
||||
|
||||
venv_path = install_dir / "venv"
|
||||
|
||||
if venv_path.exists():
|
||||
print_warning("Virtual environment already exists")
|
||||
response = input("Recreate it? (y/N): ").strip().lower()
|
||||
if response == "y":
|
||||
shutil.rmtree(venv_path)
|
||||
else:
|
||||
print_info("Using existing virtual environment")
|
||||
return venv_path
|
||||
|
||||
print_info("Creating virtual environment...")
|
||||
try:
|
||||
run_command([sys.executable, "-m", "venv", str(venv_path)])
|
||||
print_success("Virtual environment created")
|
||||
except Exception as e:
|
||||
print_error(f"Failed to create virtual environment: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
return venv_path
|
||||
|
||||
|
||||
def install_python_dependencies(venv_path: Path, install_dir: Path) -> bool:
|
||||
"""Install Python packages in virtual environment."""
|
||||
print_header("Installing Python Dependencies")
|
||||
|
||||
pip_path = venv_path / "bin" / "pip"
|
||||
if not pip_path.exists():
|
||||
pip_path = venv_path / "Scripts" / "pip.exe" # Windows
|
||||
|
||||
requirements_file = install_dir / "requirements.txt"
|
||||
|
||||
print_info("Upgrading pip...")
|
||||
try:
|
||||
run_command([str(pip_path), "install", "--upgrade", "pip"])
|
||||
except:
|
||||
print_warning("Failed to upgrade pip, continuing...")
|
||||
|
||||
print_info("Installing required packages...")
|
||||
try:
|
||||
run_command([str(pip_path), "install", "-r", str(requirements_file)])
|
||||
print_success("Python dependencies installed")
|
||||
return True
|
||||
except Exception as e:
|
||||
print_error(f"Failed to install Python dependencies: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def get_installation_directory() -> Path:
|
||||
"""Ask user for installation directory."""
|
||||
print_header("Installation Directory")
|
||||
|
||||
default_dir = Path.home() / "Documents" / "LBRY-Downloader"
|
||||
|
||||
print_info(f"Default: {default_dir}")
|
||||
response = input(f"Enter installation directory [{default_dir}]: ").strip()
|
||||
|
||||
if response:
|
||||
install_dir = Path(response).expanduser().resolve()
|
||||
else:
|
||||
install_dir = default_dir
|
||||
|
||||
# Create directory if it doesn't exist
|
||||
install_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
print_success(f"Installation directory: {install_dir}")
|
||||
return install_dir
|
||||
|
||||
|
||||
def get_download_location() -> str:
|
||||
"""Ask user for download location."""
|
||||
print_header("Download Location")
|
||||
|
||||
print_info("Where would you like to store downloaded content?")
|
||||
print_info(
|
||||
"(This is the base directory - each channel will have its own subfolder)"
|
||||
)
|
||||
print()
|
||||
|
||||
default_location = "~/Documents/lbry-downloads"
|
||||
print_info(f"Default: {default_location}")
|
||||
|
||||
while True:
|
||||
response = input(f"Enter download location [{default_location}]: ").strip()
|
||||
|
||||
if not response:
|
||||
location = default_location
|
||||
else:
|
||||
location = response
|
||||
|
||||
# Expand and check
|
||||
expanded_path = Path(location).expanduser().resolve()
|
||||
|
||||
try:
|
||||
# Try to create it
|
||||
expanded_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Check if writable
|
||||
test_file = expanded_path / ".write_test"
|
||||
try:
|
||||
test_file.touch()
|
||||
test_file.unlink()
|
||||
print_success(f"Download location: {expanded_path}")
|
||||
return str(location)
|
||||
except PermissionError:
|
||||
print_error(f"Cannot write to {expanded_path}")
|
||||
print_info("Please choose a different location or fix permissions")
|
||||
except Exception as e:
|
||||
print_error(f"Invalid path: {e}")
|
||||
|
||||
|
||||
def get_channels() -> List[dict]:
|
||||
"""Ask user for channels to download."""
|
||||
print_header("Channel Configuration")
|
||||
|
||||
print_info("Enter the channels you want to download from.")
|
||||
print_info("You can add more later by editing the config file.")
|
||||
print()
|
||||
print_info("Supported formats:")
|
||||
print_info(" - Odysee URL: https://odysee.com/@ChannelName:1")
|
||||
print_info(" - LBRY URI: lbry://@ChannelName#1")
|
||||
print_info(" - Just press Enter when done")
|
||||
print()
|
||||
|
||||
channels = []
|
||||
|
||||
while True:
|
||||
channel_num = len(channels) + 1
|
||||
url = input(f"Channel {channel_num} URL (or Enter to finish): ").strip()
|
||||
|
||||
if not url:
|
||||
if len(channels) == 0:
|
||||
print_warning(
|
||||
"No channels added. You can add them later in the config file."
|
||||
)
|
||||
break
|
||||
|
||||
# Basic validation
|
||||
if not ("odysee.com" in url or "lbry://" in url or url.startswith("@")):
|
||||
print_warning("URL format looks unusual. Make sure it's correct.")
|
||||
confirm = input("Use this URL anyway? (Y/n): ").strip().lower()
|
||||
if confirm == "n":
|
||||
continue
|
||||
|
||||
channels.append({"input": url, "enabled": True})
|
||||
print_success(f"Added: {url}")
|
||||
|
||||
return channels
|
||||
|
||||
|
||||
def get_download_limit() -> int:
|
||||
"""Ask user for download limit."""
|
||||
print_header("Download Limit")
|
||||
|
||||
print_info("How many items should be downloaded per channel per run?")
|
||||
print_info(" - Lower numbers = safer for testing")
|
||||
print_info(" - Higher numbers = faster initial sync")
|
||||
print_info(" - Downloads most recent items first")
|
||||
print()
|
||||
|
||||
print_info("Recommended values:")
|
||||
print_info(" - 5-10: Good for testing or large channels")
|
||||
print_info(" - 20-50: Moderate syncing")
|
||||
print_info(" - 0: Download everything (use with caution!)")
|
||||
print()
|
||||
|
||||
while True:
|
||||
response = input("Download limit per channel [10]: ").strip()
|
||||
|
||||
if not response:
|
||||
return 10
|
||||
|
||||
try:
|
||||
limit = int(response)
|
||||
if limit < 0:
|
||||
print_error("Please enter 0 or a positive number")
|
||||
continue
|
||||
|
||||
if limit == 0:
|
||||
confirm = (
|
||||
input("Are you sure you want to download ALL items? (y/N): ")
|
||||
.strip()
|
||||
.lower()
|
||||
)
|
||||
if confirm != "y":
|
||||
continue
|
||||
print_warning("Download limit set to ALL items")
|
||||
else:
|
||||
print_success(f"Download limit set to {limit} items per channel")
|
||||
|
||||
return limit
|
||||
except ValueError:
|
||||
print_error("Please enter a number")
|
||||
|
||||
|
||||
def create_config(
|
||||
download_location: str, channels: List[dict], download_limit: int
|
||||
) -> str:
|
||||
"""Generate configuration file content."""
|
||||
config = f"""# LBRY Downloader Configuration
|
||||
# Generated by setup script on {platform.platform()}
|
||||
|
||||
lbrynet:
|
||||
# URL of the local LBRY daemon (lbrynet)
|
||||
api_url: "http://127.0.0.1:5279"
|
||||
# Timeout for daemon requests in seconds
|
||||
timeout_seconds: 60
|
||||
|
||||
general:
|
||||
# Base directory for all downloads and state
|
||||
base_dir: "{download_location}"
|
||||
# Path to the state database
|
||||
state_file: "{download_location}/state/database.json"
|
||||
# Number of concurrent downloads (keep low to avoid daemon issues)
|
||||
max_workers: 2
|
||||
# Logging level: DEBUG, INFO, WARNING, ERROR, CRITICAL
|
||||
log_level: "INFO"
|
||||
# If true, show what would be downloaded without actually downloading
|
||||
dry_run: false
|
||||
# Verify that previously downloaded files still exist
|
||||
verify_existing_files: true
|
||||
# Write SHA256 checksums for downloaded files
|
||||
write_checksums: true
|
||||
# Filename handling: 'original' or 'safe' (aggressively sanitized)
|
||||
filename_mode: "original"
|
||||
# Include reposts in downloads (default: false)
|
||||
include_reposts: false
|
||||
# Number of claims to fetch per page from the daemon
|
||||
channel_page_size: 50
|
||||
# Keep records of claims even if they disappear from the channel
|
||||
keep_missing_claim_records: true
|
||||
# Download limit: Number of most recent downloads per channel (0 = all)
|
||||
download_limit: {download_limit}
|
||||
|
||||
channels:
|
||||
"""
|
||||
|
||||
for channel in channels:
|
||||
config += f''' - input: "{channel["input"]}"
|
||||
enabled: true
|
||||
'''
|
||||
|
||||
if not channels:
|
||||
config += """ # Add your channels here:
|
||||
# - input: "https://odysee.com/@ChannelName:1"
|
||||
# enabled: true
|
||||
"""
|
||||
|
||||
return config
|
||||
|
||||
|
||||
def create_launcher_script(install_dir: Path, venv_path: Path) -> None:
|
||||
"""Create launcher scripts for easy execution."""
|
||||
print_header("Creating Launcher Scripts")
|
||||
|
||||
# Create bin directory
|
||||
bin_dir = install_dir / "bin"
|
||||
bin_dir.mkdir(exist_ok=True)
|
||||
|
||||
# Create main launcher
|
||||
launcher = bin_dir / "lbry-downloader"
|
||||
launcher_content = f"""#!/bin/bash
|
||||
# LBRY Downloader Launcher
|
||||
|
||||
INSTALL_DIR="{install_dir}"
|
||||
VENV_DIR="{venv_path}"
|
||||
|
||||
cd "$INSTALL_DIR"
|
||||
source "$VENV_DIR/bin/activate"
|
||||
|
||||
python main.py "$@"
|
||||
"""
|
||||
|
||||
launcher.write_text(launcher_content)
|
||||
launcher.chmod(0o755)
|
||||
print_success(f"Created launcher: {launcher}")
|
||||
|
||||
# Create quick test script
|
||||
test_script = bin_dir / "lbry-test"
|
||||
test_content = f"""#!/bin/bash
|
||||
# Quick test script
|
||||
|
||||
INSTALL_DIR="{install_dir}"
|
||||
VENV_DIR="{venv_path}"
|
||||
|
||||
cd "$INSTALL_DIR"
|
||||
source "$VENV_DIR/bin/activate"
|
||||
|
||||
echo "Testing LBRY Downloader..."
|
||||
python main.py --dry-run "$@"
|
||||
"""
|
||||
|
||||
test_script.write_text(test_content)
|
||||
test_script.chmod(0o755)
|
||||
print_success(f"Created test script: {test_script}")
|
||||
|
||||
# Check if we should add to PATH
|
||||
shell_rc = Path.home() / ".bashrc"
|
||||
if not shell_rc.exists():
|
||||
shell_rc = Path.home() / ".zshrc"
|
||||
|
||||
if shell_rc.exists():
|
||||
print_info(
|
||||
f"To use 'lbry-downloader' from anywhere, add this to your {shell_rc.name}:"
|
||||
)
|
||||
print(f' export PATH="{bin_dir}:$PATH"')
|
||||
|
||||
|
||||
def run_setup() -> None:
|
||||
"""Main setup function."""
|
||||
print(f"""
|
||||
{Colors.BOLD}╔══════════════════════════════════════════════════════════════╗
|
||||
║ LBRY Downloader - Interactive Setup ║
|
||||
║ ║
|
||||
║ This script will help you install and configure the ║
|
||||
║ LBRY Downloader tool. ║
|
||||
╚══════════════════════════════════════════════════════════════╝{Colors.END}
|
||||
""")
|
||||
|
||||
# Check Python version
|
||||
if not check_python_version():
|
||||
print_error("Python 3.8+ is required. Please upgrade Python.")
|
||||
sys.exit(1)
|
||||
|
||||
# Check system dependencies
|
||||
if not check_system_dependencies():
|
||||
print_info("Some system dependencies are missing.")
|
||||
response = input("Install system dependencies? (Y/n): ").strip().lower()
|
||||
if response != "n":
|
||||
if not install_system_dependencies():
|
||||
print_warning("Failed to install some dependencies")
|
||||
response = input("Continue anyway? (y/N): ").strip().lower()
|
||||
if response != "y":
|
||||
sys.exit(1)
|
||||
|
||||
# Get installation directory
|
||||
install_dir = get_installation_directory()
|
||||
|
||||
# Copy files if not running from install directory
|
||||
current_dir = Path(__file__).parent.resolve()
|
||||
if current_dir != install_dir:
|
||||
print_header("Copying Files")
|
||||
print_info(f"Copying from {current_dir} to {install_dir}")
|
||||
|
||||
# Copy Python files
|
||||
for pattern in ["*.py", "*.md", "*.txt", "*.toml", "*.example", "Makefile"]:
|
||||
for file in current_dir.glob(pattern):
|
||||
if file.name != "setup.py": # Don't copy setup script itself
|
||||
shutil.copy2(file, install_dir)
|
||||
print_success(f"Copied {file.name}")
|
||||
|
||||
# Copy tests directory
|
||||
tests_src = current_dir / "tests"
|
||||
tests_dst = install_dir / "tests"
|
||||
if tests_src.exists():
|
||||
if tests_dst.exists():
|
||||
shutil.rmtree(tests_dst)
|
||||
shutil.copytree(tests_src, tests_dst)
|
||||
print_success("Copied tests/")
|
||||
|
||||
# Setup virtual environment
|
||||
venv_path = setup_virtual_environment(install_dir)
|
||||
|
||||
# Install Python dependencies
|
||||
if not install_python_dependencies(venv_path, install_dir):
|
||||
print_error("Failed to install Python dependencies")
|
||||
sys.exit(1)
|
||||
|
||||
# Get configuration from user
|
||||
download_location = get_download_location()
|
||||
channels = get_channels()
|
||||
download_limit = get_download_limit()
|
||||
|
||||
# Create config file
|
||||
print_header("Creating Configuration")
|
||||
config_content = create_config(download_location, channels, download_limit)
|
||||
|
||||
config_path = Path(download_location).expanduser() / "config.yaml"
|
||||
config_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
config_path.write_text(config_content)
|
||||
print_success(f"Created configuration: {config_path}")
|
||||
|
||||
# Create necessary directories
|
||||
(config_path.parent / "state").mkdir(exist_ok=True)
|
||||
(config_path.parent / "channels").mkdir(exist_ok=True)
|
||||
print_success("Created directory structure")
|
||||
|
||||
# Create launcher scripts
|
||||
create_launcher_script(install_dir, venv_path)
|
||||
|
||||
# Final instructions
|
||||
print(f"""
|
||||
{Colors.BOLD}{Colors.GREEN}╔══════════════════════════════════════════════════════════════╗
|
||||
║ Setup Complete! ║
|
||||
╚══════════════════════════════════════════════════════════════╝{Colors.END}
|
||||
|
||||
Next steps:
|
||||
|
||||
1. Start the LBRY daemon:
|
||||
lbrynet start
|
||||
|
||||
(If lbrynet is not installed, see LINUX_SETUP.md)
|
||||
|
||||
2. Test the configuration:
|
||||
cd {install_dir}
|
||||
./bin/lbry-test
|
||||
|
||||
Or run directly:
|
||||
source venv/bin/activate
|
||||
python main.py --dry-run
|
||||
|
||||
3. Run the downloader:
|
||||
./bin/lbry-downloader
|
||||
|
||||
Or:
|
||||
python main.py
|
||||
|
||||
Configuration file:
|
||||
{config_path}
|
||||
|
||||
To add more channels later, edit the config file:
|
||||
nano {config_path}
|
||||
|
||||
Documentation:
|
||||
- README.md - Project overview
|
||||
- LINUX_SETUP.md - Detailed Linux setup
|
||||
- QUICKSTART.md - Quick start guide
|
||||
- config.yaml.example - Configuration examples
|
||||
|
||||
Happy downloading!
|
||||
""")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
run_setup()
|
||||
except KeyboardInterrupt:
|
||||
print("\n\nSetup interrupted by user.")
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
print_error(f"Setup failed: {e}")
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
+186
@@ -0,0 +1,186 @@
|
||||
"""State database management with atomic writes."""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from models import StateDatabase, Channel, Claim
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class StateDbError(Exception):
|
||||
"""Raised when state database operations fail."""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class StateDb:
|
||||
"""Manages the persistent state database."""
|
||||
|
||||
def __init__(self, state_file: str):
|
||||
"""
|
||||
Initialize state database.
|
||||
|
||||
Args:
|
||||
state_file: Path to the state database JSON file.
|
||||
"""
|
||||
self.state_file = Path(state_file)
|
||||
self.data = StateDatabase()
|
||||
self._loaded = False
|
||||
|
||||
def load(self) -> StateDatabase:
|
||||
"""
|
||||
Load state from disk or initialize new state.
|
||||
|
||||
Returns:
|
||||
Loaded StateDatabase instance.
|
||||
"""
|
||||
if self.state_file.exists():
|
||||
try:
|
||||
with open(self.state_file, "r") as f:
|
||||
raw_data = json.load(f)
|
||||
|
||||
self.data = StateDatabase.from_dict(raw_data)
|
||||
self._loaded = True
|
||||
logger.debug(f"Loaded state from {self.state_file}")
|
||||
|
||||
# Validate schema version
|
||||
if self.data.schema_version != 1:
|
||||
logger.warning(
|
||||
f"State database schema version is {self.data.schema_version}, "
|
||||
f"expected 1. You may need to migrate."
|
||||
)
|
||||
|
||||
except json.JSONDecodeError as e:
|
||||
logger.error(f"Failed to parse state database: {e}")
|
||||
raise StateDbError(f"State database is corrupted: {e}")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load state database: {e}")
|
||||
raise StateDbError(f"Failed to load state database: {e}")
|
||||
else:
|
||||
# Initialize new state
|
||||
self.data = StateDatabase()
|
||||
self._loaded = True
|
||||
logger.debug(f"Initialized new state database at {self.state_file}")
|
||||
|
||||
return self.data
|
||||
|
||||
def save(self) -> None:
|
||||
"""
|
||||
Save state to disk atomically.
|
||||
|
||||
Writes to a temporary file first, then renames to ensure
|
||||
the database is never in a partially written state.
|
||||
"""
|
||||
if not self._loaded:
|
||||
raise StateDbError("Cannot save state before loading")
|
||||
|
||||
# Ensure parent directory exists
|
||||
self.state_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Write to temporary file
|
||||
temp_file = self.state_file.with_suffix(".tmp")
|
||||
|
||||
try:
|
||||
with open(temp_file, "w") as f:
|
||||
json.dump(self.data.to_dict(), f, indent=2, default=str)
|
||||
f.flush()
|
||||
os.fsync(f.fileno())
|
||||
|
||||
# Atomic rename
|
||||
shutil.move(str(temp_file), str(self.state_file))
|
||||
|
||||
logger.debug(f"Saved state to {self.state_file}")
|
||||
|
||||
except Exception as e:
|
||||
# Clean up temp file on failure
|
||||
if temp_file.exists():
|
||||
try:
|
||||
temp_file.unlink()
|
||||
except:
|
||||
pass
|
||||
raise StateDbError(f"Failed to save state database: {e}")
|
||||
|
||||
def get_channel(self, channel_claim_id: str) -> Optional[Channel]:
|
||||
"""Get a channel by its claim ID."""
|
||||
return self.data.channels.get(channel_claim_id)
|
||||
|
||||
def set_channel(self, channel: Channel) -> None:
|
||||
"""Store or update a channel record."""
|
||||
self.data.channels[channel.channel_claim_id] = channel
|
||||
|
||||
def get_claim(self, claim_id: str) -> Optional[Claim]:
|
||||
"""Get a claim by its claim ID."""
|
||||
return self.data.claims.get(claim_id)
|
||||
|
||||
def set_claim(self, claim: Claim) -> None:
|
||||
"""Store or update a claim record."""
|
||||
self.data.claims[claim.claim_id] = claim
|
||||
|
||||
def update_last_run(self) -> None:
|
||||
"""Update the last run timestamp."""
|
||||
self.data.last_run = datetime.now(timezone.utc).isoformat()
|
||||
|
||||
def log_run_history(
|
||||
self,
|
||||
run_history_file: str,
|
||||
channel_claim_id: str,
|
||||
claim_id: str,
|
||||
action: str,
|
||||
version_token: str,
|
||||
status: str,
|
||||
message: Optional[str] = None,
|
||||
) -> None:
|
||||
"""
|
||||
Append an entry to the run history log.
|
||||
|
||||
Args:
|
||||
run_history_file: Path to the run history JSONL file.
|
||||
channel_claim_id: Channel claim ID.
|
||||
claim_id: Claim ID.
|
||||
action: Action taken (e.g., 'download_new', 'skip_existing').
|
||||
version_token: Version token.
|
||||
status: Status (e.g., 'success', 'failure').
|
||||
message: Optional message.
|
||||
"""
|
||||
entry = {
|
||||
"ts": datetime.now(timezone.utc).isoformat(),
|
||||
"channel_claim_id": channel_claim_id,
|
||||
"claim_id": claim_id,
|
||||
"action": action,
|
||||
"version_token": version_token,
|
||||
"status": status,
|
||||
}
|
||||
|
||||
if message:
|
||||
entry["message"] = message
|
||||
|
||||
history_path = Path(run_history_file)
|
||||
history_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
with open(history_path, "a") as f:
|
||||
json.dump(entry, f)
|
||||
f.write("\n")
|
||||
|
||||
def get_stats(self) -> Dict[str, int]:
|
||||
"""Get database statistics."""
|
||||
total_versions = sum(len(claim.versions) for claim in self.data.claims.values())
|
||||
downloaded_versions = sum(
|
||||
1
|
||||
for claim in self.data.claims.values()
|
||||
for version in claim.versions.values()
|
||||
if version.downloaded
|
||||
)
|
||||
|
||||
return {
|
||||
"channels": len(self.data.channels),
|
||||
"claims": len(self.data.claims),
|
||||
"total_versions": total_versions,
|
||||
"downloaded_versions": downloaded_versions,
|
||||
}
|
||||
Whitespace-only changes.
@@ -0,0 +1,137 @@
|
||||
"""Tests for configuration loading."""
|
||||
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import os
|
||||
|
||||
from config_loader import load_config, ConfigError, create_default_config
|
||||
from models import Config
|
||||
|
||||
|
||||
class TestLoadConfig:
|
||||
def test_missing_config_file(self):
|
||||
with pytest.raises(ConfigError) as exc_info:
|
||||
load_config("/nonexistent/path/config.yaml")
|
||||
|
||||
assert "not found" in str(exc_info.value)
|
||||
|
||||
def test_empty_channels_list(self, tmp_path):
|
||||
config_file = tmp_path / "config.yaml"
|
||||
config_file.write_text("""
|
||||
lbrynet:
|
||||
api_url: "http://127.0.0.1:5279"
|
||||
|
||||
general:
|
||||
base_dir: "~/test"
|
||||
|
||||
channels: []
|
||||
""")
|
||||
|
||||
with pytest.raises(ConfigError) as exc_info:
|
||||
load_config(str(config_file))
|
||||
|
||||
assert "at least one channel" in str(exc_info.value)
|
||||
|
||||
def test_all_channels_disabled(self, tmp_path):
|
||||
config_file = tmp_path / "config.yaml"
|
||||
config_file.write_text("""
|
||||
lbrynet:
|
||||
api_url: "http://127.0.0.1:5279"
|
||||
|
||||
general:
|
||||
base_dir: "~/test"
|
||||
|
||||
channels:
|
||||
- input: "https://odysee.com/@Test:1"
|
||||
enabled: false
|
||||
""")
|
||||
|
||||
with pytest.raises(ConfigError) as exc_info:
|
||||
load_config(str(config_file))
|
||||
|
||||
assert "at least one channel" in str(exc_info.value).lower()
|
||||
|
||||
def test_valid_config(self, tmp_path):
|
||||
config_file = tmp_path / "config.yaml"
|
||||
config_file.write_text("""
|
||||
lbrynet:
|
||||
api_url: "http://127.0.0.1:5279"
|
||||
timeout_seconds: 120
|
||||
|
||||
general:
|
||||
base_dir: "~/Documents/lbry-downloads"
|
||||
max_workers: 3
|
||||
log_level: "DEBUG"
|
||||
|
||||
channels:
|
||||
- input: "https://odysee.com/@Test:1"
|
||||
enabled: true
|
||||
tags_include: ["tech"]
|
||||
tags_exclude: ["spam"]
|
||||
""")
|
||||
|
||||
config = load_config(str(config_file))
|
||||
|
||||
assert isinstance(config, Config)
|
||||
assert config.lbrynet.api_url == "http://127.0.0.1:5279"
|
||||
assert config.lbrynet.timeout_seconds == 120
|
||||
assert config.general.max_workers == 3
|
||||
assert config.general.log_level == "DEBUG"
|
||||
assert len(config.channels) == 1
|
||||
assert config.channels[0].input == "https://odysee.com/@Test:1"
|
||||
assert config.channels[0].tags_include == ["tech"]
|
||||
|
||||
def test_invalid_log_level(self, tmp_path):
|
||||
config_file = tmp_path / "config.yaml"
|
||||
config_file.write_text("""
|
||||
lbrynet:
|
||||
api_url: "http://127.0.0.1:5279"
|
||||
|
||||
general:
|
||||
base_dir: "~/test"
|
||||
log_level: "INVALID"
|
||||
|
||||
channels:
|
||||
- input: "https://odysee.com/@Test:1"
|
||||
enabled: true
|
||||
""")
|
||||
|
||||
with pytest.raises(ConfigError) as exc_info:
|
||||
load_config(str(config_file))
|
||||
|
||||
assert "log_level" in str(exc_info.value)
|
||||
|
||||
def test_invalid_filename_mode(self, tmp_path):
|
||||
config_file = tmp_path / "config.yaml"
|
||||
config_file.write_text("""
|
||||
lbrynet:
|
||||
api_url: "http://127.0.0.1:5279"
|
||||
|
||||
general:
|
||||
base_dir: "~/test"
|
||||
filename_mode: "invalid"
|
||||
|
||||
channels:
|
||||
- input: "https://odysee.com/@Test:1"
|
||||
enabled: true
|
||||
""")
|
||||
|
||||
with pytest.raises(ConfigError) as exc_info:
|
||||
load_config(str(config_file))
|
||||
|
||||
assert "filename_mode" in str(exc_info.value)
|
||||
|
||||
|
||||
class TestDefaultConfig:
|
||||
def test_create_default_config(self):
|
||||
config = create_default_config()
|
||||
|
||||
assert "lbrynet:" in config
|
||||
assert "general:" in config
|
||||
assert "channels:" in config
|
||||
assert "http://127.0.0.1:5279" in config
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
@@ -0,0 +1,50 @@
|
||||
"""Tests for download limit feature."""
|
||||
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import shutil
|
||||
|
||||
from models import Channel, Config, GeneralConfig, StateDatabase, DownloadAction
|
||||
|
||||
|
||||
class TestDownloadLimit:
|
||||
"""Test the download limit functionality."""
|
||||
|
||||
def test_download_limit_zero_means_all(self):
|
||||
"""Test that download_limit=0 means download all (no limit)."""
|
||||
config = Config()
|
||||
config.general = GeneralConfig()
|
||||
config.general.download_limit = 0
|
||||
|
||||
# download_limit of 0 should mean no limit
|
||||
assert config.general.download_limit == 0
|
||||
|
||||
def test_download_limit_default_is_10(self):
|
||||
"""Test that default download_limit is 10."""
|
||||
config = Config()
|
||||
config.general = GeneralConfig()
|
||||
|
||||
# Default should be 10
|
||||
assert config.general.download_limit == 10
|
||||
|
||||
def test_negative_download_limit_invalid(self):
|
||||
"""Test that negative download_limit is invalid."""
|
||||
from config_loader import ConfigError, _parse_config
|
||||
|
||||
raw_config = {
|
||||
"lbrynet": {},
|
||||
"general": {
|
||||
"download_limit": -5 # Invalid
|
||||
},
|
||||
"channels": [{"input": "https://odysee.com/@Test:1", "enabled": True}],
|
||||
}
|
||||
|
||||
with pytest.raises(ConfigError) as exc_info:
|
||||
_parse_config(raw_config)
|
||||
|
||||
assert "download_limit" in str(exc_info.value).lower()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
@@ -0,0 +1,211 @@
|
||||
"""Tests for planner module."""
|
||||
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import shutil
|
||||
|
||||
from planner import Planner
|
||||
from models import Channel, Config, GeneralConfig, StateDatabase, Claim, ClaimVersion
|
||||
from utils import create_channel_folder_name, create_claim_folder_name
|
||||
|
||||
|
||||
class TestPlanner:
|
||||
def setup_method(self):
|
||||
self.temp_dir = tempfile.mkdtemp()
|
||||
self.config = Config()
|
||||
self.config.general = GeneralConfig()
|
||||
self.config.general.verify_existing_files = True
|
||||
self.state = StateDatabase()
|
||||
self.planner = Planner(self.config, self.state, self.temp_dir)
|
||||
|
||||
def teardown_method(self):
|
||||
shutil.rmtree(self.temp_dir, ignore_errors=True)
|
||||
|
||||
def test_new_claim_action(self):
|
||||
channel = Channel(
|
||||
input="test",
|
||||
normalized_uri="lbry://@Test#1",
|
||||
channel_claim_id="channel123",
|
||||
channel_name="@Test",
|
||||
folder="Test__channel123",
|
||||
)
|
||||
|
||||
claim_data = {
|
||||
"claim_id": "claim456",
|
||||
"name": "my-video",
|
||||
"value_type": "stream",
|
||||
"value": {"source": {"sd_hash": "abc123"}, "title": "My Video"},
|
||||
"permanent_url": "lbry://@Test#1/my-video#claim456",
|
||||
"txid": "tx789",
|
||||
"nout": 0,
|
||||
}
|
||||
|
||||
actions = self.planner.process_channel_claims(channel, [claim_data])
|
||||
|
||||
assert len(actions) == 1
|
||||
assert actions[0].action == "download_new"
|
||||
assert actions[0].claim_id == "claim456"
|
||||
|
||||
def test_skip_existing_claim(self):
|
||||
channel = Channel(
|
||||
input="test",
|
||||
normalized_uri="lbry://@Test#1",
|
||||
channel_claim_id="channel123",
|
||||
channel_name="@Test",
|
||||
folder="Test__channel123",
|
||||
)
|
||||
|
||||
# Add existing claim to state
|
||||
claim_folder = create_claim_folder_name("my-video", "claim456")
|
||||
claim = Claim(
|
||||
claim_id="claim456",
|
||||
channel_claim_id="channel123",
|
||||
name="my-video",
|
||||
permanent_url="lbry://@Test#1/my-video#claim456",
|
||||
claim_folder=f"channels/Test__channel123/claims/{claim_folder}",
|
||||
)
|
||||
|
||||
# Create version token matching the claim data
|
||||
from utils import generate_version_token
|
||||
|
||||
version_token = generate_version_token(sd_hash="abc123")
|
||||
|
||||
claim.versions[version_token] = ClaimVersion(
|
||||
version_token=version_token,
|
||||
txid="tx789",
|
||||
nout=0,
|
||||
sd_hash="abc123",
|
||||
downloaded=True,
|
||||
file_relpath=f"channels/Test__channel123/claims/{claim_folder}/versions/{version_token}/video.mp4",
|
||||
)
|
||||
|
||||
self.state.claims["claim456"] = claim
|
||||
|
||||
# Create the "downloaded" file
|
||||
version_dir = (
|
||||
Path(self.temp_dir)
|
||||
/ f"channels/Test__channel123/claims/{claim_folder}/versions/{version_token}"
|
||||
)
|
||||
version_dir.mkdir(parents=True)
|
||||
(version_dir / "video.mp4").touch()
|
||||
|
||||
claim_data = {
|
||||
"claim_id": "claim456",
|
||||
"name": "my-video",
|
||||
"value_type": "stream",
|
||||
"value": {"source": {"sd_hash": "abc123"}, "title": "My Video"},
|
||||
"permanent_url": "lbry://@Test#1/my-video#claim456",
|
||||
"txid": "tx789",
|
||||
"nout": 0,
|
||||
}
|
||||
|
||||
actions = self.planner.process_channel_claims(channel, [claim_data])
|
||||
|
||||
assert len(actions) == 1
|
||||
assert actions[0].action == "skip_existing"
|
||||
|
||||
def test_new_version_detected(self):
|
||||
channel = Channel(
|
||||
input="test",
|
||||
normalized_uri="lbry://@Test#1",
|
||||
channel_claim_id="channel123",
|
||||
channel_name="@Test",
|
||||
folder="Test__channel123",
|
||||
)
|
||||
|
||||
# Add existing claim with old version
|
||||
claim_folder = create_claim_folder_name("my-video", "claim456")
|
||||
claim = Claim(
|
||||
claim_id="claim456",
|
||||
channel_claim_id="channel123",
|
||||
name="my-video",
|
||||
permanent_url="lbry://@Test#1/my-video#claim456",
|
||||
claim_folder=f"channels/Test__channel123/claims/{claim_folder}",
|
||||
)
|
||||
|
||||
old_version_token = "sd_oldhash123"
|
||||
claim.versions[old_version_token] = ClaimVersion(
|
||||
version_token=old_version_token,
|
||||
txid="tx789",
|
||||
nout=0,
|
||||
sd_hash="oldhash123",
|
||||
downloaded=True,
|
||||
)
|
||||
|
||||
self.state.claims["claim456"] = claim
|
||||
|
||||
# New claim data with different sd_hash
|
||||
claim_data = {
|
||||
"claim_id": "claim456",
|
||||
"name": "my-video",
|
||||
"value_type": "stream",
|
||||
"value": {
|
||||
"source": {"sd_hash": "newhash456"}, # Different!
|
||||
"title": "My Video Updated",
|
||||
},
|
||||
"permanent_url": "lbry://@Test#1/my-video#claim456",
|
||||
"txid": "tx999",
|
||||
"nout": 0,
|
||||
}
|
||||
|
||||
actions = self.planner.process_channel_claims(channel, [claim_data])
|
||||
|
||||
assert len(actions) == 1
|
||||
assert actions[0].action == "download_new_version"
|
||||
|
||||
def test_non_downloadable_claim_skipped(self):
|
||||
channel = Channel(
|
||||
input="test",
|
||||
normalized_uri="lbry://@Test#1",
|
||||
channel_claim_id="channel123",
|
||||
channel_name="@Test",
|
||||
folder="Test__channel123",
|
||||
)
|
||||
|
||||
# Channel type claim (not downloadable)
|
||||
claim_data = {
|
||||
"claim_id": "claim456",
|
||||
"name": "@SomeChannel",
|
||||
"value_type": "channel",
|
||||
"value": {},
|
||||
"permanent_url": "lbry://@SomeChannel#claim456",
|
||||
}
|
||||
|
||||
actions = self.planner.process_channel_claims(channel, [claim_data])
|
||||
|
||||
assert len(actions) == 0
|
||||
|
||||
def test_custom_download_path(self):
|
||||
"""Test that custom download_path is used for channel."""
|
||||
custom_path = "/custom/download/location"
|
||||
channel = Channel(
|
||||
input="test",
|
||||
normalized_uri="lbry://@Test#1",
|
||||
channel_claim_id="channel123",
|
||||
channel_name="@Test",
|
||||
folder="Test__channel123",
|
||||
download_path=custom_path,
|
||||
)
|
||||
|
||||
claim_data = {
|
||||
"claim_id": "claim456",
|
||||
"name": "my-video",
|
||||
"value_type": "stream",
|
||||
"value": {"source": {"sd_hash": "abc123"}, "title": "My Video"},
|
||||
"permanent_url": "lbry://@Test#1/my-video#claim456",
|
||||
"txid": "tx789",
|
||||
"nout": 0,
|
||||
}
|
||||
|
||||
actions = self.planner.process_channel_claims(channel, [claim_data])
|
||||
|
||||
assert len(actions) == 1
|
||||
assert actions[0].action == "download_new"
|
||||
# Check that target_dir uses custom path
|
||||
assert custom_path in actions[0].target_dir
|
||||
assert "Test__channel123" in actions[0].target_dir
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
@@ -0,0 +1,135 @@
|
||||
"""Tests for state database."""
|
||||
|
||||
import json
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import shutil
|
||||
|
||||
from state_db import StateDb
|
||||
from models import Channel, Claim, ClaimVersion
|
||||
|
||||
|
||||
class TestStateDb:
|
||||
def setup_method(self):
|
||||
self.temp_dir = tempfile.mkdtemp()
|
||||
self.state_file = Path(self.temp_dir) / "test_state.json"
|
||||
|
||||
def teardown_method(self):
|
||||
shutil.rmtree(self.temp_dir, ignore_errors=True)
|
||||
|
||||
def test_load_new_database(self):
|
||||
db = StateDb(str(self.state_file))
|
||||
state = db.load()
|
||||
|
||||
assert state.schema_version == 1
|
||||
assert state.channels == {}
|
||||
assert state.claims == {}
|
||||
|
||||
def test_save_and_load(self):
|
||||
db = StateDb(str(self.state_file))
|
||||
db.load()
|
||||
|
||||
# Add a channel
|
||||
channel = Channel(
|
||||
input="https://odysee.com/@Test:1",
|
||||
normalized_uri="lbry://@Test#1",
|
||||
channel_claim_id="abc123",
|
||||
channel_name="@Test",
|
||||
folder="Test__abc123",
|
||||
)
|
||||
db.set_channel(channel)
|
||||
db.save()
|
||||
|
||||
# Load fresh
|
||||
db2 = StateDb(str(self.state_file))
|
||||
state = db2.load()
|
||||
|
||||
assert "abc123" in state.channels
|
||||
assert state.channels["abc123"].channel_name == "@Test"
|
||||
|
||||
def test_channel_operations(self):
|
||||
db = StateDb(str(self.state_file))
|
||||
db.load()
|
||||
|
||||
channel = Channel(
|
||||
input="test",
|
||||
normalized_uri="lbry://@Test#1",
|
||||
channel_claim_id="xyz789",
|
||||
channel_name="@Test",
|
||||
folder="Test__xyz789",
|
||||
)
|
||||
|
||||
db.set_channel(channel)
|
||||
retrieved = db.get_channel("xyz789")
|
||||
|
||||
assert retrieved is not None
|
||||
assert retrieved.channel_name == "@Test"
|
||||
assert db.get_channel("nonexistent") is None
|
||||
|
||||
def test_claim_operations(self):
|
||||
db = StateDb(str(self.state_file))
|
||||
db.load()
|
||||
|
||||
claim = Claim(
|
||||
claim_id="claim123",
|
||||
channel_claim_id="channel456",
|
||||
name="my-video",
|
||||
permanent_url="lbry://@Test#1/my-video#claim123",
|
||||
claim_folder="channels/Test__channel456/claims/my-video__claim123",
|
||||
)
|
||||
|
||||
db.set_claim(claim)
|
||||
retrieved = db.get_claim("claim123")
|
||||
|
||||
assert retrieved is not None
|
||||
assert retrieved.name == "my-video"
|
||||
|
||||
def test_run_history_logging(self):
|
||||
db = StateDb(str(self.state_file))
|
||||
db.load()
|
||||
|
||||
history_file = Path(self.temp_dir) / "history.jsonl"
|
||||
|
||||
db.log_run_history(
|
||||
str(history_file),
|
||||
"channel123",
|
||||
"claim456",
|
||||
"download_new",
|
||||
"sd_abc123",
|
||||
"success",
|
||||
)
|
||||
|
||||
assert history_file.exists()
|
||||
|
||||
with open(history_file) as f:
|
||||
line = json.loads(f.readline())
|
||||
assert line["action"] == "download_new"
|
||||
assert line["status"] == "success"
|
||||
|
||||
def test_get_stats(self):
|
||||
db = StateDb(str(self.state_file))
|
||||
db.load()
|
||||
|
||||
# Add test data
|
||||
claim = Claim(
|
||||
claim_id="c1",
|
||||
channel_claim_id="ch1",
|
||||
name="video1",
|
||||
permanent_url="lbry://test",
|
||||
claim_folder="test",
|
||||
)
|
||||
claim.versions["v1"] = ClaimVersion(
|
||||
version_token="v1", txid="tx1", nout=0, downloaded=True
|
||||
)
|
||||
|
||||
db.set_claim(claim)
|
||||
|
||||
stats = db.get_stats()
|
||||
assert stats["claims"] == 1
|
||||
assert stats["total_versions"] == 1
|
||||
assert stats["downloaded_versions"] == 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
@@ -0,0 +1,115 @@
|
||||
"""Tests for utility functions."""
|
||||
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
|
||||
from utils import (
|
||||
sanitize_filename,
|
||||
slugify,
|
||||
generate_version_token,
|
||||
expand_path,
|
||||
is_downloadable_claim,
|
||||
normalize_odysee_url,
|
||||
)
|
||||
|
||||
|
||||
class TestSanitizeFilename:
|
||||
def test_basic_sanitization(self):
|
||||
assert sanitize_filename("hello/world") == "hello_world"
|
||||
assert sanitize_filename("file:name") == "file_name"
|
||||
assert sanitize_filename('file"name') == "file_name"
|
||||
|
||||
def test_whitespace_collapse(self):
|
||||
assert sanitize_filename("hello world") == "hello world"
|
||||
|
||||
def test_trailing_cleanup(self):
|
||||
assert sanitize_filename("hello.") == "hello"
|
||||
assert sanitize_filename("hello ") == "hello"
|
||||
|
||||
def test_length_limit(self):
|
||||
long_name = "a" * 200
|
||||
result = sanitize_filename(long_name)
|
||||
assert len(result) <= 120
|
||||
|
||||
def test_extension_preservation(self):
|
||||
result = sanitize_filename("my/file:name.txt")
|
||||
assert result.endswith(".txt")
|
||||
|
||||
|
||||
class TestSlugify:
|
||||
def test_basic_slug(self):
|
||||
assert slugify("Hello World") == "hello-world"
|
||||
assert slugify("Test!!!") == "test"
|
||||
|
||||
def test_special_chars_removed(self):
|
||||
assert slugify("hello@world#123") == "helloworld123"
|
||||
|
||||
def test_multiple_dashes_collapsed(self):
|
||||
assert slugify("hello---world") == "hello-world"
|
||||
|
||||
|
||||
class TestGenerateVersionToken:
|
||||
def test_sd_hash_priority(self):
|
||||
token = generate_version_token(
|
||||
sd_hash="abc123", stream_hash="def456", txid="ghi789", nout=0
|
||||
)
|
||||
assert token.startswith("sd_")
|
||||
|
||||
def test_stream_hash_fallback(self):
|
||||
token = generate_version_token(stream_hash="def456", txid="ghi789", nout=0)
|
||||
assert token.startswith("stream_")
|
||||
|
||||
def test_txid_fallback(self):
|
||||
token = generate_version_token(txid="ghi789", nout=5)
|
||||
assert token.startswith("tx_")
|
||||
|
||||
def test_metadata_fallback(self):
|
||||
token = generate_version_token(metadata={"name": "test", "id": 123})
|
||||
assert token.startswith("meta_")
|
||||
|
||||
|
||||
class TestExpandPath:
|
||||
def test_home_expansion(self):
|
||||
path = expand_path("~/test")
|
||||
assert not str(path).startswith("~")
|
||||
assert path.is_absolute()
|
||||
|
||||
|
||||
class TestIsDownloadableClaim:
|
||||
def test_stream_claim_with_source(self):
|
||||
claim = {"value_type": "stream", "value": {"source": {"sd_hash": "abc123"}}}
|
||||
assert is_downloadable_claim(claim) is True
|
||||
|
||||
def test_stream_claim_with_stream_hash(self):
|
||||
claim = {"value_type": "stream", "value": {}, "stream_hash": "abc123"}
|
||||
assert is_downloadable_claim(claim) is True
|
||||
|
||||
def test_non_stream_claim(self):
|
||||
claim = {"value_type": "channel"}
|
||||
assert is_downloadable_claim(claim) is False
|
||||
|
||||
def test_stream_without_source(self):
|
||||
claim = {"value_type": "stream", "value": {}}
|
||||
assert is_downloadable_claim(claim) is False
|
||||
|
||||
|
||||
class TestNormalizeOdyseeUrl:
|
||||
def test_https_odysee(self):
|
||||
result = normalize_odysee_url("https://odysee.com/@Channel:1")
|
||||
assert result == "lbry://@Channel:1"
|
||||
|
||||
def test_http_odysee(self):
|
||||
result = normalize_odysee_url("http://odysee.com/@Channel:1")
|
||||
assert result == "lbry://@Channel:1"
|
||||
|
||||
def test_already_lbry(self):
|
||||
result = normalize_odysee_url("lbry://@Channel:1")
|
||||
assert result == "lbry://@Channel:1"
|
||||
|
||||
def test_short_form(self):
|
||||
result = normalize_odysee_url("@Channel:1")
|
||||
assert result == "lbry://@Channel:1"
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
Executable
+254
@@ -0,0 +1,254 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
LBRY Downloader - Uninstall Script
|
||||
|
||||
This script removes the LBRY Downloader installation.
|
||||
It will ask for confirmation before deleting any files.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class Colors:
|
||||
GREEN = "\033[92m"
|
||||
YELLOW = "\033[93m"
|
||||
RED = "\033[91m"
|
||||
BLUE = "\033[94m"
|
||||
BOLD = "\033[1m"
|
||||
END = "\033[0m"
|
||||
|
||||
|
||||
def print_success(msg: str):
|
||||
print(f"{Colors.GREEN}✓{Colors.END} {msg}")
|
||||
|
||||
|
||||
def print_warning(msg: str):
|
||||
print(f"{Colors.YELLOW}⚠{Colors.END} {msg}")
|
||||
|
||||
|
||||
def print_error(msg: str):
|
||||
print(f"{Colors.RED}✗{Colors.END} {msg}")
|
||||
|
||||
|
||||
def print_info(msg: str):
|
||||
print(f"{Colors.BLUE}ℹ{Colors.END} {msg}")
|
||||
|
||||
|
||||
def print_header(msg: str):
|
||||
print(f"\n{Colors.BOLD}{msg}{Colors.END}")
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
def find_installation() -> Path:
|
||||
"""Try to find the installation directory."""
|
||||
# Check common locations
|
||||
possible_locations = [
|
||||
Path.home() / "Documents" / "LBRY-Downloader",
|
||||
Path.home() / "LBRY-Downloader",
|
||||
Path("/opt/LBRY-Downloader"),
|
||||
Path("/usr/local/LBRY-Downloader"),
|
||||
]
|
||||
|
||||
# Check current directory
|
||||
current = Path.cwd()
|
||||
if (current / "main.py").exists() and (current / "requirements.txt").exists():
|
||||
possible_locations.insert(0, current)
|
||||
|
||||
# Check if any of these exist
|
||||
for location in possible_locations:
|
||||
if location.exists() and (location / "main.py").exists():
|
||||
return location
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def get_download_location() -> Path:
|
||||
"""Try to find the download location from config."""
|
||||
config_paths = [
|
||||
Path.home() / "Documents" / "lbry-downloads" / "config.yaml",
|
||||
Path.home() / "lbry-downloads" / "config.yaml",
|
||||
]
|
||||
|
||||
for config_path in config_paths:
|
||||
if config_path.exists():
|
||||
try:
|
||||
with open(config_path) as f:
|
||||
content = f.read()
|
||||
|
||||
# Simple parsing to find base_dir
|
||||
for line in content.split("\n"):
|
||||
if "base_dir:" in line:
|
||||
path = line.split(":", 1)[1].strip().strip('"').strip("'")
|
||||
if path.startswith("~"):
|
||||
path = path.replace("~", str(Path.home()))
|
||||
return Path(path).resolve()
|
||||
except:
|
||||
pass
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def remove_directory(path: Path, description: str) -> bool:
|
||||
"""Remove a directory with confirmation."""
|
||||
if not path.exists():
|
||||
print_info(f"{description} not found at {path}")
|
||||
return False
|
||||
|
||||
print_warning(f"Found {description}: {path}")
|
||||
|
||||
# Show size if it's a directory
|
||||
if path.is_dir():
|
||||
try:
|
||||
total_size = sum(f.stat().st_size for f in path.rglob("*") if f.is_file())
|
||||
size_mb = total_size / (1024 * 1024)
|
||||
print_info(f"Size: {size_mb:.1f} MB")
|
||||
except:
|
||||
pass
|
||||
|
||||
response = input(f"Remove {description}? (y/N): ").strip().lower()
|
||||
|
||||
if response == "y":
|
||||
try:
|
||||
if path.is_dir():
|
||||
shutil.rmtree(path)
|
||||
else:
|
||||
path.unlink()
|
||||
print_success(f"Removed {description}")
|
||||
return True
|
||||
except Exception as e:
|
||||
print_error(f"Failed to remove {description}: {e}")
|
||||
return False
|
||||
else:
|
||||
print_info(f"Skipped {description}")
|
||||
return False
|
||||
|
||||
|
||||
def remove_from_path(install_dir: Path) -> None:
|
||||
"""Remove launcher from PATH if it was added."""
|
||||
print_header("Checking PATH Configuration")
|
||||
|
||||
bin_dir = install_dir / "bin"
|
||||
if not bin_dir.exists():
|
||||
return
|
||||
|
||||
shell_rc_files = [
|
||||
Path.home() / ".bashrc",
|
||||
Path.home() / ".zshrc",
|
||||
Path.home() / ".bash_profile",
|
||||
]
|
||||
|
||||
for rc_file in shell_rc_files:
|
||||
if not rc_file.exists():
|
||||
continue
|
||||
|
||||
try:
|
||||
content = rc_file.read_text()
|
||||
bin_str = str(bin_dir)
|
||||
|
||||
if bin_str in content:
|
||||
print_warning(f"Found PATH modification in {rc_file}")
|
||||
print_info("You should manually remove this line from the file:")
|
||||
for i, line in enumerate(content.split("\n"), 1):
|
||||
if bin_str in line and "PATH" in line:
|
||||
print(f" Line {i}: {line}")
|
||||
except:
|
||||
pass
|
||||
|
||||
|
||||
def main():
|
||||
print(f"""
|
||||
{Colors.BOLD}╔══════════════════════════════════════════════════════════════╗
|
||||
║ LBRY Downloader - Uninstall Script ║
|
||||
╚══════════════════════════════════════════════════════════════╝{Colors.END}
|
||||
|
||||
This script will help you uninstall LBRY Downloader.
|
||||
""")
|
||||
|
||||
# Find installation
|
||||
install_dir = find_installation()
|
||||
|
||||
if install_dir:
|
||||
print_info(f"Found installation at: {install_dir}")
|
||||
else:
|
||||
print_warning("Could not automatically find installation")
|
||||
response = input(
|
||||
"Enter installation directory (or press Enter to skip): "
|
||||
).strip()
|
||||
if response:
|
||||
install_dir = Path(response).expanduser().resolve()
|
||||
|
||||
# Find download location
|
||||
download_dir = get_download_location()
|
||||
|
||||
if download_dir:
|
||||
print_info(f"Found download location at: {download_dir}")
|
||||
else:
|
||||
default_dl = Path.home() / "Documents" / "lbry-downloads"
|
||||
if default_dl.exists():
|
||||
download_dir = default_dl
|
||||
print_info(f"Found download location at: {download_dir}")
|
||||
|
||||
# Show summary and ask for confirmation
|
||||
print_header("Uninstall Summary")
|
||||
|
||||
items_to_remove = []
|
||||
|
||||
if install_dir and install_dir.exists():
|
||||
items_to_remove.append(("Installation", install_dir))
|
||||
|
||||
if download_dir and download_dir.exists():
|
||||
items_to_remove.append(("Downloads", download_dir))
|
||||
|
||||
if not items_to_remove:
|
||||
print_info("Nothing found to uninstall")
|
||||
return
|
||||
|
||||
print("The following items will be removed:")
|
||||
for name, path in items_to_remove:
|
||||
print(f" - {name}: {path}")
|
||||
|
||||
print()
|
||||
print_warning("This action cannot be undone!")
|
||||
response = input("\nProceed with uninstallation? (yes/no): ").strip().lower()
|
||||
|
||||
if response != "yes":
|
||||
print_info("Uninstallation cancelled")
|
||||
return
|
||||
|
||||
# Remove items
|
||||
print_header("Removing Files")
|
||||
|
||||
for name, path in items_to_remove:
|
||||
remove_directory(path, name)
|
||||
|
||||
# Check PATH
|
||||
if install_dir:
|
||||
remove_from_path(install_dir)
|
||||
|
||||
# Summary
|
||||
print_header("Uninstallation Complete")
|
||||
print_success("LBRY Downloader has been removed from your system")
|
||||
|
||||
print("\nOptional cleanup:")
|
||||
print(" - Remove lbrynet if you no longer need it")
|
||||
print(" - Remove any cron jobs you may have set up")
|
||||
print(" - Remove PATH modifications from your shell config")
|
||||
|
||||
print("\nThank you for trying LBRY Downloader!")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except KeyboardInterrupt:
|
||||
print("\n\nUninstallation cancelled")
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
print_error(f"Uninstallation failed: {e}")
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
@@ -0,0 +1,303 @@
|
||||
"""Utility functions for LBRY Downloader."""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
|
||||
def sanitize_filename(name: str, max_length: int = 120) -> str:
|
||||
r"""
|
||||
Sanitize a string for use as a filesystem name.
|
||||
|
||||
Rules:
|
||||
- Replace /, \, :, *, ?, ", <, >, | with underscore
|
||||
- Collapse repeated whitespace
|
||||
- Trim trailing periods/spaces
|
||||
- Limit length
|
||||
"""
|
||||
# Replace invalid characters
|
||||
sanitized = re.sub(r'[\\/*?:"<>|]', "_", name)
|
||||
|
||||
# Collapse whitespace
|
||||
sanitized = re.sub(r"\s+", " ", sanitized)
|
||||
|
||||
# Trim trailing periods and spaces
|
||||
sanitized = sanitized.rstrip(". ")
|
||||
|
||||
# Limit length while preserving extension
|
||||
if len(sanitized) > max_length:
|
||||
# Try to preserve extension
|
||||
parts = sanitized.rsplit(".", 1)
|
||||
if len(parts) == 2 and len(parts[1]) <= 10:
|
||||
name_part, ext = parts
|
||||
available = max_length - len(ext) - 1
|
||||
sanitized = name_part[:available] + "." + ext
|
||||
else:
|
||||
sanitized = sanitized[:max_length]
|
||||
|
||||
return sanitized
|
||||
|
||||
|
||||
def slugify(text: str) -> str:
|
||||
"""
|
||||
Create a URL-safe slug from text.
|
||||
Lowercase, replace spaces with dashes, remove special chars.
|
||||
"""
|
||||
# Lowercase
|
||||
text = text.lower()
|
||||
|
||||
# Replace spaces with dashes
|
||||
text = re.sub(r"\s+", "-", text)
|
||||
|
||||
# Remove non-alphanumeric except dashes
|
||||
text = re.sub(r"[^a-z0-9-]", "", text)
|
||||
|
||||
# Collapse multiple dashes
|
||||
text = re.sub(r"-+", "-", text)
|
||||
|
||||
# Trim dashes
|
||||
text = text.strip("-")
|
||||
|
||||
return text
|
||||
|
||||
|
||||
def generate_version_token(
|
||||
sd_hash: Optional[str] = None,
|
||||
stream_hash: Optional[str] = None,
|
||||
txid: Optional[str] = None,
|
||||
nout: Optional[int] = None,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Generate a deterministic version token for a claim version.
|
||||
|
||||
Priority order:
|
||||
1. sd_hash
|
||||
2. stream_hash
|
||||
3. txid:nout
|
||||
4. hash of normalized metadata JSON
|
||||
"""
|
||||
if sd_hash:
|
||||
return f"sd_{sd_hash[:16]}"
|
||||
|
||||
if stream_hash:
|
||||
return f"stream_{stream_hash[:16]}"
|
||||
|
||||
if txid is not None and nout is not None:
|
||||
return f"tx_{txid[:16]}_{nout}"
|
||||
|
||||
if metadata:
|
||||
# Sort keys for deterministic hash
|
||||
metadata_str = json.dumps(metadata, sort_keys=True)
|
||||
hash_val = hashlib.sha256(metadata_str.encode()).hexdigest()[:16]
|
||||
return f"meta_{hash_val}"
|
||||
|
||||
# Fallback to timestamp-based (should not happen in practice)
|
||||
ts = datetime.now(timezone.utc).isoformat()
|
||||
hash_val = hashlib.sha256(ts.encode()).hexdigest()[:16]
|
||||
return f"fallback_{hash_val}"
|
||||
|
||||
|
||||
def compute_sha256(filepath: Path) -> str:
|
||||
"""Compute SHA256 checksum of a file."""
|
||||
sha256_hash = hashlib.sha256()
|
||||
with open(filepath, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(8192), b""):
|
||||
sha256_hash.update(chunk)
|
||||
return sha256_hash.hexdigest()
|
||||
|
||||
|
||||
def expand_path(path: str) -> Path:
|
||||
"""Expand ~ and environment variables in a path."""
|
||||
import os
|
||||
|
||||
expanded = os.path.expandvars(path)
|
||||
return Path(expanded).expanduser().resolve()
|
||||
|
||||
|
||||
def format_timestamp(dt: Optional[datetime] = None) -> str:
|
||||
"""Format a timestamp in ISO format with timezone."""
|
||||
if dt is None:
|
||||
dt = datetime.now(timezone.utc)
|
||||
return dt.isoformat()
|
||||
|
||||
|
||||
def parse_timestamp(ts: str) -> datetime:
|
||||
"""Parse an ISO format timestamp."""
|
||||
return datetime.fromisoformat(ts)
|
||||
|
||||
|
||||
def create_channel_folder_name(channel_name: str, channel_claim_id: str) -> str:
|
||||
"""Create a folder name for a channel."""
|
||||
sanitized = sanitize_filename(channel_name.replace("@", ""))
|
||||
# Truncate channel name portion if needed
|
||||
max_name_len = 50
|
||||
if len(sanitized) > max_name_len:
|
||||
sanitized = sanitized[:max_name_len]
|
||||
return f"{sanitized}__{channel_claim_id}"
|
||||
|
||||
|
||||
def create_claim_folder_name(claim_name: str, claim_id: str) -> str:
|
||||
"""Create a folder name for a claim."""
|
||||
sanitized = sanitize_filename(claim_name)
|
||||
# Truncate claim name portion if needed
|
||||
max_name_len = 50
|
||||
if len(sanitized) > max_name_len:
|
||||
sanitized = sanitized[:max_name_len]
|
||||
return f"{sanitized}__{claim_id}"
|
||||
|
||||
|
||||
def is_downloadable_claim(claim_value: Dict[str, Any]) -> bool:
|
||||
"""
|
||||
Determine if a claim represents downloadable content.
|
||||
|
||||
Returns True if the claim has stream/source data that can be downloaded.
|
||||
"""
|
||||
value_type = claim_value.get("value_type", "").lower()
|
||||
|
||||
# Must be a stream type claim
|
||||
if value_type != "stream":
|
||||
return False
|
||||
|
||||
value = claim_value.get("value", {})
|
||||
|
||||
# Check for source information
|
||||
source = value.get("source", {})
|
||||
if source:
|
||||
# Has source info - likely downloadable
|
||||
sd_hash = source.get("sd_hash")
|
||||
if sd_hash:
|
||||
return True
|
||||
|
||||
# Check for stream hash as fallback
|
||||
stream_hash = claim_value.get("stream_hash")
|
||||
if stream_hash:
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def normalize_odysee_url(url: str) -> str:
|
||||
"""
|
||||
Convert an Odysee URL to a LBRY URI format.
|
||||
|
||||
Examples:
|
||||
- https://odysee.com/@SomeChannel:1 -> lbry://@SomeChannel#1
|
||||
- @SomeChannel:1 -> lbry://@SomeChannel#1
|
||||
"""
|
||||
# Remove https://odysee.com/ prefix if present
|
||||
if url.startswith("https://odysee.com/"):
|
||||
url = url[len("https://odysee.com/") :]
|
||||
elif url.startswith("http://odysee.com/"):
|
||||
url = url[len("http://odysee.com/") :]
|
||||
|
||||
# Ensure lbry:// prefix
|
||||
if not url.startswith("lbry://"):
|
||||
url = f"lbry://{url}"
|
||||
|
||||
return url
|
||||
|
||||
|
||||
def extract_channel_info(resolved_data: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""
|
||||
Extract normalized channel information from a resolve response.
|
||||
"""
|
||||
claim = resolved_data
|
||||
|
||||
# Handle wrapped response format
|
||||
if "claim" in resolved_data:
|
||||
claim = resolved_data["claim"]
|
||||
|
||||
value = claim.get("value", {})
|
||||
|
||||
return {
|
||||
"claim_id": claim.get("claim_id"),
|
||||
"name": claim.get("name"),
|
||||
"normalized_name": claim.get("normalized_name"),
|
||||
"permanent_url": claim.get("permanent_url"),
|
||||
"short_url": claim.get("short_url"),
|
||||
"canonical_url": claim.get("canonical_url"),
|
||||
"value_type": claim.get("value_type"),
|
||||
"title": value.get("title"),
|
||||
"description": value.get("description"),
|
||||
}
|
||||
|
||||
|
||||
def extract_claim_metadata(claim_data: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""
|
||||
Extract normalized metadata from a claim response.
|
||||
"""
|
||||
# Handle different response formats
|
||||
if "claim" in claim_data:
|
||||
claim = claim_data["claim"]
|
||||
else:
|
||||
claim = claim_data
|
||||
|
||||
value = claim.get("value", {})
|
||||
source = value.get("source", {})
|
||||
|
||||
# Get fee info if present
|
||||
fee = None
|
||||
if "fee" in value:
|
||||
fee = value["fee"]
|
||||
|
||||
# Determine file name from source
|
||||
file_name = source.get("name")
|
||||
|
||||
# Get media type
|
||||
source_media_type = source.get("media_type")
|
||||
|
||||
return {
|
||||
"claim_id": claim.get("claim_id"),
|
||||
"name": claim.get("name"),
|
||||
"normalized_name": claim.get("normalized_name"),
|
||||
"claim_type": claim.get("claim_type"),
|
||||
"value_type": claim.get("value_type"),
|
||||
"permanent_url": claim.get("permanent_url"),
|
||||
"short_url": claim.get("short_url"),
|
||||
"canonical_url": claim.get("canonical_url"),
|
||||
"timestamp": claim.get("timestamp"),
|
||||
"release_time": value.get("release_time"),
|
||||
"title": value.get("title"),
|
||||
"description": value.get("description"),
|
||||
"sd_hash": source.get("sd_hash"),
|
||||
"stream_hash": claim.get("stream_hash"),
|
||||
"txid": claim.get("txid"),
|
||||
"nout": claim.get("nout"),
|
||||
"tags": value.get("tags", []),
|
||||
"languages": value.get("languages", []),
|
||||
"fee": fee,
|
||||
"source_name": source.get("name"),
|
||||
"source_media_type": source_media_type,
|
||||
"file_name": file_name,
|
||||
}
|
||||
|
||||
|
||||
def format_summary(
|
||||
channels_scanned: int,
|
||||
claims_examined: int,
|
||||
new_downloads: int,
|
||||
new_versions: int,
|
||||
skipped_existing: int,
|
||||
redownloaded_missing: int,
|
||||
failures: int,
|
||||
) -> str:
|
||||
"""Format a run summary string."""
|
||||
lines = [
|
||||
"Run complete.",
|
||||
f"Channels scanned: {channels_scanned}",
|
||||
f"Claims examined: {claims_examined}",
|
||||
f"New downloads: {new_downloads}",
|
||||
f"New versions: {new_versions}",
|
||||
f"Skipped existing: {skipped_existing}",
|
||||
]
|
||||
|
||||
if redownloaded_missing > 0:
|
||||
lines.append(f"Redownloaded missing: {redownloaded_missing}")
|
||||
|
||||
lines.append(f"Failures: {failures}")
|
||||
|
||||
return "\n".join(lines)
|
||||
Reference in new issue
Block a user