mirror of
https://github.com/PR0M3TH3AN/Marlin.git
synced 2025-09-08 07:08:44 +00:00
update
This commit is contained in:
thePR0M3TH3AN
212
README.md
212
README.md
@@ -2,20 +2,21 @@
|
||||
|
||||
# Marlin
|
||||
|
||||
**Marlin** is a lightweight, metadata-driven file indexer that runs entirely on your computer. It scans folders, stores paths and file stats in SQLite, lets you add hierarchical **tags** and **custom attributes**, takes automatic snapshots, and offers instant full-text search with FTS5. Nothing ever leaves your machine.
|
||||
**Marlin** is a lightweight, metadata-driven file indexer that runs 100 % on your computer. It scans folders, stores paths and file stats in SQLite, lets you attach hierarchical **tags** and **custom attributes**, takes automatic snapshots, and offers instant full-text search via FTS5.
|
||||
*No cloud, no telemetry – your data never leaves the machine.*
|
||||
|
||||
---
|
||||
|
||||
## Feature highlights
|
||||
|
||||
| Area | What you get |
|
||||
|----------------|---------------------------------------------------------------------------------|
|
||||
| **Safety** | Timestamped backups `marlin backup` and one-command restore `marlin restore` |
|
||||
| **Upgrades** | Automatic schema migrations + dynamic column adds |
|
||||
| **Indexing** | Fast multi-path scanner (WAL mode) |
|
||||
| **Metadata** | Hierarchical tags (`project/alpha`) & key-value attributes (`reviewed=yes`) |
|
||||
| **Search** | Prefix-aware FTS5, optional `--exec` action per hit |
|
||||
| **DX / Logs** | Readable tracing (`RUST_LOG=debug …`) |
|
||||
| Area | What you get |
|
||||
| -------------- | --------------------------------------------------------------------------------- |
|
||||
| **Safety** | Timestamped backups (`marlin backup`) and one-command restore (`marlin restore`) |
|
||||
| **Resilience** | Versioned, idempotent schema migrations – zero-downtime upgrades |
|
||||
| **Indexing** | Fast multi-path scanner with SQLite WAL concurrency |
|
||||
| **Metadata** | Hierarchical tags (`project/alpha`) & key-value attributes (`reviewed=yes`) |
|
||||
| **Search** | Prefix-aware FTS5 across paths, tags, and attributes; optional `--exec` per match |
|
||||
| **DX / Logs** | Structured tracing (`RUST_LOG=debug`) for every operation |
|
||||
|
||||
---
|
||||
|
||||
@@ -29,18 +30,18 @@
|
||||
▲ search / exec └──────┬──────┘
|
||||
└────────── backup / restore ▼
|
||||
timestamped snapshots
|
||||
````
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Requirement | Why |
|
||||
| ------------------ | -------------------------------------- |
|
||||
| **Rust** ≥ 1.77 | Build toolchain (`rustup.rs`) |
|
||||
| C build essentials | `gcc`, `make`, etc. for bundled SQLite |
|
||||
| Requirement | Why |
|
||||
| ------------------ | ----------------------------- |
|
||||
| **Rust** ≥ 1.77 | Build toolchain (`rustup.rs`) |
|
||||
| C build essentials | Builds bundled SQLite (Linux) |
|
||||
|
||||
*(Windows/macOS: let the Rust installer pull the matching build tools.)*
|
||||
macOS & Windows users: let the Rust installer pull the matching build tools.
|
||||
|
||||
---
|
||||
|
||||
@@ -50,8 +51,7 @@
|
||||
git clone https://github.com/yourname/marlin.git
|
||||
cd marlin
|
||||
cargo build --release
|
||||
# optional: add to PATH
|
||||
sudo install -Dm755 target/release/marlin /usr/local/bin/marlin
|
||||
sudo install -Dm755 target/release/marlin /usr/local/bin/marlin # optional
|
||||
```
|
||||
|
||||
---
|
||||
@@ -59,21 +59,21 @@ sudo install -Dm755 target/release/marlin /usr/local/bin/marlin
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
marlin init # create DB
|
||||
marlin init # create DB (idempotent)
|
||||
marlin scan ~/Pictures ~/Documents # index files
|
||||
marlin tag "~/Pictures/**/*.jpg" photos/trip-2024 # add tag
|
||||
marlin attr set "~/Documents/**/*.pdf" reviewed yes
|
||||
marlin search reviewed --exec "xdg-open {}" # open hits
|
||||
marlin search reviewed --exec "xdg-open {}" # open matches
|
||||
marlin backup # snapshot DB
|
||||
```
|
||||
|
||||
### Database location
|
||||
|
||||
* **Linux** `~/.local/share/marlin/index.db`
|
||||
* **macOS** `~/Library/Application Support/marlin/index.db`
|
||||
* **macOS** `~/Library/Application Support/marlin/index.db`
|
||||
* **Windows** `%APPDATA%\marlin\index.db`
|
||||
|
||||
Override:
|
||||
Override with:
|
||||
|
||||
```bash
|
||||
export MARLIN_DB_PATH=/path/to/custom.db
|
||||
@@ -106,74 +106,170 @@ restore <snapshot.db> replace DB with snapshot
|
||||
|
||||
## Backups & restore
|
||||
|
||||
* **Create snapshot**
|
||||
*Create snapshot*
|
||||
|
||||
```bash
|
||||
marlin backup
|
||||
# → ~/.local/share/marlin/backups/backup_2025-05-14_22-15-30.db
|
||||
```
|
||||
```bash
|
||||
marlin backup
|
||||
# → ~/.local/share/marlin/backups/backup_2025-05-14_22-15-30.db
|
||||
```
|
||||
|
||||
* **Restore snapshot**
|
||||
*Restore snapshot*
|
||||
|
||||
```bash
|
||||
marlin restore ~/.local/share/marlin/backups/backup_2025-05-14_22-15-30.db
|
||||
```
|
||||
```bash
|
||||
marlin restore ~/.local/share/marlin/backups/backup_2025-05-14_22-15-30.db
|
||||
```
|
||||
|
||||
Marlin automatically takes a safety backup before any schema migration.
|
||||
Marlin also takes an **automatic safety backup before every schema migration**.
|
||||
|
||||
---
|
||||
|
||||
## Upgrading to a new build
|
||||
## Upgrading
|
||||
|
||||
```bash
|
||||
cargo install --path . --force # rebuild & overwrite installed binary
|
||||
cargo install --path . --force # rebuild & replace installed binary
|
||||
```
|
||||
|
||||
Backups + dynamic migrations mean your data is preserved across upgrades.
|
||||
The versioned migration system preserves your data across upgrades.
|
||||
|
||||
---
|
||||
|
||||
## Roadmap
|
||||
|
||||
| Milestone | Focus |
|
||||
| --------- | -------------------------------------------------- |
|
||||
| **M1** | `tags://` virtual folder • attribute search DSL |
|
||||
| **M2** | Real-time sync service • change-log diff viewer |
|
||||
| **M3** | Natural-language query builder |
|
||||
| **M4** | Plug-in marketplace • mobile (read-only) companion |
|
||||
See [`ROADMAP.md`](./ROADMAP.md) for the full development plan.
|
||||
|
||||
---
|
||||
|
||||
## Five-minute tutorial
|
||||
## Five-Minute Quickstart
|
||||
|
||||
Paste & run each block in your terminal.
|
||||
|
||||
---
|
||||
|
||||
### 0 Prepare & build
|
||||
|
||||
```bash
|
||||
# 0. Playground
|
||||
mkdir -p ~/marlin_demo/{Projects/{Alpha,Beta},Media/Photos,Docs}
|
||||
echo "Alpha draft" > ~/marlin_demo/Projects/Alpha/draft.txt
|
||||
echo "Receipt PDF" > ~/marlin_demo/Docs/receipt.pdf
|
||||
echo "fake jpg" > ~/marlin_demo/Media/Photos/vacation.jpg
|
||||
# Clone or cd into your Marlin repo
|
||||
cd ~/Documents/GitHub/Marlin
|
||||
|
||||
# 1. Init & scan
|
||||
# Build the release binary
|
||||
cargo build --release
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 1 Install on your PATH
|
||||
|
||||
```bash
|
||||
sudo install -Dm755 target/release/marlin /usr/local/bin/marlin
|
||||
```
|
||||
|
||||
> Now `marlin` is available everywhere.
|
||||
|
||||
---
|
||||
|
||||
### 2 Prepare a clean demo directory
|
||||
|
||||
```bash
|
||||
rm -rf ~/marlin_demo
|
||||
mkdir -p ~/marlin_demo/{Projects/{Alpha,Beta},Media/Photos,Docs}
|
||||
|
||||
printf "Alpha draft\n" > ~/marlin_demo/Projects/Alpha/draft.txt
|
||||
printf "Beta notes\n" > ~/marlin_demo/Projects/Beta/notes.md
|
||||
printf "Receipt PDF\n" > ~/marlin_demo/Docs/receipt.pdf
|
||||
printf "fake jpg\n" > ~/marlin_demo/Media/Photos/vacation.jpg
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3 Initialize & index files
|
||||
|
||||
```bash
|
||||
# Use --verbose if you want full debug traces:
|
||||
marlin init
|
||||
marlin scan ~/marlin_demo
|
||||
|
||||
# 2. Tags & attributes
|
||||
marlin tag "~/marlin_demo/Projects/Alpha/**/*" project/alpha
|
||||
# or, to see every path tested:
|
||||
marlin --verbose init
|
||||
marlin --verbose scan ~/marlin_demo
|
||||
```
|
||||
|
||||
> **Tip:** Rerun `marlin scan` after you add/remove/modify files; only changed files get re-indexed.
|
||||
|
||||
---
|
||||
|
||||
### 4 Attach tags & attributes
|
||||
|
||||
```bash
|
||||
# Tag everything under “Alpha”
|
||||
marlin tag "~/marlin_demo/Projects/Alpha/**/*" project/alpha
|
||||
|
||||
# Mark all PDFs as reviewed
|
||||
marlin attr set "~/marlin_demo/**/*.pdf" reviewed yes
|
||||
|
||||
# 3. Search
|
||||
marlin search alpha
|
||||
marlin search reviewed --exec "echo Found: {}"
|
||||
|
||||
# 4. Snapshot & restore
|
||||
marlin backup
|
||||
marlin restore ~/.local/share/marlin/backups/backup_YYYY-MM-DD_HH-MM-SS.db
|
||||
# (or with debug)
|
||||
marlin --verbose tag "~/marlin_demo/Projects/Alpha/**/*" project/alpha
|
||||
marlin --verbose attr set "~/marlin_demo/**/*.pdf" reviewed yes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5 Search your index
|
||||
|
||||
```bash
|
||||
# By tag or filename
|
||||
marlin search alpha
|
||||
|
||||
# Combined terms (AND across path+attrs)
|
||||
marlin search "reviewed AND pdf"
|
||||
|
||||
# Run a command on each hit
|
||||
marlin search reviewed --exec "echo HIT → {}"
|
||||
|
||||
# If things aren’t matching, add --verbose to see the underlying FTS query:
|
||||
marlin --verbose search "reviewed AND pdf"
|
||||
```
|
||||
|
||||
> `{}` in `--exec` is replaced with each file’s path.
|
||||
|
||||
---
|
||||
|
||||
### 6 Backup & restore
|
||||
|
||||
```bash
|
||||
# Snapshot and store its name
|
||||
snap=$(marlin backup | awk '{print $NF}')
|
||||
|
||||
# Simulate data loss
|
||||
rm ~/.local/share/marlin/index.db
|
||||
|
||||
# Restore instantly
|
||||
marlin restore "$snap"
|
||||
|
||||
# Verify your files still show up
|
||||
marlin search reviewed
|
||||
```
|
||||
|
||||
> Backups live under `~/.local/share/marlin/backups` by default.
|
||||
|
||||
##### What you just exercised
|
||||
|
||||
| Command | Purpose |
|
||||
| ----------------- | ----------------------------------------- |
|
||||
| `marlin init` | Create / upgrade the SQLite database |
|
||||
| `marlin scan` | Walk directories and (re)index files |
|
||||
| `marlin tag` | Attach hierarchical tags |
|
||||
| `marlin attr set` | Add/overwrite custom key-value attributes |
|
||||
| `marlin search` | FTS5 search across path / tags / attrs |
|
||||
| `--exec` | Pipe hits into any shell command |
|
||||
| `marlin backup` | Timestamped snapshot of the DB |
|
||||
| `marlin restore` | Replace live DB with a chosen snapshot |
|
||||
|
||||
That’s the complete surface area of Marlin today—feel free to play around or
|
||||
point the scanner at real folders.
|
||||
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
MIT – see `LICENSE`
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user