Marlin/README.md

Marlin

Marlin is a lightweight, metadata-driven file indexer that runs 100 % on your computer. It scans folders, stores paths and file stats in SQLite, lets you attach hierarchical tags and custom attributes, takes automatic snapshots, and offers instant full-text search via FTS5.
No cloud, no telemetry – your data never leaves the machine.

---

Feature highlights

Area	What you get
Safety	Timestamped backups (marlin backup) and one-command restore (marlin restore)
Resilience	Versioned, idempotent schema migrations – zero-downtime upgrades
Indexing	Fast multi-path scanner with SQLite WAL concurrency
Metadata	Hierarchical tags (project/alpha) & key-value attributes (reviewed=yes)
Search	Prefix-aware FTS5 across paths, tags, and attributes; optional --exec per match
DX / Logs	Structured tracing (RUST_LOG=debug) for every operation

---

How it works

┌──────────────┐  marlin scan          ┌─────────────┐
│  your files  │ ─────────────────────▶│   SQLite    │
│ (any folder) │                      │  files/tags │
└──────────────┘   tag / attr          │ attrs / FTS │
        ▲  search / exec              └──────┬──────┘
        └────────── backup / restore          ▼
                                     timestamped snapshots

---

Prerequisites

Requirement	Why
Rust ≥ 1.77	Build toolchain (rustup.rs)
C build essentials	Builds bundled SQLite (Linux)

macOS & Windows users: let the Rust installer pull the matching build tools.

---

Build & install

git clone https://github.com/yourname/marlin.git
cd marlin
cargo build --release

# (Optional) Install the binary into your PATH:
sudo install -Dm755 target/release/marlin /usr/local/bin/marlin

Quick start

For a concise walkthrough, see Quick start & Demo.

Testing

Below is a repeat-able 3-step flow you can use every time you pull fresh code.

---

0 Prepare once

# Run once (or add to ~/.bashrc) so debug + release artefacts land
# in the same predictable place.  Speeds-up future builds.
export CARGO_TARGET_DIR=target

---

1 Build the new binary

git pull             # grab the latest commit
cargo build --release
sudo install -Dm755 target/release/marlin /usr/local/bin/marlin

• cargo build --release – builds the optimised binary.
• install … – copies it into your $PATH so marlin on the CLI is the fresh one.

---

2 Run the smoke-test suite

# Runs the end-to-end test we added in tests/e2e.rs
cargo test --test e2e -- --nocapture

• --test e2e – compiles and runs only tests/e2e.rs; other unit-tests are skipped (add them later if you like).
• --nocapture – streams stdout/stderr so you can watch each CLI step in real time.
• Exit-code 0 ➜ everything passed. Any non-zero exit or a red ✗ line means a step failed; the assert’s diff will show the command and its output.

---

3 (Optionally) run all tests

cargo test --all -- --nocapture

This will execute:

• unit tests in src/**
• every file in tests/
• doc-tests

If you wire “cargo test --all” into CI (GitHub Actions, GitLab, etc.), pushes that break a workflow will be rejected automatically.

---

One-liner helper (copy/paste)

git pull && cargo build --release &&
sudo install -Dm755 target/release/marlin /usr/local/bin/marlin &&
cargo test --test e2e -- --nocapture

Stick that in a shell alias (alias marlin-ci='…') and you’ve got a 5-second upgrade-and-verify loop.

Database location

• Linux ~/.local/share/marlin/index.db
• macOS ~/Library/Application Support/marlin/index.db
• Windows %APPDATA%\marlin\index.db

Override with:

export MARLIN_DB_PATH=/path/to/custom.db

---

CLI reference

marlin <COMMAND> [ARGS]

init                             create / migrate database
scan   <PATHS>...                walk directories & index files
tag    "<glob>" <tag_path>       add hierarchical tag
attr   set <pattern> <key> <value>  manage custom attributes
attr   ls <path>
search <query> [--exec CMD]      FTS5 query, optionally run CMD on each hit
backup                           create timestamped snapshot in backups/
restore <snapshot.db>            replace DB with snapshot
completions <shell>              generate shell completions

Attribute subcommands

Command	Example
attr set	marlin attr set ~/Docs/**/*.pdf reviewed yes
attr ls	marlin attr ls ~/Docs/report.pdf

---

Backups & restore

Create snapshot

marlin backup
# → ~/.local/share/marlin/backups/backup_2025-05-14_22-15-30.db

Restore snapshot

marlin restore ~/.local/share/marlin/backups/backup_2025-05-14_22-15-30.db

Marlin also takes an automatic safety backup before every non-init command.

---

Upgrading

cargo install --path . --force    # rebuild & replace installed binary

The versioned migration system preserves your data across upgrades.

---

License

MIT – see LICENSE