mirror of
https://github.com/PR0M3TH3AN/Marlin.git
synced 2025-09-08 07:08:44 +00:00
thePR0M3TH3AN
226
README.md
226
README.md
@@ -1,194 +1,80 @@
|
||||
|
||||
# Marlin ― Delivery Road‑map **v3.2**
|
||||
|
||||
# Marlin
|
||||
*Engineering‑ready – revised 2025‑05‑18*
|
||||
|
||||
**Marlin** is a lightweight, metadata-driven file indexer that runs **100 % on your computer**. It scans folders, stores paths and file stats in SQLite, lets you attach hierarchical **tags** and **custom attributes**, keeps timestamped **snapshots**, and offers instant full-text search via FTS5. _No cloud, no telemetry – your data never leaves the machine._
|
||||
> **Legend** △ engineering artefact ✦ user‑visible deliverable
|
||||
|
||||
---
|
||||
|
||||
## Feature highlights
|
||||
## 0 · Methodology primer (what “Done” means)
|
||||
|
||||
| Area | What you get |
|
||||
| ------------------- | ----------------------------------------------------------------------------------------------------- |
|
||||
| **Safety** | Timestamped backups (`marlin backup`) and one-command restore (`marlin restore`) |
|
||||
| **Resilience** | Versioned, idempotent schema migrations – zero-downtime upgrades |
|
||||
| **Indexing** | Fast multi-path scanner with SQLite WAL concurrency |
|
||||
| **Metadata** | Hierarchical tags (`project/alpha`) & key-value attributes (`reviewed=yes`) |
|
||||
| **Relations** | Typed file ↔ file links (`marlin link`) with backlinks viewer |
|
||||
| **Collections / Views** | Named playlists (`marlin coll`) & saved searches (`marlin view`) for instant recall |
|
||||
| **Search** | Prefix-aware FTS5 across paths, tags, attrs & links; optional `--exec` per match <br>(grep-style context snippets coming Q3) |
|
||||
| **DX / Logs** | Structured tracing (`RUST_LOG=debug`) for every operation |
|
||||
| Theme | Project rule‑of‑thumb |
|
||||
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Branching** | Trunk‑based. Feature branch → PR → 2 reviews → squash‑merge. |
|
||||
| **Spec first** | Each epic begins with a **Design Proposal (DP‑xxx)** in `/docs/adr/` containing schema diffs, example CLI session, perf targets. |
|
||||
| **Coverage** | Tarpaulin gate ≥ 85 % **on lines touched this sprint** (checked in CI). |
|
||||
| **Perf gate** | Cold‑start P95 ≤ 3 s on 100 k files **unless overridden in DP**. Regressions fail CI. |
|
||||
| **Docs** | CLI flags & examples land in `README.md` **same PR**. Docs tables (CLI cheatsheet, TUI key‑map) are auto‑generated during the build. |
|
||||
| **Demo** | Closing each epic yields a ≤ 2‑min asciinema or GIF in `docs/demos/`. |
|
||||
|
||||
---
|
||||
|
||||
## How it works
|
||||
## 1 · Bird’s‑eye table (engineering details + deliverables)
|
||||
|
||||
```text
|
||||
┌──────────────┐ marlin scan ┌─────────────┐
|
||||
│ your files │ ─────────────────────▶│ SQLite │
|
||||
│ (any folder) │ │ files/tags │
|
||||
└──────────────┘ tag / attr / link │ attrs / FTS │
|
||||
▲ search / exec └──────┬──────┘
|
||||
└────────── backup / restore ▼
|
||||
timestamped snapshots
|
||||
````
|
||||
| Phase / Sprint | Timeline | Focus & Rationale | ✦ Key UX Deliverables | △ Engineering artefacts / tasks | Definition of Done |
|
||||
| ----------------------------------------------- | ----------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
|
||||
| **Sprint 0 — Bootstrap & CI Baseline** | **2025‑Q2<br>(now → 30 May)** | CI scaffolding, coverage, crate split | — | • Split repo into **`libmarlin` (core)** + **`cli-bin`** + **`tui-bin`** <br>• Tarpaulin coverage + Hyperfine perf jobs wired <br>• `build.rs` renders CLI cheatsheet from `commands.yaml` <br>• Docs / cheatsheet autogen step in GitHub Actions | `cargo test --all` passes with coverage gate ≥ 85 %; docs artefacts appear in build; crates compile. |
|
||||
| **Sprint α — Bedrock & Metadata Domains** | **31 May → 13 Jun 2025** | Lock schema v1.1, first metadata objects | • CLI stubs: `marlin link / coll / view` <br>• `marlin demo` interactive tour | • **DP‑001 Schema v1.1** (ER + migration scripts) <br>• Unit tests (`escape_fts`, `determine_scan_root`) <br>• GitHub Action for SQL dry‑run | 100 % migrations green; demo prints ✅; logo badge shows schema version. |
|
||||
| **Epic 1 — Live‑Watch Mode & Backup Prune** | **2025‑Q2** | Continuous indexing via FS events; backups never explode | • `marlin watch <dir>` (inotify / FSEvents) <br>• `backup --prune N` (auto‑prune pre‑ and post‑command) | • **DP‑002** file‑watch life‑cycle & debounce strategy <br>• Change‑table schema storing dirty file IDs <br>• Nightly prune CI job | 8 h stress‑watch alters 10 k files → < 1 % missed; backup dir size ≤ N; watch CPU idle < 3 %. |
|
||||
| **Epic 2 — Dirty‑scan optimisation** | **2025‑Q2** | Re‑index only paths marked dirty by watch table | • `scan --dirty` | • Reuse change‑table from watch; Hyperfine benchmark script committed | Dirty‑scan runtime ≤ 15 % full scan on 100 k corpus; bench job passes. |
|
||||
| **Phase 3 — Content FTS + Annotations** | 2025‑Q3 | Grep snippets, inline notes | • `search -C3` grep‑style context <br>• `annotate add/list` | • **DP‑004** content‑blob strategy (inline vs ext‑table) <br>• `syntect` highlight PoC | Indexes 1 GB corpus ≤ 30 min; snippet CLI golden tests pass. |
|
||||
| **Phase 4 — Versioning & De‑duplication** | 2025‑Q3 | Historic diffs, SHA‑256 dedupe | • `scan --rehash` <br>• `version diff <file>` | • **DP‑005** hash column + Bloom‑de‑dupe research | Diff on 10 MB file ≤ 500 ms; duplicate sets emitted by CLI. |
|
||||
| **Phase 5 — Tag Aliases & Semantic Booster** | 2025‑Q3 | Tame tag sprawl; start AI hints | • `tag alias add/ls/rm` <br>• `tag suggest`, `summary` | • **DP‑006** embeddings size & k‑NN search bench | 95 % alias look‑ups resolved in one hop; suggest query ≤ 150 ms. |
|
||||
| **Phase 6 — Search DSL v2 & Smart Views** | 2025‑Q4 | AND/OR, ranges, structured grammar; smart folders | • New `nom` grammar <br>• Legacy parser behind **`--legacy-search`** (warn on use) | • **DP‑007** BNF + 30 acceptance strings <br>• Lexer fuzz tests (`cargo‑fuzz`) | Old queries keep working; 0 panics in fuzz run ≥ 1 M cases. |
|
||||
| **Phase 7 — Structured Workflows & Templates** | 2025‑Q4 | State graph, relationship templates | • `state set/log` <br>• `template apply` | • **DP‑008** workflow tables & YAML template spec <br>• Sample template e2e tests | Create template, apply to 20 files → all attrs/link rows present; illegal transitions blocked. |
|
||||
| **Phase 8 — TUI v1 + Lightweight Integrations** | 2026‑Q1 | Keyboard UI, VS Code sidebar | • **`marlin‑tui`** binary (tiling panes, key‑map) <br>• Read‑only VS Code sidebar | • **DP‑009** TUI redraw budget & key‑map <br>• Crate split fully consumed | TUI binary ≤ 2 MB; scroll redraw ≤ 4 ms; VS Code extension loads index. |
|
||||
| **Phase 9 — Dolphin Sidebar (MVP)** | 2026‑Q1 | Peek metadata inline in KDE Dolphin | • Qt/KIO sidebar | • **DP‑010** DB/IP bridge (D‑Bus vs UNIX socket) <br>• CMake packaging script | Sidebar opens ≤ 150 ms; passes KDE lint. |
|
||||
| **Phase 10 — Full GUI & Multi‑device Sync** | 2026‑Q2 | Visual editor + optional sync backend | • Electron/Qt hybrid explorer UI <br>• Select & integrate sync (LiteFS / Postgres) | • **DP‑011** sync back‑end trade‑study <br>• Busy‑timeout/retry strategy for multi‑writer mode | CRUD round‑trip < 2 s between two nodes; 25 GUI e2e tests green. |
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
### 2 · Feature cross‑matrix (quick look‑ups)
|
||||
|
||||
| Requirement | Why |
|
||||
| ------------------ | ----------------------------- |
|
||||
| **Rust ≥ 1.77** | Build toolchain (`rustup.rs`) |
|
||||
| C build essentials | Builds bundled SQLite (Linux) |
|
||||
|
||||
macOS & Windows users: let the Rust installer pull the matching build tools.
|
||||
| Capability | Sprint / Phase | CLI / GUI element | Linked DP |
|
||||
| -------------------------- | -------------- | ----------------------------------- | --------- |
|
||||
| Crate split & docs autogen | S0 | — | – |
|
||||
| Tarpaulin coverage gate | S0 | — | – |
|
||||
| Watch mode (FS events) | Epic 1 | `marlin watch .` | DP‑002 |
|
||||
| Backup auto‑prune | Epic 1 | `backup --prune N` | – |
|
||||
| Dirty‑scan | Epic 2 | `scan --dirty` | DP‑002 |
|
||||
| Grep snippets | Phase 3 | `search -C3 …` | DP‑004 |
|
||||
| Hash / dedupe | Phase 4 | `scan --rehash` | DP‑005 |
|
||||
| Tag aliases | Phase 5 | `tag alias` commands | DP‑006 |
|
||||
| Search DSL v2 | Phase 6 | new grammar, `--legacy-search` flag | DP‑007 |
|
||||
| Relationship templates | Phase 7 | `template new/apply` | DP‑008 |
|
||||
| TUI v1 | Phase 8 | `marlin‑tui` | DP‑009 |
|
||||
|
||||
---
|
||||
|
||||
## Build & install
|
||||
## 3 · Milestone acceptance checklist
|
||||
|
||||
```bash
|
||||
git clone https://github.com/PR0M3TH3AN/Marlin.git
|
||||
cd Marlin
|
||||
cargo build --release
|
||||
Before a milestone is declared **shipped**:
|
||||
|
||||
# (Optional) install into your PATH
|
||||
sudo install -Dm755 target/release/marlin /usr/local/bin/marlin
|
||||
```
|
||||
* [ ] **Spec** DP‑xxx merged with schema diff, ASCII‑cast demo
|
||||
* [ ] **Tests** Tarpaulin ≥ 85 % on changed lines; all suites green
|
||||
* [ ] **Perf guard** script passes on CI matrix (Ubuntu 22, macOS 14)
|
||||
* [ ] **Docs** auto‑regenerated; README & cheatsheet updated
|
||||
* [ ] **Demo** asciinema/GIF committed and linked in release notes
|
||||
* [ ] **Release tag** pushed; Cargo binary uploaded to GitHub Releases
|
||||
|
||||
---
|
||||
|
||||
## Quick start
|
||||
## 4 · Next immediate actions
|
||||
|
||||
For a concise walkthrough—including **links, collections and views**—see
|
||||
[**Quick start & Demo**](marlin_demo.md).
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
Below is a **repeat-able 3-step flow** you can use **every time you pull fresh code**.
|
||||
|
||||
### 0 Prepare once
|
||||
|
||||
```bash
|
||||
# Put build artefacts in one place (faster incremental builds)
|
||||
export CARGO_TARGET_DIR=target
|
||||
```
|
||||
|
||||
### 1 Build the new binary
|
||||
|
||||
```bash
|
||||
git pull
|
||||
cargo build --release
|
||||
sudo install -Dm755 target/release/marlin /usr/local/bin/marlin
|
||||
```
|
||||
|
||||
### 2 Run the smoke-test suite
|
||||
|
||||
```bash
|
||||
cargo test --test e2e -- --nocapture
|
||||
```
|
||||
|
||||
*Streams CLI output live; exit-code 0 = all good.*
|
||||
|
||||
### 3 (Optionally) run **all** tests
|
||||
|
||||
```bash
|
||||
cargo test --all -- --nocapture
|
||||
```
|
||||
|
||||
This now covers:
|
||||
|
||||
* unit tests in `src/**`
|
||||
* positive & negative integration suites (`tests/pos.rs`, `tests/neg.rs`)
|
||||
* doc-tests
|
||||
|
||||
#### One-liner helper
|
||||
|
||||
```bash
|
||||
git pull && cargo build --release &&
|
||||
sudo install -Dm755 target/release/marlin /usr/local/bin/marlin &&
|
||||
cargo test --test e2e -- --nocapture
|
||||
```
|
||||
|
||||
Alias it as `marlin-ci` for a 5-second upgrade-and-verify loop.
|
||||
|
||||
---
|
||||
|
||||
### Database location
|
||||
|
||||
| OS | Default path |
|
||||
| ----------- | ----------------------------------------------- |
|
||||
| **Linux** | `~/.local/share/marlin/index.db` |
|
||||
| **macOS** | `~/Library/Application Support/marlin/index.db` |
|
||||
| **Windows** | `%APPDATA%\marlin\index.db` |
|
||||
|
||||
Override:
|
||||
|
||||
```bash
|
||||
export MARLIN_DB_PATH=/path/to/custom.db
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CLI reference
|
||||
|
||||
```text
|
||||
marlin <COMMAND> [ARGS]
|
||||
|
||||
init create / migrate DB **and perform an initial scan of the cwd**
|
||||
scan <PATHS>... walk directories & (re)index files
|
||||
tag "<glob>" <tag_path> add hierarchical tag
|
||||
attr set <pattern> <key> <val> set or update custom attribute
|
||||
attr ls <path> list attributes
|
||||
link add|rm|list|backlinks manage typed file-to-file relations
|
||||
coll create|add|list manage named collections (“playlists”)
|
||||
view save|list|exec save and run smart views (saved queries)
|
||||
search <query> [--exec CMD] FTS5 query; optionally run CMD per hit
|
||||
backup create timestamped snapshot in `backups/`
|
||||
restore <snapshot.db> replace DB with snapshot
|
||||
completions <shell> generate shell completions
|
||||
```
|
||||
|
||||
### Attribute sub-commands
|
||||
|
||||
| Command | Example |
|
||||
| ----------- | ------------------------------------------------ |
|
||||
| `attr set` | `marlin attr set ~/Docs/**/*.pdf reviewed yes` |
|
||||
| `attr ls` | `marlin attr ls ~/Docs/report.pdf` |
|
||||
| JSON output | `marlin --format=json attr ls ~/Docs/report.pdf` |
|
||||
|
||||
---
|
||||
|
||||
## Backups & restore
|
||||
|
||||
```bash
|
||||
marlin backup
|
||||
# → ~/.local/share/marlin/backups/backup_2025-05-14_22-15-30.db
|
||||
```
|
||||
|
||||
```bash
|
||||
marlin restore ~/.local/share/marlin/backups/backup_2025-05-14_22-15-30.db
|
||||
```
|
||||
|
||||
> Marlin also creates an **automatic safety backup before every non-`init` command.**
|
||||
> *Auto-prune (`backup --prune <N>`) lands in Q2.*
|
||||
|
||||
---
|
||||
|
||||
## Upgrading
|
||||
|
||||
```bash
|
||||
cargo install --path . --force # rebuild & replace installed binary
|
||||
```
|
||||
|
||||
Versioned migrations preserve your data across upgrades.
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
MIT – see [`LICENSE`](LICENSE).
|
||||
| # | Task | Owner | Due |
|
||||
| - | ------------------------------ | ------ | ------------- |
|
||||
| 1 | Crate split + CI baseline | @alice | **26 May 25** |
|
||||
| 2 | Tarpaulin + Hyperfine jobs | @bob | **26 May 25** |
|
||||
| 3 | **DP‑001 Schema v1.1** draft | @carol | **30 May 25** |
|
||||
| 4 | backup prune CLI + nightly job | @dave | **05 Jun 25** |
|
||||
|
||||
|
||||
98
roadmap.md
98
roadmap.md
@@ -1,59 +1,75 @@
|
||||
# Marlin Roadmap 2025 → 2026 📜
|
||||
# Marlin ― Delivery Road-map **v3**
|
||||
|
||||
This document outlines the **official delivery plan** for Marlin over the next four quarters.
|
||||
Every work-item below is *time-boxed, testable,* and traceable back to an end-user benefit.
|
||||
*Engineering-ready version — updated 2025-05-17*
|
||||
|
||||
> **Legend**
|
||||
> ✅ = item added/clarified in the latest planning round
|
||||
> Δ = new sub-deliverable (wasn’t in the previous version)
|
||||
> **△** = engineering artefact (spec / ADR / perf target) **✦** = user-visible deliverable
|
||||
|
||||
---
|
||||
|
||||
## 1 Bird’s-eye Table
|
||||
## 0 · Methodology primer (what “Done” means)
|
||||
|
||||
| Phase / Sprint | Timeline | Focus & Rationale | Key Deliverables (Δ = new) | | |
|
||||
| ----------------------------------------------- | ------------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------ |
|
||||
| **Sprint α – Bedrock & Metadata Domains** | **2025-Q2 (now → 6 Jun)** | Stabilise schema & CI; land first metadata domains with discoverability. | Δ CI: `cargo test` + SQL dry-run<br>Δ Unit tests (`determine_scan_root`, `escape_fts`)<br>Δ Coverage: e2e `attr --format=json`<br>Δ Refactor: move `naive_substring_search` to shared util<br>Migrations: `links`, `collections`, `views`<br>CLI stubs: `link`, `coll`, `view`<br>`marlin demo` walkthrough | | |
|
||||
| **Epic 1 – Scale & Reliability** | 2025-Q2 | Keep scans fast; bullet-proof CI at 100 k files. | Δ Dirty-flag column + `scan --dirty`<br>Benchmarks: full vs dirty scan (100 k)<br>Replace per-row triggers with periodic rebuild<br>CI edge-case tests | | |
|
||||
| **Epic 2 – Live Mode & Self-Pruning Backups** | 2025-Q2 | Continuous indexing & hygiene—Marlin “just works”. | Δ `marlin watch [dir]` (notify/FSEvents)<br>Δ `backup --prune <N>` + auto-prune post-scan<br>Daily / PR-merge prune in CI | | |
|
||||
| **Phase 3 – Content FTS & Annotations** | 2025-Q3 | Index file bodies, grep-style context, inline notes. | `files.content` + migration<br>Extend `files_fts` (context snippets `-C`)<br>`annotations` table + triggers<br>CLI \`annotate add | list\` | |
|
||||
| **Phase 4 – Versioning & Deduplication** | 2025-Q3 | History, diffs & duplicate detection. | `files.hash` (SHA-256)<br>`scan --rehash` refresh<br>CLI `version diff <file>` | | |
|
||||
| **Phase 5 – Tag Aliases & Semantic Booster** | 2025-Q3 | Tame tag sprawl; seed AI-powered suggestions. | `canonical_id` on `tags`; CLI `tag alias …`<br>`embeddings` table + `scan --embed`<br>CLI `tag suggest`, `similarity scan`, `summary <file>` | | |
|
||||
| **Phase 6 – Search DSL v2 & Smart Views** | 2025-Q4 | Robust grammar + virtual folders. | Replace parser with **`nom`** grammar (`AND`, `OR`, `()` …)<br>CLI \`view save | list | exec\` with aliases & paging |
|
||||
| **Phase 7 – Structured Workflows** | 2025-Q4 | First-class task / state / reminder / event life-cycles. | ✅ State engine (`files.state`, `state_changes`)<br>CLI \`state set | transitions add | log`<br>✅ Task extractor (`tasks` table) + CLI<br>`templates`+ validation<br>CLI`remind …`, `event …`, `timeline\` |
|
||||
| **Phase 8 – Lightweight Integrations** | 2026-Q1 | Surface Marlin in editors / terminal. | VS Code & TUI extension (tags / attrs / links / notes) | | |
|
||||
| **Phase 9 – Dolphin Sidebar Plugin (MVP)** | 2026-Q1 | Read-only Qt sidebar for Linux file managers. | Qt plug-in: tags, attrs, links, annotations | | |
|
||||
| **Phase 10 – Full Edit UI & Multi-Device Sync** | 2026-Q2 | In-place metadata editor & optional sync layer. | GUI editors (tags, views, tasks, reminders, events)<br>Pick/implement sync backend (rqlite, Litestream, …) | | |
|
||||
| Theme | Project rule-of-thumb |
|
||||
| -------------- | -------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Branching** | Trunk-based. Feature branches → PR → 2 reviews → squash-merge. |
|
||||
| **Spec first** | Every epic starts with a **Design Proposal (DP-xxx)** in `/docs/adr/`. Include schema diffs, example CLI session, perf budget. |
|
||||
| **Tests** | Unit + integration coverage ≥ 85 % on lines **touched in the sprint** (checked by Tarpaulin). |
|
||||
| **Perf gate** | Cold start P95 ≤ 3 s on 100 k files **unless overridden in DP**. Regressions fail CI. |
|
||||
| **Docs** | CLI flags & examples land in `README.md` **same PR** that ships the code. |
|
||||
| **Demo** | Closing each epic produces a 2-min asciinema or gif in `docs/demos/`. |
|
||||
|
||||
---
|
||||
|
||||
## 2 Narrative & Dependencies
|
||||
## 1 · Bird’s-eye table (now includes engineering columns)
|
||||
|
||||
1. **Lock down core schema & demo** *(Sprint α).*
|
||||
Developers get immediate feedback via the `marlin demo` command while CI ensures migrations never regress.
|
||||
|
||||
2. **Scale & Live Mode** *(Epics 1-2).*
|
||||
Dirty scanning, file-watching and auto-pruned backups guarantee snappy, hands-off operation even on six-figure corpora.
|
||||
|
||||
3. **Richer Search** *(Phases 3-6).*
|
||||
Body-content FTS + grep-style snippets lay the groundwork; `nom` grammar then elevates power-user queries and smart views.
|
||||
|
||||
4. **Workflow Layers** *(Phase 7).*
|
||||
State transitions, tasks and reminders turn Marlin from a passive index into an active workflow engine.
|
||||
|
||||
5. **UX Expansions** *(Phases 8-10).*
|
||||
Start lightweight (VS Code / TUI), graduate to a read-only Dolphin plug-in, then ship full editing & sync for multi-device teams.
|
||||
|
||||
Every outer milestone depends only on the completion of the rows above it, **so shipping discipline in early sprints de-risks the headline features down the line.**
|
||||
| Phase / Sprint | Timeline | Focus & Rationale | ✦ Key UX Deliverables | △ Engineering artefacts / tasks | Definition of Done | | | |
|
||||
| --------------------------------------------- | ----------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------- |
|
||||
| **Sprint α — Bedrock & Metadata Domains** | **2025-Q2<br>(now → 06 Jun)** | Lock schema, smoking-fast CI, first metadata objects. | • CLI stubs: `marlin link / coll / view`<br>• `marlin demo` interactive tour | • DP-001 Schema v1.1 (ER + migration scripts)<br>• Unit tests (`escape_fts`, `determine_scan_root`)<br>• GitHub Action for SQL dry-run | 100 % migrations green on CI; demo command prints green tick | | | |
|
||||
| **Epic 1 — Scale & Reliability** | 2025-Q2 | Stay fast @ 100 k files | • `scan --dirty` (re-index touched rows only) | • DP-002 Dirty-flag design + FTS rebuild cadence<br>• Hyperfine benchmark script committed | Dirty scan vs full ≤ 15 % runtime on 100 k corpus; benchmark job passes | | | |
|
||||
| **Epic 2 — Live Mode & Self-Pruning Backups** | 2025-Q2 | “Just works” indexing, DB never explodes | • `marlin watch <dir>` (notify/FSEvents)<br>• `backup --prune N` & auto-prune | • DP-003 file-watcher life-cycle & debouncing<br>• Integration test with inotify-sim <br>• Cron-style GitHub job for nightly prune | 8 h stress-watch alters 10 k files < 1 % misses; backup dir ≤ N | | | |
|
||||
| **Phase 3 — Content FTS + Annotations** | 2025-Q3 | Search inside files, leave notes | • Grep-style snippet output (`-C3`)<br>• \`marlin annotate add | list\` | • DP-004 content-blob strategy (inline vs ext-table)<br>• Syntax-highlight via `syntect` PoC<br>• New FTS triggers unit-tested | Indexes 1 GB corpus in ≤ 30 min; snippet CLI passes golden-file tests | | |
|
||||
| **Phase 4 — Versioning & Deduplication** | 2025-Q3 | Historic diffs, detect dupes | • `scan --rehash` (SHA-256)<br>• `version diff <file>` | • DP-005 hash column + Bloom-de-dupe<br>• Binary diff adapter research | Diff on 10 MB file ≤ 500 ms; dupes listed via CLI | | | |
|
||||
| **Phase 5 — Tag Aliases & Semantic Booster** | 2025-Q3 | Tame tag sprawl, start AI hints | • \`tag alias add | ls | rm`<br>• `tag suggest`, `summary\` | • DP-006 embeddings size & model choice<br>• Vector store schema + k-NN index bench | 95 % of “foo/bar\~foo” alias look-ups resolve in one hop; suggest CLI returns ≤ 150 ms | |
|
||||
| **Phase 6 — Search DSL v2 & Smart Views** | 2025-Q4 | Pro-grade query language | • New `nom` grammar: AND/OR, parentheses, ranges | • DP-007 BNF + 30 acceptance strings<br>• Lexer fuzz-tests with `cargo-fuzz` | Old queries keep working (migration shim); 0 crashes in fuzz run ≥ 1 M cases | | | |
|
||||
| **Phase 7 — Structured Workflows** | 2025-Q4 | Tasks, state, reminders, templates | • \`state set | transitions add | log`<br>• `task scan | list`<br>• **NEW:** `template apply\` for relationship templates | • DP-008 Workflow tables & validation<br>• Sample YAML template spec + CLI expansion tests | Create template, apply to 20 files → all attrs/link rows present; state graph denies illegal transitions |
|
||||
| **Phase 8 — Lightweight Integrations** | 2026-Q1 | First “shell” GUIs | • VS Code side-bar (read-only)<br>• **TUI v1** (tag tree ▸ file list ▸ preview) | • DP-009 TUI key-map & redraw budget<br>• Crate split `marlin_core`, `marlin_tui` | TUI binary ≤ 2.0 MB; 10 k row scroll ≤ 4 ms redraw | | | |
|
||||
| **Phase 9 — Dolphin Sidebar (MVP)** | 2026-Q1 | Peek metadata in KDE file-manager | • Qt-plugin showing tags, attrs, links | • DP-010 DB/IP bridge (D-Bus vs UNIX socket)<br>• CMake packaging script | Sidebar opens in ≤ 150 ms; passes KDE lint | | | |
|
||||
| **Phase 10 — Full GUI & Multi-device Sync** | 2026-Q2 | Edit metadata visually, sync option | • Electron/Qt hybrid explorer UI<br>• Pick & integrate sync backend | • DP-011 sync back-end trade-study<br>• UI e2e tests in Playwright | Round-trip CRUD between two nodes in < 2 s; 25 GUI tests green | | | |
|
||||
|
||||
---
|
||||
|
||||
## 3 Next Steps
|
||||
### 2 · Feature cross-matrix (quick look-ups)
|
||||
|
||||
* **Sprint α kickoff:** break deliverables into stories, estimate, assign.
|
||||
* **Add roadmap as `docs/ROADMAP.md`** (this file).
|
||||
* Wire a **Checklist issue** on GitHub: one task per Δ bullet for instant tracking.
|
||||
| Capability | Sprint / Phase | CLI flag or GUI element | Linked DP |
|
||||
| ------------------------------------- | -------------- | ---------------------------------- | --------- |
|
||||
| Relationship **templates** | P7 | `template new`, `template apply` | DP-008 |
|
||||
| Positive / negative filter combinator | P6 | DSL `+tag:foo -tag:bar date>=2025` | DP-007 |
|
||||
| Dirty-scan optimisation | E1 | `scan --dirty` | DP-002 |
|
||||
| Watch-mode | E2 | `marlin watch .` | DP-003 |
|
||||
| Grep snippets | P3 | `search -C3 "foo"` | DP-004 |
|
||||
| Hash / dedupe | P4 | `scan --rehash` | DP-005 |
|
||||
|
||||
---
|
||||
|
||||
*Last updated · 2025-05-16*
|
||||
## 3 · Milestone acceptance checklist
|
||||
|
||||
Before a milestone is declared “shipped”:
|
||||
|
||||
* [ ] **Spec** merged (DP-xxx) with schema diff & example ASCII-cast
|
||||
* [ ] **Unit & integration tests** ≥ 85 % coverage on changed lines
|
||||
* [ ] **Perf guard-rail** script passes on CI matrix (Ubuntu 22, macOS 14)
|
||||
* [ ] **Docs** — CLI man-page, README table row, roadmap ticked
|
||||
* [ ] **Demo** uploaded to `docs/demos/` and linked in release notes
|
||||
* [ ] **Release tag** pushed; Cargo binary on GitHub Releases
|
||||
|
||||
---
|
||||
|
||||
### 4 · Next immediate actions
|
||||
|
||||
1. **Write DP-001 (Schema v1.1)** — owner @alice, due 21 May
|
||||
2. **Set up Tarpaulin & Hyperfine jobs** — @bob, due 23 May
|
||||
3. **Spike dirty-flag logic** — @carol 2 days time-box, outcome in DP-002
|
||||
|
||||
---
|
||||
|
||||
> *This roadmap now contains both product-level “what” and engineering-level “how/when/prove it”. It should allow a new contributor to jump in, pick the matching DP, and know exactly the bar they must clear for their code to merge.*
|
||||
|
||||
200
spec-details/TUI+Query-DSL+Templates.md
Normal file
200
spec-details/TUI+Query-DSL+Templates.md
Normal file
@@ -0,0 +1,200 @@
|
||||
# Marlin — TUI + Query-DSL + Templates
|
||||
|
||||
**Integration Specification · v0.1 (2025-05-18)**
|
||||
|
||||
---
|
||||
|
||||
## 0 · Scope
|
||||
|
||||
This document turns the **concept sketch** (suck-less TUI) **plus** the new
|
||||
*relationship-template* and *positive/negative filter* ideas into a concrete
|
||||
development plan that extends the existing Marlin code-base without breaking
|
||||
CLI workflows.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Feature Set
|
||||
|
||||
| Epic | Deliverable | Short Description |
|
||||
| -------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------ |
|
||||
| **T-1** | **Marlin-TUI** binary | One-screen, tiling, key-centred interface (sketch in §2). |
|
||||
| **Q-1** | **Query DSL v2** | Lexer + parser + planner supporting `AND/OR`, `NOT`, parentheses, ranges, comparison ops on `size`, `mtime`… |
|
||||
| **T-2** | **Relationship templates** | Named templates with typed fields (key + value-type) that auto-populate attributes when applied. |
|
||||
| **Glue** | Shared **`libmarlin`** crate | Re-export public APIs so both CLI and TUI call the same Rust functions. |
|
||||
|
||||
These map cleanly onto the public roadmap:
|
||||
|
||||
* T-1 → *Phase 6: Search DSL v2 & Smart Views* (UI portion)
|
||||
* Q-1 → *Phase 6* (parser)
|
||||
* T-2 → *Phase 7: Structured Workflows* (templates subsection)
|
||||
|
||||
---
|
||||
|
||||
## 2 · TUI Reference Layout & Key-Map
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Marlin ▸ ~/demo (q quit · ? help) │
|
||||
├───────────────┬─────────────────────────────────────────────┤
|
||||
│ Tags / Views │ Path / Name │ ⓘ Attributes │
|
||||
│───────────────│────────────────────────┼────────────────────│
|
||||
│ project │ api/design.md │ status=pending │
|
||||
│ ├ alpha │ roadmap.xlsx │ reviewed=yes │
|
||||
│ │ ├ draft │ ... │ tags=project/alpha │
|
||||
│ │ └ final │ (42/812 shown) │ links: 2 outgoing │
|
||||
│ <Views> │ grep: "(?i)todo" │ ───── Preview ───── │
|
||||
│ • urgent │ │ - [ ] TODO write… │
|
||||
└───────────────┴────────────────────────┴────────────────────┘
|
||||
```
|
||||
|
||||
### Essential Keys
|
||||
|
||||
| Key | Effect | CLI equivalence |
|
||||
| --------------- | ------------------------- | -------------------------------- |
|
||||
| `/expr⏎` | Regex/FTS filter | `marlin search` |
|
||||
| `;key=val⏎` | Attr filter | `marlin search attr:key=val` |
|
||||
| `:` | Command prompt | raw CLI passthrough |
|
||||
| `Space` | Mark/unmark | adds to in-memory selection list |
|
||||
| `:tag foo/bar⏎` | Tag marked files | `marlin tag` |
|
||||
| `Enter` | open file in `$EDITOR` | — |
|
||||
| `Tab` | toggle right preview pane | — |
|
||||
|
||||
Full table lives in **`docs/tui_keys.md`** (auto-generated from
|
||||
`src/tui/keymap.rs`).
|
||||
|
||||
---
|
||||
|
||||
## 3 · Architecture Impact
|
||||
|
||||
### 3.1 Crate refactor (Week 1)
|
||||
|
||||
```
|
||||
marlin/
|
||||
├── cli-bin/ ← binary crate (keeps current main.rs)
|
||||
├── tui-bin/ ← NEW binary crate (Marlin-TUI)
|
||||
├── libmarlin/ ← NEW library: DB helpers, scan, query, templates
|
||||
└── Cargo.toml
|
||||
```
|
||||
|
||||
*Export surface*:
|
||||
|
||||
```rust
|
||||
pub struct Marlin { /* connection, config … */ }
|
||||
impl Marlin {
|
||||
pub fn search(&self, q: Query) -> Result<Vec<PathBuf>>;
|
||||
pub fn tag(&self, files: &[PathBuf], tag: &str) -> Result<()>;
|
||||
pub fn apply_template(&self, files: &[PathBuf], tpl: &Template) -> Result<()>;
|
||||
/* … */
|
||||
}
|
||||
```
|
||||
|
||||
CLI keeps 100 % of its flags; it just calls `libmarlin`.
|
||||
|
||||
TUI links against the same crate → zero business-logic duplication.
|
||||
|
||||
---
|
||||
|
||||
### 3.2 Query-DSL v2 (Week 2-3)
|
||||
|
||||
* **Lexer:** [`logos`](https://crates.io/crates/logos) (<500 LOC).
|
||||
* **Parser:** `nom` (top-down precedence climbing).
|
||||
* **AST → Plan:**
|
||||
|
||||
```rust
|
||||
struct Plan {
|
||||
fts_match: Option<String>, // tags_text/path/attrs_text
|
||||
sql_where: Vec<String>, // files.size > ?, files.mtime BETWEEN ? ?
|
||||
params: Vec<SqlArg>,
|
||||
}
|
||||
```
|
||||
* **Planner** decides when to fall back to pure SQL (e.g. `size>1M AND NOT tag:archive` uses an FTS sub-query joined on rowid).
|
||||
|
||||
Backwards compatibility: old space-separated syntax is parsed first; if it fails,
|
||||
DSL v2 is attempted, so no user breakage.
|
||||
|
||||
---
|
||||
|
||||
### 3.3 Templates (Week 4)
|
||||
|
||||
**Schema (migration 0005):**
|
||||
|
||||
```sql
|
||||
CREATE TABLE templates (
|
||||
id INTEGER PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
description TEXT
|
||||
);
|
||||
CREATE TABLE template_fields (
|
||||
id INTEGER PRIMARY KEY,
|
||||
template_id INTEGER NOT NULL REFERENCES templates(id),
|
||||
key TEXT NOT NULL,
|
||||
value_type TEXT NOT NULL -- 'text' | 'int' | 'date'
|
||||
);
|
||||
```
|
||||
|
||||
**CLI additions**
|
||||
|
||||
```
|
||||
marlin template list|show|add|rm
|
||||
marlin template apply <name> <glob>
|
||||
```
|
||||
|
||||
`Templater` helper validates type coercion before inserting into `attributes`.
|
||||
|
||||
TUI: `:apply <tpl>` applies to marked files and prompts inline for field values
|
||||
(stenographic prompt à la `git commit --template`).
|
||||
|
||||
---
|
||||
|
||||
## 4 · Implementation Phases & Timeline
|
||||
|
||||
| Week | Milestone / Task | Owner | Exit Criteria |
|
||||
| ---- | ------------------------------------------ | -------- | ------------------------- |
|
||||
| 1 | Split repo into `libmarlin`, adapt CLI | core dev | `cargo test --all` green |
|
||||
| 2 | Lexer + unit tests | core dev | 95 % coverage |
|
||||
| 3 | Parser + Planner; replace `run_search()` | core dev | DSL green tests pass |
|
||||
| 4 | Template schema, CLI, tests | DB dev | `template apply` e2e test |
|
||||
| 5 | **TUI MVP**<br>draw loop + panes + key nav | UI dev | Can browse & open files |
|
||||
| 6 | TUI ↔ Marlin API glue<br>mark, tag, search | UI dev | All key-map smoke tests |
|
||||
| 7 | Preview cache, file-watch refresh | UI dev | CPU < 5 % idle load |
|
||||
| 8 | Docs refresh, release notes | all | v0.2.0 tag |
|
||||
|
||||
---
|
||||
|
||||
## 5 · Testing & CI
|
||||
|
||||
* **DSL golden-file tests** (`tests/dsl/*.txt` → expected SQL).
|
||||
* **SQLite snapshot tests** for templates.
|
||||
* **TUI headless tests** via `termwiz`’s `TerminalTest` harness.
|
||||
* GitHub Actions matrix unchanged; new crate just builds in.
|
||||
|
||||
---
|
||||
|
||||
## 6 · Risks & Mitigations
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
| ---------------------------- | -------------- | ------------------------------------------------------------ |
|
||||
| Parser ambiguity | broken queries | formal grammar + fuzzy tests |
|
||||
| TUI performance on huge dirs | lag | incremental drawing + LRU file stat cache |
|
||||
| Schema lock-in for templates | hard to evolve | keep `value_type` generic (text) until needs prove otherwise |
|
||||
|
||||
---
|
||||
|
||||
## 7 · What Stays Back-Compatible?
|
||||
|
||||
* **CLI syntax**: existing commands & search strings still work.
|
||||
* **Database**: 0005 migration is additive.
|
||||
* **Automation scripts**: unchanged; TUI is *optional* (`marlin-tui` binary).
|
||||
|
||||
---
|
||||
|
||||
## 8 · Why It’s Worth It
|
||||
|
||||
* **Differentiator**: Few local-first tools combine rich metadata, SQL-grade
|
||||
filtering, and both script-friendly CLI *and* keyboard-driven TUI.
|
||||
* **Gateway to GUI**: The API we expose for TUI is the same one the
|
||||
future GTK/Qt desktop app can use.
|
||||
|
||||
---
|
||||
|
||||
**Let the hacking begin!**
|
||||
Reference in New Issue
Block a user