From 6157ac5233a9a395266b31269d21f3e75316244f Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Sun, 18 May 2025 14:25:38 -0400 Subject: [PATCH] update --- README.md | 226 ++++++------------------ roadmap.md | 98 +++++----- spec-details/TUI+Query-DSL+Templates.md | 200 +++++++++++++++++++++ 3 files changed, 313 insertions(+), 211 deletions(-) create mode 100644 spec-details/TUI+Query-DSL+Templates.md diff --git a/README.md b/README.md index 824c79a..1249495 100644 --- a/README.md +++ b/README.md @@ -1,194 +1,80 @@ -![Marlin Logo](https://raw.githubusercontent.com/PR0M3TH3AN/Marlin/refs/heads/main/assets/png/marlin_logo.png) +# Marlin ― Delivery Road‑map **v3.2** -# Marlin +*Engineering‑ready – revised 2025‑05‑18* -**Marlin** is a lightweight, metadata-driven file indexer that runs **100 % on your computer**. It scans folders, stores paths and file stats in SQLite, lets you attach hierarchical **tags** and **custom attributes**, keeps timestamped **snapshots**, and offers instant full-text search via FTS5. _No cloud, no telemetry – your data never leaves the machine._ +> **Legend** △ engineering artefact ✦ user‑visible deliverable --- -## Feature highlights +## 0 · Methodology primer  (what “Done” means) -| Area | What you get | -| ------------------- | ----------------------------------------------------------------------------------------------------- | -| **Safety** | Timestamped backups (`marlin backup`) and one-command restore (`marlin restore`) | -| **Resilience** | Versioned, idempotent schema migrations – zero-downtime upgrades | -| **Indexing** | Fast multi-path scanner with SQLite WAL concurrency | -| **Metadata** | Hierarchical tags (`project/alpha`) & key-value attributes (`reviewed=yes`) | -| **Relations** | Typed file ↔ file links (`marlin link`) with backlinks viewer | -| **Collections / Views** | Named playlists (`marlin coll`) & saved searches (`marlin view`) for instant recall | -| **Search** | Prefix-aware FTS5 across paths, tags, attrs & links; optional `--exec` per match
(grep-style context snippets coming Q3) | -| **DX / Logs** | Structured tracing (`RUST_LOG=debug`) for every operation | +| Theme | Project rule‑of‑thumb | +| -------------- | ------------------------------------------------------------------------------------------------------------------------------------- | +| **Branching** | Trunk‑based. Feature branch → PR → 2 reviews → squash‑merge. | +| **Spec first** | Each epic begins with a **Design Proposal (DP‑xxx)** in `/docs/adr/` containing schema diffs, example CLI session, perf targets. | +| **Coverage** | Tarpaulin gate ≥ 85 % **on lines touched this sprint** (checked in CI). | +| **Perf gate** | Cold‑start P95 ≤ 3 s on 100 k files **unless overridden in DP**. Regressions fail CI. | +| **Docs** | CLI flags & examples land in `README.md` **same PR**. Docs tables (CLI cheatsheet, TUI key‑map) are auto‑generated during the build. | +| **Demo** | Closing each epic yields a ≤ 2‑min asciinema or GIF in `docs/demos/`. | --- -## How it works +## 1 · Bird’s‑eye table (engineering details + deliverables) -```text -┌──────────────┐ marlin scan ┌─────────────┐ -│ your files │ ─────────────────────▶│ SQLite │ -│ (any folder) │ │ files/tags │ -└──────────────┘ tag / attr / link │ attrs / FTS │ - ▲ search / exec └──────┬──────┘ - └────────── backup / restore ▼ - timestamped snapshots -```` +| Phase / Sprint | Timeline | Focus & Rationale | ✦ Key UX Deliverables | △ Engineering artefacts / tasks | Definition of Done | +| ----------------------------------------------- | ----------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | +| **Sprint 0 — Bootstrap & CI Baseline** | **2025‑Q2
(now → 30 May)** | CI scaffolding, coverage, crate split | — | • Split repo into **`libmarlin` (core)** + **`cli-bin`** + **`tui-bin`**
• Tarpaulin coverage + Hyperfine perf jobs wired
• `build.rs` renders CLI cheatsheet from `commands.yaml`
• Docs / cheatsheet autogen step in GitHub Actions | `cargo test --all` passes with coverage gate ≥ 85 %; docs artefacts appear in build; crates compile. | +| **Sprint α — Bedrock & Metadata Domains** | **31 May → 13 Jun 2025** | Lock schema v1.1, first metadata objects | • CLI stubs: `marlin link / coll / view`
• `marlin demo` interactive tour | • **DP‑001 Schema v1.1** (ER + migration scripts)
• Unit tests (`escape_fts`, `determine_scan_root`)
• GitHub Action for SQL dry‑run | 100 % migrations green; demo prints ✅; logo badge shows schema version. | +| **Epic 1 — Live‑Watch Mode & Backup Prune** | **2025‑Q2** | Continuous indexing via FS events; backups never explode | • `marlin watch ` (inotify / FSEvents)
• `backup --prune N` (auto‑prune pre‑ and post‑command) | • **DP‑002** file‑watch life‑cycle & debounce strategy
• Change‑table schema storing dirty file IDs
• Nightly prune CI job | 8 h stress‑watch alters 10 k files → < 1 % missed; backup dir size ≤ N; watch CPU idle < 3 %. | +| **Epic 2 — Dirty‑scan optimisation** | **2025‑Q2** | Re‑index only paths marked dirty by watch table | • `scan --dirty` | • Reuse change‑table from watch; Hyperfine benchmark script committed | Dirty‑scan runtime ≤ 15 % full scan on 100 k corpus; bench job passes. | +| **Phase 3 — Content FTS + Annotations** | 2025‑Q3 | Grep snippets, inline notes | • `search -C3` grep‑style context
• `annotate add/list` | • **DP‑004** content‑blob strategy (inline vs ext‑table)
• `syntect` highlight PoC | Indexes 1 GB corpus ≤ 30 min; snippet CLI golden tests pass. | +| **Phase 4 — Versioning & De‑duplication** | 2025‑Q3 | Historic diffs, SHA‑256 dedupe | • `scan --rehash`
• `version diff ` | • **DP‑005** hash column + Bloom‑de‑dupe research | Diff on 10 MB file ≤ 500 ms; duplicate sets emitted by CLI. | +| **Phase 5 — Tag Aliases & Semantic Booster** | 2025‑Q3 | Tame tag sprawl; start AI hints | • `tag alias add/ls/rm`
• `tag suggest`, `summary` | • **DP‑006** embeddings size & k‑NN search bench | 95 % alias look‑ups resolved in one hop; suggest query ≤ 150 ms. | +| **Phase 6 — Search DSL v2 & Smart Views** | 2025‑Q4 | AND/OR, ranges, structured grammar; smart folders | • New `nom` grammar
• Legacy parser behind **`--legacy-search`** (warn on use) | • **DP‑007** BNF + 30 acceptance strings
• Lexer fuzz tests (`cargo‑fuzz`) | Old queries keep working; 0 panics in fuzz run ≥ 1 M cases. | +| **Phase 7 — Structured Workflows & Templates** | 2025‑Q4 | State graph, relationship templates | • `state set/log`
• `template apply` | • **DP‑008** workflow tables & YAML template spec
• Sample template e2e tests | Create template, apply to 20 files → all attrs/link rows present; illegal transitions blocked. | +| **Phase 8 — TUI v1 + Lightweight Integrations** | 2026‑Q1 | Keyboard UI, VS Code sidebar | • **`marlin‑tui`** binary (tiling panes, key‑map)
• Read‑only VS Code sidebar | • **DP‑009** TUI redraw budget & key‑map
• Crate split fully consumed | TUI binary ≤ 2 MB; scroll redraw ≤ 4 ms; VS Code extension loads index. | +| **Phase 9 — Dolphin Sidebar (MVP)** | 2026‑Q1 | Peek metadata inline in KDE Dolphin | • Qt/KIO sidebar | • **DP‑010** DB/IP bridge (D‑Bus vs UNIX socket)
• CMake packaging script | Sidebar opens ≤ 150 ms; passes KDE lint. | +| **Phase 10 — Full GUI & Multi‑device Sync** | 2026‑Q2 | Visual editor + optional sync backend | • Electron/Qt hybrid explorer UI
• Select & integrate sync (LiteFS / Postgres) | • **DP‑011** sync back‑end trade‑study
• Busy‑timeout/retry strategy for multi‑writer mode | CRUD round‑trip < 2 s between two nodes; 25 GUI e2e tests green. | --- -## Prerequisites +### 2 · Feature cross‑matrix (quick look‑ups) -| Requirement | Why | -| ------------------ | ----------------------------- | -| **Rust ≥ 1.77** | Build toolchain (`rustup.rs`) | -| C build essentials | Builds bundled SQLite (Linux) | - -macOS & Windows users: let the Rust installer pull the matching build tools. +| Capability | Sprint / Phase | CLI / GUI element | Linked DP | +| -------------------------- | -------------- | ----------------------------------- | --------- | +| Crate split & docs autogen | S0 | — | – | +| Tarpaulin coverage gate | S0 | — | – | +| Watch mode (FS events) | Epic 1 | `marlin watch .` | DP‑002 | +| Backup auto‑prune | Epic 1 | `backup --prune N` | – | +| Dirty‑scan | Epic 2 | `scan --dirty` | DP‑002 | +| Grep snippets | Phase 3 | `search -C3 …` | DP‑004 | +| Hash / dedupe | Phase 4 | `scan --rehash` | DP‑005 | +| Tag aliases | Phase 5 | `tag alias` commands | DP‑006 | +| Search DSL v2 | Phase 6 | new grammar, `--legacy-search` flag | DP‑007 | +| Relationship templates | Phase 7 | `template new/apply` | DP‑008 | +| TUI v1 | Phase 8 | `marlin‑tui` | DP‑009 | --- -## Build & install +## 3 · Milestone acceptance checklist -```bash -git clone https://github.com/PR0M3TH3AN/Marlin.git -cd Marlin -cargo build --release +Before a milestone is declared **shipped**: -# (Optional) install into your PATH -sudo install -Dm755 target/release/marlin /usr/local/bin/marlin -``` +* [ ] **Spec** DP‑xxx merged with schema diff, ASCII‑cast demo +* [ ] **Tests** Tarpaulin ≥ 85 % on changed lines; all suites green +* [ ] **Perf guard** script passes on CI matrix (Ubuntu 22, macOS 14) +* [ ] **Docs** auto‑regenerated; README & cheatsheet updated +* [ ] **Demo** asciinema/GIF committed and linked in release notes +* [ ] **Release tag** pushed; Cargo binary uploaded to GitHub Releases --- -## Quick start +## 4 · Next immediate actions -For a concise walkthrough—including **links, collections and views**—see -[**Quick start & Demo**](marlin_demo.md). - ---- - -## Testing - -Below is a **repeat-able 3-step flow** you can use **every time you pull fresh code**. - -### 0 Prepare once - -```bash -# Put build artefacts in one place (faster incremental builds) -export CARGO_TARGET_DIR=target -``` - -### 1 Build the new binary - -```bash -git pull -cargo build --release -sudo install -Dm755 target/release/marlin /usr/local/bin/marlin -``` - -### 2 Run the smoke-test suite - -```bash -cargo test --test e2e -- --nocapture -``` - -*Streams CLI output live; exit-code 0 = all good.* - -### 3 (Optionally) run **all** tests - -```bash -cargo test --all -- --nocapture -``` - -This now covers: - -* unit tests in `src/**` -* positive & negative integration suites (`tests/pos.rs`, `tests/neg.rs`) -* doc-tests - -#### One-liner helper - -```bash -git pull && cargo build --release && -sudo install -Dm755 target/release/marlin /usr/local/bin/marlin && -cargo test --test e2e -- --nocapture -``` - -Alias it as `marlin-ci` for a 5-second upgrade-and-verify loop. - ---- - -### Database location - -| OS | Default path | -| ----------- | ----------------------------------------------- | -| **Linux** | `~/.local/share/marlin/index.db` | -| **macOS** | `~/Library/Application Support/marlin/index.db` | -| **Windows** | `%APPDATA%\marlin\index.db` | - -Override: - -```bash -export MARLIN_DB_PATH=/path/to/custom.db -``` - ---- - -## CLI reference - -```text -marlin [ARGS] - -init create / migrate DB **and perform an initial scan of the cwd** -scan ... walk directories & (re)index files -tag "" add hierarchical tag -attr set set or update custom attribute -attr ls list attributes -link add|rm|list|backlinks manage typed file-to-file relations -coll create|add|list manage named collections (“playlists”) -view save|list|exec save and run smart views (saved queries) -search [--exec CMD] FTS5 query; optionally run CMD per hit -backup create timestamped snapshot in `backups/` -restore replace DB with snapshot -completions generate shell completions -``` - -### Attribute sub-commands - -| Command | Example | -| ----------- | ------------------------------------------------ | -| `attr set` | `marlin attr set ~/Docs/**/*.pdf reviewed yes` | -| `attr ls` | `marlin attr ls ~/Docs/report.pdf` | -| JSON output | `marlin --format=json attr ls ~/Docs/report.pdf` | - ---- - -## Backups & restore - -```bash -marlin backup -# → ~/.local/share/marlin/backups/backup_2025-05-14_22-15-30.db -``` - -```bash -marlin restore ~/.local/share/marlin/backups/backup_2025-05-14_22-15-30.db -``` - -> Marlin also creates an **automatic safety backup before every non-`init` command.** -> *Auto-prune (`backup --prune `) lands in Q2.* - ---- - -## Upgrading - -```bash -cargo install --path . --force # rebuild & replace installed binary -``` - -Versioned migrations preserve your data across upgrades. - ---- - -## License - -MIT – see [`LICENSE`](LICENSE). +| # | Task | Owner | Due | +| - | ------------------------------ | ------ | ------------- | +| 1 | Crate split + CI baseline | @alice | **26 May 25** | +| 2 | Tarpaulin + Hyperfine jobs | @bob | **26 May 25** | +| 3 | **DP‑001 Schema v1.1** draft | @carol | **30 May 25** | +| 4 | backup prune CLI + nightly job | @dave | **05 Jun 25** | diff --git a/roadmap.md b/roadmap.md index c45e256..79a2c44 100644 --- a/roadmap.md +++ b/roadmap.md @@ -1,59 +1,75 @@ -# Marlin Roadmap 2025 → 2026 📜 +# Marlin ― Delivery Road-map **v3** -This document outlines the **official delivery plan** for Marlin over the next four quarters. -Every work-item below is *time-boxed, testable,* and traceable back to an end-user benefit. +*Engineering-ready version — updated 2025-05-17* > **Legend** -> ✅ = item added/clarified in the latest planning round -> Δ = new sub-deliverable (wasn’t in the previous version) +> **△** = engineering artefact (spec / ADR / perf target)  **✦** = user-visible deliverable --- -## 1 Bird’s-eye Table +## 0 · Methodology primer (what “Done” means) -| Phase / Sprint | Timeline | Focus & Rationale | Key Deliverables (Δ = new) | | | -| ----------------------------------------------- | ------------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------ | -| **Sprint α – Bedrock & Metadata Domains** | **2025-Q2 (now → 6 Jun)** | Stabilise schema & CI; land first metadata domains with discoverability. | Δ CI: `cargo test` + SQL dry-run
Δ Unit tests (`determine_scan_root`, `escape_fts`)
Δ Coverage: e2e `attr --format=json`
Δ Refactor: move `naive_substring_search` to shared util
Migrations: `links`, `collections`, `views`
CLI stubs: `link`, `coll`, `view`
`marlin demo` walkthrough | | | -| **Epic 1 – Scale & Reliability** | 2025-Q2 | Keep scans fast; bullet-proof CI at 100 k files. | Δ Dirty-flag column + `scan --dirty`
Benchmarks: full vs dirty scan (100 k)
Replace per-row triggers with periodic rebuild
CI edge-case tests | | | -| **Epic 2 – Live Mode & Self-Pruning Backups** | 2025-Q2 | Continuous indexing & hygiene—Marlin “just works”. | Δ `marlin watch [dir]` (notify/FSEvents)
Δ `backup --prune ` + auto-prune post-scan
Daily / PR-merge prune in CI | | | -| **Phase 3 – Content FTS & Annotations** | 2025-Q3 | Index file bodies, grep-style context, inline notes. | `files.content` + migration
Extend `files_fts` (context snippets `-C`)
`annotations` table + triggers
CLI \`annotate add | list\` | | -| **Phase 4 – Versioning & Deduplication** | 2025-Q3 | History, diffs & duplicate detection. | `files.hash` (SHA-256)
`scan --rehash` refresh
CLI `version diff ` | | | -| **Phase 5 – Tag Aliases & Semantic Booster** | 2025-Q3 | Tame tag sprawl; seed AI-powered suggestions. | `canonical_id` on `tags`; CLI `tag alias …`
`embeddings` table + `scan --embed`
CLI `tag suggest`, `similarity scan`, `summary ` | | | -| **Phase 6 – Search DSL v2 & Smart Views** | 2025-Q4 | Robust grammar + virtual folders. | Replace parser with **`nom`** grammar (`AND`, `OR`, `()` …)
CLI \`view save | list | exec\` with aliases & paging | -| **Phase 7 – Structured Workflows** | 2025-Q4 | First-class task / state / reminder / event life-cycles. | ✅ State engine (`files.state`, `state_changes`)
CLI \`state set | transitions add | log`
✅ Task extractor (`tasks` table) + CLI
`templates`+ validation
CLI`remind …`, `event …`, `timeline\` | -| **Phase 8 – Lightweight Integrations** | 2026-Q1 | Surface Marlin in editors / terminal. | VS Code & TUI extension (tags / attrs / links / notes) | | | -| **Phase 9 – Dolphin Sidebar Plugin (MVP)** | 2026-Q1 | Read-only Qt sidebar for Linux file managers. | Qt plug-in: tags, attrs, links, annotations | | | -| **Phase 10 – Full Edit UI & Multi-Device Sync** | 2026-Q2 | In-place metadata editor & optional sync layer. | GUI editors (tags, views, tasks, reminders, events)
Pick/implement sync backend (rqlite, Litestream, …) | | | +| Theme | Project rule-of-thumb | +| -------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| **Branching** | Trunk-based. Feature branches → PR → 2 reviews → squash-merge. | +| **Spec first** | Every epic starts with a **Design Proposal (DP-xxx)** in `/docs/adr/`. Include schema diffs, example CLI session, perf budget. | +| **Tests** | Unit + integration coverage ≥ 85 % on lines **touched in the sprint** (checked by Tarpaulin). | +| **Perf gate** | Cold start P95 ≤ 3 s on 100 k files **unless overridden in DP**. Regressions fail CI. | +| **Docs** | CLI flags & examples land in `README.md` **same PR** that ships the code. | +| **Demo** | Closing each epic produces a 2-min asciinema or gif in `docs/demos/`. | --- -## 2 Narrative & Dependencies +## 1 · Bird’s-eye table (now includes engineering columns) -1. **Lock down core schema & demo** *(Sprint α).* - Developers get immediate feedback via the `marlin demo` command while CI ensures migrations never regress. - -2. **Scale & Live Mode** *(Epics 1-2).* - Dirty scanning, file-watching and auto-pruned backups guarantee snappy, hands-off operation even on six-figure corpora. - -3. **Richer Search** *(Phases 3-6).* - Body-content FTS + grep-style snippets lay the groundwork; `nom` grammar then elevates power-user queries and smart views. - -4. **Workflow Layers** *(Phase 7).* - State transitions, tasks and reminders turn Marlin from a passive index into an active workflow engine. - -5. **UX Expansions** *(Phases 8-10).* - Start lightweight (VS Code / TUI), graduate to a read-only Dolphin plug-in, then ship full editing & sync for multi-device teams. - -Every outer milestone depends only on the completion of the rows above it, **so shipping discipline in early sprints de-risks the headline features down the line.** +| Phase / Sprint | Timeline | Focus & Rationale | ✦ Key UX Deliverables | △ Engineering artefacts / tasks | Definition of Done | | | | +| --------------------------------------------- | ----------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------- | +| **Sprint α — Bedrock & Metadata Domains** | **2025-Q2
(now → 06 Jun)** | Lock schema, smoking-fast CI, first metadata objects. | • CLI stubs: `marlin link / coll / view`
• `marlin demo` interactive tour | • DP-001 Schema v1.1 (ER + migration scripts)
• Unit tests (`escape_fts`, `determine_scan_root`)
• GitHub Action for SQL dry-run | 100 % migrations green on CI; demo command prints green tick | | | | +| **Epic 1 — Scale & Reliability** | 2025-Q2 | Stay fast @ 100 k files | • `scan --dirty` (re-index touched rows only) | • DP-002 Dirty-flag design + FTS rebuild cadence
• Hyperfine benchmark script committed | Dirty scan vs full ≤ 15 % runtime on 100 k corpus; benchmark job passes | | | | +| **Epic 2 — Live Mode & Self-Pruning Backups** | 2025-Q2 | “Just works” indexing, DB never explodes | • `marlin watch ` (notify/FSEvents)
• `backup --prune N` & auto-prune | • DP-003 file-watcher life-cycle & debouncing
• Integration test with inotify-sim
• Cron-style GitHub job for nightly prune | 8 h stress-watch alters 10 k files < 1 % misses; backup dir ≤ N | | | | +| **Phase 3 — Content FTS + Annotations** | 2025-Q3 | Search inside files, leave notes | • Grep-style snippet output (`-C3`)
• \`marlin annotate add | list\` | • DP-004 content-blob strategy (inline vs ext-table)
• Syntax-highlight via `syntect` PoC
• New FTS triggers unit-tested | Indexes 1 GB corpus in ≤ 30 min; snippet CLI passes golden-file tests | | | +| **Phase 4 — Versioning & Deduplication** | 2025-Q3 | Historic diffs, detect dupes | • `scan --rehash` (SHA-256)
• `version diff ` | • DP-005 hash column + Bloom-de-dupe
• Binary diff adapter research | Diff on 10 MB file ≤ 500 ms; dupes listed via CLI | | | | +| **Phase 5 — Tag Aliases & Semantic Booster** | 2025-Q3 | Tame tag sprawl, start AI hints | • \`tag alias add | ls | rm`
• `tag suggest`, `summary\` | • DP-006 embeddings size & model choice
• Vector store schema + k-NN index bench | 95 % of “foo/bar\~foo” alias look-ups resolve in one hop; suggest CLI returns ≤ 150 ms | | +| **Phase 6 — Search DSL v2 & Smart Views** | 2025-Q4 | Pro-grade query language | • New `nom` grammar: AND/OR, parentheses, ranges | • DP-007 BNF + 30 acceptance strings
• Lexer fuzz-tests with `cargo-fuzz` | Old queries keep working (migration shim); 0 crashes in fuzz run ≥ 1 M cases | | | | +| **Phase 7 — Structured Workflows** | 2025-Q4 | Tasks, state, reminders, templates | • \`state set | transitions add | log`
• `task scan | list`
• **NEW:** `template apply\` for relationship templates | • DP-008 Workflow tables & validation
• Sample YAML template spec + CLI expansion tests | Create template, apply to 20 files → all attrs/link rows present; state graph denies illegal transitions | +| **Phase 8 — Lightweight Integrations** | 2026-Q1 | First “shell” GUIs | • VS Code side-bar (read-only)
• **TUI v1** (tag tree ▸ file list ▸ preview) | • DP-009 TUI key-map & redraw budget
• Crate split `marlin_core`, `marlin_tui` | TUI binary ≤ 2.0 MB; 10 k row scroll ≤ 4 ms redraw | | | | +| **Phase 9 — Dolphin Sidebar (MVP)** | 2026-Q1 | Peek metadata in KDE file-manager | • Qt-plugin showing tags, attrs, links | • DP-010 DB/IP bridge (D-Bus vs UNIX socket)
• CMake packaging script | Sidebar opens in ≤ 150 ms; passes KDE lint | | | | +| **Phase 10 — Full GUI & Multi-device Sync** | 2026-Q2 | Edit metadata visually, sync option | • Electron/Qt hybrid explorer UI
• Pick & integrate sync backend | • DP-011 sync back-end trade-study
• UI e2e tests in Playwright | Round-trip CRUD between two nodes in < 2 s; 25 GUI tests green | | | | --- -## 3 Next Steps +### 2 · Feature cross-matrix (quick look-ups) -* **Sprint α kickoff:** break deliverables into stories, estimate, assign. -* **Add roadmap as `docs/ROADMAP.md`** (this file). -* Wire a **Checklist issue** on GitHub: one task per Δ bullet for instant tracking. +| Capability | Sprint / Phase | CLI flag or GUI element | Linked DP | +| ------------------------------------- | -------------- | ---------------------------------- | --------- | +| Relationship **templates** | P7 | `template new`, `template apply` | DP-008 | +| Positive / negative filter combinator | P6 | DSL `+tag:foo -tag:bar date>=2025` | DP-007 | +| Dirty-scan optimisation | E1 | `scan --dirty` | DP-002 | +| Watch-mode | E2 | `marlin watch .` | DP-003 | +| Grep snippets | P3 | `search -C3 "foo"` | DP-004 | +| Hash / dedupe | P4 | `scan --rehash` | DP-005 | --- -*Last updated · 2025-05-16* +## 3 · Milestone acceptance checklist + +Before a milestone is declared “shipped”: + +* [ ] **Spec** merged (DP-xxx) with schema diff & example ASCII-cast +* [ ] **Unit & integration tests** ≥ 85 % coverage on changed lines +* [ ] **Perf guard-rail** script passes on CI matrix (Ubuntu 22, macOS 14) +* [ ] **Docs** — CLI man-page, README table row, roadmap ticked +* [ ] **Demo** uploaded to `docs/demos/` and linked in release notes +* [ ] **Release tag** pushed; Cargo binary on GitHub Releases + +--- + +### 4 · Next immediate actions + +1. **Write DP-001 (Schema v1.1)** — owner @alice, due 21 May +2. **Set up Tarpaulin & Hyperfine jobs** — @bob, due 23 May +3. **Spike dirty-flag logic** — @carol 2 days time-box, outcome in DP-002 + +--- + +> *This roadmap now contains both product-level “what” and engineering-level “how/when/prove it”. It should allow a new contributor to jump in, pick the matching DP, and know exactly the bar they must clear for their code to merge.* diff --git a/spec-details/TUI+Query-DSL+Templates.md b/spec-details/TUI+Query-DSL+Templates.md new file mode 100644 index 0000000..858e02a --- /dev/null +++ b/spec-details/TUI+Query-DSL+Templates.md @@ -0,0 +1,200 @@ +# Marlin — TUI + Query-DSL + Templates + +**Integration Specification · v0.1 (2025-05-18)** + +--- + +## 0 · Scope + +This document turns the **concept sketch** (suck-less TUI) **plus** the new +*relationship-template* and *positive/negative filter* ideas into a concrete +development plan that extends the existing Marlin code-base without breaking +CLI workflows. + +--- + +## 1 · Feature Set + +| Epic | Deliverable | Short Description | +| -------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------ | +| **T-1** | **Marlin-TUI** binary | One-screen, tiling, key-centred interface (sketch in §2). | +| **Q-1** | **Query DSL v2** | Lexer + parser + planner supporting `AND/OR`, `NOT`, parentheses, ranges, comparison ops on `size`, `mtime`… | +| **T-2** | **Relationship templates** | Named templates with typed fields (key + value-type) that auto-populate attributes when applied. | +| **Glue** | Shared **`libmarlin`** crate | Re-export public APIs so both CLI and TUI call the same Rust functions. | + +These map cleanly onto the public roadmap: + +* T-1 → *Phase 6: Search DSL v2 & Smart Views* (UI portion) +* Q-1 → *Phase 6* (parser) +* T-2 → *Phase 7: Structured Workflows* (templates subsection) + +--- + +## 2 · TUI Reference Layout & Key-Map + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Marlin ▸ ~/demo (q quit · ? help) │ +├───────────────┬─────────────────────────────────────────────┤ +│ Tags / Views │ Path / Name │ ⓘ Attributes │ +│───────────────│────────────────────────┼────────────────────│ +│ project │ api/design.md │ status=pending │ +│ ├ alpha │ roadmap.xlsx │ reviewed=yes │ +│ │ ├ draft │ ... │ tags=project/alpha │ +│ │ └ final │ (42/812 shown) │ links: 2 outgoing │ +│ │ grep: "(?i)todo" │ ───── Preview ───── │ +│ • urgent │ │ - [ ] TODO write… │ +└───────────────┴────────────────────────┴────────────────────┘ +``` + +### Essential Keys + +| Key | Effect | CLI equivalence | +| --------------- | ------------------------- | -------------------------------- | +| `/expr⏎` | Regex/FTS filter | `marlin search` | +| `;key=val⏎` | Attr filter | `marlin search attr:key=val` | +| `:` | Command prompt | raw CLI passthrough | +| `Space` | Mark/unmark | adds to in-memory selection list | +| `:tag foo/bar⏎` | Tag marked files | `marlin tag` | +| `Enter` | open file in `$EDITOR` | — | +| `Tab` | toggle right preview pane | — | + +Full table lives in **`docs/tui_keys.md`** (auto-generated from +`src/tui/keymap.rs`). + +--- + +## 3 · Architecture Impact + +### 3.1 Crate refactor (Week 1) + +``` +marlin/ +├── cli-bin/ ← binary crate (keeps current main.rs) +├── tui-bin/ ← NEW binary crate (Marlin-TUI) +├── libmarlin/ ← NEW library: DB helpers, scan, query, templates +└── Cargo.toml +``` + +*Export surface*: + +```rust +pub struct Marlin { /* connection, config … */ } +impl Marlin { + pub fn search(&self, q: Query) -> Result>; + pub fn tag(&self, files: &[PathBuf], tag: &str) -> Result<()>; + pub fn apply_template(&self, files: &[PathBuf], tpl: &Template) -> Result<()>; + /* … */ +} +``` + +CLI keeps 100 % of its flags; it just calls `libmarlin`. + +TUI links against the same crate → zero business-logic duplication. + +--- + +### 3.2 Query-DSL v2 (Week 2-3) + +* **Lexer:** [`logos`](https://crates.io/crates/logos) (<500 LOC). +* **Parser:** `nom` (top-down precedence climbing). +* **AST → Plan:** + + ```rust + struct Plan { + fts_match: Option, // tags_text/path/attrs_text + sql_where: Vec, // files.size > ?, files.mtime BETWEEN ? ? + params: Vec, + } + ``` +* **Planner** decides when to fall back to pure SQL (e.g. `size>1M AND NOT tag:archive` uses an FTS sub-query joined on rowid). + +Backwards compatibility: old space-separated syntax is parsed first; if it fails, +DSL v2 is attempted, so no user breakage. + +--- + +### 3.3 Templates (Week 4) + +**Schema (migration 0005):** + +```sql +CREATE TABLE templates ( + id INTEGER PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + description TEXT +); +CREATE TABLE template_fields ( + id INTEGER PRIMARY KEY, + template_id INTEGER NOT NULL REFERENCES templates(id), + key TEXT NOT NULL, + value_type TEXT NOT NULL -- 'text' | 'int' | 'date' +); +``` + +**CLI additions** + +``` +marlin template list|show|add|rm +marlin template apply +``` + +`Templater` helper validates type coercion before inserting into `attributes`. + +TUI: `:apply ` applies to marked files and prompts inline for field values +(stenographic prompt à la `git commit --template`). + +--- + +## 4 · Implementation Phases & Timeline + +| Week | Milestone / Task | Owner | Exit Criteria | +| ---- | ------------------------------------------ | -------- | ------------------------- | +| 1 | Split repo into `libmarlin`, adapt CLI | core dev | `cargo test --all` green | +| 2 | Lexer + unit tests | core dev | 95 % coverage | +| 3 | Parser + Planner; replace `run_search()` | core dev | DSL green tests pass | +| 4 | Template schema, CLI, tests | DB dev | `template apply` e2e test | +| 5 | **TUI MVP**
draw loop + panes + key nav | UI dev | Can browse & open files | +| 6 | TUI ↔ Marlin API glue
mark, tag, search | UI dev | All key-map smoke tests | +| 7 | Preview cache, file-watch refresh | UI dev | CPU < 5 % idle load | +| 8 | Docs refresh, release notes | all | v0.2.0 tag | + +--- + +## 5 · Testing & CI + +* **DSL golden-file tests** (`tests/dsl/*.txt` → expected SQL). +* **SQLite snapshot tests** for templates. +* **TUI headless tests** via `termwiz`’s `TerminalTest` harness. +* GitHub Actions matrix unchanged; new crate just builds in. + +--- + +## 6 · Risks & Mitigations + +| Risk | Impact | Mitigation | +| ---------------------------- | -------------- | ------------------------------------------------------------ | +| Parser ambiguity | broken queries | formal grammar + fuzzy tests | +| TUI performance on huge dirs | lag | incremental drawing + LRU file stat cache | +| Schema lock-in for templates | hard to evolve | keep `value_type` generic (text) until needs prove otherwise | + +--- + +## 7 · What Stays Back-Compatible? + +* **CLI syntax**: existing commands & search strings still work. +* **Database**: 0005 migration is additive. +* **Automation scripts**: unchanged; TUI is *optional* (`marlin-tui` binary). + +--- + +## 8 · Why It’s Worth It + +* **Differentiator**: Few local-first tools combine rich metadata, SQL-grade + filtering, and both script-friendly CLI *and* keyboard-driven TUI. +* **Gateway to GUI**: The API we expose for TUI is the same one the + future GTK/Qt desktop app can use. + +--- + +**Let the hacking begin!**