mirror of
https://github.com/PR0M3TH3AN/Marlin.git
synced 2025-09-08 07:08:44 +00:00
update
This commit is contained in:
thePR0M3TH3AN
183
docs/marlin_demo.md
Normal file
183
docs/marlin_demo.md
Normal file
@@ -0,0 +1,183 @@
|
||||
# Marlin Demo 🚀
|
||||
|
||||
Below is a **“hello-world” walk-through** that matches the current `main`
|
||||
branch (auto-scan on `marlin init`, no more forced-migration chatter, cleaner
|
||||
build). Everything runs offline on a throw-away directory under `~/marlin_demo`.
|
||||
|
||||
---
|
||||
|
||||
## 0 Build & install Marlin
|
||||
|
||||
```bash
|
||||
# inside the repo
|
||||
export CARGO_TARGET_DIR=target # <-- speeds up future builds (once)
|
||||
cargo build --release # build the new binary
|
||||
sudo install -Dm755 target/release/marlin /usr/local/bin/marlin
|
||||
# (cargo install --path . --locked --force works too)
|
||||
````
|
||||
|
||||
---
|
||||
|
||||
## 1 Create the demo tree
|
||||
|
||||
```bash
|
||||
rm -rf ~/marlin_demo
|
||||
mkdir -p ~/marlin_demo/{Projects/{Alpha,Beta,Gamma},Logs,Reports,Scripts,Media/Photos}
|
||||
# (zsh users: quote the pattern or enable braceexpand first)
|
||||
|
||||
# ── Projects ───────────────────────────────────────────────────
|
||||
cat <<EOF > ~/marlin_demo/Projects/Alpha/draft1.md
|
||||
# Alpha draft 1
|
||||
- [ ] TODO: outline architecture
|
||||
- [ ] TODO: write tests
|
||||
EOF
|
||||
cat <<EOF > ~/marlin_demo/Projects/Alpha/draft2.md
|
||||
# Alpha draft 2
|
||||
- [x] TODO: outline architecture
|
||||
- [ ] TODO: implement feature X
|
||||
EOF
|
||||
cat <<EOF > ~/marlin_demo/Projects/Beta/notes.md
|
||||
Beta meeting notes:
|
||||
|
||||
- decided on roadmap
|
||||
- ACTION: follow-up with design team
|
||||
EOF
|
||||
cat <<EOF > ~/marlin_demo/Projects/Beta/final.md
|
||||
# Beta Final
|
||||
All tasks complete. Ready to ship!
|
||||
EOF
|
||||
cat <<EOF > ~/marlin_demo/Projects/Gamma/TODO.txt
|
||||
Gamma tasks:
|
||||
TODO: refactor module Y
|
||||
EOF
|
||||
|
||||
# ── Logs & Reports ─────────────────────────────────────────────
|
||||
echo "2025-05-15 12:00:00 INFO Starting app" > ~/marlin_demo/Logs/app.log
|
||||
echo "2025-05-15 12:01:00 ERROR Oops, crash" >> ~/marlin_demo/Logs/app.log
|
||||
echo "2025-05-15 00:00:00 INFO System check OK" > ~/marlin_demo/Logs/system.log
|
||||
printf "Q1 financials\n" > ~/marlin_demo/Reports/Q1_report.pdf
|
||||
|
||||
# ── Scripts & Media ────────────────────────────────────────────
|
||||
cat <<'EOF' > ~/marlin_demo/Scripts/deploy.sh
|
||||
#!/usr/bin/env bash
|
||||
echo "Deploying version $1…"
|
||||
EOF
|
||||
chmod +x ~/marlin_demo/Scripts/deploy.sh
|
||||
echo "JPEGDATA" > ~/marlin_demo/Media/Photos/event.jpg
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2 Initialise **and** index (one step)
|
||||
|
||||
```bash
|
||||
cd ~/marlin_demo # run init from the folder you want indexed
|
||||
marlin init # • creates or migrates DB
|
||||
# • runs *first* full scan of this directory
|
||||
```
|
||||
|
||||
Add more directories later with `marlin scan <dir>`.
|
||||
|
||||
---
|
||||
|
||||
## 3 Tagging examples
|
||||
|
||||
```bash
|
||||
# Tag all project markdown as ‘project/md’
|
||||
marlin tag '~/marlin_demo/Projects/**/*.md' project/md
|
||||
|
||||
# Tag your logs
|
||||
marlin tag '~/marlin_demo/Logs/**/*.log' logs/app
|
||||
|
||||
# Tag everything under Beta as ‘project/beta’
|
||||
marlin tag '~/marlin_demo/Projects/Beta/**/*' project/beta
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4 Set custom attributes
|
||||
|
||||
```bash
|
||||
marlin attr set '~/marlin_demo/Projects/Beta/final.md' status complete
|
||||
marlin attr set '~/marlin_demo/Reports/*.pdf' reviewed yes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5 Play with search / exec hooks
|
||||
|
||||
```bash
|
||||
marlin search TODO
|
||||
marlin search tag:project/md
|
||||
marlin search 'tag:logs/app AND ERROR'
|
||||
marlin search 'attr:status=complete'
|
||||
marlin search 'attr:reviewed=yes AND pdf'
|
||||
marlin search 'attr:reviewed=yes' --exec 'xdg-open {}'
|
||||
marlin --format=json search 'attr:status=complete' # machine-readable output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6 Verbose mode
|
||||
|
||||
```bash
|
||||
marlin --verbose scan ~/marlin_demo # watch debug logs stream by
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7 Snapshot & restore
|
||||
|
||||
```bash
|
||||
snap=$(marlin backup | awk '{print $NF}')
|
||||
rm ~/.local/share/marlin/index.db # simulate disaster
|
||||
marlin restore "$snap"
|
||||
marlin search TODO # still works
|
||||
```
|
||||
|
||||
*(Reminder: Marlin also makes an **auto-backup** before every non-`init`
|
||||
command, so manual snapshots are extra insurance.)*
|
||||
|
||||
---
|
||||
|
||||
## 8 Linking demo
|
||||
|
||||
```bash
|
||||
touch ~/marlin_demo/foo.txt ~/marlin_demo/bar.txt
|
||||
marlin scan ~/marlin_demo # index the new files
|
||||
|
||||
foo=~/marlin_demo/foo.txt
|
||||
bar=~/marlin_demo/bar.txt
|
||||
|
||||
marlin link add "$foo" "$bar" --type references # create typed link
|
||||
marlin link list "$foo" # outgoing links from foo
|
||||
marlin link backlinks "$bar" # incoming links to bar
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9 Collections & smart views
|
||||
|
||||
```bash
|
||||
# Collection
|
||||
marlin coll create SetA
|
||||
marlin coll add SetA '~/marlin_demo/Projects/**/*.md'
|
||||
marlin coll list SetA
|
||||
|
||||
# Saved view (smart folder)
|
||||
marlin view save tasks 'attr:status=complete OR TODO'
|
||||
marlin view exec tasks
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Recap
|
||||
|
||||
* `cargo build --release` + `sudo install …` is still the build path.
|
||||
* **`marlin init`** scans the **current working directory** on first run.
|
||||
* Scan again only when you add *new* directories (`marlin scan …`).
|
||||
* Auto-backups happen before every command; manual `marlin backup` gives you extra restore points.
|
||||
|
||||
Happy organising!
|
||||
|
||||
```
|
||||
74
docs/roadmap.md
Normal file
74
docs/roadmap.md
Normal file
@@ -0,0 +1,74 @@
|
||||
# Marlin ― Delivery Road‑map **v3**
|
||||
|
||||
*Engineering‑ready version — updated 2025‑05‑17*
|
||||
|
||||
> **Legend**
|
||||
> **△** = engineering artefact (spec / ADR / perf target) **✦** = user-visible deliverable
|
||||
|
||||
---
|
||||
|
||||
## 0 · Methodology primer (what “Done” means)
|
||||
|
||||
| Theme | Project rule-of-thumb |
|
||||
| -------------- | -------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Branching** | Trunk-based. Feature branches → PR → 2 reviews → squash-merge. |
|
||||
| **Spec first** | Every epic starts with a **Design Proposal (DP-xxx)** in `/docs/adr/`. Include schema diffs, example CLI session, perf budget. |
|
||||
| **Tests** | Unit + integration coverage ≥ 85 % on lines **touched in the sprint** (checked by Tarpaulin). |
|
||||
| **Perf gate** | Cold start P95 ≤ 3 s on 100 k files **unless overridden in DP**. Regressions fail CI. |
|
||||
| **Docs** | CLI flags & examples land in `README.md` **same PR** that ships the code. |
|
||||
| **Demo** | Closing each epic produces a 2-min asciinema or gif in `docs/demos/`. |
|
||||
|
||||
---
|
||||
|
||||
## 1 · Bird’s‑eye table (now includes engineering columns)
|
||||
|
||||
| Phase / Sprint | Timeline | Focus & Rationale | ✦ Key UX Deliverables | △ Engineering artefacts / tasks | Definition of Done |
|
||||
| --------------------------------------------- | -------- | ---------------------------------------- | -------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
|
||||
| **Epic 1 — Scale & Reliability** | 2025-Q2 | Stay fast @ 100 k files | • `scan --dirty` (re-index touched rows only) | • DP-002 Dirty-flag design + FTS rebuild cadence<br>• Hyperfine benchmark script committed | Dirty scan vs full ≤ 15 % runtime on 100 k corpus; benchmark job passes |
|
||||
| **Epic 2 — Live Mode & Self‑Pruning Backups** | 2025-Q2 | “Just works” indexing, DB never explodes | • `marlin watch <dir>` (notify/FSEvents)<br>• `backup --prune N` & auto-prune | • DP-003 file-watcher life-cycle & debouncing<br>• Integration test with inotify-sim <br>• Cron-style GitHub job for nightly prune | 8 h stress-watch alters 10 k files < 1 % misses; backup dir ≤ N |
|
||||
| **Phase 3 — Content FTS + Annotations** | 2025-Q3 | Search inside files, leave notes | • Grep-style snippet output (`-C3`)<br>• `marlin annotate add/list` | • DP-004 content-blob strategy (inline vs ext-table)<br>• Syntax-highlight via `syntect` PoC<br>• New FTS triggers unit-tested | Indexes 1 GB corpus in ≤ 30 min; snippet CLI passes golden-file tests |
|
||||
| **Phase 4 — Versioning & Deduplication** | 2025-Q3 | Historic diffs, detect dupes | • `scan --rehash` (SHA-256)<br>• `version diff <file>` | • DP-005 hash column + Bloom-de-dupe<br>• Binary diff adapter research | Diff on 10 MB file ≤ 500 ms; dupes listed via CLI |
|
||||
| **Phase 5 — Tag Aliases & Semantic Booster** | 2025-Q3 | Tame tag sprawl, start AI hints | • `tag alias add/ls/rm`<br>• `tag suggest`, `summary` | • DP-006 embeddings size & model choice<br>• Vector store schema + k-NN index bench | 95 % of “foo/bar\~foo” alias look-ups resolve in one hop; suggest CLI returns ≤ 150 ms |
|
||||
| **Phase 6 — Search DSL v2 & Smart Views** | 2025-Q4 | Pro-grade query language | • New `nom` grammar: AND/OR, parentheses, ranges | • DP-007 BNF + 30 acceptance strings<br>• Lexer fuzz-tests with `cargo-fuzz` | Old queries keep working (migration shim); 0 crashes in fuzz run ≥ 1 M cases |
|
||||
| **Phase 7 — Structured Workflows** | 2025-Q4 | Tasks, state, reminders, templates | • `state set/transitions add/log`<br>• `task scan/list`<br>• **NEW:** `template apply` | • DP-008 Workflow tables & validation<br>• Sample YAML template spec + CLI expansion tests | Create template, apply to 20 files → all attrs/link rows present; state graph denies illegal transitions |
|
||||
| **Phase 8 — Lightweight Integrations** | 2026-Q1 | First “shell” GUIs | • VS Code side-bar (read-only)<br>• **TUI v1** (tag tree ▸ file list ▸ preview) | • DP-009 TUI key-map & redraw budget<br>• Crate split `marlin_core`, `marlin_tui` | TUI binary ≤ 2.0 MB; 10 k row scroll ≤ 4 ms redraw |
|
||||
| **Phase 9 — Dolphin Sidebar (MVP)** | 2026-Q1 | Peek metadata in KDE file-manager | • Qt-plugin showing tags, attrs, links | • DP-010 DB/IP bridge (D‑Bus vs UNIX socket)<br>• CMake packaging script | Sidebar opens in ≤ 150 ms; passes KDE lint |
|
||||
| **Phase 10 — Full GUI & Multi-device Sync** | 2026-Q2 | Edit metadata visually, sync option | • Electron/Qt hybrid explorer UI<br>• Pick & integrate sync backend | • DP-011 sync back-end trade-study<br>• UI e2e tests in Playwright | Round-trip CRUD between two nodes in < 2 s; 25 GUI tests green |
|
||||
|
||||
---
|
||||
|
||||
### 2 · Feature cross-matrix (quick look-ups)
|
||||
|
||||
| Capability | Sprint / Phase | CLI flag or GUI element | Linked DP |
|
||||
| ------------------------------------- | -------------- | ---------------------------------- | --------- |
|
||||
| Relationship **templates** | P7 | `template new`, `template apply` | DP-008 |
|
||||
| Positive / negative filter combinator | P6 | DSL `+tag:foo -tag:bar date>=2025` | DP-007 |
|
||||
| Dirty-scan optimisation | E1 | `scan --dirty` | DP-002 |
|
||||
| Watch-mode | E2 | `marlin watch .` | DP-003 |
|
||||
| Grep snippets | P3 | `search -C3 "foo"` | DP-004 |
|
||||
| Hash / dedupe | P4 | `scan --rehash` | DP-005 |
|
||||
|
||||
---
|
||||
|
||||
## 3 · Milestone acceptance checklist
|
||||
|
||||
Before a milestone is declared “shipped”:
|
||||
|
||||
* [ ] **Spec** merged (DP-xxx) with schema diff & example ASCII-cast
|
||||
* [ ] **Unit & integration tests** ≥ 85 % coverage on changed lines
|
||||
* [ ] **Perf guard-rail** script passes on CI matrix (Ubuntu 22, macOS 14)
|
||||
* [ ] **Docs** — CLI man-page, README table row, roadmap ticked
|
||||
* [ ] **Demo** uploaded to `docs/demos/` and linked in release notes
|
||||
* [ ] **Release tag** pushed; Cargo binary on GitHub Releases
|
||||
|
||||
---
|
||||
|
||||
### 4 · Next immediate actions
|
||||
|
||||
1. **Write DP-001 (Schema v1.1)** — owner @alice, due 21 May
|
||||
2. **Set up Tarpaulin & Hyperfine jobs** — @bob, due 23 May
|
||||
3. **Spike dirty-flag logic** — @carol 2 days time-box, outcome in DP-002
|
||||
|
||||
---
|
||||
|
||||
> *This roadmap now contains both product-level “what” and engineering-level “how/when/prove it”. It should allow a new contributor to jump in, pick the matching DP, and know exactly the bar they must clear for their code to merge.*
|
||||
200
docs/spec-details/TUI+Query-DSL+Templates.md
Normal file
200
docs/spec-details/TUI+Query-DSL+Templates.md
Normal file
@@ -0,0 +1,200 @@
|
||||
# Marlin — TUI + Query-DSL + Templates
|
||||
|
||||
**Integration Specification · v0.1 (2025-05-18)**
|
||||
|
||||
---
|
||||
|
||||
## 0 · Scope
|
||||
|
||||
This document turns the **concept sketch** (suck-less TUI) **plus** the new
|
||||
*relationship-template* and *positive/negative filter* ideas into a concrete
|
||||
development plan that extends the existing Marlin code-base without breaking
|
||||
CLI workflows.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Feature Set
|
||||
|
||||
| Epic | Deliverable | Short Description |
|
||||
| -------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------ |
|
||||
| **T-1** | **Marlin-TUI** binary | One-screen, tiling, key-centred interface (sketch in §2). |
|
||||
| **Q-1** | **Query DSL v2** | Lexer + parser + planner supporting `AND/OR`, `NOT`, parentheses, ranges, comparison ops on `size`, `mtime`… |
|
||||
| **T-2** | **Relationship templates** | Named templates with typed fields (key + value-type) that auto-populate attributes when applied. |
|
||||
| **Glue** | Shared **`libmarlin`** crate | Re-export public APIs so both CLI and TUI call the same Rust functions. |
|
||||
|
||||
These map cleanly onto the public roadmap:
|
||||
|
||||
* T-1 → *Phase 6: Search DSL v2 & Smart Views* (UI portion)
|
||||
* Q-1 → *Phase 6* (parser)
|
||||
* T-2 → *Phase 7: Structured Workflows* (templates subsection)
|
||||
|
||||
---
|
||||
|
||||
## 2 · TUI Reference Layout & Key-Map
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Marlin ▸ ~/demo (q quit · ? help) │
|
||||
├───────────────┬─────────────────────────────────────────────┤
|
||||
│ Tags / Views │ Path / Name │ ⓘ Attributes │
|
||||
│───────────────│────────────────────────┼────────────────────│
|
||||
│ project │ api/design.md │ status=pending │
|
||||
│ ├ alpha │ roadmap.xlsx │ reviewed=yes │
|
||||
│ │ ├ draft │ ... │ tags=project/alpha │
|
||||
│ │ └ final │ (42/812 shown) │ links: 2 outgoing │
|
||||
│ <Views> │ grep: "(?i)todo" │ ───── Preview ───── │
|
||||
│ • urgent │ │ - [ ] TODO write… │
|
||||
└───────────────┴────────────────────────┴────────────────────┘
|
||||
```
|
||||
|
||||
### Essential Keys
|
||||
|
||||
| Key | Effect | CLI equivalence |
|
||||
| --------------- | ------------------------- | -------------------------------- |
|
||||
| `/expr⏎` | Regex/FTS filter | `marlin search` |
|
||||
| `;key=val⏎` | Attr filter | `marlin search attr:key=val` |
|
||||
| `:` | Command prompt | raw CLI passthrough |
|
||||
| `Space` | Mark/unmark | adds to in-memory selection list |
|
||||
| `:tag foo/bar⏎` | Tag marked files | `marlin tag` |
|
||||
| `Enter` | open file in `$EDITOR` | — |
|
||||
| `Tab` | toggle right preview pane | — |
|
||||
|
||||
Full table lives in **`docs/tui_keys.md`** (auto-generated from
|
||||
`src/tui/keymap.rs`).
|
||||
|
||||
---
|
||||
|
||||
## 3 · Architecture Impact
|
||||
|
||||
### 3.1 Crate refactor (Week 1)
|
||||
|
||||
```
|
||||
marlin/
|
||||
├── cli-bin/ ← binary crate (keeps current main.rs)
|
||||
├── tui-bin/ ← NEW binary crate (Marlin-TUI)
|
||||
├── libmarlin/ ← NEW library: DB helpers, scan, query, templates
|
||||
└── Cargo.toml
|
||||
```
|
||||
|
||||
*Export surface*:
|
||||
|
||||
```rust
|
||||
pub struct Marlin { /* connection, config … */ }
|
||||
impl Marlin {
|
||||
pub fn search(&self, q: Query) -> Result<Vec<PathBuf>>;
|
||||
pub fn tag(&self, files: &[PathBuf], tag: &str) -> Result<()>;
|
||||
pub fn apply_template(&self, files: &[PathBuf], tpl: &Template) -> Result<()>;
|
||||
/* … */
|
||||
}
|
||||
```
|
||||
|
||||
CLI keeps 100 % of its flags; it just calls `libmarlin`.
|
||||
|
||||
TUI links against the same crate → zero business-logic duplication.
|
||||
|
||||
---
|
||||
|
||||
### 3.2 Query-DSL v2 (Week 2-3)
|
||||
|
||||
* **Lexer:** [`logos`](https://crates.io/crates/logos) (<500 LOC).
|
||||
* **Parser:** `nom` (top-down precedence climbing).
|
||||
* **AST → Plan:**
|
||||
|
||||
```rust
|
||||
struct Plan {
|
||||
fts_match: Option<String>, // tags_text/path/attrs_text
|
||||
sql_where: Vec<String>, // files.size > ?, files.mtime BETWEEN ? ?
|
||||
params: Vec<SqlArg>,
|
||||
}
|
||||
```
|
||||
* **Planner** decides when to fall back to pure SQL (e.g. `size>1M AND NOT tag:archive` uses an FTS sub-query joined on rowid).
|
||||
|
||||
Backwards compatibility: old space-separated syntax is parsed first; if it fails,
|
||||
DSL v2 is attempted, so no user breakage.
|
||||
|
||||
---
|
||||
|
||||
### 3.3 Templates (Week 4)
|
||||
|
||||
**Schema (migration 0005):**
|
||||
|
||||
```sql
|
||||
CREATE TABLE templates (
|
||||
id INTEGER PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
description TEXT
|
||||
);
|
||||
CREATE TABLE template_fields (
|
||||
id INTEGER PRIMARY KEY,
|
||||
template_id INTEGER NOT NULL REFERENCES templates(id),
|
||||
key TEXT NOT NULL,
|
||||
value_type TEXT NOT NULL -- 'text' | 'int' | 'date'
|
||||
);
|
||||
```
|
||||
|
||||
**CLI additions**
|
||||
|
||||
```
|
||||
marlin template list|show|add|rm
|
||||
marlin template apply <name> <glob>
|
||||
```
|
||||
|
||||
`Templater` helper validates type coercion before inserting into `attributes`.
|
||||
|
||||
TUI: `:apply <tpl>` applies to marked files and prompts inline for field values
|
||||
(stenographic prompt à la `git commit --template`).
|
||||
|
||||
---
|
||||
|
||||
## 4 · Implementation Phases & Timeline
|
||||
|
||||
| Week | Milestone / Task | Owner | Exit Criteria |
|
||||
| ---- | ------------------------------------------ | -------- | ------------------------- |
|
||||
| 1 | Split repo into `libmarlin`, adapt CLI | core dev | `cargo test --all` green |
|
||||
| 2 | Lexer + unit tests | core dev | 95 % coverage |
|
||||
| 3 | Parser + Planner; replace `run_search()` | core dev | DSL green tests pass |
|
||||
| 4 | Template schema, CLI, tests | DB dev | `template apply` e2e test |
|
||||
| 5 | **TUI MVP**<br>draw loop + panes + key nav | UI dev | Can browse & open files |
|
||||
| 6 | TUI ↔ Marlin API glue<br>mark, tag, search | UI dev | All key-map smoke tests |
|
||||
| 7 | Preview cache, file-watch refresh | UI dev | CPU < 5 % idle load |
|
||||
| 8 | Docs refresh, release notes | all | v0.2.0 tag |
|
||||
|
||||
---
|
||||
|
||||
## 5 · Testing & CI
|
||||
|
||||
* **DSL golden-file tests** (`tests/dsl/*.txt` → expected SQL).
|
||||
* **SQLite snapshot tests** for templates.
|
||||
* **TUI headless tests** via `termwiz`’s `TerminalTest` harness.
|
||||
* GitHub Actions matrix unchanged; new crate just builds in.
|
||||
|
||||
---
|
||||
|
||||
## 6 · Risks & Mitigations
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
| ---------------------------- | -------------- | ------------------------------------------------------------ |
|
||||
| Parser ambiguity | broken queries | formal grammar + fuzzy tests |
|
||||
| TUI performance on huge dirs | lag | incremental drawing + LRU file stat cache |
|
||||
| Schema lock-in for templates | hard to evolve | keep `value_type` generic (text) until needs prove otherwise |
|
||||
|
||||
---
|
||||
|
||||
## 7 · What Stays Back-Compatible?
|
||||
|
||||
* **CLI syntax**: existing commands & search strings still work.
|
||||
* **Database**: 0005 migration is additive.
|
||||
* **Automation scripts**: unchanged; TUI is *optional* (`marlin-tui` binary).
|
||||
|
||||
---
|
||||
|
||||
## 8 · Why It’s Worth It
|
||||
|
||||
* **Differentiator**: Few local-first tools combine rich metadata, SQL-grade
|
||||
filtering, and both script-friendly CLI *and* keyboard-driven TUI.
|
||||
* **Gateway to GUI**: The API we expose for TUI is the same one the
|
||||
future GTK/Qt desktop app can use.
|
||||
|
||||
---
|
||||
|
||||
**Let the hacking begin!**
|
||||
186
docs/vision.md
Normal file
186
docs/vision.md
Normal file
@@ -0,0 +1,186 @@
|
||||
# Marlin – Metadata‑Driven File Explorer
|
||||
|
||||
*Version 2 – 12 May 2025*
|
||||
|
||||
---
|
||||
|
||||
## 1 Key Features & Functionality
|
||||
|
||||
| Feature Area | Capabilities |
|
||||
| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Tagging System** | • Unlimited, hierarchical or flat tags.<br>• Alias/synonym support with precedence rules (admin‑defined canonical name).<br>• **Bulk tag editing** via multi‑select context menu.<br>• Folder‑to‑Tag import with optional *watch & sync* mode so new sub‑folders inherit tags automatically. |
|
||||
| **Custom Metadata Attributes** | • User‑defined fields (text, number, date, enum, boolean).<br>• Per‑template **Custom Metadata Schemas** (e.g. *Photo* → *Date, Location*). |
|
||||
| **File Relationships** | • Typed, directional or bidirectional links (*related to*, *duplicate of*, *cites*…).<br>• Plugin API can register new relationship sets. |
|
||||
| **Version Control for Metadata** | • Every change logged; unlimited roll‑back.<br>• Side‑by‑side diff viewer and *blame* panel showing *who/when/what*.<br>• Offline edits stored locally and merged (Git‑style optimistic merge with conflict prompts). |
|
||||
| **Advanced Search & Smart Folders** | • Structured query syntax: `tag:ProjectX AND author:Alice`.<br>• Natural‑language search (*"files Alice edited last month"*) with toggle to exact mode.<br>• Visual Query Builder showing live query string.<br>• Saved queries appear as virtual “smart folders” that update in real‑time. |
|
||||
| **User Interface** | • Sidebar: tags, attributes, relationships.<br>• Drag‑and‑drop tagging; inline metadata editor.<br>• Search bar with auto‑complete (Bloom filter backed).<br>• **Dual View Mode** – metadata vs traditional folder; remembers preference per location.<br>• **Interactive 60‑second tour** on first launch plus contextual tooltip help. |
|
||||
| **Collaboration** | • Real‑time metadata sync across devices via cloud or self‑hosted relay.<br>• Conflict handling as per Version Control.<br>• Role‑based permissions (read / write / admin) on tags & attributes. |
|
||||
| **Performance & Scale** | • Sharded/distributed index optional for >1 M files.<br>• Query cache with LRU eviction.<br>• Target metrics (100 k files): cold start ≤ 3 s, complex query ≤ 150 ms (stretch 50 ms). |
|
||||
| **Backup & Restore** | • Scheduled encrypted backups; export to JSON / XML.<br>• One‑click restore from any point‑in‑time snapshot. |
|
||||
| **Extensibility** | • Plug‑in system (TypeScript/JS) – see §2.4.<br>• Python scripting hook for automation and batch tasks.<br>• REST/IPC API for external tools. |
|
||||
|
||||
---
|
||||
|
||||
## 2 Technical Implementation
|
||||
|
||||
### 2.1 Core Stack
|
||||
|
||||
| Component | Primary Choice | Notes |
|
||||
| -------------- | -------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
|
||||
| File Manager | **Dolphin (KDE)** KIO‑based plug‑ins | GTK users can install a Nautilus extension (feature‑parity subset). |
|
||||
| Metadata Store | **SQLite + FTS5** (single‑user) → optional **LiteFS/Postgres** for replication & multi‑user scale. | Per‑row AES‑GCM encryption for sensitive fields; keys stored in OS keyring. |
|
||||
| Indexer Daemon | Rust service using `notify` (inotify on Linux, FSEvents on macOS). | 100 ms debounce batches, async SQLite writes. |
|
||||
| Cache | In‑memory LRU + Bloom filter for auto‑complete. | |
|
||||
|
||||
### 2.2 Database Schema (simplified)
|
||||
|
||||
```text
|
||||
files(id PK, path, inode, size, mtime, ctime, hash)
|
||||
tags(id PK, name, parent_id, canonical_id)
|
||||
file_tags(file_id FK, tag_id FK)
|
||||
attributes(id PK, file_id FK, key, value, value_type)
|
||||
relationships(id PK, src_file_id FK, dst_file_id FK, rel_type, direction)
|
||||
change_log(change_id PK, object_table, object_id, op, actor, ts, payload_json)
|
||||
```
|
||||
|
||||
### 2.3 Sync & Conflict Resolution
|
||||
|
||||
1. Each client appends to **change\_log** (CRDT‑compatible delta).
|
||||
2. Delta sync via WebSocket; server merges and re‑broadcasts.
|
||||
3. Conflicts → *Conflict Queue* UI (choose theirs / mine / merge).
|
||||
|
||||
### 2.4 Plugin API (TypeScript)
|
||||
|
||||
```ts
|
||||
export interface MarlinPlugin {
|
||||
onInit(ctx: CoreContext): void;
|
||||
extendSchema?(db: Database): void; // e.g. add new relationship table
|
||||
addCommands?(ui: UIContext): void; // register menus, actions
|
||||
}
|
||||
```
|
||||
|
||||
Plugins run in a sandboxed process with whitelisted IPC calls.
|
||||
|
||||
---
|
||||
|
||||
## 3 UX & Accessibility
|
||||
|
||||
* **Keyboard‑only workflow** audit (Tab / Shift‑Tab / Space toggles).
|
||||
* High‑contrast theme; adheres to WCAG 2.1 AA.
|
||||
* `Ctrl+Alt+V` toggles Dual View.
|
||||
* Generated query string shown live under Visual Builder – educates power users.
|
||||
|
||||
---
|
||||
|
||||
## 4 Performance Budget
|
||||
|
||||
| Metric | MVP | Stretch |
|
||||
| ------------------------ | --------- | ---------- |
|
||||
| Cold start (100 k files) | ≤ 3 s | 1 s |
|
||||
| Complex AND/OR query | ≤ 150 ms | 50 ms |
|
||||
| Sustained inserts | 5 k ops/s | 20 k ops/s |
|
||||
|
||||
Benchmarks run nightly; regressions block merge.
|
||||
|
||||
---
|
||||
|
||||
## 5 Security & Privacy
|
||||
|
||||
* **Role‑based ACL** on tags/attributes.
|
||||
* Per‑change audit trail; logs rotated to cold storage (≥ 90 days online).
|
||||
* Plugins confined by seccomp/AppArmor; no direct disk/network unless declared.
|
||||
|
||||
---
|
||||
|
||||
## 6 Packaging & Distribution
|
||||
|
||||
* **Flatpak** (GNOME/KDE) and **AppImage** for portable builds.
|
||||
* Background service runs as a systemd user unit: `--user marlin-indexerd.service`.
|
||||
* CLI (`marlin-cli`) packaged for headless servers & CI.
|
||||
|
||||
---
|
||||
|
||||
## 7 Roadmap
|
||||
|
||||
| Milestone | Scope | Timeline |
|
||||
| --------- | ----------------------------------------------------------------------------- | -------- |
|
||||
| **M1** | Tagging, attributes, virtual folders, SQLite, Dolphin plug‑in | 6 weeks |
|
||||
| **M2** | Sync service, version control, CLI | +6 weeks |
|
||||
| **M3** | NLP search, Visual Builder, distributed index prototype | +6 weeks |
|
||||
| **M4** | Plugin marketplace, enterprise auth (LDAP/OIDC), mobile companion (view‑only) | +8 weeks |
|
||||
|
||||
---
|
||||
|
||||
## 8 Branding
|
||||
|
||||
* **Name**: **Marlin** – fast, precise.
|
||||
* Icon: stylised sailfish fin forming a folder corner.
|
||||
* Tagline: *“Cut through clutter.”*
|
||||
* Domain: `marlin‑explorer.io` (availability checked 2025‑05‑12).
|
||||
|
||||
---
|
||||
|
||||
## 9 Quick‑Win Checklist (Sprint 0)
|
||||
|
||||
* [ ] Implement bulk metadata editor UI
|
||||
* [ ] Write conflict‑resolution spec & unit tests
|
||||
* [ ] Build diff viewer prototype
|
||||
* [ ] Keyboard‑only navigation audit
|
||||
* [ ] Establish performance CI with sample 100 k file corpus
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## 10 Development Plan (Outline)
|
||||
|
||||
### 10.1 Process & Methodology
|
||||
|
||||
* **Framework** – 2‑week Scrum sprints with Jira backlog, GitHub Projects mirror for public issues.
|
||||
* **Branching** – Trunk‑based: feature branches → PR → required CI & code‑review approvals (2).*Main* auto‑deploys nightly Flatpak.
|
||||
* **Definition of Done** – Code + unit tests + docs + passing CI + demo video (for UI work).
|
||||
* **CI/CD** – GitHub Actions matrix (Ubuntu 22.04, KDE Neon, Fedora 39) → Flatpak / AppImage artefacts, `cargo clippy`, coverage gate ≥ 85 %.
|
||||
|
||||
### 10.2 Team & Roles (FTE‑equivalent)
|
||||
|
||||
| Role | Core Skills | Allocation |
|
||||
| ----------------------------- | -------------------------------- | ---------- |
|
||||
| Lead Engineer | Rust, Qt/Kirigami, KIO | 1.0 |
|
||||
| Backend Engineer | Rust, LiteFS/Postgres, WebSocket | 1.0 |
|
||||
| Full‑stack / Plug‑in Engineer | TypeScript, Node, IPC | 0.8 |
|
||||
| UX / QA | Figma, accessibility, Playwright | 0.5 |
|
||||
| DevOps (fractional) | CI, Flatpak, security hardening | 0.2 |
|
||||
|
||||
### 10.3 Roadmap → Sprint‑level Tasks
|
||||
|
||||
| Sprint | Goal | Key Tasks | Exit Criteria |
|
||||
| ---------------------- | -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
|
||||
| **S0 (2 wks)** | Project bootstrap | • Repo + CI skeleton<br>• SQLite schema + migrations<br>• `marlin-cli init` & basic scan<br>• Hyperfine perf baseline | CLI scans dir; tests pass; artefact builds |
|
||||
| **S1–3 (M1, 6 wks)** | Tagging + virtual folders MVP | • Indexer daemon in Rust<br>• CRUD tags/attributes via CLI & DB<br>• Dolphin plug‑in: sidebar + tag view<br>• KIO `tags://` virtual folder<br>• Bulk‑edit dialog | 100 k‑file corpus cold‑start ≤ 3 s; user can tag files & navigate `tags://Urgent` |
|
||||
| **S4–6 (M2, 6 wks)** | Sync & version control | • Change‑log table + diff viewer<br>• LiteFS replication PoC<br>• WebSocket delta sync<br>• Conflict queue UI + last‑write‑wins fallback | Two devices sync metadata in <1 s round‑trip; rollback works |
|
||||
| **S7–9 (M3, 6 wks)** | NLP search & Visual Builder | • Integrate Tantivy FTS + ONNX intent model<br>• Toggle exact vs natural search<br>• QML Visual Builder with live query string | NL query "docs Alice edited last week" returns expected set in ≤ 300 ms |
|
||||
| **S10–13 (M4, 8 wks)** | Plug‑in marketplace & mobile companion | • IPC sandbox + manifest spec<br>• Sample plug‑ins (image EXIF auto‑tagger)<br>• Flutter read‑only client<br>• LDAP/OIDC enterprise auth | First external plug‑in published; mobile app lists smart folders |
|
||||
|
||||
### 10.4 Tooling & Infrastructure
|
||||
|
||||
* **Issue tracking** – Jira → labels `component/indexer`, `component/ui`.
|
||||
* **Docs** – mkdocs‑material hosted on GitHub Pages; automatic diagram generation via `cargo doc` + Mermaid.
|
||||
* **Nightly Perf Benchmarks** – Run in CI against 10 k, 100 k, 1 M synthetic corpora; fail build if P95 query > target.
|
||||
* **Security** – Dependabot, Trivy scans, optional SLSA level 2 provenance for releases.
|
||||
|
||||
### 10.5 Risks & Mitigations
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
| ------------------------------ | ---------------- | --------------------------------------------------------------------------- |
|
||||
| CRDT complexity | Delays M2 | Ship LWW first; schedule CRDT refactor post‑launch |
|
||||
| File system event overflow | Index corruption | Debounce & auto‑fallback to full rescan; alert user |
|
||||
| Cross‑distro packaging pain | Adoption drops | Stick to Flatpak; AppImage only for power users; collect telemetry (opt‑in) |
|
||||
| Scaling >1 M files on slow HDD | Perf complaints | Offer "index on SSD" wizard; tune FTS page cache |
|
||||
|
||||
### 10.6 Budget & Timeline Snapshot
|
||||
|
||||
* **Total dev time** ≈ 30 weeks.
|
||||
* **Buffer** +10 % (3 weeks) for holidays & unknowns → **33 weeks** (\~8 months).
|
||||
* **Rough budget** (3 FTE avg × 33 wks × \$150 k/yr) ≈ **\$285 k** payroll + \$15 k ops / tooling.
|
||||
|
||||
---
|
||||
Reference in New Issue
Block a user