From e6d22ba6053e420a59b28debc8711d524cf201f8 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Thu, 15 May 2025 17:14:40 -0400 Subject: [PATCH] update --- roadmap.md | 56 +++++++++++++++++++++++++----------------------------- 1 file changed, 26 insertions(+), 30 deletions(-) diff --git a/roadmap.md b/roadmap.md index 9032721..4535881 100644 --- a/roadmap.md +++ b/roadmap.md @@ -1,40 +1,36 @@ -Here’s the updated roadmap with each new feature slotted in where its dependencies are best met: +Here’s a slimmed-down, re-organized roadmap that groups related work into bigger milestones, highlights key deliverables (including a “demo” command and grep-style context snippets), and stages integrations for maximal developer velocity: -| Phase | Focus | Why now? | Key deliverables | -| -------------------------- | -------------------------------------- | --------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | -| **1. 2025-Q2 – “Bedrock”** | Migrations + CI baseline + core schema | We’ve stabilized migrations; now add foundational tables for links, groups, views | • CI: `cargo test` + `cargo sqlx migrate run --dry-run`
• New migrations: | +| Phase / Sprint | Timeline | Focus & Rationale | Key Deliverables | +| ---------------------------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | +| **Sprint α: Bedrock & Metadata Domains** | **2025-Q2 (now–6/01)** | Stabilize core schema, migrations, CI, and introduce the first metadata domains **with discoverability**. | - **CI:** `cargo test` + SQL migration dry-run coverage
- **Migrations:** | -* `links(src_file,dst_file,link_type)` -* `collections(name)` + `collection_files` -* `views(name,query)`
• CLI stubs for `marlin link` / `unlink` / `list-links` / `backlinks`, `marlin coll` and `marlin view` | - \| **2. 2025-Q2** | Leaner FTS maintenance | Per-row triggers don’t scale past \~100 k files | • Replace per-row triggers with a “dirty” flag + periodic rebuild
• Benchmark end-to-end on 100 k files | - \| **2.1 2025-Q2** | Dirty-row FTS + CI | Prep for both scale and live-watcher—avoid full rebuilds on every change | • `scan --dirty` reindexes only changed files
• CI coverage for dirty-scan edge cases | - \| **2.2 2025-Q2** | Live file watching | Offer true “working-dir” mode—auto-scan on FS events | • `marlin watch [dir]` via `notify` crate
• Incremental scan on create/modify/delete/rename | - \| **2.3 2025-Q2** | Self-pruning backups | Instant protection and bounded storage—no manual snapshot cleanup | • `marlin backup --prune ` flag
• Post-scan hook to prune to latest 10
• Daily prune automation (cron or CI) | - \| **3. 2025-Q3** | FTS5 content indexing & annotations | Full-text search over file bodies + per-file notes/highlights | • Add `files.content` column + migration
• Extend `files_fts` to include `content`
• New `annotations` table + FTS triggers
• CLI: `marlin annotate add|list` | - \| **4. 2025-Q3** | Content hashing, dedup & versioning | Detect duplicates, track history, enable diffs | • Populate `files.hash` with SHA-256
• `scan --rehash` option
• CLI: `marlin version diff ` | - \| **5. 2025-Q3** | Tag aliases/canonicals & semantic/AI enhancements | Control tag sprawl and lay groundwork for AI-driven suggestions | • Enforce `canonical_id` on `tags` + `tag alias add|ls|rm` CLI
• Create `embeddings` table
• `scan --embed` to generate vectors
• CLI: `marlin tag suggest`, `marlin summary `, `marlin similarity scan` | - \| **6. 2025-Q4** | Search DSL v2 & Smart Views | More powerful query grammar + reusable “virtual folders” | • Replace ad-hoc parser with a `nom`-based grammar
• CLI: `marlin view save|list|exec` | - \| **7. 2025-Q4** | Attribute templates, states, tasks & timeline | Structured metadata unlocks workflows, reminders & temporal context | • `templates` + `template_fields` tables + validation
• CLI: +* `links(src,dst,type)` +* `collections(name)+collection_files` +* `views(name,query)`
- **CLI stubs** for `marlin link|unlink|list|backlinks`, `marlin coll`, `marlin view`
- **`marlin demo`** command that walks you through a mini-drive-by tutorial of link/coll/view flows | + \| **Epic 1: Scale & Reliability** | **2025-Q2** | Prototype “dirty-row” FTS to avoid per-row triggers, benchmark at 100 k files, and shore up CI for edge cases. | - Dirty-flag + `scan --dirty` reindex only changed rows
- Replace per-row triggers with periodic FTS rebuild
- End-to-end benchmarks on \~100 k files
- CI tests for dirty-scan edge-cases | + \| **Epic 2: Live Mode & Self-Pruning Backups** | **2025-Q2** | Automate continuous indexing & backup hygiene so Marlin “just works” in a real workspace. | - `marlin watch [dir]` via `notify` crate (create/modify/delete/rename)
- `backup --prune ` flag + post-scan auto-prune to latest N
- Daily/pr-merge auto-prune automation in CI | + \| **Phase 3: Content FTS & Annotations** | **2025-Q3** | Go beyond metadata—index file bodies, provide grep-style context, and add annotation support. | - New `files.content` column + migration
- Extend `files_fts` to include `content` and emit context snippets (`-C` style)
- `annotations` table + FTS triggers
- CLI: `marlin annotate add|list` | + \| **Phase 4: Versioning & Deduplication** | **2025-Q3** | Enable history, diffs, and duplicate detection with content hashing. | - Compute & store `files.hash` (SHA256)
- `scan --rehash` to refresh hashes
- CLI: `marlin version diff ` to show changes between revisions | + \| **Phase 5: Tag Aliases & Semantic Enhancements** | **2025-Q3** | Tame tag sprawl and seed AI-powered suggestions via embeddings. | - Enforce `canonical_id` on `tags`; CLI: `marlin tag alias add|ls|rm`
- `embeddings` table + `scan --embed` to generate vectors
- CLI: `marlin tag suggest`, `marlin summary `, `marlin similarity scan` | + \| **Phase 6: Search DSL v2 & Smart Views** | **2025-Q4** | Offer a robust query grammar and “virtual folders” powered by views. | - Swap ad-hoc parser for a `nom`-based grammar with `AND`, `OR`, parentheses, wildcards…
- CLI: `marlin view save|list|exec` with aliases and pagination | + \| **Phase 7: Structured Workflows** | **2025-Q4** | Unlock full task, state, reminder & event workflows directly on files. | - `templates` + `template_fields` + validation engine
- CLI: * `marlin state set|transitions add|state log` * `marlin task scan|task list` * `marlin remind set ""` -* `marlin event add ""`, `marlin timeline` | - \| **8. 2026-Q1** | Dolphin read-only plugin | Surface metadata, links, annotations in native file manager | • Qt sidebar showing tags, attributes, links, annotations | - \| **9. 2026-Q1** | Full edit UI | After proving read-only stable, add in-place editing | • Tag editor, collection & view managers, state/task/event dialogs | - \| **10. 2026-Q2** | Multi-device sync | Final frontier: optional sync/replication layer | • Choose sync backend (rqlite / Litestream / bespoke)
• Support read-only mounts for remote indexes | +* `marlin event add ""` + `marlin timeline` | + \| **Phase 8: Lightweight Integrations** | **2026-Q1** | Surface Marlin inside your editor/terminal before diving into a full GUI. | - **VSCode & terminal UI extension**: file-tree sidebar showing tags/attrs/links/annotations | + \| **Phase 9: Dolphin Sidebar Plugin (MVP)** | **2026-Q1** | Prototype a read-only Qt sidebar for Linux file managers—peek metadata without leaving your file browser. | - Qt plugin showing tags, attributes, links, and annotations alongside files | + \| **Phase 10: Full Edit UI & Multi-Device Sync** | **2026-Q2** | Ship an in-place metadata editor and optional sync layer for distributed workflows. | - Tag & view editors, task/reminder/event dialogs in GUI
- Choose/implement sync backend (rqlite, Litestream or custom) for optional read-only remote mounts | --- -### Current sprint (ends **2025-06-01**) +### Why this order? -1. FTS rebuild prototype (dirty-rows) – measure on 50 k files -2. `backup --prune` implementation + auto-prune hook -3. Integration tests for tag/attr workflows on Windows via GitHub Actions -4. **New:** basic `links`, `collections`, `views` migrations + CLI stubs +1. **Lock down core schema & domains** (links, collections, views) **with a “demo” helper** so users can explore right away. +2. **Scale & CI** unlocks safe indexing at volume, then +3. **Live mode & auto-prune** keep your index fresh without manual steps. +4. **Content FTS + annotations** builds on an efficient, reliable plumbing layer—you’ll love grep-style context snippets. +5. **Versioning & semantic layers** ride atop a stable full-text index and annotation system. +6. **Advanced queries & workflows** expand power users’ toolsets before branching into GUIs and sync. -**Development principles remain**: - -* Local-first, offline-capable -* Ship code = ship migrations -* Instrumentation first (trace spans & timings on all new commands) +This grouping ensures every new layer rests on a solid, tested foundation—maximizing both developer speed and user delight.