mirror of
https://github.com/PR0M3TH3AN/Marlin.git
synced 2026-11-03 04:11:00 +00:00
187 lines
14 KiB
Markdown
187 lines
14 KiB
Markdown
# Marlin – Metadata‑Driven File Explorer
|
||
|
||
*Version 2 – 12 May 2025*
|
||
|
||
---
|
||
|
||
## 1 Key Features & Functionality
|
||
|
||
| Feature Area | Capabilities |
|
||
| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||
| **Tagging System** | • Unlimited, hierarchical or flat tags.<br>• Alias/synonym support via admin‑defined mappings (canonical names resolved at query time).<br>• **Bulk tag editing** via multi‑select context menu.<br>• Folder‑to‑Tag import with optional *watch & sync* mode so new sub‑folders inherit tags automatically. |
|
||
| **Custom Metadata Attributes** | • User‑defined fields (text, number, date, enum, boolean).<br>• Per‑template **Custom Metadata Schemas** (e.g. *Photo* → *Date, Location*). |
|
||
| **File Relationships** | • Typed, directional or bidirectional links (*related to*, *duplicate of*, *cites*…).<br>• Plugin API can register new relationship sets. |
|
||
| **Version Control for Metadata** | • Every change logged; unlimited roll‑back.<br>• Side‑by‑side diff viewer and *blame* panel showing *who/when/what*.<br>• Offline edits stored locally and merged (Git‑style optimistic merge with conflict prompts). |
|
||
| **Advanced Search & Smart Folders** | • Structured query syntax: `tag:ProjectX AND author:Alice`.<br>• Natural‑language search (*"files Alice edited last month"*) with toggle to exact mode.<br>• Visual Query Builder showing live query string.<br>• Saved queries appear as virtual “smart folders” that update in real‑time. |
|
||
| **User Interface** | • Sidebar: tags, attributes, relationships.<br>• Drag‑and‑drop tagging; inline metadata editor.<br>• Search bar with auto‑complete (Bloom filter backed).<br>• **Dual View Mode** – metadata vs traditional folder; remembers preference per location.<br>• **Interactive 60‑second tour** on first launch plus contextual tooltip help. |
|
||
| **Collaboration** | • Real‑time metadata sync across devices via cloud or self‑hosted relay.<br>• Conflict handling as per Version Control.<br>• Role‑based permissions (read / write / admin) on tags & attributes. |
|
||
| **Performance & Scale** | • Sharded/distributed index optional for >1 M files.<br>• Query cache with LRU eviction.<br>• Target metrics (100 k files): cold start ≤ 3 s, complex query ≤ 150 ms (stretch 50 ms). |
|
||
| **Backup & Restore** | • Scheduled encrypted backups; export to JSON / XML.<br>• One‑click restore from any point‑in‑time snapshot. |
|
||
| **Extensibility** | • Plug‑in system (TypeScript/JS) – see §2.4.<br>• Python scripting hook for automation and batch tasks.<br>• REST/IPC API for external tools. |
|
||
|
||
---
|
||
|
||
## 2 Technical Implementation
|
||
|
||
### 2.1 Core Stack
|
||
|
||
| Component | Primary Choice | Notes |
|
||
| -------------- | -------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
|
||
| File Manager | **Dolphin (KDE)** KIO‑based plug‑ins | GTK users can install a Nautilus extension (feature‑parity subset). |
|
||
| Metadata Store | **SQLite + FTS5** (single‑user) → optional **LiteFS/Postgres** for replication & multi‑user scale. | Per‑row AES‑GCM encryption for sensitive fields; keys stored in OS keyring. |
|
||
| Indexer Daemon | Rust service using `notify` (inotify on Linux, FSEvents on macOS). | 100 ms debounce batches, async SQLite writes. |
|
||
| Cache | In‑memory LRU + Bloom filter for auto‑complete. | |
|
||
|
||
### 2.2 Database Schema (simplified)
|
||
|
||
```text
|
||
files(id PK, path, inode, size, mtime, ctime, hash)
|
||
tags(id PK, name, parent_id)
|
||
file_tags(file_id FK, tag_id FK)
|
||
attributes(id PK, file_id FK, key, value, value_type)
|
||
relationships(id PK, src_file_id FK, dst_file_id FK, rel_type, direction)
|
||
change_log(change_id PK, object_table, object_id, op, actor, ts, payload_json)
|
||
```
|
||
|
||
### 2.3 Sync & Conflict Resolution
|
||
|
||
1. Each client appends to **change\_log** (CRDT‑compatible delta).
|
||
2. Delta sync via WebSocket; server merges and re‑broadcasts.
|
||
3. Conflicts → *Conflict Queue* UI (choose theirs / mine / merge).
|
||
|
||
### 2.4 Plugin API (TypeScript)
|
||
|
||
```ts
|
||
export interface MarlinPlugin {
|
||
onInit(ctx: CoreContext): void;
|
||
extendSchema?(db: Database): void; // e.g. add new relationship table
|
||
addCommands?(ui: UIContext): void; // register menus, actions
|
||
}
|
||
```
|
||
|
||
Plugins run in a sandboxed process with whitelisted IPC calls.
|
||
|
||
---
|
||
|
||
## 3 UX & Accessibility
|
||
|
||
* **Keyboard‑only workflow** audit (Tab / Shift‑Tab / Space toggles).
|
||
* High‑contrast theme; adheres to WCAG 2.1 AA.
|
||
* `Ctrl+Alt+V` toggles Dual View.
|
||
* Generated query string shown live under Visual Builder – educates power users.
|
||
|
||
---
|
||
|
||
## 4 Performance Budget
|
||
|
||
| Metric | MVP | Stretch |
|
||
| ------------------------ | --------- | ---------- |
|
||
| Cold start (100 k files) | ≤ 3 s | 1 s |
|
||
| Complex AND/OR query | ≤ 150 ms | 50 ms |
|
||
| Sustained inserts | 5 k ops/s | 20 k ops/s |
|
||
|
||
Benchmarks run nightly; regressions block merge.
|
||
|
||
---
|
||
|
||
## 5 Security & Privacy
|
||
|
||
* **Role‑based ACL** on tags/attributes.
|
||
* Per‑change audit trail; logs rotated to cold storage (≥ 90 days online).
|
||
* Plugins confined by seccomp/AppArmor; no direct disk/network unless declared.
|
||
|
||
---
|
||
|
||
## 6 Packaging & Distribution
|
||
|
||
* **Flatpak** (GNOME/KDE) and **AppImage** for portable builds.
|
||
* Background service runs as a systemd user unit: `--user marlin-indexerd.service`.
|
||
* CLI (`marlin-cli`) packaged for headless servers & CI.
|
||
|
||
---
|
||
|
||
## 7 Roadmap
|
||
|
||
| Milestone | Scope | Timeline |
|
||
| --------- | ----------------------------------------------------------------------------- | -------- |
|
||
| **M1** | Tagging, attributes, virtual folders, SQLite, Dolphin plug‑in | 6 weeks |
|
||
| **M2** | Sync service, version control, CLI | +6 weeks |
|
||
| **M3** | NLP search, Visual Builder, distributed index prototype | +6 weeks |
|
||
| **M4** | Plugin marketplace, enterprise auth (LDAP/OIDC), mobile companion (view‑only) | +8 weeks |
|
||
|
||
---
|
||
|
||
## 8 Branding
|
||
|
||
* **Name**: **Marlin** – fast, precise.
|
||
* Icon: stylised sailfish fin forming a folder corner.
|
||
* Tagline: *“Cut through clutter.”*
|
||
* Domain: `marlin‑explorer.io` (availability checked 2025‑05‑12).
|
||
|
||
---
|
||
|
||
## 9 Quick‑Win Checklist (Sprint 0)
|
||
|
||
* [ ] Implement bulk metadata editor UI
|
||
* [ ] Write conflict‑resolution spec & unit tests
|
||
* [ ] Build diff viewer prototype
|
||
* [ ] Keyboard‑only navigation audit
|
||
* [ ] Establish performance CI with sample 100 k file corpus
|
||
|
||
---
|
||
|
||
---
|
||
|
||
## 10 Development Plan (Outline)
|
||
|
||
### 10.1 Process & Methodology
|
||
|
||
* **Framework** – 2‑week Scrum sprints with Jira backlog, GitHub Projects mirror for public issues.
|
||
* **Branching** – Trunk‑based: feature branches → PR → required CI & code‑review approvals (2).*Main* auto‑deploys nightly Flatpak.
|
||
* **Definition of Done** – Code + unit tests + docs + passing CI + demo video (for UI work).
|
||
* **CI/CD** – GitHub Actions matrix (Ubuntu 22.04, KDE Neon, Fedora 39) → Flatpak / AppImage artefacts, `cargo clippy`, coverage gate ≥ 85 %.
|
||
|
||
### 10.2 Team & Roles (FTE‑equivalent)
|
||
|
||
| Role | Core Skills | Allocation |
|
||
| ----------------------------- | -------------------------------- | ---------- |
|
||
| Lead Engineer | Rust, Qt/Kirigami, KIO | 1.0 |
|
||
| Backend Engineer | Rust, LiteFS/Postgres, WebSocket | 1.0 |
|
||
| Full‑stack / Plug‑in Engineer | TypeScript, Node, IPC | 0.8 |
|
||
| UX / QA | Figma, accessibility, Playwright | 0.5 |
|
||
| DevOps (fractional) | CI, Flatpak, security hardening | 0.2 |
|
||
|
||
### 10.3 Roadmap → Sprint‑level Tasks
|
||
|
||
| Sprint | Goal | Key Tasks | Exit Criteria |
|
||
| ---------------------- | -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
|
||
| **S0 (2 wks)** | Project bootstrap | • Repo + CI skeleton<br>• SQLite schema + migrations<br>• `marlin-cli init` & basic scan<br>• Hyperfine perf baseline | CLI scans dir; tests pass; artefact builds |
|
||
| **S1–3 (M1, 6 wks)** | Tagging + virtual folders MVP | • Indexer daemon in Rust<br>• CRUD tags/attributes via CLI & DB<br>• Dolphin plug‑in: sidebar + tag view<br>• KIO `tags://` virtual folder<br>• Bulk‑edit dialog | 100 k‑file corpus cold‑start ≤ 3 s; user can tag files & navigate `tags://Urgent` |
|
||
| **S4–6 (M2, 6 wks)** | Sync & version control | • Change‑log table + diff viewer<br>• LiteFS replication PoC<br>• WebSocket delta sync<br>• Conflict queue UI + last‑write‑wins fallback | Two devices sync metadata in <1 s round‑trip; rollback works |
|
||
| **S7–9 (M3, 6 wks)** | NLP search & Visual Builder | • Integrate Tantivy FTS + ONNX intent model<br>• Toggle exact vs natural search<br>• QML Visual Builder with live query string | NL query "docs Alice edited last week" returns expected set in ≤ 300 ms |
|
||
| **S10–13 (M4, 8 wks)** | Plug‑in marketplace & mobile companion | • IPC sandbox + manifest spec<br>• Sample plug‑ins (image EXIF auto‑tagger)<br>• Flutter read‑only client<br>• LDAP/OIDC enterprise auth | First external plug‑in published; mobile app lists smart folders |
|
||
|
||
### 10.4 Tooling & Infrastructure
|
||
|
||
* **Issue tracking** – Jira → labels `component/indexer`, `component/ui`.
|
||
* **Docs** – mkdocs‑material hosted on GitHub Pages; automatic diagram generation via `cargo doc` + Mermaid.
|
||
* **Nightly Perf Benchmarks** – Run in CI against 10 k, 100 k, 1 M synthetic corpora; fail build if P95 query > target.
|
||
* **Security** – Dependabot, Trivy scans, optional SLSA level 2 provenance for releases.
|
||
|
||
### 10.5 Risks & Mitigations
|
||
|
||
| Risk | Impact | Mitigation |
|
||
| ------------------------------ | ---------------- | --------------------------------------------------------------------------- |
|
||
| CRDT complexity | Delays M2 | Ship LWW first; schedule CRDT refactor post‑launch |
|
||
| File system event overflow | Index corruption | Debounce & auto‑fallback to full rescan; alert user |
|
||
| Cross‑distro packaging pain | Adoption drops | Stick to Flatpak; AppImage only for power users; collect telemetry (opt‑in) |
|
||
| Scaling >1 M files on slow HDD | Perf complaints | Offer "index on SSD" wizard; tune FTS page cache |
|
||
|
||
### 10.6 Budget & Timeline Snapshot
|
||
|
||
* **Total dev time** ≈ 30 weeks.
|
||
* **Buffer** +10 % (3 weeks) for holidays & unknowns → **33 weeks** (\~8 months).
|
||
* **Rough budget** (3 FTE avg × 33 wks × \$150 k/yr) ≈ **\$285 k** payroll + \$15 k ops / tooling.
|
||
|
||
---
|