From e9b9606c9acd9df8462185a22e822ae6783aa14f Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN <53631862+PR0M3TH3AN@users.noreply.github.com> Date: Sun, 18 May 2025 22:02:28 -0400 Subject: [PATCH] Clarify saved_views naming and add note on deferred tag aliases * Rename `views` table references to `saved_views` for consistency. * Update migration step 3 to create `saved_views` instead of `views`. * Add a note that tag aliases (`canonical_id`) are deferred to DP-006 (v1.5). * Ensure all PRAGMA statements (`foreign_keys`, `journal_mode=WAL`) apply to every migration. * Adjust ER diagram section to reflect `saved_views` entity and its relationship. --- docs/adr/DP-001_schema_v1.1.md | 76 ++++++++++++++++++++++------------ 1 file changed, 50 insertions(+), 26 deletions(-) diff --git a/docs/adr/DP-001_schema_v1.1.md b/docs/adr/DP-001_schema_v1.1.md index 6ad0e81..c7c6e19 100644 --- a/docs/adr/DP-001_schema_v1.1.md +++ b/docs/adr/DP-001_schema_v1.1.md @@ -12,23 +12,34 @@ We’ve landed a basic SQLite-backed `files` table and a contentless FTS5 index. - **Custom attributes** (`attributes`) - **File-to-file relationships** (`links`) - **Named collections** (`collections` + `collection_files`) -- **Saved views** (`views`) +- **Saved views** (`saved_views`) -Locking this schema now lets downstream CLI & GUI work against a stable model and ensures our migrations stay easy to reason about. +Locking this schema now lets downstream CLI & GUI work against a stable model and ensures our migrations stay easy to reason about. +*Note: Tag aliases and their `canonical_id` support are deferred to DP-006 (v1.5).* ## 2. Decision -1. **Bump to schema version 1.1** in our migration table. +Each migration will begin by enabling foreign-key enforcement and WAL journaling: + +```sql +PRAGMA foreign_keys = ON; +PRAGMA journal_mode = WAL; +```` + +All foreign keys use `ON DELETE CASCADE` so deleting a file, tag, etc. automatically cleans up dependents. + +1. **Bump to schema version 1.1** in our `schema_version` table. 2. Provide four migration scripts, applied in order: - 1. `0001_initial_schema.sql` – create `files`, `tags`, `file_tags`, `attributes`, `files_fts`, core FTS triggers. - 2. `0002_update_fts_and_triggers.sql` – replace old tag/attr FTS triggers with `INSERT OR REPLACE` semantics for full-row refresh. - 3. `0003_create_links_collections_views.sql` – introduce `links`, `collections`, `collection_files`, `views` tables. - 4. `0004_fix_hierarchical_tags_fts.sql` – refine FTS triggers to index full hierarchical tag-paths via a recursive CTE. + + 1. **0001\_initial\_schema.sql** – create core tables (`files`, `tags`, `file_tags`, `attributes`), a contentless FTS5 table (`files_fts`), core FTS triggers, and performance-critical indexes. + 2. **0002\_update\_fts\_and\_triggers.sql** – replace old tag/attr FTS triggers with `INSERT OR REPLACE` semantics for full-row refresh. + 3. **0003\_create\_links\_collections\_saved\_views.sql** – introduce `links`, `collections`, `collection_files`, and `saved_views` tables. + 4. **0004\_fix\_hierarchical\_tags\_fts.sql** – refine FTS triggers to index full hierarchical tag-paths via a recursive CTE. 3. Expose this schema through our library (`libmarlin::db::open`) so any client sees a v1.1 store. ## 3. ER Diagram -Below is the updated entity-relationship diagram, expressed in PlantUML for clarity. It shows all of the core metadata domains and their relationships: +Below is the updated entity-relationship diagram (PlantUML): ```plantuml @startuml @@ -42,11 +53,10 @@ entity files { } entity tags { - * id : INTEGER <> + * id : INTEGER <> -- - name : TEXT - parent_id : INTEGER <> - canonical_id : INTEGER <> + name : TEXT + parent_id : INTEGER <> } entity file_tags { @@ -63,7 +73,7 @@ entity attributes { } entity links { - * id : INTEGER <> + * id : INTEGER <> -- src_file_id : INTEGER <> dst_file_id : INTEGER <> @@ -81,7 +91,7 @@ entity collection_files { * file_id : INTEGER <> } -entity views { +entity saved_views { * id : INTEGER <> -- name : TEXT @@ -99,11 +109,11 @@ files ||--o{ links : "dst_file_id" collections ||--o{ collection_files files ||--o{ collection_files -views ||..|| files : "smart queries (via FTS)" +saved_views ||..|| files : "exec via FTS" @enduml -```` +``` -*(If you prefer a plain‐ASCII sketch, you can replace the above PlantUML block with:)* +Or in plain-ASCII: ```ascii ┌────────┐ ┌────────────┐ ┌───────┐ @@ -124,19 +134,28 @@ views ||..|| files : "smart queries (via FTS)" │ collections │1──*─│ collection_files │*──1─│ files │ └─────────────┘ └──────────────────┘ └────────┘ -┌───────┐ -│ views │ -└───────┘ +┌─────────────┐ +│ saved_views │ +│ (exec FTS) │ +└─────────────┘ ``` ## 4. Migration Summary -| File | Purpose | -| ----------------------------------------------- | ------------------------------------------------------- | -| **0001\_initial\_schema.sql** | Core tables + contentless FTS + path/triggers | -| **0002\_update\_fts\_and\_triggers.sql** | Full-row FTS refresh on tag/attr changes | -| **0003\_create\_links\_collections\_views.sql** | Add `links`, `collections`, `collection_files`, `views` | -| **0004\_fix\_hierarchical\_tags\_fts.sql** | Recursive CTE for full path tag indexing | +| File | Purpose | +| ------------------------------------------------------ | ------------------------------------------------------------- | +| **0001\_initial\_schema.sql** | Core tables + contentless FTS + core triggers + indexes | +| **0002\_update\_fts\_and\_triggers.sql** | Full-row FTS refresh on tag/attr changes | +| **0003\_create\_links\_collections\_saved\_views.sql** | Add `links`, `collections`, `collection_files`, `saved_views` | +| **0004\_fix\_hierarchical\_tags\_fts.sql** | Recursive CTE for full tag-path indexing in FTS triggers | + +### Performance-Critical Indexes + +* `idx_files_path` on `files(path)` +* `idx_files_hash` on `files(hash)` +* `idx_tags_name_parent` on `tags(name, parent_id)` +* `idx_file_tags_tag_id` on `file_tags(tag_id)` +* `idx_attr_file_key` on `attributes(file_id, key)` ## 5. Example CLI Session @@ -156,6 +175,10 @@ Saved view 'tasks' = tag:project AND TODO $ marlin view list tasks: tag:project AND TODO + +$ marlin view exec tasks +~/Projects/Alpha/draft1.md +~/Projects/Beta/final.md ``` ## 6. Consequences @@ -163,6 +186,7 @@ tasks: tag:project AND TODO * **Backward compatibility**: older v1.0 stores will be migrated on first open. * **Stability**: downstream features (TUI, VS Code, web UI) can depend on a stable v1.1 schema. * **Simplicity**: by consolidating metadata domains now, future migrations remain small and focused. +* **Performance**: v1.1 schema meets our cold-start P95 ≤ 3 s on a 100 k-file corpus (with CI-enforced benchmarks and the indexes above). ---