Make TUI v3 the default and add index0 foundations

This commit is contained in:
thePR0M3TH3AN committed 2026-03-05 21:52:39 -05:00
1 parent 0fa8aebc1b
commit e8a2b81e7b
17 files changed
+3005 -76

No files matched your search

+29
View File
@@ -37,6 +37,7 @@ Use these docs as inputs, but treat this file as the decision layer:
- TUI v2 integration execution plan (current working plan): `docs/tui_v2_integration_execution_plan_2026-03-02.md`
- TUI v2 mockup gap audit: `docs/tui_v2_mockup_gap_audit_2026-03-02.md`
- Semantic vector index integration plan: `docs/semantic_vector_index_plan.md`
- Index0 atlas execution plan: `docs/index0_atlas_execution_plan_2026-03-05.md`
- Cutover memo: `docs/tui_v2_cutover_decision.md`
- Security readiness: `docs/security_readiness_checklist.md`
- Supply-chain integrity: `docs/supply_chain_release_integrity.md`
@@ -79,6 +80,28 @@ Use these docs as inputs, but treat this file as the decision layer:
- Phase C complete baseline (legacy + TUI v2 semantic workflows and mode toggles)
- Next: incremental mutation hooks + hybrid quality hardening
### Index0 Atlas Roadmap
- New architecture/spec track drafted:
- `docs/index0_atlas_execution_plan_2026-03-05.md`
- Phase 0 implementation spec drafted:
- `docs/index0_phase0_spec_2026-03-05.md`
- Phase 1 foundation slice landed:
- reserved `_system.index0` namespace normalization
- deterministic `index0` merge helper
- Nostr manifest optional `index0` metadata support
- focused tests in `src/tests/test_index0_core.py`
- Intent:
- add canonical encrypted atlas state inside the existing payload for audit, hierarchy wayfinding, and future KB/chat navigation
- Key implementation direction:
- reserved `_system.index0` namespace inside the synced payload
- deterministic merge + checkpointed hash-chain validation
- rebuildable wayfinder views layered on top
- Sequencing:
- Phase 0 spec/schema is drafted
- Phase 1 foundation is partially implemented
- next implementation slice: event emission hooks from entry CRUD/link flows
- implementation phases should not preempt current TUI cutover-critical or security-readiness blockers
## 4) Recommended Next Steps (Priority Order)
1. Execute Phases A-C from `docs/tui_v2_integration_execution_plan_2026-03-02.md`:
@@ -91,6 +114,12 @@ Use these docs as inputs, but treat this file as the decision layer:
: unified lexical+tag+semantic search integration.
5. Execute Phase F:
: final per-board fidelity pass and strict mockup closeout.
6. Start Index0 Atlas Phase 0 from `docs/index0_atlas_execution_plan_2026-03-05.md`:
: lock schema, merge contract extension, manifest checkpoint fields, and compaction rules.
7. After cutover-critical TUI/security slices stabilize, execute Index0 Atlas Phases 1-3:
: canonical storage, event emission hooks, and manifest checkpoint validation.
8. After semantic hardening resumes, execute Index0 Atlas Phase 5:
: atlas read APIs/views and semantic interplay.
## 5) Next Slice Definition (Recommended Immediate Work)
@@ -0,0 +1,613 @@
# Index0 Atlas Execution Plan (2026-03-05)
Status: Proposed
Branch target: `beta`
Scope: Core index payload, Nostr sync, hierarchy navigation, future KB and agent workflows
## 1) Goal
Implement `index0` as the canonical encrypted atlas for SeedPass:
1. record what exists
2. record what changed
3. expose deterministic wayfinding state for users and agents
4. preserve existing SeedPass sync, merge, and security guarantees
Working slogan:
`Index0 is the atlas: it records what exists, what changed, and how to navigate it.`
## 2) Beta Branch Baseline To Reuse
This plan is grounded in the current `beta` branch, not a greenfield design.
Existing contracts already in place:
1. Nostr sync uses manifest + snapshot chunks + deltas:
- manifest `30070`
- snapshot chunk `30071`
- delta `30072`
- references: `docs/SPEC.md`, `docs/nostr_setup.md`, `src/nostr/backup_models.py`, `src/nostr/snapshot.py`
2. Deterministic convergence already exists through `modified_ts_hash_tombstone_v2`:
- references: `docs/sync_conflict_contract.md`, `src/seedpass/core/sync_conflict.py`
3. Retention caps already exist for unbounded metadata growth:
- current example: tombstone retention cap in `src/seedpass/core/sync_conflict.py`
4. Hierarchy and namespace reset are already part of the product model:
- key hierarchy: `docs/SPEC.md`
- namespace reset / `nostr_account_idx`: `docs/nostr_namespace_reset.md`, `src/seedpass/core/state_manager.py`
5. Entry graph links already provide typed relationships between artifacts:
- reference: `docs/entry_graph.md`
6. Semantic retrieval already has the correct architectural shape for derived caches:
- local-only derived index, not canonical sync state
- reference: `docs/semantic_vector_index_plan.md`
## 3) Core Design Decision
`index0` should live inside the existing encrypted index payload and ride the existing snapshot/delta pipeline.
Recommendation for `beta`:
1. Do not create a separate database or alternate sync protocol.
2. Do not put `index0` records into the user `entries` map initially.
3. Add a reserved top-level system namespace in the canonical payload, then expose entry-like adapters later if needed.
Recommended payload shape:
```json
{
"schema_version": 3,
"entries": {},
"_sync_meta": {},
"_system": {
"index0": {
"schema_version": 1,
"events": {},
"checkpoints": {},
"canonical_views": {},
"view_manifest": {},
"heads": {},
"stats": {}
}
}
}
```
## 4) Why `_system.index0` Instead Of User `entries`
The current `beta` code strongly assumes user entries are numeric IDs:
1. create/update flows allocate integer IDs in `src/seedpass/core/entry_management.py`
2. retrieve/modify/list paths index into `entries[str(index)]`
3. several CLI/TUI/API paths expect user-facing entry IDs to remain numeric
Shoving reserved `index0` records directly into `entries` would create unnecessary compatibility risk.
So the practical `beta` implementation is:
1. canonical storage inside the same synced payload
2. separate reserved namespace for system-owned records
3. optional projection into UI/API later as a virtual view layer
This still satisfies the main requirement: `index0` remains first-class canonical state inside the same encrypted, merged, synced vault.
## 5) What `index0` Must Contain
`index0` should contain three record classes.
### 5.1 `index0_event`
Ground-truth change records.
Purpose:
1. append change history
2. capture hierarchy and relationship mutations
3. support audit, sync replay validation, and activity feeds
Minimum fields:
```json
{
"event_id": "sha256(...)",
"event_type": "entry_created",
"subject_type": "entry",
"subject_id": "42",
"subject_kind": "document",
"scope_path": "seed/<root>/managed/<child>",
"actor_type": "user",
"actor_id": "<profile fingerprint or agent id>",
"writer_id": "<logical writer id>",
"modified_ts": 1760000000,
"prev_hash": "<optional prior hash in writer stream>",
"classification": "internal",
"partition": "standard",
"policy_ref": "<optional policy stamp or rule id>",
"payload_ref": {
"entry_id": "42"
},
"links": [
{
"target_id": "42",
"relation": "describes"
}
],
"tags": ["kb", "chat", "audit"],
"visibility": "private",
"integrity_hash": "sha256(...)"
}
```
Initial event types:
1. `entry_created`
2. `entry_modified`
3. `entry_deleted`
4. `entry_archived`
5. `entry_restored`
6. `link_added`
7. `link_removed`
8. `profile_added`
9. `profile_removed`
10. `agent_added`
11. `agent_removed`
12. `sync_snapshot_published`
13. `sync_delta_applied`
14. `nostr_message_indexed`
15. `document_indexed`
### 5.2 `index0_checkpoint`
Deterministic integrity and compaction anchors.
Purpose:
1. summarize older event windows
2. cap growth
3. store checkpoint hashes that manifests can reference
Minimum fields:
```json
{
"checkpoint_id": "cp-000001",
"window_start_ts": 1760000000,
"window_end_ts": 1760086400,
"event_count": 512,
"head_hashes": {
"writer:<root>": "abc123",
"writer:<managed-user>": "def456"
},
"summary_hash": "sha256(...)",
"rollup": {
"events_by_type": {
"entry_modified": 300,
"nostr_message_indexed": 90
},
"subjects_by_kind": {
"document": 120,
"note": 90
}
},
"modified_ts": 1760086400
}
```
### 5.3 `index0_view`
Derived, regeneratable navigation caches.
Purpose:
1. speed up browsing
2. avoid recomputing common hierarchy and activity summaries
3. support future agent navigation and hybrid search routing
Initial views:
1. `children_of:<scope>`
2. `recent_activity:<scope>`
3. `top_tags:<scope>`
4. `counts_by_kind:<scope>`
5. `hot_nodes:<scope>`
6. `conversation_index:<scope>`
Views are not the source of truth. They must always be rebuildable from canonical entries plus `index0_event` and `index0_checkpoint`.
### 5.4 Canonical vs local derived views
This is an important design refinement for `beta`.
Do not sync every derived navigation or search view.
Recommended split:
1. `canonical_views`:
- low-cardinality, deterministic, cross-device-useful summaries
- examples: `children_of`, `counts_by_kind`, compact `recent_activity`
2. local derived views:
- expensive, high-churn, or personalization-oriented caches
- examples: semantic neighbor expansions, ranking caches, UI-specific hotlists
Rules:
1. only sync views that materially improve restore/bootstrap/navigation across devices
2. keep search-heavy and high-cardinality projections local
3. treat `view_manifest` as the contract for what canonical views exist and how they are versioned
This keeps `index0` useful without recreating a second bloated database inside the vault payload.
## 6) Hash Chaining And Tamper Evidence
For `beta`, use lightweight tamper evidence instead of a full Merkle epoch system.
### 6.1 Event integrity
Each `index0_event` stores:
1. `integrity_hash`: canonical hash of the event payload
2. `prev_hash`: optional hash of the prior event in the same logical writer stream
3. `writer_id`: deterministic stream identifier
### 6.2 Why writer streams instead of one global chain
A single global append-only chain is fragile under concurrent multi-device mutation.
The current `beta` branch is already multi-device and delta-merge oriented, so the safer MVP is:
1. hash-chain per logical writer stream
2. deterministic checkpoint rollups that record the set of stream heads
3. manifest-level checkpoint metadata for restore validation
This preserves tamper evidence without forcing total ordering across all concurrent writers.
### 6.3 Manifest checkpoint metadata
Extend manifest payloads to optionally include:
```json
{
"index0_checkpoint_id": "cp-000001",
"index0_checkpoint_hash": "sha256(...)",
"index0_stream_heads": {
"writer:<root>": "abc123"
}
}
```
Validation path on restore:
1. restore latest snapshot
2. replay newer deltas
3. recompute current `index0` checkpoint/head material
4. compare against manifest metadata
5. raise warning or fail closed based on policy setting
## 7) Deterministic Merge Contract For `index0`
`index0` must not bypass the current merge contract. It should extend it.
Required behavior:
1. base entries still merge with `modified_ts_hash_tombstone_v2`
2. `_system.index0.events` merges by `event_id`
3. identical `event_id` payloads are idempotent
4. equal-subject competing events resolve deterministically using:
- `modified_ts`
- canonical event hash
- tombstone/deletion precedence where applicable
5. checkpoints and views are regenerated or selected deterministically
Implementation direction:
1. add a dedicated `merge_system_index0(...)` helper beside `merge_index_payloads(...)`
2. fold its result into the canonical merged payload
3. keep all timestamps replay-safe and derived from payload data, not wall clock, where merge state is computed
## 8) Deterministic Compaction
`index0` will grow quickly if it tracks chat, docs, and agent activity.
MVP compaction rules:
1. keep recent raw events under a deterministic retention window
2. roll older events into checkpoints
3. preserve enough raw events to support recent activity and recent forensic review
4. preserve checkpoint hashes indefinitely or under a much larger cap
Recommended deterministic policy:
1. retain last `N` raw events per writer stream
2. retain last `M` days of raw events globally
3. compact older events into fixed windows:
- daily first
- weekly later if scale requires
4. compute checkpoint IDs from canonical window bounds, not device-local counters
Important:
Compaction must be deterministic from payload state alone so two devices compacting the same data converge to the same post-compaction state.
Recommended initial defaults:
1. keep checkpoints indefinitely until an explicit larger cap is required
2. keep only compact canonical views in sync
3. rebuild local heavy views on demand after sync or unlock
## 9) Nostr And Communication Scope
The goal is not to make `index0` hold every byte of communication content.
Recommended split:
1. canonical communication artifacts remain normal vault data or future dedicated entry kinds
2. `index0` records:
- existence
- ownership/visibility metadata
- hierarchy placement
- conversation references
- recent activity summaries
3. optional future message/document bodies continue to live in encrypted entries or high-risk partitions
This keeps `index0` useful as a map without turning it into a giant bottleneck payload.
Retention and exposure rules:
1. public Nostr references may be stored as lightweight refs in canonical vault data
2. private DM content should default to encrypted entry payloads or high-risk partitions, not inline `index0_event` bodies
3. `index0` should store enough metadata to navigate to a message, not enough to become the message store
## 10) Hierarchy And Permission Model
`index0` must not weaken SeedPass security posture.
Rules:
1. `index0` is encrypted inside the same profile payload as the rest of the vault
2. `index0` only reveals what the active profile is already authorized to decrypt
3. manager access to subordinate data should continue to use the existing hierarchy and profile-loading model
4. future agent access should reuse approval/export/partition controls rather than adding an `index0` bypass
5. event metadata for high-risk partitions should be minimal and policy-aware
Policy alignment:
1. reuse policy stamps / policy-as-code identifiers where policy-shaped access affects atlas visibility
2. reuse audit expectations from the agent autonomy plan rather than inventing a second audit vocabulary
3. keep `classification` and `partition` fields small, deterministic, and non-secret
Institutional knowledge model:
1. each user/agent continues producing canonical artifacts in their scope
2. `index0` links those artifacts into a searchable atlas
3. manager-level profiles can build broader navigation views when they already have legitimate access to those scopes
## 11) Interaction With Existing Roadmaps
### 11.1 TUI v2 plan
`index0` is not a prerequisite for the current TUI v2 parity/cutover work.
However, once present it directly improves:
1. real hierarchy navigation
2. recent activity surfaces
3. context-aware wayfinding
4. future conversation and KB sidebars
### 11.2 Semantic index plan
The semantic vector index should remain local-only derived state.
Recommended relationship:
1. `index0` is canonical synced atlas data
2. semantic vectors remain device-local derived retrieval data
3. semantic chunking can consume `index0_view` and graph context as ranking features
### 11.3 Entry graph plan
`index0` should reuse `links` conventions instead of inventing a second graph model.
## 12) Implementation Touchpoints In `beta`
Primary files likely to change:
1. `src/seedpass/core/sync_conflict.py`
- add deterministic merge support for `_system.index0`
2. `src/seedpass/core/vault.py`
- load/save/migration handling for `_system.index0`
3. `src/seedpass/core/encryption.py`
- merge path for restored payloads
4. `src/nostr/backup_models.py`
- manifest metadata extension for `index0` checkpoint fields
5. `src/nostr/snapshot.py`
- publish/fetch manifest support for `index0` checkpoint metadata
6. `src/seedpass/core/entry_management.py`
- emit `index0_event` records on create/update/delete/link actions
7. `src/seedpass/core/api.py`
- expose atlas inspection/status endpoints
8. `src/seedpass/core/semantic_index.py`
- optional future use of `index0_view` context during hybrid retrieval
9. `src/seedpass/core/state_manager.py`
- only if local rebuild or validation state must be persisted
10. `src/seedpass/core/agent_export_policy.py`
- if atlas reads need policy-shaped export or redaction behavior
## 13) Delivery Phases
## Phase 0: Spec And Schema Lock
Deliver:
1. finalize `_system.index0` payload shape
2. finalize event/checkpoint/view schemas
3. define manifest extension fields
4. define deterministic compaction contract
Exit criteria:
1. schema doc merged
2. migration strategy documented
3. no open ambiguity on user-entry ID compatibility
## Phase 1: Canonical Storage + Merge
Deliver:
1. payload migrations for `_system.index0`
2. `merge_system_index0(...)`
3. event ID and integrity hash helpers
4. deterministic checkpoint selection rules
Exit criteria:
1. replay idempotency holds
2. stale payloads cannot override newer `index0` state
3. merge order independence proven by tests
## Phase 2: Event Emission Hooks
Deliver:
1. emit events on entry CRUD
2. emit events on archive/restore
3. emit events on link add/remove
4. emit events on profile/agent lifecycle operations where already represented in core flows
Exit criteria:
1. user-visible mutations update `index0` automatically
2. no manual rebuild required for core audit path
## Phase 3: Manifest Checkpoints + Validation
Deliver:
1. extend manifest schema
2. write checkpoint metadata during snapshot publish
3. validate checkpoint/head metadata during restore
4. define warning vs hard-fail policy
Exit criteria:
1. snapshot/delta replay can verify `index0` checkpoint continuity
2. failure states are explicit and test-covered
## Phase 4: Deterministic Compaction
Deliver:
1. raw-event retention windows
2. checkpoint rollup generation
3. deterministic pruning
4. migration path for older payloads without checkpoints
Exit criteria:
1. event growth remains bounded
2. two devices compacting the same payload converge
## Phase 5: Read APIs And Wayfinder Views
Deliver:
1. atlas status API/CLI surface
2. recent activity feed
3. children-of and counts-by-kind views
4. hot-nodes and top-tags views
Exit criteria:
1. user/agent can navigate from atlas state without scanning the full vault
2. views can be rebuilt from canonical data
## Phase 6: Communication And KB Integration
Deliver:
1. add event hooks for future Nostr chat/document workflows
2. add message/conversation references
3. integrate atlas cues into KB and semantic search experiences
4. keep raw private communication content out of canonical atlas events
Exit criteria:
1. institutional knowledge use cases work without changing cryptographic source-of-truth rules
## 14) Testing Plan
Required new tests:
1. unit:
- event hash generation
- writer-stream hash chaining
- checkpoint hash generation
- deterministic compaction selection
2. merge:
- order independence
- replay idempotency
- stale event rejection
- concurrent writer-stream head preservation
3. migration:
- old payload without `_system.index0` upgrades cleanly
- namespace reset still works
4. Nostr sync:
- manifest metadata round-trip
- snapshot + delta restore validation
- checkpoint mismatch handling
5. integration:
- CRUD emits events
- link operations emit events
- archive/delete semantics converge
6. scale:
- large KB/chat event volume
- compaction boundedness
Likely test homes:
1. `src/tests/test_delta_merge.py`
2. new `src/tests/test_index0_merge.py`
3. new `src/tests/test_index0_compaction.py`
4. new `src/tests/test_index0_manifest_validation.py`
5. targeted API/CLI/TUI tests once read surfaces exist
## 15) Sequencing Against The Current Dev Plan
Recommended sequencing in `beta`:
1. now:
- Phase 0 spec and schema lock
- no conflict with current TUI work
2. after current cutover-critical TUI/security slices are stable:
- Phases 1 through 3
3. after semantic hybrid hardening resumes:
- Phase 5 atlas read surfaces and semantic interplay
4. after communication entry types or Nostr DM indexing lands:
- Phase 6
Practical interpretation:
1. start the architecture/spec work immediately
2. do not let `index0` disrupt TUI parity closeout or current security readiness gates
3. implement core atlas plumbing as the next major core-data track once current cutover work is no longer the top release blocker
## 16) Open Design Questions
These should be resolved in Phase 0 before code lands:
1. should `writer_id` be profile-fingerprint based, device based, or actor based?
2. should checkpoint mismatch fail restore or warn and continue by default?
3. which communication payloads are canonical vault entries versus external references?
4. should manager-level aggregate atlas views be materialized or computed on demand?
5. should `writer_id` default to profile scope first and only later split by agent identity where needed?
## 17) Bottom Line
For the current `beta` branch, the right implementation is:
1. keep `index0` inside the existing encrypted snapshot/delta payload
2. model it as a reserved `_system.index0` atlas namespace
3. use lightweight hash-chained writer streams plus manifest checkpoint hashes
4. compact deterministically
5. expose fast navigation through rebuildable views
This gives SeedPass a canonical launch pad and map for institutional knowledge without abandoning the sync, determinism, and hierarchy model that `beta` already has.
+620
View File
@@ -0,0 +1,620 @@
# Index0 Phase 0 Implementation Spec (2026-03-05)
Status: Proposed
Branch target: `beta`
Depends on: `docs/index0_atlas_execution_plan_2026-03-05.md`
## 1) Goal
Lock the Phase 0 implementation contract for `index0` so coding can begin without schema drift.
This document defines:
1. canonical payload schema
2. manifest extension fields
3. normalization and hashing rules
4. deterministic merge rules
5. deterministic compaction defaults
6. first implementation slice
7. first test matrix
## 2) Canonical Payload Contract
`index0` lives inside the same encrypted vault payload in a reserved namespace:
```json
{
"schema_version": 3,
"entries": {},
"_sync_meta": {},
"_system": {
"index0": {
"schema_version": 1,
"events": {},
"checkpoints": {},
"canonical_views": {},
"view_manifest": {},
"heads": {},
"stats": {}
}
}
}
```
Normalization requirements:
1. missing `_system` must be treated as `{}`
2. missing `_system.index0` must be initialized during load or merge
3. all `events`, `checkpoints`, `canonical_views`, `heads`, and `stats` members must be JSON objects
4. all keys must be ASCII strings
5. the canonical JSON hash function must use `sort_keys=True` and compact separators `(",", ":")`
## 3) Schema Definitions
### 3.1 `index0_event`
Storage key:
- `events[event_id]`
Required fields:
```json
{
"event_id": "e:sha256hex",
"event_type": "entry_created",
"subject_type": "entry",
"subject_id": "42",
"subject_kind": "document",
"scope_path": "seed/<root>/managed/<child>",
"actor_type": "user",
"actor_id": "<profile fingerprint or agent id>",
"writer_id": "writer:<scope-or-profile>",
"modified_ts": 1760000000,
"prev_hash": "",
"classification": "internal",
"partition": "standard",
"payload_ref": {
"entry_id": "42"
},
"links": [],
"tags": [],
"visibility": "private",
"integrity_hash": "sha256hex"
}
```
Optional fields:
```json
{
"policy_ref": "<policy stamp or stable rule id>",
"source": "entry_management",
"source_event_id": "<nostr event id or external ref>",
"summary": "<short safe summary>"
}
```
Enum constraints:
1. `event_type`:
- `entry_created`
- `entry_modified`
- `entry_deleted`
- `entry_archived`
- `entry_restored`
- `link_added`
- `link_removed`
- `profile_added`
- `profile_removed`
- `agent_added`
- `agent_removed`
- `sync_snapshot_published`
- `sync_delta_applied`
- `nostr_message_indexed`
- `document_indexed`
2. `subject_type`:
- `entry`
- `profile`
- `agent`
- `conversation`
- `document`
- `sync`
3. `actor_type`:
- `user`
- `agent`
- `system`
4. `classification`:
- `public`
- `internal`
- `restricted`
- `secret`
5. `partition`:
- `standard`
- `high_risk`
6. `visibility`:
- `public`
- `private`
- `policy_scoped`
Normalization rules:
1. `event_id`, `subject_id`, `actor_id`, and `writer_id` are stored as strings
2. `modified_ts` must be an integer greater than zero
3. `prev_hash` defaults to `""`
4. `links` must use the same canonical link normalization rules already used for entries
5. `tags` must be unique and lexicographically sorted
6. `summary` must never contain secret material
7. `integrity_hash` is computed from the canonical event body excluding the `integrity_hash` field itself
### 3.2 `index0_checkpoint`
Storage key:
- `checkpoints[checkpoint_id]`
Required fields:
```json
{
"checkpoint_id": "cp:day:2026-03-05:writer:<root>",
"window_type": "day",
"window_key": "2026-03-05",
"writer_id": "writer:<root>",
"window_start_ts": 1760000000,
"window_end_ts": 1760086399,
"event_count": 512,
"head_hash": "sha256hex",
"summary_hash": "sha256hex",
"rollup": {
"events_by_type": {},
"subjects_by_kind": {},
"subjects": []
},
"modified_ts": 1760086399
}
```
Normalization rules:
1. initial implementation supports only `window_type = "day"`
2. `checkpoint_id` must be deterministic from `window_type`, `window_key`, and `writer_id`
3. `subjects` must be a sorted unique list and capped deterministically
4. `summary_hash` is computed from the canonical checkpoint body excluding `summary_hash`
### 3.3 `canonical_view`
Storage key:
- `canonical_views[view_id]`
Required fields:
```json
{
"view_id": "children_of:seed/<root>",
"view_type": "children_of",
"scope_path": "seed/<root>",
"source_checkpoint_ids": [],
"source_event_ids": [],
"data": {},
"modified_ts": 1760086400,
"view_hash": "sha256hex"
}
```
Normalization rules:
1. only compact low-cardinality views are eligible for canonical sync in Phase 1
2. `data` must be deterministic and sorted by stable keys
3. `view_hash` is computed from the canonical view body excluding `view_hash`
### 3.4 `view_manifest`
Purpose:
1. declare canonical synced views
2. version view builders
3. make local rebuild policy explicit
Schema:
```json
{
"version": 1,
"canonical_view_types": [
"children_of",
"counts_by_kind",
"recent_activity"
],
"local_only_view_types": [
"hot_nodes",
"conversation_index",
"semantic_neighbors"
],
"builder_versions": {
"children_of": 1,
"counts_by_kind": 1,
"recent_activity": 1
}
}
```
### 3.5 `heads`
Purpose:
Track current writer-stream head hashes.
Schema:
```json
{
"writer:<root>": {
"event_id": "e:sha256hex",
"head_hash": "sha256hex",
"modified_ts": 1760086400
}
}
```
Rules:
1. each `writer_id` has at most one head record
2. higher `modified_ts` wins
3. equal timestamp resolves by lexicographically larger `head_hash`
### 3.6 `stats`
Initial synced stats:
```json
{
"event_count": 1280,
"checkpoint_count": 14,
"writer_count": 3,
"last_compaction_ts": 1760086400,
"last_validation_ts": 1760086400
}
```
Stats are advisory and deterministically derived. They are never the source of truth.
## 4) Manifest Extension Contract
The current manifest already carries snapshot metadata. Extend it with optional `index0` fields:
```json
{
"ver": 1,
"algo": "gzip",
"chunks": [],
"delta_since": 1760086400,
"nonce": "base64...",
"index0": {
"schema_version": 1,
"checkpoint_ids": [
"cp:day:2026-03-05:writer:<root>"
],
"checkpoint_hashes": {
"cp:day:2026-03-05:writer:<root>": "sha256hex"
},
"stream_heads": {
"writer:<root>": "sha256hex"
}
}
}
```
Manifest rules:
1. manifest `index0` section is optional for backward compatibility
2. restore must accept manifests with no `index0`
3. manifest `index0` metadata is validation material, not source-of-truth state
4. if present, it must be recomputable from the decrypted payload after replay
## 5) Hashing And ID Rules
### 5.1 Canonical helper
Use the same canonical JSON discipline already used by the sync layer:
```python
json.dumps(value, sort_keys=True, separators=(",", ":"))
```
### 5.2 Event body hash
Pseudo:
```python
def canonical_event_body(event):
body = dict(event)
body.pop("integrity_hash", None)
return body
def compute_event_hash(event):
return sha256(canonical(canonical_event_body(event)))
```
### 5.3 Event ID
For Phase 1, event IDs should be content-addressed and deterministic:
```python
event_id = "e:" + compute_event_hash(event)
```
This intentionally makes duplicate event publication idempotent.
### 5.4 Head hash
Pseudo:
```python
def compute_head_hash(event):
marker = {
"event_id": event["event_id"],
"integrity_hash": event["integrity_hash"],
"prev_hash": event.get("prev_hash", ""),
"writer_id": event["writer_id"],
}
return sha256(canonical(marker))
```
### 5.5 Checkpoint ID
Pseudo:
```python
checkpoint_id = f"cp:day:{window_key}:{writer_id}"
```
### 5.6 Checkpoint hash
Pseudo:
```python
def compute_checkpoint_hash(checkpoint):
body = dict(checkpoint)
body.pop("summary_hash", None)
return sha256(canonical(body))
```
## 6) Writer Model
Phase 1 writer rule:
1. `writer_id` defaults to profile scope, not device instance
2. format:
- root profile: `writer:profile:<fingerprint>`
- managed scope: `writer:profile:<fingerprint>:managed:<subject>`
- agent scope later if needed: `writer:agent:<agent_id>`
Rationale:
1. profile-scoped writers are simpler and stable across devices
2. this fits current `beta` multi-device deterministic merge better than device-scoped streams
3. agent-specific streams can be added later without invalidating the contract
## 7) Deterministic Merge Rules
`merge_index_payloads(...)` remains the top-level orchestrator.
Add:
```python
merge_system_index0(current_index0, incoming_index0) -> merged_index0
```
### 7.1 Event merge
Rules:
1. merge by `event_id`
2. if only one side contains an event, keep it
3. if both sides contain the same `event_id` and canonical bodies match, keep one
4. if both sides contain the same `event_id` but bodies differ, pick the event with lexicographically larger canonical body hash and record a validation warning path later
This last case should be treated as corruption-suspicious and covered by tests even if it should not occur in normal generation.
### 7.2 Head merge
Rules:
1. merge by `writer_id`
2. higher `modified_ts` wins
3. equal timestamp resolves by larger `head_hash`
### 7.3 Checkpoint merge
Rules:
1. merge by `checkpoint_id`
2. if both sides contain the same `checkpoint_id` with different bodies, select lexicographically larger `summary_hash`
3. equal bodies remain idempotent
### 7.4 Canonical view merge
Rules:
1. merge by `view_id`
2. higher `modified_ts` wins
3. equal timestamp resolves by larger `view_hash`
4. local-only views must never be written into canonical synced payload
### 7.5 Stats merge
Rules:
1. stats are recomputed from merged payload where practical
2. if not recomputed, use deterministic max/union semantics only
## 8) Merge Pseudocode
```python
def merge_system_index0(current, incoming):
cur = normalize_index0(current)
inc = normalize_index0(incoming)
merged_events = merge_events(cur["events"], inc["events"])
merged_heads = merge_heads(cur["heads"], inc["heads"])
merged_checkpoints = merge_checkpoints(
cur["checkpoints"], inc["checkpoints"]
)
merged_views = merge_canonical_views(
cur["canonical_views"], inc["canonical_views"], cur["view_manifest"], inc["view_manifest"]
)
merged_view_manifest = merge_view_manifest(
cur["view_manifest"], inc["view_manifest"]
)
merged = {
"schema_version": max(cur["schema_version"], inc["schema_version"]),
"events": merged_events,
"checkpoints": merged_checkpoints,
"canonical_views": merged_views,
"view_manifest": merged_view_manifest,
"heads": merged_heads,
"stats": recompute_stats(
merged_events, merged_checkpoints, merged_heads
),
}
return compact_index0(merged)
```
## 9) Deterministic Compaction Defaults
Phase 1 compaction must stay intentionally simple.
Defaults:
1. raw event retention per writer:
- keep last `512` raw events
2. raw event retention by age:
- keep last `30` days
3. checkpoint window:
- daily
4. canonical view retention:
- keep only latest view per `view_id`
5. checkpoint subject list cap:
- `128`
Compaction order:
1. group events by `writer_id`
2. within each writer, sort by:
- `modified_ts`
- `event_id`
3. materialize daily checkpoints for prunable windows
4. prune raw events only after checkpoint exists
5. recompute heads and stats
Guarantee:
If two devices compact the same normalized payload, they must produce the same payload.
## 10) Initial Builder Scope
The first code slice should not try to solve the full knowledge-base product.
Phase 1 implementation should include only:
1. `_system.index0` schema initialization and migration
2. event emission for:
- entry create
- entry modify
- entry delete
- entry archive
- entry restore
- link add
- link remove
3. merge support
4. head maintenance
5. manifest `index0` metadata round-trip
Deferred until later phases:
1. communication-specific event emission
2. local rebuild daemons
3. UI-facing atlas panels
4. semantic interplay
5. policy-filtered atlas export
## 11) Failure Handling
Rules:
1. malformed `index0` objects must normalize to empty structures where safe
2. malformed individual events/checkpoints/views should be dropped during normalization if required fields are invalid
3. manifest `index0` mismatches should warn by default in the first implementation
4. hard-fail behavior can be added later behind policy/config
## 12) Test Matrix
### 12.1 Unit
1. `normalize_index0` fills defaults and drops malformed objects
2. `compute_event_hash` is stable
3. `compute_head_hash` is stable
4. `compute_checkpoint_hash` is stable
5. tag normalization is deterministic
6. checkpoint ID generation is deterministic
### 12.2 Merge
1. event merge is idempotent
2. event merge is order independent
3. conflicting same-`event_id` bodies resolve deterministically
4. head merge prefers newer `modified_ts`
5. equal timestamp head merge prefers larger `head_hash`
6. view merge rejects local-only view types in canonical payload
### 12.3 Replay / vault boundary
1. payload with `_system.index0` survives encrypt/decrypt round-trip
2. repeated merge through vault boundary is idempotent
3. stale payload replay cannot remove newer heads/checkpoints
### 12.4 Migration
1. old payload without `_system` upgrades cleanly
2. old payload with malformed `_system.index0` repairs cleanly
3. existing entry CRUD remains unchanged
### 12.5 Nostr manifest
1. manifest serializes and parses optional `index0`
2. restore accepts legacy manifest with no `index0`
3. restore warning path triggers on checkpoint/head mismatch
### 12.6 Integration
1. add entry emits `entry_created`
2. modify entry emits `entry_modified`
3. delete path emits `entry_deleted`
4. archive/restore emit expected events
5. link add/remove emit expected events
## 13) Implementation Order
Recommended coding order:
1. add normalization + hashing helpers in a new `index0` module
2. extend merge logic with `merge_system_index0`
3. extend manifest dataclass and serializer/parser
4. add payload migration/init in vault load path
5. emit events from entry-management hooks
6. add tests in lockstep
## 14) Definition Of Phase 0 Complete
Phase 0 is complete when:
1. this schema contract is accepted
2. manifest extension fields are accepted
3. merge and compaction rules are accepted
4. first implementation slice is accepted
5. test inventory is accepted
At that point, implementation can move directly into Phase 1 without reopening architecture questions.