mirror of
https://github.com/PR0M3TH3AN/TORCH.git
synced 2026-10-27 07:51:52 +00:00
feat(memory): add append-only event contract with build/verify CLI
This commit is contained in:
thePR0M3TH3AN committed
1 parent
20ae04ed8a
commit
393ba35f32
12 files changed
+921
-5
No files matched your search
@@ -149,6 +149,18 @@ npm run scheduler:weekly
|
||||
The `torch-lock` CLI also supports memory operations:
|
||||
|
||||
```bash
|
||||
# Append one schema-validated memory event (append-only)
|
||||
npx --no-install torch-lock memory add \
|
||||
--agent codex \
|
||||
--topic memory \
|
||||
--message "Lock preflight failed on relay outage; rerun with fallback relays."
|
||||
|
||||
# Build deterministic memory_update.md from memory/events/**/*.json
|
||||
npx --no-install torch-lock memory build
|
||||
|
||||
# Verify schema + duplicates + determinism + generated-file no-diff
|
||||
npx --no-install torch-lock memory verify
|
||||
|
||||
# List memories (optional filters: --agent, --type, --tags, --pinned)
|
||||
npx --no-install torch-lock list-memories --limit 5
|
||||
|
||||
@@ -163,6 +175,12 @@ npx --no-install torch-lock unpin-memory --id <memory-id>
|
||||
npx --no-install torch-lock memory-stats
|
||||
```
|
||||
|
||||
Memory event contract:
|
||||
- Source: `memory/events/YYYY/MM/DD/<timestamp>_<id>.json` (append-only).
|
||||
- Schema: `memory/schema/memory-event.schema.json`.
|
||||
- Generated file: `memory_update.md` with auto-generated header.
|
||||
- Full contract: `docs/memory-events-contract.md`.
|
||||
|
||||
### Prompt Governance & Versioning CLI
|
||||
|
||||
Agent prompts are changed through a governance workflow that archives the previous version before overwriting, enabling full rollback.
|
||||
|
||||
@@ -94,10 +94,11 @@ At the start of a session, the scheduler retrieves relevant memories based on yo
|
||||
- **Action:** Read this file to learn from past agents (e.g., "The integration test suite is flaky on the login step", "Use `node:test` instead of Jest").
|
||||
|
||||
#### Writing Memory
|
||||
To save a learning for future agents, create a memory file before finishing your task.
|
||||
- **File:** `memory-update.md` (in the repository root).
|
||||
- **Content:** Concise, high-value insights.
|
||||
- **Mechanism:** The scheduler will automatically ingest this file after your run completes.
|
||||
To save a learning for future agents, write an append-only memory event and rebuild generated memory docs.
|
||||
- **Source of truth:** `memory/events/YYYY/MM/DD/<timestamp>_<id>.json` (append-only, never edit old events).
|
||||
- **Generated output:** `memory_update.md` (auto-generated; do not hand-edit).
|
||||
- **Commands:** `torch-lock memory add`, `torch-lock memory build`, `torch-lock memory verify`.
|
||||
- **Mechanism:** `memory verify` enforces schema validity, duplicate-id protection, deterministic rendering, and no-diff output checks.
|
||||
|
||||
**Good Memories:**
|
||||
- "Fix for `EADDRINUSE` in tests: ensure server.close() is called in `after()`."
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
# Memory Events Contract
|
||||
|
||||
This contract defines the merge-safe memory pipeline used by `torch-lock memory`.
|
||||
|
||||
## Source of Truth
|
||||
|
||||
- Append-only event files:
|
||||
- `memory/events/YYYY/MM/DD/<timestamp>_<id>.json`
|
||||
- Event schema:
|
||||
- `memory/schema/memory-event.schema.json`
|
||||
- Generated artifact:
|
||||
- `memory_update.md`
|
||||
|
||||
## Commands
|
||||
|
||||
### `torch-lock memory add`
|
||||
|
||||
Writes exactly one new event file.
|
||||
|
||||
Required flags:
|
||||
- `--agent`
|
||||
- `--topic`
|
||||
- one of `--message` or `--message-file`
|
||||
|
||||
Optional flags:
|
||||
- `--created-at` (RFC3339 UTC, default: now)
|
||||
- `--id` (UUIDv7, default: generated)
|
||||
- `--run-id`
|
||||
- `--tags` (comma-separated)
|
||||
- `--source`
|
||||
- `--project` (defaults to `TORCH`; non-`TORCH` is rejected by schema)
|
||||
- `--events-dir` (default: `memory/events`)
|
||||
- `--schema-path` (default: `memory/schema/memory-event.schema.json`)
|
||||
|
||||
Behavior:
|
||||
- Validates event against schema contract before writing.
|
||||
- Enforces append-only naming: `<timestamp>_<id>.json`.
|
||||
- Fails if target file already exists.
|
||||
|
||||
### `torch-lock memory build`
|
||||
|
||||
Builds deterministic markdown from all events.
|
||||
|
||||
Optional flags:
|
||||
- `--events-dir` (default: `memory/events`)
|
||||
- `--schema-path` (default: `memory/schema/memory-event.schema.json`)
|
||||
- `--output` (default: `memory_update.md`)
|
||||
|
||||
Behavior:
|
||||
- Loads and validates all event JSON files.
|
||||
- Enforces filename/date/id contract.
|
||||
- Fails on duplicate ids.
|
||||
- Sorts by `(created_at ASC, id ASC, relative_path ASC)`.
|
||||
- Emits auto-generated header:
|
||||
- `<!-- AUTO-GENERATED: DO NOT EDIT. Source: memory/events -->`
|
||||
- Normalizes output to LF newlines and one trailing newline.
|
||||
|
||||
### `torch-lock memory verify`
|
||||
|
||||
Verification gate for CI and branch safety.
|
||||
|
||||
Optional flags:
|
||||
- `--events-dir` (default: `memory/events`)
|
||||
- `--schema-path` (default: `memory/schema/memory-event.schema.json`)
|
||||
- `--output` (default: `memory_update.md`)
|
||||
|
||||
Behavior:
|
||||
- Re-validates all events + path contract.
|
||||
- Re-renders twice (determinism guard).
|
||||
- Fails if output file is missing header.
|
||||
- Fails if output file bytes differ from deterministic render (no-diff gate).
|
||||
|
||||
## Exit Codes
|
||||
|
||||
- `0`: success
|
||||
- `1`: usage error (missing/invalid CLI arguments)
|
||||
- `4`: memory contract violation
|
||||
- schema invalid
|
||||
- filename/date mismatch
|
||||
- duplicate event id
|
||||
- deterministic render mismatch
|
||||
- generated file missing header
|
||||
- generated file out of date
|
||||
@@ -0,0 +1,68 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://torch.local/schemas/memory-event.schema.json",
|
||||
"title": "TORCH Memory Event",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"id",
|
||||
"created_at",
|
||||
"project",
|
||||
"agent",
|
||||
"topic",
|
||||
"content"
|
||||
],
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string",
|
||||
"description": "UUIDv7",
|
||||
"pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$"
|
||||
},
|
||||
"created_at": {
|
||||
"type": "string",
|
||||
"description": "UTC timestamp in RFC3339 format",
|
||||
"format": "date-time",
|
||||
"pattern": "Z$"
|
||||
},
|
||||
"project": {
|
||||
"type": "string",
|
||||
"const": "TORCH"
|
||||
},
|
||||
"agent": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"maxLength": 64,
|
||||
"pattern": "^[a-zA-Z0-9_.-]+$"
|
||||
},
|
||||
"topic": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"maxLength": 128
|
||||
},
|
||||
"content": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"maxLength": 20000
|
||||
},
|
||||
"run_id": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"maxLength": 128
|
||||
},
|
||||
"tags": {
|
||||
"type": "array",
|
||||
"maxItems": 20,
|
||||
"items": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"maxLength": 64
|
||||
},
|
||||
"uniqueItems": true
|
||||
},
|
||||
"source": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"maxLength": 256
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -38,6 +38,7 @@
|
||||
"scripts/check-innerhtml.mjs",
|
||||
"scripts/test-audit/",
|
||||
"scripts/memory/",
|
||||
"memory/schema/",
|
||||
"migrations/",
|
||||
"skills/",
|
||||
"dashboard/",
|
||||
@@ -75,6 +76,9 @@
|
||||
"test:playwright:ui": "playwright test --ui",
|
||||
"torch:memory:list": "node bin/torch-lock.mjs list-memories",
|
||||
"torch:memory:inspect": "node bin/torch-lock.mjs inspect-memory",
|
||||
"torch:memory:add": "node bin/torch-lock.mjs memory add",
|
||||
"torch:memory:build": "node bin/torch-lock.mjs memory build",
|
||||
"torch:memory:verify": "node bin/torch-lock.mjs memory verify",
|
||||
"torch:remove": "node bin/torch-lock.mjs remove"
|
||||
},
|
||||
"dependencies": {
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
test_integrity_note:
|
||||
change_type: "new_tests"
|
||||
scenarios:
|
||||
- id: SCN-memory-event-add-append-only
|
||||
given: "A repo with memory schema configured"
|
||||
when: "An operator runs `torch-lock memory add`"
|
||||
then: "Exactly one new schema-valid event file is created in a dated append-only path"
|
||||
- id: SCN-memory-build-deterministic-order
|
||||
given: "Multiple memory event files with sortable timestamps and IDs"
|
||||
when: "An operator runs `torch-lock memory build`"
|
||||
then: "Generated markdown is deterministic and sorted by created_at then id"
|
||||
- id: SCN-memory-verify-no-diff-gate
|
||||
given: "A hand-edited or stale generated memory markdown file"
|
||||
when: "An operator runs `torch-lock memory verify`"
|
||||
then: "Verification fails with a contract error requiring rebuild"
|
||||
- id: SCN-memory-verify-duplicate-id
|
||||
given: "Two events share the same id"
|
||||
when: "An operator runs `torch-lock memory verify`"
|
||||
then: "Verification fails and reports duplicate id"
|
||||
observable_outcomes:
|
||||
- "JSON event files written at memory/events/YYYY/MM/DD/<timestamp>_<id>.json"
|
||||
- "Generated memory_update.md includes deterministic header and ordering"
|
||||
- "CLI returns non-zero ExitError code 4 on contract violations"
|
||||
determinism_controls:
|
||||
- "Fixed created_at values in tests"
|
||||
- "Stable sort key contract (created_at, id, relative_path)"
|
||||
- "Byte-level equality checks for generated output"
|
||||
anti_cheat_rationale:
|
||||
prevents:
|
||||
- "hard-coded return value"
|
||||
- "over-mocking internal logic"
|
||||
- "snapshot rubber-stamping"
|
||||
- "retry/sleep-based flake masking"
|
||||
relaxation:
|
||||
did_relax_any_assertion: false
|
||||
if_true_explain_spec_basis: ""
|
||||
@@ -42,6 +42,15 @@ export function parseArgs(argv) {
|
||||
status: null,
|
||||
list: false,
|
||||
output: null,
|
||||
topic: null,
|
||||
runId: null,
|
||||
message: null,
|
||||
messageFile: null,
|
||||
eventsDir: null,
|
||||
schemaPath: null,
|
||||
createdAt: null,
|
||||
source: null,
|
||||
project: null,
|
||||
};
|
||||
let i = 0;
|
||||
|
||||
@@ -53,6 +62,10 @@ export function parseArgs(argv) {
|
||||
args.subcommand = argv[1];
|
||||
i = 2;
|
||||
}
|
||||
if (args.command === 'memory' && argv.length > 1 && !argv[1].startsWith('-')) {
|
||||
args.subcommand = argv[1];
|
||||
i = 2;
|
||||
}
|
||||
}
|
||||
|
||||
for (; i < argv.length; i++) {
|
||||
@@ -147,6 +160,42 @@ export function parseArgs(argv) {
|
||||
args.target = argv[++i];
|
||||
} else if (arg.startsWith('--target=')) {
|
||||
args.target = arg.split('=')[1];
|
||||
} else if (arg === '--topic') {
|
||||
args.topic = argv[++i];
|
||||
} else if (arg.startsWith('--topic=')) {
|
||||
args.topic = arg.split('=')[1];
|
||||
} else if (arg === '--run-id') {
|
||||
args.runId = argv[++i];
|
||||
} else if (arg.startsWith('--run-id=')) {
|
||||
args.runId = arg.split('=')[1];
|
||||
} else if (arg === '--message') {
|
||||
args.message = argv[++i];
|
||||
} else if (arg.startsWith('--message=')) {
|
||||
args.message = arg.split('=')[1];
|
||||
} else if (arg === '--message-file') {
|
||||
args.messageFile = argv[++i];
|
||||
} else if (arg.startsWith('--message-file=')) {
|
||||
args.messageFile = arg.split('=')[1];
|
||||
} else if (arg === '--events-dir') {
|
||||
args.eventsDir = argv[++i];
|
||||
} else if (arg.startsWith('--events-dir=')) {
|
||||
args.eventsDir = arg.split('=')[1];
|
||||
} else if (arg === '--schema-path') {
|
||||
args.schemaPath = argv[++i];
|
||||
} else if (arg.startsWith('--schema-path=')) {
|
||||
args.schemaPath = arg.split('=')[1];
|
||||
} else if (arg === '--created-at') {
|
||||
args.createdAt = argv[++i];
|
||||
} else if (arg.startsWith('--created-at=')) {
|
||||
args.createdAt = arg.split('=')[1];
|
||||
} else if (arg === '--source') {
|
||||
args.source = argv[++i];
|
||||
} else if (arg.startsWith('--source=')) {
|
||||
args.source = arg.split('=')[1];
|
||||
} else if (arg === '--project') {
|
||||
args.project = argv[++i];
|
||||
} else if (arg.startsWith('--project=')) {
|
||||
args.project = arg.split('=')[1];
|
||||
} else if (arg === '--content') {
|
||||
args.content = argv[++i];
|
||||
} else if (arg.startsWith('--content=')) {
|
||||
|
||||
@@ -0,0 +1,467 @@
|
||||
import fs from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
import crypto from 'node:crypto';
|
||||
import { ExitError } from './errors.mjs';
|
||||
|
||||
const AUTO_HEADER = '<!-- AUTO-GENERATED: DO NOT EDIT. Source: memory/events -->';
|
||||
const DEFAULT_EVENTS_DIR = 'memory/events';
|
||||
const DEFAULT_SCHEMA_PATH = 'memory/schema/memory-event.schema.json';
|
||||
const DEFAULT_OUTPUT_PATH = 'memory_update.md';
|
||||
|
||||
const UUID_V7_REGEX = /^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/;
|
||||
const VALID_AGENT_REGEX = /^[a-zA-Z0-9_.-]+$/;
|
||||
|
||||
function ensureObject(value, context) {
|
||||
if (!value || typeof value !== 'object' || Array.isArray(value)) {
|
||||
throw new ExitError(4, `${context} must be an object`);
|
||||
}
|
||||
}
|
||||
|
||||
function hasOwn(obj, key) {
|
||||
return Object.prototype.hasOwnProperty.call(obj, key);
|
||||
}
|
||||
|
||||
function normalizeToLf(text) {
|
||||
return String(text).replace(/\r\n/g, '\n').replace(/\r/g, '\n');
|
||||
}
|
||||
|
||||
function normalizeIsoToSecondPrecision(iso) {
|
||||
const date = new Date(iso);
|
||||
if (Number.isNaN(date.getTime())) return null;
|
||||
return date.toISOString().replace(/\.\d{3}Z$/, 'Z');
|
||||
}
|
||||
|
||||
function toCompactTimestamp(createdAtIso) {
|
||||
const normalized = normalizeIsoToSecondPrecision(createdAtIso);
|
||||
if (!normalized) return null;
|
||||
return normalized.replace(/[-:]/g, '').replace('.000', '');
|
||||
}
|
||||
|
||||
function generateUuidV7() {
|
||||
const bytes = crypto.randomBytes(16);
|
||||
const nowMs = BigInt(Date.now());
|
||||
|
||||
bytes[0] = Number((nowMs >> 40n) & 0xffn);
|
||||
bytes[1] = Number((nowMs >> 32n) & 0xffn);
|
||||
bytes[2] = Number((nowMs >> 24n) & 0xffn);
|
||||
bytes[3] = Number((nowMs >> 16n) & 0xffn);
|
||||
bytes[4] = Number((nowMs >> 8n) & 0xffn);
|
||||
bytes[5] = Number(nowMs & 0xffn);
|
||||
|
||||
bytes[6] = (bytes[6] & 0x0f) | 0x70;
|
||||
bytes[8] = (bytes[8] & 0x3f) | 0x80;
|
||||
|
||||
const hex = bytes.toString('hex');
|
||||
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20, 32)}`;
|
||||
}
|
||||
|
||||
async function readJsonFile(filePath, contextLabel) {
|
||||
let raw;
|
||||
try {
|
||||
raw = await fs.readFile(filePath, 'utf8');
|
||||
} catch (error) {
|
||||
throw new ExitError(4, `${contextLabel}: unable to read ${filePath}: ${error.message}`);
|
||||
}
|
||||
|
||||
try {
|
||||
return JSON.parse(raw);
|
||||
} catch (error) {
|
||||
throw new ExitError(4, `${contextLabel}: invalid JSON in ${filePath}: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function readSchema(schemaPath) {
|
||||
const schema = await readJsonFile(schemaPath, 'schema validation failed');
|
||||
ensureObject(schema, 'Schema');
|
||||
if (schema.type !== 'object') {
|
||||
throw new ExitError(4, 'schema validation failed: top-level schema type must be object');
|
||||
}
|
||||
return schema;
|
||||
}
|
||||
|
||||
function validateStringField(errors, event, field, options = {}) {
|
||||
const value = event[field];
|
||||
if (value == null) return;
|
||||
if (typeof value !== 'string') {
|
||||
errors.push(`${field} must be a string`);
|
||||
return;
|
||||
}
|
||||
if (typeof options.minLength === 'number' && value.length < options.minLength) {
|
||||
errors.push(`${field} must be at least ${options.minLength} characters`);
|
||||
}
|
||||
if (typeof options.maxLength === 'number' && value.length > options.maxLength) {
|
||||
errors.push(`${field} must be at most ${options.maxLength} characters`);
|
||||
}
|
||||
if (options.pattern && !options.pattern.test(value)) {
|
||||
errors.push(`${field} does not match required format`);
|
||||
}
|
||||
if (options.const && value !== options.const) {
|
||||
errors.push(`${field} must equal ${options.const}`);
|
||||
}
|
||||
}
|
||||
|
||||
function validateTags(errors, event) {
|
||||
if (!hasOwn(event, 'tags')) return;
|
||||
const { tags } = event;
|
||||
if (!Array.isArray(tags)) {
|
||||
errors.push('tags must be an array when provided');
|
||||
return;
|
||||
}
|
||||
if (tags.length > 20) {
|
||||
errors.push('tags must contain at most 20 items');
|
||||
}
|
||||
const seen = new Set();
|
||||
for (const [index, tag] of tags.entries()) {
|
||||
if (typeof tag !== 'string') {
|
||||
errors.push(`tags[${index}] must be a string`);
|
||||
continue;
|
||||
}
|
||||
if (tag.length < 1 || tag.length > 64) {
|
||||
errors.push(`tags[${index}] must be 1..64 characters`);
|
||||
}
|
||||
if (seen.has(tag)) {
|
||||
errors.push('tags must be unique');
|
||||
break;
|
||||
}
|
||||
seen.add(tag);
|
||||
}
|
||||
}
|
||||
|
||||
function validateEventObject(event, schemaPath) {
|
||||
const errors = [];
|
||||
|
||||
ensureObject(event, 'Event');
|
||||
const allowedProps = new Set([
|
||||
'id',
|
||||
'created_at',
|
||||
'project',
|
||||
'agent',
|
||||
'topic',
|
||||
'content',
|
||||
'run_id',
|
||||
'tags',
|
||||
'source',
|
||||
]);
|
||||
|
||||
const requiredProps = ['id', 'created_at', 'project', 'agent', 'topic', 'content'];
|
||||
for (const key of requiredProps) {
|
||||
if (!hasOwn(event, key)) {
|
||||
errors.push(`missing required field: ${key}`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const key of Object.keys(event)) {
|
||||
if (!allowedProps.has(key)) {
|
||||
errors.push(`unexpected field: ${key}`);
|
||||
}
|
||||
}
|
||||
|
||||
validateStringField(errors, event, 'id', { pattern: UUID_V7_REGEX });
|
||||
validateStringField(errors, event, 'created_at');
|
||||
validateStringField(errors, event, 'project', { const: 'TORCH' });
|
||||
validateStringField(errors, event, 'agent', { minLength: 1, maxLength: 64, pattern: VALID_AGENT_REGEX });
|
||||
validateStringField(errors, event, 'topic', { minLength: 1, maxLength: 128 });
|
||||
validateStringField(errors, event, 'content', { minLength: 1, maxLength: 20000 });
|
||||
validateStringField(errors, event, 'run_id', { minLength: 1, maxLength: 128 });
|
||||
validateStringField(errors, event, 'source', { minLength: 1, maxLength: 256 });
|
||||
validateTags(errors, event);
|
||||
|
||||
if (typeof event.created_at === 'string') {
|
||||
if (!event.created_at.endsWith('Z')) {
|
||||
errors.push('created_at must be UTC and end with Z');
|
||||
}
|
||||
const normalized = normalizeIsoToSecondPrecision(event.created_at);
|
||||
if (!normalized) {
|
||||
errors.push('created_at must be a valid RFC3339 timestamp');
|
||||
}
|
||||
}
|
||||
|
||||
if (errors.length > 0) {
|
||||
throw new ExitError(4, `schema validation failed (${schemaPath}): ${errors.join('; ')}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function ensureDir(dirPath) {
|
||||
await fs.mkdir(dirPath, { recursive: true });
|
||||
}
|
||||
|
||||
async function collectEventFiles(eventsDir) {
|
||||
async function walk(dir) {
|
||||
const entries = await fs.readdir(dir, { withFileTypes: true }).catch((error) => {
|
||||
if (error.code === 'ENOENT') return [];
|
||||
throw error;
|
||||
});
|
||||
|
||||
const files = await Promise.all(entries.map(async (entry) => {
|
||||
const full = path.join(dir, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
return walk(full);
|
||||
}
|
||||
return full.endsWith('.json') ? [full] : [];
|
||||
}));
|
||||
|
||||
return files.flat();
|
||||
}
|
||||
|
||||
const absoluteEventsDir = path.resolve(process.cwd(), eventsDir);
|
||||
const files = await walk(absoluteEventsDir);
|
||||
return files.sort((a, b) => a.localeCompare(b));
|
||||
}
|
||||
|
||||
function createdAtParts(createdAt) {
|
||||
const normalized = normalizeIsoToSecondPrecision(createdAt);
|
||||
if (!normalized) return null;
|
||||
return {
|
||||
normalized,
|
||||
year: normalized.slice(0, 4),
|
||||
month: normalized.slice(5, 7),
|
||||
day: normalized.slice(8, 10),
|
||||
compact: toCompactTimestamp(normalized),
|
||||
};
|
||||
}
|
||||
|
||||
function validatePathContract(filePath, eventsDir, event) {
|
||||
const absoluteEventsDir = path.resolve(process.cwd(), eventsDir);
|
||||
const absoluteFile = path.resolve(filePath);
|
||||
const relative = path.relative(absoluteEventsDir, absoluteFile);
|
||||
const relPosix = relative.split(path.sep).join('/');
|
||||
const parts = relPosix.split('/');
|
||||
|
||||
if (parts.length < 4) {
|
||||
throw new ExitError(4, `filename/date mismatch: ${relPosix} must be YYYY/MM/DD/<timestamp>_<id>.json`);
|
||||
}
|
||||
|
||||
const fileName = parts[parts.length - 1];
|
||||
const [baseIdPart] = fileName.split('.json');
|
||||
const underscore = baseIdPart.indexOf('_');
|
||||
if (underscore <= 0) {
|
||||
throw new ExitError(4, `filename/date mismatch: ${relPosix} must include "<timestamp>_<id>.json"`);
|
||||
}
|
||||
|
||||
const fileCompactTs = baseIdPart.slice(0, underscore);
|
||||
const fileId = baseIdPart.slice(underscore + 1);
|
||||
const dateParts = createdAtParts(event.created_at);
|
||||
if (!dateParts) {
|
||||
throw new ExitError(4, `filename/date mismatch: invalid created_at in ${relPosix}`);
|
||||
}
|
||||
|
||||
const [year, month, day] = parts;
|
||||
if (year !== dateParts.year || month !== dateParts.month || day !== dateParts.day) {
|
||||
throw new ExitError(4, `filename/date mismatch: ${relPosix} date path does not match created_at`);
|
||||
}
|
||||
if (fileCompactTs !== dateParts.compact) {
|
||||
throw new ExitError(4, `filename/date mismatch: ${relPosix} timestamp prefix does not match created_at`);
|
||||
}
|
||||
if (fileId !== event.id) {
|
||||
throw new ExitError(4, `filename/date mismatch: ${relPosix} id suffix does not match event.id`);
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeContent(content) {
|
||||
const lines = normalizeToLf(content).split('\n').map((line) => line.replace(/[ \t]+$/g, ''));
|
||||
return lines.join('\n').trim();
|
||||
}
|
||||
|
||||
function renderMarkdown(events) {
|
||||
const out = [AUTO_HEADER, '# Memory Update', ''];
|
||||
let currentDate = null;
|
||||
|
||||
for (const event of events) {
|
||||
const date = event.created_at.slice(0, 10);
|
||||
if (date !== currentDate) {
|
||||
if (currentDate !== null) out.push('');
|
||||
out.push(`## ${date}`, '');
|
||||
currentDate = date;
|
||||
}
|
||||
|
||||
out.push(`### ${event.created_at} · ${event.topic} · ${event.agent}`);
|
||||
out.push(`- id: ${event.id}`);
|
||||
if (event.run_id) out.push(`- run_id: ${event.run_id}`);
|
||||
if (Array.isArray(event.tags) && event.tags.length > 0) out.push(`- tags: ${event.tags.join(', ')}`);
|
||||
if (event.source) out.push(`- source: ${event.source}`);
|
||||
out.push('');
|
||||
out.push(normalizeContent(event.content));
|
||||
out.push('');
|
||||
}
|
||||
|
||||
const normalized = out.join('\n').replace(/[ \t]+\n/g, '\n');
|
||||
return `${normalized.trimEnd()}\n`;
|
||||
}
|
||||
|
||||
async function loadAndValidateEvents({ eventsDir, schemaPath }) {
|
||||
await readSchema(schemaPath);
|
||||
const files = await collectEventFiles(eventsDir);
|
||||
const events = [];
|
||||
const seenIds = new Set();
|
||||
|
||||
for (const filePath of files) {
|
||||
const event = await readJsonFile(filePath, 'schema validation failed');
|
||||
validateEventObject(event, schemaPath);
|
||||
validatePathContract(filePath, eventsDir, event);
|
||||
if (seenIds.has(event.id)) {
|
||||
throw new ExitError(4, `duplicate id detected: ${event.id}`);
|
||||
}
|
||||
seenIds.add(event.id);
|
||||
|
||||
events.push({
|
||||
...event,
|
||||
__path: path.relative(path.resolve(process.cwd(), eventsDir), path.resolve(filePath)).split(path.sep).join('/'),
|
||||
created_at: normalizeIsoToSecondPrecision(event.created_at),
|
||||
});
|
||||
}
|
||||
|
||||
events.sort((a, b) =>
|
||||
a.created_at.localeCompare(b.created_at)
|
||||
|| a.id.localeCompare(b.id)
|
||||
|| a.__path.localeCompare(b.__path));
|
||||
|
||||
return events;
|
||||
}
|
||||
|
||||
async function writeFileEnsuringDir(filePath, content) {
|
||||
const absolute = path.resolve(process.cwd(), filePath);
|
||||
await ensureDir(path.dirname(absolute));
|
||||
await fs.writeFile(absolute, content, 'utf8');
|
||||
}
|
||||
|
||||
function getContentFromArgs(args) {
|
||||
if (args.message && args.messageFile) {
|
||||
throw new ExitError(1, 'choose either --message or --message-file');
|
||||
}
|
||||
if (args.message) {
|
||||
return args.message;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
async function resolveAddContent(args) {
|
||||
const inline = getContentFromArgs(args);
|
||||
if (inline != null) return inline;
|
||||
|
||||
if (args.messageFile) {
|
||||
try {
|
||||
return await fs.readFile(args.messageFile, 'utf8');
|
||||
} catch (error) {
|
||||
throw new ExitError(1, `unable to read --message-file: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
throw new ExitError(1, 'memory add requires --message or --message-file');
|
||||
}
|
||||
|
||||
export async function cmdMemoryAdd(args) {
|
||||
const eventsDir = args.eventsDir || DEFAULT_EVENTS_DIR;
|
||||
const schemaPath = args.schemaPath || DEFAULT_SCHEMA_PATH;
|
||||
|
||||
if (!args.agent) throw new ExitError(1, 'memory add requires --agent');
|
||||
if (!args.topic) throw new ExitError(1, 'memory add requires --topic');
|
||||
|
||||
const message = await resolveAddContent(args);
|
||||
const createdAt = args.createdAt || new Date().toISOString().replace(/\.\d{3}Z$/, 'Z');
|
||||
const id = args.id || generateUuidV7();
|
||||
const project = args.project || 'TORCH';
|
||||
const tags = Array.isArray(args.tags) && args.tags.length > 0 ? args.tags : undefined;
|
||||
|
||||
const event = {
|
||||
id,
|
||||
created_at: createdAt,
|
||||
project,
|
||||
agent: args.agent,
|
||||
topic: args.topic,
|
||||
content: message,
|
||||
...(args.runId ? { run_id: args.runId } : {}),
|
||||
...(tags ? { tags } : {}),
|
||||
...(args.source ? { source: args.source } : {}),
|
||||
};
|
||||
|
||||
await readSchema(schemaPath);
|
||||
validateEventObject(event, schemaPath);
|
||||
|
||||
const dateParts = createdAtParts(event.created_at);
|
||||
if (!dateParts) {
|
||||
throw new ExitError(4, 'created_at must be a valid RFC3339 timestamp');
|
||||
}
|
||||
|
||||
const relativePath = path.posix.join(
|
||||
dateParts.year,
|
||||
dateParts.month,
|
||||
dateParts.day,
|
||||
`${dateParts.compact}_${event.id}.json`,
|
||||
);
|
||||
const outputPath = path.resolve(process.cwd(), eventsDir, relativePath);
|
||||
await ensureDir(path.dirname(outputPath));
|
||||
|
||||
try {
|
||||
await fs.access(outputPath);
|
||||
throw new ExitError(4, `event file already exists: ${outputPath}`);
|
||||
} catch (error) {
|
||||
if (error instanceof ExitError) throw error;
|
||||
if (error.code !== 'ENOENT') {
|
||||
throw new ExitError(4, `unable to access event file path: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
const serialized = `${JSON.stringify(event, null, 2)}\n`;
|
||||
await fs.writeFile(outputPath, serialized, 'utf8');
|
||||
|
||||
console.log(JSON.stringify({
|
||||
ok: true,
|
||||
id: event.id,
|
||||
created_at: normalizeIsoToSecondPrecision(event.created_at),
|
||||
path: path.relative(process.cwd(), outputPath).split(path.sep).join('/'),
|
||||
}, null, 2));
|
||||
}
|
||||
|
||||
export async function cmdMemoryBuild(args) {
|
||||
const eventsDir = args.eventsDir || DEFAULT_EVENTS_DIR;
|
||||
const schemaPath = args.schemaPath || DEFAULT_SCHEMA_PATH;
|
||||
const outputPath = args.output || DEFAULT_OUTPUT_PATH;
|
||||
|
||||
const events = await loadAndValidateEvents({ eventsDir, schemaPath });
|
||||
const rendered = renderMarkdown(events);
|
||||
await writeFileEnsuringDir(outputPath, rendered);
|
||||
|
||||
console.log(JSON.stringify({
|
||||
ok: true,
|
||||
output: outputPath,
|
||||
events: events.length,
|
||||
}, null, 2));
|
||||
}
|
||||
|
||||
export async function cmdMemoryVerify(args) {
|
||||
const eventsDir = args.eventsDir || DEFAULT_EVENTS_DIR;
|
||||
const schemaPath = args.schemaPath || DEFAULT_SCHEMA_PATH;
|
||||
const outputPath = args.output || DEFAULT_OUTPUT_PATH;
|
||||
|
||||
const events = await loadAndValidateEvents({ eventsDir, schemaPath });
|
||||
const first = renderMarkdown(events);
|
||||
const second = renderMarkdown(events);
|
||||
|
||||
if (first !== second) {
|
||||
throw new ExitError(4, 'determinism check failed: repeated render produced different output');
|
||||
}
|
||||
|
||||
const absoluteOutput = path.resolve(process.cwd(), outputPath);
|
||||
let existing = '';
|
||||
try {
|
||||
existing = await fs.readFile(absoluteOutput, 'utf8');
|
||||
} catch (error) {
|
||||
if (error.code !== 'ENOENT') {
|
||||
throw new ExitError(4, `unable to read output file ${outputPath}: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (existing && !existing.startsWith(AUTO_HEADER)) {
|
||||
throw new ExitError(4, `${outputPath} missing generated-file header`);
|
||||
}
|
||||
|
||||
if (existing !== first) {
|
||||
throw new ExitError(4, `${outputPath} is out of date; run "torch-lock memory build"`);
|
||||
}
|
||||
|
||||
console.log(JSON.stringify({
|
||||
ok: true,
|
||||
verified_events: events.length,
|
||||
output: outputPath,
|
||||
}, null, 2));
|
||||
}
|
||||
+6
-1
@@ -49,6 +49,7 @@ Commands:
|
||||
init [--force] Initialize torch/ directory in current project
|
||||
update [--force] Update torch/ configuration (backups, merges)
|
||||
remove [--force] Remove all TORCH files and configuration from project
|
||||
memory <add|build|verify> Append/build/verify append-only memory events
|
||||
|
||||
list-memories [--agent <id>] [--type <type>] [--tags <a,b>] [--pinned <true|false>] [--full]
|
||||
inspect-memory --id <memoryId>
|
||||
@@ -65,6 +66,9 @@ Options:
|
||||
--json Emit compact single-line JSON
|
||||
--json-file Write JSON output to a file path
|
||||
--quiet Suppress stderr progress logs (pairs well with --json)
|
||||
--events-dir Memory events directory (default: memory/events)
|
||||
--schema-path Memory event schema path (default: memory/schema/memory-event.schema.json)
|
||||
--output Output path for generated memory markdown (default: memory_update.md)
|
||||
|
||||
Environment:
|
||||
NOSTR_LOCK_NAMESPACE Namespace prefix for lock tags (default: torch)
|
||||
@@ -80,4 +84,5 @@ Exit codes:
|
||||
0 Success
|
||||
1 Usage error
|
||||
2 Relay/network error
|
||||
3 Lock denied (already locked or race lost)`;
|
||||
3 Lock denied (already locked or race lost)
|
||||
4 Memory contract validation failed`;
|
||||
+22
@@ -45,6 +45,7 @@ import { cmdCheck } from './cmd-check.mjs';
|
||||
import { cmdList } from './cmd-list.mjs';
|
||||
import { cmdComplete } from './cmd-complete.mjs';
|
||||
import { cmdDoctor } from './cmd-doctor.mjs';
|
||||
import { cmdMemoryAdd, cmdMemoryBuild, cmdMemoryVerify } from './cmd-memory.mjs';
|
||||
|
||||
useWebSocketImplementation(WebSocket);
|
||||
|
||||
@@ -402,6 +403,27 @@ const COMMAND_HANDLERS = {
|
||||
await cmdBackup({ output: args.output });
|
||||
}
|
||||
},
|
||||
memory: async (args) => {
|
||||
if (!args.subcommand) {
|
||||
console.error('ERROR: Missing subcommand for memory (add, build, verify)');
|
||||
throw new ExitError(1, 'Missing subcommand');
|
||||
}
|
||||
if (args.subcommand === 'add') {
|
||||
await cmdMemoryAdd(args);
|
||||
return;
|
||||
}
|
||||
if (args.subcommand === 'build') {
|
||||
await cmdMemoryBuild(args);
|
||||
return;
|
||||
}
|
||||
if (args.subcommand === 'verify') {
|
||||
await cmdMemoryVerify(args);
|
||||
return;
|
||||
}
|
||||
|
||||
console.error(`ERROR: Unknown memory subcommand: ${args.subcommand}`);
|
||||
throw new ExitError(1, 'Unknown memory subcommand');
|
||||
},
|
||||
};
|
||||
|
||||
function usage() {
|
||||
|
||||
@@ -78,3 +78,31 @@ test('parseArgs should parse memory flags with equals', () => {
|
||||
assert.strictEqual(args.limit, 10);
|
||||
assert.strictEqual(args.offset, 5);
|
||||
});
|
||||
|
||||
test('parseArgs should parse memory subcommands and event contract flags', () => {
|
||||
const args = parseArgs([
|
||||
'memory',
|
||||
'add',
|
||||
'--agent', 'codex',
|
||||
'--topic', 'memory',
|
||||
'--message', 'hello',
|
||||
'--run-id', 'run_1',
|
||||
'--events-dir', 'memory/events',
|
||||
'--schema-path', 'memory/schema/memory-event.schema.json',
|
||||
'--created-at', '2026-03-01T15:20:11Z',
|
||||
'--source', 'scheduler',
|
||||
'--project', 'TORCH',
|
||||
]);
|
||||
|
||||
assert.strictEqual(args.command, 'memory');
|
||||
assert.strictEqual(args.subcommand, 'add');
|
||||
assert.strictEqual(args.agent, 'codex');
|
||||
assert.strictEqual(args.topic, 'memory');
|
||||
assert.strictEqual(args.message, 'hello');
|
||||
assert.strictEqual(args.runId, 'run_1');
|
||||
assert.strictEqual(args.eventsDir, 'memory/events');
|
||||
assert.strictEqual(args.schemaPath, 'memory/schema/memory-event.schema.json');
|
||||
assert.strictEqual(args.createdAt, '2026-03-01T15:20:11Z');
|
||||
assert.strictEqual(args.source, 'scheduler');
|
||||
assert.strictEqual(args.project, 'TORCH');
|
||||
});
|
||||
@@ -0,0 +1,135 @@
|
||||
import { test } from 'node:test';
|
||||
import assert from 'node:assert';
|
||||
import fs from 'node:fs/promises';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { cmdMemoryAdd, cmdMemoryBuild, cmdMemoryVerify } from '../src/cmd-memory.mjs';
|
||||
import { ExitError } from '../src/errors.mjs';
|
||||
|
||||
const SCHEMA_PATH = 'memory/schema/memory-event.schema.json';
|
||||
|
||||
async function withTempProject(fn) {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'torch-memory-cmd-'));
|
||||
const cwd = process.cwd();
|
||||
process.chdir(root);
|
||||
try {
|
||||
await fs.mkdir('memory/events', { recursive: true });
|
||||
await fs.mkdir('memory/schema', { recursive: true });
|
||||
const schemaSource = path.resolve(cwd, SCHEMA_PATH);
|
||||
await fs.copyFile(schemaSource, path.join(root, SCHEMA_PATH));
|
||||
await fn(root);
|
||||
} finally {
|
||||
process.chdir(cwd);
|
||||
}
|
||||
}
|
||||
|
||||
async function writeEvent(root, event) {
|
||||
const compact = event.created_at.replace(/[-:]/g, '').replace('.000', '');
|
||||
const year = event.created_at.slice(0, 4);
|
||||
const month = event.created_at.slice(5, 7);
|
||||
const day = event.created_at.slice(8, 10);
|
||||
const dir = path.join(root, 'memory', 'events', year, month, day);
|
||||
await fs.mkdir(dir, { recursive: true });
|
||||
const filePath = path.join(dir, `${compact}_${event.id}.json`);
|
||||
await fs.writeFile(filePath, `${JSON.stringify(event, null, 2)}\n`, 'utf8');
|
||||
}
|
||||
|
||||
test('memory add writes one append-only event file with schema-valid shape', async () => {
|
||||
await withTempProject(async () => {
|
||||
const createdAt = '2026-03-01T15:20:11Z';
|
||||
await cmdMemoryAdd({
|
||||
subcommand: 'add',
|
||||
agent: 'codex',
|
||||
topic: 'memory',
|
||||
message: 'Append-only event created.',
|
||||
createdAt,
|
||||
runId: 'run_abc123',
|
||||
tags: ['architecture', 'memory'],
|
||||
source: 'test',
|
||||
});
|
||||
|
||||
const eventDir = path.resolve('memory/events/2026/03/01');
|
||||
const files = await fs.readdir(eventDir);
|
||||
assert.strictEqual(files.length, 1);
|
||||
const json = JSON.parse(await fs.readFile(path.join(eventDir, files[0]), 'utf8'));
|
||||
assert.strictEqual(json.project, 'TORCH');
|
||||
assert.strictEqual(json.created_at, createdAt);
|
||||
assert.strictEqual(json.agent, 'codex');
|
||||
assert.ok(json.id.includes('-7'));
|
||||
});
|
||||
});
|
||||
|
||||
test('memory build renders deterministic markdown sorted by created_at then id', async () => {
|
||||
await withTempProject(async (root) => {
|
||||
await writeEvent(root, {
|
||||
id: '0195505d-0000-7000-8000-000000000002',
|
||||
created_at: '2026-03-01T15:20:11Z',
|
||||
project: 'TORCH',
|
||||
agent: 'codex',
|
||||
topic: 'memory',
|
||||
content: 'second',
|
||||
});
|
||||
await writeEvent(root, {
|
||||
id: '0195505d-0000-7000-8000-000000000001',
|
||||
created_at: '2026-03-01T15:20:11Z',
|
||||
project: 'TORCH',
|
||||
agent: 'codex',
|
||||
topic: 'memory',
|
||||
content: 'first',
|
||||
});
|
||||
|
||||
await cmdMemoryBuild({ subcommand: 'build' });
|
||||
const rendered = await fs.readFile('memory_update.md', 'utf8');
|
||||
const firstPos = rendered.indexOf('000000000001');
|
||||
const secondPos = rendered.indexOf('000000000002');
|
||||
assert.ok(firstPos > 0);
|
||||
assert.ok(secondPos > firstPos);
|
||||
assert.ok(rendered.startsWith('<!-- AUTO-GENERATED: DO NOT EDIT. Source: memory/events -->'));
|
||||
});
|
||||
});
|
||||
|
||||
test('memory verify fails when generated output is stale', async () => {
|
||||
await withTempProject(async (root) => {
|
||||
await writeEvent(root, {
|
||||
id: '0195505d-0000-7000-8000-000000000001',
|
||||
created_at: '2026-03-01T15:20:11Z',
|
||||
project: 'TORCH',
|
||||
agent: 'codex',
|
||||
topic: 'memory',
|
||||
content: 'expected output',
|
||||
});
|
||||
await fs.writeFile('memory_update.md', 'manually edited\n', 'utf8');
|
||||
|
||||
await assert.rejects(
|
||||
async () => cmdMemoryVerify({ subcommand: 'verify' }),
|
||||
(error) => error instanceof ExitError && error.code === 4 && /out of date|missing generated-file header/i.test(error.message),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
test('memory verify fails on duplicate event ids', async () => {
|
||||
await withTempProject(async (root) => {
|
||||
const sharedId = '0195505d-0000-7000-8000-000000000001';
|
||||
await writeEvent(root, {
|
||||
id: sharedId,
|
||||
created_at: '2026-03-01T15:20:11Z',
|
||||
project: 'TORCH',
|
||||
agent: 'codex',
|
||||
topic: 'memory',
|
||||
content: 'first',
|
||||
});
|
||||
await writeEvent(root, {
|
||||
id: sharedId,
|
||||
created_at: '2026-03-01T15:20:12Z',
|
||||
project: 'TORCH',
|
||||
agent: 'codex',
|
||||
topic: 'memory',
|
||||
content: 'second',
|
||||
});
|
||||
|
||||
await assert.rejects(
|
||||
async () => cmdMemoryVerify({ subcommand: 'verify' }),
|
||||
(error) => error instanceof ExitError && error.code === 4 && /duplicate id/i.test(error.message),
|
||||
);
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user