runtime: recover stopped legacy sessions and add bounded continuation

This commit is contained in:
thePR0M3TH3AN committed 2026-10-02 23:49:59 -04:00
1 parent 25a3c7c1b5
commit 11dabd2450
11 files changed
+662 -1

No files matched your search

+20
View File
@@ -2,6 +2,26 @@
Repository state outranks conversation memory. Query ownership before crossing a domain boundary.
## Owner-requested restart and continuation repair (2026-10-03 UTC)
The owner said "yeah fix the issue" after the arbiter verified that the fleet
had stopped making progress. The arbiter implemented a schema-compatible,
owner-only manual legacy recovery and a bounded deterministic continuation pilot.
These are source changes pending independent QA and installed-release acceptance,
not completed umbrella tasks. Read
docs/spec_changes/2026-10-03-legacy-runtime-recovery.md. The arbiter may apply exact
tested stopped-invocation recovery and resume the existing seven identities with
the source CLI. Preserve old unknown/failure receipts and existing unique commits.
The source controller may launch only the explicitly approved existing roster,
under its daily turn cap, event deduplication and per-identity physical turn guard.
It does not grant agents permission to start providers or bypass budgets, resource
leases, approval boundaries or task ownership. Existing legacy manager-wake
configuration stays disabled; this source pilot is a distinct, bounded mechanism.
Manager should process current inbox and route real next work rather than leaving
it at a proposal-only loop. QA may independently review the exact new source and
scenario packet now; no fabricated QA acceptance is implied by owner implementation.
## Owner priority update: reliability before visual polish (2026-10-03 UTC)
The owner requested that TORCH's operating defects be fixed first so the fleet
+57
View File
@@ -2811,3 +2811,60 @@ test_integrity_note:
did_relax_any_assertion: false
if_true_explain_spec_basis: ""
```
# Owner-requested legacy recovery and bounded continuation (2026-10-03)
Implementation and behavioral basis:
[legacy runtime recovery](spec_changes/2026-10-03-legacy-runtime-recovery.md).
Existing assertions are unchanged; the new scenarios are in
`test/fleet/legacy-runtime-recovery.test.mjs`.
```yaml
test_integrity_note:
change_type: new_tests
scenarios:
- id: SCN-legacy-runtime-recovery
given: "A stopped exact invocation and retained unknown identity"
when: "The owner recovers presence"
then: "One owner audit, offline presence and preserved task/session/unknown"
- id: SCN-legacy-runtime-recovery-refusal
given: "Live or mismatched invocation/process evidence"
when: "Proof is requested"
then: "Recovery refuses without mutation"
- id: SCN-legacy-runtime-recovery-cas
given: "Stale, expired, forged or newly live evidence"
when: "Recovery applies"
then: "Transaction refuses without success audit"
- id: SCN-legacy-runtime-recovery-authority
given: "A specialist or incomplete expected snapshot"
when: "Owner recovery is attempted"
then: "Authority/snapshot validation refuses"
- id: SCN-runtime-turn-guard
given: "A physically live executor reporting idle"
when: "A second turn tries to start"
then: "The executor mutex refuses overlap until terminal release"
- id: SCN-runtime-continuation
given: "Owner-enabled roster and a new durable signal"
when: "Turns repeat, hit a cap or are paused"
then: "Only new eligible signals launch within the owner daily cap"
- id: SCN-runtime-continuation-pause
given: "Two eligible turns"
when: "The owner pauses during the first"
then: "The second does not start and pause is retained"
- id: SCN-legacy-runtime-recovery-project
given: "Cloned project/session metadata in distinct checkouts"
when: "Proof crosses checkout boundaries"
then: "Recovery refuses and leaves the other identity untouched"
observable_outcomes:
- "Persisted identity, task/session, owner audit, launch count and refusal codes"
determinism_controls:
- "Hermetic repositories, fixed clock and OS-boundary fixtures; no live provider"
anti_cheat_rationale:
prevents:
- "Boolean stopped proof or stop-as-PASS"
- "Stale/cross-project mutations and peer impersonation"
- "Duplicate physical turns or unchanged-signal busy loops"
- "Losing pause or refunding interrupted attempts"
relaxation:
did_relax_any_assertion: false
if_true_explain_spec_basis: ""
```
+8
View File
@@ -22,6 +22,14 @@ states, dependencies and evidence. Blocked remains blocked; automatic wakes stay
disabled until qualified and explicitly activated. Older pause/install/status
statements below are historical snapshots, not current runtime evidence.
Owner-requested recovery/continuation implementation is described in
[the source pilot specification](spec_changes/2026-10-03-legacy-runtime-recovery.md).
Do not close the runtime-recovery, persistent-operations or event-wake umbrellas
from focused tests alone. Release/Fleet Operations still need installer-owned
permanent timer provisioning/removal, installed artifact qualification and restart
recovery of retained crashed turn/controller guards. The transient host pilot does
not survive reboot, and its daily turn cap is not a token/dollar usage governor.
## Priority UI cycle: compact cards and agent Chat
Owner-approved [Console UI spec](CONSOLE_CHAT_AND_COMPACT_CARDS_SPEC.md).
@@ -0,0 +1,93 @@
# Manual owner recovery for retained unknown executor outcomes
The owner requested fixing the stalled fleet. Legacy runtime records lacked a
recovery API: three physically stopped executors remained `working/unknown` and
correctly refused duplicate launches. This change adds an owner-only, previewed
`recover-runtime` operation without schema changes or automatic recovery.
The owner explicitly selects a project-scoped systemd service invocation. Its
working directory and exact area argument must match this installation; its
terminal timestamp must match the retained identity update within five seconds
(systemd displays timestamps with one-second precision). Both service PIDs must
be zero, its cgroup absent, and no visible same-user native process may reference
the session or registered workspace. Physical state is checked again at apply.
Unreadable process visibility, an active service, foreign invocation or changed
identity refuses recovery. This is a **manual legacy operator binding**, not
cryptographic session provenance or protection against arbitrary same-user code.
Evidence is process-local, branded, single-use and expires after sixty seconds.
The service transaction compares the exact state/runtime/session/update snapshot,
sets presence to offline and writes an owner-attributed audit. Session, current
task and the unknown outcome are preserved. No task, test or launch is completed;
no process is killed, lease removed, schema migrated or provider started.
Preview and apply use the same implementation. Apply additionally requires the
session and update timestamp from the preview. Hosts/providers without verifiable
Linux/systemd evidence refuse this legacy recovery; no Boolean fallback exists.
## Bounded continuation pilot
`torch continuation enable --max-turns-per-day 12 --yes` approves only the
currently activated identity roster. `status` previews eligible identities;
`tick --limit 3 --yes` runs at most three normal guarded `up` operations. New
unacknowledged messages or assigned/in-progress task revision changes supply the
signal; an unchanged signal is not retried. Active/unknown identities, held turn
guards, detach, disabled policy and the daily UTC attempt cap prevent launch.
Attempts are charged before invocation, including failed/interrupted attempts.
The cap limits turns, **not dollars or token consumption**. Unknown quota remains
unknown. This controller makes no product/approval/assignment decisions.
`torch continuation pause --yes` prevents subsequent launches. Policy and attempt
ledger are separate atomic files so an in-flight turn cannot overwrite a pause.
Pause does not kill an already-running turn. A controller lock serializes ticks;
a crashed lock is retained rather than expired automatically. Each executor has
a separate physical turn guard held through classification and final presence,
so an agent reporting idle cannot permit overlapping native turns.
The first host pilot uses an explicitly installed transient user-systemd timer.
It survives this chat ending, but not a host reboot. Permanent installer-owned
provisioning/removal and installed artifact qualification remain existing backlog
work; this is not a claim that the alpha.3 installed package has been upgraded.
```yaml
test_integrity_note:
change_type: new_tests
scenarios:
- id: SCN-legacy-runtime-recovery
given: "An exact retained working/unknown identity and stopped project invocation"
when: "The owner previews and applies recovery"
then: "Presence becomes offline once; session, task and unknown outcome persist with one owner audit"
- id: SCN-legacy-runtime-recovery-refusal
given: "Live, foreign, mismatched or incomplete stopped-executor evidence"
when: "Recovery evidence is requested"
then: "Recovery refuses without identity mutation"
- id: SCN-legacy-runtime-recovery-cas
given: "Issued proof followed by a changed identity, live process, expiration or forgery"
when: "Recovery is applied"
then: "Transactional recovery refuses and creates no success audit"
- id: SCN-legacy-runtime-recovery-authority
given: "A specialist actor or incomplete expected snapshot"
when: "Owner recovery is requested"
then: "Authority or snapshot validation refuses"
- id: SCN-runtime-turn-guard
given: "A physical executor holds an identity guard and reports idle early"
when: "Another executor attempts the same identity"
then: "It refuses until the first executor finishes and releases its own guard"
- id: SCN-runtime-continuation
given: "An explicitly enabled roster, a new durable signal and a fixed daily turn cap"
when: "Continuation ticks, repeats a signal, exhausts its cap or is paused"
then: "Only eligible new signals launch; deduplication, cap, owner authority and pause remain enforced"
observable_outcomes:
- "Durable presence, session and task state; owner-attributed audit; explicit refusal codes"
determinism_controls:
- "Hermetic Git/install fixtures, fixed clock and injected OS observation boundary"
anti_cheat_rationale:
prevents:
- "Caller Boolean as stopped proof"
- "Process stop promoted to successful task or test"
- "Stale snapshot overwrites or silent peer impersonation"
- "Expired, foreign or replayed proof"
relaxation:
did_relax_any_assertion: false
if_true_explain_spec_basis: ""
```
+49
View File
@@ -70,6 +70,8 @@ import {
} from './design/architect.mjs';
import { FleetEvolutionService } from './evolution/service.mjs';
import { HierarchyEvolutionService } from './evolution/hierarchy.mjs';
import { issueStoppedExecutorEvidence } from './runtime/stopped-executor-evidence.mjs';
import { configureContinuation, planContinuation, runContinuation } from './runtime/continuation.mjs';
import { ConvergenceService } from './convergence/service.mjs';
import {
attachForge, detachForge, forgeStatus, planForgeAttach, planForgeDetach,
@@ -95,6 +97,8 @@ Usage:
torch install --restore [--dry-run] [--yes] [--json]
torch setup [--parent <path>] [--dry-run] [--yes] [--json]
torch up [--fresh] [--only <id,id>] [--dry-run] [--yes] [--json]
torch recover-runtime --area <id> --unit <service> --invocation <id> [--expected-session <id> --expected-updated-at <time> --yes] [--json]
torch continuation <status|enable|pause|tick> [--max-turns-per-day <1-48>] [--limit <1-3>] [--yes] [--json]
torch down [--dry-run] [--yes] [--json]
torch capture [--json]
torch detach [--dry-run] [--yes] [--json]
@@ -576,6 +580,51 @@ export async function runCli(argv = process.argv.slice(2), {
process.stdout.write(HELP);
return 0;
}
if (command === 'continuation') {
const control = openControlPlane({ repositoryRoot: optionValue(argv, '--repo') ?? cwd, env });
try {
const operation = argv[1] ?? 'status';
const backlog = new BacklogService({ repositoryRoot: control.repositoryRoot, controlPlane: control });
if (operation === 'status' || !argv.includes('--yes') || argv.includes('--dry-run')) {
print(planContinuation(control, backlog), { json }); return 0;
}
if (operation === 'enable' || operation === 'pause') {
print(configureContinuation(control, { actorId: 'owner', enabled: operation === 'enable',
maxTurnsPerDay: Number(optionValue(argv, '--max-turns-per-day') ?? 12) }), { json }); return 0;
}
if (operation !== 'tick') throw new TorchError('Unknown continuation operation.', { code: 'UNKNOWN_COMMAND' });
print(await runContinuation(control, backlog, { actorId: 'owner', limit: Number(optionValue(argv, '--limit') ?? 3),
launch: areaId => {
const result = spawn(process.execPath, [fileURLToPath(new URL('../bin/torch.mjs', import.meta.url)),
'up', '--only', areaId, '--yes', '--json'], { cwd: control.repositoryRoot, env,
encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], maxBuffer: 256 * 1024 });
return result.error || result.signal ? null : result.status;
} }), { json });
return 0;
} finally { control.close(); }
}
if (command === 'recover-runtime') {
const control = openControlPlane({ repositoryRoot: optionValue(argv, '--repo') ?? cwd, env });
try {
const areaId = optionValue(argv, '--area');
const issued = issueStoppedExecutorEvidence({ controlPlane: control, actorId: 'owner', areaId,
unit: optionValue(argv, '--unit'), invocationId: optionValue(argv, '--invocation') });
if (!argv.includes('--yes') || argv.includes('--dry-run')) {
print({ ...issued.plan, requiresApproval: true }, { json });
return 0;
}
const expected = issued.plan.expectedIdentity;
if (optionValue(argv, '--expected-session') !== expected.runtimeSessionId
|| optionValue(argv, '--expected-updated-at') !== expected.updatedAt) {
throw new TorchError('Apply requires the exact session and updated timestamp from the recovery preview.', {
code: 'RUNTIME_RECOVERY_SNAPSHOT_CONFLICT',
});
}
print(control.recoverOwnerStoppedUnknownIdentity({ actorId: 'owner', areaId,
expectedIdentity: expected, evidence: issued.evidence }), { json });
return 0;
} finally { control.close(); }
}
if (command === 'runtimes') {
const operation = argv[1] ?? 'list';
if (operation === 'update') {
+32 -1
View File
@@ -7,6 +7,7 @@ import { readInstallManifest } from '../kernel/install.mjs';
import { projectStatePath } from '../kernel/paths.mjs';
import { loadProjectConfig } from '../kernel/config.mjs';
import { organizationGraphFromConfig } from '../kernel/organization.mjs';
import { consumeStoppedExecutorEvidence } from '../runtime/stopped-executor-evidence.mjs';
export const PRESENCE_STATES = Object.freeze([
'starting', 'working', 'waiting', 'idle', 'stale', 'stopping', 'offline',
@@ -459,6 +460,36 @@ export class ControlPlane {
return { messageId: id, recipient: recipientId, acknowledgedAt };
}
recoverOwnerStoppedUnknownIdentity({ actorId, areaId, expectedIdentity, evidence } = {}) {
const owner = this.assertOwnerActor(actorId);
const id = this.assertIdentity(areaId);
if (!expectedIdentity || ['state', 'runtime', 'runtimeSessionId', 'updatedAt'].some(key => !expectedIdentity[key])) {
throw new TorchError('Recovery requires the complete previewed identity snapshot.', { code: 'RUNTIME_RECOVERY_SNAPSHOT_REQUIRED' });
}
this.database.exec('BEGIN IMMEDIATE');
try {
const current = this.identity(id);
if (current.state !== 'working' || !current.summary?.includes('Runtime executor completion is unknown')
|| ['state', 'runtime', 'runtimeSessionId', 'updatedAt'].some(key => current[key] !== expectedIdentity[key])) {
throw new TorchError('Identity changed since recovery was reviewed.', { code: 'RUNTIME_RECOVERY_SNAPSHOT_CONFLICT' });
}
const facts = consumeStoppedExecutorEvidence(evidence, { projectId: this.projectId, repositoryRoot: this.repositoryRoot, identity: current });
const now = this.clock().toISOString();
const result = this.database.prepare(`UPDATE identities SET state = 'offline', summary = ?, heartbeat_at = ?, updated_at = ?
WHERE area_id = ? AND state = ? AND runtime = ? AND runtime_session_id = ? AND updated_at = ?`)
.run('Owner verified stopped legacy executor; prior outcome remains unknown.', now, now,
id, current.state, current.runtime, current.runtimeSessionId, current.updatedAt);
if (result.changes !== 1) throw new TorchError('Recovery lost its identity snapshot.', { code: 'RUNTIME_RECOVERY_SNAPSHOT_CONFLICT' });
const audit = this.auditOwnerAction({ actorId: owner, operation: 'runtime.legacy-unknown.recover-stopped',
entityType: 'identity', entityId: id, details: { previousIdentity: expectedIdentity, ...facts } });
this.database.exec('COMMIT');
return { identity: this.identity(id), auditId: audit.id, priorOutcome: 'unknown', sessionsStarted: false, mutationPerformed: true };
} catch (error) {
this.database.exec('ROLLBACK');
throw error;
}
}
reportStatus({ areaId, state, summary, runtime, runtimeSessionId, task } = {}) {
const id = this.assertIdentity(areaId);
const normalizedState = requiredText(state, 'state');
@@ -751,7 +782,7 @@ export class ControlPlane {
auditOwnerAction({ actorId, operation, entityType, entityId = null, details = null } = {}) {
const actor = this.assertOwnerActor(actorId);
if (!['artifact.feedback', 'backlog.priority', 'backlog.create', 'agent.request', 'approval.decide', 'approval.notify', 'forge.sync', 'canonical.fetch', 'profile.change', 'schedule.launcher.install', 'schedule.launcher.reconcile', 'schedule.wake.recover', 'hierarchy.conclude-pilot', 'check.recover-prepared'].includes(operation)) {
if (!['artifact.feedback', 'backlog.priority', 'backlog.create', 'agent.request', 'approval.decide', 'approval.notify', 'forge.sync', 'canonical.fetch', 'profile.change', 'schedule.launcher.install', 'schedule.launcher.reconcile', 'schedule.wake.recover', 'hierarchy.conclude-pilot', 'check.recover-prepared', 'runtime.legacy-unknown.recover-stopped', 'runtime.continuation.configure'].includes(operation)) {
throw new TorchError('This owner audit surface does not authorize the requested operation', {
code: 'OWNER_OPERATION_UNSUPPORTED', details: { operation },
});
+92
View File
@@ -0,0 +1,92 @@
import { createHash, randomUUID } from 'node:crypto';
import { existsSync, mkdirSync, readFileSync, renameSync, rmdirSync, writeFileSync } from 'node:fs';
import { join } from 'node:path';
import { TorchError } from '../kernel/errors.mjs';
function statePath(control) { return join(control.stateRoot, 'continuation.json'); }
function read(control) {
const policyPath = join(control.stateRoot, 'continuation-policy.json');
const state = { enabled: false, maxTurnsPerDay: 12, areas: [], day: null, attempts: 0, handled: {},
...(existsSync(statePath(control)) ? JSON.parse(readFileSync(statePath(control), 'utf8')) : {}),
...(existsSync(policyPath) ? JSON.parse(readFileSync(policyPath, 'utf8')) : {}) };
if (typeof state.enabled !== 'boolean' || !Number.isInteger(state.maxTurnsPerDay)
|| state.maxTurnsPerDay < 1 || state.maxTurnsPerDay > 48 || !Array.isArray(state.areas)
|| !Number.isInteger(state.attempts) || state.attempts < 0 || !state.handled || typeof state.handled !== 'object') {
throw new TorchError('Continuation policy or attempt ledger is invalid; launch refused.', { code: 'CONTINUATION_POLICY_INVALID' });
}
return state;
}
function save(control, state, policy = false) {
const path = policy ? join(control.stateRoot, 'continuation-policy.json') : statePath(control);
const temporary = `${path}.${randomUUID()}.tmp`;
const value = policy ? { enabled: state.enabled, maxTurnsPerDay: state.maxTurnsPerDay, areas: state.areas }
: { day: state.day, attempts: state.attempts, handled: state.handled };
writeFileSync(temporary, JSON.stringify(value), { flag: 'wx', mode: 0o600 });
renameSync(temporary, path);
}
export function configureContinuation(control, { actorId, enabled, maxTurnsPerDay = 12 } = {}) {
control.assertOwnerActor(actorId);
if (typeof enabled !== 'boolean' || !Number.isInteger(maxTurnsPerDay) || maxTurnsPerDay < 1 || maxTurnsPerDay > 48) {
throw new TorchError('Continuation requires an explicit enable/pause and 1–48 daily turns.', { code: 'CONTINUATION_POLICY_INVALID' });
}
const state = { ...read(control), enabled, maxTurnsPerDay, areas: control.listAgents().map(a => a.areaId) };
save(control, state, true);
control.auditOwnerAction({ actorId, operation: 'runtime.continuation.configure', entityType: 'project', entityId: control.projectId,
details: { enabled, maxTurnsPerDay, areas: state.areas } });
return { enabled, maxTurnsPerDay, areas: state.areas, sessionsStarted: false };
}
export function planContinuation(control, backlog, { at = new Date(), limit = 3 } = {}) {
const state = read(control);
const day = at.toISOString().slice(0, 10);
const attempts = state.day === day ? state.attempts : 0;
if (!Number.isInteger(limit) || limit < 1 || limit > 3) throw new TorchError('Continuation limit must be 1–3.', { code: 'CONTINUATION_POLICY_INVALID' });
const metadata = JSON.parse(readFileSync(join(control.stateRoot, 'project.json'), 'utf8'));
if (!state.enabled || metadata.detachedAt || attempts >= state.maxTurnsPerDay) {
return { enabled: state.enabled, reason: metadata.detachedAt ? 'detached' : !state.enabled ? 'paused' : 'daily-turn-cap', candidates: [], attempts };
}
const tasks = backlog.list();
const candidates = [];
for (const areaId of state.areas) {
const agent = control.getAgent(areaId);
if (!['idle', 'offline', 'waiting'].includes(agent.state)
|| existsSync(join(control.stateRoot, 'sessions', 'turn-guards', areaId))) continue;
const messages = control.readMessages({ recipient: areaId, unacknowledgedOnly: true, limit: 1000 });
const active = tasks.filter(t => t.owner === areaId && ['assigned', 'in_progress'].includes(t.state));
if (!messages.length && !active.length) continue;
const fingerprint = createHash('sha256').update(JSON.stringify({ messages: messages.map(m => m.id), tasks: active.map(t => [t.id, t.revision]) })).digest('hex');
if (state.handled[areaId] === fingerprint) continue;
candidates.push({ areaId, fingerprint, messageCount: messages.length, taskIds: active.map(t => t.id) });
}
candidates.sort((a, b) => Number(b.areaId === 'session-manager') - Number(a.areaId === 'session-manager'));
return { enabled: true, day, attempts, remainingTurns: state.maxTurnsPerDay - attempts,
candidates: candidates.slice(0, Math.min(limit, state.maxTurnsPerDay - attempts)), mutationPerformed: false };
}
export async function runContinuation(control, backlog, { actorId, launch, at, clock = () => at ?? new Date(), limit = 3 } = {}) {
control.assertOwnerActor(actorId);
if (typeof launch !== 'function') throw new TorchError('Continuation needs an authorized launcher.', { code: 'RUNTIME_EXECUTION_NOT_AUTHORIZED' });
const lock = join(control.stateRoot, 'continuation.lock');
try { mkdirSync(lock); } catch (error) {
if (error.code === 'EEXIST') return { reason: 'controller-active-or-unreconciled', launched: [] };
throw error;
}
try {
const plan = planContinuation(control, backlog, { at: clock(), limit });
const launched = [];
for (const candidate of plan.candidates) {
// Fresh plan and policy check before each launch; a concurrent pause wins.
const current = planContinuation(control, backlog, { at: clock(), limit: 3 });
if (!current.candidates.some(c => c.areaId === candidate.areaId && c.fingerprint === candidate.fingerprint)) continue;
const state = read(control);
if (state.day !== current.day) { state.day = current.day; state.attempts = 0; }
state.attempts++;
state.handled[candidate.areaId] = candidate.fingerprint;
save(control, state); // Charge/reserve before launch; interrupted attempts are never refunded/retried silently.
const status = await launch(candidate.areaId);
launched.push({ areaId: candidate.areaId, status });
}
return { launched, reason: plan.reason ?? null };
} finally { rmdirSync(lock); }
}
+3
View File
@@ -11,6 +11,7 @@ import {
costCeilingPlanIssues, profileCapabilityIssues, resolveLaunchPolicy, validateRuntimeAdapter,
} from '../adapters/runtime.mjs';
import { hierarchyOrder } from './hierarchy-order.mjs';
import { withRuntimeTurnGuard } from './turn-guard.mjs';
const MCP_ENTRY = fileURLToPath(new URL('../mcp/server.mjs', import.meta.url));
@@ -570,6 +571,7 @@ export function startFleet({ plan, controlPlane, executor, adapters } = {}) {
const runtimes = adapterMap({ adapters });
for (const action of plan.actions) {
controlPlane.assertIdentity(action.areaId);
withRuntimeTurnGuard(controlPlane, action.areaId, () => {
writeCombinedPrompt(action);
const result = executor({
...action.launch, areaId: action.areaId, runtime: action.runtime,
@@ -634,6 +636,7 @@ export function startFleet({ plan, controlPlane, executor, adapters } = {}) {
? `${action.mode} turn completed` : `${action.mode} requested`,
});
started.push({ areaId: action.areaId, runtimeSessionId: identity.runtimeSessionId, mode: action.mode });
});
}
if (plan.stateRoot) {
const metadata = projectMetadata(plan.stateRoot);
+106
View File
@@ -0,0 +1,106 @@
import { spawnSync } from 'node:child_process';
import { createHash } from 'node:crypto';
import { readFileSync, readdirSync, readlinkSync, statSync } from 'node:fs';
import { basename, join, resolve } from 'node:path';
import { TorchError } from '../kernel/errors.mjs';
const issued = new WeakMap();
const TTL_MS = 60_000;
function refuse(message) {
throw new TorchError(message, { code: 'RUNTIME_RECOVERY_UNVERIFIED' });
}
export function readStoppedUnit(unit) {
const result = spawnSync('systemctl', ['--user', 'show', unit,
'--property=Id,ActiveState,SubState,MainPID,ControlPID,InvocationID,ExecStart,WorkingDirectory,ControlGroup,ExecMainExitTimestamp'],
{ encoding: 'utf8', timeout: 5000, maxBuffer: 64 * 1024 });
if (result.error || result.signal || result.status !== 0) refuse('Cannot verify the owner-selected service invocation.');
return Object.fromEntries(result.stdout.trim().split('\n').map(line => {
const index = line.indexOf('=');
return [line.slice(0, index), line.slice(index + 1)];
}));
}
export function readNativeProcesses() {
if (process.platform !== 'linux') refuse('Legacy process inspection is supported only on Linux.');
const processes = [];
for (const name of readdirSync('/proc').filter(value => /^\d+$/.test(value))) {
const directory = join('/proc', name);
try {
if (statSync(directory).uid !== process.getuid()) continue;
const argv = readFileSync(join(directory, 'cmdline'), 'utf8').split('\0').filter(Boolean);
if (!argv.length) continue;
const native = argv.some(value => /^(?:codex|claude|pi)(?:\.js)?$/.test(basename(value)));
if (native) processes.push({ pid: Number(name), argv, cwd: readlinkSync(join(directory, 'cwd')) });
} catch (error) {
if (error.code !== 'ENOENT' && error.code !== 'ESRCH') refuse('Native process visibility is incomplete; recovery refused.');
}
}
return processes;
}
function verify(record) {
const unit = record.readUnit(record.unit);
const prefix = `torch-${record.projectId}-`;
if (typeof record.unit !== 'string' || !record.unit.startsWith(prefix) || !/^[A-Za-z0-9_.-]+\.service$/.test(record.unit)
|| unit.Id !== record.unit || !/^[a-f0-9]{32}$/.test(unit.InvocationID ?? '')
|| unit.InvocationID !== record.invocationId
|| !['failed', 'inactive'].includes(unit.ActiveState)
|| !['failed', 'dead'].includes(unit.SubState)
|| unit.MainPID !== '0' || unit.ControlPID !== '0' || unit.ControlGroup !== ''
|| resolve(unit.WorkingDirectory || '/') !== record.repositoryRoot) {
refuse('The selected project service is not a verified stopped invocation.');
}
// Explicit legacy operator binding, not a claim of cryptographic session provenance.
// Exact area token and terminal timestamp must agree with the retained unknown row.
if (!(unit.ExecStart ?? '').split(/[\s;]+/).includes(record.identity.areaId)
|| !Number.isFinite(Date.parse(unit.ExecMainExitTimestamp))
|| Math.abs(Date.parse(unit.ExecMainExitTimestamp) - Date.parse(record.identity.updatedAt)) > 5000) {
refuse('Service area or terminal timestamp does not match the retained unknown identity.');
}
for (const processRecord of record.readProcesses()) {
if (processRecord.argv.includes(record.identity.runtimeSessionId)
|| processRecord.argv.includes(record.worktree)
|| resolve(processRecord.cwd) === record.worktree) {
refuse('A native process still references the selected session or workspace.');
}
}
return { unit: record.unit, invocationId: unit.InvocationID,
terminalAt: new Date(Date.parse(unit.ExecMainExitTimestamp)).toISOString(),
outcome: 'unknown', verification: 'owner-selected-legacy-invocation' };
}
export function issueStoppedExecutorEvidence({ controlPlane, actorId, areaId, unit, invocationId,
readUnit = readStoppedUnit, readProcesses = readNativeProcesses, now = () => Date.now() } = {}) {
controlPlane.assertOwnerActor(actorId);
const identity = controlPlane.getAgent(areaId);
if (identity.state !== 'working' || !identity.runtimeSessionId || !identity.updatedAt
|| !identity.summary?.includes('Runtime executor completion is unknown')) {
refuse('Only an exact retained working/unknown identity can use legacy recovery.');
}
const worktree = controlPlane.manifest.external?.find(entry => entry.type === 'worktree' && entry.area === areaId)?.path;
if (!worktree) refuse('No registered workspace exists for this identity.');
const record = { identity: Object.freeze({ areaId, state: identity.state, runtime: identity.runtime,
runtimeSessionId: identity.runtimeSessionId, updatedAt: identity.updatedAt }),
unit, invocationId, projectId: controlPlane.projectId, repositoryRoot: resolve(controlPlane.repositoryRoot),
worktree: resolve(worktree), readUnit, readProcesses, now, issuedAt: now() };
const facts = verify(record);
const evidence = Object.freeze({});
issued.set(evidence, record);
return { evidence, plan: { areaId, expectedIdentity: record.identity, ...facts,
mutationPerformed: false, effect: 'Recover presence to offline; preserve task, session and unknown outcome. No provider launch.' } };
}
export function consumeStoppedExecutorEvidence(evidence, { projectId, repositoryRoot, identity } = {}) {
const record = issued.get(evidence);
if (!record) refuse('Recovery requires fresh Runtime-issued evidence, not caller text or a Boolean.');
issued.delete(evidence);
if (record.projectId !== projectId || record.repositoryRoot !== resolve(repositoryRoot)
|| record.now() < record.issuedAt
|| record.now() - record.issuedAt > TTL_MS
|| ['areaId', 'state', 'runtime', 'runtimeSessionId', 'updatedAt'].some(key => record.identity[key] !== identity[key])) {
refuse('Recovery evidence is stale, expired or belongs to another identity.');
}
const facts = verify(record); // Recheck physical state immediately before the transactional mutation.
return { ...facts, digest: createHash('sha256').update(JSON.stringify({ identity: record.identity, ...facts })).digest('hex') };
}
+31
View File
@@ -0,0 +1,31 @@
import { randomUUID } from 'node:crypto';
import { mkdirSync, readFileSync, rmdirSync, unlinkSync, writeFileSync } from 'node:fs';
import { join } from 'node:path';
import { TorchError } from '../kernel/errors.mjs';
// One physical executor per identity, including the interval after an agent
// reports idle but before its harness actually exits. Crashed guards never expire.
export function withRuntimeTurnGuard(controlPlane, areaId, run) {
controlPlane.assertIdentity(areaId);
if (!/^[a-z0-9][a-z0-9-]*$/.test(areaId)) throw new TorchError('Invalid runtime guard identity.', { code: 'INVALID_RUNTIME_INPUT' });
const parent = join(controlPlane.stateRoot, 'sessions', 'turn-guards');
mkdirSync(parent, { recursive: true });
const directory = join(parent, areaId);
try { mkdirSync(directory); } catch (error) {
if (error.code === 'EEXIST') throw new TorchError('An executor already holds this identity turn guard.', {
code: 'RUNTIME_TURN_ACTIVE', details: { areaId },
});
throw error;
}
const path = join(directory, 'owner.json');
const nonce = randomUUID();
writeFileSync(path, JSON.stringify({ nonce, pid: process.pid, areaId, projectId: controlPlane.projectId }), { flag: 'wx', mode: 0o600 });
try { return run(); } finally {
// Delete only the exact guard created by this invocation; preserve foreign/changed guards.
const record = JSON.parse(readFileSync(path, 'utf8'));
if (record.nonce === nonce && record.pid === process.pid) {
unlinkSync(path);
rmdirSync(directory);
}
}
}
+171
View File
@@ -0,0 +1,171 @@
import assert from 'node:assert/strict';
import { execFileSync } from 'node:child_process';
import { mkdtempSync, readFileSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import test from 'node:test';
import { analyzeRepository } from '../../src/kernel/analyze.mjs';
import { inspectRepository } from '../../src/kernel/git.mjs';
import { proposeDomains } from '../../src/kernel/domains.mjs';
import { installProject } from '../../src/kernel/install.mjs';
import { openControlPlane } from '../../src/control-plane/service.mjs';
import { issueStoppedExecutorEvidence } from '../../src/runtime/stopped-executor-evidence.mjs';
import { withRuntimeTurnGuard } from '../../src/runtime/turn-guard.mjs';
import { configureContinuation, planContinuation, runContinuation } from '../../src/runtime/continuation.mjs';
function fixture() {
const root = mkdtempSync(join(tmpdir(), 'torch-legacy-recovery-'));
for (const args of [['init', '-b', 'main'], ['config', 'user.email', '[email protected]'],
['config', 'user.name', 'Recovery Fixture']]) execFileSync('git', args, { cwd: root, stdio: 'pipe' });
writeFileSync(join(root, 'app.js'), 'export const ready = true;\n');
execFileSync('git', ['add', '.'], { cwd: root });
execFileSync('git', ['commit', '-m', 'fixture'], { cwd: root, stdio: 'pipe' });
const repository = inspectRepository(root);
const proposal = proposeDomains({ repository, analysis: analyzeRepository(repository) });
proposal.review = { status: 'approved', reviewedAt: '2026-10-03T00:00:00Z', reviewedBy: 'owner', notes: [] };
const env = { ...process.env, XDG_DATA_HOME: join(root, '.data') };
installProject({ repository, proposal, env, projectId: 'recovery-fixture' });
const actualPath = join(root, '.torch', 'install-manifest.json');
const manifest = JSON.parse(readFileSync(actualPath, 'utf8'));
const areaId = proposal.domains[0].id;
manifest.external.push({ type: 'worktree', area: areaId, path: join(root, 'worker') });
writeFileSync(actualPath, JSON.stringify(manifest));
let time = Date.parse('2026-10-03T00:00:00Z');
const control = openControlPlane({ repositoryRoot: root, env, clock: () => new Date(time) });
control.reportStatus({ areaId, state: 'working', runtime: 'codex', runtimeSessionId: 'fixture-session',
task: 'TASK-preserved', summary: 'Runtime executor completion is unknown; duplicate launch is blocked pending reconciliation.' });
const identity = control.getAgent(areaId);
const unit = `torch-${control.projectId}-worker.service`;
const properties = { Id: unit, ActiveState: 'failed', SubState: 'failed', MainPID: '0', ControlPID: '0',
ControlGroup: '', InvocationID: 'a'.repeat(32), ExecStart: `node /tmp/probe.mjs ${areaId} ;`,
WorkingDirectory: root, ExecMainExitTimestamp: identity.updatedAt };
let processes = [];
const issue = () => issueStoppedExecutorEvidence({ controlPlane: control, actorId: 'owner', areaId, unit,
invocationId: 'a'.repeat(32), readUnit: () => ({ ...properties }), readProcesses: () => processes, now: () => time });
const apply = issued => control.recoverOwnerStoppedUnknownIdentity({ actorId: 'owner', areaId,
expectedIdentity: issued.plan.expectedIdentity, evidence: issued.evidence });
return { control, identity, areaId, properties, issue, apply,
advance: ms => { time += ms; }, processes: value => { processes = value; } };
}
test('SCN-legacy-runtime-recovery: owner recovery preserves task/session/unknown and records exactly one owner audit', () => {
const f = fixture();
try {
const issued = f.issue();
assert.equal(f.control.getAgent(f.areaId).state, 'working');
const result = f.apply(issued);
assert.equal(result.identity.state, 'offline');
assert.equal(result.identity.currentTask, 'TASK-preserved');
assert.equal(result.identity.runtimeSessionId, 'fixture-session');
assert.equal(result.priorOutcome, 'unknown');
assert.equal(result.sessionsStarted, false);
const audits = f.control.readAudit({ limit: 100 }).filter(x => x.operation === 'runtime.legacy-unknown.recover-stopped');
assert.equal(audits.length, 1);
assert.equal(audits[0].actorId, 'owner');
assert.equal(audits[0].details.outcome, 'unknown');
assert.throws(() => f.apply(issued), { code: 'RUNTIME_RECOVERY_SNAPSHOT_CONFLICT' });
} finally { f.control.close(); }
});
test('SCN-legacy-runtime-recovery-refusal: live, foreign invocation, area, time and process mismatches cannot issue proof', () => {
for (const patch of [{ MainPID: '99' }, { ControlPID: '99' }, { ActiveState: 'active' },
{ InvocationID: 'b'.repeat(32) }, { ExecStart: 'node other-area' },
{ ExecMainExitTimestamp: '2026-10-02T00:00:00Z' }, { ControlGroup: '/live' }]) {
const f = fixture();
try { Object.assign(f.properties, patch); assert.throws(f.issue, { code: 'RUNTIME_RECOVERY_UNVERIFIED' });
assert.equal(f.control.getAgent(f.areaId).state, 'working'); } finally { f.control.close(); }
}
const f = fixture();
try { f.processes([{ pid: 999, argv: ['codex', 'resume', 'fixture-session'], cwd: '/tmp' }]);
assert.throws(f.issue, { code: 'RUNTIME_RECOVERY_UNVERIFIED' }); } finally { f.control.close(); }
});
test('SCN-legacy-runtime-recovery-cas: changed snapshot, live recheck, expiry and caller-forged proof refuse without mutation', () => {
for (const scenario of ['changed', 'live', 'expired', 'forged']) {
const f = fixture();
try {
const issued = f.issue();
if (scenario === 'changed') { f.advance(1); f.control.reportStatus({ areaId: f.areaId, state: 'working', runtimeSessionId: 'different', summary: 'Runtime executor completion is unknown' }); }
if (scenario === 'live') f.properties.MainPID = '99';
if (scenario === 'expired') f.advance(60_001);
if (scenario === 'forged') issued.evidence = { stopped: true };
assert.throws(() => f.apply(issued));
assert.equal(f.control.getAgent(f.areaId).state, 'working');
assert.equal(f.control.readAudit({ limit: 100 }).filter(x => x.operation === 'runtime.legacy-unknown.recover-stopped').length, 0);
} finally { f.control.close(); }
}
});
test('SCN-legacy-runtime-recovery-authority: specialist and incomplete snapshot cannot use owner recovery', () => {
const f = fixture();
try {
const issued = f.issue();
assert.throws(() => f.control.recoverOwnerStoppedUnknownIdentity({ actorId: f.areaId, areaId: f.areaId,
expectedIdentity: issued.plan.expectedIdentity, evidence: issued.evidence }), { code: 'OWNER_AUTHORITY_REQUIRED' });
assert.throws(() => f.control.recoverOwnerStoppedUnknownIdentity({ actorId: 'owner', areaId: f.areaId,
expectedIdentity: {}, evidence: issued.evidence }), { code: 'RUNTIME_RECOVERY_SNAPSHOT_REQUIRED' });
assert.equal(f.control.getAgent(f.areaId).state, 'working');
} finally { f.control.close(); }
});
test('SCN-runtime-turn-guard: overlapping executors refuse even if an agent reports idle; terminal turns release their own guard', () => {
const f = fixture();
try {
let launches = 0;
withRuntimeTurnGuard(f.control, f.areaId, () => {
launches++;
f.control.reportStatus({ areaId: f.areaId, state: 'idle', summary: 'Agent reported idle before physical exit' });
assert.throws(() => withRuntimeTurnGuard(f.control, f.areaId, () => { launches++; }), { code: 'RUNTIME_TURN_ACTIVE' });
});
withRuntimeTurnGuard(f.control, f.areaId, () => { launches++; });
assert.equal(launches, 2);
} finally { f.control.close(); }
});
test('SCN-runtime-continuation: explicit enable, event deduplication, daily cap and pause govern authorized turns', async () => {
const f = fixture();
const backlog = { list: () => [] };
const at = new Date('2026-10-03T00:00:00Z');
try {
f.control.reportStatus({ areaId: f.areaId, state: 'idle', summary: 'Physical turn completed' });
f.control.sendOwnerRequest({ actorId: 'owner', recipient: f.areaId, body: 'Continue assigned work' });
assert.equal(planContinuation(f.control, backlog, { at }).reason, 'paused');
assert.throws(() => configureContinuation(f.control, { actorId: f.areaId, enabled: true }), { code: 'OWNER_AUTHORITY_REQUIRED' });
configureContinuation(f.control, { actorId: 'owner', enabled: true, maxTurnsPerDay: 1 });
assert.equal(planContinuation(f.control, backlog, { at }).candidates.length, 1);
let calls = 0;
const run = () => runContinuation(f.control, backlog, { actorId: 'owner', at, launch: async area => { assert.equal(area, f.areaId); calls++; return 0; } });
assert.equal((await run()).launched.length, 1);
assert.equal((await run()).launched.length, 0);
f.control.sendOwnerRequest({ actorId: 'owner', recipient: f.areaId, body: 'New work' });
assert.equal(planContinuation(f.control, backlog, { at }).reason, 'daily-turn-cap');
configureContinuation(f.control, { actorId: 'owner', enabled: true, maxTurnsPerDay: 2 });
assert.equal(planContinuation(f.control, backlog, { at }).candidates.length, 1);
configureContinuation(f.control, { actorId: 'owner', enabled: false, maxTurnsPerDay: 2 });
assert.equal((await run()).launched.length, 0);
assert.equal(calls, 1);
} finally { f.control.close(); }
});
test('SCN-runtime-continuation-pause: a pause during one turn prevents another queued turn', async () => {
const f = fixture();
try {
f.control.reportStatus({ areaId: f.areaId, state: 'idle', summary: 'Idle' });
for (const recipient of [f.areaId, 'session-manager']) f.control.sendOwnerRequest({ actorId: 'owner', recipient, body: 'Review current work' });
configureContinuation(f.control, { actorId: 'owner', enabled: true, maxTurnsPerDay: 3 });
const result = await runContinuation(f.control, { list: () => [] }, { actorId: 'owner',
launch: async () => { configureContinuation(f.control, { actorId: 'owner', enabled: false, maxTurnsPerDay: 3 }); return 0; } });
assert.equal(result.launched.length, 1);
assert.equal(planContinuation(f.control, { list: () => [] }).reason, 'paused');
} finally { f.control.close(); }
});
test('SCN-legacy-runtime-recovery-project: evidence cannot be applied to a different checkout with cloned project/session fields', () => {
const first = fixture();
const second = fixture();
try {
assert.throws(() => second.control.recoverOwnerStoppedUnknownIdentity({ actorId: 'owner', areaId: second.areaId,
expectedIdentity: second.identity, evidence: first.issue().evidence }), { code: 'RUNTIME_RECOVERY_UNVERIFIED' });
assert.equal(second.control.getAgent(second.areaId).state, 'working');
} finally { first.control.close(); second.control.close(); }
});