mirror of
https://github.com/PR0M3TH3AN/TORCH.git
synced 2026-10-27 07:51:52 +00:00
Merge commit '3b1bfe953ad593c5c6b14f85bb78ed79e28742db' into torch/provider-runtime
# Conflicts: # docs/TEST_INTEGRITY.md # test/fleet/mcp-adapter.test.mjs
This commit is contained in:
commit
bc2febae88
13 files changed
+791
-3
No files matched your search
@@ -2,6 +2,44 @@
|
||||
|
||||
Repository state outranks conversation memory. Query ownership before crossing a domain boundary.
|
||||
|
||||
## Owner-requested restart and continuation repair (2026-10-03 UTC)
|
||||
|
||||
The owner said "yeah fix the issue" after the arbiter verified that the fleet
|
||||
had stopped making progress. The arbiter implemented a schema-compatible,
|
||||
owner-only manual legacy recovery and a bounded deterministic continuation pilot.
|
||||
These are source changes pending independent QA and installed-release acceptance,
|
||||
not completed umbrella tasks. Read
|
||||
docs/spec_changes/2026-10-03-legacy-runtime-recovery.md. The arbiter may apply exact
|
||||
tested stopped-invocation recovery and resume the existing seven identities with
|
||||
the source CLI. Preserve old unknown/failure receipts and existing unique commits.
|
||||
|
||||
The source controller may launch only the explicitly approved existing roster,
|
||||
under its daily turn cap, event deduplication and per-identity physical turn guard.
|
||||
It does not grant agents permission to start providers or bypass budgets, resource
|
||||
leases, approval boundaries or task ownership. Existing legacy manager-wake
|
||||
configuration stays disabled; this source pilot is a distinct, bounded mechanism.
|
||||
Manager should process current inbox and route real next work rather than leaving
|
||||
it at a proposal-only loop. QA may independently review the exact new source and
|
||||
scenario packet now; no fabricated QA acceptance is implied by owner implementation.
|
||||
|
||||
## Owner priority update: reliability before visual polish (2026-10-03 UTC)
|
||||
|
||||
The owner requested that TORCH's operating defects be fixed first so the fleet
|
||||
benefits from improved operation. This supersedes the UI-first ordering below,
|
||||
not the existing resume authorization or safety boundaries. The authoritative
|
||||
backlog priorities were changed through the revision-checked owner API.
|
||||
|
||||
Sequence the smallest safe current-schema stopped-unknown recovery and CLI
|
||||
qualification first; then truthful lifecycle/status and their source-safety
|
||||
prerequisites; then qualified bounded continuation/event-driven wakes under
|
||||
pause, budget and authority policy; then actionable operational attention.
|
||||
Compact cards/detail modals and coordinator Chat remain high-priority work after
|
||||
these reliability gates. Do not require the entire future company architecture
|
||||
to deliver a safe minimal recovery. Reprioritization does not resolve blockers,
|
||||
grant test consent, enable automatic wakes or launch agents. Preserve task owners,
|
||||
dependencies and evidence; explicitly block/resequence existing assignments when
|
||||
needed rather than replacing them silently or creating concurrent assignments.
|
||||
|
||||
## Owner-approved fleet resume and priority polish cycle (2026-10-02)
|
||||
|
||||
The owner explicitly said: "We got a reset. lets get torch as pollished as we can.
|
||||
|
||||
@@ -2811,6 +2811,67 @@ test_integrity_note:
|
||||
did_relax_any_assertion: false
|
||||
if_true_explain_spec_basis: ""
|
||||
```
|
||||
# Owner-requested legacy recovery and bounded continuation (2026-10-03)
|
||||
|
||||
Implementation and behavioral basis:
|
||||
[legacy runtime recovery](spec_changes/2026-10-03-legacy-runtime-recovery.md).
|
||||
Existing assertions are unchanged; the new scenarios are in
|
||||
`test/fleet/legacy-runtime-recovery.test.mjs`.
|
||||
|
||||
```yaml
|
||||
test_integrity_note:
|
||||
change_type: new_tests
|
||||
scenarios:
|
||||
- id: SCN-legacy-runtime-recovery
|
||||
given: "A stopped exact invocation and retained unknown identity"
|
||||
when: "The owner recovers presence"
|
||||
then: "One owner audit, offline presence and preserved task/session/unknown"
|
||||
- id: SCN-legacy-runtime-recovery-refusal
|
||||
given: "Live or mismatched invocation/process evidence"
|
||||
when: "Proof is requested"
|
||||
then: "Recovery refuses without mutation"
|
||||
- id: SCN-legacy-runtime-recovery-cas
|
||||
given: "Stale, expired, forged or newly live evidence"
|
||||
when: "Recovery applies"
|
||||
then: "Transaction refuses without success audit"
|
||||
- id: SCN-legacy-runtime-recovery-authority
|
||||
given: "A specialist or incomplete expected snapshot"
|
||||
when: "Owner recovery is attempted"
|
||||
then: "Authority/snapshot validation refuses"
|
||||
- id: SCN-runtime-turn-guard
|
||||
given: "A physically live executor reporting idle"
|
||||
when: "A second turn tries to start"
|
||||
then: "The executor mutex refuses overlap until terminal release"
|
||||
- id: SCN-runtime-continuation
|
||||
given: "Owner-enabled roster and a new durable signal"
|
||||
when: "Turns repeat, hit a cap or are paused"
|
||||
then: "Only new eligible signals launch within the owner daily cap"
|
||||
- id: SCN-runtime-continuation-signals
|
||||
given: "Two eligible identities with multiple unread messages"
|
||||
when: "A bounded batch starts and only acknowledgements change afterward"
|
||||
then: "Both starts occur concurrently, a competing controller refuses, and acknowledgements alone cause no wake"
|
||||
- id: SCN-runtime-continuation-pause
|
||||
given: "Two eligible turns"
|
||||
when: "The owner pauses during the first"
|
||||
then: "The second does not start and pause is retained"
|
||||
- id: SCN-legacy-runtime-recovery-project
|
||||
given: "Cloned project/session metadata in distinct checkouts"
|
||||
when: "Proof crosses checkout boundaries"
|
||||
then: "Recovery refuses and leaves the other identity untouched"
|
||||
observable_outcomes:
|
||||
- "Persisted identity, task/session, owner audit, launch count and refusal codes"
|
||||
determinism_controls:
|
||||
- "Hermetic repositories, fixed clock and OS-boundary fixtures; no live provider"
|
||||
anti_cheat_rationale:
|
||||
prevents:
|
||||
- "Boolean stopped proof or stop-as-PASS"
|
||||
- "Stale/cross-project mutations and peer impersonation"
|
||||
- "Duplicate physical turns or unchanged-signal busy loops"
|
||||
- "Losing pause or refunding interrupted attempts"
|
||||
relaxation:
|
||||
did_relax_any_assertion: false
|
||||
if_true_explain_spec_basis: ""
|
||||
```
|
||||
|
||||
```yaml
|
||||
test_integrity_note:
|
||||
|
||||
@@ -1,5 +1,35 @@
|
||||
# TORCH remaining work
|
||||
|
||||
## Current owner priority: reliability first (2026-10-03 UTC)
|
||||
|
||||
This ordering supersedes the UI-first ordering below. The existing authoritative
|
||||
backlog, not this index, controls task state and assignment.
|
||||
|
||||
1. Safe current-schema runtime recovery (`TASK-company-runtime-recovery`) and
|
||||
combined CLI qualification (`TASK-pilot-cli-suite-timeout`).
|
||||
2. Truthful startup/session status (`TASK-pilot-starting-presence`) and its
|
||||
existing Kernel reservation/Work candidate-safety prerequisites.
|
||||
3. Persistent operations and bounded event-driven continuation
|
||||
(`TASK-conversation-persistent-operations`,
|
||||
`TASK-conversation-event-driven-wakes`), followed by actual scheduler
|
||||
qualification (`TASK-conversation-scheduler-qualification`).
|
||||
4. Specific actionable operational warnings (`TASK-console-attention-actions`).
|
||||
5. Compact cards/detail modals and coordinator-first Chat remain high priority.
|
||||
|
||||
The first four groups and incomplete technical prerequisites are urgent;
|
||||
scheduler qualification is high. These priority-only changes preserve owners,
|
||||
states, dependencies and evidence. Blocked remains blocked; automatic wakes stay
|
||||
disabled until qualified and explicitly activated. Older pause/install/status
|
||||
statements below are historical snapshots, not current runtime evidence.
|
||||
|
||||
Owner-requested recovery/continuation implementation is described in
|
||||
[the source pilot specification](spec_changes/2026-10-03-legacy-runtime-recovery.md).
|
||||
Do not close the runtime-recovery, persistent-operations or event-wake umbrellas
|
||||
from focused tests alone. Release/Fleet Operations still need installer-owned
|
||||
permanent timer provisioning/removal, installed artifact qualification and restart
|
||||
recovery of retained crashed turn/controller guards. The transient host pilot does
|
||||
not survive reboot, and its daily turn cap is not a token/dollar usage governor.
|
||||
|
||||
## Priority UI cycle: compact cards and agent Chat
|
||||
|
||||
Owner-approved [Console UI spec](CONSOLE_CHAT_AND_COMPACT_CARDS_SPEC.md).
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
# Manual owner recovery for retained unknown executor outcomes
|
||||
|
||||
The owner requested fixing the stalled fleet. Legacy runtime records lacked a
|
||||
recovery API: three physically stopped executors remained `working/unknown` and
|
||||
correctly refused duplicate launches. This change adds an owner-only, previewed
|
||||
`recover-runtime` operation without schema changes or automatic recovery.
|
||||
|
||||
The owner explicitly selects a project-scoped systemd service invocation. Its
|
||||
working directory and exact area argument must match this installation; its
|
||||
terminal timestamp must match the retained identity update within five seconds
|
||||
(systemd displays timestamps with one-second precision). Both service PIDs must
|
||||
be zero, its cgroup absent, and no visible same-user native process may reference
|
||||
the session or registered workspace. Physical state is checked again at apply.
|
||||
Unreadable process visibility, an active service, foreign invocation or changed
|
||||
identity refuses recovery. This is a **manual legacy operator binding**, not
|
||||
cryptographic session provenance or protection against arbitrary same-user code.
|
||||
|
||||
Evidence is process-local, branded, single-use and expires after sixty seconds.
|
||||
The service transaction compares the exact state/runtime/session/update snapshot,
|
||||
sets presence to offline and writes an owner-attributed audit. Session, current
|
||||
task and the unknown outcome are preserved. No task, test or launch is completed;
|
||||
no process is killed, lease removed, schema migrated or provider started.
|
||||
|
||||
Preview and apply use the same implementation. Apply additionally requires the
|
||||
session and update timestamp from the preview. Hosts/providers without verifiable
|
||||
Linux/systemd evidence refuse this legacy recovery; no Boolean fallback exists.
|
||||
|
||||
## Bounded continuation pilot
|
||||
|
||||
`torch continuation enable --max-turns-per-day 12 --yes` approves only the
|
||||
currently activated identity roster. `status` previews eligible identities;
|
||||
`tick --limit 3 --yes` runs at most three normal guarded `up` operations. New
|
||||
unacknowledged messages or assigned/in-progress task revision changes supply the
|
||||
signal; an unchanged signal is not retried. Active/unknown identities, held turn
|
||||
guards, detach, disabled policy and the daily UTC attempt cap prevent launch.
|
||||
Attempts are charged before invocation, including failed/interrupted attempts.
|
||||
The cap limits turns, **not dollars or token consumption**. Unknown quota remains
|
||||
unknown. This controller makes no product/approval/assignment decisions.
|
||||
|
||||
`torch continuation pause --yes` prevents subsequent launches. Policy and attempt
|
||||
ledger are separate atomic files so an in-flight turn cannot overwrite a pause.
|
||||
Pause does not kill an already-running turn. A controller lock serializes ticks;
|
||||
a crashed lock is retained rather than expired automatically. Each executor has
|
||||
a separate physical turn guard held through classification and final presence,
|
||||
so an agent reporting idle cannot permit overlapping native turns.
|
||||
|
||||
The first host pilot uses an explicitly installed transient user-systemd timer.
|
||||
It survives this chat ending, but not a host reboot. Permanent installer-owned
|
||||
provisioning/removal and installed artifact qualification remain existing backlog
|
||||
work; this is not a claim that the alpha.3 installed package has been upgraded.
|
||||
|
||||
```yaml
|
||||
test_integrity_note:
|
||||
change_type: new_tests
|
||||
scenarios:
|
||||
- id: SCN-legacy-runtime-recovery
|
||||
given: "An exact retained working/unknown identity and stopped project invocation"
|
||||
when: "The owner previews and applies recovery"
|
||||
then: "Presence becomes offline once; session, task and unknown outcome persist with one owner audit"
|
||||
- id: SCN-legacy-runtime-recovery-refusal
|
||||
given: "Live, foreign, mismatched or incomplete stopped-executor evidence"
|
||||
when: "Recovery evidence is requested"
|
||||
then: "Recovery refuses without identity mutation"
|
||||
- id: SCN-legacy-runtime-recovery-cas
|
||||
given: "Issued proof followed by a changed identity, live process, expiration or forgery"
|
||||
when: "Recovery is applied"
|
||||
then: "Transactional recovery refuses and creates no success audit"
|
||||
- id: SCN-legacy-runtime-recovery-authority
|
||||
given: "A specialist actor or incomplete expected snapshot"
|
||||
when: "Owner recovery is requested"
|
||||
then: "Authority or snapshot validation refuses"
|
||||
- id: SCN-runtime-turn-guard
|
||||
given: "A physical executor holds an identity guard and reports idle early"
|
||||
when: "Another executor attempts the same identity"
|
||||
then: "It refuses until the first executor finishes and releases its own guard"
|
||||
- id: SCN-runtime-continuation
|
||||
given: "An explicitly enabled roster, a new durable signal and a fixed daily turn cap"
|
||||
when: "Continuation ticks, repeats a signal, exhausts its cap or is paused"
|
||||
then: "Only eligible new signals launch; deduplication, cap, owner authority and pause remain enforced"
|
||||
observable_outcomes:
|
||||
- "Durable presence, session and task state; owner-attributed audit; explicit refusal codes"
|
||||
determinism_controls:
|
||||
- "Hermetic Git/install fixtures, fixed clock and injected OS observation boundary"
|
||||
anti_cheat_rationale:
|
||||
prevents:
|
||||
- "Caller Boolean as stopped proof"
|
||||
- "Process stop promoted to successful task or test"
|
||||
- "Stale snapshot overwrites or silent peer impersonation"
|
||||
- "Expired, foreign or replayed proof"
|
||||
relaxation:
|
||||
did_relax_any_assertion: false
|
||||
if_true_explain_spec_basis: ""
|
||||
```
|
||||
@@ -0,0 +1,53 @@
|
||||
# Owner repair and bounded continuation pilot
|
||||
|
||||
## Candidate and verification
|
||||
|
||||
Canonical development branch: `rewrite/portable-agent-fleet`.
|
||||
|
||||
- `11dabd2`: exact stopped-unknown owner recovery, physical turn guard, opt-in continuation.
|
||||
- `4c42715`: reviewed Runtime MCP XDG runtime-directory forwarding, cherry-picked without its unfinished broader branch.
|
||||
- `f823714`: bounded parallel turns and signal watermarks that do not wake on acknowledgements alone.
|
||||
- Complete suite: 266 PASS, 4 skipped, zero failures/cancellations (`/tmp/torch-continuation-final-suite.log`). Syntax check and full lint PASS. Subsequent MCP focused suite: 6 PASS after removing unnecessary string escapes, with identical assertion semantics.
|
||||
|
||||
These are local/source verification results, not independent QA acceptance or an installed-release upgrade. No acceptance assertions or test timeouts were relaxed.
|
||||
|
||||
## Actual recovery
|
||||
|
||||
Owner-selected legacy units were verified terminal, with no live matching native process. The new public source CLI recovered these identities to offline, retaining prior unknown outcomes and runtime/session/task data:
|
||||
|
||||
| Identity | Owner audit |
|
||||
| --- | --- |
|
||||
| owner-console | cfae5b3d-695b-4657-95f0-cdba230741db |
|
||||
| qa | 158bc692-e775-49a2-987f-8d9d3e8ffb9a |
|
||||
| release-self-host | 7f618cd6-86cd-471f-a5f0-d7f03f8c6fb4 |
|
||||
|
||||
This is an explicit owner-selected legacy invocation binding, not cryptographic session provenance. Old failed units remain evidence. No processes were killed, task/test results forged, reservations fabricated, or schema migrations applied.
|
||||
|
||||
## Actual pilot
|
||||
|
||||
Project: `3dc11932-6e3c-42aa-9a8a-ca7b8f818b54`.
|
||||
|
||||
Transient user timer/service: `torch-3dc11932-6e3c-42aa-9a8a-ca7b8f818b54-continuation-pilot`.
|
||||
|
||||
First activation: 2026-10-03 04:00:18 UTC. Source CLI starts at most three existing approved identities concurrently, charges attempts before launch, and checks again two minutes after the previous batch exits. At verification, three attempts were charged and physical guards existed for session-manager, project-kernel and provider-runtime; their source `up` child processes were running. Daily cap: 12 turns, UTC day. This is a turn cap, NOT measured token/cost governance. No additional identities were authorized.
|
||||
|
||||
Monitoring URL returned HTTP 200: <http://127.0.0.1:4174/console>.
|
||||
|
||||
Pause future turns from the canonical repository:
|
||||
|
||||
```sh
|
||||
node bin/torch.mjs continuation pause --yes --json
|
||||
```
|
||||
|
||||
Pause does not kill already-started turns. Inspect policy/eligibility with `node bin/torch.mjs continuation status --json`. The timer remains a cheap deterministic check when paused or capped.
|
||||
|
||||
## Remaining qualification
|
||||
|
||||
- Independent QA of this exact composed candidate and real native completion/continuation.
|
||||
- Startup presence truthfulness: physical launches may still display idle until their first status report.
|
||||
- Installer-owned persistent timer registration, removal, upgrade and reboot qualification. This transient timer survives chat exit, not host reboot.
|
||||
- Crash reconciliation for retained controller/turn locks; no time-based expiry silently permits overlapping executors.
|
||||
- Real usage governor, runtime/model/auth qualification and operational failure reporting.
|
||||
- Existing umbrella tasks remain open; the previously observed CLI timeout still needs a root-cause diagnosis despite this passing run.
|
||||
|
||||
Installed alpha.3 runtime and console service were not replaced. The older schedule framework's automatic model wakes remain disabled; the source continuation pilot is separately authorized.
|
||||
+53
-1
@@ -14,7 +14,7 @@ import { startConsole } from './console/server.mjs';
|
||||
import { classifyRecoverability, createLocalCanonical, planLocalCanonical } from './canonical/local.mjs';
|
||||
import { openControlPlane } from './control-plane/service.mjs';
|
||||
import { fetchCanonicalObjects, planCanonicalFetch } from './canonical/fetch.mjs';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { spawn as spawnProcess, spawnSync } from 'node:child_process';
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import {
|
||||
mkdirSync, mkdtempSync, renameSync, rmSync, statSync, writeFileSync,
|
||||
@@ -70,6 +70,8 @@ import {
|
||||
} from './design/architect.mjs';
|
||||
import { FleetEvolutionService } from './evolution/service.mjs';
|
||||
import { HierarchyEvolutionService } from './evolution/hierarchy.mjs';
|
||||
import { issueStoppedExecutorEvidence } from './runtime/stopped-executor-evidence.mjs';
|
||||
import { configureContinuation, planContinuation, runContinuation } from './runtime/continuation.mjs';
|
||||
import { ConvergenceService } from './convergence/service.mjs';
|
||||
import {
|
||||
attachForge, detachForge, forgeStatus, planForgeAttach, planForgeDetach,
|
||||
@@ -95,6 +97,8 @@ Usage:
|
||||
torch install --restore [--dry-run] [--yes] [--json]
|
||||
torch setup [--parent <path>] [--dry-run] [--yes] [--json]
|
||||
torch up [--fresh] [--only <id,id>] [--dry-run] [--yes] [--json]
|
||||
torch recover-runtime --area <id> --unit <service> --invocation <id> [--expected-session <id> --expected-updated-at <time> --yes] [--json]
|
||||
torch continuation <status|enable|pause|tick> [--max-turns-per-day <1-48>] [--limit <1-3>] [--yes] [--json]
|
||||
torch down [--dry-run] [--yes] [--json]
|
||||
torch capture [--json]
|
||||
torch detach [--dry-run] [--yes] [--json]
|
||||
@@ -576,6 +580,54 @@ export async function runCli(argv = process.argv.slice(2), {
|
||||
process.stdout.write(HELP);
|
||||
return 0;
|
||||
}
|
||||
if (command === 'continuation') {
|
||||
const control = openControlPlane({ repositoryRoot: optionValue(argv, '--repo') ?? cwd, env });
|
||||
try {
|
||||
const operation = argv[1] ?? 'status';
|
||||
const backlog = new BacklogService({ repositoryRoot: control.repositoryRoot, controlPlane: control });
|
||||
if (operation === 'status' || !argv.includes('--yes') || argv.includes('--dry-run')) {
|
||||
print(planContinuation(control, backlog), { json }); return 0;
|
||||
}
|
||||
if (operation === 'enable' || operation === 'pause') {
|
||||
print(configureContinuation(control, { actorId: 'owner', enabled: operation === 'enable',
|
||||
maxTurnsPerDay: Number(optionValue(argv, '--max-turns-per-day') ?? 12) }), { json }); return 0;
|
||||
}
|
||||
if (operation !== 'tick') throw new TorchError('Unknown continuation operation.', { code: 'UNKNOWN_COMMAND' });
|
||||
print(await runContinuation(control, backlog, { actorId: 'owner', limit: Number(optionValue(argv, '--limit') ?? 3),
|
||||
launch: areaId => new Promise(resolveTurn => {
|
||||
const child = spawnProcess(process.execPath, [fileURLToPath(new URL('../bin/torch.mjs', import.meta.url)),
|
||||
'up', '--only', areaId, '--yes', '--json'], { cwd: control.repositoryRoot, env,
|
||||
stdio: ['ignore', 'pipe', 'pipe'] });
|
||||
child.stdout.resume(); child.stderr.resume();
|
||||
let failed = false;
|
||||
child.on('error', () => { failed = true; });
|
||||
child.on('close', (code, signal) => resolveTurn(failed || signal ? null : code));
|
||||
}) }), { json });
|
||||
return 0;
|
||||
} finally { control.close(); }
|
||||
}
|
||||
if (command === 'recover-runtime') {
|
||||
const control = openControlPlane({ repositoryRoot: optionValue(argv, '--repo') ?? cwd, env });
|
||||
try {
|
||||
const areaId = optionValue(argv, '--area');
|
||||
const issued = issueStoppedExecutorEvidence({ controlPlane: control, actorId: 'owner', areaId,
|
||||
unit: optionValue(argv, '--unit'), invocationId: optionValue(argv, '--invocation') });
|
||||
if (!argv.includes('--yes') || argv.includes('--dry-run')) {
|
||||
print({ ...issued.plan, requiresApproval: true }, { json });
|
||||
return 0;
|
||||
}
|
||||
const expected = issued.plan.expectedIdentity;
|
||||
if (optionValue(argv, '--expected-session') !== expected.runtimeSessionId
|
||||
|| optionValue(argv, '--expected-updated-at') !== expected.updatedAt) {
|
||||
throw new TorchError('Apply requires the exact session and updated timestamp from the recovery preview.', {
|
||||
code: 'RUNTIME_RECOVERY_SNAPSHOT_CONFLICT',
|
||||
});
|
||||
}
|
||||
print(control.recoverOwnerStoppedUnknownIdentity({ actorId: 'owner', areaId,
|
||||
expectedIdentity: expected, evidence: issued.evidence }), { json });
|
||||
return 0;
|
||||
} finally { control.close(); }
|
||||
}
|
||||
if (command === 'runtimes') {
|
||||
const operation = argv[1] ?? 'list';
|
||||
if (operation === 'update') {
|
||||
|
||||
@@ -7,6 +7,7 @@ import { readInstallManifest } from '../kernel/install.mjs';
|
||||
import { projectStatePath } from '../kernel/paths.mjs';
|
||||
import { loadProjectConfig } from '../kernel/config.mjs';
|
||||
import { organizationGraphFromConfig } from '../kernel/organization.mjs';
|
||||
import { consumeStoppedExecutorEvidence } from '../runtime/stopped-executor-evidence.mjs';
|
||||
|
||||
export const PRESENCE_STATES = Object.freeze([
|
||||
'starting', 'working', 'waiting', 'idle', 'stale', 'stopping', 'offline',
|
||||
@@ -459,6 +460,36 @@ export class ControlPlane {
|
||||
return { messageId: id, recipient: recipientId, acknowledgedAt };
|
||||
}
|
||||
|
||||
recoverOwnerStoppedUnknownIdentity({ actorId, areaId, expectedIdentity, evidence } = {}) {
|
||||
const owner = this.assertOwnerActor(actorId);
|
||||
const id = this.assertIdentity(areaId);
|
||||
if (!expectedIdentity || ['state', 'runtime', 'runtimeSessionId', 'updatedAt'].some(key => !expectedIdentity[key])) {
|
||||
throw new TorchError('Recovery requires the complete previewed identity snapshot.', { code: 'RUNTIME_RECOVERY_SNAPSHOT_REQUIRED' });
|
||||
}
|
||||
this.database.exec('BEGIN IMMEDIATE');
|
||||
try {
|
||||
const current = this.identity(id);
|
||||
if (current.state !== 'working' || !current.summary?.includes('Runtime executor completion is unknown')
|
||||
|| ['state', 'runtime', 'runtimeSessionId', 'updatedAt'].some(key => current[key] !== expectedIdentity[key])) {
|
||||
throw new TorchError('Identity changed since recovery was reviewed.', { code: 'RUNTIME_RECOVERY_SNAPSHOT_CONFLICT' });
|
||||
}
|
||||
const facts = consumeStoppedExecutorEvidence(evidence, { projectId: this.projectId, repositoryRoot: this.repositoryRoot, identity: current });
|
||||
const now = this.clock().toISOString();
|
||||
const result = this.database.prepare(`UPDATE identities SET state = 'offline', summary = ?, heartbeat_at = ?, updated_at = ?
|
||||
WHERE area_id = ? AND state = ? AND runtime = ? AND runtime_session_id = ? AND updated_at = ?`)
|
||||
.run('Owner verified stopped legacy executor; prior outcome remains unknown.', now, now,
|
||||
id, current.state, current.runtime, current.runtimeSessionId, current.updatedAt);
|
||||
if (result.changes !== 1) throw new TorchError('Recovery lost its identity snapshot.', { code: 'RUNTIME_RECOVERY_SNAPSHOT_CONFLICT' });
|
||||
const audit = this.auditOwnerAction({ actorId: owner, operation: 'runtime.legacy-unknown.recover-stopped',
|
||||
entityType: 'identity', entityId: id, details: { previousIdentity: expectedIdentity, ...facts } });
|
||||
this.database.exec('COMMIT');
|
||||
return { identity: this.identity(id), auditId: audit.id, priorOutcome: 'unknown', sessionsStarted: false, mutationPerformed: true };
|
||||
} catch (error) {
|
||||
this.database.exec('ROLLBACK');
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
reportStatus({ areaId, state, summary, runtime, runtimeSessionId, task } = {}) {
|
||||
const id = this.assertIdentity(areaId);
|
||||
const normalizedState = requiredText(state, 'state');
|
||||
@@ -751,7 +782,7 @@ export class ControlPlane {
|
||||
|
||||
auditOwnerAction({ actorId, operation, entityType, entityId = null, details = null } = {}) {
|
||||
const actor = this.assertOwnerActor(actorId);
|
||||
if (!['artifact.feedback', 'backlog.priority', 'backlog.create', 'agent.request', 'approval.decide', 'approval.notify', 'forge.sync', 'canonical.fetch', 'profile.change', 'schedule.launcher.install', 'schedule.launcher.reconcile', 'schedule.wake.recover', 'hierarchy.conclude-pilot', 'check.recover-prepared'].includes(operation)) {
|
||||
if (!['artifact.feedback', 'backlog.priority', 'backlog.create', 'agent.request', 'approval.decide', 'approval.notify', 'forge.sync', 'canonical.fetch', 'profile.change', 'schedule.launcher.install', 'schedule.launcher.reconcile', 'schedule.wake.recover', 'hierarchy.conclude-pilot', 'check.recover-prepared', 'runtime.legacy-unknown.recover-stopped', 'runtime.continuation.configure'].includes(operation)) {
|
||||
throw new TorchError('This owner audit surface does not authorize the requested operation', {
|
||||
code: 'OWNER_OPERATION_UNSUPPORTED', details: { operation },
|
||||
});
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
import { createHash, randomUUID } from 'node:crypto';
|
||||
import { existsSync, mkdirSync, readFileSync, renameSync, rmdirSync, writeFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { TorchError } from '../kernel/errors.mjs';
|
||||
|
||||
function statePath(control) { return join(control.stateRoot, 'continuation.json'); }
|
||||
function read(control) {
|
||||
const policyPath = join(control.stateRoot, 'continuation-policy.json');
|
||||
const state = { enabled: false, maxTurnsPerDay: 12, areas: [], day: null, attempts: 0, handled: {},
|
||||
...(existsSync(statePath(control)) ? JSON.parse(readFileSync(statePath(control), 'utf8')) : {}),
|
||||
...(existsSync(policyPath) ? JSON.parse(readFileSync(policyPath, 'utf8')) : {}) };
|
||||
if (typeof state.enabled !== 'boolean' || !Number.isInteger(state.maxTurnsPerDay)
|
||||
|| state.maxTurnsPerDay < 1 || state.maxTurnsPerDay > 48 || !Array.isArray(state.areas)
|
||||
|| !Number.isInteger(state.attempts) || state.attempts < 0 || !state.handled || typeof state.handled !== 'object') {
|
||||
throw new TorchError('Continuation policy or attempt ledger is invalid; launch refused.', { code: 'CONTINUATION_POLICY_INVALID' });
|
||||
}
|
||||
return state;
|
||||
}
|
||||
function save(control, state, policy = false) {
|
||||
const path = policy ? join(control.stateRoot, 'continuation-policy.json') : statePath(control);
|
||||
const temporary = `${path}.${randomUUID()}.tmp`;
|
||||
const value = policy ? { enabled: state.enabled, maxTurnsPerDay: state.maxTurnsPerDay, areas: state.areas }
|
||||
: { day: state.day, attempts: state.attempts, handled: state.handled };
|
||||
writeFileSync(temporary, JSON.stringify(value), { flag: 'wx', mode: 0o600 });
|
||||
renameSync(temporary, path);
|
||||
}
|
||||
|
||||
export function configureContinuation(control, { actorId, enabled, maxTurnsPerDay = 12 } = {}) {
|
||||
control.assertOwnerActor(actorId);
|
||||
if (typeof enabled !== 'boolean' || !Number.isInteger(maxTurnsPerDay) || maxTurnsPerDay < 1 || maxTurnsPerDay > 48) {
|
||||
throw new TorchError('Continuation requires an explicit enable/pause and 1–48 daily turns.', { code: 'CONTINUATION_POLICY_INVALID' });
|
||||
}
|
||||
const state = { ...read(control), enabled, maxTurnsPerDay, areas: control.listAgents().map(a => a.areaId) };
|
||||
save(control, state, true);
|
||||
control.auditOwnerAction({ actorId, operation: 'runtime.continuation.configure', entityType: 'project', entityId: control.projectId,
|
||||
details: { enabled, maxTurnsPerDay, areas: state.areas } });
|
||||
return { enabled, maxTurnsPerDay, areas: state.areas, sessionsStarted: false };
|
||||
}
|
||||
|
||||
export function planContinuation(control, backlog, { at = new Date(), limit = 3 } = {}) {
|
||||
const state = read(control);
|
||||
const day = at.toISOString().slice(0, 10);
|
||||
const attempts = state.day === day ? state.attempts : 0;
|
||||
if (!Number.isInteger(limit) || limit < 1 || limit > 3) throw new TorchError('Continuation limit must be 1–3.', { code: 'CONTINUATION_POLICY_INVALID' });
|
||||
const metadata = JSON.parse(readFileSync(join(control.stateRoot, 'project.json'), 'utf8'));
|
||||
if (!state.enabled || metadata.detachedAt || attempts >= state.maxTurnsPerDay) {
|
||||
return { enabled: state.enabled, reason: metadata.detachedAt ? 'detached' : !state.enabled ? 'paused' : 'daily-turn-cap', candidates: [], attempts };
|
||||
}
|
||||
const tasks = backlog.list();
|
||||
const candidates = [];
|
||||
for (const areaId of state.areas) {
|
||||
const agent = control.getAgent(areaId);
|
||||
if (!['idle', 'offline', 'waiting'].includes(agent.state)
|
||||
|| existsSync(join(control.stateRoot, 'sessions', 'turn-guards', areaId))) continue;
|
||||
const messages = control.readMessages({ recipient: areaId, unacknowledgedOnly: true, limit: 1000 });
|
||||
const active = tasks.filter(t => t.owner === areaId && ['assigned', 'in_progress'].includes(t.state));
|
||||
const previous = state.handled[areaId] ?? {};
|
||||
const freshMessages = messages.filter(m => !previous.messageIds?.includes(m.id));
|
||||
const freshTasks = active.filter(t => previous.taskRevisions?.[t.id] !== t.revision);
|
||||
if (!freshMessages.length && !freshTasks.length) continue;
|
||||
const signal = { messageIds: messages.map(m => m.id), taskRevisions: Object.fromEntries(active.map(t => [t.id, t.revision])) };
|
||||
const fingerprint = createHash('sha256').update(JSON.stringify(signal)).digest('hex');
|
||||
candidates.push({ areaId, fingerprint, signal, messageCount: freshMessages.length, taskIds: freshTasks.map(t => t.id) });
|
||||
}
|
||||
candidates.sort((a, b) => Number(b.areaId === 'session-manager') - Number(a.areaId === 'session-manager'));
|
||||
return { enabled: true, day, attempts, remainingTurns: state.maxTurnsPerDay - attempts,
|
||||
candidates: candidates.slice(0, Math.min(limit, state.maxTurnsPerDay - attempts)), mutationPerformed: false };
|
||||
}
|
||||
|
||||
export async function runContinuation(control, backlog, { actorId, launch, at, clock = () => at ?? new Date(), limit = 3 } = {}) {
|
||||
control.assertOwnerActor(actorId);
|
||||
if (typeof launch !== 'function') throw new TorchError('Continuation needs an authorized launcher.', { code: 'RUNTIME_EXECUTION_NOT_AUTHORIZED' });
|
||||
const lock = join(control.stateRoot, 'continuation.lock');
|
||||
try { mkdirSync(lock); } catch (error) {
|
||||
if (error.code === 'EEXIST') return { reason: 'controller-active-or-unreconciled', launched: [] };
|
||||
throw error;
|
||||
}
|
||||
try {
|
||||
const plan = planContinuation(control, backlog, { at: clock(), limit });
|
||||
const pending = [];
|
||||
for (const candidate of plan.candidates) {
|
||||
// Fresh plan and policy check before each launch; a concurrent pause wins.
|
||||
const current = planContinuation(control, backlog, { at: clock(), limit: 3 });
|
||||
if (!current.candidates.some(c => c.areaId === candidate.areaId && c.fingerprint === candidate.fingerprint)) continue;
|
||||
const state = read(control);
|
||||
if (state.day !== current.day) { state.day = current.day; state.attempts = 0; }
|
||||
state.attempts++;
|
||||
state.handled[candidate.areaId] = candidate.signal;
|
||||
save(control, state); // Charge/reserve before launch; interrupted attempts are never refunded/retried silently.
|
||||
let result;
|
||||
try { result = launch(candidate.areaId); } catch { result = null; }
|
||||
pending.push(Promise.resolve(result).catch(() => null).then(status => ({ areaId: candidate.areaId, status })));
|
||||
}
|
||||
return { launched: await Promise.all(pending), reason: plan.reason ?? null };
|
||||
} finally { rmdirSync(lock); }
|
||||
}
|
||||
@@ -11,6 +11,7 @@ import {
|
||||
costCeilingPlanIssues, profileCapabilityIssues, resolveLaunchPolicy, validateRuntimeAdapter,
|
||||
} from '../adapters/runtime.mjs';
|
||||
import { hierarchyOrder } from './hierarchy-order.mjs';
|
||||
import { withRuntimeTurnGuard } from './turn-guard.mjs';
|
||||
|
||||
const MCP_ENTRY = fileURLToPath(new URL('../mcp/server.mjs', import.meta.url));
|
||||
|
||||
@@ -570,6 +571,7 @@ export function startFleet({ plan, controlPlane, executor, adapters } = {}) {
|
||||
const runtimes = adapterMap({ adapters });
|
||||
for (const action of plan.actions) {
|
||||
controlPlane.assertIdentity(action.areaId);
|
||||
withRuntimeTurnGuard(controlPlane, action.areaId, () => {
|
||||
writeCombinedPrompt(action);
|
||||
const result = executor({
|
||||
...action.launch, areaId: action.areaId, runtime: action.runtime,
|
||||
@@ -634,6 +636,7 @@ export function startFleet({ plan, controlPlane, executor, adapters } = {}) {
|
||||
? `${action.mode} turn completed` : `${action.mode} requested`,
|
||||
});
|
||||
started.push({ areaId: action.areaId, runtimeSessionId: identity.runtimeSessionId, mode: action.mode });
|
||||
});
|
||||
}
|
||||
if (plan.stateRoot) {
|
||||
const metadata = projectMetadata(plan.stateRoot);
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { readFileSync, readdirSync, readlinkSync, statSync } from 'node:fs';
|
||||
import { basename, join, resolve } from 'node:path';
|
||||
import { TorchError } from '../kernel/errors.mjs';
|
||||
|
||||
const issued = new WeakMap();
|
||||
const TTL_MS = 60_000;
|
||||
function refuse(message) {
|
||||
throw new TorchError(message, { code: 'RUNTIME_RECOVERY_UNVERIFIED' });
|
||||
}
|
||||
|
||||
export function readStoppedUnit(unit) {
|
||||
const result = spawnSync('systemctl', ['--user', 'show', unit,
|
||||
'--property=Id,ActiveState,SubState,MainPID,ControlPID,InvocationID,ExecStart,WorkingDirectory,ControlGroup,ExecMainExitTimestamp'],
|
||||
{ encoding: 'utf8', timeout: 5000, maxBuffer: 64 * 1024 });
|
||||
if (result.error || result.signal || result.status !== 0) refuse('Cannot verify the owner-selected service invocation.');
|
||||
return Object.fromEntries(result.stdout.trim().split('\n').map(line => {
|
||||
const index = line.indexOf('=');
|
||||
return [line.slice(0, index), line.slice(index + 1)];
|
||||
}));
|
||||
}
|
||||
|
||||
export function readNativeProcesses() {
|
||||
if (process.platform !== 'linux') refuse('Legacy process inspection is supported only on Linux.');
|
||||
const processes = [];
|
||||
for (const name of readdirSync('/proc').filter(value => /^\d+$/.test(value))) {
|
||||
const directory = join('/proc', name);
|
||||
try {
|
||||
if (statSync(directory).uid !== process.getuid()) continue;
|
||||
const argv = readFileSync(join(directory, 'cmdline'), 'utf8').split('\0').filter(Boolean);
|
||||
if (!argv.length) continue;
|
||||
const native = argv.some(value => /^(?:codex|claude|pi)(?:\.js)?$/.test(basename(value)));
|
||||
if (native) processes.push({ pid: Number(name), argv, cwd: readlinkSync(join(directory, 'cwd')) });
|
||||
} catch (error) {
|
||||
if (error.code !== 'ENOENT' && error.code !== 'ESRCH') refuse('Native process visibility is incomplete; recovery refused.');
|
||||
}
|
||||
}
|
||||
return processes;
|
||||
}
|
||||
|
||||
function verify(record) {
|
||||
const unit = record.readUnit(record.unit);
|
||||
const prefix = `torch-${record.projectId}-`;
|
||||
if (typeof record.unit !== 'string' || !record.unit.startsWith(prefix) || !/^[A-Za-z0-9_.-]+\.service$/.test(record.unit)
|
||||
|| unit.Id !== record.unit || !/^[a-f0-9]{32}$/.test(unit.InvocationID ?? '')
|
||||
|| unit.InvocationID !== record.invocationId
|
||||
|| !['failed', 'inactive'].includes(unit.ActiveState)
|
||||
|| !['failed', 'dead'].includes(unit.SubState)
|
||||
|| unit.MainPID !== '0' || unit.ControlPID !== '0' || unit.ControlGroup !== ''
|
||||
|| resolve(unit.WorkingDirectory || '/') !== record.repositoryRoot) {
|
||||
refuse('The selected project service is not a verified stopped invocation.');
|
||||
}
|
||||
// Explicit legacy operator binding, not a claim of cryptographic session provenance.
|
||||
// Exact area token and terminal timestamp must agree with the retained unknown row.
|
||||
if (!(unit.ExecStart ?? '').split(/[\s;]+/).includes(record.identity.areaId)
|
||||
|| !Number.isFinite(Date.parse(unit.ExecMainExitTimestamp))
|
||||
|| Math.abs(Date.parse(unit.ExecMainExitTimestamp) - Date.parse(record.identity.updatedAt)) > 5000) {
|
||||
refuse('Service area or terminal timestamp does not match the retained unknown identity.');
|
||||
}
|
||||
for (const processRecord of record.readProcesses()) {
|
||||
if (processRecord.argv.includes(record.identity.runtimeSessionId)
|
||||
|| processRecord.argv.includes(record.worktree)
|
||||
|| resolve(processRecord.cwd) === record.worktree) {
|
||||
refuse('A native process still references the selected session or workspace.');
|
||||
}
|
||||
}
|
||||
return { unit: record.unit, invocationId: unit.InvocationID,
|
||||
terminalAt: new Date(Date.parse(unit.ExecMainExitTimestamp)).toISOString(),
|
||||
outcome: 'unknown', verification: 'owner-selected-legacy-invocation' };
|
||||
}
|
||||
|
||||
export function issueStoppedExecutorEvidence({ controlPlane, actorId, areaId, unit, invocationId,
|
||||
readUnit = readStoppedUnit, readProcesses = readNativeProcesses, now = () => Date.now() } = {}) {
|
||||
controlPlane.assertOwnerActor(actorId);
|
||||
const identity = controlPlane.getAgent(areaId);
|
||||
if (identity.state !== 'working' || !identity.runtimeSessionId || !identity.updatedAt
|
||||
|| !identity.summary?.includes('Runtime executor completion is unknown')) {
|
||||
refuse('Only an exact retained working/unknown identity can use legacy recovery.');
|
||||
}
|
||||
const worktree = controlPlane.manifest.external?.find(entry => entry.type === 'worktree' && entry.area === areaId)?.path;
|
||||
if (!worktree) refuse('No registered workspace exists for this identity.');
|
||||
const record = { identity: Object.freeze({ areaId, state: identity.state, runtime: identity.runtime,
|
||||
runtimeSessionId: identity.runtimeSessionId, updatedAt: identity.updatedAt }),
|
||||
unit, invocationId, projectId: controlPlane.projectId, repositoryRoot: resolve(controlPlane.repositoryRoot),
|
||||
worktree: resolve(worktree), readUnit, readProcesses, now, issuedAt: now() };
|
||||
const facts = verify(record);
|
||||
const evidence = Object.freeze({});
|
||||
issued.set(evidence, record);
|
||||
return { evidence, plan: { areaId, expectedIdentity: record.identity, ...facts,
|
||||
mutationPerformed: false, effect: 'Recover presence to offline; preserve task, session and unknown outcome. No provider launch.' } };
|
||||
}
|
||||
|
||||
export function consumeStoppedExecutorEvidence(evidence, { projectId, repositoryRoot, identity } = {}) {
|
||||
const record = issued.get(evidence);
|
||||
if (!record) refuse('Recovery requires fresh Runtime-issued evidence, not caller text or a Boolean.');
|
||||
issued.delete(evidence);
|
||||
if (record.projectId !== projectId || record.repositoryRoot !== resolve(repositoryRoot)
|
||||
|| record.now() < record.issuedAt
|
||||
|| record.now() - record.issuedAt > TTL_MS
|
||||
|| ['areaId', 'state', 'runtime', 'runtimeSessionId', 'updatedAt'].some(key => record.identity[key] !== identity[key])) {
|
||||
refuse('Recovery evidence is stale, expired or belongs to another identity.');
|
||||
}
|
||||
const facts = verify(record); // Recheck physical state immediately before the transactional mutation.
|
||||
return { ...facts, digest: createHash('sha256').update(JSON.stringify({ identity: record.identity, ...facts })).digest('hex') };
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import { mkdirSync, readFileSync, rmdirSync, unlinkSync, writeFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { TorchError } from '../kernel/errors.mjs';
|
||||
|
||||
// One physical executor per identity, including the interval after an agent
|
||||
// reports idle but before its harness actually exits. Crashed guards never expire.
|
||||
export function withRuntimeTurnGuard(controlPlane, areaId, run) {
|
||||
controlPlane.assertIdentity(areaId);
|
||||
if (!/^[a-z0-9][a-z0-9-]*$/.test(areaId)) throw new TorchError('Invalid runtime guard identity.', { code: 'INVALID_RUNTIME_INPUT' });
|
||||
const parent = join(controlPlane.stateRoot, 'sessions', 'turn-guards');
|
||||
mkdirSync(parent, { recursive: true });
|
||||
const directory = join(parent, areaId);
|
||||
try { mkdirSync(directory); } catch (error) {
|
||||
if (error.code === 'EEXIST') throw new TorchError('An executor already holds this identity turn guard.', {
|
||||
code: 'RUNTIME_TURN_ACTIVE', details: { areaId },
|
||||
});
|
||||
throw error;
|
||||
}
|
||||
const path = join(directory, 'owner.json');
|
||||
const nonce = randomUUID();
|
||||
writeFileSync(path, JSON.stringify({ nonce, pid: process.pid, areaId, projectId: controlPlane.projectId }), { flag: 'wx', mode: 0o600 });
|
||||
try { return run(); } finally {
|
||||
// Delete only the exact guard created by this invocation; preserve foreign/changed guards.
|
||||
const record = JSON.parse(readFileSync(path, 'utf8'));
|
||||
if (record.nonce === nonce && record.pid === process.pid) {
|
||||
unlinkSync(path);
|
||||
rmdirSync(directory);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,194 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { mkdtempSync, readFileSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import test from 'node:test';
|
||||
import { analyzeRepository } from '../../src/kernel/analyze.mjs';
|
||||
import { inspectRepository } from '../../src/kernel/git.mjs';
|
||||
import { proposeDomains } from '../../src/kernel/domains.mjs';
|
||||
import { installProject } from '../../src/kernel/install.mjs';
|
||||
import { openControlPlane } from '../../src/control-plane/service.mjs';
|
||||
import { issueStoppedExecutorEvidence } from '../../src/runtime/stopped-executor-evidence.mjs';
|
||||
import { withRuntimeTurnGuard } from '../../src/runtime/turn-guard.mjs';
|
||||
import { configureContinuation, planContinuation, runContinuation } from '../../src/runtime/continuation.mjs';
|
||||
|
||||
function fixture() {
|
||||
const root = mkdtempSync(join(tmpdir(), 'torch-legacy-recovery-'));
|
||||
for (const args of [['init', '-b', 'main'], ['config', 'user.email', '[email protected]'],
|
||||
['config', 'user.name', 'Recovery Fixture']]) execFileSync('git', args, { cwd: root, stdio: 'pipe' });
|
||||
writeFileSync(join(root, 'app.js'), 'export const ready = true;\n');
|
||||
execFileSync('git', ['add', '.'], { cwd: root });
|
||||
execFileSync('git', ['commit', '-m', 'fixture'], { cwd: root, stdio: 'pipe' });
|
||||
const repository = inspectRepository(root);
|
||||
const proposal = proposeDomains({ repository, analysis: analyzeRepository(repository) });
|
||||
proposal.review = { status: 'approved', reviewedAt: '2026-10-03T00:00:00Z', reviewedBy: 'owner', notes: [] };
|
||||
const env = { ...process.env, XDG_DATA_HOME: join(root, '.data') };
|
||||
installProject({ repository, proposal, env, projectId: 'recovery-fixture' });
|
||||
const actualPath = join(root, '.torch', 'install-manifest.json');
|
||||
const manifest = JSON.parse(readFileSync(actualPath, 'utf8'));
|
||||
const areaId = proposal.domains[0].id;
|
||||
manifest.external.push({ type: 'worktree', area: areaId, path: join(root, 'worker') });
|
||||
writeFileSync(actualPath, JSON.stringify(manifest));
|
||||
let time = Date.parse('2026-10-03T00:00:00Z');
|
||||
const control = openControlPlane({ repositoryRoot: root, env, clock: () => new Date(time) });
|
||||
control.reportStatus({ areaId, state: 'working', runtime: 'codex', runtimeSessionId: 'fixture-session',
|
||||
task: 'TASK-preserved', summary: 'Runtime executor completion is unknown; duplicate launch is blocked pending reconciliation.' });
|
||||
const identity = control.getAgent(areaId);
|
||||
const unit = `torch-${control.projectId}-worker.service`;
|
||||
const properties = { Id: unit, ActiveState: 'failed', SubState: 'failed', MainPID: '0', ControlPID: '0',
|
||||
ControlGroup: '', InvocationID: 'a'.repeat(32), ExecStart: `node /tmp/probe.mjs ${areaId} ;`,
|
||||
WorkingDirectory: root, ExecMainExitTimestamp: identity.updatedAt };
|
||||
let processes = [];
|
||||
const issue = () => issueStoppedExecutorEvidence({ controlPlane: control, actorId: 'owner', areaId, unit,
|
||||
invocationId: 'a'.repeat(32), readUnit: () => ({ ...properties }), readProcesses: () => processes, now: () => time });
|
||||
const apply = issued => control.recoverOwnerStoppedUnknownIdentity({ actorId: 'owner', areaId,
|
||||
expectedIdentity: issued.plan.expectedIdentity, evidence: issued.evidence });
|
||||
return { control, identity, areaId, properties, issue, apply,
|
||||
advance: ms => { time += ms; }, processes: value => { processes = value; } };
|
||||
}
|
||||
|
||||
test('SCN-legacy-runtime-recovery: owner recovery preserves task/session/unknown and records exactly one owner audit', () => {
|
||||
const f = fixture();
|
||||
try {
|
||||
const issued = f.issue();
|
||||
assert.equal(f.control.getAgent(f.areaId).state, 'working');
|
||||
const result = f.apply(issued);
|
||||
assert.equal(result.identity.state, 'offline');
|
||||
assert.equal(result.identity.currentTask, 'TASK-preserved');
|
||||
assert.equal(result.identity.runtimeSessionId, 'fixture-session');
|
||||
assert.equal(result.priorOutcome, 'unknown');
|
||||
assert.equal(result.sessionsStarted, false);
|
||||
const audits = f.control.readAudit({ limit: 100 }).filter(x => x.operation === 'runtime.legacy-unknown.recover-stopped');
|
||||
assert.equal(audits.length, 1);
|
||||
assert.equal(audits[0].actorId, 'owner');
|
||||
assert.equal(audits[0].details.outcome, 'unknown');
|
||||
assert.throws(() => f.apply(issued), { code: 'RUNTIME_RECOVERY_SNAPSHOT_CONFLICT' });
|
||||
} finally { f.control.close(); }
|
||||
});
|
||||
|
||||
test('SCN-legacy-runtime-recovery-refusal: live, foreign invocation, area, time and process mismatches cannot issue proof', () => {
|
||||
for (const patch of [{ MainPID: '99' }, { ControlPID: '99' }, { ActiveState: 'active' },
|
||||
{ InvocationID: 'b'.repeat(32) }, { ExecStart: 'node other-area' },
|
||||
{ ExecMainExitTimestamp: '2026-10-02T00:00:00Z' }, { ControlGroup: '/live' }]) {
|
||||
const f = fixture();
|
||||
try { Object.assign(f.properties, patch); assert.throws(f.issue, { code: 'RUNTIME_RECOVERY_UNVERIFIED' });
|
||||
assert.equal(f.control.getAgent(f.areaId).state, 'working'); } finally { f.control.close(); }
|
||||
}
|
||||
const f = fixture();
|
||||
try { f.processes([{ pid: 999, argv: ['codex', 'resume', 'fixture-session'], cwd: '/tmp' }]);
|
||||
assert.throws(f.issue, { code: 'RUNTIME_RECOVERY_UNVERIFIED' }); } finally { f.control.close(); }
|
||||
});
|
||||
|
||||
test('SCN-legacy-runtime-recovery-cas: changed snapshot, live recheck, expiry and caller-forged proof refuse without mutation', () => {
|
||||
for (const scenario of ['changed', 'live', 'expired', 'forged']) {
|
||||
const f = fixture();
|
||||
try {
|
||||
const issued = f.issue();
|
||||
if (scenario === 'changed') { f.advance(1); f.control.reportStatus({ areaId: f.areaId, state: 'working', runtimeSessionId: 'different', summary: 'Runtime executor completion is unknown' }); }
|
||||
if (scenario === 'live') f.properties.MainPID = '99';
|
||||
if (scenario === 'expired') f.advance(60_001);
|
||||
if (scenario === 'forged') issued.evidence = { stopped: true };
|
||||
assert.throws(() => f.apply(issued));
|
||||
assert.equal(f.control.getAgent(f.areaId).state, 'working');
|
||||
assert.equal(f.control.readAudit({ limit: 100 }).filter(x => x.operation === 'runtime.legacy-unknown.recover-stopped').length, 0);
|
||||
} finally { f.control.close(); }
|
||||
}
|
||||
});
|
||||
|
||||
test('SCN-legacy-runtime-recovery-authority: specialist and incomplete snapshot cannot use owner recovery', () => {
|
||||
const f = fixture();
|
||||
try {
|
||||
const issued = f.issue();
|
||||
assert.throws(() => f.control.recoverOwnerStoppedUnknownIdentity({ actorId: f.areaId, areaId: f.areaId,
|
||||
expectedIdentity: issued.plan.expectedIdentity, evidence: issued.evidence }), { code: 'OWNER_AUTHORITY_REQUIRED' });
|
||||
assert.throws(() => f.control.recoverOwnerStoppedUnknownIdentity({ actorId: 'owner', areaId: f.areaId,
|
||||
expectedIdentity: {}, evidence: issued.evidence }), { code: 'RUNTIME_RECOVERY_SNAPSHOT_REQUIRED' });
|
||||
assert.equal(f.control.getAgent(f.areaId).state, 'working');
|
||||
} finally { f.control.close(); }
|
||||
});
|
||||
|
||||
test('SCN-runtime-turn-guard: overlapping executors refuse even if an agent reports idle; terminal turns release their own guard', () => {
|
||||
const f = fixture();
|
||||
try {
|
||||
let launches = 0;
|
||||
withRuntimeTurnGuard(f.control, f.areaId, () => {
|
||||
launches++;
|
||||
f.control.reportStatus({ areaId: f.areaId, state: 'idle', summary: 'Agent reported idle before physical exit' });
|
||||
assert.throws(() => withRuntimeTurnGuard(f.control, f.areaId, () => { launches++; }), { code: 'RUNTIME_TURN_ACTIVE' });
|
||||
});
|
||||
withRuntimeTurnGuard(f.control, f.areaId, () => { launches++; });
|
||||
assert.equal(launches, 2);
|
||||
} finally { f.control.close(); }
|
||||
});
|
||||
|
||||
test('SCN-runtime-continuation: explicit enable, event deduplication, daily cap and pause govern authorized turns', async () => {
|
||||
const f = fixture();
|
||||
const backlog = { list: () => [] };
|
||||
const at = new Date('2026-10-03T00:00:00Z');
|
||||
try {
|
||||
f.control.reportStatus({ areaId: f.areaId, state: 'idle', summary: 'Physical turn completed' });
|
||||
f.control.sendOwnerRequest({ actorId: 'owner', recipient: f.areaId, body: 'Continue assigned work' });
|
||||
assert.equal(planContinuation(f.control, backlog, { at }).reason, 'paused');
|
||||
assert.throws(() => configureContinuation(f.control, { actorId: f.areaId, enabled: true }), { code: 'OWNER_AUTHORITY_REQUIRED' });
|
||||
configureContinuation(f.control, { actorId: 'owner', enabled: true, maxTurnsPerDay: 1 });
|
||||
assert.equal(planContinuation(f.control, backlog, { at }).candidates.length, 1);
|
||||
let calls = 0;
|
||||
const run = () => runContinuation(f.control, backlog, { actorId: 'owner', at, launch: async area => { assert.equal(area, f.areaId); calls++; return 0; } });
|
||||
assert.equal((await run()).launched.length, 1);
|
||||
assert.equal((await run()).launched.length, 0);
|
||||
f.control.sendOwnerRequest({ actorId: 'owner', recipient: f.areaId, body: 'New work' });
|
||||
assert.equal(planContinuation(f.control, backlog, { at }).reason, 'daily-turn-cap');
|
||||
configureContinuation(f.control, { actorId: 'owner', enabled: true, maxTurnsPerDay: 2 });
|
||||
assert.equal(planContinuation(f.control, backlog, { at }).candidates.length, 1);
|
||||
configureContinuation(f.control, { actorId: 'owner', enabled: false, maxTurnsPerDay: 2 });
|
||||
assert.equal((await run()).launched.length, 0);
|
||||
assert.equal(calls, 1);
|
||||
} finally { f.control.close(); }
|
||||
});
|
||||
|
||||
test('SCN-runtime-continuation-signals: acknowledgements do not wake; eligible turns start concurrently within the limit', async () => {
|
||||
const f = fixture();
|
||||
try {
|
||||
f.control.reportStatus({ areaId: f.areaId, state: 'idle', summary: 'Idle' });
|
||||
const messages = [];
|
||||
for (const recipient of [f.areaId, 'session-manager']) {
|
||||
messages.push(f.control.sendOwnerRequest({ actorId: 'owner', recipient, body: 'Review current work' }));
|
||||
f.control.sendOwnerRequest({ actorId: 'owner', recipient, body: 'Second handoff' });
|
||||
}
|
||||
configureContinuation(f.control, { actorId: 'owner', enabled: true, maxTurnsPerDay: 4 });
|
||||
const completions = [];
|
||||
const running = runContinuation(f.control, { list: () => [] }, { actorId: 'owner', limit: 2,
|
||||
launch: () => new Promise(resolve => completions.push(resolve)) });
|
||||
assert.equal(completions.length, 2, 'Both starts occur without waiting for the first provider to finish');
|
||||
assert.equal((await runContinuation(f.control, { list: () => [] }, { actorId: 'owner', launch: () => 0 })).reason,
|
||||
'controller-active-or-unreconciled');
|
||||
completions.forEach(resolve => resolve(0));
|
||||
assert.equal((await running).launched.length, 2);
|
||||
for (const message of messages) f.control.ackMessage({ recipient: message.recipient, messageId: message.id });
|
||||
assert.equal(planContinuation(f.control, { list: () => [] }).candidates.length, 0);
|
||||
} finally { f.control.close(); }
|
||||
});
|
||||
|
||||
test('SCN-runtime-continuation-pause: a pause during one turn prevents another queued turn', async () => {
|
||||
const f = fixture();
|
||||
try {
|
||||
f.control.reportStatus({ areaId: f.areaId, state: 'idle', summary: 'Idle' });
|
||||
for (const recipient of [f.areaId, 'session-manager']) f.control.sendOwnerRequest({ actorId: 'owner', recipient, body: 'Review current work' });
|
||||
configureContinuation(f.control, { actorId: 'owner', enabled: true, maxTurnsPerDay: 3 });
|
||||
const result = await runContinuation(f.control, { list: () => [] }, { actorId: 'owner',
|
||||
launch: async () => { configureContinuation(f.control, { actorId: 'owner', enabled: false, maxTurnsPerDay: 3 }); return 0; } });
|
||||
assert.equal(result.launched.length, 1);
|
||||
assert.equal(planContinuation(f.control, { list: () => [] }).reason, 'paused');
|
||||
} finally { f.control.close(); }
|
||||
});
|
||||
|
||||
test('SCN-legacy-runtime-recovery-project: evidence cannot be applied to a different checkout with cloned project/session fields', () => {
|
||||
const first = fixture();
|
||||
const second = fixture();
|
||||
try {
|
||||
assert.throws(() => second.control.recoverOwnerStoppedUnknownIdentity({ actorId: 'owner', areaId: second.areaId,
|
||||
expectedIdentity: second.identity, evidence: first.issue().evidence }), { code: 'RUNTIME_RECOVERY_UNVERIFIED' });
|
||||
assert.equal(second.control.getAgent(second.areaId).state, 'working');
|
||||
} finally { first.control.close(); second.control.close(); }
|
||||
});
|
||||
@@ -259,7 +259,7 @@ test('SCN-codex-mcp-runtime-dir: only a valid Linux parent runtime directory is
|
||||
areaId: context.worker, runtimeSessionId: 'thread-runtime-dir', worktree: '/tmp/codex-runtime-dir',
|
||||
message: 'Resume.', mcp,
|
||||
});
|
||||
const expected = `mcp_servers.${mcp.name}.env_vars=[\"XDG_RUNTIME_DIR\"]`;
|
||||
const expected = `mcp_servers.${mcp.name}.env_vars=["XDG_RUNTIME_DIR"]`;
|
||||
for (const launch of [create, resume]) {
|
||||
assert.equal(launch.launch.args.includes(expected), true);
|
||||
assert.equal(launch.launch.args.some((arg) => /env_vars=/.test(arg) && arg !== expected), false);
|
||||
|
||||
Reference in new issue
Block a user