mirror of
https://github.com/PR0M3TH3AN/TORCH.git
synced 2026-11-06 04:42:24 +00:00
runtime: recover stopped legacy sessions and add bounded continuation
This commit is contained in:
1 parent
25a3c7c1b5
commit
11dabd2450
11 files changed
+662
-1
No files matched your search
@@ -2,6 +2,26 @@
|
||||
|
||||
Repository state outranks conversation memory. Query ownership before crossing a domain boundary.
|
||||
|
||||
## Owner-requested restart and continuation repair (2026-10-03 UTC)
|
||||
|
||||
The owner said "yeah fix the issue" after the arbiter verified that the fleet
|
||||
had stopped making progress. The arbiter implemented a schema-compatible,
|
||||
owner-only manual legacy recovery and a bounded deterministic continuation pilot.
|
||||
These are source changes pending independent QA and installed-release acceptance,
|
||||
not completed umbrella tasks. Read
|
||||
docs/spec_changes/2026-10-03-legacy-runtime-recovery.md. The arbiter may apply exact
|
||||
tested stopped-invocation recovery and resume the existing seven identities with
|
||||
the source CLI. Preserve old unknown/failure receipts and existing unique commits.
|
||||
|
||||
The source controller may launch only the explicitly approved existing roster,
|
||||
under its daily turn cap, event deduplication and per-identity physical turn guard.
|
||||
It does not grant agents permission to start providers or bypass budgets, resource
|
||||
leases, approval boundaries or task ownership. Existing legacy manager-wake
|
||||
configuration stays disabled; this source pilot is a distinct, bounded mechanism.
|
||||
Manager should process current inbox and route real next work rather than leaving
|
||||
it at a proposal-only loop. QA may independently review the exact new source and
|
||||
scenario packet now; no fabricated QA acceptance is implied by owner implementation.
|
||||
|
||||
## Owner priority update: reliability before visual polish (2026-10-03 UTC)
|
||||
|
||||
The owner requested that TORCH's operating defects be fixed first so the fleet
|
||||
|
||||
@@ -2811,3 +2811,60 @@ test_integrity_note:
|
||||
did_relax_any_assertion: false
|
||||
if_true_explain_spec_basis: ""
|
||||
```
|
||||
# Owner-requested legacy recovery and bounded continuation (2026-10-03)
|
||||
|
||||
Implementation and behavioral basis:
|
||||
[legacy runtime recovery](spec_changes/2026-10-03-legacy-runtime-recovery.md).
|
||||
Existing assertions are unchanged; the new scenarios are in
|
||||
`test/fleet/legacy-runtime-recovery.test.mjs`.
|
||||
|
||||
```yaml
|
||||
test_integrity_note:
|
||||
change_type: new_tests
|
||||
scenarios:
|
||||
- id: SCN-legacy-runtime-recovery
|
||||
given: "A stopped exact invocation and retained unknown identity"
|
||||
when: "The owner recovers presence"
|
||||
then: "One owner audit, offline presence and preserved task/session/unknown"
|
||||
- id: SCN-legacy-runtime-recovery-refusal
|
||||
given: "Live or mismatched invocation/process evidence"
|
||||
when: "Proof is requested"
|
||||
then: "Recovery refuses without mutation"
|
||||
- id: SCN-legacy-runtime-recovery-cas
|
||||
given: "Stale, expired, forged or newly live evidence"
|
||||
when: "Recovery applies"
|
||||
then: "Transaction refuses without success audit"
|
||||
- id: SCN-legacy-runtime-recovery-authority
|
||||
given: "A specialist or incomplete expected snapshot"
|
||||
when: "Owner recovery is attempted"
|
||||
then: "Authority/snapshot validation refuses"
|
||||
- id: SCN-runtime-turn-guard
|
||||
given: "A physically live executor reporting idle"
|
||||
when: "A second turn tries to start"
|
||||
then: "The executor mutex refuses overlap until terminal release"
|
||||
- id: SCN-runtime-continuation
|
||||
given: "Owner-enabled roster and a new durable signal"
|
||||
when: "Turns repeat, hit a cap or are paused"
|
||||
then: "Only new eligible signals launch within the owner daily cap"
|
||||
- id: SCN-runtime-continuation-pause
|
||||
given: "Two eligible turns"
|
||||
when: "The owner pauses during the first"
|
||||
then: "The second does not start and pause is retained"
|
||||
- id: SCN-legacy-runtime-recovery-project
|
||||
given: "Cloned project/session metadata in distinct checkouts"
|
||||
when: "Proof crosses checkout boundaries"
|
||||
then: "Recovery refuses and leaves the other identity untouched"
|
||||
observable_outcomes:
|
||||
- "Persisted identity, task/session, owner audit, launch count and refusal codes"
|
||||
determinism_controls:
|
||||
- "Hermetic repositories, fixed clock and OS-boundary fixtures; no live provider"
|
||||
anti_cheat_rationale:
|
||||
prevents:
|
||||
- "Boolean stopped proof or stop-as-PASS"
|
||||
- "Stale/cross-project mutations and peer impersonation"
|
||||
- "Duplicate physical turns or unchanged-signal busy loops"
|
||||
- "Losing pause or refunding interrupted attempts"
|
||||
relaxation:
|
||||
did_relax_any_assertion: false
|
||||
if_true_explain_spec_basis: ""
|
||||
```
|
||||
@@ -22,6 +22,14 @@ states, dependencies and evidence. Blocked remains blocked; automatic wakes stay
|
||||
disabled until qualified and explicitly activated. Older pause/install/status
|
||||
statements below are historical snapshots, not current runtime evidence.
|
||||
|
||||
Owner-requested recovery/continuation implementation is described in
|
||||
[the source pilot specification](spec_changes/2026-10-03-legacy-runtime-recovery.md).
|
||||
Do not close the runtime-recovery, persistent-operations or event-wake umbrellas
|
||||
from focused tests alone. Release/Fleet Operations still need installer-owned
|
||||
permanent timer provisioning/removal, installed artifact qualification and restart
|
||||
recovery of retained crashed turn/controller guards. The transient host pilot does
|
||||
not survive reboot, and its daily turn cap is not a token/dollar usage governor.
|
||||
|
||||
## Priority UI cycle: compact cards and agent Chat
|
||||
|
||||
Owner-approved [Console UI spec](CONSOLE_CHAT_AND_COMPACT_CARDS_SPEC.md).
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
# Manual owner recovery for retained unknown executor outcomes
|
||||
|
||||
The owner requested fixing the stalled fleet. Legacy runtime records lacked a
|
||||
recovery API: three physically stopped executors remained `working/unknown` and
|
||||
correctly refused duplicate launches. This change adds an owner-only, previewed
|
||||
`recover-runtime` operation without schema changes or automatic recovery.
|
||||
|
||||
The owner explicitly selects a project-scoped systemd service invocation. Its
|
||||
working directory and exact area argument must match this installation; its
|
||||
terminal timestamp must match the retained identity update within five seconds
|
||||
(systemd displays timestamps with one-second precision). Both service PIDs must
|
||||
be zero, its cgroup absent, and no visible same-user native process may reference
|
||||
the session or registered workspace. Physical state is checked again at apply.
|
||||
Unreadable process visibility, an active service, foreign invocation or changed
|
||||
identity refuses recovery. This is a **manual legacy operator binding**, not
|
||||
cryptographic session provenance or protection against arbitrary same-user code.
|
||||
|
||||
Evidence is process-local, branded, single-use and expires after sixty seconds.
|
||||
The service transaction compares the exact state/runtime/session/update snapshot,
|
||||
sets presence to offline and writes an owner-attributed audit. Session, current
|
||||
task and the unknown outcome are preserved. No task, test or launch is completed;
|
||||
no process is killed, lease removed, schema migrated or provider started.
|
||||
|
||||
Preview and apply use the same implementation. Apply additionally requires the
|
||||
session and update timestamp from the preview. Hosts/providers without verifiable
|
||||
Linux/systemd evidence refuse this legacy recovery; no Boolean fallback exists.
|
||||
|
||||
## Bounded continuation pilot
|
||||
|
||||
`torch continuation enable --max-turns-per-day 12 --yes` approves only the
|
||||
currently activated identity roster. `status` previews eligible identities;
|
||||
`tick --limit 3 --yes` runs at most three normal guarded `up` operations. New
|
||||
unacknowledged messages or assigned/in-progress task revision changes supply the
|
||||
signal; an unchanged signal is not retried. Active/unknown identities, held turn
|
||||
guards, detach, disabled policy and the daily UTC attempt cap prevent launch.
|
||||
Attempts are charged before invocation, including failed/interrupted attempts.
|
||||
The cap limits turns, **not dollars or token consumption**. Unknown quota remains
|
||||
unknown. This controller makes no product/approval/assignment decisions.
|
||||
|
||||
`torch continuation pause --yes` prevents subsequent launches. Policy and attempt
|
||||
ledger are separate atomic files so an in-flight turn cannot overwrite a pause.
|
||||
Pause does not kill an already-running turn. A controller lock serializes ticks;
|
||||
a crashed lock is retained rather than expired automatically. Each executor has
|
||||
a separate physical turn guard held through classification and final presence,
|
||||
so an agent reporting idle cannot permit overlapping native turns.
|
||||
|
||||
The first host pilot uses an explicitly installed transient user-systemd timer.
|
||||
It survives this chat ending, but not a host reboot. Permanent installer-owned
|
||||
provisioning/removal and installed artifact qualification remain existing backlog
|
||||
work; this is not a claim that the alpha.3 installed package has been upgraded.
|
||||
|
||||
```yaml
|
||||
test_integrity_note:
|
||||
change_type: new_tests
|
||||
scenarios:
|
||||
- id: SCN-legacy-runtime-recovery
|
||||
given: "An exact retained working/unknown identity and stopped project invocation"
|
||||
when: "The owner previews and applies recovery"
|
||||
then: "Presence becomes offline once; session, task and unknown outcome persist with one owner audit"
|
||||
- id: SCN-legacy-runtime-recovery-refusal
|
||||
given: "Live, foreign, mismatched or incomplete stopped-executor evidence"
|
||||
when: "Recovery evidence is requested"
|
||||
then: "Recovery refuses without identity mutation"
|
||||
- id: SCN-legacy-runtime-recovery-cas
|
||||
given: "Issued proof followed by a changed identity, live process, expiration or forgery"
|
||||
when: "Recovery is applied"
|
||||
then: "Transactional recovery refuses and creates no success audit"
|
||||
- id: SCN-legacy-runtime-recovery-authority
|
||||
given: "A specialist actor or incomplete expected snapshot"
|
||||
when: "Owner recovery is requested"
|
||||
then: "Authority or snapshot validation refuses"
|
||||
- id: SCN-runtime-turn-guard
|
||||
given: "A physical executor holds an identity guard and reports idle early"
|
||||
when: "Another executor attempts the same identity"
|
||||
then: "It refuses until the first executor finishes and releases its own guard"
|
||||
- id: SCN-runtime-continuation
|
||||
given: "An explicitly enabled roster, a new durable signal and a fixed daily turn cap"
|
||||
when: "Continuation ticks, repeats a signal, exhausts its cap or is paused"
|
||||
then: "Only eligible new signals launch; deduplication, cap, owner authority and pause remain enforced"
|
||||
observable_outcomes:
|
||||
- "Durable presence, session and task state; owner-attributed audit; explicit refusal codes"
|
||||
determinism_controls:
|
||||
- "Hermetic Git/install fixtures, fixed clock and injected OS observation boundary"
|
||||
anti_cheat_rationale:
|
||||
prevents:
|
||||
- "Caller Boolean as stopped proof"
|
||||
- "Process stop promoted to successful task or test"
|
||||
- "Stale snapshot overwrites or silent peer impersonation"
|
||||
- "Expired, foreign or replayed proof"
|
||||
relaxation:
|
||||
did_relax_any_assertion: false
|
||||
if_true_explain_spec_basis: ""
|
||||
```
|
||||
+49
@@ -70,6 +70,8 @@ import {
|
||||
} from './design/architect.mjs';
|
||||
import { FleetEvolutionService } from './evolution/service.mjs';
|
||||
import { HierarchyEvolutionService } from './evolution/hierarchy.mjs';
|
||||
import { issueStoppedExecutorEvidence } from './runtime/stopped-executor-evidence.mjs';
|
||||
import { configureContinuation, planContinuation, runContinuation } from './runtime/continuation.mjs';
|
||||
import { ConvergenceService } from './convergence/service.mjs';
|
||||
import {
|
||||
attachForge, detachForge, forgeStatus, planForgeAttach, planForgeDetach,
|
||||
@@ -95,6 +97,8 @@ Usage:
|
||||
torch install --restore [--dry-run] [--yes] [--json]
|
||||
torch setup [--parent <path>] [--dry-run] [--yes] [--json]
|
||||
torch up [--fresh] [--only <id,id>] [--dry-run] [--yes] [--json]
|
||||
torch recover-runtime --area <id> --unit <service> --invocation <id> [--expected-session <id> --expected-updated-at <time> --yes] [--json]
|
||||
torch continuation <status|enable|pause|tick> [--max-turns-per-day <1-48>] [--limit <1-3>] [--yes] [--json]
|
||||
torch down [--dry-run] [--yes] [--json]
|
||||
torch capture [--json]
|
||||
torch detach [--dry-run] [--yes] [--json]
|
||||
@@ -576,6 +580,51 @@ export async function runCli(argv = process.argv.slice(2), {
|
||||
process.stdout.write(HELP);
|
||||
return 0;
|
||||
}
|
||||
if (command === 'continuation') {
|
||||
const control = openControlPlane({ repositoryRoot: optionValue(argv, '--repo') ?? cwd, env });
|
||||
try {
|
||||
const operation = argv[1] ?? 'status';
|
||||
const backlog = new BacklogService({ repositoryRoot: control.repositoryRoot, controlPlane: control });
|
||||
if (operation === 'status' || !argv.includes('--yes') || argv.includes('--dry-run')) {
|
||||
print(planContinuation(control, backlog), { json }); return 0;
|
||||
}
|
||||
if (operation === 'enable' || operation === 'pause') {
|
||||
print(configureContinuation(control, { actorId: 'owner', enabled: operation === 'enable',
|
||||
maxTurnsPerDay: Number(optionValue(argv, '--max-turns-per-day') ?? 12) }), { json }); return 0;
|
||||
}
|
||||
if (operation !== 'tick') throw new TorchError('Unknown continuation operation.', { code: 'UNKNOWN_COMMAND' });
|
||||
print(await runContinuation(control, backlog, { actorId: 'owner', limit: Number(optionValue(argv, '--limit') ?? 3),
|
||||
launch: areaId => {
|
||||
const result = spawn(process.execPath, [fileURLToPath(new URL('../bin/torch.mjs', import.meta.url)),
|
||||
'up', '--only', areaId, '--yes', '--json'], { cwd: control.repositoryRoot, env,
|
||||
encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], maxBuffer: 256 * 1024 });
|
||||
return result.error || result.signal ? null : result.status;
|
||||
} }), { json });
|
||||
return 0;
|
||||
} finally { control.close(); }
|
||||
}
|
||||
if (command === 'recover-runtime') {
|
||||
const control = openControlPlane({ repositoryRoot: optionValue(argv, '--repo') ?? cwd, env });
|
||||
try {
|
||||
const areaId = optionValue(argv, '--area');
|
||||
const issued = issueStoppedExecutorEvidence({ controlPlane: control, actorId: 'owner', areaId,
|
||||
unit: optionValue(argv, '--unit'), invocationId: optionValue(argv, '--invocation') });
|
||||
if (!argv.includes('--yes') || argv.includes('--dry-run')) {
|
||||
print({ ...issued.plan, requiresApproval: true }, { json });
|
||||
return 0;
|
||||
}
|
||||
const expected = issued.plan.expectedIdentity;
|
||||
if (optionValue(argv, '--expected-session') !== expected.runtimeSessionId
|
||||
|| optionValue(argv, '--expected-updated-at') !== expected.updatedAt) {
|
||||
throw new TorchError('Apply requires the exact session and updated timestamp from the recovery preview.', {
|
||||
code: 'RUNTIME_RECOVERY_SNAPSHOT_CONFLICT',
|
||||
});
|
||||
}
|
||||
print(control.recoverOwnerStoppedUnknownIdentity({ actorId: 'owner', areaId,
|
||||
expectedIdentity: expected, evidence: issued.evidence }), { json });
|
||||
return 0;
|
||||
} finally { control.close(); }
|
||||
}
|
||||
if (command === 'runtimes') {
|
||||
const operation = argv[1] ?? 'list';
|
||||
if (operation === 'update') {
|
||||
|
||||
@@ -7,6 +7,7 @@ import { readInstallManifest } from '../kernel/install.mjs';
|
||||
import { projectStatePath } from '../kernel/paths.mjs';
|
||||
import { loadProjectConfig } from '../kernel/config.mjs';
|
||||
import { organizationGraphFromConfig } from '../kernel/organization.mjs';
|
||||
import { consumeStoppedExecutorEvidence } from '../runtime/stopped-executor-evidence.mjs';
|
||||
|
||||
export const PRESENCE_STATES = Object.freeze([
|
||||
'starting', 'working', 'waiting', 'idle', 'stale', 'stopping', 'offline',
|
||||
@@ -459,6 +460,36 @@ export class ControlPlane {
|
||||
return { messageId: id, recipient: recipientId, acknowledgedAt };
|
||||
}
|
||||
|
||||
recoverOwnerStoppedUnknownIdentity({ actorId, areaId, expectedIdentity, evidence } = {}) {
|
||||
const owner = this.assertOwnerActor(actorId);
|
||||
const id = this.assertIdentity(areaId);
|
||||
if (!expectedIdentity || ['state', 'runtime', 'runtimeSessionId', 'updatedAt'].some(key => !expectedIdentity[key])) {
|
||||
throw new TorchError('Recovery requires the complete previewed identity snapshot.', { code: 'RUNTIME_RECOVERY_SNAPSHOT_REQUIRED' });
|
||||
}
|
||||
this.database.exec('BEGIN IMMEDIATE');
|
||||
try {
|
||||
const current = this.identity(id);
|
||||
if (current.state !== 'working' || !current.summary?.includes('Runtime executor completion is unknown')
|
||||
|| ['state', 'runtime', 'runtimeSessionId', 'updatedAt'].some(key => current[key] !== expectedIdentity[key])) {
|
||||
throw new TorchError('Identity changed since recovery was reviewed.', { code: 'RUNTIME_RECOVERY_SNAPSHOT_CONFLICT' });
|
||||
}
|
||||
const facts = consumeStoppedExecutorEvidence(evidence, { projectId: this.projectId, repositoryRoot: this.repositoryRoot, identity: current });
|
||||
const now = this.clock().toISOString();
|
||||
const result = this.database.prepare(`UPDATE identities SET state = 'offline', summary = ?, heartbeat_at = ?, updated_at = ?
|
||||
WHERE area_id = ? AND state = ? AND runtime = ? AND runtime_session_id = ? AND updated_at = ?`)
|
||||
.run('Owner verified stopped legacy executor; prior outcome remains unknown.', now, now,
|
||||
id, current.state, current.runtime, current.runtimeSessionId, current.updatedAt);
|
||||
if (result.changes !== 1) throw new TorchError('Recovery lost its identity snapshot.', { code: 'RUNTIME_RECOVERY_SNAPSHOT_CONFLICT' });
|
||||
const audit = this.auditOwnerAction({ actorId: owner, operation: 'runtime.legacy-unknown.recover-stopped',
|
||||
entityType: 'identity', entityId: id, details: { previousIdentity: expectedIdentity, ...facts } });
|
||||
this.database.exec('COMMIT');
|
||||
return { identity: this.identity(id), auditId: audit.id, priorOutcome: 'unknown', sessionsStarted: false, mutationPerformed: true };
|
||||
} catch (error) {
|
||||
this.database.exec('ROLLBACK');
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
reportStatus({ areaId, state, summary, runtime, runtimeSessionId, task } = {}) {
|
||||
const id = this.assertIdentity(areaId);
|
||||
const normalizedState = requiredText(state, 'state');
|
||||
@@ -751,7 +782,7 @@ export class ControlPlane {
|
||||
|
||||
auditOwnerAction({ actorId, operation, entityType, entityId = null, details = null } = {}) {
|
||||
const actor = this.assertOwnerActor(actorId);
|
||||
if (!['artifact.feedback', 'backlog.priority', 'backlog.create', 'agent.request', 'approval.decide', 'approval.notify', 'forge.sync', 'canonical.fetch', 'profile.change', 'schedule.launcher.install', 'schedule.launcher.reconcile', 'schedule.wake.recover', 'hierarchy.conclude-pilot', 'check.recover-prepared'].includes(operation)) {
|
||||
if (!['artifact.feedback', 'backlog.priority', 'backlog.create', 'agent.request', 'approval.decide', 'approval.notify', 'forge.sync', 'canonical.fetch', 'profile.change', 'schedule.launcher.install', 'schedule.launcher.reconcile', 'schedule.wake.recover', 'hierarchy.conclude-pilot', 'check.recover-prepared', 'runtime.legacy-unknown.recover-stopped', 'runtime.continuation.configure'].includes(operation)) {
|
||||
throw new TorchError('This owner audit surface does not authorize the requested operation', {
|
||||
code: 'OWNER_OPERATION_UNSUPPORTED', details: { operation },
|
||||
});
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
import { createHash, randomUUID } from 'node:crypto';
|
||||
import { existsSync, mkdirSync, readFileSync, renameSync, rmdirSync, writeFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { TorchError } from '../kernel/errors.mjs';
|
||||
|
||||
function statePath(control) { return join(control.stateRoot, 'continuation.json'); }
|
||||
function read(control) {
|
||||
const policyPath = join(control.stateRoot, 'continuation-policy.json');
|
||||
const state = { enabled: false, maxTurnsPerDay: 12, areas: [], day: null, attempts: 0, handled: {},
|
||||
...(existsSync(statePath(control)) ? JSON.parse(readFileSync(statePath(control), 'utf8')) : {}),
|
||||
...(existsSync(policyPath) ? JSON.parse(readFileSync(policyPath, 'utf8')) : {}) };
|
||||
if (typeof state.enabled !== 'boolean' || !Number.isInteger(state.maxTurnsPerDay)
|
||||
|| state.maxTurnsPerDay < 1 || state.maxTurnsPerDay > 48 || !Array.isArray(state.areas)
|
||||
|| !Number.isInteger(state.attempts) || state.attempts < 0 || !state.handled || typeof state.handled !== 'object') {
|
||||
throw new TorchError('Continuation policy or attempt ledger is invalid; launch refused.', { code: 'CONTINUATION_POLICY_INVALID' });
|
||||
}
|
||||
return state;
|
||||
}
|
||||
function save(control, state, policy = false) {
|
||||
const path = policy ? join(control.stateRoot, 'continuation-policy.json') : statePath(control);
|
||||
const temporary = `${path}.${randomUUID()}.tmp`;
|
||||
const value = policy ? { enabled: state.enabled, maxTurnsPerDay: state.maxTurnsPerDay, areas: state.areas }
|
||||
: { day: state.day, attempts: state.attempts, handled: state.handled };
|
||||
writeFileSync(temporary, JSON.stringify(value), { flag: 'wx', mode: 0o600 });
|
||||
renameSync(temporary, path);
|
||||
}
|
||||
|
||||
export function configureContinuation(control, { actorId, enabled, maxTurnsPerDay = 12 } = {}) {
|
||||
control.assertOwnerActor(actorId);
|
||||
if (typeof enabled !== 'boolean' || !Number.isInteger(maxTurnsPerDay) || maxTurnsPerDay < 1 || maxTurnsPerDay > 48) {
|
||||
throw new TorchError('Continuation requires an explicit enable/pause and 1–48 daily turns.', { code: 'CONTINUATION_POLICY_INVALID' });
|
||||
}
|
||||
const state = { ...read(control), enabled, maxTurnsPerDay, areas: control.listAgents().map(a => a.areaId) };
|
||||
save(control, state, true);
|
||||
control.auditOwnerAction({ actorId, operation: 'runtime.continuation.configure', entityType: 'project', entityId: control.projectId,
|
||||
details: { enabled, maxTurnsPerDay, areas: state.areas } });
|
||||
return { enabled, maxTurnsPerDay, areas: state.areas, sessionsStarted: false };
|
||||
}
|
||||
|
||||
export function planContinuation(control, backlog, { at = new Date(), limit = 3 } = {}) {
|
||||
const state = read(control);
|
||||
const day = at.toISOString().slice(0, 10);
|
||||
const attempts = state.day === day ? state.attempts : 0;
|
||||
if (!Number.isInteger(limit) || limit < 1 || limit > 3) throw new TorchError('Continuation limit must be 1–3.', { code: 'CONTINUATION_POLICY_INVALID' });
|
||||
const metadata = JSON.parse(readFileSync(join(control.stateRoot, 'project.json'), 'utf8'));
|
||||
if (!state.enabled || metadata.detachedAt || attempts >= state.maxTurnsPerDay) {
|
||||
return { enabled: state.enabled, reason: metadata.detachedAt ? 'detached' : !state.enabled ? 'paused' : 'daily-turn-cap', candidates: [], attempts };
|
||||
}
|
||||
const tasks = backlog.list();
|
||||
const candidates = [];
|
||||
for (const areaId of state.areas) {
|
||||
const agent = control.getAgent(areaId);
|
||||
if (!['idle', 'offline', 'waiting'].includes(agent.state)
|
||||
|| existsSync(join(control.stateRoot, 'sessions', 'turn-guards', areaId))) continue;
|
||||
const messages = control.readMessages({ recipient: areaId, unacknowledgedOnly: true, limit: 1000 });
|
||||
const active = tasks.filter(t => t.owner === areaId && ['assigned', 'in_progress'].includes(t.state));
|
||||
if (!messages.length && !active.length) continue;
|
||||
const fingerprint = createHash('sha256').update(JSON.stringify({ messages: messages.map(m => m.id), tasks: active.map(t => [t.id, t.revision]) })).digest('hex');
|
||||
if (state.handled[areaId] === fingerprint) continue;
|
||||
candidates.push({ areaId, fingerprint, messageCount: messages.length, taskIds: active.map(t => t.id) });
|
||||
}
|
||||
candidates.sort((a, b) => Number(b.areaId === 'session-manager') - Number(a.areaId === 'session-manager'));
|
||||
return { enabled: true, day, attempts, remainingTurns: state.maxTurnsPerDay - attempts,
|
||||
candidates: candidates.slice(0, Math.min(limit, state.maxTurnsPerDay - attempts)), mutationPerformed: false };
|
||||
}
|
||||
|
||||
export async function runContinuation(control, backlog, { actorId, launch, at, clock = () => at ?? new Date(), limit = 3 } = {}) {
|
||||
control.assertOwnerActor(actorId);
|
||||
if (typeof launch !== 'function') throw new TorchError('Continuation needs an authorized launcher.', { code: 'RUNTIME_EXECUTION_NOT_AUTHORIZED' });
|
||||
const lock = join(control.stateRoot, 'continuation.lock');
|
||||
try { mkdirSync(lock); } catch (error) {
|
||||
if (error.code === 'EEXIST') return { reason: 'controller-active-or-unreconciled', launched: [] };
|
||||
throw error;
|
||||
}
|
||||
try {
|
||||
const plan = planContinuation(control, backlog, { at: clock(), limit });
|
||||
const launched = [];
|
||||
for (const candidate of plan.candidates) {
|
||||
// Fresh plan and policy check before each launch; a concurrent pause wins.
|
||||
const current = planContinuation(control, backlog, { at: clock(), limit: 3 });
|
||||
if (!current.candidates.some(c => c.areaId === candidate.areaId && c.fingerprint === candidate.fingerprint)) continue;
|
||||
const state = read(control);
|
||||
if (state.day !== current.day) { state.day = current.day; state.attempts = 0; }
|
||||
state.attempts++;
|
||||
state.handled[candidate.areaId] = candidate.fingerprint;
|
||||
save(control, state); // Charge/reserve before launch; interrupted attempts are never refunded/retried silently.
|
||||
const status = await launch(candidate.areaId);
|
||||
launched.push({ areaId: candidate.areaId, status });
|
||||
}
|
||||
return { launched, reason: plan.reason ?? null };
|
||||
} finally { rmdirSync(lock); }
|
||||
}
|
||||
@@ -11,6 +11,7 @@ import {
|
||||
costCeilingPlanIssues, profileCapabilityIssues, resolveLaunchPolicy, validateRuntimeAdapter,
|
||||
} from '../adapters/runtime.mjs';
|
||||
import { hierarchyOrder } from './hierarchy-order.mjs';
|
||||
import { withRuntimeTurnGuard } from './turn-guard.mjs';
|
||||
|
||||
const MCP_ENTRY = fileURLToPath(new URL('../mcp/server.mjs', import.meta.url));
|
||||
|
||||
@@ -570,6 +571,7 @@ export function startFleet({ plan, controlPlane, executor, adapters } = {}) {
|
||||
const runtimes = adapterMap({ adapters });
|
||||
for (const action of plan.actions) {
|
||||
controlPlane.assertIdentity(action.areaId);
|
||||
withRuntimeTurnGuard(controlPlane, action.areaId, () => {
|
||||
writeCombinedPrompt(action);
|
||||
const result = executor({
|
||||
...action.launch, areaId: action.areaId, runtime: action.runtime,
|
||||
@@ -634,6 +636,7 @@ export function startFleet({ plan, controlPlane, executor, adapters } = {}) {
|
||||
? `${action.mode} turn completed` : `${action.mode} requested`,
|
||||
});
|
||||
started.push({ areaId: action.areaId, runtimeSessionId: identity.runtimeSessionId, mode: action.mode });
|
||||
});
|
||||
}
|
||||
if (plan.stateRoot) {
|
||||
const metadata = projectMetadata(plan.stateRoot);
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { readFileSync, readdirSync, readlinkSync, statSync } from 'node:fs';
|
||||
import { basename, join, resolve } from 'node:path';
|
||||
import { TorchError } from '../kernel/errors.mjs';
|
||||
|
||||
const issued = new WeakMap();
|
||||
const TTL_MS = 60_000;
|
||||
function refuse(message) {
|
||||
throw new TorchError(message, { code: 'RUNTIME_RECOVERY_UNVERIFIED' });
|
||||
}
|
||||
|
||||
export function readStoppedUnit(unit) {
|
||||
const result = spawnSync('systemctl', ['--user', 'show', unit,
|
||||
'--property=Id,ActiveState,SubState,MainPID,ControlPID,InvocationID,ExecStart,WorkingDirectory,ControlGroup,ExecMainExitTimestamp'],
|
||||
{ encoding: 'utf8', timeout: 5000, maxBuffer: 64 * 1024 });
|
||||
if (result.error || result.signal || result.status !== 0) refuse('Cannot verify the owner-selected service invocation.');
|
||||
return Object.fromEntries(result.stdout.trim().split('\n').map(line => {
|
||||
const index = line.indexOf('=');
|
||||
return [line.slice(0, index), line.slice(index + 1)];
|
||||
}));
|
||||
}
|
||||
|
||||
export function readNativeProcesses() {
|
||||
if (process.platform !== 'linux') refuse('Legacy process inspection is supported only on Linux.');
|
||||
const processes = [];
|
||||
for (const name of readdirSync('/proc').filter(value => /^\d+$/.test(value))) {
|
||||
const directory = join('/proc', name);
|
||||
try {
|
||||
if (statSync(directory).uid !== process.getuid()) continue;
|
||||
const argv = readFileSync(join(directory, 'cmdline'), 'utf8').split('\0').filter(Boolean);
|
||||
if (!argv.length) continue;
|
||||
const native = argv.some(value => /^(?:codex|claude|pi)(?:\.js)?$/.test(basename(value)));
|
||||
if (native) processes.push({ pid: Number(name), argv, cwd: readlinkSync(join(directory, 'cwd')) });
|
||||
} catch (error) {
|
||||
if (error.code !== 'ENOENT' && error.code !== 'ESRCH') refuse('Native process visibility is incomplete; recovery refused.');
|
||||
}
|
||||
}
|
||||
return processes;
|
||||
}
|
||||
|
||||
function verify(record) {
|
||||
const unit = record.readUnit(record.unit);
|
||||
const prefix = `torch-${record.projectId}-`;
|
||||
if (typeof record.unit !== 'string' || !record.unit.startsWith(prefix) || !/^[A-Za-z0-9_.-]+\.service$/.test(record.unit)
|
||||
|| unit.Id !== record.unit || !/^[a-f0-9]{32}$/.test(unit.InvocationID ?? '')
|
||||
|| unit.InvocationID !== record.invocationId
|
||||
|| !['failed', 'inactive'].includes(unit.ActiveState)
|
||||
|| !['failed', 'dead'].includes(unit.SubState)
|
||||
|| unit.MainPID !== '0' || unit.ControlPID !== '0' || unit.ControlGroup !== ''
|
||||
|| resolve(unit.WorkingDirectory || '/') !== record.repositoryRoot) {
|
||||
refuse('The selected project service is not a verified stopped invocation.');
|
||||
}
|
||||
// Explicit legacy operator binding, not a claim of cryptographic session provenance.
|
||||
// Exact area token and terminal timestamp must agree with the retained unknown row.
|
||||
if (!(unit.ExecStart ?? '').split(/[\s;]+/).includes(record.identity.areaId)
|
||||
|| !Number.isFinite(Date.parse(unit.ExecMainExitTimestamp))
|
||||
|| Math.abs(Date.parse(unit.ExecMainExitTimestamp) - Date.parse(record.identity.updatedAt)) > 5000) {
|
||||
refuse('Service area or terminal timestamp does not match the retained unknown identity.');
|
||||
}
|
||||
for (const processRecord of record.readProcesses()) {
|
||||
if (processRecord.argv.includes(record.identity.runtimeSessionId)
|
||||
|| processRecord.argv.includes(record.worktree)
|
||||
|| resolve(processRecord.cwd) === record.worktree) {
|
||||
refuse('A native process still references the selected session or workspace.');
|
||||
}
|
||||
}
|
||||
return { unit: record.unit, invocationId: unit.InvocationID,
|
||||
terminalAt: new Date(Date.parse(unit.ExecMainExitTimestamp)).toISOString(),
|
||||
outcome: 'unknown', verification: 'owner-selected-legacy-invocation' };
|
||||
}
|
||||
|
||||
export function issueStoppedExecutorEvidence({ controlPlane, actorId, areaId, unit, invocationId,
|
||||
readUnit = readStoppedUnit, readProcesses = readNativeProcesses, now = () => Date.now() } = {}) {
|
||||
controlPlane.assertOwnerActor(actorId);
|
||||
const identity = controlPlane.getAgent(areaId);
|
||||
if (identity.state !== 'working' || !identity.runtimeSessionId || !identity.updatedAt
|
||||
|| !identity.summary?.includes('Runtime executor completion is unknown')) {
|
||||
refuse('Only an exact retained working/unknown identity can use legacy recovery.');
|
||||
}
|
||||
const worktree = controlPlane.manifest.external?.find(entry => entry.type === 'worktree' && entry.area === areaId)?.path;
|
||||
if (!worktree) refuse('No registered workspace exists for this identity.');
|
||||
const record = { identity: Object.freeze({ areaId, state: identity.state, runtime: identity.runtime,
|
||||
runtimeSessionId: identity.runtimeSessionId, updatedAt: identity.updatedAt }),
|
||||
unit, invocationId, projectId: controlPlane.projectId, repositoryRoot: resolve(controlPlane.repositoryRoot),
|
||||
worktree: resolve(worktree), readUnit, readProcesses, now, issuedAt: now() };
|
||||
const facts = verify(record);
|
||||
const evidence = Object.freeze({});
|
||||
issued.set(evidence, record);
|
||||
return { evidence, plan: { areaId, expectedIdentity: record.identity, ...facts,
|
||||
mutationPerformed: false, effect: 'Recover presence to offline; preserve task, session and unknown outcome. No provider launch.' } };
|
||||
}
|
||||
|
||||
export function consumeStoppedExecutorEvidence(evidence, { projectId, repositoryRoot, identity } = {}) {
|
||||
const record = issued.get(evidence);
|
||||
if (!record) refuse('Recovery requires fresh Runtime-issued evidence, not caller text or a Boolean.');
|
||||
issued.delete(evidence);
|
||||
if (record.projectId !== projectId || record.repositoryRoot !== resolve(repositoryRoot)
|
||||
|| record.now() < record.issuedAt
|
||||
|| record.now() - record.issuedAt > TTL_MS
|
||||
|| ['areaId', 'state', 'runtime', 'runtimeSessionId', 'updatedAt'].some(key => record.identity[key] !== identity[key])) {
|
||||
refuse('Recovery evidence is stale, expired or belongs to another identity.');
|
||||
}
|
||||
const facts = verify(record); // Recheck physical state immediately before the transactional mutation.
|
||||
return { ...facts, digest: createHash('sha256').update(JSON.stringify({ identity: record.identity, ...facts })).digest('hex') };
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import { mkdirSync, readFileSync, rmdirSync, unlinkSync, writeFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { TorchError } from '../kernel/errors.mjs';
|
||||
|
||||
// One physical executor per identity, including the interval after an agent
|
||||
// reports idle but before its harness actually exits. Crashed guards never expire.
|
||||
export function withRuntimeTurnGuard(controlPlane, areaId, run) {
|
||||
controlPlane.assertIdentity(areaId);
|
||||
if (!/^[a-z0-9][a-z0-9-]*$/.test(areaId)) throw new TorchError('Invalid runtime guard identity.', { code: 'INVALID_RUNTIME_INPUT' });
|
||||
const parent = join(controlPlane.stateRoot, 'sessions', 'turn-guards');
|
||||
mkdirSync(parent, { recursive: true });
|
||||
const directory = join(parent, areaId);
|
||||
try { mkdirSync(directory); } catch (error) {
|
||||
if (error.code === 'EEXIST') throw new TorchError('An executor already holds this identity turn guard.', {
|
||||
code: 'RUNTIME_TURN_ACTIVE', details: { areaId },
|
||||
});
|
||||
throw error;
|
||||
}
|
||||
const path = join(directory, 'owner.json');
|
||||
const nonce = randomUUID();
|
||||
writeFileSync(path, JSON.stringify({ nonce, pid: process.pid, areaId, projectId: controlPlane.projectId }), { flag: 'wx', mode: 0o600 });
|
||||
try { return run(); } finally {
|
||||
// Delete only the exact guard created by this invocation; preserve foreign/changed guards.
|
||||
const record = JSON.parse(readFileSync(path, 'utf8'));
|
||||
if (record.nonce === nonce && record.pid === process.pid) {
|
||||
unlinkSync(path);
|
||||
rmdirSync(directory);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,171 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { mkdtempSync, readFileSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import test from 'node:test';
|
||||
import { analyzeRepository } from '../../src/kernel/analyze.mjs';
|
||||
import { inspectRepository } from '../../src/kernel/git.mjs';
|
||||
import { proposeDomains } from '../../src/kernel/domains.mjs';
|
||||
import { installProject } from '../../src/kernel/install.mjs';
|
||||
import { openControlPlane } from '../../src/control-plane/service.mjs';
|
||||
import { issueStoppedExecutorEvidence } from '../../src/runtime/stopped-executor-evidence.mjs';
|
||||
import { withRuntimeTurnGuard } from '../../src/runtime/turn-guard.mjs';
|
||||
import { configureContinuation, planContinuation, runContinuation } from '../../src/runtime/continuation.mjs';
|
||||
|
||||
function fixture() {
|
||||
const root = mkdtempSync(join(tmpdir(), 'torch-legacy-recovery-'));
|
||||
for (const args of [['init', '-b', 'main'], ['config', 'user.email', '[email protected]'],
|
||||
['config', 'user.name', 'Recovery Fixture']]) execFileSync('git', args, { cwd: root, stdio: 'pipe' });
|
||||
writeFileSync(join(root, 'app.js'), 'export const ready = true;\n');
|
||||
execFileSync('git', ['add', '.'], { cwd: root });
|
||||
execFileSync('git', ['commit', '-m', 'fixture'], { cwd: root, stdio: 'pipe' });
|
||||
const repository = inspectRepository(root);
|
||||
const proposal = proposeDomains({ repository, analysis: analyzeRepository(repository) });
|
||||
proposal.review = { status: 'approved', reviewedAt: '2026-10-03T00:00:00Z', reviewedBy: 'owner', notes: [] };
|
||||
const env = { ...process.env, XDG_DATA_HOME: join(root, '.data') };
|
||||
installProject({ repository, proposal, env, projectId: 'recovery-fixture' });
|
||||
const actualPath = join(root, '.torch', 'install-manifest.json');
|
||||
const manifest = JSON.parse(readFileSync(actualPath, 'utf8'));
|
||||
const areaId = proposal.domains[0].id;
|
||||
manifest.external.push({ type: 'worktree', area: areaId, path: join(root, 'worker') });
|
||||
writeFileSync(actualPath, JSON.stringify(manifest));
|
||||
let time = Date.parse('2026-10-03T00:00:00Z');
|
||||
const control = openControlPlane({ repositoryRoot: root, env, clock: () => new Date(time) });
|
||||
control.reportStatus({ areaId, state: 'working', runtime: 'codex', runtimeSessionId: 'fixture-session',
|
||||
task: 'TASK-preserved', summary: 'Runtime executor completion is unknown; duplicate launch is blocked pending reconciliation.' });
|
||||
const identity = control.getAgent(areaId);
|
||||
const unit = `torch-${control.projectId}-worker.service`;
|
||||
const properties = { Id: unit, ActiveState: 'failed', SubState: 'failed', MainPID: '0', ControlPID: '0',
|
||||
ControlGroup: '', InvocationID: 'a'.repeat(32), ExecStart: `node /tmp/probe.mjs ${areaId} ;`,
|
||||
WorkingDirectory: root, ExecMainExitTimestamp: identity.updatedAt };
|
||||
let processes = [];
|
||||
const issue = () => issueStoppedExecutorEvidence({ controlPlane: control, actorId: 'owner', areaId, unit,
|
||||
invocationId: 'a'.repeat(32), readUnit: () => ({ ...properties }), readProcesses: () => processes, now: () => time });
|
||||
const apply = issued => control.recoverOwnerStoppedUnknownIdentity({ actorId: 'owner', areaId,
|
||||
expectedIdentity: issued.plan.expectedIdentity, evidence: issued.evidence });
|
||||
return { control, identity, areaId, properties, issue, apply,
|
||||
advance: ms => { time += ms; }, processes: value => { processes = value; } };
|
||||
}
|
||||
|
||||
test('SCN-legacy-runtime-recovery: owner recovery preserves task/session/unknown and records exactly one owner audit', () => {
|
||||
const f = fixture();
|
||||
try {
|
||||
const issued = f.issue();
|
||||
assert.equal(f.control.getAgent(f.areaId).state, 'working');
|
||||
const result = f.apply(issued);
|
||||
assert.equal(result.identity.state, 'offline');
|
||||
assert.equal(result.identity.currentTask, 'TASK-preserved');
|
||||
assert.equal(result.identity.runtimeSessionId, 'fixture-session');
|
||||
assert.equal(result.priorOutcome, 'unknown');
|
||||
assert.equal(result.sessionsStarted, false);
|
||||
const audits = f.control.readAudit({ limit: 100 }).filter(x => x.operation === 'runtime.legacy-unknown.recover-stopped');
|
||||
assert.equal(audits.length, 1);
|
||||
assert.equal(audits[0].actorId, 'owner');
|
||||
assert.equal(audits[0].details.outcome, 'unknown');
|
||||
assert.throws(() => f.apply(issued), { code: 'RUNTIME_RECOVERY_SNAPSHOT_CONFLICT' });
|
||||
} finally { f.control.close(); }
|
||||
});
|
||||
|
||||
test('SCN-legacy-runtime-recovery-refusal: live, foreign invocation, area, time and process mismatches cannot issue proof', () => {
|
||||
for (const patch of [{ MainPID: '99' }, { ControlPID: '99' }, { ActiveState: 'active' },
|
||||
{ InvocationID: 'b'.repeat(32) }, { ExecStart: 'node other-area' },
|
||||
{ ExecMainExitTimestamp: '2026-10-02T00:00:00Z' }, { ControlGroup: '/live' }]) {
|
||||
const f = fixture();
|
||||
try { Object.assign(f.properties, patch); assert.throws(f.issue, { code: 'RUNTIME_RECOVERY_UNVERIFIED' });
|
||||
assert.equal(f.control.getAgent(f.areaId).state, 'working'); } finally { f.control.close(); }
|
||||
}
|
||||
const f = fixture();
|
||||
try { f.processes([{ pid: 999, argv: ['codex', 'resume', 'fixture-session'], cwd: '/tmp' }]);
|
||||
assert.throws(f.issue, { code: 'RUNTIME_RECOVERY_UNVERIFIED' }); } finally { f.control.close(); }
|
||||
});
|
||||
|
||||
test('SCN-legacy-runtime-recovery-cas: changed snapshot, live recheck, expiry and caller-forged proof refuse without mutation', () => {
|
||||
for (const scenario of ['changed', 'live', 'expired', 'forged']) {
|
||||
const f = fixture();
|
||||
try {
|
||||
const issued = f.issue();
|
||||
if (scenario === 'changed') { f.advance(1); f.control.reportStatus({ areaId: f.areaId, state: 'working', runtimeSessionId: 'different', summary: 'Runtime executor completion is unknown' }); }
|
||||
if (scenario === 'live') f.properties.MainPID = '99';
|
||||
if (scenario === 'expired') f.advance(60_001);
|
||||
if (scenario === 'forged') issued.evidence = { stopped: true };
|
||||
assert.throws(() => f.apply(issued));
|
||||
assert.equal(f.control.getAgent(f.areaId).state, 'working');
|
||||
assert.equal(f.control.readAudit({ limit: 100 }).filter(x => x.operation === 'runtime.legacy-unknown.recover-stopped').length, 0);
|
||||
} finally { f.control.close(); }
|
||||
}
|
||||
});
|
||||
|
||||
test('SCN-legacy-runtime-recovery-authority: specialist and incomplete snapshot cannot use owner recovery', () => {
|
||||
const f = fixture();
|
||||
try {
|
||||
const issued = f.issue();
|
||||
assert.throws(() => f.control.recoverOwnerStoppedUnknownIdentity({ actorId: f.areaId, areaId: f.areaId,
|
||||
expectedIdentity: issued.plan.expectedIdentity, evidence: issued.evidence }), { code: 'OWNER_AUTHORITY_REQUIRED' });
|
||||
assert.throws(() => f.control.recoverOwnerStoppedUnknownIdentity({ actorId: 'owner', areaId: f.areaId,
|
||||
expectedIdentity: {}, evidence: issued.evidence }), { code: 'RUNTIME_RECOVERY_SNAPSHOT_REQUIRED' });
|
||||
assert.equal(f.control.getAgent(f.areaId).state, 'working');
|
||||
} finally { f.control.close(); }
|
||||
});
|
||||
|
||||
test('SCN-runtime-turn-guard: overlapping executors refuse even if an agent reports idle; terminal turns release their own guard', () => {
|
||||
const f = fixture();
|
||||
try {
|
||||
let launches = 0;
|
||||
withRuntimeTurnGuard(f.control, f.areaId, () => {
|
||||
launches++;
|
||||
f.control.reportStatus({ areaId: f.areaId, state: 'idle', summary: 'Agent reported idle before physical exit' });
|
||||
assert.throws(() => withRuntimeTurnGuard(f.control, f.areaId, () => { launches++; }), { code: 'RUNTIME_TURN_ACTIVE' });
|
||||
});
|
||||
withRuntimeTurnGuard(f.control, f.areaId, () => { launches++; });
|
||||
assert.equal(launches, 2);
|
||||
} finally { f.control.close(); }
|
||||
});
|
||||
|
||||
test('SCN-runtime-continuation: explicit enable, event deduplication, daily cap and pause govern authorized turns', async () => {
|
||||
const f = fixture();
|
||||
const backlog = { list: () => [] };
|
||||
const at = new Date('2026-10-03T00:00:00Z');
|
||||
try {
|
||||
f.control.reportStatus({ areaId: f.areaId, state: 'idle', summary: 'Physical turn completed' });
|
||||
f.control.sendOwnerRequest({ actorId: 'owner', recipient: f.areaId, body: 'Continue assigned work' });
|
||||
assert.equal(planContinuation(f.control, backlog, { at }).reason, 'paused');
|
||||
assert.throws(() => configureContinuation(f.control, { actorId: f.areaId, enabled: true }), { code: 'OWNER_AUTHORITY_REQUIRED' });
|
||||
configureContinuation(f.control, { actorId: 'owner', enabled: true, maxTurnsPerDay: 1 });
|
||||
assert.equal(planContinuation(f.control, backlog, { at }).candidates.length, 1);
|
||||
let calls = 0;
|
||||
const run = () => runContinuation(f.control, backlog, { actorId: 'owner', at, launch: async area => { assert.equal(area, f.areaId); calls++; return 0; } });
|
||||
assert.equal((await run()).launched.length, 1);
|
||||
assert.equal((await run()).launched.length, 0);
|
||||
f.control.sendOwnerRequest({ actorId: 'owner', recipient: f.areaId, body: 'New work' });
|
||||
assert.equal(planContinuation(f.control, backlog, { at }).reason, 'daily-turn-cap');
|
||||
configureContinuation(f.control, { actorId: 'owner', enabled: true, maxTurnsPerDay: 2 });
|
||||
assert.equal(planContinuation(f.control, backlog, { at }).candidates.length, 1);
|
||||
configureContinuation(f.control, { actorId: 'owner', enabled: false, maxTurnsPerDay: 2 });
|
||||
assert.equal((await run()).launched.length, 0);
|
||||
assert.equal(calls, 1);
|
||||
} finally { f.control.close(); }
|
||||
});
|
||||
|
||||
test('SCN-runtime-continuation-pause: a pause during one turn prevents another queued turn', async () => {
|
||||
const f = fixture();
|
||||
try {
|
||||
f.control.reportStatus({ areaId: f.areaId, state: 'idle', summary: 'Idle' });
|
||||
for (const recipient of [f.areaId, 'session-manager']) f.control.sendOwnerRequest({ actorId: 'owner', recipient, body: 'Review current work' });
|
||||
configureContinuation(f.control, { actorId: 'owner', enabled: true, maxTurnsPerDay: 3 });
|
||||
const result = await runContinuation(f.control, { list: () => [] }, { actorId: 'owner',
|
||||
launch: async () => { configureContinuation(f.control, { actorId: 'owner', enabled: false, maxTurnsPerDay: 3 }); return 0; } });
|
||||
assert.equal(result.launched.length, 1);
|
||||
assert.equal(planContinuation(f.control, { list: () => [] }).reason, 'paused');
|
||||
} finally { f.control.close(); }
|
||||
});
|
||||
|
||||
test('SCN-legacy-runtime-recovery-project: evidence cannot be applied to a different checkout with cloned project/session fields', () => {
|
||||
const first = fixture();
|
||||
const second = fixture();
|
||||
try {
|
||||
assert.throws(() => second.control.recoverOwnerStoppedUnknownIdentity({ actorId: 'owner', areaId: second.areaId,
|
||||
expectedIdentity: second.identity, evidence: first.issue().evidence }), { code: 'RUNTIME_RECOVERY_UNVERIFIED' });
|
||||
assert.equal(second.control.getAgent(second.areaId).state, 'working');
|
||||
} finally { first.control.close(); second.control.close(); }
|
||||
});
|
||||
Reference in new issue
Block a user