diff --git a/.torch/prompts/COMMON.md b/.torch/prompts/COMMON.md index 56986db..c5c5a41 100644 --- a/.torch/prompts/COMMON.md +++ b/.torch/prompts/COMMON.md @@ -2,6 +2,44 @@ Repository state outranks conversation memory. Query ownership before crossing a domain boundary. +## Owner-requested restart and continuation repair (2026-10-03 UTC) + +The owner said "yeah fix the issue" after the arbiter verified that the fleet +had stopped making progress. The arbiter implemented a schema-compatible, +owner-only manual legacy recovery and a bounded deterministic continuation pilot. +These are source changes pending independent QA and installed-release acceptance, +not completed umbrella tasks. Read +docs/spec_changes/2026-10-03-legacy-runtime-recovery.md. The arbiter may apply exact +tested stopped-invocation recovery and resume the existing seven identities with +the source CLI. Preserve old unknown/failure receipts and existing unique commits. + +The source controller may launch only the explicitly approved existing roster, +under its daily turn cap, event deduplication and per-identity physical turn guard. +It does not grant agents permission to start providers or bypass budgets, resource +leases, approval boundaries or task ownership. Existing legacy manager-wake +configuration stays disabled; this source pilot is a distinct, bounded mechanism. +Manager should process current inbox and route real next work rather than leaving +it at a proposal-only loop. QA may independently review the exact new source and +scenario packet now; no fabricated QA acceptance is implied by owner implementation. + +## Owner priority update: reliability before visual polish (2026-10-03 UTC) + +The owner requested that TORCH's operating defects be fixed first so the fleet +benefits from improved operation. This supersedes the UI-first ordering below, +not the existing resume authorization or safety boundaries. The authoritative +backlog priorities were changed through the revision-checked owner API. + +Sequence the smallest safe current-schema stopped-unknown recovery and CLI +qualification first; then truthful lifecycle/status and their source-safety +prerequisites; then qualified bounded continuation/event-driven wakes under +pause, budget and authority policy; then actionable operational attention. +Compact cards/detail modals and coordinator Chat remain high-priority work after +these reliability gates. Do not require the entire future company architecture +to deliver a safe minimal recovery. Reprioritization does not resolve blockers, +grant test consent, enable automatic wakes or launch agents. Preserve task owners, +dependencies and evidence; explicitly block/resequence existing assignments when +needed rather than replacing them silently or creating concurrent assignments. + ## Owner-approved fleet resume and priority polish cycle (2026-10-02) The owner explicitly said: "We got a reset. lets get torch as pollished as we can. diff --git a/docs/TEST_INTEGRITY.md b/docs/TEST_INTEGRITY.md index 2b19a3b..aacb180 100644 --- a/docs/TEST_INTEGRITY.md +++ b/docs/TEST_INTEGRITY.md @@ -2811,6 +2811,67 @@ test_integrity_note: did_relax_any_assertion: false if_true_explain_spec_basis: "" ``` +# Owner-requested legacy recovery and bounded continuation (2026-10-03) + +Implementation and behavioral basis: +[legacy runtime recovery](spec_changes/2026-10-03-legacy-runtime-recovery.md). +Existing assertions are unchanged; the new scenarios are in +`test/fleet/legacy-runtime-recovery.test.mjs`. + +```yaml +test_integrity_note: + change_type: new_tests + scenarios: + - id: SCN-legacy-runtime-recovery + given: "A stopped exact invocation and retained unknown identity" + when: "The owner recovers presence" + then: "One owner audit, offline presence and preserved task/session/unknown" + - id: SCN-legacy-runtime-recovery-refusal + given: "Live or mismatched invocation/process evidence" + when: "Proof is requested" + then: "Recovery refuses without mutation" + - id: SCN-legacy-runtime-recovery-cas + given: "Stale, expired, forged or newly live evidence" + when: "Recovery applies" + then: "Transaction refuses without success audit" + - id: SCN-legacy-runtime-recovery-authority + given: "A specialist or incomplete expected snapshot" + when: "Owner recovery is attempted" + then: "Authority/snapshot validation refuses" + - id: SCN-runtime-turn-guard + given: "A physically live executor reporting idle" + when: "A second turn tries to start" + then: "The executor mutex refuses overlap until terminal release" + - id: SCN-runtime-continuation + given: "Owner-enabled roster and a new durable signal" + when: "Turns repeat, hit a cap or are paused" + then: "Only new eligible signals launch within the owner daily cap" + - id: SCN-runtime-continuation-signals + given: "Two eligible identities with multiple unread messages" + when: "A bounded batch starts and only acknowledgements change afterward" + then: "Both starts occur concurrently, a competing controller refuses, and acknowledgements alone cause no wake" + - id: SCN-runtime-continuation-pause + given: "Two eligible turns" + when: "The owner pauses during the first" + then: "The second does not start and pause is retained" + - id: SCN-legacy-runtime-recovery-project + given: "Cloned project/session metadata in distinct checkouts" + when: "Proof crosses checkout boundaries" + then: "Recovery refuses and leaves the other identity untouched" + observable_outcomes: + - "Persisted identity, task/session, owner audit, launch count and refusal codes" + determinism_controls: + - "Hermetic repositories, fixed clock and OS-boundary fixtures; no live provider" + anti_cheat_rationale: + prevents: + - "Boolean stopped proof or stop-as-PASS" + - "Stale/cross-project mutations and peer impersonation" + - "Duplicate physical turns or unchanged-signal busy loops" + - "Losing pause or refunding interrupted attempts" + relaxation: + did_relax_any_assertion: false + if_true_explain_spec_basis: "" +``` ```yaml test_integrity_note: diff --git a/docs/TODO.md b/docs/TODO.md index 10a1d9b..e422a66 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -1,5 +1,35 @@ # TORCH remaining work +## Current owner priority: reliability first (2026-10-03 UTC) + +This ordering supersedes the UI-first ordering below. The existing authoritative +backlog, not this index, controls task state and assignment. + +1. Safe current-schema runtime recovery (`TASK-company-runtime-recovery`) and + combined CLI qualification (`TASK-pilot-cli-suite-timeout`). +2. Truthful startup/session status (`TASK-pilot-starting-presence`) and its + existing Kernel reservation/Work candidate-safety prerequisites. +3. Persistent operations and bounded event-driven continuation + (`TASK-conversation-persistent-operations`, + `TASK-conversation-event-driven-wakes`), followed by actual scheduler + qualification (`TASK-conversation-scheduler-qualification`). +4. Specific actionable operational warnings (`TASK-console-attention-actions`). +5. Compact cards/detail modals and coordinator-first Chat remain high priority. + +The first four groups and incomplete technical prerequisites are urgent; +scheduler qualification is high. These priority-only changes preserve owners, +states, dependencies and evidence. Blocked remains blocked; automatic wakes stay +disabled until qualified and explicitly activated. Older pause/install/status +statements below are historical snapshots, not current runtime evidence. + +Owner-requested recovery/continuation implementation is described in +[the source pilot specification](spec_changes/2026-10-03-legacy-runtime-recovery.md). +Do not close the runtime-recovery, persistent-operations or event-wake umbrellas +from focused tests alone. Release/Fleet Operations still need installer-owned +permanent timer provisioning/removal, installed artifact qualification and restart +recovery of retained crashed turn/controller guards. The transient host pilot does +not survive reboot, and its daily turn cap is not a token/dollar usage governor. + ## Priority UI cycle: compact cards and agent Chat Owner-approved [Console UI spec](CONSOLE_CHAT_AND_COMPACT_CARDS_SPEC.md). diff --git a/docs/spec_changes/2026-10-03-legacy-runtime-recovery.md b/docs/spec_changes/2026-10-03-legacy-runtime-recovery.md new file mode 100644 index 0000000..405af54 --- /dev/null +++ b/docs/spec_changes/2026-10-03-legacy-runtime-recovery.md @@ -0,0 +1,93 @@ +# Manual owner recovery for retained unknown executor outcomes + +The owner requested fixing the stalled fleet. Legacy runtime records lacked a +recovery API: three physically stopped executors remained `working/unknown` and +correctly refused duplicate launches. This change adds an owner-only, previewed +`recover-runtime` operation without schema changes or automatic recovery. + +The owner explicitly selects a project-scoped systemd service invocation. Its +working directory and exact area argument must match this installation; its +terminal timestamp must match the retained identity update within five seconds +(systemd displays timestamps with one-second precision). Both service PIDs must +be zero, its cgroup absent, and no visible same-user native process may reference +the session or registered workspace. Physical state is checked again at apply. +Unreadable process visibility, an active service, foreign invocation or changed +identity refuses recovery. This is a **manual legacy operator binding**, not +cryptographic session provenance or protection against arbitrary same-user code. + +Evidence is process-local, branded, single-use and expires after sixty seconds. +The service transaction compares the exact state/runtime/session/update snapshot, +sets presence to offline and writes an owner-attributed audit. Session, current +task and the unknown outcome are preserved. No task, test or launch is completed; +no process is killed, lease removed, schema migrated or provider started. + +Preview and apply use the same implementation. Apply additionally requires the +session and update timestamp from the preview. Hosts/providers without verifiable +Linux/systemd evidence refuse this legacy recovery; no Boolean fallback exists. + +## Bounded continuation pilot + +`torch continuation enable --max-turns-per-day 12 --yes` approves only the +currently activated identity roster. `status` previews eligible identities; +`tick --limit 3 --yes` runs at most three normal guarded `up` operations. New +unacknowledged messages or assigned/in-progress task revision changes supply the +signal; an unchanged signal is not retried. Active/unknown identities, held turn +guards, detach, disabled policy and the daily UTC attempt cap prevent launch. +Attempts are charged before invocation, including failed/interrupted attempts. +The cap limits turns, **not dollars or token consumption**. Unknown quota remains +unknown. This controller makes no product/approval/assignment decisions. + +`torch continuation pause --yes` prevents subsequent launches. Policy and attempt +ledger are separate atomic files so an in-flight turn cannot overwrite a pause. +Pause does not kill an already-running turn. A controller lock serializes ticks; +a crashed lock is retained rather than expired automatically. Each executor has +a separate physical turn guard held through classification and final presence, +so an agent reporting idle cannot permit overlapping native turns. + +The first host pilot uses an explicitly installed transient user-systemd timer. +It survives this chat ending, but not a host reboot. Permanent installer-owned +provisioning/removal and installed artifact qualification remain existing backlog +work; this is not a claim that the alpha.3 installed package has been upgraded. + +```yaml +test_integrity_note: + change_type: new_tests + scenarios: + - id: SCN-legacy-runtime-recovery + given: "An exact retained working/unknown identity and stopped project invocation" + when: "The owner previews and applies recovery" + then: "Presence becomes offline once; session, task and unknown outcome persist with one owner audit" + - id: SCN-legacy-runtime-recovery-refusal + given: "Live, foreign, mismatched or incomplete stopped-executor evidence" + when: "Recovery evidence is requested" + then: "Recovery refuses without identity mutation" + - id: SCN-legacy-runtime-recovery-cas + given: "Issued proof followed by a changed identity, live process, expiration or forgery" + when: "Recovery is applied" + then: "Transactional recovery refuses and creates no success audit" + - id: SCN-legacy-runtime-recovery-authority + given: "A specialist actor or incomplete expected snapshot" + when: "Owner recovery is requested" + then: "Authority or snapshot validation refuses" + - id: SCN-runtime-turn-guard + given: "A physical executor holds an identity guard and reports idle early" + when: "Another executor attempts the same identity" + then: "It refuses until the first executor finishes and releases its own guard" + - id: SCN-runtime-continuation + given: "An explicitly enabled roster, a new durable signal and a fixed daily turn cap" + when: "Continuation ticks, repeats a signal, exhausts its cap or is paused" + then: "Only eligible new signals launch; deduplication, cap, owner authority and pause remain enforced" + observable_outcomes: + - "Durable presence, session and task state; owner-attributed audit; explicit refusal codes" + determinism_controls: + - "Hermetic Git/install fixtures, fixed clock and injected OS observation boundary" + anti_cheat_rationale: + prevents: + - "Caller Boolean as stopped proof" + - "Process stop promoted to successful task or test" + - "Stale snapshot overwrites or silent peer impersonation" + - "Expired, foreign or replayed proof" + relaxation: + did_relax_any_assertion: false + if_true_explain_spec_basis: "" +``` diff --git a/reports/audit/runtime-recovery-continuation-2026-10-03.md b/reports/audit/runtime-recovery-continuation-2026-10-03.md new file mode 100644 index 0000000..81343e2 --- /dev/null +++ b/reports/audit/runtime-recovery-continuation-2026-10-03.md @@ -0,0 +1,53 @@ +# Owner repair and bounded continuation pilot + +## Candidate and verification + +Canonical development branch: `rewrite/portable-agent-fleet`. + +- `11dabd2`: exact stopped-unknown owner recovery, physical turn guard, opt-in continuation. +- `4c42715`: reviewed Runtime MCP XDG runtime-directory forwarding, cherry-picked without its unfinished broader branch. +- `f823714`: bounded parallel turns and signal watermarks that do not wake on acknowledgements alone. +- Complete suite: 266 PASS, 4 skipped, zero failures/cancellations (`/tmp/torch-continuation-final-suite.log`). Syntax check and full lint PASS. Subsequent MCP focused suite: 6 PASS after removing unnecessary string escapes, with identical assertion semantics. + +These are local/source verification results, not independent QA acceptance or an installed-release upgrade. No acceptance assertions or test timeouts were relaxed. + +## Actual recovery + +Owner-selected legacy units were verified terminal, with no live matching native process. The new public source CLI recovered these identities to offline, retaining prior unknown outcomes and runtime/session/task data: + +| Identity | Owner audit | +| --- | --- | +| owner-console | cfae5b3d-695b-4657-95f0-cdba230741db | +| qa | 158bc692-e775-49a2-987f-8d9d3e8ffb9a | +| release-self-host | 7f618cd6-86cd-471f-a5f0-d7f03f8c6fb4 | + +This is an explicit owner-selected legacy invocation binding, not cryptographic session provenance. Old failed units remain evidence. No processes were killed, task/test results forged, reservations fabricated, or schema migrations applied. + +## Actual pilot + +Project: `3dc11932-6e3c-42aa-9a8a-ca7b8f818b54`. + +Transient user timer/service: `torch-3dc11932-6e3c-42aa-9a8a-ca7b8f818b54-continuation-pilot`. + +First activation: 2026-10-03 04:00:18 UTC. Source CLI starts at most three existing approved identities concurrently, charges attempts before launch, and checks again two minutes after the previous batch exits. At verification, three attempts were charged and physical guards existed for session-manager, project-kernel and provider-runtime; their source `up` child processes were running. Daily cap: 12 turns, UTC day. This is a turn cap, NOT measured token/cost governance. No additional identities were authorized. + +Monitoring URL returned HTTP 200: . + +Pause future turns from the canonical repository: + +```sh +node bin/torch.mjs continuation pause --yes --json +``` + +Pause does not kill already-started turns. Inspect policy/eligibility with `node bin/torch.mjs continuation status --json`. The timer remains a cheap deterministic check when paused or capped. + +## Remaining qualification + +- Independent QA of this exact composed candidate and real native completion/continuation. +- Startup presence truthfulness: physical launches may still display idle until their first status report. +- Installer-owned persistent timer registration, removal, upgrade and reboot qualification. This transient timer survives chat exit, not host reboot. +- Crash reconciliation for retained controller/turn locks; no time-based expiry silently permits overlapping executors. +- Real usage governor, runtime/model/auth qualification and operational failure reporting. +- Existing umbrella tasks remain open; the previously observed CLI timeout still needs a root-cause diagnosis despite this passing run. + +Installed alpha.3 runtime and console service were not replaced. The older schedule framework's automatic model wakes remain disabled; the source continuation pilot is separately authorized. diff --git a/src/cli.mjs b/src/cli.mjs index e136d1d..6a169fc 100644 --- a/src/cli.mjs +++ b/src/cli.mjs @@ -14,7 +14,7 @@ import { startConsole } from './console/server.mjs'; import { classifyRecoverability, createLocalCanonical, planLocalCanonical } from './canonical/local.mjs'; import { openControlPlane } from './control-plane/service.mjs'; import { fetchCanonicalObjects, planCanonicalFetch } from './canonical/fetch.mjs'; -import { spawnSync } from 'node:child_process'; +import { spawn as spawnProcess, spawnSync } from 'node:child_process'; import { randomUUID } from 'node:crypto'; import { mkdirSync, mkdtempSync, renameSync, rmSync, statSync, writeFileSync, @@ -70,6 +70,8 @@ import { } from './design/architect.mjs'; import { FleetEvolutionService } from './evolution/service.mjs'; import { HierarchyEvolutionService } from './evolution/hierarchy.mjs'; +import { issueStoppedExecutorEvidence } from './runtime/stopped-executor-evidence.mjs'; +import { configureContinuation, planContinuation, runContinuation } from './runtime/continuation.mjs'; import { ConvergenceService } from './convergence/service.mjs'; import { attachForge, detachForge, forgeStatus, planForgeAttach, planForgeDetach, @@ -95,6 +97,8 @@ Usage: torch install --restore [--dry-run] [--yes] [--json] torch setup [--parent ] [--dry-run] [--yes] [--json] torch up [--fresh] [--only ] [--dry-run] [--yes] [--json] + torch recover-runtime --area --unit --invocation [--expected-session --expected-updated-at