From 25a3c7c1b5ebbd91c1122d6bde884c1ae78ac582 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN Date: Fri, 2 Oct 2026 22:27:25 -0400 Subject: [PATCH 1/5] fleet: prioritize operational reliability before console polish --- .torch/prompts/COMMON.md | 18 ++++++++++++++++++ docs/TODO.md | 22 ++++++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/.torch/prompts/COMMON.md b/.torch/prompts/COMMON.md index 56986db..261d491 100644 --- a/.torch/prompts/COMMON.md +++ b/.torch/prompts/COMMON.md @@ -2,6 +2,24 @@ Repository state outranks conversation memory. Query ownership before crossing a domain boundary. +## Owner priority update: reliability before visual polish (2026-10-03 UTC) + +The owner requested that TORCH's operating defects be fixed first so the fleet +benefits from improved operation. This supersedes the UI-first ordering below, +not the existing resume authorization or safety boundaries. The authoritative +backlog priorities were changed through the revision-checked owner API. + +Sequence the smallest safe current-schema stopped-unknown recovery and CLI +qualification first; then truthful lifecycle/status and their source-safety +prerequisites; then qualified bounded continuation/event-driven wakes under +pause, budget and authority policy; then actionable operational attention. +Compact cards/detail modals and coordinator Chat remain high-priority work after +these reliability gates. Do not require the entire future company architecture +to deliver a safe minimal recovery. Reprioritization does not resolve blockers, +grant test consent, enable automatic wakes or launch agents. Preserve task owners, +dependencies and evidence; explicitly block/resequence existing assignments when +needed rather than replacing them silently or creating concurrent assignments. + ## Owner-approved fleet resume and priority polish cycle (2026-10-02) The owner explicitly said: "We got a reset. lets get torch as pollished as we can. diff --git a/docs/TODO.md b/docs/TODO.md index 10a1d9b..44ea4c8 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -1,5 +1,27 @@ # TORCH remaining work +## Current owner priority: reliability first (2026-10-03 UTC) + +This ordering supersedes the UI-first ordering below. The existing authoritative +backlog, not this index, controls task state and assignment. + +1. Safe current-schema runtime recovery (`TASK-company-runtime-recovery`) and + combined CLI qualification (`TASK-pilot-cli-suite-timeout`). +2. Truthful startup/session status (`TASK-pilot-starting-presence`) and its + existing Kernel reservation/Work candidate-safety prerequisites. +3. Persistent operations and bounded event-driven continuation + (`TASK-conversation-persistent-operations`, + `TASK-conversation-event-driven-wakes`), followed by actual scheduler + qualification (`TASK-conversation-scheduler-qualification`). +4. Specific actionable operational warnings (`TASK-console-attention-actions`). +5. Compact cards/detail modals and coordinator-first Chat remain high priority. + +The first four groups and incomplete technical prerequisites are urgent; +scheduler qualification is high. These priority-only changes preserve owners, +states, dependencies and evidence. Blocked remains blocked; automatic wakes stay +disabled until qualified and explicitly activated. Older pause/install/status +statements below are historical snapshots, not current runtime evidence. + ## Priority UI cycle: compact cards and agent Chat Owner-approved [Console UI spec](CONSOLE_CHAT_AND_COMPACT_CARDS_SPEC.md). From 11dabd2450f5cb40104557a684c3efbf763f3900 Mon Sep 17 00:00:00 2001 From: thePR0M3TH3AN Date: Fri, 2 Oct 2026 23:49:59 -0400 Subject: [PATCH 2/5] runtime: recover stopped legacy sessions and add bounded continuation --- .torch/prompts/COMMON.md | 20 ++ docs/TEST_INTEGRITY.md | 57 ++++++ docs/TODO.md | 8 + .../2026-10-03-legacy-runtime-recovery.md | 93 ++++++++++ src/cli.mjs | 49 +++++ src/control-plane/service.mjs | 33 +++- src/runtime/continuation.mjs | 92 ++++++++++ src/runtime/lifecycle.mjs | 3 + src/runtime/stopped-executor-evidence.mjs | 106 +++++++++++ src/runtime/turn-guard.mjs | 31 ++++ test/fleet/legacy-runtime-recovery.test.mjs | 171 ++++++++++++++++++ 11 files changed, 662 insertions(+), 1 deletion(-) create mode 100644 docs/spec_changes/2026-10-03-legacy-runtime-recovery.md create mode 100644 src/runtime/continuation.mjs create mode 100644 src/runtime/stopped-executor-evidence.mjs create mode 100644 src/runtime/turn-guard.mjs create mode 100644 test/fleet/legacy-runtime-recovery.test.mjs diff --git a/.torch/prompts/COMMON.md b/.torch/prompts/COMMON.md index 261d491..c5c5a41 100644 --- a/.torch/prompts/COMMON.md +++ b/.torch/prompts/COMMON.md @@ -2,6 +2,26 @@ Repository state outranks conversation memory. Query ownership before crossing a domain boundary. +## Owner-requested restart and continuation repair (2026-10-03 UTC) + +The owner said "yeah fix the issue" after the arbiter verified that the fleet +had stopped making progress. The arbiter implemented a schema-compatible, +owner-only manual legacy recovery and a bounded deterministic continuation pilot. +These are source changes pending independent QA and installed-release acceptance, +not completed umbrella tasks. Read +docs/spec_changes/2026-10-03-legacy-runtime-recovery.md. The arbiter may apply exact +tested stopped-invocation recovery and resume the existing seven identities with +the source CLI. Preserve old unknown/failure receipts and existing unique commits. + +The source controller may launch only the explicitly approved existing roster, +under its daily turn cap, event deduplication and per-identity physical turn guard. +It does not grant agents permission to start providers or bypass budgets, resource +leases, approval boundaries or task ownership. Existing legacy manager-wake +configuration stays disabled; this source pilot is a distinct, bounded mechanism. +Manager should process current inbox and route real next work rather than leaving +it at a proposal-only loop. QA may independently review the exact new source and +scenario packet now; no fabricated QA acceptance is implied by owner implementation. + ## Owner priority update: reliability before visual polish (2026-10-03 UTC) The owner requested that TORCH's operating defects be fixed first so the fleet diff --git a/docs/TEST_INTEGRITY.md b/docs/TEST_INTEGRITY.md index a35e033..ee96a2a 100644 --- a/docs/TEST_INTEGRITY.md +++ b/docs/TEST_INTEGRITY.md @@ -2811,3 +2811,60 @@ test_integrity_note: did_relax_any_assertion: false if_true_explain_spec_basis: "" ``` +# Owner-requested legacy recovery and bounded continuation (2026-10-03) + +Implementation and behavioral basis: +[legacy runtime recovery](spec_changes/2026-10-03-legacy-runtime-recovery.md). +Existing assertions are unchanged; the new scenarios are in +`test/fleet/legacy-runtime-recovery.test.mjs`. + +```yaml +test_integrity_note: + change_type: new_tests + scenarios: + - id: SCN-legacy-runtime-recovery + given: "A stopped exact invocation and retained unknown identity" + when: "The owner recovers presence" + then: "One owner audit, offline presence and preserved task/session/unknown" + - id: SCN-legacy-runtime-recovery-refusal + given: "Live or mismatched invocation/process evidence" + when: "Proof is requested" + then: "Recovery refuses without mutation" + - id: SCN-legacy-runtime-recovery-cas + given: "Stale, expired, forged or newly live evidence" + when: "Recovery applies" + then: "Transaction refuses without success audit" + - id: SCN-legacy-runtime-recovery-authority + given: "A specialist or incomplete expected snapshot" + when: "Owner recovery is attempted" + then: "Authority/snapshot validation refuses" + - id: SCN-runtime-turn-guard + given: "A physically live executor reporting idle" + when: "A second turn tries to start" + then: "The executor mutex refuses overlap until terminal release" + - id: SCN-runtime-continuation + given: "Owner-enabled roster and a new durable signal" + when: "Turns repeat, hit a cap or are paused" + then: "Only new eligible signals launch within the owner daily cap" + - id: SCN-runtime-continuation-pause + given: "Two eligible turns" + when: "The owner pauses during the first" + then: "The second does not start and pause is retained" + - id: SCN-legacy-runtime-recovery-project + given: "Cloned project/session metadata in distinct checkouts" + when: "Proof crosses checkout boundaries" + then: "Recovery refuses and leaves the other identity untouched" + observable_outcomes: + - "Persisted identity, task/session, owner audit, launch count and refusal codes" + determinism_controls: + - "Hermetic repositories, fixed clock and OS-boundary fixtures; no live provider" + anti_cheat_rationale: + prevents: + - "Boolean stopped proof or stop-as-PASS" + - "Stale/cross-project mutations and peer impersonation" + - "Duplicate physical turns or unchanged-signal busy loops" + - "Losing pause or refunding interrupted attempts" + relaxation: + did_relax_any_assertion: false + if_true_explain_spec_basis: "" +``` diff --git a/docs/TODO.md b/docs/TODO.md index 44ea4c8..e422a66 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -22,6 +22,14 @@ states, dependencies and evidence. Blocked remains blocked; automatic wakes stay disabled until qualified and explicitly activated. Older pause/install/status statements below are historical snapshots, not current runtime evidence. +Owner-requested recovery/continuation implementation is described in +[the source pilot specification](spec_changes/2026-10-03-legacy-runtime-recovery.md). +Do not close the runtime-recovery, persistent-operations or event-wake umbrellas +from focused tests alone. Release/Fleet Operations still need installer-owned +permanent timer provisioning/removal, installed artifact qualification and restart +recovery of retained crashed turn/controller guards. The transient host pilot does +not survive reboot, and its daily turn cap is not a token/dollar usage governor. + ## Priority UI cycle: compact cards and agent Chat Owner-approved [Console UI spec](CONSOLE_CHAT_AND_COMPACT_CARDS_SPEC.md). diff --git a/docs/spec_changes/2026-10-03-legacy-runtime-recovery.md b/docs/spec_changes/2026-10-03-legacy-runtime-recovery.md new file mode 100644 index 0000000..405af54 --- /dev/null +++ b/docs/spec_changes/2026-10-03-legacy-runtime-recovery.md @@ -0,0 +1,93 @@ +# Manual owner recovery for retained unknown executor outcomes + +The owner requested fixing the stalled fleet. Legacy runtime records lacked a +recovery API: three physically stopped executors remained `working/unknown` and +correctly refused duplicate launches. This change adds an owner-only, previewed +`recover-runtime` operation without schema changes or automatic recovery. + +The owner explicitly selects a project-scoped systemd service invocation. Its +working directory and exact area argument must match this installation; its +terminal timestamp must match the retained identity update within five seconds +(systemd displays timestamps with one-second precision). Both service PIDs must +be zero, its cgroup absent, and no visible same-user native process may reference +the session or registered workspace. Physical state is checked again at apply. +Unreadable process visibility, an active service, foreign invocation or changed +identity refuses recovery. This is a **manual legacy operator binding**, not +cryptographic session provenance or protection against arbitrary same-user code. + +Evidence is process-local, branded, single-use and expires after sixty seconds. +The service transaction compares the exact state/runtime/session/update snapshot, +sets presence to offline and writes an owner-attributed audit. Session, current +task and the unknown outcome are preserved. No task, test or launch is completed; +no process is killed, lease removed, schema migrated or provider started. + +Preview and apply use the same implementation. Apply additionally requires the +session and update timestamp from the preview. Hosts/providers without verifiable +Linux/systemd evidence refuse this legacy recovery; no Boolean fallback exists. + +## Bounded continuation pilot + +`torch continuation enable --max-turns-per-day 12 --yes` approves only the +currently activated identity roster. `status` previews eligible identities; +`tick --limit 3 --yes` runs at most three normal guarded `up` operations. New +unacknowledged messages or assigned/in-progress task revision changes supply the +signal; an unchanged signal is not retried. Active/unknown identities, held turn +guards, detach, disabled policy and the daily UTC attempt cap prevent launch. +Attempts are charged before invocation, including failed/interrupted attempts. +The cap limits turns, **not dollars or token consumption**. Unknown quota remains +unknown. This controller makes no product/approval/assignment decisions. + +`torch continuation pause --yes` prevents subsequent launches. Policy and attempt +ledger are separate atomic files so an in-flight turn cannot overwrite a pause. +Pause does not kill an already-running turn. A controller lock serializes ticks; +a crashed lock is retained rather than expired automatically. Each executor has +a separate physical turn guard held through classification and final presence, +so an agent reporting idle cannot permit overlapping native turns. + +The first host pilot uses an explicitly installed transient user-systemd timer. +It survives this chat ending, but not a host reboot. Permanent installer-owned +provisioning/removal and installed artifact qualification remain existing backlog +work; this is not a claim that the alpha.3 installed package has been upgraded. + +```yaml +test_integrity_note: + change_type: new_tests + scenarios: + - id: SCN-legacy-runtime-recovery + given: "An exact retained working/unknown identity and stopped project invocation" + when: "The owner previews and applies recovery" + then: "Presence becomes offline once; session, task and unknown outcome persist with one owner audit" + - id: SCN-legacy-runtime-recovery-refusal + given: "Live, foreign, mismatched or incomplete stopped-executor evidence" + when: "Recovery evidence is requested" + then: "Recovery refuses without identity mutation" + - id: SCN-legacy-runtime-recovery-cas + given: "Issued proof followed by a changed identity, live process, expiration or forgery" + when: "Recovery is applied" + then: "Transactional recovery refuses and creates no success audit" + - id: SCN-legacy-runtime-recovery-authority + given: "A specialist actor or incomplete expected snapshot" + when: "Owner recovery is requested" + then: "Authority or snapshot validation refuses" + - id: SCN-runtime-turn-guard + given: "A physical executor holds an identity guard and reports idle early" + when: "Another executor attempts the same identity" + then: "It refuses until the first executor finishes and releases its own guard" + - id: SCN-runtime-continuation + given: "An explicitly enabled roster, a new durable signal and a fixed daily turn cap" + when: "Continuation ticks, repeats a signal, exhausts its cap or is paused" + then: "Only eligible new signals launch; deduplication, cap, owner authority and pause remain enforced" + observable_outcomes: + - "Durable presence, session and task state; owner-attributed audit; explicit refusal codes" + determinism_controls: + - "Hermetic Git/install fixtures, fixed clock and injected OS observation boundary" + anti_cheat_rationale: + prevents: + - "Caller Boolean as stopped proof" + - "Process stop promoted to successful task or test" + - "Stale snapshot overwrites or silent peer impersonation" + - "Expired, foreign or replayed proof" + relaxation: + did_relax_any_assertion: false + if_true_explain_spec_basis: "" +``` diff --git a/src/cli.mjs b/src/cli.mjs index e136d1d..0b7a0f7 100644 --- a/src/cli.mjs +++ b/src/cli.mjs @@ -70,6 +70,8 @@ import { } from './design/architect.mjs'; import { FleetEvolutionService } from './evolution/service.mjs'; import { HierarchyEvolutionService } from './evolution/hierarchy.mjs'; +import { issueStoppedExecutorEvidence } from './runtime/stopped-executor-evidence.mjs'; +import { configureContinuation, planContinuation, runContinuation } from './runtime/continuation.mjs'; import { ConvergenceService } from './convergence/service.mjs'; import { attachForge, detachForge, forgeStatus, planForgeAttach, planForgeDetach, @@ -95,6 +97,8 @@ Usage: torch install --restore [--dry-run] [--yes] [--json] torch setup [--parent ] [--dry-run] [--yes] [--json] torch up [--fresh] [--only ] [--dry-run] [--yes] [--json] + torch recover-runtime --area --unit --invocation [--expected-session --expected-updated-at