diff --git a/docs/TODO.md b/docs/TODO.md index 238ffea4..d830d640 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -2,6 +2,18 @@ ## dispatch +- 2026-10-23 resume: actual QA4 bootEPERM/release consumed; browser/runtimechecks0, prior mobile strict failures unchanged. HLS sole full/normal grant12:32:43Z at clean79a9a130 after safe merge latest672ca953; START/results pending. Health cooling77→71C/RAM68.3%/swap90% advisory; one heavy max. QA host escalation disposition/browser grant later; no app-fix/cron/autoapproval-effect claim. Overnight client wake left composer yet unread55642s, recovery now consumed. + +- 2026-10-22 storage3+4DONE: storage4 product7edc7ea8/closure8317da4d independently durable status/log/ancestry; worker normalgateexit0/discovery436/build29cbe6f0 artifactPASS, no independent436start audit. QA4 now sole bounded mobile browser grant21:04:02; HLS candidatecdc398b0/merged79a9a130 next full after QA release. Dispatch exact candidate35/35 natural321ms; worker129focused/19mutants/lint11. HLS gates/preview/campaign remain unfinished. + +- 2026-10-22 HLS follow-on: dispatch independent actual observer/controller/helper scratch atd04d501e12=7pass/5fail/cancelled0/naturalexit1/165.17ms. Final-owner GET/timer/promise retained, rejoin stale, late healthy cache, body-read disposal fail; no media-slot leak. Controller HLS handoff + optional manifest-helper cancellation only routed with strict spec/integrity/regressions. Focused/mutants/lint only while storage4 holds heavy slot; torrent/RangeGET remain separate, no HLS fix landed. + +- 2026-10-22 HEAD slice productf41ac656/evidenced04d501e durable status/log/ancestry verified. Worker normal435unique/buildartifact/nativeChromium+Firefox controlledHEAD pass; loadeddata synthetic, no decoder/transport-byte proof. All client resources released20:51:23Z. Storage4 sole full/normal START ACK20:52:16; client9 HLS read-only measurement ACK, broader9 remainsOPEN; QA4 HOLD. Integrated preview and other campaign priorities remain unfinished. + +- 2026-10-22 client9 product normally landedf41ac656: dispatch durable status/log/ancestry0, worker gateexit0/435unique starts. Sequential build active, separate bounded HEAD browser after build conditionally allocated; results pending. Storage4 candidateb21ce86c focused16/9mutants/lint11 ready, next full slot after client release; QA4 held. Integrated preview and broader campaign gates unfinished. + +- 2026-10-22 restart20:41UTC: a79d49df verified durably landed26e9a787, queue0. Fleet health cooling74 to66C/load3.81 to3.30, >20GiB available RAM, swap52 to51% sole suggested-limit crossing. Client9 sole full/normal slot granted; START/results pending. Storage4/QA4 held. Own sync blocked read-only Git metadata; no new product/tests/build/browser qualification. + - 2026-10-22 storage4 registeredfeed/JSDOM independent4=2pass/2fail:obsolete result restores deletedDOM/overwrites replacementExplore;coordinator-only separate fix routed afterstorage3 submit. Storage3 fullgate/build grant active(client9 next full,QA4 mobile held/sourceprep). Client9 exactcandidate independent23pass. No product landing yet. - 2026-10-22 QA3 native originalfixture3pass/no retries,CI2assertions unreproduced/open(logs403/no artifact),no fixjustified. QA4 now actual-app mobile360/390/1280 strictoverflow/source/focus/touch investigation,exclusiveboundedChromium/noCSSwrites. Remaining6priority campaign not complete. diff --git a/docs/agents/BITVID-SCHEDULER.md b/docs/agents/BITVID-SCHEDULER.md new file mode 100644 index 00000000..1b60e1ee --- /dev/null +++ b/docs/agents/BITVID-SCHEDULER.md @@ -0,0 +1,15 @@ +# Bitvid scheduled-job policy + +Owner direction, 2026-10-23: "I would like to not have extra processes like this running when the fleet is down or paused". + +All seven installed Bitvid cron jobs enter through `tools/fleet/bin/bitvid-schedule-guard.mjs`, not directly through `sched.sh`. + +- A missing/dead/stopped interactive fleet means no scheduled headless launch. At least one configured interactive area must have a running record and a live PID. +- `fleet/PAUSED` is represented by the existing state-directory `PAUSED` marker. It unconditionally blocks every job, including the weekly restart job; `ignorePause` does not bypass this outer guard. +- A running scheduled job is checked every second. On pause/down, the guard sends TERM only to the process group it created. Fleet's nested process runners forward termination to their own agent group. A bounded KILL fallback handles a group that does not exit. +- A cron tick while down or paused starts only the brief guard check, which exits without an AI session or persistent watcher. There is no always-on supervisor added. +- Unpausing and bringing the fleet up lets the next normal cron tick run; this policy does not automatically resume a paused fleet. +- Existing quiet hours, per-job non-overlap locks and time limits still apply inside `sched.sh`. The one-heavy-job convention still applies. +- This is a Bitvid-specific file, separate from the kit-managed scheduler. Kit upgrades should retain it. `fleet crontab` prints generic, unguarded commands: do not overwrite the installed guarded entries without preserving this policy. + +Validation: `env -u NODE_TEST_CONTEXT node --test tools/fleet/test/bitvid-schedule-guard.test.mjs` passed 8 tests with natural exit; fixtures only, no real fleet shutdown/pause or headless model run. Test integrity: new behavioral coverage, no existing assertions/gates changed. Process stop tests use a disposable sleep child and verify the unrelated test parent remains alive. Nested production runner cleanup relies on the kit's existing TERM forwarding; this is not a real-model stop qualification. diff --git a/docs/agents/PROGRAM.md b/docs/agents/PROGRAM.md index 8d6a8a9e..7c800d0e 100644 --- a/docs/agents/PROGRAM.md +++ b/docs/agents/PROGRAM.md @@ -1,6 +1,6 @@ # Program — the owner's priorities and rulings -Last updated: 2026-10-22 +Last updated: 2026-10-23 ## Current milestone @@ -17,6 +17,12 @@ Local-review handoff: storage2/history6 and demonstrated cold-feed blockerclient ## Rulings log (newest first) +## 2026-10-23 — Existing seven schedules authorized in interpreter session + +> "All seven jobs shown above" — owner, relayed by interpreter at2026-10-23T12:32:43Z; not typed in dispatch. + +Interpreter reports installing the existing checkin, hygiene, triage, summary, winddown, restart and resources jobs in user crontab, preserving prior entries and verifying readback. Scoped PATH/FLEET_CONFIG; resources every15minutes, only while laptop awake. Installation/readback are interpreter evidence, not independently verified here. This records scope and provenance; dispatch did not install or alter cron. One-heavy-job/resource guidance and all restricted-action limits remain. It adds no push/deploy/spend/delete/credential authority. Interpreter separately reports owner-approved MCP-only autoapproval setting saved; effective state in existing sessions is unverified and does not override a denied action. + ## 2026-10-22 — Six-priority campaign resumed via interpreter > "yeah lets get the fleet spun up and address all of these" — owner, relayed by interpreter at 2026-10-22T14:02:34Z; not typed in dispatch. diff --git a/fleet/GOAL.md b/fleet/GOAL.md index 152cf0a3..e89a5071 100644 --- a/fleet/GOAL.md +++ b/fleet/GOAL.md @@ -50,3 +50,9 @@ Resume existing client worker for current-HEAD active URL-media cancellation rep > "yeah lets get the fleet spun up and address all of these" — owner, relayed by interpreter at 2026-10-22T14:02:34Z; not typed in dispatch. The relay identifies six priorities: Firefox deterministic CI failure; remaining active HEAD/HLS/torrent cancellation; playback reliability promises; feed/view tombstone stale-refresh; Amber sessions; mobile usability. This supersedes the prior idle/local-review and quiet-hours hold for this foreground campaign. Routine investigation, evidence-routed fixes, tests/builds and normal local unstable landing are approved. No push/deploy/spend/production data/real wallet/credential/live signer/scheduler authority. Track all six in docs/agents/CAMPAIGN-2026-10-22.md. Preserve worktrees, use three existing workers with disjoint scope, stage before fixes, serialize browser/full gates, keep app.js single-writer. Final integrated local build/preview and honest live/manual limits required. + +## 2026-10-23 — Existing schedule activation reported + +> "All seven jobs shown above" — owner, relayed by interpreter12:32:43Z; not typed in dispatch. + +Interpreter reports installed/readback-verified existing seven schedules: checkin,hygiene,triage,summary,winddown,restart,resources. This supersedes earlier no-scheduler-activation status only for these jobs, with resources every15minutes while laptop awake. Installation and execution not independently verified by dispatch. Preserve one-heavy-job limit and all restricted-action boundaries; no dispatch cron/settings edits. diff --git a/fleet/LOG.md b/fleet/LOG.md index db406e5d..14a0edaf 100644 --- a/fleet/LOG.md +++ b/fleet/LOG.md @@ -169,3 +169,24 @@ - Client9 dispatch-independent11mode audit via /tmp/bitvid-dispatch-head-audit.mjs: disposed/same-turn/later exit1(3),shared/cross/legacy/live-success/error/timeout/late-success/error exit0(8),allcancelled0/naturalexit. Finalowner leavesHEAD1/abort0/timer1(5000ms)/unsettled/inflight;media slots0. Routed controller HEAD-only lifecycle cleanup+strict spec/integrity/regressions,including ignoring-abort/replacement/handoff controls;HLS/torrent/RangeGET remain separate/open. QA genuineartifact missing;offline exactcachedpackage reconstruction explicitly routed(no fetch/install scripts),before app-ready narrowbrowser. - QA3 native runner completed originalinstrumentedTest/globalsetup/unchanged3selectors:3pass/0fail/0flaky/0skip/retries0/workers1,naturalexit0/9955ms,release~14:22:11UTC. Worker evidence inspected earlier scratch controls;CI2terminal failures unreproduced,environment/access limits kept. No speculative fix/rerun;retain checkpoint docs and redirect QA4 mobile actualapp360/390/1280 strictoverflow+source/focus/touch,investigation only/exclusiveboundedChromium. Firefox remains open,not silently dropped. - Dispatch exact89f8923c archive registered feed/JSDOM repro4=2pass/2fail/cancelled0/naturalexit1/203.65ms confirms old refresh restores deletedDOM and overwrites replacement Explore metadata;validdelay/disposal controls pass. Separate storage4 coordinator-only lifecycle/result guard routed afterstorage3 tip frozen/submitted. No app.js writer;fullgate slot alreadygrantedstorage3,client9 next,QA4 heldsourceprep. Candidatecliente887ceaf independentnewHEAD23/23 naturalexit0/282.24ms;worker60focused/16mutants/lint11 retained,not rerun. No livebrowser/CPU/cloud proof. + +## 2026-10-22 — Maintenance restart assessment + +- Read current rules/context/goal/program/resume/mailbox. Existing approved three-worker team alive/unread0; no launches. a79d49df durable26e9a787, queue0/no drain. Two health snapshots cooling74 to66C, load3.81 to3.30/8, RAM>20GiB, swap52 to51% sole suggested-limit crossing; sole client9 full/normal slot granted20:41:26Z. START/results not yet verified; storage4/QA4 held. +- Interpreter next-action reply and client grant delivered by MCP mailbox under current direct owner startup/mailbox instruction. No approval-config/cron/non-fleet-process changes. Own ff-only sync hit read-only Git metadata; branch/history/dirty restart note preserved. No new tests/build/browser/product qualification. +- Client9 ACK/START20:42:22Z sole normal gate at merged09f52bdd5024d8b691500fc8d7181e6fe2a4336f; clean/runtime equivalence/diffcheck0 worker-reported, gate outcome pending. Worker health77C/RAM66%/swap51%. QA HOLD ACK20:42:17; storage focused-only restriction sent. Client wake deferred/already-read (no extra typing); QA wake submitted; storage wake deferred to waiter. No unattended-completion claim. + +- Client9 product09f52bdd durablef41ac656 verified20:47UTC(status/log/ancestry0);435unique normalgate starts/exit0 worker-reported. Sequential build active; one bounded controlled native HEAD smoke after build conditionally granted. Storage4 READYb21ce86c diff/evidence inspected(16focused/9killedmutants/lint11 worker-reported),next full slot after client release; QA4 held. Fleet health64C/65.8%RAM/swap51% only crossed suggestion. No new dispatch tests/build/browser; integrated preview/live/HLS/torrent remain unverified. +- Client9 reports build/artifact naturalexit0 and heavy RELEASE20:48Z; conditional bounded HEAD browser grant already delivered, START/results pending. Interpreter checkpoint read/replied20:49; no cron/autoapproval/healthconfig changes. Health61.1C/RAM66.6%/swap51% advisory only. Storage4 next full slot,QA4 HOLD; no duplicate full. +- Client9 native browser START20:49:24Z consumed: actual controller/observer/native fetch+AbortController+IntersectionObserver with local controlled HEAD and synthetic media completion; build alreadyexit0, no overlap. Outcome/release pending; docs0219bd31 pending, not assumed landed. Dispatch health58.9C/RAM66.6%/swap51%. Storage4 next full,QA4 HOLD. +- Client HEAD smoke release corrected to actual message20:51:23Z; Chromium+Firefox each3HEAD/1signalabort/cancellednull1/healthy2/cachewrites2/stalebadge0/slots0, synthetic media and4deniedCDN attempts each; source inspected, no decoder/byte/universalabort proof. Evidence4f1d9548 durabled04d501e independently status/log/ancestry0. Storage4 sole full/normal grant20:52:02,START ACK20:52:16; sequential build after green. Client HLS read-only measurement ACK20:52:16, no writes/browser/full/build; keep9OPEN. QA4 held. +- HLS reproduction independently12=7pass/5fail/0cancelled/naturalexit1/165.17ms atd04d501e: orphan GET/timer/promise, stale replacement/late cache/body-read disposal. Temporary controller HLS handoff+manifest-helper optional cancellation ownership and strict spec/integrity/tests routed; no player/engine/torrent/RangeGET scope. Storage4 retains heavy slot; client focused/mutants/lint only, QA4 held. Evidence test_logs/TEST_LOG_2026-10-22_20-54-00_dispatch-hls.md. +- Client HLS routing ACK20:55:32 consumed; focused implementation only. Repro docsfb0e671d independentlyQUEUED behind alive drain277174, not landed. Health burst86.1C/clientCPU58% then77.1C/client0% within14s, RAM64.2%/swap51%; no sustained-over85 finding or extra heavy launch. Storage4 sole gate,QA4 held. +- Storage4 product888807ba durably7edc7ea8/closure265ce426 durably8317da4d independently status/log/ancestry0. Worker normalgateexit0/discovery436(no independent436start audit),build29cbe6f0/artifactPASS; released20:58Z. QA4 sole bounded mobile browser granted21:04:02 on verified frozen artifact/provenance; HLS next full after QA release. Clientcandidatecdc398b0 merged79a9a130 inspected; dispatch direct new suite35/35 naturalexit0/321.06635ms, worker129focused/19mutants/lint11. No HLS full/build/browser/landing yet. + +## 2026-10-23 — Mail recovery and HLS gate allocation + +- Actual mailbox consumed: QA4 previous localserverEPERM/release with runtime0/browser0; client clean79a9a130 preserved; interpreter reports owner-approved MCP autoapproval config only, not independently verified/effective and no config touched here. Cron still pending/uninstalled. +- Wake log client21:04:05 submitted/leftcomposer but unread55642s, recovery12:31:27 submitted1Enter; mail now consumed. Busy interpreter wake deferred/already-read, no waiter typing. +- Fleethealth77→71C/RAM68.3%/~21GiB available/load1.04/8/swap90% only advisory stable/slightly falling; queue0. Granted sole HLS full/normal12:32:43Z, sequentialbuild aftergreen/freshhealth, no browser overlap. QA HOLD; sandbox EPERM inspection/normal host escalation staged after release, no bypass/rejected-action retry. Results/integratedpreview/campaign unfinished. +- Client HLS START12:33:14 at mergedd7184f07, clean/inputsequal workerreport; gate outcome pending. Health81.1C/RAM66.3%/swap89%. Interpreter reports owner quote "All seven jobs shown above" and all7 existing schedules installed/readback; recorded PROGRAM/GOAL/RULE with provenance, no independent install/execution/config-effect proof. No settings/cron mutation here; restricted authority unchanged. Busy interpreter wake waiter typed/submitted12:33:13, actual mail consumed. diff --git a/fleet/RULE-CHANGES.md b/fleet/RULE-CHANGES.md index 0f7a420b..afa6b008 100644 --- a/fleet/RULE-CHANGES.md +++ b/fleet/RULE-CHANGES.md @@ -7,8 +7,14 @@ override a stale system prompt. Nothing filters them down to a window of days. The session manager adds a line here whenever a rule, a prompt, a tool's usage, or a landing rule changes. +## 2026-10-23 + +- Interpreter12:32:43Z reports owner quote "All seven jobs shown above" and installation/readback of existing checkin/hygiene/triage/summary/winddown/restart/resources schedules, scoped PATH/FLEET_CONFIG and preserved prior crontab. Record as relay/interpreter evidence, not independent verification or dispatch install authority. Earlier no-cron-installed statements are historical; no execution-success claim. One heavy job maximum and restricted-action boundaries remain; no denied-action retry or assumed effective MCP approval config. + ## 2026-10-22 +- Client9 HLS follow-on explicitly routed after HEAD productf41ac656/evidenced04d501e durable. Dispatch independently ran actual observer/controller/cache/probeHlsManifest scratch12=7pass/5fail/cancelled0/naturalexit1/165.17ms atd04d501e: final-owner GET1/abort0/timer1/unsettled, stale same-source rejoin and late cache; body-read disposal also fails, media slots0. Temporary ownership ONLY js/ui/urlHealthController.js HLS lifecycle handoff + js/services/hlsPlayback.js manifest-probe optional cancellation plumbing across fetch/body + unique focused lifecycle/helper tests and strict spec/integrity notes. Explicit Stage3 contract extends final-owner prompt null settlement/abort/timer-hook cleanup/no obsolete cache/fallback/mirror/badge writes to HLS fetch AND body-read, even ignored abort or absent AbortController. Shared/cross-controller/legacy/live success/error/timeout and same-source replacement identity preserved; existing helper callers/signature behavior preserved without lifecycle. No HLS engine/player/session/config changes, torrent/RangeGET/app.js excluded. Focused+mutants+lint only, no full/build/browser/submission until resource grant; storage4 currently sole full/normal slot, QA4 held. + - Storage4 separately routed after dispatch exact89f8923c registered Recent/Explore+productionstages+actualVideoListView/Card JSDOM reproduction4=2pass/2fail/naturalexit1/203.65ms: old sameview result restores deletedDOM and old Recent overwrites remounted Explore metadata. Temporary ownership ONLY js/app/feedCoordinator.js async request/result/error/lifecycle eligibility + unique new registered lifecycle tests/notes. No app.js/view/engine/service/buffer edits for4; preserve live rendering/source/filter/moderation/ranking/return contracts. Guard stale success AND failure metadata/render/telemetry/sideeffects, newer refresh/view identity/disposal, preserve valid delayed pipeline/fallback. Freeze+submit existingstorage3 candidate first; storage4 separatecommit/item/normalgate, focused/mutants/lint then resource request. Current slot storage3 full/build;client9 next full;QA4 mobile HOLD/sourceprep, no unannounced competing jobs. - QA3 native original instrumentedTest/globalsetup/3callbacks passed3/0fail/0flaky/0skip/retries0 with private genuine Buffer/public-deny guard; CI2 terminal assertions remain unreproduced, no causal fix. Retain docs-only checkpoint, keep CI issue open pending equivalent evidence; no speculative concurrency/repeated local runs. Existing QA moves to QA4 actual-app mobile Upload overflow360/390 plus1280 control, focus/touch/source-culprit measurement first, scratch/unique evidence only. Exclusive bounded Chromium browser slot granted after START; no product/CSS/tests/config/full/build before reproduced cause+ownership routing. Preserve original strict overflow failure, genuine fixture dependency provenance, preview3000, and live/manual limits. diff --git a/test_logs/TEST_LOG_2026-10-22_19-21-00_dispatch-restart.md b/test_logs/TEST_LOG_2026-10-22_19-21-00_dispatch-restart.md new file mode 100644 index 00000000..36c22f19 --- /dev/null +++ b/test_logs/TEST_LOG_2026-10-22_19-21-00_dispatch-restart.md @@ -0,0 +1,8 @@ +# Dispatch restart checkpoint +- Read current rules/context/goal/program/resume/repo memory and actual mailbox. Prompt sync confirmed by interpreter19:19:04. +- git log unstable:75bc92f3a47c06ead43c3e715ef8793475635518; queue0/lander stopped. No tests/build/new worker launched by dispatch. +- CLI mailbox interpreter reply19:20:17 and client9 grant19:20:28 delivered; wake log reports no-window. Host MCP status earlier reported alive; CLI status cannot establish worker termination. Actual incoming worker startup messages confirm sessions resumed. +- Resource constraint consumed19:20:34 supersedes heavy grant. free/uptime/sensors/PSI exit0:19GiB available,swap1.1/2GiB,package68C,load10.20/8.19/5.17,CPU PSI avg60 17.74%,memory PSI0. Hold new heavy launches. +- MCP mailbox_send resource hold to client rejected: outgoing coordination reserved to user; prior review forbade retry; no trusted direct override. No retry/workaround. Local .task records hold and ambiguity; worker receipt/execution unverified. +- Latest original dispatch wake swallowed after2Enter; subsequent incoming wakes skipped-busy while dispatch midturn. Actual mailbox consumed. +- No new product/test/landing qualification. Six campaign priorities remain open. Prior storage3 gate434 evidence not rerun. diff --git a/test_logs/TEST_LOG_2026-10-22_20-54-00_dispatch-hls.md b/test_logs/TEST_LOG_2026-10-22_20-54-00_dispatch-hls.md new file mode 100644 index 00000000..66bdc77a --- /dev/null +++ b/test_logs/TEST_LOG_2026-10-22_20-54-00_dispatch-hls.md @@ -0,0 +1,6 @@ +# HLS cancellation independent reproduction + +- Source under test: worker clean branch d04d501e (git log printed), actual observer/controller/cache/probeHlsManifest via /tmp/client9-hls-repro.mjs. Script inspected; controlled fetch/timers/JSDOM, no network. +- Command: env -u NODE_TEST_CONTEXT -u UNIT_TEST_SHARD node /tmp/client9-hls-repro.mjs +- Result: natural exit1,12 tests,7pass/5fail/0cancelled/0skipped,165.167587ms. Final-owner GET1/abort0/timer1/unsettled/inflight unchanged; same-turn/later replacement shares old request; late success writes healthy cache; disposal during body-read fails. Shared/cross-controller/legacy/live-success/error/timeout/body-shared controls pass. Media slots0/queue0, disposed badge checking. +- No code/tests edited by dispatch, no full/browser/build run; no real transport/decoder/live playback proof. Routed optional lifecycle at controller/helper boundary with explicit strict HLS contract, preserving legacy/live callers. diff --git a/test_logs/TEST_LOG_2026-10-22_21-04-00_dispatch-hls-candidate.md b/test_logs/TEST_LOG_2026-10-22_21-04-00_dispatch-hls-candidate.md new file mode 100644 index 00000000..d40a26f2 --- /dev/null +++ b/test_logs/TEST_LOG_2026-10-22_21-04-00_dispatch-hls-candidate.md @@ -0,0 +1,6 @@ +# HLS candidate verification + +- Git printed candidate cdc398b0280346d19fb5d7a61025168a7d55de5e and merged79a9a130 from worker; inspected exact controller/helper delta via git show. Optional lifecycle-only change, existing helper callers retained. +- Command: env -u NODE_TEST_CONTEXT -u UNIT_TEST_SHARD node /home/user/Documents/GitHub/bitvid-client-reliability/tests/url-health-hls-lifecycle.test.mjs +- Result naturalexit0,35tests/35pass/0fail/0cancelled/0skipped,321.06635ms. Controlled actual observer/controller/helper fetch/body, native-path handoff and hook identity controls. +- Worker broader129/6files/19killedmutants/lint11 not rerun. Full/build/browser unrun for HLS; no realnetwork/decoder/byte or torrent/RangeGET proof. Storage4 landing independently checked status/log/ancestry; QA4 next sole bounded mobile browser, HLS full waits. diff --git a/tools/fleet/bin/bitvid-schedule-guard.mjs b/tools/fleet/bin/bitvid-schedule-guard.mjs new file mode 100644 index 00000000..66d51c59 --- /dev/null +++ b/tools/fleet/bin/bitvid-schedule-guard.mjs @@ -0,0 +1,56 @@ +#!/usr/bin/env node +// Bitvid policy: cron may work only while an interactive fleet is alive and unpaused. +// Kept separate from sched.sh so kit updates cannot restore ignorePause behavior. +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { loadConfig, pidAlive, killProcessGroup, runProcessGroup } from './lib.mjs'; + +export function inactiveReason(c) { + if (fs.existsSync(path.join(c.stateDir, 'PAUSED'))) return 'fleet paused'; + for (const area of c.areas) { + try { + const run = JSON.parse(fs.readFileSync(path.join(c.stateDir, 'runs', `${area.id}.json`), 'utf8')); + if (run.status === 'running' && Number.isInteger(run.pid) && run.pid > 1 && pidAlive(run.pid)) return null; + } catch { /* Missing or invalid state is not permission to launch. */ } + } + return 'fleet down'; +} + +export async function guardSchedule(c, job, { scheduler = path.join(path.dirname(fileURLToPath(import.meta.url)), 'sched.sh'), pollMs = 1000 } = {}) { + if (!c.schedules.some((entry) => entry.name === job)) throw new Error(`Unknown scheduled job: ${job}`); + const reason = inactiveReason(c); + if (reason) return { status: 0, reason }; + let poll; + let force; + let stopped; + const result = await runProcessGroup({ + cmd: '/bin/bash', args: [scheduler, job], cwd: c.repo, env: process.env, + onSpawn(pid) { + poll = setInterval(() => { + const why = inactiveReason(c); + if (!why || stopped) return; + stopped = why; + // Only the process group this invocation created. Nested fleet runners + // receive TERM and forward it to their own agent process groups. + killProcessGroup(pid, 'SIGTERM'); + force = setTimeout(() => killProcessGroup(pid, 'SIGKILL'), 3000); + }, pollMs); + }, + }).finally(() => { + clearInterval(poll); + clearTimeout(force); + }); + return { status: stopped ? 0 : (result.status ?? 1), reason: stopped ?? null }; +} + +if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + try { + const c = loadConfig(); + const result = await guardSchedule(c, process.argv[2]); + process.exitCode = result.status; + } catch (error) { + console.error(`Bitvid schedule guard: ${error.message}`); + process.exitCode = 1; + } +} diff --git a/tools/fleet/test/bitvid-schedule-guard.test.mjs b/tools/fleet/test/bitvid-schedule-guard.test.mjs new file mode 100644 index 00000000..6a54c452 --- /dev/null +++ b/tools/fleet/test/bitvid-schedule-guard.test.mjs @@ -0,0 +1,67 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { spawn } from 'node:child_process'; +import { guardSchedule } from '../bin/bitvid-schedule-guard.mjs'; + +const moduleUrl = new URL('../bin/bitvid-schedule-guard.mjs', import.meta.url).href; +function fixture(t) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'bitvid-schedule-')); + t.after(() => fs.rmSync(root, { recursive: true, force: true })); + const c = { repo: root, stateDir: root, areas: [{ id: 'dispatch' }], schedules: [{ name: 'resources' }, { name: 'restart' }] }; + fs.mkdirSync(path.join(root, 'runs')); + const record = path.join(root, 'runs', 'dispatch.json'); + const scheduler = path.join(root, 'schedule.sh'); + fs.writeFileSync(scheduler, 'echo started\nprintf yes > "'+root+'/launched"\n'); + const live = () => fs.writeFileSync(record, JSON.stringify({ status: 'running', pid: process.pid })); + return { c, root, record, scheduler, live }; +} + +for (const job of ['resources', 'restart']) { + test(`${job} does not launch while down`, async (t) => { + const f = fixture(t); + assert.deepEqual(await guardSchedule(f.c, job, { scheduler: f.scheduler }), { status: 0, reason: 'fleet down' }); + assert.equal(fs.existsSync(path.join(f.root, 'launched')), false); + }); + test(`${job} cannot bypass pause even with a live fleet`, async (t) => { + const f = fixture(t); f.live(); fs.writeFileSync(path.join(f.root, 'PAUSED'), 'owner pause'); + assert.deepEqual(await guardSchedule(f.c, job, { scheduler: f.scheduler }), { status: 0, reason: 'fleet paused' }); + assert.equal(fs.existsSync(path.join(f.root, 'launched')), false); + }); +} + +test('stopped record pointing at a live unrelated process cannot start a job', async (t) => { + const f = fixture(t); + fs.writeFileSync(f.record, JSON.stringify({ status: 'stopped', pid: process.pid })); + assert.equal((await guardSchedule(f.c, 'resources', { scheduler: f.scheduler })).reason, 'fleet down'); +}); + +test('live unpaused fleet launches a short job and exits naturally', async (t) => { + const f = fixture(t); f.live(); + assert.deepEqual(await guardSchedule(f.c, 'resources', { scheduler: f.scheduler }), { status: 0, reason: null }); + assert.equal(fs.readFileSync(path.join(f.root, 'launched'), 'utf8'), 'yes'); +}); + +for (const transition of ['pause', 'down']) { + test(`already-running scheduled child is stopped on ${transition}`, { timeout: 5000 }, async (t) => { + const f = fixture(t); f.live(); + fs.writeFileSync(f.scheduler, 'echo started\nexec sleep 600\n'); + const code = `import {guardSchedule} from ${JSON.stringify(moduleUrl)}; const r=await guardSchedule(${JSON.stringify(f.c)},'resources',${JSON.stringify({scheduler:f.scheduler,pollMs:25})}); console.log(JSON.stringify(r)); process.exitCode=r.status;`; + const child = spawn(process.execPath, ['--input-type=module', '-e', code], { stdio: ['ignore','pipe','pipe'] }); + t.after(() => { if (child.exitCode === null) child.kill('SIGTERM'); }); + let output = ''; + const exited = new Promise((resolve, reject) => { child.once('error',reject); child.once('close',(status)=>resolve(status)); }); + await new Promise((resolve, reject) => { + child.once('error', reject); + child.stdout.on('data', chunk => { output += chunk; if (output.includes('started\n')) resolve(); }); + child.once('exit', () => { if (!output.includes('started\n')) reject(new Error('child exited before ready')); }); + }); + if (transition === 'pause') fs.writeFileSync(path.join(f.root,'PAUSED'),'pause'); + else fs.writeFileSync(f.record, JSON.stringify({status:'stopped',pid:process.pid})); + assert.equal(await exited, 0); + assert.match(output, transition === 'pause' ? /fleet paused/ : /fleet down/); + assert.doesNotThrow(() => process.kill(process.pid, 0)); + }); +}