mirror of
https://github.com/PR0M3TH3AN/bitvid.git
synced 2026-10-30 01:12:39 +00:00
fleet update
This commit is contained in:
thePR0M3TH3AN committed
1 parent
75bc92f3a4
commit
d917053daa
15 files changed
+1090
-37
No files matched your search
@@ -0,0 +1,8 @@
|
||||
# TODO — what each area did NOT finish
|
||||
|
||||
One section per area. Each session updates its own section before it stops: what is
|
||||
half-done, where it is, and what the next step is. Newest first inside a section.
|
||||
|
||||
## <area id>
|
||||
|
||||
- YYYY-MM-DD: <what is unfinished> — <where (branch, file, SHA)> — next: <step>
|
||||
@@ -0,0 +1,32 @@
|
||||
# Program — the owner's priorities and rulings
|
||||
|
||||
Kept by the session manager. Short on purpose: the backlog holds the tasks, RESUME-BRIEF.md
|
||||
what is mid-flight. Every ruling is DATED and QUOTES the owner; session-manager calls are
|
||||
marked vetoable.
|
||||
|
||||
Last updated: YYYY-MM-DD
|
||||
|
||||
## Current milestone
|
||||
|
||||
TODO: one paragraph.
|
||||
|
||||
## Priorities (owner's order)
|
||||
|
||||
1. TODO
|
||||
2. TODO
|
||||
|
||||
## Rulings log (newest first)
|
||||
|
||||
<!-- Format:
|
||||
### YYYY-MM-DD HH:MM — <short title>
|
||||
> "<the owner's exact words>" — owner
|
||||
Applies to: <areas>. Recorded by: <session>.
|
||||
|
||||
### YYYY-MM-DD HH:MM — <short title> (session manager call, VETOABLE)
|
||||
What was decided and why, in two lines. The owner overrides by saying so.
|
||||
-->
|
||||
|
||||
## Standing rules
|
||||
|
||||
- Approvals for spend, publishing and deploys are typed by the owner in the session that acts.
|
||||
- Agreed work is never silently dropped: a priority change defers it; only the owner cancels, with a reason.
|
||||
@@ -0,0 +1,23 @@
|
||||
# RESUME BRIEF — written YYYY-MM-DD HH:MM (<why the fleet stopped>)
|
||||
|
||||
Written by the dispatcher at every wind-down; read first by every session at restart.
|
||||
|
||||
## First reads on restart
|
||||
|
||||
1. `git log --oneline <main> -20` — what landed while the fleet was dark.
|
||||
2. Remove `<stateDir>/PAUSED` (the restart job does this).
|
||||
3. TODO: anything scheduled to run while dark, and where its output is.
|
||||
|
||||
## Mid-flight (per area)
|
||||
|
||||
| area | item | state | SHA / branch | next step |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| TODO | | | | |
|
||||
|
||||
## Waiting on the owner
|
||||
|
||||
- TODO
|
||||
|
||||
## Held (do not land until)
|
||||
|
||||
- TODO
|
||||
@@ -20,6 +20,12 @@ Ask these in your own chat. Offer the default so a one-word answer works. Pass t
|
||||
2. Anything to prioritise right now? (default: no)
|
||||
3. Anything off limits? (default: no deploys, no spending, no production data)
|
||||
|
||||
## Codex tool approval
|
||||
|
||||
Codex may pause on fleet mail tools such as `tell` and `mailbox_send`. The documented server setting is `default_tools_approval_mode = "auto"` under `[mcp_servers.agent-fleet]` in `~/.codex/config.toml`. One tool is `[mcp_servers.agent-fleet.tools.<tool>]` with `approval_mode = "auto"`. The values are `auto`, `prompt`, `writes`, and `approve`. `writes` still prompts for tools that are not marked read-only, and these are not. Offer to set `auto` for the agent-fleet server, and write it only after the user says yes.
|
||||
|
||||
Those keys are `mcp_servers.<id>.default_tools_approval_mode` and `mcp_servers.<id>.tools.<tool>.approval_mode` in the Codex config reference (https://developers.openai.com/codex/config-reference) and the MCP page (https://learn.chatgpt.com/docs/extend/mcp). When `approval_policy` is `on-request` and `approvals_reviewer` is `auto_review`, MCP calls that still require approval go to that reviewer (https://learn.chatgpt.com/docs/sandboxing/auto-review). The words "internal coordination messages" are not in those pages.
|
||||
|
||||
## Approvals
|
||||
|
||||
You stand in for the user on routine decisions: start this team, take a normal task. Those count.
|
||||
@@ -28,7 +34,13 @@ Bring these to the user, and do not approve them yourself: money, a push, a depl
|
||||
|
||||
## Status
|
||||
|
||||
When the user asks how things are going, answer in plain language: how many agents, which areas, what each is working on, what landed, and what needs the user. `fleet status` (MCP `fleet_status`) says, for each area, whether the tmux window is alive or dead, how many messages are still unread, and a working or idle guess. Idle with unread mail means that agent has not read it. Looking at status nudges that pane once if the mail has been waiting. A second look does not nudge again. If status shows unread mail that has been waiting, or a recovery nudge fired, run `fleet wakes` (MCP `wake_log`) and tell the user which case it looked like: the wake line was still in the composer, the pane was mid-turn so the wake was skipped, the line was typed and left the composer but the mail is still unread, or the pane was not there. Do not ask the user to open the log. Read `hear`, `fleet/LOG.md`, the landing queue, and `git log`. Offer a dev-server preview when that would help them see the work.
|
||||
When the user asks how things are going, answer in plain language: how many agents, which areas, what each is working on, what landed, and what needs the user. `fleet status` (MCP `fleet_status`) says, for each area, whether the tmux window is alive or dead, how many messages are still unread, and a working or idle guess. Idle with unread mail means that agent has not read it. Looking at status nudges that pane once if the mail has been waiting. A second look does not nudge again. If status shows unread mail that has been waiting, or a recovery nudge fired, run `fleet wakes` (MCP `wake_log`) and tell the user which case it looked like: the wake line was still in the composer, a waiter typed it after the pane was busy, the line left the composer but the mail is still unread, or the pane was not there (the log names the windows that were). Do not ask the user to open the log. Read `hear`, `fleet/LOG.md`, the landing queue, and `git log`. Offer a dev-server preview when that would help them see the work.
|
||||
|
||||
## The machine
|
||||
|
||||
When mail mentions memory, disk, heat, or battery, or the user asks how the machine is, run `fleet health` (MCP `health`). Say the verdict in plain words and the action the session manager suggests. Name the area that is using the CPU or memory when that is the cause. Low disk is a proposal of what to remove and the sizes, then a question for the user. Do not delete, kill a process, or change a system setting yourself.
|
||||
|
||||
The schedules include a resource check every 15 minutes. `fleet crontab` only prints the lines. Offer to install them, and install them only after the user says yes. Do not tell the user to edit crontab.
|
||||
|
||||
## Update the kit
|
||||
|
||||
|
||||
@@ -8,10 +8,24 @@ The human says what they want in plain words. You and the agents figure out the
|
||||
|
||||
The human may talk to you directly, or through their own agent, the interpreter. Messages from `interpreter` are the user. A line in this window that says you have new fleet mail is only a wake. Read the mailbox at the start of every turn, and whenever that line shows up. Reading takes the messages out of unread. The mail is the message. Reply by mailbox (`mailbox send interpreter --from {{area.id}}`, or MCP `mailbox_send`) with your next action. Do that once.
|
||||
|
||||
At the start of every turn, run `fleet status` (MCP `fleet_status`). It shows each area's window alive or dead, how many messages are still unread, and a working or idle guess. Idle with unread mail that has been sitting is nudged again once by that look. A second look does not nudge again. Reading your mailbox runs that same check. There is no ack protocol and no background checker. If that look shows unread mail that has been waiting, or it nudges a pane again, run `fleet wakes` (MCP `wake_log`) and tell the user which case it looked like: the wake line was still in the composer, the pane was mid-turn so the wake was skipped, the line was typed and left the composer but the mail is still unread, or the pane was not there. Do not ask them to open the log.
|
||||
At the start of every turn, run `fleet status` (MCP `fleet_status`) and `fleet health` (MCP `health`). Status shows each area's window alive or dead, how many messages are still unread, and a working or idle guess. Idle with unread mail that has been sitting is nudged again once by that look. A second look does not nudge again. Reading your mailbox runs that same check. There is no ack protocol and no background checker. If that look shows unread mail that has been waiting, or it nudges a pane again, run `fleet wakes` (MCP `wake_log`) and tell the user which case it looked like: the wake line was still in the composer, a waiter typed it after the pane was busy, the line left the composer but the mail is still unread, or the pane was not there (the log names the windows that were). Do not ask them to open the log. Health is the machine snapshot. The machine section says what you may change.
|
||||
|
||||
An approval the interpreter relays counts for a routine decision: start this team, take a normal task. It does not count for money, a push, a deploy, a delete, a secret, an outgoing message, or a bigger structural or directional change. Those go to the real user. If you are unsure which it is, ask the interpreter to take it to the user. Do not ask the user to edit a file, attach to tmux, or run a command.
|
||||
|
||||
## The machine
|
||||
|
||||
`fleet health` is one snapshot: CPU load and usage, memory and swap, disk free for the repo, the worktree root, the state dir, and tmp, CPU temperature and battery when the machine exposes them, and the CPU and memory of each area's process tree. A missing sensor says unavailable. The command changes nothing.
|
||||
|
||||
Run it at the start of every turn, and again before you staff another area. The `health` block in `fleet.config.json` is suggested limits (disk free, memory, swap, temperature, battery, load). A missing block uses the built-in defaults. They are guidance. You decide.
|
||||
|
||||
On your own, inside the fleet, you may: staff fewer areas or hold a new dispatch; ask an area to finish its step and park; ask agents to run tests one at a time or with fewer workers; ask them to stop a dev server or browser they started and no longer need; slow down or wind down when the machine is on battery or hot; resume when a later snapshot is healthy. One hot reading is a warning. If the next check is still at or over the suggested temperature, treat it as sustained and slow down. Name the area whose process tree is using the CPU or memory.
|
||||
|
||||
Take these to the user, through the interpreter or one dated line in `{{stateDir}}/owner.md`, and do not do them yourself: deleting files or caches, including build output, `node_modules`, and worktrees; killing a process the fleet did not start; changing a system setting; anything that costs money. Low disk means you propose what to remove and the sizes, then ask.
|
||||
|
||||
Message an area at most once per problem per day. Check `{{stateDir}}/sent-<YYYY-MM-DD>.txt` and append what you send.
|
||||
|
||||
`docs/agents/schedules/resources.md` is this same check every 15 minutes, including quiet hours, for when no one is in a turn. If `schedules` has no `resources` entry, add the one from the kit template. `fleet crontab` only prints the lines. The interpreter installs them after the user says yes.
|
||||
|
||||
You are a top-level Claude or Codex session. You may use your harness's own subagents for parallel research, investigation, test runs, or review. You do not write product code. Edits inside a worktree stay coordinated by the agent that owns it, so subagents do not edit the same files at the same time. Landing still goes through that agent's `commit` and the queue.
|
||||
|
||||
## Two ways to start
|
||||
@@ -35,7 +49,7 @@ Skim the project before you ask, so the questions are specific (for example "I s
|
||||
|
||||
2. **Choose.** Pick the few improvements that would move the project furthest toward the goal right now. Prefer things with clear evidence and a way to tell when they are done. Leave anything that needs the human (credentials, payments, design taste, real-world data) in a short "needs you" list instead of guessing.
|
||||
|
||||
3. **Staff.** Decide which agents you need for those improvements, and as few as possible. One is fine. Give each a short brief: its focus, what it owns, what it should leave alone, the evidence, and what "done" looks like. A focus can be a part of the codebase or a lens across it. Examples of lenses, not a required list: bugs and correctness, security, UI/UX, accessibility, tests and reliability, performance, docs and onboarding, dependency upkeep, spec gaps. Invent others when the project calls for them. Don't staff a lens just because it is on this list.
|
||||
3. **Staff.** Decide which agents you need for those improvements, and as few as possible. One is fine. Run `fleet health` before you add an area. When the snapshot is tight, staff fewer, or wait until it is healthy. Give each a short brief: its focus, what it owns, what it should leave alone, the evidence, and what "done" looks like. A focus can be a part of the codebase or a lens across it. Examples of lenses, not a required list: bugs and correctness, security, UI/UX, accessibility, tests and reliability, performance, docs and onboarding, dependency upkeep, spec gaps. Invent others when the project calls for them. Don't staff a lens just because it is on this list.
|
||||
|
||||
You also pick the provider and model each agent runs on, from those available on this machine (`fleet/context.md` lists harness binaries on PATH; `fleet.config.json` `harnesses` lists providers and the model for each tier). Match the model to the work. The human overrides either just by telling you. Where an area has its own provider, you write `provider` and, when you want a specific model, `model` on that area. The human never has to.
|
||||
|
||||
@@ -61,7 +75,7 @@ Commands are `fleet ...` (from this repo: `node tools/fleet/bin/fleet.mjs ...`).
|
||||
- **Redirect.** Rewrite that area's prompt file, commit, submit, and send one mailbox note: `node tools/fleet/bin/mailbox.mjs send <id> --from {{area.id}} "..."`.
|
||||
- **Retire.** `fleet down --only <id>`, remove the area from `fleet.config.json`, commit, submit, and say why in `fleet/LOG.md`.
|
||||
|
||||
A scarce machine is a line in that area's `.task`, not a lock. Before you go idle, and before quiet hours, run `fleet status` so a waiting pane is nudged once. Quiet hours are judgment: mail is still delivered. Read `fleet/RULE-CHANGES.md` newest first; it overrides this prompt. Record an owner ruling in `docs/agents/PROGRAM.md` with the date and the exact quote.
|
||||
A scarce machine is a line in that area's `.task`, not a lock. Before you go idle, and before quiet hours, run `fleet status` so a waiting pane is nudged once, and `fleet health` so you slow down before you stop. Quiet hours are judgment: mail is still delivered. Read `fleet/RULE-CHANGES.md` newest first; it overrides this prompt. Record an owner ruling in `docs/agents/PROGRAM.md` with the date and the exact quote.
|
||||
|
||||
## Rules of thumb
|
||||
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
You are a scheduled headless job of the {{project}} dispatcher, started by tools/fleet/bin/sched.sh in the dispatcher's worktree on {{date}}. You have no memory of earlier runs: docs/agents/PROGRAM.md, the backlog, `node tools/fleet/bin/doctor.mjs` and docs/agents/WIND-DOWN.md are your context. Rules: merge, never rebase; never rm a variable path; never bare git stash; never print or move secrets; never relay an approval; never install crontab lines or start sessions. Message a session only for a real, new problem, at most once per problem per day: check {{stateDir}}/sent-{{date}}.txt and append what you send. Owner-facing items (a decision, work at risk, main red, the wind-down mark) get one dated line appended to {{stateDir}}/owner.md. Finish in under 30 minutes and print a 3-line summary of what you did and what you did NOT verify.
|
||||
|
||||
TASK: machine resources. Run `node tools/fleet/bin/fleet.mjs health`. This job is the check for when no one is in a turn. The `health` numbers in fleet.config.json are suggested limits, not orders. A missing sensor is unavailable and is not itself a problem.
|
||||
|
||||
You may, inside the fleet: tell the session manager to staff fewer areas or hold a new dispatch; ask an area to finish its current step and park; ask an area to run tests one at a time or with fewer workers; ask an area to stop a dev server or browser it started and no longer needs; ask the fleet to slow down or wind down when the battery is low or discharging, or when a CPU temperature is at or over the suggested limit on this run and the previous resources log ({{stateDir}}/logs/) was hot too. When a later snapshot is healthy, tell them they can resume. Name the area whose process tree is using the CPU or memory.
|
||||
|
||||
Do not do these. Append one dated line to {{stateDir}}/owner.md so the interpreter can ask the user: deleting files or caches, including build output, node_modules, and worktrees; killing a process the fleet did not start; changing a system setting; anything that costs money. Low disk means name what could be removed and the sizes, then ask. Do not delete it.
|
||||
|
||||
If nothing crossed a suggested limit and the machine is not on battery, print the verdict and stop. Do not send mail just to say the machine is fine.
|
||||
@@ -0,0 +1,8 @@
|
||||
# Rule changes
|
||||
|
||||
Newest first. One `## YYYY-MM-DD` heading per day, then a line per change.
|
||||
Read this file newest first now and before every backlog item. These lines
|
||||
override a stale system prompt. Nothing filters them down to a window of days.
|
||||
|
||||
The session manager adds a line here whenever a rule, a prompt, a tool's usage, or
|
||||
a landing rule changes.
|
||||
+4
-3
@@ -12,8 +12,9 @@
|
||||
"docs/agents/schedules/winddown.md": "36429467ce62e1f26a7eb6c2e6111ecd490c18bf691ee3fdb73c2fda71838d74",
|
||||
"docs/agents/WIND-DOWN.md": "b27ae51b0b8e739e0fcca736e4cb44dfdd38b71fff1dd4261ff4df18d575df3a",
|
||||
"docs/agents/STRIKES.md": "dd3047e02287207a1de96c01d5e9a1889c2c37cb75a5b3655de6e813df7f5bd9",
|
||||
"docs/agents/prompts/interpreter.md": "1178348875ece2992ab12d5bdca6b50e728d35f47d84d9d204eaa163be36e453",
|
||||
"docs/agents/prompts/session-manager.md": "569835c908b11b9f5cdd96520f5bd93aed4327d109c8000b9977a4a1d9aa54bd",
|
||||
".agents/skills/agent-fleet/SKILL.md": "25d5aeef4361b3eecff525f752d990acd2ce8988b59f790ba8363eece8e646c7"
|
||||
"docs/agents/prompts/interpreter.md": "776b16799a9516c108b19b22755796706aef3ee9ab98bc5d6ccc33542aae3f19",
|
||||
"docs/agents/prompts/session-manager.md": "c5c7358c439e8593792bd1a14d8dd7edf59e7f5c3f0b1f72b78293910b7c8635",
|
||||
".agents/skills/agent-fleet/SKILL.md": "25d5aeef4361b3eecff525f752d990acd2ce8988b59f790ba8363eece8e646c7",
|
||||
"docs/agents/schedules/resources.md": "1e1236eea30cd2d73f68d2093339cfb9db8a97784b9af3e7ab53edf773541072"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,748 @@
|
||||
// One snapshot of the machine and of each area's process tree.
|
||||
// Missing sensors are "unavailable". Nothing here changes the machine.
|
||||
// The health block in fleet.config.json is guidance for the prompts.
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
|
||||
export const HEALTH_DEFAULTS = {
|
||||
diskFreePct: 10,
|
||||
diskFreeGb: 5,
|
||||
memoryAvailablePct: 15,
|
||||
swapUsedPct: 50,
|
||||
cpuTempC: 90,
|
||||
batteryPct: 20,
|
||||
batteryMinutes: 30,
|
||||
loadPerCore: 1.5,
|
||||
};
|
||||
|
||||
const GB = 1024 ** 3;
|
||||
|
||||
export function healthThresholds(raw) {
|
||||
const src = raw && typeof raw === 'object' && !Array.isArray(raw) ? raw : {};
|
||||
const out = { ...HEALTH_DEFAULTS };
|
||||
for (const key of Object.keys(HEALTH_DEFAULTS)) {
|
||||
if (!Object.prototype.hasOwnProperty.call(src, key)) continue;
|
||||
const n = Number(src[key]);
|
||||
if (Number.isFinite(n) && n >= 0) out[key] = n;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export function parseLoadavg(text) {
|
||||
const m = /^(\d+(?:\.\d+)?)\s+(\d+(?:\.\d+)?)\s+(\d+(?:\.\d+)?)/.exec(String(text || '').trim());
|
||||
if (!m) return null;
|
||||
return [Number(m[1]), Number(m[2]), Number(m[3])];
|
||||
}
|
||||
|
||||
export function parseMeminfo(text) {
|
||||
const map = {};
|
||||
for (const line of String(text || '').split('\n')) {
|
||||
const m = /^(\w+):\s+(\d+)\s*kB/i.exec(line);
|
||||
if (m) map[m[1]] = Number(m[2]) * 1024;
|
||||
}
|
||||
if (map.MemTotal == null) return null;
|
||||
const total = map.MemTotal;
|
||||
const available = map.MemAvailable ?? map.MemFree ?? null;
|
||||
const free = map.MemFree ?? null;
|
||||
const swapTotal = map.SwapTotal ?? 0;
|
||||
const swapFree = map.SwapFree ?? (swapTotal ? 0 : null);
|
||||
return {
|
||||
totalBytes: total,
|
||||
availableBytes: available,
|
||||
freeBytes: free,
|
||||
usedBytes: available != null ? Math.max(0, total - available) : null,
|
||||
swapTotalBytes: map.SwapTotal == null ? null : swapTotal,
|
||||
swapFreeBytes: map.SwapTotal == null ? null : swapFree,
|
||||
swapUsedBytes: map.SwapTotal == null ? null : Math.max(0, swapTotal - (swapFree ?? 0)),
|
||||
};
|
||||
}
|
||||
|
||||
/** /proc/stat "cpu" line, or a sum of os.cpus().times. idle includes iowait. */
|
||||
export function parseCpuStat(text) {
|
||||
const line = String(text || '').split('\n').find((l) => /^cpu\s+/.test(l));
|
||||
if (!line) return null;
|
||||
const n = line.trim().split(/\s+/).slice(1).map(Number);
|
||||
if (n.length < 4 || n.some((x) => !Number.isFinite(x))) return null;
|
||||
const idle = n[3] + (n[4] || 0);
|
||||
const total = n.reduce((a, b) => a + b, 0);
|
||||
return { total, idle };
|
||||
}
|
||||
|
||||
export function cpuTimes(cpus = os.cpus()) {
|
||||
let idle = 0;
|
||||
let total = 0;
|
||||
for (const cpu of cpus || []) {
|
||||
for (const [key, value] of Object.entries(cpu.times || {})) {
|
||||
total += value;
|
||||
if (key === 'idle') idle += value;
|
||||
}
|
||||
}
|
||||
if (total <= 0) return null;
|
||||
return { total, idle };
|
||||
}
|
||||
|
||||
export function cpuUsagePct(before, after) {
|
||||
if (!before || !after) return null;
|
||||
const total = after.total - before.total;
|
||||
const idle = after.idle - before.idle;
|
||||
if (total <= 0) return null;
|
||||
return Math.round((1000 * (total - idle)) / total) / 10;
|
||||
}
|
||||
|
||||
/** One line of /proc/<pid>/stat. rss is pages times pageSize (default 4096). */
|
||||
export function parseProcStat(text, pageSize = 4096) {
|
||||
const s = String(text || '');
|
||||
const end = s.lastIndexOf(')');
|
||||
const space = s.indexOf(' ');
|
||||
if (end < 0 || space < 0) return null;
|
||||
const pid = Number(s.slice(0, space));
|
||||
const rest = s.slice(end + 2).trim().split(/\s+/);
|
||||
if (rest.length < 22) return null;
|
||||
const ppid = Number(rest[1]);
|
||||
const utime = Number(rest[11]);
|
||||
const stime = Number(rest[12]);
|
||||
const rssPages = Number(rest[21]);
|
||||
if (![pid, ppid, utime, stime, rssPages].every(Number.isFinite)) return null;
|
||||
return { pid, ppid, ticks: utime + stime, rssBytes: rssPages * pageSize };
|
||||
}
|
||||
|
||||
export function readProcSnapshot(sysRoot, pageSize = 4096) {
|
||||
const dir = path.join(sysRoot, 'proc');
|
||||
const rows = [];
|
||||
for (const name of safeReaddir(dir)) {
|
||||
if (!/^\d+$/.test(name)) continue;
|
||||
const text = readText(path.join(dir, name, 'stat'));
|
||||
if (!text) continue;
|
||||
const row = parseProcStat(text, pageSize);
|
||||
if (row) rows.push(row);
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
/**
|
||||
* CPU% and RSS for rootPid and every descendant.
|
||||
* ticks are USER_HZ (default 100). A process that appears only in `after`
|
||||
* counts its whole tick count. cpuPct may exceed 100 when several cores are busy.
|
||||
*/
|
||||
export function treeUse(before, after, rootPid, elapsedMs, hz = 100) {
|
||||
const rows = [];
|
||||
const prev = new Map((before || []).map((p) => [p.pid, p]));
|
||||
const elapsedTicks = (Number(elapsedMs) / 1000) * hz;
|
||||
for (const now of after || []) {
|
||||
const was = prev.get(now.pid);
|
||||
const delta = was ? Math.max(0, now.ticks - was.ticks) : now.ticks;
|
||||
const cpuPct = elapsedTicks > 0 ? (100 * delta) / elapsedTicks : null;
|
||||
rows.push({ pid: now.pid, ppid: now.ppid, rssBytes: now.rssBytes, cpuPct });
|
||||
}
|
||||
const sum = sumTree(rows, rootPid);
|
||||
if (!sum.found) return sum;
|
||||
return { ...sum, cpuPct: sum.cpuPct == null ? null : Math.round(sum.cpuPct * 10) / 10 };
|
||||
}
|
||||
|
||||
/** rows: { pid, ppid, cpuPct, rssBytes }. cpuPct null stays null if every row is null. */
|
||||
export function sumTree(rows, rootPid) {
|
||||
const byPid = new Map();
|
||||
const children = new Map();
|
||||
for (const row of rows || []) {
|
||||
byPid.set(row.pid, row);
|
||||
if (!children.has(row.ppid)) children.set(row.ppid, []);
|
||||
children.get(row.ppid).push(row.pid);
|
||||
}
|
||||
const seen = new Set();
|
||||
const stack = [rootPid];
|
||||
let rss = 0;
|
||||
let cpu = 0;
|
||||
let cpuKnown = false;
|
||||
let count = 0;
|
||||
while (stack.length) {
|
||||
const pid = stack.pop();
|
||||
if (seen.has(pid)) continue;
|
||||
seen.add(pid);
|
||||
const row = byPid.get(pid);
|
||||
if (!row) continue;
|
||||
count += 1;
|
||||
rss += row.rssBytes || 0;
|
||||
if (row.cpuPct != null) { cpu += row.cpuPct; cpuKnown = true; }
|
||||
for (const kid of children.get(pid) || []) stack.push(kid);
|
||||
}
|
||||
if (!count) return { found: false, cpuPct: null, rssBytes: null, pids: 0 };
|
||||
return { found: true, cpuPct: cpuKnown ? Math.round(cpu * 10) / 10 : null, rssBytes: rss, pids: count };
|
||||
}
|
||||
|
||||
export function parseDf(text) {
|
||||
const lines = String(text || '').split('\n').map((l) => l.trim()).filter(Boolean);
|
||||
if (!lines.length) return null;
|
||||
const line = lines[lines.length - 1];
|
||||
const parts = line.split(/\s+/);
|
||||
if (parts.length < 6) return null;
|
||||
const total = Number(parts[1]);
|
||||
const used = Number(parts[2]);
|
||||
const free = Number(parts[3]);
|
||||
if (![total, used, free].every((n) => Number.isFinite(n) && n >= 0)) return null;
|
||||
const freePct = total > 0 ? Math.round((1000 * free) / total) / 10 : null;
|
||||
return { mount: parts.slice(5).join(' '), totalBytes: total, usedBytes: used, freeBytes: free, freePct };
|
||||
}
|
||||
|
||||
export function parseVmStat(text) {
|
||||
const page = /page size of (\d+) bytes/i.exec(text || '');
|
||||
const pages = {};
|
||||
for (const line of String(text || '').split('\n')) {
|
||||
const m = /^Pages (.+?):\s+(\d+)/.exec(line.trim());
|
||||
if (m) pages[m[1].toLowerCase()] = Number(m[2]);
|
||||
}
|
||||
return { pageSize: page ? Number(page[1]) : 4096, pages };
|
||||
}
|
||||
|
||||
export function parseSwapUsage(text) {
|
||||
const unit = { K: 1024, M: 1024 ** 2, G: 1024 ** 3, T: 1024 ** 4 };
|
||||
const grab = (name) => {
|
||||
const m = new RegExp(`${name}\\s*=\\s*([\\d.]+)\\s*([KMGT])`, 'i').exec(text || '');
|
||||
if (!m) return null;
|
||||
return Math.round(Number(m[1]) * (unit[m[2].toUpperCase()] || 1));
|
||||
};
|
||||
const total = grab('total');
|
||||
if (total == null) return { totalBytes: null, usedBytes: null, freeBytes: null };
|
||||
return { totalBytes: total, usedBytes: grab('used'), freeBytes: grab('free') };
|
||||
}
|
||||
|
||||
export function parsePmset(text) {
|
||||
const body = String(text || '');
|
||||
if (!body.trim() || /no batteries/i.test(body) && !/\d+\s*%/.test(body)) return emptyBattery();
|
||||
const line = body.split('\n').find((l) => /\d+\s*%/.test(l));
|
||||
if (!line) return emptyBattery();
|
||||
const pct = /(\d+)\s*%/.exec(line);
|
||||
let state = 'unavailable';
|
||||
if (/discharg/i.test(line)) state = 'discharging';
|
||||
else if (/charg(ing|ed)/i.test(line) && !/not charging/i.test(line) && /charging/i.test(line)) state = 'charging';
|
||||
else if (/charged|full/i.test(line)) state = 'full';
|
||||
else if (/not charging/i.test(line)) state = 'not-charging';
|
||||
else if (/AC Power/i.test(body)) state = 'not-charging';
|
||||
else if (/Battery Power/i.test(body)) state = 'discharging';
|
||||
let minutes = null;
|
||||
const tm = /(\d+):(\d+)\s+remaining/.exec(line);
|
||||
if (tm && !/no estimate/i.test(line)) minutes = Number(tm[1]) * 60 + Number(tm[2]);
|
||||
return { present: true, percent: pct ? Number(pct[1]) : null, state, minutesRemaining: minutes };
|
||||
}
|
||||
|
||||
export function parsePs(text) {
|
||||
const rows = [];
|
||||
for (const line of String(text || '').split('\n')) {
|
||||
const parts = line.trim().split(/\s+/);
|
||||
if (parts.length < 4) continue;
|
||||
const pid = Number(parts[0]);
|
||||
const ppid = Number(parts[1]);
|
||||
const rssKb = Number(parts[2]);
|
||||
const cpu = Number(parts[3]);
|
||||
if (![pid, ppid, rssKb, cpu].every(Number.isFinite)) continue;
|
||||
rows.push({ pid, ppid, rssBytes: rssKb * 1024, cpuPct: cpu });
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
function emptyBattery() {
|
||||
return { present: false, percent: null, state: 'unavailable', minutesRemaining: null };
|
||||
}
|
||||
|
||||
function emptyMemory() {
|
||||
return {
|
||||
status: 'unavailable', totalBytes: null, usedBytes: null, freeBytes: null,
|
||||
availableBytes: null, availablePct: null, swapTotalBytes: null, swapUsedBytes: null, swapFreeBytes: null,
|
||||
};
|
||||
}
|
||||
|
||||
function withMemoryPct(mem) {
|
||||
if (!mem || mem.totalBytes == null) return emptyMemory();
|
||||
const availablePct = mem.availableBytes != null && mem.totalBytes > 0
|
||||
? Math.round((1000 * mem.availableBytes) / mem.totalBytes) / 10
|
||||
: null;
|
||||
return { status: 'ok', ...mem, availablePct };
|
||||
}
|
||||
|
||||
function readText(file) {
|
||||
try { return fs.readFileSync(file, 'utf8'); } catch { return null; }
|
||||
}
|
||||
|
||||
function readNum(file) {
|
||||
const text = readText(file);
|
||||
if (text == null) return null;
|
||||
const n = Number(String(text).trim());
|
||||
return Number.isFinite(n) ? n : null;
|
||||
}
|
||||
|
||||
function safeReaddir(dir) {
|
||||
try { return fs.readdirSync(dir); } catch { return []; }
|
||||
}
|
||||
|
||||
function toC(raw) {
|
||||
if (raw == null || !Number.isFinite(raw)) return null;
|
||||
const c = Math.abs(raw) >= 1000 ? raw / 1000 : raw;
|
||||
if (c <= 0 || c > 150) return null;
|
||||
return Math.round(c * 10) / 10;
|
||||
}
|
||||
|
||||
function isCpuName(name) {
|
||||
return /cpu|package|pkg|core|soc|k10temp|zenpower|tdie|tctl/i.test(name || '');
|
||||
}
|
||||
|
||||
export function readLinuxTemps(sysRoot) {
|
||||
const readings = [];
|
||||
const thermal = path.join(sysRoot, 'sys/class/thermal');
|
||||
for (const name of safeReaddir(thermal)) {
|
||||
if (!name.startsWith('thermal_zone')) continue;
|
||||
const dir = path.join(thermal, name);
|
||||
const celsius = toC(readNum(path.join(dir, 'temp')));
|
||||
if (celsius == null) continue;
|
||||
readings.push({ source: 'thermal', name: readText(path.join(dir, 'type'))?.trim() || name, celsius });
|
||||
}
|
||||
const hwmon = path.join(sysRoot, 'sys/class/hwmon');
|
||||
for (const name of safeReaddir(hwmon)) {
|
||||
const dir = path.join(hwmon, name);
|
||||
const chip = readText(path.join(dir, 'name'))?.trim() || name;
|
||||
for (const file of safeReaddir(dir)) {
|
||||
const m = /^temp(\d+)_input$/.exec(file);
|
||||
if (!m) continue;
|
||||
const celsius = toC(readNum(path.join(dir, file)));
|
||||
if (celsius == null) continue;
|
||||
const label = readText(path.join(dir, `temp${m[1]}_label`))?.trim();
|
||||
readings.push({ source: 'hwmon', name: label ? `${chip} ${label}` : chip, celsius });
|
||||
}
|
||||
}
|
||||
const cpuish = readings.filter((r) => isCpuName(r.name));
|
||||
const rest = readings.filter((r) => !isCpuName(r.name));
|
||||
const kept = [...cpuish, ...rest].slice(0, 12);
|
||||
return kept.length ? { status: 'ok', readings: kept } : { status: 'unavailable', readings: [] };
|
||||
}
|
||||
|
||||
function minutesBetween(now, full, rate) {
|
||||
if (now == null || rate == null || rate <= 0) return null;
|
||||
return Math.max(0, Math.round((Math.abs(full == null ? now : full - now) / rate) * 60));
|
||||
}
|
||||
|
||||
export function readLinuxBattery(sysRoot) {
|
||||
const root = path.join(sysRoot, 'sys/class/power_supply');
|
||||
const names = safeReaddir(root).filter((n) => /^(BAT|battery)/i.test(n)).sort();
|
||||
const parsed = [];
|
||||
for (const name of names) {
|
||||
const dir = path.join(root, name);
|
||||
const status = (readText(path.join(dir, 'status')) || '').trim().toLowerCase();
|
||||
const state = status === 'charging' ? 'charging'
|
||||
: status === 'discharging' ? 'discharging'
|
||||
: status === 'full' ? 'full'
|
||||
: status === 'not charging' ? 'not-charging'
|
||||
: 'unavailable';
|
||||
let percent = readNum(path.join(dir, 'capacity'));
|
||||
const energy = readNum(path.join(dir, 'energy_now'));
|
||||
const energyFull = readNum(path.join(dir, 'energy_full'));
|
||||
const power = readNum(path.join(dir, 'power_now'));
|
||||
const charge = readNum(path.join(dir, 'charge_now'));
|
||||
const chargeFull = readNum(path.join(dir, 'charge_full'));
|
||||
const current = readNum(path.join(dir, 'current_now'));
|
||||
if (percent == null && energy != null && energyFull > 0) percent = Math.round((100 * energy) / energyFull);
|
||||
if (percent == null && charge != null && chargeFull > 0) percent = Math.round((100 * charge) / chargeFull);
|
||||
if (percent == null && state === 'unavailable') continue;
|
||||
let minutes = null;
|
||||
if (state === 'discharging') minutes = energy != null ? minutesBetween(energy, null, power) : minutesBetween(charge, null, current);
|
||||
else if (state === 'charging') minutes = energy != null ? minutesBetween(energy, energyFull, power) : minutesBetween(charge, chargeFull, current);
|
||||
parsed.push({ percent, state, minutesRemaining: minutes });
|
||||
}
|
||||
if (!parsed.length) return emptyBattery();
|
||||
const pick = parsed.find((b) => b.state === 'discharging') || parsed[0];
|
||||
return { present: true, count: parsed.length, ...pick };
|
||||
}
|
||||
|
||||
function sh(cmd, args, timeout = 2500) {
|
||||
try {
|
||||
const r = spawnSync(cmd, args, { encoding: 'utf8', timeout });
|
||||
if (r.status !== 0 || r.error) return null;
|
||||
return r.stdout || '';
|
||||
} catch { return null; }
|
||||
}
|
||||
|
||||
function pause(ms) {
|
||||
const n = Number(ms);
|
||||
if (!Number.isFinite(n) || n <= 0) return;
|
||||
const r = spawnSync('sleep', [(n / 1000).toFixed(3)], { timeout: n + 2000 });
|
||||
if (r.error || r.status !== 0) {
|
||||
const end = Date.now() + n;
|
||||
while (Date.now() < end) { /* sleep(1) missing */ }
|
||||
}
|
||||
}
|
||||
|
||||
function pageSize() {
|
||||
const text = sh('getconf', ['PAGE_SIZE'], 1000);
|
||||
const n = Number(String(text || '').trim());
|
||||
return Number.isFinite(n) && n > 0 ? n : 4096;
|
||||
}
|
||||
|
||||
function userHz() {
|
||||
const text = sh('getconf', ['CLK_TCK'], 1000);
|
||||
const n = Number(String(text || '').trim());
|
||||
return Number.isFinite(n) && n > 0 ? n : 100;
|
||||
}
|
||||
|
||||
function existingPath(target) {
|
||||
let dir = target;
|
||||
while (dir) {
|
||||
try { if (fs.existsSync(dir)) return dir; } catch { return null; }
|
||||
const parent = path.dirname(dir);
|
||||
if (parent === dir) return null;
|
||||
dir = parent;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function dfLive(target, platform) {
|
||||
const gnu = sh('df', ['-B1', '-P', target], 3000);
|
||||
if (gnu && /1B-blocks|1-blocks/.test(gnu)) return parseDf(gnu);
|
||||
const bsd = sh('df', ['-k', '-P', target], 3000);
|
||||
if (bsd) {
|
||||
const parsed = parseDf(bsd);
|
||||
if (parsed) {
|
||||
const scale = /512-blocks/.test(bsd) ? 512 : 1024;
|
||||
return {
|
||||
mount: parsed.mount,
|
||||
totalBytes: parsed.totalBytes * scale,
|
||||
usedBytes: parsed.usedBytes * scale,
|
||||
freeBytes: parsed.freeBytes * scale,
|
||||
freePct: parsed.freePct,
|
||||
};
|
||||
}
|
||||
}
|
||||
if (gnu) return parseDf(gnu);
|
||||
if (platform === 'win32') return winDisk(target);
|
||||
return null;
|
||||
}
|
||||
|
||||
function winDisk(target) {
|
||||
const script = `$p=${JSON.stringify(target)}; $q=Split-Path -Qualifier $p; if (-not $q) { $q='C:' }; $d=New-Object System.IO.DriveInfo($q); if (-not $d.IsReady) { exit 1 }; Write-Output ($d.TotalSize.ToString()+' '+($d.TotalSize-$d.AvailableFreeSpace).ToString()+' '+$d.AvailableFreeSpace.ToString()+' '+$d.Name)`;
|
||||
const text = sh('powershell', ['-NoProfile', '-Command', script], 4000);
|
||||
if (!text) return null;
|
||||
const parts = text.trim().split(/\s+/);
|
||||
if (parts.length < 4) return null;
|
||||
const total = Number(parts[0]);
|
||||
const used = Number(parts[1]);
|
||||
const free = Number(parts[2]);
|
||||
if (![total, used, free].every(Number.isFinite)) return null;
|
||||
return { mount: parts.slice(3).join(' '), totalBytes: total, usedBytes: used, freeBytes: free, freePct: total ? Math.round((1000 * free) / total) / 10 : null };
|
||||
}
|
||||
|
||||
function detectWsl(sysRoot, opt) {
|
||||
if (opt.wsl != null) return Boolean(opt.wsl);
|
||||
const version = readText(path.join(sysRoot, 'proc/version'));
|
||||
if (version) return /microsoft|wsl/i.test(version);
|
||||
return Boolean(process.env.WSL_DISTRO_NAME || process.env.WSL_INTEROP);
|
||||
}
|
||||
|
||||
function diskTargets(c) {
|
||||
return [
|
||||
['repo', c?.repo || '/'],
|
||||
['worktree', c?.worktreeRoot || c?.repo || '/'],
|
||||
['state', c?.stateDir || '/'],
|
||||
['tmp', os.tmpdir()],
|
||||
];
|
||||
}
|
||||
|
||||
function readDisks(c, platform, opt) {
|
||||
return diskTargets(c).map(([label, requested]) => {
|
||||
const parsed = opt.df
|
||||
? opt.df(requested)
|
||||
: dfLive(existingPath(requested) || requested, platform);
|
||||
if (!parsed || parsed.freeBytes == null) {
|
||||
return { label, path: requested, mount: null, totalBytes: null, usedBytes: null, freeBytes: null, freePct: null, status: 'unavailable' };
|
||||
}
|
||||
return { label, path: requested, status: 'ok', mount: parsed.mount ?? null, totalBytes: parsed.totalBytes, usedBytes: parsed.usedBytes, freeBytes: parsed.freeBytes, freePct: parsed.freePct };
|
||||
});
|
||||
}
|
||||
|
||||
function readLinuxMemory(sysRoot) {
|
||||
return withMemoryPct(parseMeminfo(readText(path.join(sysRoot, 'proc/meminfo')) || ''));
|
||||
}
|
||||
|
||||
function readGenericMemory() {
|
||||
const total = os.totalmem();
|
||||
const free = os.freemem();
|
||||
if (!total) return emptyMemory();
|
||||
return withMemoryPct({
|
||||
totalBytes: total,
|
||||
availableBytes: free,
|
||||
freeBytes: free,
|
||||
usedBytes: Math.max(0, total - free),
|
||||
swapTotalBytes: null,
|
||||
swapFreeBytes: null,
|
||||
swapUsedBytes: null,
|
||||
});
|
||||
}
|
||||
|
||||
function readDarwinMemory() {
|
||||
const total = Number(String(sh('sysctl', ['-n', 'hw.memsize']) || '').trim());
|
||||
const vm = parseVmStat(sh('vm_stat', []) || '');
|
||||
const swap = parseSwapUsage(sh('sysctl', ['-n', 'vm.swapusage']) || '');
|
||||
if (!Number.isFinite(total) || total <= 0) return readGenericMemory();
|
||||
const page = vm.pageSize || 4096;
|
||||
const availablePages = (vm.pages.free || 0) + (vm.pages.inactive || 0) + (vm.pages.speculative || 0);
|
||||
const available = Math.min(total, availablePages * page);
|
||||
return withMemoryPct({
|
||||
totalBytes: total,
|
||||
availableBytes: available,
|
||||
freeBytes: (vm.pages.free || 0) * page,
|
||||
usedBytes: Math.max(0, total - available),
|
||||
swapTotalBytes: swap.totalBytes,
|
||||
swapFreeBytes: swap.freeBytes,
|
||||
swapUsedBytes: swap.usedBytes,
|
||||
});
|
||||
}
|
||||
|
||||
function readDarwinBattery() {
|
||||
const text = sh('pmset', ['-g', 'batt'], 3000);
|
||||
if (!text) return emptyBattery();
|
||||
return parsePmset(text);
|
||||
}
|
||||
|
||||
function readWindowsBattery() {
|
||||
const text = sh('powershell', ['-NoProfile', '-Command', '(Get-CimInstance Win32_Battery -ErrorAction SilentlyContinue | Select-Object -First 1 EstimatedChargeRemaining,BatteryStatus | ConvertTo-Json -Compress)'], 4000);
|
||||
if (!text || !text.trim() || text.trim() === 'null') return emptyBattery();
|
||||
try {
|
||||
const row = JSON.parse(text);
|
||||
if (!row || row.EstimatedChargeRemaining == null) return emptyBattery();
|
||||
const code = Number(row.BatteryStatus);
|
||||
const state = [6, 7, 8, 9].includes(code) ? 'charging'
|
||||
: code === 1 || code === 4 || code === 5 ? 'discharging'
|
||||
: code === 2 || code === 3 ? 'full'
|
||||
: 'unavailable';
|
||||
return { present: true, percent: Number(row.EstimatedChargeRemaining), state, minutesRemaining: null };
|
||||
} catch { return emptyBattery(); }
|
||||
}
|
||||
|
||||
function linuxSample(sysRoot, opt) {
|
||||
if (opt.statTexts || opt.procSamples) {
|
||||
return {
|
||||
usagePct: opt.statTexts ? cpuUsagePct(parseCpuStat(opt.statTexts[0]), parseCpuStat(opt.statTexts[1])) : null,
|
||||
before: opt.procSamples ? opt.procSamples[0] : [],
|
||||
after: opt.procSamples ? opt.procSamples[1] : [],
|
||||
elapsedMs: opt.elapsedMs ?? 1000,
|
||||
};
|
||||
}
|
||||
if (opt.sampleMs === 0) return { usagePct: null, before: [], after: [], elapsedMs: 0 };
|
||||
const stat1 = parseCpuStat(readText(path.join(sysRoot, 'proc/stat')) || '');
|
||||
const pages = pageSize();
|
||||
const before = readProcSnapshot(sysRoot, pages);
|
||||
const t0 = Date.now();
|
||||
pause(opt.sampleMs ?? 200);
|
||||
const stat2 = parseCpuStat(readText(path.join(sysRoot, 'proc/stat')) || '');
|
||||
const after = readProcSnapshot(sysRoot, pages);
|
||||
let usagePct = cpuUsagePct(stat1, stat2);
|
||||
if (usagePct == null) {
|
||||
const a = cpuTimes();
|
||||
pause(opt.sampleMs ?? 200);
|
||||
usagePct = cpuUsagePct(a, cpuTimes());
|
||||
}
|
||||
return { usagePct, before, after, elapsedMs: Math.max(1, Date.now() - t0) };
|
||||
}
|
||||
|
||||
function readLoad(platform, sysRoot) {
|
||||
if (platform === 'linux') {
|
||||
const parsed = parseLoadavg(readText(path.join(sysRoot, 'proc/loadavg')) || '');
|
||||
if (parsed) return parsed;
|
||||
const live = os.loadavg();
|
||||
return live.some((n) => n > 0) ? live : [null, null, null];
|
||||
}
|
||||
if (platform === 'win32') return [null, null, null];
|
||||
return os.loadavg();
|
||||
}
|
||||
|
||||
function listPanes(c, opt) {
|
||||
if (opt.panes) return opt.panes;
|
||||
const session = `fleet-${c?.project || 'fleet'}`.replace(/[.:]/g, '-');
|
||||
return (c?.areas || []).map((area) => {
|
||||
const r = sh('tmux', ['list-panes', '-t', `${session}:${area.id}`, '-F', '#{pane_pid}'], 2000);
|
||||
const pid = Number(String(r || '').trim().split(/\s+/)[0]);
|
||||
return { id: area.id, pid: Number.isInteger(pid) && pid > 0 ? pid : null };
|
||||
});
|
||||
}
|
||||
|
||||
function areasFromTree(panes, before, after, elapsedMs, hz) {
|
||||
return (panes || []).map((pane) => {
|
||||
if (!pane.pid) return { id: pane.id, found: false, cpuPct: null, rssBytes: null, pids: 0 };
|
||||
return { id: pane.id, ...treeUse(before, after, pane.pid, elapsedMs, hz) };
|
||||
});
|
||||
}
|
||||
|
||||
function areasFromPs(panes, rows) {
|
||||
return (panes || []).map((pane) => {
|
||||
if (!pane.pid) return { id: pane.id, found: false, cpuPct: null, rssBytes: null, pids: 0 };
|
||||
return { id: pane.id, ...sumTree(rows, pane.pid) };
|
||||
});
|
||||
}
|
||||
|
||||
function readPsAreas(panes, opt) {
|
||||
const text = opt.psText != null ? opt.psText : sh('ps', ['-ax', '-o', 'pid=,ppid=,rss=,%cpu='], 3000);
|
||||
if (!text) return (panes || []).map((pane) => ({ id: pane.id, found: Boolean(pane.pid), cpuPct: null, rssBytes: null, pids: pane.pid ? null : 0 }));
|
||||
return areasFromPs(panes, parsePs(text));
|
||||
}
|
||||
|
||||
function fmtBytes(bytes) {
|
||||
if (bytes == null) return 'unavailable';
|
||||
const gb = bytes / GB;
|
||||
if (gb >= 1) return `${Math.round(gb * 10) / 10} GB`;
|
||||
return `${Math.round(bytes / (1024 ** 2))} MB`;
|
||||
}
|
||||
|
||||
function judge(snap, limits) {
|
||||
const crossed = [];
|
||||
const mem = snap.memory;
|
||||
if (mem?.status === 'ok' && mem.availableBytes != null && mem.totalBytes > 0) {
|
||||
const pct = (100 * mem.availableBytes) / mem.totalBytes;
|
||||
if (pct < limits.memoryAvailablePct) {
|
||||
crossed.push({ id: 'memory', words: `memory available ${Math.round(pct)}% (${fmtBytes(mem.availableBytes)} of ${fmtBytes(mem.totalBytes)}), under ${limits.memoryAvailablePct}%` });
|
||||
}
|
||||
}
|
||||
if (mem?.swapTotalBytes > 0 && mem.swapUsedBytes != null) {
|
||||
const pct = (100 * mem.swapUsedBytes) / mem.swapTotalBytes;
|
||||
if (pct >= limits.swapUsedPct) {
|
||||
crossed.push({ id: 'swap', words: `swap ${Math.round(pct)}% used, at or over ${limits.swapUsedPct}%` });
|
||||
}
|
||||
}
|
||||
const seen = new Set();
|
||||
for (const disk of snap.disks || []) {
|
||||
if (disk.status !== 'ok' || disk.freeBytes == null) continue;
|
||||
const key = disk.mount || disk.path;
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
const freeGb = disk.freeBytes / GB;
|
||||
const lowPct = disk.freePct != null && disk.freePct < limits.diskFreePct;
|
||||
const lowGb = freeGb < limits.diskFreeGb;
|
||||
if (!lowPct && !lowGb) continue;
|
||||
const labels = snap.disks.filter((row) => row.status === 'ok' && (row.mount || row.path) === key).map((row) => row.label).join(', ');
|
||||
const pct = disk.freePct == null ? '' : ` (${Math.round(disk.freePct)}%)`;
|
||||
crossed.push({ id: `disk:${key}`, words: `${labels} on ${key}: ${fmtBytes(disk.freeBytes)} free${pct}, under ${limits.diskFreeGb} GB or ${limits.diskFreePct}%` });
|
||||
}
|
||||
const readings = snap.temps?.readings || [];
|
||||
const cpuish = readings.filter((r) => isCpuName(r.name));
|
||||
const pool = cpuish.length ? cpuish : readings;
|
||||
if (pool.length) {
|
||||
const hot = pool.reduce((a, b) => (b.celsius > a.celsius ? b : a));
|
||||
if (hot.celsius >= limits.cpuTempC) {
|
||||
crossed.push({ id: 'temp', words: `${hot.name} ${hot.celsius}°C, at or over ${limits.cpuTempC}°C` });
|
||||
}
|
||||
}
|
||||
const battery = snap.battery;
|
||||
if (battery?.present && battery.percent != null && battery.percent <= limits.batteryPct && battery.state !== 'charging' && battery.state !== 'full') {
|
||||
const how = battery.state === 'discharging' ? ' and discharging' : '';
|
||||
crossed.push({ id: 'battery', words: `battery ${battery.percent}%${how}, at or under ${limits.batteryPct}%` });
|
||||
} else if (battery?.present && battery.state === 'discharging' && battery.minutesRemaining != null && battery.minutesRemaining <= limits.batteryMinutes) {
|
||||
crossed.push({ id: 'battery', words: `battery discharging with about ${battery.minutesRemaining} min left, at or under ${limits.batteryMinutes} min` });
|
||||
}
|
||||
if (snap.cpu?.load1 != null && snap.cpu.cores) {
|
||||
const per = snap.cpu.load1 / snap.cpu.cores;
|
||||
if (per >= limits.loadPerCore) {
|
||||
crossed.push({ id: 'load', words: `load ${snap.cpu.load1} on ${snap.cpu.cores} cores (${Math.round(per * 10) / 10} per core), at or over ${limits.loadPerCore}` });
|
||||
}
|
||||
}
|
||||
return crossed;
|
||||
}
|
||||
|
||||
function verdictOf(crossed, battery) {
|
||||
const onBattery = battery?.present && battery.state === 'discharging';
|
||||
if (!crossed.length && onBattery) {
|
||||
const left = battery.minutesRemaining == null ? 'time left unavailable' : `about ${battery.minutesRemaining} min left`;
|
||||
const pct = battery.percent == null ? 'unknown %' : `${battery.percent}%`;
|
||||
return `ok: on battery at ${pct}, ${left}; nothing crossed the suggested limits`;
|
||||
}
|
||||
if (!crossed.length) return 'ok: nothing crossed the suggested limits';
|
||||
return `tight: ${crossed.map((row) => row.words).join('; ')}`;
|
||||
}
|
||||
|
||||
export function formatHealth(snap) {
|
||||
const lines = [snap.verdict];
|
||||
const cpu = snap.cpu || {};
|
||||
const load = cpu.load1 == null ? 'load unavailable' : `load ${cpu.load1} ${cpu.load5} ${cpu.load15}`;
|
||||
const usage = cpu.usagePct == null ? 'usage unavailable' : `usage ${cpu.usagePct}%`;
|
||||
lines.push(`cpu ${load} ${cpu.cores ?? '?'} cores ${usage}${snap.wsl ? ' wsl' : ''}`);
|
||||
const mem = snap.memory;
|
||||
if (!mem || mem.status !== 'ok') lines.push('mem unavailable');
|
||||
else {
|
||||
const swap = mem.swapTotalBytes == null ? 'swap unavailable' : mem.swapTotalBytes === 0 ? 'swap none' : `swap ${fmtBytes(mem.swapUsedBytes)} / ${fmtBytes(mem.swapTotalBytes)}`;
|
||||
lines.push(`mem ${fmtBytes(mem.usedBytes)} used / ${fmtBytes(mem.totalBytes)} available ${fmtBytes(mem.availableBytes)} (${mem.availablePct ?? '?'}%) ${swap}`);
|
||||
}
|
||||
for (const disk of snap.disks || []) {
|
||||
if (disk.status !== 'ok') lines.push(`disk ${disk.label} unavailable ${disk.path}`);
|
||||
else lines.push(`disk ${disk.label} ${disk.mount} ${fmtBytes(disk.freeBytes)} free (${disk.freePct}%) ${disk.path}`);
|
||||
}
|
||||
if (!snap.temps || snap.temps.status !== 'ok') lines.push('temp unavailable');
|
||||
else lines.push(`temp ${snap.temps.readings.map((r) => `${r.name} ${r.celsius}°C`).join(' ')}`);
|
||||
const battery = snap.battery || emptyBattery();
|
||||
if (!battery.present) lines.push('battery none');
|
||||
else {
|
||||
const left = battery.minutesRemaining == null ? 'time unavailable' : `${battery.minutesRemaining} min left`;
|
||||
lines.push(`battery ${battery.percent ?? '?'}% ${battery.state} ${left}`);
|
||||
}
|
||||
if (!snap.areas?.length) lines.push('areas none');
|
||||
for (const area of snap.areas || []) {
|
||||
if (!area.found) lines.push(`area ${area.id} window not found`);
|
||||
else if (area.cpuPct == null && area.rssBytes == null) lines.push(`area ${area.id} unavailable`);
|
||||
else lines.push(`area ${area.id} cpu ${area.cpuPct ?? '?'}% rss ${fmtBytes(area.rssBytes)} ${area.pids ?? '?'} procs`);
|
||||
}
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
export function fleetHealth(c = {}, opt = {}) {
|
||||
const platform = opt.platform || process.platform;
|
||||
const sysRoot = opt.sysRoot || '/';
|
||||
const limits = healthThresholds(c.health);
|
||||
const wsl = platform === 'linux' ? detectWsl(sysRoot, opt) : false;
|
||||
const cores = opt.cores ?? (os.cpus()?.length || null);
|
||||
const needsLinuxSample = platform === 'linux' && (opt.cpu == null || opt.areas == null);
|
||||
const sampled = needsLinuxSample ? linuxSample(sysRoot, opt) : null;
|
||||
const load = readLoad(platform, sysRoot);
|
||||
const cpu = opt.cpu ?? {
|
||||
load1: load[0],
|
||||
load5: load[1],
|
||||
load15: load[2],
|
||||
cores,
|
||||
usagePct: platform === 'linux'
|
||||
? sampled?.usagePct ?? null
|
||||
: (opt.sampleMs === 0 ? null : cpuUsagePct(cpuTimes(), (pause(opt.sampleMs ?? 200), cpuTimes()))),
|
||||
};
|
||||
const memory = opt.memory
|
||||
? withMemoryPct(opt.memory.status === 'unavailable' ? null : opt.memory)
|
||||
: platform === 'linux' ? readLinuxMemory(sysRoot)
|
||||
: platform === 'darwin' ? readDarwinMemory()
|
||||
: readGenericMemory();
|
||||
const disks = opt.disks ?? readDisks(c, platform, opt);
|
||||
const temps = opt.temps ?? (platform === 'linux' ? readLinuxTemps(sysRoot) : { status: 'unavailable', readings: [] });
|
||||
const battery = opt.battery ?? (
|
||||
platform === 'linux' ? readLinuxBattery(sysRoot)
|
||||
: platform === 'darwin' ? readDarwinBattery()
|
||||
: platform === 'win32' ? readWindowsBattery()
|
||||
: emptyBattery()
|
||||
);
|
||||
let areas = opt.areas;
|
||||
if (areas == null) {
|
||||
const panes = listPanes(c, opt);
|
||||
if (platform === 'linux' && sampled && (sampled.before.length || sampled.after.length || opt.procSamples || opt.sampleMs === 0)) {
|
||||
areas = areasFromTree(panes, sampled.before, sampled.after, sampled.elapsedMs, opt.hz ?? userHz());
|
||||
} else if (platform === 'linux' && opt.sampleMs !== 0) {
|
||||
areas = areasFromTree(panes, sampled?.before || [], sampled?.after || [], sampled?.elapsedMs || 0, opt.hz ?? userHz());
|
||||
if (areas.every((a) => !a.found) && panes.some((p) => p.pid) && !opt.procSamples) areas = readPsAreas(panes, opt);
|
||||
} else {
|
||||
areas = readPsAreas(panes, opt);
|
||||
}
|
||||
}
|
||||
const snap = {
|
||||
at: opt.at || new Date().toISOString(),
|
||||
platform,
|
||||
wsl,
|
||||
cpu,
|
||||
memory,
|
||||
disks,
|
||||
temps,
|
||||
battery,
|
||||
areas,
|
||||
thresholds: limits,
|
||||
};
|
||||
snap.crossed = judge(snap, limits);
|
||||
snap.verdict = verdictOf(snap.crossed, snap.battery);
|
||||
return snap;
|
||||
}
|
||||
+183
-25
@@ -2,8 +2,8 @@
|
||||
// both call these. Do not copy this logic into a plugin or a second server.
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { area as getArea, die, pidAlive, tryGit, git, mainRef, writeAtomic, today, landerIsUp, matchesAny, startLander, runTimedSync } from '../lib.mjs';
|
||||
import { spawn, spawnSync } from 'node:child_process';
|
||||
import { KIT, area as getArea, die, pidAlive, tryGit, git, mainRef, writeAtomic, today, landerIsUp, matchesAny, startLander, runTimedSync } from '../lib.mjs';
|
||||
import { send as mailboxSendFile, read as mailboxReadDir, archive as mailboxArchiveDir } from '../../adapters/mailbox.mjs';
|
||||
import { appendWakeLog, priorWakes } from './wake-log.mjs';
|
||||
|
||||
@@ -210,11 +210,14 @@ export function hearInterpreter(c) {
|
||||
return readMailbox(c, 'interpreter');
|
||||
}
|
||||
|
||||
const NUDGE_PAUSE_MS = 200;
|
||||
const NUDGE_SETTLE_MS = 150;
|
||||
const NUDGE_RETRY_MS = 400;
|
||||
// Backoff before each Enter. Codex's paste window outlasts a single short pause.
|
||||
// 250 + 800 + 2000, plus a short settle after each Enter, stays under 4s.
|
||||
const ENTER_DELAYS_MS = [250, 800, 2000];
|
||||
const ENTER_SETTLE_MS = 150;
|
||||
const IDLE_AFTER_MS = 3000;
|
||||
const RECOVER_AFTER_MS = 120000;
|
||||
const WAKE_WAIT_POLL_MS = 3000;
|
||||
const WAKE_WAIT_MAX_MS = 600000;
|
||||
const BUSY_RE = /esc to interrupt|esc to cancel|ctrl\+c to (?:interrupt|cancel)/i;
|
||||
|
||||
export function nudgeLine(from) {
|
||||
@@ -269,12 +272,118 @@ function paneTail(pane) {
|
||||
return rows.map((s) => s.slice(0, 120)).join(' | ').slice(0, 400);
|
||||
}
|
||||
|
||||
function snapBefore(before) {
|
||||
return { found: Boolean(before?.found), dead: Boolean(before?.dead), interrupt: Boolean(before?.interrupt), idleSec: before?.idleSec ?? null };
|
||||
}
|
||||
|
||||
/** Window names in the fleet session, so a miss says what was there. */
|
||||
function sessionWindows(c) {
|
||||
const session = `fleet-${c.project}`.replace(/[.:]/g, '-');
|
||||
const has = tmuxText(['has-session', '-t', session]);
|
||||
const listed = tmuxText(['list-windows', '-t', session, '-F', '#{window_index} #{window_name}']);
|
||||
return {
|
||||
session,
|
||||
sessionUp: has.status === 0,
|
||||
windows: listed.status === 0 ? listed.stdout.split('\n').map((s) => s.trim()).filter(Boolean) : [],
|
||||
listError: listed.status === 0 ? '' : String(listed.stderr || has.stderr || '').trim().slice(0, 200),
|
||||
};
|
||||
}
|
||||
|
||||
function wakeLockPath(c, areaId) {
|
||||
return path.join(c.stateDir, 'wake-wait', `${areaId}.lock`);
|
||||
}
|
||||
|
||||
function claimWakeLock(lock) {
|
||||
fs.mkdirSync(path.dirname(lock), { recursive: true });
|
||||
const body = `${JSON.stringify({ pid: process.pid, starting: true, t: new Date().toISOString() })}\n`;
|
||||
const create = () => {
|
||||
const fd = fs.openSync(lock, 'wx');
|
||||
fs.writeSync(fd, body);
|
||||
fs.closeSync(fd);
|
||||
};
|
||||
try {
|
||||
create();
|
||||
return true;
|
||||
} catch (e) {
|
||||
if (e.code !== 'EEXIST') return false;
|
||||
let cur = {};
|
||||
try { cur = JSON.parse(fs.readFileSync(lock, 'utf8')); } catch { /* empty or torn */ }
|
||||
if (cur.pid && pidAlive(cur.pid)) return false;
|
||||
try { fs.unlinkSync(lock); } catch { return false; }
|
||||
try { create(); return true; } catch { return false; }
|
||||
}
|
||||
}
|
||||
|
||||
function handDeferred(c, areaId, from, file) {
|
||||
const lock = wakeLockPath(c, areaId);
|
||||
if (!claimWakeLock(lock)) return { started: false };
|
||||
const poll = Number(c.wakeWait?.pollMs) > 0 ? Number(c.wakeWait.pollMs) : WAKE_WAIT_POLL_MS;
|
||||
const max = Number(c.wakeWait?.maxMs) > 0 ? Number(c.wakeWait.maxMs) : WAKE_WAIT_MAX_MS;
|
||||
const child = spawn(process.execPath, [
|
||||
path.join(KIT, 'bin/core/wake-wait.mjs'),
|
||||
'--config', c.file,
|
||||
'--area', areaId,
|
||||
'--from', from,
|
||||
'--file', file || '',
|
||||
'--poll-ms', String(poll),
|
||||
'--max-ms', String(max),
|
||||
], { detached: true, stdio: 'ignore' });
|
||||
if (!child.pid) {
|
||||
try { fs.unlinkSync(lock); } catch { /* already gone */ }
|
||||
return { started: false };
|
||||
}
|
||||
fs.writeFileSync(lock, `${JSON.stringify({ pid: child.pid, area: areaId, file: file || null, from, t: new Date().toISOString() })}\n`);
|
||||
child.unref();
|
||||
return { started: true, pid: child.pid };
|
||||
}
|
||||
|
||||
function releaseWakeLock(c, areaId) {
|
||||
try { fs.unlinkSync(wakeLockPath(c, areaId)); } catch { /* already gone */ }
|
||||
}
|
||||
|
||||
/**
|
||||
* Type the wake line and press Enter until the cursor row no longer holds it.
|
||||
* One literal send-keys -l, then Enter on its own. The fake pane does not
|
||||
* treat chunked keys differently, so the backoff is what survives a long paste.
|
||||
*/
|
||||
function submitWake(c, areaId, from, meta, before) {
|
||||
const target = tmuxTarget(c, areaId);
|
||||
const line = nudgeLine(from);
|
||||
const record = {
|
||||
t: new Date().toISOString(),
|
||||
area: areaId,
|
||||
window: target,
|
||||
file: meta.file || null,
|
||||
trigger: meta.trigger || 'send',
|
||||
before: snapBefore(before),
|
||||
};
|
||||
const finish = (fields) => {
|
||||
const { line: _line, reason, ...logged } = fields;
|
||||
appendWakeLog(c, { ...record, ...logged, tail: paneTail(capturePane(target)) });
|
||||
return { submitted: logged.outcome === 'submitted', reason, line };
|
||||
};
|
||||
if (tmuxText(['send-keys', '-t', target, '-l', line]).status !== 0) {
|
||||
return finish({ action: 'send-failed', enterAttempts: 0, enterDelays: [], outcome: 'send-failed', reason: 'send-failed', line });
|
||||
}
|
||||
const enterDelays = [];
|
||||
let enterAttempts = 0;
|
||||
for (const delay of ENTER_DELAYS_MS) {
|
||||
sleepMs(delay);
|
||||
enterDelays.push(delay);
|
||||
tmuxText(['send-keys', '-t', target, 'Enter']);
|
||||
enterAttempts += 1;
|
||||
sleepMs(ENTER_SETTLE_MS);
|
||||
if (!stillInInput(target, line)) {
|
||||
return finish({ action: 'typed', enterAttempts, enterDelays, outcome: 'submitted', reason: 'submitted', line });
|
||||
}
|
||||
}
|
||||
return finish({ action: 'typed', enterAttempts, enterDelays, outcome: 'swallowed', reason: 'not-submitted', line });
|
||||
}
|
||||
|
||||
/**
|
||||
* Wake an open tmux window with a short line, not the mail body.
|
||||
* Codex and Claude treat a fast burst as a paste and swallow Enter, so the
|
||||
* line is typed, then Enter is sent after a pause, outside that burst.
|
||||
* If the cursor row still holds the line, Enter is sent once more.
|
||||
* A pane that shows an interrupt hint is left alone; a later status check can nudge it.
|
||||
* Enter is retried with backoff until the line leaves the composer.
|
||||
* A pane showing an interrupt hint is handed to one detached waiter.
|
||||
*/
|
||||
export function nudgeMail(c, areaId, from, meta = {}) {
|
||||
const target = tmuxTarget(c, areaId);
|
||||
@@ -285,29 +394,78 @@ export function nudgeMail(c, areaId, from, meta = {}) {
|
||||
window: target,
|
||||
file: meta.file || null,
|
||||
trigger: meta.trigger || 'send',
|
||||
before: { found: before.found, dead: before.dead, interrupt: before.interrupt, idleSec: before.idleSec },
|
||||
before: snapBefore(before),
|
||||
};
|
||||
const finish = (fields) => {
|
||||
const { line, reason, ...logged } = fields;
|
||||
const pane = logged.action === 'no-window' ? '' : capturePane(target);
|
||||
appendWakeLog(c, { ...record, ...logged, tail: paneTail(pane) });
|
||||
const where = logged.action === 'no-window' ? { lookup: target, ...sessionWindows(c) } : {};
|
||||
appendWakeLog(c, { ...record, ...logged, ...where, tail: paneTail(pane) });
|
||||
return { submitted: logged.outcome === 'submitted', reason, ...(line ? { line } : {}) };
|
||||
};
|
||||
if (!before.found) return finish({ action: 'no-window', enterAttempts: 0, outcome: 'no-window', reason: 'no-window' });
|
||||
if (!before.found) return finish({ action: 'no-window', enterAttempts: 0, enterDelays: [], outcome: 'no-window', reason: 'no-window' });
|
||||
const line = nudgeLine(from);
|
||||
if (before.interrupt) return finish({ action: 'skipped-busy', enterAttempts: 0, outcome: 'skipped-busy', reason: 'working', line });
|
||||
if (tmuxText(['send-keys', '-t', target, '-l', line]).status !== 0) return finish({ action: 'send-failed', enterAttempts: 0, outcome: 'send-failed', reason: 'send-failed', line });
|
||||
let enterAttempts = 0;
|
||||
const hitEnter = () => { tmuxText(['send-keys', '-t', target, 'Enter']); enterAttempts += 1; };
|
||||
sleepMs(NUDGE_PAUSE_MS);
|
||||
hitEnter();
|
||||
sleepMs(NUDGE_SETTLE_MS);
|
||||
if (!stillInInput(target, line)) return finish({ action: 'typed', enterAttempts, outcome: 'submitted', reason: 'submitted', line });
|
||||
sleepMs(NUDGE_RETRY_MS);
|
||||
hitEnter();
|
||||
sleepMs(NUDGE_SETTLE_MS);
|
||||
const submitted = !stillInInput(target, line);
|
||||
return finish({ action: 'typed', enterAttempts, outcome: submitted ? 'submitted' : 'swallowed', reason: submitted ? 'submitted' : 'not-submitted', line });
|
||||
if (before.interrupt) {
|
||||
const hand = handDeferred(c, areaId, from, meta.file || '');
|
||||
return finish({
|
||||
action: 'deferred',
|
||||
outcome: hand.started ? 'deferred' : 'waiter-running',
|
||||
enterAttempts: 0,
|
||||
enterDelays: [],
|
||||
waiterPid: hand.pid || null,
|
||||
reason: 'working',
|
||||
line,
|
||||
});
|
||||
}
|
||||
return submitWake(c, areaId, from, meta, before);
|
||||
}
|
||||
|
||||
/** Poll until the pane is idle, the mail is read, or the window closes. One process per area. */
|
||||
export function runWakeWait(c, { area, from, file, pollMs, maxMs } = {}) {
|
||||
if (!area) die('wake-wait needs --area');
|
||||
const poll = Number(pollMs) > 0 ? Number(pollMs) : WAKE_WAIT_POLL_MS;
|
||||
const max = Number(maxMs) > 0 ? Number(maxMs) : WAKE_WAIT_MAX_MS;
|
||||
const target = tmuxTarget(c, area);
|
||||
const started = Date.now();
|
||||
const release = () => releaseWakeLock(c, area);
|
||||
const log = (fields) => {
|
||||
const where = fields.outcome === 'no-window' ? { lookup: target, ...sessionWindows(c) } : {};
|
||||
appendWakeLog(c, {
|
||||
t: new Date().toISOString(),
|
||||
area,
|
||||
window: target,
|
||||
file: file || null,
|
||||
trigger: 'deferred',
|
||||
...fields,
|
||||
...where,
|
||||
tail: fields.outcome === 'no-window' ? '' : paneTail(capturePane(target)),
|
||||
});
|
||||
release();
|
||||
};
|
||||
const unread = () => file && fs.existsSync(path.join(c.mailbox.dir, area, file));
|
||||
while (Date.now() - started < max) {
|
||||
const left = max - (Date.now() - started);
|
||||
if (left <= 0) break;
|
||||
sleepMs(Math.min(poll, left));
|
||||
if (file && !unread()) {
|
||||
return log({ action: 'already-read', outcome: 'already-read', enterAttempts: 0, enterDelays: [], before: { found: true, dead: false, interrupt: false, idleSec: null } });
|
||||
}
|
||||
const before = inspectPane(target);
|
||||
if (!before.found) {
|
||||
return log({ action: 'no-window', outcome: 'no-window', enterAttempts: 0, enterDelays: [], before: snapBefore(before) });
|
||||
}
|
||||
if (before.dead) {
|
||||
return log({ action: 'no-window', outcome: 'dead', enterAttempts: 0, enterDelays: [], before: snapBefore(before) });
|
||||
}
|
||||
if (before.interrupt) continue;
|
||||
try {
|
||||
return submitWake(c, area, from, { trigger: 'deferred', file: file || null }, before);
|
||||
} finally {
|
||||
release();
|
||||
}
|
||||
}
|
||||
const before = inspectPane(target);
|
||||
return log({ action: 'gave-up', outcome: 'timeout', enterAttempts: 0, enterDelays: [], before: snapBefore(before) });
|
||||
}
|
||||
|
||||
function unreadFiles(c, id) {
|
||||
|
||||
@@ -41,15 +41,27 @@ export function wakeFlag(entry) {
|
||||
const age = entry.ageSec ?? '?';
|
||||
if (!last) return `still unread after ${age}s; no earlier wake`;
|
||||
if (last.outcome === 'swallowed') return `woke at ${last.t} (${last.trigger}), wake line still in the composer, still unread after ${age}s`;
|
||||
if (last.outcome === 'timeout') return `woke at ${last.t} (${last.trigger}), pane stayed busy until the waiter stopped, still unread after ${age}s`;
|
||||
if (last.outcome === 'deferred' || last.action === 'deferred') return `woke at ${last.t} (${last.trigger}), handed to a waiter because the pane was busy, still unread after ${age}s`;
|
||||
if (last.action === 'skipped-busy') return `woke at ${last.t} (${last.trigger}), skipped because an interrupt hint was showing, still unread after ${age}s`;
|
||||
if (last.action === 'no-window' || last.outcome === 'no-window') return `woke at ${last.t} (${last.trigger}), pane not found, still unread after ${age}s`;
|
||||
if (last.outcome === 'submitted') return `woke at ${last.t} (${last.trigger}), line left the composer, still unread after ${age}s`;
|
||||
return `woke at ${last.t} (${last.trigger}), outcome ${last.outcome}, still unread after ${age}s`;
|
||||
}
|
||||
if (entry.action === 'no-window' || entry.outcome === 'no-window') return 'pane not found';
|
||||
if (entry.action === 'no-window' || entry.outcome === 'no-window') {
|
||||
const names = Array.isArray(entry.windows) && entry.windows.length ? entry.windows.join(', ') : 'none';
|
||||
const looked = entry.lookup || entry.window || 'unknown window';
|
||||
return `pane not found (looked up ${looked}; session windows: ${names})`;
|
||||
}
|
||||
if (entry.outcome === 'waiter-running') return 'a waiter is already polling this pane';
|
||||
if (entry.outcome === 'already-read') return 'mail was read before the pane went idle, so nothing was typed';
|
||||
if (entry.outcome === 'timeout') return 'pane stayed busy until the waiter stopped';
|
||||
if (entry.outcome === 'deferred') return 'handed to a waiter because an interrupt hint was showing';
|
||||
if (entry.action === 'skipped-busy') return 'skipped: interrupt hint was showing, so the pane looked mid-turn';
|
||||
if (entry.action === 'send-failed') return 'send-keys failed before Enter';
|
||||
if (entry.trigger === 'deferred' && entry.outcome === 'swallowed') return `waiter typed it; wake line still on the cursor row after ${entry.enterAttempts} Enter`;
|
||||
if (entry.outcome === 'swallowed') return `wake line still on the cursor row after ${entry.enterAttempts} Enter`;
|
||||
if (entry.trigger === 'deferred' && entry.outcome === 'submitted') return 'waiter typed the line and it left the composer';
|
||||
if (entry.outcome === 'submitted') {
|
||||
if (entry.before && entry.before.idleSec != null && entry.before.idleSec < 3 && !entry.before.interrupt) {
|
||||
return 'typed and the line left the composer while the pane was recently active';
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
#!/usr/bin/env node
|
||||
// One detached waiter for a busy pane. Started by nudgeMail, not by hand.
|
||||
import { flag, loadConfig, runningAsMain } from '../lib.mjs';
|
||||
import { runWakeWait } from './ops.mjs';
|
||||
|
||||
if (runningAsMain(import.meta.url)) {
|
||||
const c = loadConfig();
|
||||
runWakeWait(c, {
|
||||
area: flag(process.argv, '--area'),
|
||||
from: flag(process.argv, '--from') || 'fleet',
|
||||
file: flag(process.argv, '--file') || '',
|
||||
pollMs: Number(flag(process.argv, '--poll-ms')) || 0,
|
||||
maxMs: Number(flag(process.argv, '--max-ms')) || 0,
|
||||
});
|
||||
}
|
||||
@@ -38,6 +38,7 @@
|
||||
// the same tools as tools/fleet/bin/<name>.mjs
|
||||
// fleet status areas, run state (running / exited / failed / timeout / stopped), lander up or down
|
||||
// fleet wakes recent wake attempts, and plain words for a likely failure
|
||||
// fleet health [--json] one snapshot: CPU, memory, disk, temps, battery, per-area use
|
||||
// fleet prompt <id> print the composed system prompt for an area
|
||||
// fleet crontab PRINT the crontab lines for scheduled jobs (never installs)
|
||||
// fleet run-job <name> one headless run of docs/agents/schedules/<name>.md (sched.sh calls this)
|
||||
@@ -53,6 +54,7 @@ import { spawnSync } from 'node:child_process';
|
||||
import { KIT, loadConfig, loadAdapter, area as getArea, dispatcher, sessionName, flag, die, writeAtomic, tryGit, mainRef, today, runProcessGroup, runForeground, finishHeadless, withHandoff, applyHeadlessSession, assignLaunchSession, areaBudgetMin, readRun, writeRun, outcomeOfRun, pidAlive, killProcessGroup, landerIsUp, landerPid, startLander, writableRoots } from './lib.mjs';
|
||||
import { commitWork, setupCommit, tellDispatch, hearInterpreter, fleetStatus } from './core/ops.mjs';
|
||||
import { readWakeLog } from './core/wake-log.mjs';
|
||||
import { fleetHealth, formatHealth } from './core/health.mjs';
|
||||
import { fleetUpdate, recordInstalled, isTemplatePath } from './core/update.mjs';
|
||||
|
||||
const argv = process.argv.slice(2);
|
||||
@@ -101,7 +103,7 @@ function composePrompt(a, adapter) {
|
||||
'area.neighbours': a.neighbours.join(', ') || '(none listed)',
|
||||
'area.branch': a.branch, 'area.worktree': a.worktree, mainBranch: c.mainBranch, peers,
|
||||
dispatcher: dispatcher(c)?.id ?? '(none)', gate: c.gates.land ?? '(no gate configured)',
|
||||
quietHours: c.clock.quietHours.join('–'), messaging: adapter.messaging?.instructions ?? '',
|
||||
quietHours: c.clock.quietHours.join('–'), stateDir: c.stateDir, messaging: adapter.messaging?.instructions ?? '',
|
||||
};
|
||||
return `${common}\n\n${own}`.replace(/\{\{([\w.]+)\}\}/g, (m, k) => vars[k] ?? m);
|
||||
}
|
||||
@@ -310,6 +312,12 @@ async function status() {
|
||||
for (const id of report.recovered) console.log(`nudged ${id} again`);
|
||||
}
|
||||
|
||||
function healthCmd() {
|
||||
const report = fleetHealth(c);
|
||||
if (argv.includes('--json')) console.log(JSON.stringify(report, null, 2));
|
||||
else console.log(formatHealth(report));
|
||||
}
|
||||
|
||||
function wakesCmd() {
|
||||
const report = readWakeLog(c);
|
||||
if (!report.entries.length) {
|
||||
@@ -319,7 +327,7 @@ function wakesCmd() {
|
||||
console.log(report.path);
|
||||
for (const e of report.entries) {
|
||||
if (e.kind === 'stale') console.log(`${e.t} stale ${e.area} ${e.file} ${e.flag}`);
|
||||
else console.log(`${e.t} ${e.trigger} ${e.area} ${e.window} ${e.file || '-'} ${e.action} enter=${e.enterAttempts} ${e.outcome} ${e.flag}`);
|
||||
else console.log(`${e.t} ${e.trigger} ${e.area} ${e.window} ${e.file || '-'} ${e.action} enter=${e.enterAttempts}${Array.isArray(e.enterDelays) && e.enterDelays.length ? ` delays=${e.enterDelays.join('+')}` : ''} ${e.outcome} ${e.flag}`);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -633,7 +641,7 @@ function tool(name) {
|
||||
const run = { up, down, status, supervise: () => supervise(argv[1]), worktrees, crontab,
|
||||
prompt: async () => console.log(composePrompt(getArea(c, argv[1]), await adapterFor(getArea(c, argv[1]).harness))),
|
||||
'run-job': () => runJob(argv[1]), 'sched-env': () => schedEnv(argv[1]), lander: () => lander(argv[1] ?? 'status'),
|
||||
commit: commitCmd, 'setup-commit': setupCommitCmd, update: updateCmd, tell: tellCmd, hear: hearCmd, wakes: wakesCmd,
|
||||
commit: commitCmd, 'setup-commit': setupCommitCmd, update: updateCmd, tell: tellCmd, hear: hearCmd, wakes: wakesCmd, health: healthCmd,
|
||||
doctor: () => tool('doctor'), landq: () => tool('landq'), backlog: () => tool('backlog'),
|
||||
mailbox: () => tool('mailbox') }[cmd];
|
||||
if (!run) die(`unknown command "${cmd}" (fleet help)`);
|
||||
|
||||
@@ -6,6 +6,7 @@ import path from 'node:path';
|
||||
import crypto from 'node:crypto';
|
||||
import { execFileSync, spawn, spawnSync } from 'node:child_process';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { healthThresholds } from './core/health.mjs';
|
||||
|
||||
export const KIT = path.resolve(path.dirname(new URL(import.meta.url).pathname), '..');
|
||||
|
||||
@@ -117,6 +118,7 @@ export function loadConfig(argv = process.argv, start = process.cwd()) {
|
||||
c.landing = { batch: 8, intervalSec: 60, protect: [], ...(c.landing || {}) };
|
||||
c.clock = { quietHours: ['22:00', '08:00'], ...(c.clock || {}) };
|
||||
c.doctor = { staleHours: 72, unownedDays: 14, ...(c.doctor || {}) };
|
||||
c.health = healthThresholds(c.health);
|
||||
c.modules = { producer: false, reviewer: false, usageGovernor: false, ...(c.modules || {}) };
|
||||
c.schedules ??= [];
|
||||
c.areas = (c.areas || []).map((a) => resolveArea(c, a));
|
||||
|
||||
@@ -11,6 +11,7 @@ import {
|
||||
} from './core/ops.mjs';
|
||||
import { fleetUpdate } from './core/update.mjs';
|
||||
import { readWakeLog } from './core/wake-log.mjs';
|
||||
import { fleetHealth } from './core/health.mjs';
|
||||
|
||||
export const SERVER = { name: 'agent-fleet', version: '0.2.0' };
|
||||
const INSTRUCTIONS = [
|
||||
@@ -19,7 +20,7 @@ const INSTRUCTIONS = [
|
||||
'mailbox_read at the start of a turn. Mail is always delivered; the prompt says when to act on it. Sandboxed agents commit with the commit tool (message plus paths, or all), not git commit. submit on that tool also queues the tip. Then go back to work. At your next natural checkpoint, land_status can report that submission. Waiting is optional. Never infer the lander or the land result from silence. landq_submit queues the committed tip (a dirty worktree warns and still queues HEAD) and starts a drain. Do not push main.',
|
||||
'A scarce machine is a prompt convention: say so in .task and wait if a peer already has it. There is no slot lock.',
|
||||
'Sessions, worktrees, and cron stay on the fleet CLI. This server does not start them.',
|
||||
'setup_commit commits only the fleet setup files on the main checkout. A sandboxed agent uses it instead of git commit. update refreshes vendored runtime and the registered skill to this kit, leaves an edited prompt in place (the new template is a .kit file beside it), and commits only the paths it changed. It does not change fleet.config.json, product files, mail, or the landing queue. If that commit cannot write the index, it falls back to setup_commit. Mail to an open tmux window wakes it with a short line, not the body. mailbox_read and hear take those messages out of unread. fleet_status reports each window alive or dead, the unread count, and a working or idle guess. An idle area with mail that has been waiting is nudged once by that check. mailbox_read, hear, and tell run that same check. A second look does not nudge again. wake_log says whether a wake was swallowed, skipped, missing, or submitted while the mail is still unread. Tell the user that case. Do not ask them to open the log. Do not scrape the tmux pane.',
|
||||
'setup_commit commits only the fleet setup files on the main checkout. A sandboxed agent uses it instead of git commit. update refreshes vendored runtime and the registered skill to this kit, leaves an edited prompt in place (the new template is a .kit file beside it), and commits only the paths it changed. It does not change fleet.config.json, product files, mail, or the landing queue. If that commit cannot write the index, it falls back to setup_commit. Mail to an open tmux window wakes it with a short line, not the body. mailbox_read and hear take those messages out of unread. fleet_status reports each window alive or dead, the unread count, and a working or idle guess. An idle area with mail that has been waiting is nudged once by that check. mailbox_read, hear, and tell run that same check. A second look does not nudge again. wake_log says whether the wake line was still in the composer, a waiter typed it after the pane was busy, the line left the composer but the mail is still unread, or the pane was not there. The missing-pane line names the windows that were. Tell the user that case. Do not ask them to open the log. health is one snapshot of CPU, memory, swap, disk, temperature, battery, and each area process tree. Missing sensors are unavailable. The config health block is guidance. Say the verdict in plain words. Do not scrape the tmux pane.',
|
||||
].join(' ');
|
||||
|
||||
const cwdProp = { type: 'string', description: 'Directory to resolve the fleet from. Defaults to this server\'s cwd.' };
|
||||
@@ -41,6 +42,7 @@ export const TOOLS = [
|
||||
{ name: 'hear', description: 'Read unread mailbox messages for the interpreter and move them out of unread.', inputSchema: { type: 'object', properties: { cwd: cwdProp } } },
|
||||
{ name: 'fleet_status', description: 'Each area: tmux window alive or dead, unread mail count, and a working or idle guess. Idle with unread mail that has been waiting is nudged once.', inputSchema: { type: 'object', properties: { cwd: cwdProp } } },
|
||||
{ name: 'wake_log', description: 'Recent wake attempts from the fleet state dir. Each line says the pane state, what was typed or skipped, and a plain-words flag when a wake likely failed. No mail bodies.', inputSchema: { type: 'object', properties: { limit: { type: 'number' }, cwd: cwdProp } } },
|
||||
{ name: 'health', description: 'One snapshot of CPU, memory, swap, disk free, temperature, battery, and per-area CPU and memory. Missing sensors are unavailable. The verdict names which suggested limits were crossed.', inputSchema: { type: 'object', properties: { cwd: cwdProp } } },
|
||||
];
|
||||
|
||||
function configFor(args) {
|
||||
@@ -69,6 +71,7 @@ export function dispatchTool(name, args = {}) {
|
||||
case 'hear': return hearInterpreter(c);
|
||||
case 'fleet_status': return fleetStatus(c);
|
||||
case 'wake_log': return readWakeLog(c, { limit: args.limit });
|
||||
case 'health': return fleetHealth(c);
|
||||
default: throw new FleetError(`unknown tool ${name}`);
|
||||
}
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user