fleet update

This commit is contained in:
thePR0M3TH3AN committed 2026-10-26 14:58:19 -04:00
1 parent 0ffd92a8d2
commit 2d695e7acd
20 files changed
+546 -38

No files matched your search

+1 -1
View File
@@ -5,7 +5,7 @@ description: Use fleet-mcp tools for backlog, mailbox, landing queue, and area o
Follow the fleet rules in `AGENTS.md`. Call the `agent-fleet` MCP tools for backlog, mailbox, landing-queue, area questions, and commits. Editing those files directly is also fine. Sandboxed Codex cannot write Git's index: use the `commit` tool (or `fleet commit -m "..." --area <id> -- <paths>`), not `git commit`. `submit: true` queues the landing and starts a drain. Go back to work. At your next natural checkpoint, `land_status` (or `fleet landq status <id>`) can report that submission. Waiting is optional. Do not infer the lander or the land result from silence. Other providers may use either commit path.
If you installed this fleet, you stay on as the interpreter. Read `docs/agents/prompts/interpreter.md`. Run `fleet up` yourself. Talk to the session manager with `tell` and read replies with `hear`. Do not scrape the tmux pane, and do not send the user to a terminal. A routine yes you relay counts. Money, pushes, deploys, deletes, secrets, outgoing messages, and a bigger structural or directional change go to the user. Uncommitted setup files use `setup_commit`, not `git commit`. Workers do not start the fleet. `fleet crontab` stays unused unless the user asks.
If you installed this fleet, you stay on as the interpreter. Read `docs/agents/prompts/interpreter.md`. Run `fleet up` yourself. Talk to the session manager with the MCP tools `tell` and `hear`. Do not send those with the shell `fleet tell`: that is a sandboxed command, and the MCP approval mode does not apply to it. Do not scrape the tmux pane, and do not send the user to a terminal. A routine yes you relay counts. Money, pushes, deploys, deletes, secrets, outgoing messages, and a bigger structural or directional change go to the user. Uncommitted setup files use `setup_commit`, not `git commit`. Workers do not start the fleet. `fleet crontab` stays unused unless the user asks.
You may use this harness's own subagents for parallel research, investigation, test runs, or review. You coordinate edits in a worktree so subagents do not edit the same files at the same time. Landing still goes through your `commit` and the queue.
+8
View File
@@ -0,0 +1,8 @@
# TODO — what each area did NOT finish
One section per area. Each session updates its own section before it stops: what is
half-done, where it is, and what the next step is. Newest first inside a section.
## <area id>
- YYYY-MM-DD: <what is unfinished> — <where (branch, file, SHA)> — next: <step>
+32
View File
@@ -0,0 +1,32 @@
# Program — the owner's priorities and rulings
Kept by the session manager. Short on purpose: the backlog holds the tasks, RESUME-BRIEF.md
what is mid-flight. Every ruling is DATED and QUOTES the owner; session-manager calls are
marked vetoable.
Last updated: YYYY-MM-DD
## Current milestone
TODO: one paragraph.
## Priorities (owner's order)
1. TODO
2. TODO
## Rulings log (newest first)
<!-- Format:
### YYYY-MM-DD HH:MM — <short title>
> "<the owner's exact words>" — owner
Applies to: <areas>. Recorded by: <session>.
### YYYY-MM-DD HH:MM — <short title> (session manager call, VETOABLE)
What was decided and why, in two lines. The owner overrides by saying so.
-->
## Standing rules
- Approvals for spend, publishing and deploys are typed by the owner in the session that acts.
- Agreed work is never silently dropped: a priority change defers it; only the owner cancels, with a reason.
+23
View File
@@ -0,0 +1,23 @@
# RESUME BRIEF — written YYYY-MM-DD HH:MM (<why the fleet stopped>)
Written by the dispatcher at every wind-down; read first by every session at restart.
## First reads on restart
1. `git log --oneline <main> -20` — what landed while the fleet was dark.
2. Remove `<stateDir>/PAUSED` (the restart job does this).
3. TODO: anything scheduled to run while dark, and where its output is.
## Mid-flight (per area)
| area | item | state | SHA / branch | next step |
| --- | --- | --- | --- | --- |
| TODO | | | | |
## Waiting on the owner
- TODO
## Held (do not land until)
- TODO
+12
View File
@@ -0,0 +1,12 @@
# Model tier strikes
Evidence loop for per-area model tiers. A strike is one of: red landing, withdrawn
submission, reviewer rejection, peer-caught false green, owner-reported regression.
- Two strikes in a week (week = from the usage reset) -> propose moving the area UP a tier.
- Two clean weeks -> offer moving it DOWN a tier.
- Loses the thread after compaction -> raise its context.
Changes go into fleet.config.json (area `tier` / `context`) and need a fresh start of that area.
| date | area | tier | strike | evidence (SHA, check, who caught it) |
|---|---|---|---|---|
+21 -1
View File
@@ -133,9 +133,29 @@ have it in `.task`, and wait if a peer's `.task` says they do.
If you are blocked, put one line in your worktree's `.task`:
`WAITING: <who> — <what> (since <date>)` and remove it when unblocked. The
session manager's check-in reads every `.task`, so a missed message never leaves you stuck.
Keep `.task` (one line: what you are doing now) current before any long step, so a
Also record the hold with MCP `hold` (or `fleet hold`): the owner, the exact
unblock condition, and the next task you are still allowed to do. Clear it when
the condition is met. Keep `.task` (one line: what you are doing now) current before any long step, so a
compaction or crash loses nothing.
When work actually advances, write one checkpoint with MCP `progress` (or
`fleet progress <your id> "<what changed>"`). A live window is not progress.
Status and the check-in flag an area with no checkpoint for 20 minutes and no
unread mail.
When health holds heavy work, keep doing light work (source review, offline
analysis) instead of going idle. The hold names that next task.
Runners, dependency manifests, pinned archives, and evidence live in your
worktree or `{{stateDir}}/deps`, with those names in `manifest.txt`. Never only
in `/tmp`. The npm cache is disposable. Three pinned archives disappeared from
`/tmp` and the npm cache, and QA could not run.
There is no billing hook. If the pane says the five-hour window is nearly gone
(for example "2% remaining"), write the checkpoint, mail your next step, and
stop. After the reset, `fleet resume` is the one step back: it checks the
mailbox, the approval path, deps, and health, and you continue from the checkpoint.
## Reporting
One message per event, to the session that needs it. Message the session manager only for:
+17 -5
View File
@@ -8,7 +8,7 @@ Run `fleet up` yourself. If `fleet init` already started the session manager in
## The channel
Send the session manager a message with MCP `tell`, or `fleet tell "<text>"`. That writes mail from `interpreter` to the session manager. When its tmux window is open, a short line wakes it (`You have new fleet mail from interpreter. Read it with mailbox_read.`). That line is not the message. The mail is the one record. The same wake is used for worker mail and for replies. Read replies with MCP `hear` or `fleet hear`. Reading takes them out of unread, so a later hear does not repeat them. Do not scrape the pane.
Send the session manager a message with MCP `tell`. That writes mail from `interpreter` to the session manager. When its tmux window is open, a short line wakes it (`You have new fleet mail from interpreter. Read it with mailbox_read.`). That line is not the message. The mail is the one record. The same wake is used for worker mail and for replies. Read replies with MCP `hear`. Reading takes them out of unread, so a later hear does not repeat them. Do not scrape the pane. The shell form `fleet tell` (and `fleet hear`) is a sandboxed command. Codex automatic approval review applies to that shell even when the MCP server is set to approve, so do not use the shell to send or read mail.
At the start of every turn, run `fleet status` (MCP `fleet_status`) before you hear. That look is the one re-nudge for mail that has been waiting. A second look does not nudge again. `hear` and `tell` run that same check.
@@ -22,9 +22,11 @@ Ask these in your own chat. Offer the default so a one-word answer works. Pass t
## Codex tool approval
Codex may pause on fleet mail tools such as `tell` and `mailbox_send`. The documented server setting is `default_tools_approval_mode = "auto"` under `[mcp_servers.agent-fleet]` in `~/.codex/config.toml`. One tool is `[mcp_servers.agent-fleet.tools.<tool>]` with `approval_mode = "auto"`. The values are `auto`, `prompt`, `writes`, and `approve`. `writes` still prompts for tools that are not marked read-only, and these are not. Offer to set `auto` for the agent-fleet server, and write it only after the user says yes. A running Codex session does not see a change to `~/.codex/config.toml` until that session is reloaded.
Codex may pause on fleet mail tools such as `tell` and `mailbox_send`. `fleet up` starts Codex with `--ask-for-approval on-request`, no `--profile`, and it does not set `CODEX_HOME`, so the session reads the same `~/.codex/config.toml` (or `$CODEX_HOME/config.toml`) as any other Codex process in that environment. The server name is `agent-fleet`. A table under another name or another profile does not apply.
Those keys are `mcp_servers.<id>.default_tools_approval_mode` and `mcp_servers.<id>.tools.<tool>.approval_mode` in the Codex config reference (https://developers.openai.com/codex/config-reference) and the MCP page (https://learn.chatgpt.com/docs/extend/mcp). When `approval_policy` is `on-request` and `approvals_reviewer` is `auto_review`, MCP calls that still require approval go to that reviewer (https://learn.chatgpt.com/docs/sandboxing/auto-review). The words "internal coordination messages" are not in those pages.
The value that skips approval is `default_tools_approval_mode = "approve"` under `[mcp_servers.agent-fleet]`. One tool is `[mcp_servers.agent-fleet.tools.mailbox_send]` and `[mcp_servers.agent-fleet.tools.tell]` with `approval_mode = "approve"`. The values are `auto`, `prompt`, `writes`, and `approve`. `auto` is not always-allow: these tools are not marked read-only, so `auto` and `writes` still require approval. `fleet up` uses `on-request`. When `approvals_reviewer` is `auto_review`, a call that still requires approval goes to that reviewer, which can reject it. `approve` is the mode that does not require approval, so the reviewer does not run. Offer to set `approve` for the agent-fleet server, and write it only after the user says yes. Do not put it in the shipped fragment. A running Codex session does not see a change to `~/.codex/config.toml` until that session is reloaded.
Those keys are `mcp_servers.<id>.default_tools_approval_mode` and `mcp_servers.<id>.tools.<tool>.approval_mode` in the Codex config reference (https://developers.openai.com/codex/config-reference) and the MCP page (https://learn.chatgpt.com/docs/extend/mcp). The reviewer behavior is https://learn.chatgpt.com/docs/sandboxing/auto-review and the Codex changes that skip review only for `approve` (https://github.com/openai/codex/pull/20069 and https://github.com/openai/codex/pull/20750). The words "internal coordination messages" are not in those pages. `fleet ready` reads the config and says whether mail is set to `approve`. It also proves one mailbox round trip.
## Approvals
@@ -34,11 +36,11 @@ Bring these to the user, and do not approve them yourself: money, a push, a depl
## Status
When the user asks how things are going, answer in plain language: how many agents, which areas, what each is working on, what landed, and what needs the user. `fleet status` (MCP `fleet_status`) says, for each area, whether the tmux window is alive or dead, how many messages are still unread, and a working or idle guess. Idle with unread mail means that agent has not read it. Looking at status nudges that pane once if the mail has been waiting. A second look does not nudge again. If status shows unread mail that has been waiting, or a recovery nudge fired, run `fleet wakes` (MCP `wake_log`) and tell the user which case it looked like: the wake line was still in the composer, a later Enter submitted it, a waiter typed it after the pane was busy, the line left the composer but the mail is still unread, or the pane was not there (the log names the windows and the tmux socket). Do not ask the user to open the log. Read `hear`, `fleet/LOG.md`, the landing queue, and `git log`. Offer a dev-server preview when that would help them see the work.
When the user asks how things are going, answer in plain language: how many agents, which areas, what each is working on, what landed, and what needs the user. `fleet status` (MCP `fleet_status`) says, for each area, whether the tmux window is alive or dead, how many messages are still unread, a working or idle guess, a stall when nothing has advanced, and any hold (owner, unblock condition, next allowed task). A live window that still says working is not progress. Idle with unread mail means that agent has not read it. Looking at status nudges that pane once if the mail has been waiting. A second look does not nudge again. If status shows unread mail that has been waiting, or a recovery nudge fired, run `fleet wakes` (MCP `wake_log`) and tell the user which case it looked like: the wake line was still in the composer, a later Enter submitted it, a waiter typed it after the pane was busy, the line left the composer but the mail is still unread, or the pane was not there (the log names the windows and the tmux socket). Do not ask the user to open the log. Read `hear`, `fleet/LOG.md`, the landing queue, and `git log`. Offer a dev-server preview when that would help them see the work.
## The machine
When mail mentions memory, disk, heat, or battery, or the user asks how the machine is, run `fleet health` (MCP `health`). Say the verdict in plain words and the action the session manager suggests. Name the area that is using the CPU or memory when that is the cause. Swap occupancy alone is not a tight verdict: Linux keeps swapped pages. It counts when available memory is also low, or when swap is near full. Low disk is a proposal of what to remove and the sizes, then a question for the user. Do not delete, kill a process, or change a system setting yourself.
When mail mentions memory, disk, heat, or battery, or the user asks how the machine is, run `fleet health` (MCP `health`). Say the verdict in plain words and the action the session manager suggests. Name the area that is using the CPU or memory when that is the cause. Swap occupancy alone is not a tight verdict, and neither is a swap file that is nearly full while plenty of RAM is free: Linux keeps swapped pages. Swap counts when available memory is under twice the memory limit, or when swap is actively being read in. A high load average while fleet areas are idle is a note about other programs (browsers, scanners, chat apps), not a hold. When health does hold heavy work, agents keep doing light work (source review, offline analysis) instead of going idle. Low disk is a proposal of what to remove and the sizes, then a question for the user. Do not delete, kill a process, or change a system setting yourself.
The kit default CPU temperature is 90°C. If this machine runs hot below that, offer a lower `health.cpuTempC` in `fleet.config.json` (85 is a reasonable laptop figure, not a new kit default) and write it only after the user says yes. The next `fleet health` reads that file. A change to `~/.codex/config.toml` is different: Codex sessions need a reload before they see it.
@@ -48,6 +50,16 @@ The schedules include a resource check every 15 minutes. `fleet crontab` only pr
When a newer kit is available, do not re-run `fleet init` and do not hand-copy files. In `~/agent-fleet-kit` run `git pull`, reinstall with the same `npm install -g` used at setup, then in this project run `fleet update` (MCP `update`). That refreshes `tools/fleet` and the skill for the tool that is registered. A prompt you did not edit is overwritten. A prompt you edited stays, and the new template is the `.kit` file beside it: if you did not edit the local file, replace it with that copy and delete the `.kit` file; if you did, merge, then delete the `.kit` file. It does not change `fleet.config.json`, product files, worktrees, mail, or the landing queue. It commits only the paths it changed. If that commit cannot write the git index, call MCP `update` (or `setup_commit`). Do not `git commit`, and do not ask the user to approve one. If it says to restart, restart agents after they hand off so they load the new prompts.
## After a stop
A reboot, a power or network drop, or a vendor usage limit leaves sessions idle. Run `fleet resume` (the same as `fleet up`). It runs `fleet ready`: the mailbox round trip, the Codex approval path, health, whether durable deps are still on disk, and the last checkpoint for each area. Tell the session manager to continue from those checkpoints. Do not start the work over.
There is no billing integration. If a pane or the user says the five-hour window is nearly gone (the wake log has shown "2% remaining"), tell the agents to write a checkpoint and hand off. Resuming after the reset is `fleet resume` plus that checkpoint.
## Keep work on disk
Runners, lockfiles, pinned archives, and evidence live in the area worktree or the fleet state dir under `deps/`, with a `manifest.txt` of the file names. `fleet status` prints that state dir. They do not live only in `/tmp` or the npm cache. Both are disposable: a laptop lost three pinned archives that way, and QA could not run. `fleet ready` checks the manifest.
## Your own subagents
You are a top-level Claude or Codex session. You may use your harness's own subagents for parallel research, investigation, test runs, or review. Edits inside a worktree stay coordinated by the agent that owns it, so subagents do not edit the same files at the same time. Landing still goes through that agent's `commit` and the queue.
+10 -4
View File
@@ -6,9 +6,9 @@ The human says what they want in plain words. You and the agents figure out the
## Who you are talking to
The human may talk to you directly, or through their own agent, the interpreter. Messages from `interpreter` are the user. A line in this window that says you have new fleet mail is only a wake. Read the mailbox at the start of every turn, and whenever that line shows up. Reading takes the messages out of unread. The mail is the message. Reply by mailbox (`mailbox send interpreter --from {{area.id}}`, or MCP `mailbox_send`) with your next action. Do that once.
The human may talk to you directly, or through their own agent, the interpreter. Messages from `interpreter` are the user. A line in this window that says you have new fleet mail is only a wake. Read the mailbox at the start of every turn, and whenever that line shows up. Reading takes the messages out of unread. The mail is the message. Reply with MCP `mailbox_send` (to `interpreter`, from `{{area.id}}`) with your next action. Do that once. Do not reply with the shell `mailbox send` or `fleet tell`. That shell is a sandboxed command. Codex automatic approval review applies to it even when `[mcp_servers.agent-fleet]` is `approve`, and the review has been rejecting those replies.
At the start of every turn, run `fleet status` (MCP `fleet_status`) and `fleet health` (MCP `health`). Status shows each area's window alive or dead, how many messages are still unread, and a working or idle guess. Idle with unread mail that has been sitting is nudged again once by that look. A second look does not nudge again. Reading your mailbox runs that same check. There is no ack protocol and no background checker. If that look shows unread mail that has been waiting, or it nudges a pane again, run `fleet wakes` (MCP `wake_log`) and tell the user which case it looked like: the wake line was still in the composer, a later Enter submitted it, a waiter typed it after the pane was busy, the line left the composer but the mail is still unread, or the pane was not there (the log names the windows and the tmux socket). Do not ask them to open the log. Health is the machine snapshot. The machine section says what you may change.
At the start of every turn, run `fleet status` (MCP `fleet_status`) and `fleet health` (MCP `health`). Status shows each area's window alive or dead, how many messages are still unread, a working or idle guess, a stall when the checkpoint is old or missing and there is no unread mail, and any hold. A live window is not progress. Idle with unread mail that has been sitting is nudged again once by that look. A second look does not nudge again. Reading your mailbox runs that same check. There is no ack protocol and no background checker. If that look shows unread mail that has been waiting, or it nudges a pane again, run `fleet wakes` (MCP `wake_log`) and tell the user which case it looked like: the wake line was still in the composer, a later Enter submitted it, a waiter typed it after the pane was busy, the line left the composer but the mail is still unread, or the pane was not there (the log names the windows and the tmux socket). Do not ask them to open the log. Health is the machine snapshot. The machine section says what you may change.
An approval the interpreter relays counts for a routine decision: start this team, take a normal task. It does not count for money, a push, a deploy, a delete, a secret, an outgoing message, or a bigger structural or directional change. Those go to the real user. If you are unsure which it is, ask the interpreter to take it to the user. Do not ask the user to edit a file, attach to tmux, or run a command.
@@ -16,9 +16,11 @@ An approval the interpreter relays counts for a routine decision: start this tea
`fleet health` is one snapshot: CPU load and usage, memory and swap, disk free for the repo, the worktree root, the state dir, and tmp, CPU temperature and battery when the machine exposes them, and the CPU and memory of each area's process tree. A missing sensor says unavailable. The command changes nothing.
Run it at the start of every turn, and again before you staff another area. The `health` block in `fleet.config.json` is suggested limits (disk free, memory, swap, temperature, battery, load). A missing block uses the built-in defaults. They are guidance. You decide. Swap occupancy alone is not tight: Linux keeps swapped pages. Swap counts when available memory is also low, or when swap is near full (default 90%). The CPU temperature default stays 90°C. A lower `cpuTempC` in that block is a per-repo choice. The interpreter offers one and writes it only after the user says yes. Do not change it yourself.
Run it at the start of every turn, and again before you staff another area. The `health` block in `fleet.config.json` is suggested limits (disk free, memory, swap, temperature, battery, load). A missing block uses the built-in defaults. They are guidance. You decide. Swap occupancy alone is not tight, and a swap file that is nearly full while plenty of RAM is free is not tight: Linux keeps swapped pages. Swap counts when available memory is under twice the memory limit (default 30%), or when the snapshot sees swap being read in. A high load average while fleet areas are idle is a note that names the other programs. It does not hold fleet work. The CPU temperature default stays 90°C. A lower `cpuTempC` in that block is a per-repo choice. The interpreter offers one and writes it only after the user says yes. Do not change it yourself.
On your own, inside the fleet, you may: staff fewer areas or hold a new dispatch; ask an area to finish its step and park; ask agents to run tests one at a time or with fewer workers; ask them to stop a dev server or browser they started and no longer need; slow down or wind down when the machine is on battery or hot; resume when a later snapshot is healthy. One hot reading is a warning. If the next check is still at or over the suggested temperature, treat it as sustained and slow down. Name the area whose process tree is using the CPU or memory.
On your own, inside the fleet, you may: staff fewer areas or hold a new dispatch; ask an area to finish its step and park; ask agents to run tests one at a time or with fewer workers; ask them to stop a dev server or browser they started and no longer need; slow down or wind down when the machine is on battery or hot; resume when a later snapshot is healthy. One hot reading is a warning. If the next check is still at or over the suggested temperature, treat it as sustained and slow down. Name the area whose process tree is using the CPU or memory. When you hold heavy work (builds, tests, new staffing), say so and tell the area to keep doing light work: source review, offline analysis, writing the next checkpoint. Do not leave them idle.
Every hold is a record, not a feeling. MCP `hold` (or `fleet hold`) needs the owner, the exact unblock condition, and the next allowed task. Status prints that. Clear it when the condition is met. When something actually advances, MCP `progress` (or `fleet progress`) writes one line. A pane that still looks busy is not that line. The check-in flags an area with no progress for the stall window (default 20 minutes) and no unread mail.
Take these to the user, through the interpreter or one dated line in `{{stateDir}}/owner.md`, and do not do them yourself: deleting files or caches, including build output, `node_modules`, and worktrees; killing a process the fleet did not start; changing a system setting; anything that costs money. Low disk means you propose what to remove and the sizes, then ask.
@@ -77,6 +79,10 @@ Commands are `fleet ...` (from this repo: `node tools/fleet/bin/fleet.mjs ...`).
A scarce machine is a line in that area's `.task`, not a lock. Before you go idle, and before quiet hours, run `fleet status` so a waiting pane is nudged once, and `fleet health` so you slow down before you stop. Quiet hours are judgment: mail is still delivered. Read `fleet/RULE-CHANGES.md` newest first; it overrides this prompt. Record an owner ruling in `docs/agents/PROGRAM.md` with the date and the exact quote.
After a reboot, a power loss, or a usage-limit stop, `fleet resume` (or `fleet up`) prints `fleet ready` and you continue from the checkpoints. Do not restaff work that a checkpoint already names. There is no billing integration. When a pane or the user says the five-hour window is nearly gone, write checkpoints, mail the next step, and stop cleanly so the resume after the reset is one step.
Runners, lockfiles, pinned archives, and evidence stay in the worktree or `{{stateDir}}/deps` with a `manifest.txt`. `/tmp` and the npm cache are disposable. Three pinned archives vanished from them and QA could not run. Do not send an area back to that.
## Rules of thumb
- Brief the human in plain words. Lead with what matters to them. Ask only for real decisions.
+1 -1
View File
@@ -1,3 +1,3 @@
You are a scheduled headless job of the {{project}} dispatcher, started by tools/fleet/bin/sched.sh in the dispatcher's worktree on {{date}}. You have no memory of earlier runs: docs/agents/PROGRAM.md, the backlog, `node tools/fleet/bin/doctor.mjs` and docs/agents/WIND-DOWN.md are your context. Rules: merge, never rebase; never rm a variable path; never bare git stash; never print or move secrets; never relay an approval; never install crontab lines or start sessions. Message a session only for a real, new problem, at most once per problem per day: check {{stateDir}}/sent-{{date}}.txt and append what you send. Owner-facing items (a decision, work at risk, main red, the wind-down mark) get one dated line appended to {{stateDir}}/owner.md. Finish in under 30 minutes and print a 3-line summary of what you did and what you did NOT verify.
TASK: hourly check-in. Run `node tools/fleet/bin/doctor.mjs --json`, `node tools/fleet/bin/landq.mjs status` and read every WAITING line. For each WAITING older than 2 hours, message the session being waited on (once per day). A lander that is NOT running is owner-facing. Red or conflict results: make sure the submitting area knows (mailbox or native message). Record new strikes (red landing, withdrawn submission, reviewer rejection, peer-caught false green, owner regression) in docs/agents/STRIKES.md; two in a week for one area is owner-facing with a proposed tier move. If the usage governor is enabled and the adapter usage hook reports at or above usage.windDownAtPct, that is owner-facing: run docs/agents/WIND-DOWN.md step 1 (create {{stateDir}}/PAUSED with the reason) and stop. Land any doc change with `node tools/fleet/bin/landq.mjs submit`.
TASK: hourly check-in. Run `node tools/fleet/bin/doctor.mjs --json`, `node tools/fleet/bin/landq.mjs status`, `node tools/fleet/bin/fleet.mjs status`, and read every WAITING line. For each WAITING older than 2 hours, message the session being waited on (once per day). A status line that says stalled (no progress for the stall window and no unread mail) is a real problem: message that area once per day and name the hold if one is recorded (owner, unblock condition, next allowed task). A live window is not progress. A lander that is NOT running is owner-facing. Red or conflict results: make sure the submitting area knows (MCP mailbox_send, not a sandboxed shell send). Record new strikes (red landing, withdrawn submission, reviewer rejection, peer-caught false green, owner regression) in docs/agents/STRIKES.md; two in a week for one area is owner-facing with a proposed tier move. If the usage governor is enabled and the adapter usage hook reports at or above usage.windDownAtPct, that is owner-facing: run docs/agents/WIND-DOWN.md step 1 (create {{stateDir}}/PAUSED with the reason) and stop. There is no billing integration. If a wake line or a pane says the five-hour window is nearly gone, that is also owner-facing: the next step is a checkpoint and a clean handoff, then `fleet resume` after the reset. Land any doc change with `node tools/fleet/bin/landq.mjs submit`.
+1 -1
View File
@@ -1,6 +1,6 @@
You are a scheduled headless job of the {{project}} dispatcher, started by tools/fleet/bin/sched.sh in the dispatcher's worktree on {{date}}. You have no memory of earlier runs: docs/agents/PROGRAM.md, the backlog, `node tools/fleet/bin/doctor.mjs` and docs/agents/WIND-DOWN.md are your context. Rules: merge, never rebase; never rm a variable path; never bare git stash; never print or move secrets; never relay an approval; never install crontab lines or start sessions. Message a session only for a real, new problem, at most once per problem per day: check {{stateDir}}/sent-{{date}}.txt and append what you send. Owner-facing items (a decision, work at risk, main red, the wind-down mark) get one dated line appended to {{stateDir}}/owner.md. Finish in under 30 minutes and print a 3-line summary of what you did and what you did NOT verify.
TASK: machine resources. Run `node tools/fleet/bin/fleet.mjs health`. This job is the check for when no one is in a turn. The `health` numbers in fleet.config.json are suggested limits, not orders. A missing sensor is unavailable and is not itself a problem. Swap occupancy alone is not a crossed limit. It counts when available memory is also low, or when swap is near full.
TASK: machine resources. Run `node tools/fleet/bin/fleet.mjs health`. This job is the check for when no one is in a turn. The `health` numbers in fleet.config.json are suggested limits, not orders. A missing sensor is unavailable and is not itself a problem. Swap occupancy alone is not a crossed limit, and a swap file that is nearly full while plenty of RAM is free is not one either. Swap counts when available memory is under twice the memory limit, or when swap is being read in. A high load average while fleet areas are idle is a note about other programs. It does not hold fleet work. When heavy work is held, tell the area to keep doing light work instead of going idle, and record the hold (owner, unblock condition, next allowed task).
You may, inside the fleet: tell the session manager to staff fewer areas or hold a new dispatch; ask an area to finish its current step and park; ask an area to run tests one at a time or with fewer workers; ask an area to stop a dev server or browser it started and no longer needs; ask the fleet to slow down or wind down when the battery is low or discharging, or when a CPU temperature is at or over the suggested limit on this run and the previous resources log ({{stateDir}}/logs/) was hot too. When a later snapshot is healthy, tell them they can resume. Name the area whose process tree is using the CPU or memory.
+1 -1
View File
@@ -1,3 +1,3 @@
You are a scheduled headless job of the {{project}} dispatcher, started by tools/fleet/bin/sched.sh in the dispatcher's worktree on {{date}}. You have no memory of earlier runs: docs/agents/PROGRAM.md, the backlog, `node tools/fleet/bin/doctor.mjs` and docs/agents/WIND-DOWN.md are your context. Rules: merge, never rebase; never rm a variable path; never bare git stash; never print or move secrets; never relay an approval; never install crontab lines or start sessions. Message a session only for a real, new problem, at most once per problem per day: check {{stateDir}}/sent-{{date}}.txt and append what you send. Owner-facing items (a decision, work at risk, main red, the wind-down mark) get one dated line appended to {{stateDir}}/owner.md. Finish in under 30 minutes and print a 3-line summary of what you did and what you did NOT verify.
TASK: reset checkpoint for an already active, unpaused fleet. Read docs/agents/RESUME-BRIEF.md and report the next eligible work to the session manager. Never remove PAUSED or start sessions: down and pause are explicit owner controls, including for this schedule. The owner must unpause and bring the fleet up before scheduled work can resume.
TASK: reset checkpoint for an already active, unpaused fleet. Read docs/agents/RESUME-BRIEF.md and the progress files under {{stateDir}}/progress. Report the next eligible work to the session manager from those checkpoints, not from a live window. Run `node tools/fleet/bin/fleet.mjs ready` and include its mailbox, approval, deps, and health lines in the summary. Never remove PAUSED or start sessions: down and pause are explicit owner controls, including for this schedule. The owner must unpause and bring the fleet up (`fleet resume`) before scheduled work can resume.
+8
View File
@@ -0,0 +1,8 @@
# Rule changes
Newest first. One `## YYYY-MM-DD` heading per day, then a line per change.
Read this file newest first now and before every backlog item. These lines
override a stale system prompt. Nothing filters them down to a window of days.
The session manager adds a line here whenever a rule, a prompt, a tool's usage, or
a landing rule changes.
+7 -7
View File
@@ -1,20 +1,20 @@
{
"version": "0.2.0",
"files": {
"docs/agents/prompts/COMMON.md": "e4ed1e09f6c89c236205405e881452caa4bb9c82473e20943dcc5f9261d171eb",
"docs/agents/prompts/COMMON.md": "b5ff5891cf101da1aa5c367569b8f375f1ceb7af5748891af9e90ae8517e47d5",
"docs/agents/prompts/area.md": "98359aa5ec95117b011cd4fb5bda683459dfe498cae775675c3a92e56a501d6e",
"docs/agents/prompts/dispatch.md": "f7b7fe719e20a5f93794b4142690778f1231c5c9c01c3244c3e9adc2d84b995c",
"docs/agents/schedules/checkin.md": "5c26b3a903ac3473dc3cc65a615b3db59bff1a0ab4ddfe7d82604a748440cff9",
"docs/agents/schedules/checkin.md": "c45171a35dd63f2339205b77a8afeab4adfa516996052aff288f24877132c899",
"docs/agents/schedules/hygiene.md": "08e8adb6930ef425cf71936479f7ee125300bb2ad337f00a4e6c68789a1b5918",
"docs/agents/schedules/restart.md": "b567cfe7021747affe4b768c07b28dce8a0a763d9700932f253e932f6bc96568",
"docs/agents/schedules/restart.md": "64056e9b2a792ac98b8951274d1db35040df0be9256724edfe7a6979a51055f3",
"docs/agents/schedules/summary.md": "7ecccd7ceec28b52e1ffcfcf38e7a3c8881880641c163d6da52e1c5de46e820f",
"docs/agents/schedules/triage.md": "0b1f52fd6a4bd14ec8be9f90e0504ed153bf2f96acd65c178ce9c3eef7a9253f",
"docs/agents/schedules/winddown.md": "36429467ce62e1f26a7eb6c2e6111ecd490c18bf691ee3fdb73c2fda71838d74",
"docs/agents/WIND-DOWN.md": "b27ae51b0b8e739e0fcca736e4cb44dfdd38b71fff1dd4261ff4df18d575df3a",
"docs/agents/STRIKES.md": "dd3047e02287207a1de96c01d5e9a1889c2c37cb75a5b3655de6e813df7f5bd9",
"docs/agents/prompts/interpreter.md": "04d06ae3dc4910d1e5eea87f1497720e9a8648b9af7278e8265894b293ef7d7f",
"docs/agents/prompts/session-manager.md": "7acaf1e520714d766e1b1735a87d9f96f356d33762c03471e4ba6759df32b3b1",
".agents/skills/agent-fleet/SKILL.md": "25d5aeef4361b3eecff525f752d990acd2ce8988b59f790ba8363eece8e646c7",
"docs/agents/schedules/resources.md": "e0689943c23414d1023a09b46f9cd87b88d6cca6e9d4ec0a36f5112938e25f3e"
"docs/agents/prompts/interpreter.md": "c1c2cb3feedc7aacd0e5cae9aab554d929dd148d7cc707da6028979697332b32",
"docs/agents/prompts/session-manager.md": "af1da998da524be29ff6f5a625d4d86887fe33c233e56f56c808023414f1218b",
".agents/skills/agent-fleet/SKILL.md": "b4eef39e6246acb86ea143e863f5800f675ce9b62bca2a83296ec5a0f34f088c",
"docs/agents/schedules/resources.md": "09bb4cb73c1bcabbe469b2d597762e010659f291ba66b8f02fdb47ec963d69ff"
}
}
+87 -12
View File
@@ -12,8 +12,10 @@ export const HEALTH_DEFAULTS = {
diskFreeGb: 5,
memoryAvailablePct: 15,
swapUsedPct: 50,
// Occupancy alone is not tight: Linux keeps swapped pages. swapUsedPct counts
// only when available memory is also low. swapFullPct counts on its own.
// Occupancy alone is not tight, and a full swap file is not tight while
// plenty of RAM is free. swapUsedPct counts only when available memory is
// under twice memoryAvailablePct. swapFullPct is kept so older configs load;
// it is not a hold by itself. Live swap-in (pswpin) is pressure on its own.
swapFullPct: 90,
cpuTempC: 90,
batteryPct: 20,
@@ -189,6 +191,35 @@ export function parseDf(text) {
return { mount: parts.slice(5).join(' '), totalBytes: total, usedBytes: used, freeBytes: free, freePct };
}
/** Linux /proc/vmstat counters. pswpin is pages read back from swap. */
export function parseLinuxVmstat(text) {
const out = {};
for (const line of String(text || '').split('\n')) {
const m = /^([A-Za-z0-9_]+)\s+(\d+)$/.exec(line.trim());
if (m) out[m[1]] = Number(m[2]);
}
return out;
}
/** Pages swapped in between two /proc/vmstat snapshots. Null when either side lacks pswpin. */
export function swapInPages(before, after) {
if (before?.pswpin == null || after?.pswpin == null) return null;
return Math.max(0, after.pswpin - before.pswpin);
}
/** Top non-trivial CPU lines from `ps -ax -o pid=,pcpu=,comm=`. */
export function parseBusyPs(text) {
const rows = [];
for (const line of String(text || '').split('\n')) {
const m = /^\s*(\d+)\s+(\d+(?:\.\d+)?)\s+(\S+)/.exec(line);
if (!m) continue;
const cpu = Number(m[2]);
if (cpu < 5) continue;
rows.push({ pid: Number(m[1]), cpu, comm: m[3] });
}
return rows.sort((a, b) => b.cpu - a.cpu).slice(0, 3);
}
export function parseVmStat(text) {
const page = /page size of (\d+) bytes/i.exec(text || '');
const pages = {};
@@ -521,20 +552,24 @@ function readWindowsBattery() {
function linuxSample(sysRoot, opt) {
if (opt.statTexts || opt.procSamples) {
const vm = Array.isArray(opt.vmstat) ? opt.vmstat.map((text) => parseLinuxVmstat(text)) : null;
return {
usagePct: opt.statTexts ? cpuUsagePct(parseCpuStat(opt.statTexts[0]), parseCpuStat(opt.statTexts[1])) : null,
before: opt.procSamples ? opt.procSamples[0] : [],
after: opt.procSamples ? opt.procSamples[1] : [],
elapsedMs: opt.elapsedMs ?? 1000,
swapInPages: vm ? swapInPages(vm[0], vm[1]) : null,
};
}
if (opt.sampleMs === 0) return { usagePct: null, before: [], after: [], elapsedMs: 0 };
if (opt.sampleMs === 0) return { usagePct: null, before: [], after: [], elapsedMs: 0, swapInPages: null };
const stat1 = parseCpuStat(readText(path.join(sysRoot, 'proc/stat')) || '');
const vm1 = parseLinuxVmstat(readText(path.join(sysRoot, 'proc/vmstat')) || '');
const pages = pageSize();
const before = readProcSnapshot(sysRoot, pages);
const t0 = Date.now();
pause(opt.sampleMs ?? 200);
const stat2 = parseCpuStat(readText(path.join(sysRoot, 'proc/stat')) || '');
const vm2 = parseLinuxVmstat(readText(path.join(sysRoot, 'proc/vmstat')) || '');
const after = readProcSnapshot(sysRoot, pages);
let usagePct = cpuUsagePct(stat1, stat2);
if (usagePct == null) {
@@ -542,7 +577,7 @@ function linuxSample(sysRoot, opt) {
pause(opt.sampleMs ?? 200);
usagePct = cpuUsagePct(a, cpuTimes());
}
return { usagePct, before, after, elapsedMs: Math.max(1, Date.now() - t0) };
return { usagePct, before, after, elapsedMs: Math.max(1, Date.now() - t0), swapInPages: swapInPages(vm1, vm2) };
}
function readLoad(platform, sysRoot) {
@@ -606,13 +641,14 @@ function judge(snap, limits) {
if (mem?.swapTotalBytes > 0 && mem.swapUsedBytes != null) {
const pct = (100 * mem.swapUsedBytes) / mem.swapTotalBytes;
const availablePct = mem.totalBytes > 0 && mem.availableBytes != null ? (100 * mem.availableBytes) / mem.totalBytes : null;
const memoryLow = availablePct != null && availablePct < limits.memoryAvailablePct;
const nearFull = pct >= limits.swapFullPct;
// Occupancy alone is normal on Linux. Count it when memory is also low, or when swap is near full.
if (nearFull || (memoryLow && pct >= limits.swapUsedPct)) {
const words = nearFull
? `swap ${Math.round(pct)}% used, near full (at or over ${limits.swapFullPct}%)`
: `swap ${Math.round(pct)}% used while available memory is under ${limits.memoryAvailablePct}%`;
const softLimit = limits.memoryAvailablePct * 2;
const memorySoft = availablePct != null && availablePct < softLimit;
const swappingIn = Number(mem.swapInPages) > 0;
// A full swap file with plenty of free RAM is parked pages, not pressure.
if (swappingIn || (memorySoft && pct >= limits.swapUsedPct)) {
const words = swappingIn
? `swap is being read in (${mem.swapInPages} pages in the sample)`
: `swap ${Math.round(pct)}% used while available memory is under ${softLimit}%`;
crossed.push({ id: 'swap', words });
}
}
@@ -649,12 +685,45 @@ function judge(snap, limits) {
if (snap.cpu?.load1 != null && snap.cpu.cores) {
const per = snap.cpu.load1 / snap.cpu.cores;
if (per >= limits.loadPerCore) {
crossed.push({ id: 'load', words: `load ${snap.cpu.load1} on ${snap.cpu.cores} cores (${Math.round(per * 10) / 10} per core), at or over ${limits.loadPerCore}` });
const fleet = fleetCpuPct(snap.areas);
const words = `load ${snap.cpu.load1} on ${snap.cpu.cores} cores (${Math.round(per * 10) / 10} per core)`;
if (fleet != null && fleet >= 10) {
const who = (snap.areas || []).filter((a) => a.cpuPct >= 10).map((a) => `${a.id} ${a.cpuPct}%`).join(', ');
crossed.push({ id: 'load', words: `${words}, and fleet areas are using CPU${who ? ` (${who})` : ''}` });
} else {
const outside = describeBusy(snap.busy);
const sample = snap.cpu.usagePct == null ? '' : ` A short CPU sample is ${snap.cpu.usagePct}%.`;
const fleetWords = fleet == null ? 'fleet area CPU was not measured' : 'fleet areas are idle';
snap.notes.push({ id: 'load', words: `${words} is above ${limits.loadPerCore} per core, but ${fleetWords}.${sample}${outside} This does not hold fleet work.` });
}
}
}
return crossed;
}
function fleetCpuPct(areas) {
const known = (areas || []).filter((a) => a.found && a.cpuPct != null);
if (!known.length) return null;
return known.reduce((sum, a) => sum + a.cpuPct, 0);
}
function describeBusy(busy) {
const rows = (busy || []).filter((row) => row?.comm && Number(row.cpu) >= 1).slice(0, 3);
if (!rows.length) return '';
return ` Busy outside the fleet: ${rows.map((row) => `${row.comm} ${row.cpu}%`).join(', ')}.`;
}
function collectBusy(snap, limits, opt) {
if (snap.cpu?.load1 == null || !snap.cpu.cores) return [];
if (snap.cpu.load1 / snap.cpu.cores < limits.loadPerCore) return [];
const fleet = fleetCpuPct(snap.areas);
if (fleet != null && fleet >= 10) return [];
if (Array.isArray(opt.busy)) return opt.busy;
if (opt.psBusy == null && (opt.sampleMs === 0 || opt.skipBusy)) return [];
const text = opt.psBusy != null ? opt.psBusy : sh('ps', ['-ax', '-o', 'pid=,pcpu=,comm='], 2000);
return parseBusyPs(text);
}
function verdictOf(crossed, battery) {
const onBattery = battery?.present && battery.state === 'discharging';
if (!crossed.length && onBattery) {
@@ -690,6 +759,7 @@ export function formatHealth(snap) {
const left = battery.minutesRemaining == null ? 'time unavailable' : `${battery.minutesRemaining} min left`;
lines.push(`battery ${battery.percent ?? '?'}% ${battery.state} ${left}`);
}
for (const note of snap.notes || []) lines.push(`note ${note.words}`);
if (!snap.areas?.length) lines.push('areas none');
for (const area of snap.areas || []) {
if (!area.found) lines.push(`area ${area.id} window not found`);
@@ -742,6 +812,9 @@ export function fleetHealth(c = {}, opt = {}) {
areas = readPsAreas(panes, opt);
}
}
if (memory?.status === 'ok') {
memory.swapInPages = opt.swapInPages != null ? opt.swapInPages : (sampled?.swapInPages ?? null);
}
const snap = {
at: opt.at || new Date().toISOString(),
platform,
@@ -753,7 +826,9 @@ export function fleetHealth(c = {}, opt = {}) {
battery,
areas,
thresholds: limits,
notes: [],
};
snap.busy = collectBusy(snap, limits, opt);
snap.crossed = judge(snap, limits);
snap.verdict = verdictOf(snap.crossed, snap.battery);
return snap;
+26 -3
View File
@@ -3,9 +3,10 @@
import fs from 'node:fs';
import path from 'node:path';
import { spawn } from 'node:child_process';
import { KIT, area as getArea, die, pidAlive, tryGit, git, mainRef, writeAtomic, today, landerIsUp, matchesAny, startLander, runTimedSync } from '../lib.mjs';
import { KIT, area as getArea, die, pidAlive, tryGit, git, mainRef, writeAtomic, today, landerIsUp, matchesAny, startLander, runTimedSync, readRun } from '../lib.mjs';
import { send as mailboxSendFile, read as mailboxReadDir, archive as mailboxArchiveDir } from '../../adapters/mailbox.mjs';
import { appendWakeLog, priorWakes } from './wake-log.mjs';
import { readHoldMap, readProgressMap, STALL_MIN } from './progress.mjs';
import { tmuxCommand, tmuxSocketFacts } from './tmux.mjs';
const STATUSES = ['open', 'in-progress', 'blocked', 'done', 'deferred'];
@@ -549,17 +550,39 @@ function senderOf(file) {
* That is the only recovery. It runs when someone looks at status, reads mail,
* hears, or tells. A filename already nudged is not nudged again. There is no timer.
*/
export function fleetStatus(c, { recoverAfterMs = RECOVER_AFTER_MS, idleAfterMs = IDLE_AFTER_MS } = {}) {
export function fleetStatus(c, { recoverAfterMs = RECOVER_AFTER_MS, idleAfterMs = IDLE_AFTER_MS, views = null } = {}) {
const state = readNudgeState(c);
let stateDirty = false;
const recovered = [];
const configured = Number(c.progress?.stallMin);
const stallMin = Number.isFinite(configured) && configured >= 0 ? configured : STALL_MIN;
const progress = readProgressMap(c);
const holds = readHoldMap(c);
const areas = c.areas.map((a) => {
const view = paneView(c, a.id, idleAfterMs);
const view = views?.[a.id] || paneView(c, a.id, idleAfterMs);
const unread = unreadFiles(c, a.id);
const mark = progress[a.id] || null;
const ageMin = mark?.at ? (Date.now() - Date.parse(mark.at)) / 60000 : null;
const row = {
id: a.id, role: a.role, harness: a.harness,
window: view.window, unread: unread.length, activity: view.activity,
progress: mark && Number.isFinite(ageMin) ? { at: mark.at, line: mark.line, ageMin: Math.round(ageMin) } : null,
hold: holds[a.id] || null,
stalled: false,
stallReason: null,
};
const quiet = view.window === 'alive' && unread.length === 0;
if (quiet && mark && Number.isFinite(ageMin) && ageMin >= stallMin) {
row.stalled = true;
row.stallReason = `no progress for ${Math.round(ageMin)} min and no unread mail`;
} else if (quiet && !mark) {
const started = Date.parse(readRun(c, a.id)?.startedAt || '');
const upMin = Number.isFinite(started) ? (Date.now() - started) / 60000 : null;
if (upMin != null && upMin >= stallMin) {
row.stalled = true;
row.stallReason = `no checkpoint for ${Math.round(upMin)} min and no unread mail`;
}
}
const oldest = unread[0];
const age = oldest ? Date.now() - oldest.mtimeMs : 0;
const already = (state[a.id] || []).includes(oldest?.file);
+70
View File
@@ -0,0 +1,70 @@
// Per-area checkpoint and hold records. Status and the check-in read them.
// There is no timer. An area writes a line when work advances, and a hold
// when something is blocked, naming who clears it and what is still allowed.
import fs from 'node:fs';
import path from 'node:path';
import { area as getArea, die, writeAtomic } from '../lib.mjs';
export const STALL_MIN = 20;
function safeId(id) {
if (!/^[a-z0-9][a-z0-9-]{0,62}$/.test(id || '')) die(`bad area "${id ?? ''}"`);
return id;
}
function fileIn(c, kind, id) {
return path.join(c.stateDir, kind, `${safeId(id)}.json`);
}
function readMap(c, kind) {
const dir = path.join(c.stateDir, kind);
const out = {};
if (!fs.existsSync(dir)) return out;
for (const name of fs.readdirSync(dir)) {
if (!name.endsWith('.json')) continue;
try { out[name.slice(0, -5)] = JSON.parse(fs.readFileSync(path.join(dir, name), 'utf8')); }
catch { /* a torn write is skipped; the next checkpoint replaces it */ }
}
return out;
}
export function readProgressMap(c) {
return readMap(c, 'progress');
}
export function readHoldMap(c) {
return readMap(c, 'holds');
}
export function writeProgress(c, area, line) {
const id = safeId(area);
getArea(c, id);
const text = String(line || '').trim();
if (!text) die('progress needs one line saying what advanced');
const rec = { at: new Date().toISOString(), area: id, line: text.slice(0, 400) };
writeAtomic(fileIn(c, 'progress', id), JSON.stringify(rec) + '\n');
return rec;
}
export function writeHold(c, area, { owner, unblock, next } = {}) {
const id = safeId(area);
getArea(c, id);
const rec = {
at: new Date().toISOString(),
area: id,
owner: String(owner || '').trim(),
unblock: String(unblock || '').trim(),
next: String(next || '').trim(),
};
if (!rec.owner || !rec.unblock || !rec.next) die('a hold needs --owner, --unblock, and --next');
writeAtomic(fileIn(c, 'holds', id), JSON.stringify(rec) + '\n');
return rec;
}
export function clearHold(c, area) {
const id = safeId(area);
getArea(c, id);
const file = fileIn(c, 'holds', id);
if (fs.existsSync(file)) fs.unlinkSync(file);
return { area: id, cleared: true };
}
+169
View File
@@ -0,0 +1,169 @@
// Startup readiness. One mailbox round trip, the Codex approval path, health,
// durable deps, and the last checkpoints. No model call and no daemon.
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { send, read, archive } from '../../adapters/mailbox.mjs';
import { fleetHealth } from './health.mjs';
import { readHoldMap, readProgressMap } from './progress.mjs';
const USAGE_RE = /five-hour|usage limit|%\s*remaining|rate limit/i;
export function codexConfigCandidates(repo, env = process.env, home = os.homedir()) {
const root = env.CODEX_HOME || path.join(home, '.codex');
return [path.join(root, 'config.toml'), path.join(repo, '.codex', 'config.toml')];
}
/**
* Read the agent-fleet approval mode out of a Codex config.toml.
* `approve` is the always-allow value. `auto` still requires approval for
* tools that are not marked read-only, and on-request auto-review then runs.
*/
export function parseCodexMcpApproval(text, server = 'agent-fleet') {
let section = '';
const result = { server: false, defaultMode: null, tools: {} };
for (const raw of String(text || '').split('\n')) {
const line = raw.replace(/\s+#.*$/, '').trim();
if (!line || line.startsWith('#')) continue;
const table = /^\[([^\]]+)\]$/.exec(line);
if (table) { section = table[1].trim(); continue; }
const kv = /^([A-Za-z0-9_.-]+)\s*=\s*(.+)$/.exec(line);
if (!kv) continue;
const val = kv[2].trim().replace(/^['"]|['"]$/g, '');
if (section === `mcp_servers.${server}` && kv[1] === 'default_tools_approval_mode') {
result.server = true;
result.defaultMode = val;
}
const tool = new RegExp(`^mcp_servers\\.${server}\\.tools\\.(.+)$`).exec(section);
if (tool && kv[1] === 'approval_mode') {
result.server = true;
result.tools[tool[1]] = val;
}
}
const sendMode = result.tools.mailbox_send;
const tellMode = result.tools.tell;
const mailOk = result.defaultMode === 'approve' || (sendMode === 'approve' && tellMode === 'approve');
return { ...result, mailOk };
}
export function readCodexApproval(repo, opt = {}) {
const files = [];
for (const file of codexConfigCandidates(repo, opt.env, opt.home)) {
if (!file || files.some((row) => row.file === file)) continue;
if (!fs.existsSync(file)) {
files.push({ file, found: false });
continue;
}
const parsed = parseCodexMcpApproval(fs.readFileSync(file, 'utf8'));
files.push({ file, found: true, ...parsed });
}
const hit = files.find((row) => row.mailOk) || files.find((row) => row.found && row.server) || null;
return {
files,
mailOk: Boolean(hit?.mailOk),
mode: hit?.defaultMode ?? null,
file: hit?.file ?? null,
note: 'fleet up runs codex with no --profile and does not set CODEX_HOME. The session reads the same config.toml as any other Codex process in that environment. The server name is agent-fleet. Shell fleet tell is a sandboxed command; default_tools_approval_mode does not apply to it.',
};
}
/** Prove send, read, and archive on an isolated mailbox. Does not wake a pane. */
export function mailboxRoundTrip(dir) {
const probe = path.join(dir, 'probe');
fs.rmSync(probe, { recursive: true, force: true });
const body = `ready ${Date.now()} ${process.pid}`;
try {
fs.mkdirSync(dir, { recursive: true });
const stamp = path.join(dir, `.write-${process.pid}`);
fs.writeFileSync(stamp, 'ok');
fs.unlinkSync(stamp);
const file = send(probe, 'ready', 'ready', body, 'round trip');
const msgs = read(probe, 'ready');
const hit = msgs.find((m) => m.text.includes(body));
if (!hit) return { ok: false, error: 'read did not see the message just written' };
archive(probe, 'ready', hit.file);
const left = read(probe, 'ready');
if (left.length) return { ok: false, error: 'archive left the message unread' };
return { ok: true, file: path.basename(file) };
} catch (e) {
return { ok: false, error: e instanceof Error ? e.message : String(e) };
} finally {
fs.rmSync(probe, { recursive: true, force: true });
}
}
export function depsReady(c) {
const root = path.join(c.stateDir, 'deps');
const manifest = path.join(root, 'manifest.txt');
const missing = [];
if (fs.existsSync(manifest)) {
for (const line of fs.readFileSync(manifest, 'utf8').split('\n')) {
const rel = line.trim();
if (!rel || rel.startsWith('#')) continue;
const file = path.resolve(root, rel);
if (file !== root && !file.startsWith(root + path.sep)) { missing.push(rel); continue; }
if (!fs.existsSync(file)) missing.push(rel);
}
}
const names = ['package.json', 'package-lock.json', 'pnpm-lock.yaml', 'yarn.lock', 'Cargo.lock', 'go.sum'];
const repoManifests = names.filter((name) => fs.existsSync(path.join(c.repo, name)));
return {
repoManifests,
manifest: fs.existsSync(manifest),
missing,
ok: missing.length === 0,
note: 'Keep runners, lockfiles, pinned archives, and evidence in the worktree or stateDir/deps. /tmp and the npm cache are disposable.',
};
}
export function usageSignal(c) {
const file = path.join(c.stateDir, 'wake-log.jsonl');
if (!fs.existsSync(file)) return { seen: false, line: null };
const lines = fs.readFileSync(file, 'utf8').split('\n').filter(Boolean).slice(-80);
const hit = [...lines].reverse().find((line) => USAGE_RE.test(line)) || null;
return { seen: Boolean(hit), line: hit ? hit.slice(0, 240) : null };
}
export function fleetReady(c, opt = {}) {
const mailbox = mailboxRoundTrip(path.join(c.stateDir, 'ready-mailbox'));
const approval = readCodexApproval(c.repo, opt);
const health = opt.health || fleetHealth(c, { sampleMs: 0, ...(opt.healthOpt || {}) });
const deps = depsReady(c);
const checkpoints = Object.values(readProgressMap(c));
const holds = Object.values(readHoldMap(c));
const usage = usageSignal(c);
return {
ok: mailbox.ok && deps.ok,
mailbox,
approval,
health: { verdict: health.verdict, notes: health.notes || [] },
deps,
checkpoints,
holds,
usage,
};
}
export function formatReady(report) {
const lines = [];
lines.push(report.mailbox.ok ? 'mailbox round trip ok' : `mailbox round trip failed: ${report.mailbox.error}`);
if (report.approval.mailOk) lines.push(`approval ${report.approval.mode || 'per-tool approve'} in ${report.approval.file}`);
else {
const seen = (report.approval.files || []).filter((row) => row.found && row.server);
const where = seen.length ? seen.map((row) => `${row.file} mode=${row.defaultMode || 'unset'}`).join('; ') : 'no [mcp_servers.agent-fleet] table found';
lines.push(`approval want default_tools_approval_mode = "approve" (or per-tool approval_mode = "approve" on mailbox_send and tell). ${where}. auto still asks, and on-request auto-review can reject the call. Offer to write approve only after the user says yes, then reload Codex.`);
}
lines.push(`health ${report.health.verdict}`);
for (const note of report.health.notes || []) lines.push(`health note ${note.words}`);
const manifests = report.deps.repoManifests.length ? report.deps.repoManifests.join(', ') : 'none in the repo root';
lines.push(`deps repo manifests: ${manifests}${report.deps.manifest ? '; stateDir/deps/manifest.txt present' : ''}`);
if (report.deps.missing.length) lines.push(`deps missing ${report.deps.missing.join(', ')}`);
lines.push(`deps note ${report.deps.note}`);
if (!report.checkpoints.length) lines.push('checkpoints none yet');
for (const row of report.checkpoints) lines.push(`checkpoint ${row.area} ${row.at} ${row.line}`);
if (!report.holds.length) lines.push('holds none');
for (const row of report.holds) lines.push(`hold ${row.area} owner ${row.owner}; unblock ${row.unblock}; next ${row.next}`);
if (report.usage.seen) lines.push('usage a recent wake line mentioned a usage limit. Checkpoint and hand off; resume from the checkpoints after the window resets.');
lines.push('resume continue from the checkpoints. A live window is not progress.');
return lines.join('\n');
}
+40
View File
@@ -39,6 +39,11 @@
// fleet status areas, run state (running / exited / failed / timeout / stopped), lander up or down
// fleet wakes recent wake attempts, and plain words for a likely failure
// fleet health [--json] one snapshot: CPU, memory, disk, temps, battery, per-area use
// fleet ready [--json] mailbox round trip, Codex approval path, health, deps, checkpoints
// fleet resume same as fleet up, after a reboot or a usage-limit stop
// fleet progress <area> <line> record what just advanced. Status uses this, not the pane
// fleet hold <area> --owner <who> --unblock <cond> --next <task>
// record a hold. --clear removes it
// fleet prompt <id> print the composed system prompt for an area
// fleet crontab PRINT the crontab lines for scheduled jobs (never installs)
// fleet run-job <name> explicit manual headless one-shot; cron uses scheduled-job
@@ -56,6 +61,8 @@ import { KIT, loadConfig, loadAdapter, area as getArea, dispatcher, sessionName,
import { commitWork, setupCommit, tellDispatch, hearInterpreter, fleetStatus } from './core/ops.mjs';
import { readWakeLog } from './core/wake-log.mjs';
import { fleetHealth, formatHealth } from './core/health.mjs';
import { fleetReady, formatReady } from './core/ready.mjs';
import { writeProgress, writeHold, clearHold } from './core/progress.mjs';
import { guardSchedule } from './core/schedule-guard.mjs';
import { tmuxCommand, rememberTmuxServer } from './core/tmux.mjs';
import { fleetUpdate, recordInstalled, isTemplatePath } from './core/update.mjs';
@@ -243,6 +250,7 @@ async function supervise(id) {
async function up() {
const fresh = has('--fresh'), bg = has('--bg'), dry = has('--dry-run');
if (!dry) console.log(formatReady(fleetReady(c)));
const sessions = readSessions();
const areas = pick();
const order = [...areas.filter((a) => a.role !== 'dispatcher'), ...areas.filter((a) => a.role === 'dispatcher')];
@@ -312,6 +320,9 @@ async function status() {
const row = report.areas.find((r) => r.id === a.id);
const live = row ? ` window=${row.window} unread=${row.unread} ${row.activity}` : '';
console.log(`${a.id.padEnd(18)} ${a.role.padEnd(10)} ${a.harness.padEnd(7)} ${`${a.tier}/${a.model ?? '-'}`.padEnd(16)} ctx=${String(a.context ?? '-').padEnd(5)} rc=${a.remoteControl ? 'on ' : 'off'} budget=${budgetMin > 0 ? `${budgetMin}m` : 'unlimited'} ${fs.existsSync(a.worktree) ? '' : 'NO-WORKTREE '}${runMark(a, sessions)}${live}`);
if (row?.stalled) console.log(` ${a.id} stalled: ${row.stallReason}`);
if (row?.hold) console.log(` ${a.id} hold: owner ${row.hold.owner}; unblock ${row.hold.unblock}; next ${row.hold.next}`);
else if (row?.progress) console.log(` ${a.id} progress: ${row.progress.line} (${row.progress.ageMin} min ago)`);
}
if (report.interpreter.unread) console.log(`interpreter unread=${report.interpreter.unread}`);
for (const id of report.recovered) console.log(`nudged ${id} again`);
@@ -323,6 +334,34 @@ function healthCmd() {
else console.log(formatHealth(report));
}
function readyCmd() {
const report = fleetReady(c);
if (argv.includes('--json')) console.log(JSON.stringify(report, null, 2));
else console.log(formatReady(report));
if (!report.ok) process.exitCode = 1;
}
function progressCmd() {
const area = argv[1];
const words = [];
for (let i = 2; i < argv.length; i++) {
if (argv[i] === '--config') { i++; continue; }
words.push(argv[i]);
}
const rec = writeProgress(c, area, words.join(' '));
console.log(`${rec.area} ${rec.at} ${rec.line}`);
}
function holdCmd() {
const area = argv[1];
if (argv.includes('--clear')) {
console.log(`cleared hold ${clearHold(c, area).area}`);
return;
}
const rec = writeHold(c, area, { owner: flag(argv, '--owner'), unblock: flag(argv, '--unblock'), next: flag(argv, '--next') });
console.log(`hold ${rec.area}: owner ${rec.owner}; unblock ${rec.unblock}; next ${rec.next}`);
}
function wakesCmd() {
const report = readWakeLog(c);
if (!report.entries.length) {
@@ -648,6 +687,7 @@ const run = { up, down, status, supervise: () => supervise(argv[1]), worktrees,
'scheduled-job': async () => { process.exitCode = (await guardSchedule(c, argv[1])).status; },
'run-job': () => runJob(argv[1]), 'sched-env': () => schedEnv(argv[1]), lander: () => lander(argv[1] ?? 'status'),
commit: commitCmd, 'setup-commit': setupCommitCmd, update: updateCmd, tell: tellCmd, hear: hearCmd, wakes: wakesCmd, health: healthCmd,
ready: readyCmd, resume: up, progress: progressCmd, hold: holdCmd,
doctor: () => tool('doctor'), landq: () => tool('landq'), backlog: () => tool('backlog'),
mailbox: () => tool('mailbox') }[cmd];
if (!run) die(`unknown command "${cmd}" (fleet help)`);
+1
View File
@@ -119,6 +119,7 @@ export function loadConfig(argv = process.argv, start = process.cwd()) {
c.clock = { quietHours: ['22:00', '08:00'], ...(c.clock || {}) };
c.doctor = { staleHours: 72, unownedDays: 14, ...(c.doctor || {}) };
c.health = healthThresholds(c.health);
c.progress = { stallMin: 20, ...(c.progress || {}) };
c.modules = { producer: false, reviewer: false, usageGovernor: false, ...(c.modules || {}) };
c.schedules ??= [];
c.areas = (c.areas || []).map((a) => resolveArea(c, a));
+11 -2
View File
@@ -12,6 +12,8 @@ import {
import { fleetUpdate } from './core/update.mjs';
import { readWakeLog } from './core/wake-log.mjs';
import { fleetHealth } from './core/health.mjs';
import { fleetReady } from './core/ready.mjs';
import { writeProgress, writeHold, clearHold } from './core/progress.mjs';
export const SERVER = { name: 'agent-fleet', version: '0.2.0' };
const INSTRUCTIONS = [
@@ -20,7 +22,8 @@ const INSTRUCTIONS = [
'mailbox_read at the start of a turn. Mail is always delivered; the prompt says when to act on it. Sandboxed agents commit with the commit tool (message plus paths, or all), not git commit. submit on that tool also queues the tip. Then go back to work. At your next natural checkpoint, land_status can report that submission. Waiting is optional. Never infer the lander or the land result from silence. landq_submit queues the committed tip (a dirty worktree warns and still queues HEAD) and starts a drain. Do not push main.',
'A scarce machine is a prompt convention: say so in .task and wait if a peer already has it. There is no slot lock.',
'Sessions, worktrees, and cron stay on the fleet CLI. This server does not start them.',
'setup_commit commits only the fleet setup files on the main checkout. A sandboxed agent uses it instead of git commit. update refreshes vendored runtime and the registered skill to this kit, leaves an edited prompt in place (the new template is a .kit file beside it), and commits only the paths it changed. It does not change fleet.config.json, product files, mail, or the landing queue. If that commit cannot write the index, it falls back to setup_commit. Mail to an open tmux window wakes it with a short line, not the body. mailbox_read and hear take those messages out of unread. fleet_status reports each window alive or dead, the unread count, and a working or idle guess. An idle area with mail that has been waiting is nudged once by that check. mailbox_read, hear, and tell run that same check. A second look does not nudge again. wake_log says whether the wake line was still in the composer, a later Enter submitted it, a waiter typed it after the pane was busy, the line left the composer but the mail is still unread, or the pane was not there. The missing-pane line names the windows and the tmux socket. Tell the user that case. Do not ask them to open the log. health is one snapshot of CPU, memory, swap, disk, temperature, battery, and each area process tree. Missing sensors are unavailable. Swap occupancy alone is not tight. The config health block is guidance. A lower cpuTempC is a per-repo edit, offered to the user, not a new default. A change to ~/.codex/config.toml needs a Codex reload. Say the verdict in plain words. Do not scrape the tmux pane.',
'Send mail with mailbox_send or tell. A shell fleet tell or mailbox send is a sandboxed command, and the MCP approval mode does not apply to it. progress writes the checkpoint that status uses to tell progress from a live window. hold records the owner, the exact unblock condition, and the next allowed task.',
'setup_commit commits only the fleet setup files on the main checkout. A sandboxed agent uses it instead of git commit. update refreshes vendored runtime and the registered skill to this kit, leaves an edited prompt in place (the new template is a .kit file beside it), and commits only the paths it changed. It does not change fleet.config.json, product files, mail, or the landing queue. If that commit cannot write the index, it falls back to setup_commit. Mail to an open tmux window wakes it with a short line, not the body. mailbox_read and hear take those messages out of unread. fleet_status reports each window alive or dead, the unread count, a working or idle guess, a stall when nothing has advanced, and any hold. An idle area with mail that has been waiting is nudged once by that check. mailbox_read, hear, and tell run that same check. A second look does not nudge again. wake_log says whether the wake line was still in the composer, a later Enter submitted it, a waiter typed it after the pane was busy, the line left the composer but the mail is still unread, or the pane was not there. The missing-pane line names the windows and the tmux socket. Tell the user that case. Do not ask them to open the log. health is one snapshot of CPU, memory, swap, disk, temperature, battery, and each area process tree. Missing sensors are unavailable. Near-full swap with plenty of available memory is not tight. Load from other programs is a note, not a hold. When health holds heavy work, keep doing light work. The config health block is guidance. A lower cpuTempC is a per-repo edit, offered to the user, not a new default. A change to ~/.codex/config.toml needs a Codex reload. Say the verdict in plain words. Do not scrape the tmux pane.',
].join(' ');
const cwdProp = { type: 'string', description: 'Directory to resolve the fleet from. Defaults to this server\'s cwd.' };
@@ -42,7 +45,10 @@ export const TOOLS = [
{ name: 'hear', description: 'Read unread mailbox messages for the interpreter and move them out of unread.', inputSchema: { type: 'object', properties: { cwd: cwdProp } } },
{ name: 'fleet_status', description: 'Each area: tmux window alive or dead, unread mail count, and a working or idle guess. Idle with unread mail that has been waiting is nudged once.', inputSchema: { type: 'object', properties: { cwd: cwdProp } } },
{ name: 'wake_log', description: 'Recent wake attempts from the fleet state dir. Each line says the pane state, what was typed or skipped, and a plain-words flag when a wake likely failed. No mail bodies.', inputSchema: { type: 'object', properties: { limit: { type: 'number' }, cwd: cwdProp } } },
{ name: 'health', description: 'One snapshot of CPU, memory, swap, disk free, temperature, battery, and per-area CPU and memory. Missing sensors are unavailable. The verdict names which suggested limits were crossed.', inputSchema: { type: 'object', properties: { cwd: cwdProp } } },
{ name: 'health', description: 'One snapshot of CPU, memory, swap, disk free, temperature, battery, and per-area CPU and memory. Missing sensors are unavailable. The verdict names which suggested limits were crossed. Near-full swap with plenty of free RAM is not tight. Load outside the fleet is a note.', inputSchema: { type: 'object', properties: { cwd: cwdProp } } },
{ name: 'ready', description: 'Startup check: one mailbox round trip, whether Codex approval for agent-fleet is approve, health, durable deps, checkpoints, and holds. Does not call a model.', inputSchema: { type: 'object', properties: { cwd: cwdProp } } },
{ name: 'progress', description: 'Record one line of real progress for an area. Status uses this to tell progress from a live window.', inputSchema: { type: 'object', properties: { area: { type: 'string' }, line: { type: 'string' }, cwd: cwdProp }, required: ['area', 'line'] } },
{ name: 'hold', description: 'Record a hold: owner, the exact unblock condition, and the next allowed task. clear removes it.', inputSchema: { type: 'object', properties: { area: { type: 'string' }, owner: { type: 'string' }, unblock: { type: 'string' }, next: { type: 'string' }, clear: { type: 'boolean' }, cwd: cwdProp }, required: ['area'] } },
];
function configFor(args) {
@@ -72,6 +78,9 @@ export function dispatchTool(name, args = {}) {
case 'fleet_status': return fleetStatus(c);
case 'wake_log': return readWakeLog(c, { limit: args.limit });
case 'health': return fleetHealth(c);
case 'ready': return fleetReady(c);
case 'progress': return writeProgress(c, args.area, args.line);
case 'hold': return args.clear ? clearHold(c, args.area) : writeHold(c, args.area, { owner: args.owner, unblock: args.unblock, next: args.next });
default: throw new FleetError(`unknown tool ${name}`);
}
}