thePR0M3TH3AN/bitvid
1
0 Fork 0
mirror of https://github.com/PR0M3TH3AN/bitvid.git synced 2026-11-02 19:02:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #2474 from PR0M3TH3AN/claude/fix-jules-task-locking-fVCuW

Browse Source 
Add Nostr-based distributed task locking for multi-agent coordination
This commit is contained in:
thePR0M3TH3AN authored and GitHub committed 2026-02-13 16:11:50 -05:00
2 parents b1be38a24b 9d4c98a872
commit fe39d9500d
7 files changed
+1160 -61
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/agents/TORCH.md
+309
View File
@@ -0,0 +1,309 @@
# TORCH: Task Orchestration via Relay-Coordinated Handoff
TORCH is a decentralized task-locking protocol for multi-agent development. It uses the Nostr protocol to coordinate AI coding agents (Claude Code, Google Jules, OpenAI Codex, etc.) working in parallel on the same repository, preventing them from colliding on the same task.
## The Problem
When multiple AI agents operate on a shared codebase, they need a way to signal "I'm working on this" so other agents don't duplicate the effort. The traditional approach uses draft pull requests as locks, but this breaks down in practice:
- **Sandboxed environments** (e.g., Google Jules) block `git push` and don't provide GitHub CLI tools.
- **Token management** adds operational overhead — GitHub PATs must be created, rotated, and injected into each agent environment.
- **Platform coupling** ties the coordination mechanism to GitHub specifically, making it non-portable.
- **Complex reasoning chains** force agents to parse jq pipelines, handle multi-method fallbacks, and perform manual race detection, consuming tokens and increasing error rates.
## The Solution
TORCH replaces all of that with a single protocol:
1. An agent **publishes a Nostr event** to claim a task.
2. Other agents **query the relay** to see what's claimed.
3. Locks **auto-expire** via NIP-40 — no cleanup required.
4. Each lock uses an **ephemeral keypair** — no secrets to manage.
The entire locking mechanism is handled by a single script (`scripts/agent/nostr-lock.mjs`) that any agent can call with two commands. The agent's reasoning chain reduces to: "run check, run lock, read exit code."
## How It Works
### Protocol Overview
```
Agent A Nostr Relays Agent B
| | |
|-- check (query locks) ---->| |
|<--- JSON: locked=[] -------| |
| | |
|-- lock (publish event) --->| |
|<--- OK (published) --------| |
| | |
|-- race check (re-query) -->| |
|<--- only my event ---------| |
| | |
| RACE CHECK: won | |
| (begin work) | |
| | |
| |<-- check (query locks) ----|
| |--- JSON: locked=[A] ------>|
| | |
| | (Agent B picks |
| | a different task) |
| | |
| (2 hours pass) | |
| |--- lock auto-expires ----->|
| | |
```
### Event Schema (NIP-78 / Kind 30078)
Each lock is a NIP-33 parameterized replaceable event:
```json
{
"kind": 30078,
"pubkey": "<ephemeral-pubkey>",
"created_at": 1739484362,
"tags": [
["d", "bitvid-lock/daily/nip-research-agent/2026-02-13"],
["t", "bitvid-agent-lock"],
["t", "bitvid-lock-daily"],
["t", "bitvid-lock-daily-2026-02-13"],
["expiration", "1739491562"]
],
"content": "{\"agent\":\"nip-research-agent\",\"cadence\":\"daily\",\"status\":\"started\",\"date\":\"2026-02-13\",\"platform\":\"jules\",\"lockedAt\":\"2026-02-13T20:06:02.000Z\",\"expiresAt\":\"2026-02-13T22:06:02.000Z\"}"
}
```
**Tag breakdown:**
| Tag | Purpose |
|-----|---------|
| `d` | NIP-33 identifier: `<namespace>/<cadence>/<agent>/<date>`. Makes the event replaceable per agent per day. |
| `t` (bitvid-agent-lock) | Broad category filter for all TORCH lock events. |
| `t` (bitvid-lock-daily) | Cadence-specific filter. |
| `t` (bitvid-lock-daily-2026-02-13) | Date-specific filter for precise relay-side queries. |
| `expiration` | NIP-40 auto-expiration timestamp. Relays may garbage-collect after this time. |
### Ephemeral Keys
Each `lock` invocation generates a fresh secp256k1 keypair:
```
generateSecretKey() → 32 random bytes → sign event → discard key
```
The private key exists only in process memory for the duration of the script. It is never written to disk, never stored, never reused. The public key is included in the event for Nostr protocol compliance but has no ongoing identity or authority.
This means:
- **No secret management.** No tokens, no `.env` files, no rotation schedules.
- **No key reuse.** Every lock is cryptographically independent.
- **No cleanup authority.** The lock can't be manually deleted (the key is gone), but it doesn't need to be — NIP-40 expiration handles it.
### Race Detection
When two agents try to claim the same task simultaneously:
1. Both agents run `check` — neither sees a lock.
2. Both agents run `lock` — both publish events to relays.
3. Both agents re-query after a 1.5-second propagation delay.
4. Both agents see two lock events for the same task.
5. The event with the **earlier `created_at`** wins.
6. The loser exits with code 3. The winner proceeds.
The losing agent's event remains on relays but is harmless — it auto-expires, and it doesn't prevent the winner from working. Other agents checking later see both events but this just means the task appears (correctly) as locked.
### Auto-Expiration
Locks expire via NIP-40 after a configurable TTL (default: 2 hours). This provides:
- **Self-healing.** If an agent crashes mid-task, its lock doesn't persist forever.
- **No stale state.** No equivalent of orphaned draft PRs blocking future runs.
- **Zero maintenance.** No cron jobs to clean up expired locks, no manual intervention.
The client-side `check` command also filters by expiration, so even if a relay hasn't garbage-collected an expired event, it won't appear in the exclusion set.
## Usage
### Commands
**Check what's locked:**
```bash
node scripts/agent/nostr-lock.mjs check --cadence daily
```
Returns JSON to stdout:
```json
{
"cadence": "daily",
"date": "2026-02-13",
"locked": ["audit-agent", "ci-health-agent"],
"available": ["const-refactor-agent", "content-audit-agent", ...],
"lockCount": 2,
"locks": [...]
}
```
**Claim a task:**
```bash
AGENT_PLATFORM=jules \
node scripts/agent/nostr-lock.mjs lock \
--agent nip-research-agent \
--cadence daily
```
Returns key=value pairs to stdout:
```
LOCK_STATUS=ok
LOCK_EVENT_ID=ea2724c76d2bd707...
LOCK_PUBKEY=7ee606dd0619e339...
LOCK_AGENT=nip-research-agent
LOCK_CADENCE=daily
LOCK_DATE=2026-02-13
LOCK_EXPIRES=1739491562
LOCK_EXPIRES_ISO=2026-02-13T22:06:02.000Z
```
Exit codes:
- `0` — Lock acquired.
- `3` — Race lost or task already locked.
- `2` — Relay error.
- `1` — Usage error.
**List all active locks (human-readable):**
```bash
node scripts/agent/nostr-lock.mjs list
```
**Test without publishing:**
```bash
node scripts/agent/nostr-lock.mjs lock --agent test --cadence daily --dry-run
```
### Configuration
| Environment Variable | Default | Description |
|---------------------|---------|-------------|
| `NOSTR_LOCK_RELAYS` | `wss://relay.damus.io,wss://nos.lol,wss://relay.primal.net` | Comma-separated relay URLs |
| `NOSTR_LOCK_TTL` | `7200` (2 hours) | Lock expiration in seconds |
| `AGENT_PLATFORM` | `unknown` | Platform identifier included in lock metadata |
## Integration with the Scheduler
The bitvid scheduler uses TORCH as its sole coordination mechanism. The full flow is documented in `docs/agents/prompts/scheduler-flow.md`, but the TORCH-specific part is just two steps:
```bash
# 1. Build exclusion set
node scripts/agent/nostr-lock.mjs check --cadence daily
# → Use the "locked" array to skip already-claimed agents
# 2. Claim the selected agent
AGENT_PLATFORM=jules \
node scripts/agent/nostr-lock.mjs lock --agent <agent-name> --cadence daily
# → Exit 0: proceed with work
# → Exit 3: pick a different agent
```
The scheduler meta prompts (`docs/agents/prompts/META_PROMPTS.md`) embed these commands directly, so agents execute them without needing to understand the underlying protocol.
## Architecture
```
┌──────────────────────────────────────────────────────────┐
│ Agent Platforms │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Claude Code │ │ Google Jules│ │ OpenAI Codex│ │
│ │ │ │ │ │ │ │
│ │ nostr-lock │ │ nostr-lock │ │ nostr-lock │ │
│ │ .mjs │ │ .mjs │ │ .mjs │ │
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
│ │ │ │ │
└─────────┼────────────────┼────────────────┼──────────────┘
│ │ │
│ WebSocket (NIP-01) │
│ │ │
┌─────────▼────────────────▼────────────────▼──────────────┐
│ Nostr Relay Network │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │relay.damus.io│ │ nos.lol │ │relay.primal │ │
│ │ │ │ │ │ .net │ │
│ │ Kind 30078 │ │ Kind 30078 │ │ Kind 30078 │ │
│ │ lock events │ │ lock events │ │ lock events │ │
│ │ (NIP-40 │ │ (NIP-40 │ │ (NIP-40 │ │
│ │ auto-exp) │ │ auto-exp) │ │ auto-exp) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │
│ Events replicate across relays. Queries go to all. │
│ Publishing succeeds if at least 1 relay accepts. │
└──────────────────────────────────────────────────────────┘
```
### Dependencies
TORCH uses only packages already in the bitvid project:
- **nostr-tools** (v2.19.4) — Key generation, event signing, relay pool
- **ws** (v8.19.0) — WebSocket for Node.js relay connections
No additional dependencies are needed.
## Why Nostr?
The bitvid project is built on Nostr. Using Nostr for agent coordination means:
1. **No new infrastructure.** Public relays already exist and are free to use.
2. **No platform coupling.** Works with any agent platform that has Node.js and WebSocket access, regardless of their relationship with GitHub.
3. **No credentials.** Ephemeral keys eliminate the entire category of secret management.
4. **Protocol-native race resolution.** NIP-33 replaceable events and relay-side ordering provide deterministic conflict resolution without custom logic.
5. **Self-healing by default.** NIP-40 expiration means locks clean themselves up.
6. **Dogfooding.** A Nostr project using Nostr for its own development tooling.
## Toward a Generalizable Drop-In
TORCH is designed to be extracted from bitvid and used in any repository. The components that would need to be generalized:
| Component | Current State | To Generalize |
|-----------|--------------|---------------|
| `nostr-lock.mjs` | Hardcoded bitvid agent rosters | Accept roster via config file or CLI flag |
| Tag namespace | `bitvid-agent-lock` | Configurable project prefix |
| `d` tag pattern | `bitvid-lock/<cadence>/<agent>/<date>` | `<project>-lock/<cadence>/<agent>/<date>` |
| Relay defaults | 3 hardcoded relays | Config file with sensible defaults |
| Scheduler flow | bitvid-specific docs | Template scheduler docs with project placeholders |
| Agent rosters | Inline arrays in the script | External `torch.config.json` or similar |
A future `torch` npm package could look like:
```bash
# Install
npm install --save-dev @bitvid/torch
# Initialize in a new repo
npx torch init --project myproject --relays wss://relay.damus.io
# Creates:
# torch.config.json — project name, relays, TTL, roster
# scripts/torch-lock.mjs — pre-configured lock script
# docs/torch-scheduler.md — template scheduler flow
# Use in agent prompts
npx torch check --cadence daily
npx torch lock --agent my-agent --cadence daily
```
The protocol itself (NIP-78 events with NIP-40 expiration) is already generic. The bitvid-specific parts are just the roster definitions and tag namespaces.
## Related Files
| File | Purpose |
|------|---------|
| `scripts/agent/nostr-lock.mjs` | TORCH lock script (check, lock, list) |
| `docs/agents/prompts/scheduler-flow.md` | Scheduler procedure using TORCH |
| `docs/agents/prompts/META_PROMPTS.md` | Agent meta prompts with TORCH commands |
| `AGENTS.md` | Task Claiming Protocol referencing TORCH |
| `scripts/agents/claim-audit.mjs` | Legacy claim audit (predates TORCH) |
## NIPs Referenced
| NIP | Usage in TORCH |
|-----|---------------|
| [NIP-01](https://github.com/nostr-protocol/nips/blob/master/01.md) | Basic event structure and relay communication |
| [NIP-33](https://github.com/nostr-protocol/nips/blob/master/33.md) | Parameterized replaceable events (d-tag uniqueness) |
| [NIP-40](https://github.com/nostr-protocol/nips/blob/master/40.md) | Expiration tag for auto-expiring locks |
| [NIP-78](https://github.com/nostr-protocol/nips/blob/master/78.md) | Application-specific data (Kind 30078) |
docs/agents/prompts/META_PROMPTS.md
+24 -20
View File
@@ -17,22 +17,24 @@ MUST 1: Set cadence config to:
- branch_prefix = agents/daily/
- prompt_dir = docs/agents/prompts/daily/
MUST 2: Run this exact preflight command first:
MUST 2: Run preflight to get the exclusion set:
curl -s "https://api.github.com/repos/PR0M3TH3AN/bitvid/pulls?state=open&per_page=100" | jq '{count: length, prs: [.[] | {number, created_at, draft, head: {ref: .head.ref}, agent: ((.head.ref | capture("^agents/daily/(?<agent>[^/]+)/")?.agent) // (.title | capture("(?<agent>[A-Za-z0-9-]+-agent)")?.agent // null))}] | sort_by(.created_at, .number)}'
node scripts/agent/nostr-lock.mjs check --cadence daily
MUST 3: Run the exact command list in this order:
Use the `locked` array from the JSON output as the exclusion set.
MUST 3: Run these commands in this order:
1) cat AGENTS.md CLAUDE.md
2) ls -1 docs/agents/task-logs/daily/ | sort | tail -n 1
3) [claim] create branch, create+commit `_started.md`, push
4) rerun preflight command from MUST 2
5) execute selected prompt from docs/agents/prompts/daily/
3) Select next roster agent not in exclusion set
4) Claim via Nostr lock:
AGENT_PLATFORM=jules node scripts/agent/nostr-lock.mjs lock --agent <agent-name> --cadence daily
Exit 0 = lock acquired, proceed. Exit 3 = race lost, go back to step 2.
5) Execute selected prompt from docs/agents/prompts/daily/
6) npm run lint
7) create `_completed.md` or `_failed.md`, commit, push
7) Create `_completed.md` or `_failed.md`, commit, push
MUST 4: Race rule: after step 3, compare only matching derived `agent` claims from the rerun. Earlier `created_at` wins; if tied, lower PR number wins. Print exactly one line: `RACE CHECK: won` or `RACE CHECK: lost (agent already claimed by PR #<number>)`.
MUST 5: If all daily agents are excluded, stop and write `_failed.md` with this exact reason: `All roster tasks currently claimed by other agents`.
MUST 4: If all daily agents are excluded, stop and write `_failed.md` with this exact reason: `All roster tasks currently claimed by other agents`.
```
## Weekly Scheduler Meta Prompt
@@ -48,20 +50,22 @@ MUST 1: Set cadence config to:
- branch_prefix = agents/weekly/
- prompt_dir = docs/agents/prompts/weekly/
MUST 2: Run this exact preflight command first:
MUST 2: Run preflight to get the exclusion set:
curl -s "https://api.github.com/repos/PR0M3TH3AN/bitvid/pulls?state=open&per_page=100" | jq '{count: length, prs: [.[] | {number, created_at, draft, head: {ref: .head.ref}, agent: ((.head.ref | capture("^agents/weekly/(?<agent>[^/]+)/")?.agent) // (.title | capture("(?<agent>[A-Za-z0-9-]+-agent)")?.agent // null))}] | sort_by(.created_at, .number)}'
node scripts/agent/nostr-lock.mjs check --cadence weekly
MUST 3: Run the exact command list in this order:
Use the `locked` array from the JSON output as the exclusion set.
MUST 3: Run these commands in this order:
1) cat AGENTS.md CLAUDE.md
2) ls -1 docs/agents/task-logs/weekly/ | sort | tail -n 1
3) [claim] create branch, create+commit `_started.md`, push
4) rerun preflight command from MUST 2
5) execute selected prompt from docs/agents/prompts/weekly/
3) Select next roster agent not in exclusion set
4) Claim via Nostr lock:
AGENT_PLATFORM=jules node scripts/agent/nostr-lock.mjs lock --agent <agent-name> --cadence weekly
Exit 0 = lock acquired, proceed. Exit 3 = race lost, go back to step 2.
5) Execute selected prompt from docs/agents/prompts/weekly/
6) npm run lint
7) create `_completed.md` or `_failed.md`, commit, push
7) Create `_completed.md` or `_failed.md`, commit, push
MUST 4: Race rule: after step 3, compare only matching derived `agent` claims from the rerun. Earlier `created_at` wins; if tied, lower PR number wins. Print exactly one line: `RACE CHECK: won` or `RACE CHECK: lost (agent already claimed by PR #<number>)`.
MUST 5: If all weekly agents are excluded, stop and write `_failed.md` with this exact reason: `All roster tasks currently claimed by other agents`.
MUST 4: If all weekly agents are excluded, stop and write `_failed.md` with this exact reason: `All roster tasks currently claimed by other agents`.
```
docs/agents/prompts/scheduler-flow.md
+56 -23
View File
@@ -1,6 +1,6 @@
# Scheduler Flow (Single Source of Truth)
Use this document for **all scheduler runs**.
Use this document for **all scheduler runs**. Task locking uses the **TORCH** protocol (see `docs/agents/TORCH.md`).
## Scheduler Override (Top-Line Rule)
@@ -27,61 +27,94 @@ During scheduler execution, the scheduler-specific instructions in this document
- `branch_prefix` = `agents/<cadence>/`
- `prompt_dir` = `docs/agents/prompts/<cadence>/`
2. **MUST** run this command first and save the JSON output as the preflight exclusion set:
2. **MUST** run the preflight check to build the exclusion set:
```bash
curl -s "https://api.github.com/repos/PR0M3TH3AN/bitvid/pulls?state=open&per_page=100" | jq '{count: length, prs: [.[] | {number, created_at, draft, head: {ref: .head.ref}, agent: ((.head.ref | capture("^agents/<cadence>/(?<agent>[^/]+)/")?.agent) // (.title | capture("(?<agent>[A-Za-z0-9-]+-agent)")?.agent // null))}] | sort_by(.created_at, .number)}'
node scripts/agent/nostr-lock.mjs check --cadence <cadence>
```
3. **MUST** derive excluded agents from command output using this order only:
1) parse `head.ref` as `agents/<cadence>/<agent-name>/...`; 2) fallback to title regex; 3) if still unknown, stop with a global lock message.
This queries Nostr relays for active lock events and returns JSON with `locked` and `available` agent lists. Use the `locked` array as the exclusion set.
4. **MUST** run this command and read both policy files before selecting an agent:
3. **MUST** read both policy files before selecting an agent:
```bash
cat AGENTS.md CLAUDE.md
```
5. **MUST** run this command to identify the latest cadence log file, then choose the next roster agent not excluded by Step 2:
4. **MUST** run this command to identify the latest cadence log file, then choose the next roster agent not in the exclusion set:
```bash
ls -1 <log_dir> | sort | tail -n 1
```
6. **MUST** stop and write a `_failed.md` log with `All roster tasks currently claimed by other agents` when every roster agent is excluded.
5. **MUST** stop and write a `_failed.md` log with `All roster tasks currently claimed by other agents` when every roster agent is excluded.
7. **MUST** claim the selected agent in this exact order:
1) create branch,
2) create and commit `<timestamp>_<agent>_started.md` in `<log_dir>`,
3) push branch and commit.
6. **MUST** claim the selected agent:
8. **MUST** re-run the same PR preflight command from Step 2 immediately after push. If an earlier claim exists for the same derived agent, print `RACE CHECK: lost (agent already claimed by PR #<number>)`, abandon the run, and return to Step 5. Otherwise print `RACE CHECK: won`.
```bash
AGENT_PLATFORM=<jules|claude-code|codex> \
node scripts/agent/nostr-lock.mjs lock \
--agent <agent-name> \
--cadence <cadence>
```
9. **MUST** execute `<prompt_dir>/<prompt-file>` end-to-end.
The script generates an ephemeral keypair, publishes a NIP-78 lock event to Nostr relays with NIP-40 auto-expiration (2 hours), and performs a built-in race check. No tokens or secrets needed.
10. **MUST** create exactly one final status file (`_completed.md` or `_failed.md`), run `npm run lint`, then commit and push.
- **Exit code 0** = lock acquired, proceed to Step 7.
- **Exit code 3** = race lost (another agent claimed first). Return to Step 2.
- **Exit code 2** = relay error. Write a `_failed.md` log with reason `Nostr relay error` and stop.
## Canonical Example Run Output
7. **MUST** execute `<prompt_dir>/<prompt-file>` end-to-end.
8. **MUST** create exactly one final status file (`_completed.md` or `_failed.md`), run `npm run lint`, then commit and push.
## Canonical Example Run
```text
$ curl -s "https://api.github.com/repos/PR0M3TH3AN/bitvid/pulls?state=open&per_page=100" | jq '{count: length, prs: [.[] | {number, created_at, draft, head: {ref: .head.ref}, agent: ((.head.ref | capture("^agents/daily/(?<agent>[^/]+)/")?.agent) // (.title | capture("(?<agent>[A-Za-z0-9-]+-agent)")?.agent // null))}] | sort_by(.created_at, .number)}'
{ "count": 12, "prs": [ ... ] }
# Step 1: Set cadence
cadence=daily
# Step 2: Preflight — check Nostr locks
$ node scripts/agent/nostr-lock.mjs check --cadence daily
{
"cadence": "daily",
"date": "2026-02-14",
"locked": ["audit-agent"],
"available": ["ci-health-agent", "const-refactor-agent", ...],
"lockCount": 1,
"locks": [...]
}
# Step 3: Read policies
$ cat AGENTS.md CLAUDE.md
# bitvid — AI Agent Guide
...
# Step 4: Check logs, select next agent
$ ls -1 docs/agents/task-logs/daily/ | sort | tail -n 1
2026-02-13_18-40-00_load-test-agent_completed.md
Selected agent: nip-research-agent
Created: docs/agents/task-logs/daily/2026-02-14_00-00-00_nip-research-agent_started.md
Pushed branch: agents/daily/nip-research-agent/2026-02-14-run
$ curl -s "https://api.github.com/repos/PR0M3TH3AN/bitvid/pulls?state=open&per_page=100" | jq '{count: length, prs: [.[] | {number, created_at, draft, head: {ref: .head.ref}, agent: ((.head.ref | capture("^agents/daily/(?<agent>[^/]+)/")?.agent) // (.title | capture("(?<agent>[A-Za-z0-9-]+-agent)")?.agent // null))}] | sort_by(.created_at, .number)}'
{ "count": 13, "prs": [ ... ] }
# Step 6: Claim via Nostr lock
$ AGENT_PLATFORM=jules node scripts/agent/nostr-lock.mjs lock --agent nip-research-agent --cadence daily
Locking: agent=nip-research-agent, cadence=daily, date=2026-02-14
Step 1: Checking for existing locks...
Step 2: Generating ephemeral keypair...
Step 3: Building lock event...
Step 4: Publishing to relays...
Published to 3/3 relays
Step 5: Race check...
RACE CHECK: won
LOCK_STATUS=ok
LOCK_EVENT_ID=ea2724c76d2bd707...
LOCK_AGENT=nip-research-agent
LOCK_CADENCE=daily
LOCK_EXPIRES_ISO=2026-02-14T02:00:00.000Z
# Step 7: Execute agent prompt
(agent runs nip-research-agent prompt...)
# Step 8: Finalize
$ npm run lint
> lint passed