diff --git a/docs/agents/PROMPT_LIBRARY_STATUS.md b/docs/agents/PROMPT_LIBRARY_STATUS.md new file mode 100644 index 00000000..3fbd88aa --- /dev/null +++ b/docs/agents/PROMPT_LIBRARY_STATUS.md @@ -0,0 +1,129 @@ +# Prompt Library Status + +Last updated: 2026-02-12 +Run type: Weekly (initial baseline) + +## Summary + +- **Total prompts**: 36 (21 daily, 15 weekly) + 2 schedulers +- **Health**: Generally good. Most prompts follow the canonical structure. + Several P1/P2 issues fixed this run; two truncated prompts require + human attention. + +## Changes This Run + +### P1 Fixes (Policy Violations) + +| File | Issue | Fix | +|------|-------|-----| +| `daily/bitvid-content-audit-agent.md` | Agent name typo: `bitvd` (missing `i`) | Fixed to `bitvid-content-audit-agent` | +| `daily/bitvid-content-audit-agent.md` | Emoji in PR title (`🌐`) | Removed emoji | +| `daily/bitvid-docs-alignment-agent.md` | Emoji in PR title (`📘`) and body (`💡🎯✅📌`) | Removed all emoji | +| `daily/bitvid-perf-agent.md` | Emoji in docs PR title (`🌐`) | Removed emoji | +| `weekly/bitvid-perf-optimization-agent.md` | Emoji in PR title (`⚡`) and body (`💡🎯📊`) | Removed all emoji | +| `daily/bitvid-style-agent.md` | Identity says `bitvid-stylelint-agent` but filename is `bitvid-style-agent.md` | Fixed identity to match filename | + +### P2 Fixes (Behavioral Defects) + +| File | Issue | Fix | +|------|-------|-----| +| `daily/bitvid-docs-code-investigator.md` | AI generation artifacts (`:contentReference[oaicite:N]{index=N}`) in 4 locations | Removed all artifacts | +| `weekly/bitvid-event-schema-agent.md` | Example uses `require()` but project is ES modules (`"type": "module"`) | Changed to `import` syntax | +| `daily/bitvid-const-refactor-agent.md` | References `scripts/find-numeric-literals.js` as if it exists (it doesn't) | Clarified the script must be created first | +| `weekly/bitvid-interop-agent.md` | File truncated mid-code-block; missing Failure Modes, Outputs sections | Closed code block, added truncation notice | +| `weekly/bitvid-smoke-agent.md` | File truncated mid-code-block; missing Failure Modes, PR, Outputs sections | Closed code block, added truncation notice | + +## Known Remaining Issues + +### Requires Human Action + +1. **bitvid-interop-agent.md** — Truncated at line 138. Missing sections: + Failure Modes, Outputs Per Run. Needs human to complete the prompt. +2. **bitvid-smoke-agent.md** — Truncated at line 156. Missing sections: + Failure Modes, PR & Commit Conventions, Outputs Per Run. Needs human + to complete the prompt. + +### P3 (Structural Inconsistencies — Low Priority) + +- Section heading styles vary: some prompts use `═══`, some use `───`, + some use `---`. This is cosmetic and does not affect behavior. +- Several prompts lack a formal "Goals & Success Criteria" header + (content is present but under different names): + - `weekly/bitvid-dead-code-agent.md` — criteria spread across HARD + GUARDRAILS and QUALITY BAR sections + - `weekly/bitvid-telemetry-agent.md` — criteria in PRIVACY & OPT-IN RULES +- Several prompts lack a formal "Failure Modes" header (content covered + in risk/security sections): + - `weekly/bitvid-perf-deepdive-agent.md` + - `weekly/bitvid-race-condition-agent.md` +- `weekly/bitvid-perf-optimization-agent.md` — `DIAGNOSIS.md`, + `BASELINE.md`, `AFTER.md` deliverable locations not specified + (could clutter repo root). +- `weekly/bitvid-pr-review-agent.md` — uses emoji in PR comment + templates (checkmarks, warning signs). Lower impact than PR titles + but inconsistent with CLAUDE.md guidance. + +### P4 (Polish — Lowest Priority) + +- Some prompts have redundant or overly verbose workflow descriptions + that could be tightened without losing clarity. +- `weekly/bitvid-event-schema-agent.md` — uses `✓` and `⚠️` in + reporting templates. Minor inconsistency. + +## Inventory + +### Daily Prompts (21) + +| # | Agent Name | Prompt File | Canonical Sections | +|---|-----------|-------------|-------------------| +| 1 | audit-agent | `bitvid-audit-agent.md` | Complete | +| 2 | ci-health-agent | `bitvid-ci-health-agent.md` | Complete | +| 3 | const-refactor-agent | `bitvid-const-refactor-agent.md` | Complete | +| 4 | content-audit-agent | `bitvid-content-audit-agent.md` | Complete | +| 5 | decompose-agent | `bitvid-decompose-agent.md` | Complete | +| 6 | deps-security-agent | `bitvid-deps-security-agent.md` | Complete | +| 7 | design-system-audit-agent | `bitvid-design-system-audit-agent.md` | Complete | +| 8 | docs-agent | `bitvid-docs-agent.md` | Complete | +| 9 | docs-alignment-agent | `bitvid-docs-alignment-agent.md` | Complete | +| 10 | docs-code-investigator | `bitvid-docs-code-investigator.md` | Complete | +| 11 | innerhtml-migration-agent | `bitvid-innerhtml-migration-agent.md` | Complete | +| 12 | known-issues-agent | `bitvid-known-issues-agent.md` | Complete | +| 13 | load-test-agent | `bitvid-load-test-agent.md` | Complete | +| 14 | nip-research-agent | `bitvid-nip-research-agent.md` | Complete | +| 15 | onboarding-audit-agent | `bitvid-onboarding-audit-agent.md` | Complete | +| 16 | perf-agent | `bitvid-perf-agent.md` | Complete | +| 17 | prompt-curator-agent | `bitvid-prompt-curator-agent.md` | Complete | +| 18 | scheduler-update-agent | `bitvid-scheduler-update-agent.md` | Complete | +| 19 | style-agent | `bitvid-style-agent.md` | Complete | +| 20 | test-audit-agent | `bitvid-test-audit-agent.md` | Complete | +| 21 | todo-triage-agent | `bitvid-todo-triage-agent.md` | Complete | + +### Weekly Prompts (15) + +| # | Agent Name | Prompt File | Canonical Sections | +|---|-----------|-------------|-------------------| +| 1 | bug-reproducer-agent | `bitvid-bug-reproducer-agent.md` | Complete | +| 2 | changelog-agent | `bitvid-changelog-agent.md` | Complete | +| 3 | dead-code-agent | `bitvid-dead-code-agent.md` | Missing formal Goals header | +| 4 | event-schema-agent | `bitvid-event-schema-agent.md` | Complete | +| 5 | frontend-console-debug-agent | `bitvid-frontend-console-debug-agent.md` | Complete | +| 6 | fuzz-agent | `bitvid-fuzz-agent.md` | Complete | +| 7 | interop-agent | `bitvid-interop-agent.md` | **Truncated** | +| 8 | perf-deepdive-agent | `bitvid-perf-deepdive-agent.md` | Missing formal Failure Modes header | +| 9 | perf-optimization-agent | `bitvid-perf-optimization-agent.md` | Complete | +| 10 | pr-review-agent | `bitvid-pr-review-agent.md` | Complete | +| 11 | race-condition-agent | `bitvid-race-condition-agent.md` | Missing formal Failure Modes header | +| 12 | refactor-agent | `bitvid-refactor-agent.md` | Complete | +| 13 | smoke-agent | `bitvid-smoke-agent.md` | **Truncated** | +| 14 | telemetry-agent | `bitvid-telemetry-agent.md` | Missing formal Goals header | +| 15 | test-coverage-agent | `bitvid-test-coverage-agent.md` | Complete | +| 16 | weekly-synthesis-agent | `bitvid-weekly-synthesis-agent.md` | Complete | + +## Positive Patterns + +- All prompts have Authority Hierarchy with `AGENTS.md` at the top. +- All prompts correctly defer to policy when conflicts arise. +- No prompt instructs agents to use `console.*` for production logging. +- All script references verified to exist (except where noted above). +- Branch naming consistently follows repo conventions. +- Security-sensitive guardrails are consistently present. diff --git a/docs/agents/RESEARCH_LOG.md b/docs/agents/RESEARCH_LOG.md new file mode 100644 index 00000000..43d9ae61 --- /dev/null +++ b/docs/agents/RESEARCH_LOG.md @@ -0,0 +1,86 @@ +# Prompt Library Research Log + +Dated entries from web research on prompt-engineering and context-engineering +best practices. Each entry records search queries, key takeaways, sources, +and what changed in the library (or why nothing changed). + +--- + +## 2026-02-12 — Initial Weekly Research + +### Search Queries +- "Anthropic context engineering prompt engineering best practices 2025 2026 agent prompts" +- "OpenAI prompt engineering guide agent system prompts best practices 2025 2026" + +### Key Sources +1. [Anthropic: Effective Context Engineering for AI Agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) +2. [Anthropic: Effective Harnesses for Long-Running Agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents) +3. [OpenAI: GPT-4.1 Prompting Guide](https://cookbook.openai.com/examples/gpt4-1_prompting_guide) +4. [OpenAI: Prompt Engineering Guide](https://platform.openai.com/docs/guides/prompt-engineering) + +### Key Takeaways + +1. **"Right altitude" of specificity** (Anthropic). Prompts should be + specific enough to direct behavior but flexible enough for the model + to apply heuristics. Avoid two extremes: hardcoded if-else logic + (brittle) and vague high-level guidance (underdetermined). Our library + is generally at the right altitude — most prompts provide specific + commands and paths while leaving room for judgment. + +2. **Structured note-taking for long-horizon agents** (Anthropic). Agents + should maintain persistent memory files (notes, to-do lists) outside + the context window. Our prompts already follow this pattern well — + most require `CONTEXT.md`, `TODO.md`, `DECISIONS.md`, `TEST_LOG.md` + as persistent state files per `AGENTS.md` Section 15. + +3. **Tool descriptions should be self-contained and non-overlapping** + (Anthropic). When agents must choose between tools/commands, the + options should be clearly distinct. Our prompts generally do this + well by specifying exact commands to run. + +4. **Minimal high-signal context** (Anthropic). "Find the smallest + possible set of high-signal tokens that maximize the likelihood of + some desired outcome." Some of our prompts are verbose — this + principle supports trimming redundant instructions. However, per our + own hard constraints, "shorter is better — but not at the cost of + clarity, safety, or specificity." + +5. **Incremental work, clean state** (Anthropic, long-running agents). + Agents should work on one feature at a time, commit progress with + descriptive messages, and leave the environment in a clean state. + Our prompts consistently enforce this with "one file per PR" and + "one issue per PR" patterns. + +6. **Build evals to measure prompt performance** (OpenAI). This is a gap + in our library — we have no systematic way to evaluate whether a + prompt revision improved agent behavior. Future work could establish + simple before/after metrics for prompt changes. + +7. **Pin to specific behavior expectations** (OpenAI). Production prompts + should have predictable, testable outcomes. Our verification sections + serve this purpose (exact commands + expected results). + +8. **Examples over exhaustive rules** (Anthropic). "For an LLM, examples + are the 'pictures' worth a thousand words." Several of our prompts + include good examples (e.g., code patterns in innerhtml-migration, + const-refactor). Some prompts could benefit from adding 1-2 concrete + examples of expected behavior. + +### Library Impact + +**Changes made this run based on research**: None. The research validated +that our library's existing patterns (structured sections, persistent +state files, specific commands, incremental work) align well with current +best practices. No findings warranted immediate changes. + +**Potential future improvements** (not applied — need evidence of benefit): +- Add concrete input/output examples to prompts that currently lack them + (e.g., scheduler-update-agent, changelog-agent). +- Consider establishing lightweight prompt evaluation criteria (e.g., + "did the agent produce a correct PR on first run?") for future + prompt revisions. +- Some prompts could be shortened by removing redundant restatements + of constraints that are already in `AGENTS.md`. However, the + redundancy may serve as a useful safety net for agents that don't + always read policy docs carefully — defer this decision to a future + run with evidence. diff --git a/docs/agents/STYLE_GUIDE.md b/docs/agents/STYLE_GUIDE.md new file mode 100644 index 00000000..44b6f37d --- /dev/null +++ b/docs/agents/STYLE_GUIDE.md @@ -0,0 +1,173 @@ +# Agent Prompt Style Guide + +This guide defines the canonical structure, naming conventions, and quality +standards for agent prompts in `/docs/agents/prompts/`. + +Authority: This guide is subordinate to `AGENTS.md` and `CLAUDE.md`. If this +guide conflicts with either, follow the higher-level document. + +--- + +## Canonical Prompt Template + +Every agent prompt should include these sections in order: + +### 1. Identity & Mission +- First line: `You are: **bitvid-**, a working inside the \`PR0M3TH3AN/bitvid\` repo.` +- One-sentence mission statement. +- The agent name in the identity line **must match** the filename (minus + `.md` extension). + +### 2. Authority Hierarchy +- Numbered list, highest authority first: + 1. `AGENTS.md` + 2. `CLAUDE.md` + 3. Domain-specific docs (varies by agent) + 4. This agent prompt (always lowest) + +### 3. Scope +- **In scope**: explicit list of files, directories, and systems. +- **Out of scope**: explicit boundaries to prevent mission drift. + +### 4. Goals & Success Criteria +- Numbered, measurable outcomes. +- Avoid vague criteria like "improve quality" — specify what "good" looks like. + +### 5. Hard Constraints +- Non-negotiable rules in imperative voice. +- Must include: "Inspect first", "Minimal changes", "No mission drift". +- Security-sensitive guardrails per `AGENTS.md`. + +### 6. Workflow +- Ordered steps with specific commands, file paths, and checks. +- Standard flow: preflight -> inspect -> plan -> implement -> verify -> document -> PR. +- Each step should name exact commands to run. + +### 7. Verification +- Exact commands to run (e.g., `npm run lint`, `npm run test:unit`). +- Expected outputs or pass criteria. +- Artifacts to capture. + +### 8. Failure Modes +- What to do when blocked, uncertain, or facing risky changes. +- Default: stop, log, open issue. +- Specific triggers for opening issues vs. attempting fixes. + +### 9. PR & Commit Conventions +- Defer to `AGENTS.md` / `CLAUDE.md` for branch naming and conventions. +- Provide commit message examples with appropriate prefixes. +- PR body requirements (what must be included). + +### 10. Outputs +- What the agent produces per run (PRs, docs, issues, reports). +- Distinguish between daily and weekly run outputs if applicable. + +Specialized sections (search patterns, metrics, taxonomies, etc.) may be +added **after** section 10. + +--- + +## Naming Conventions + +### Agent Identity +- Format: `bitvid--agent` +- The identity in the prompt **must match** the filename. +- Example: file `bitvid-style-agent.md` -> identity `bitvid-style-agent` + +### Filenames +- Pattern: `bitvid--agent.md` +- Lowercase, hyphen-separated. +- No abbreviations unless well-established in the repo. + +### Branch Names +- Follow `AGENTS.md` / `CLAUDE.md` conventions. +- Do not invent separate naming schemes for prompt work. + +### Commit Messages +- Use conventional-commit-style prefixes: + - `docs(agents): ` for prompt changes + - `fix(ai): ` for agent-discovered fixes + - `chore(ai): ` for mechanical changes +- Do not use emoji in commit messages. + +### PR Titles +- Follow repo conventions. Do not use emoji. +- Keep under 70 characters. + +--- + +## Voice & Tone + +- **Imperative** for constraints and instructions: "Do not modify crypto logic." +- **Declarative** for descriptions: "This agent audits test coverage." +- **Concise** — cut filler words. Prefer active voice. +- **Specific** — name files, commands, and paths. Avoid "the relevant file" + when you can say `js/constants.js`. + +--- + +## Common Patterns to Follow + +### Good: Specific Commands +``` +Run: `npm run test:unit` +``` + +### Bad: Vague Instructions +``` +Run the tests. +``` + +### Good: Explicit Failure Mode +``` +If lint fails with non-autofixable errors, stop and open an issue +including the exact failing command and error output. +``` + +### Bad: Implicit Failure Mode +``` +Handle errors appropriately. +``` + +### Good: Verified Path Reference +``` +Check `js/nostrEventSchemas.js` for existing builder functions. +``` + +### Bad: Assumed Path Reference +``` +Check the event schemas file. +``` + +--- + +## Anti-Patterns to Avoid + +1. **AI generation artifacts** — Remove `:contentReference`, `oaicite`, + or similar metadata left by AI tools. +2. **Emoji in PR titles/commit messages** — `CLAUDE.md` says to avoid emoji + unless explicitly requested. +3. **`require()` syntax** — The project uses ES modules (`"type": "module"`). + Always use `import`/`export` in examples. +4. **Referencing nonexistent scripts** — Verify a file exists before + referencing it. If suggesting a script to create, say so explicitly. +5. **Identity/filename mismatch** — The agent name in the first line must + match the filename. +6. **Truncated files** — Every prompt must have complete sections. If a + section is missing, the prompt is incomplete. + +--- + +## Section Heading Style + +Use either Unicode box-drawing separators or markdown `---` consistently +within a single prompt. Prefer the style already established in the file +being edited. + +Current conventions observed in the library: +- Daily prompts: mix of `═══` and `───` +- Weekly prompts: mostly `───` +- Schedulers: `───` + +Standardization of heading style is P3 priority — functional correctness +and policy alignment take precedence.