diff --git a/.agents/evals/implement/write-docs/eval.yaml b/.agents/evals/implement/write-docs/eval.yaml new file mode 100644 index 0000000..b044078 --- /dev/null +++ b/.agents/evals/implement/write-docs/eval.yaml @@ -0,0 +1,61 @@ +skill_name: write-docs + +trigger_tests: + - id: explicit-trigger-document-module + name: "Explicit trigger — document a script" + query: "Write documentation for the install.sh script" + should_trigger: true + + - id: explicit-trigger-create-docs + name: "Explicit trigger — create docs for a feature" + query: "Create docs for this feature" + should_trigger: true + + - id: implicit-trigger-readme-update + name: "Implicit trigger — outdated README section, no trigger phrase" + query: "We need to update the README section for the auth module, the current one is outdated" + should_trigger: true + + - id: negative-trigger-prd + name: "Negative — PRD request should route to to-prd" + query: "Write a PRD for the new logging feature" + should_trigger: false + + - id: negative-trigger-write-skill + name: "Negative — skill authoring request should route to write-skill" + query: "Write a skill for generating documentation automatically" + should_trigger: false + + - id: negative-trigger-skill-file + name: "Negative — SKILL.md update (skill files are self-describing)" + query: "Document how the write-docs skill works by updating its SKILL.md" + should_trigger: false + +output_tests: + - id: output-proposes-files-before-reading + name: "Deterministic — candidates proposed or approval sought before reading files" + type: deterministic + prompt: "Write documentation for the config module" + expected_output: "Skill proposes candidate files or asks the user to name specific files before reading any file content" + assertions: + - "Response proposes candidate file paths or asks the user to confirm which files to read before showing any extracted content" + - "Response does not display extracted code content or API surface without first receiving file approval" + + - id: output-gap-check-present + name: "Deterministic — gap check step present before drafting" + type: deterministic + prompt: "Write documentation for the install.sh script, audience: developer" + expected_output: "Skill presents extracted behaviour to the user and asks them to fill gaps before drafting any section" + assertions: + - "Response includes a gap check step that presents extracted behaviour and asks what the code does not explain" + - "Response does not skip directly to a drafted documentation section without presenting extracted content first" + + - id: output-never-invents-behaviour + name: "LLM rubric — no invented behaviour, all claims sourced" + type: llm-rubric + prompt: "Document the src/config.py file for internal developers" + expected_output: "Documentation where every claim is attributed to code content or explicit user input, with no invented explanations, assumptions about intent, or unverifiable behaviour claims." + assertions: + - "The skill explicitly derives each documented claim from a named source — a code line, spec section, or user statement — and does not add claims without attribution" + - "The skill does not include descriptions of caller intent, design rationale, or future behaviour that are not present in the source material" + - "If a behaviour is undocumentable (internal detail with no public spec), the skill notes it as out-of-scope rather than inventing an explanation" diff --git a/.agents/skills/write-a-skill/SKILL.md b/.agents/skills/write-a-skill/SKILL.md deleted file mode 100644 index 227f87c..0000000 --- a/.agents/skills/write-a-skill/SKILL.md +++ /dev/null @@ -1,120 +0,0 @@ ---- -name: write-a-skill -description: Create new agent skills with proper structure, progressive disclosure, and bundled resources. Use when user wants to create, write, or build a new skill. -metadata: - category: factory - version: "1.1" ---- - -# Writing Skills - -## Process - -1. **Gather requirements** - ask user about: - - What task/domain does the skill cover? - - What specific use cases should it handle? - - Does it need executable scripts or just instructions? - - Any reference materials to include? - -2. **Draft the skill** - create: - - SKILL.md with concise instructions - - Additional reference files if content exceeds 500 lines - - Utility scripts if deterministic operations needed - -3. **Review with user** - present draft and ask: - - Does this cover your use cases? - - Anything missing or unclear? - - Should any section be more/less detailed? - -## Skill Structure - -``` -skill-name/ -├── SKILL.md # Main instructions (required) -├── REFERENCE.md # Detailed docs (if needed) -├── EXAMPLES.md # Usage examples (if needed) -└── scripts/ # Utility scripts (if needed) - └── helper.js -``` - -## SKILL.md Template - -```md ---- -name: skill-name -description: Brief description of capability. Use when [specific triggers]. ---- - -# Skill Name - -## Quick start - -[Minimal working example] - -## Workflows - -[Step-by-step processes with checklists for complex tasks] - -## Advanced features - -[Link to separate files: See [REFERENCE.md](REFERENCE.md)] -``` - -## Description Requirements - -The description is **the only thing your agent sees** when deciding which skill to load. It's surfaced in the system prompt alongside all other installed skills. Your agent reads these descriptions and picks the relevant skill based on the user's request. - -**Goal**: Give your agent just enough info to know: - -1. What capability this skill provides -2. When/why to trigger it (specific keywords, contexts, file types) - -**Format**: - -- Max 1024 chars -- Write in third person -- First sentence: what it does -- Second sentence: "Use when [specific triggers]" - -**Good example**: - -``` -Extract text and tables from PDF files, fill forms, merge documents. Use when working with PDF files or when user mentions PDFs, forms, or document extraction. -``` - -**Bad example**: - -``` -Helps with documents. -``` - -The bad example gives your agent no way to distinguish this from other document skills. - -## When to Add Scripts - -Add utility scripts when: - -- Operation is deterministic (validation, formatting) -- Same code would be generated repeatedly -- Errors need explicit handling - -Scripts save tokens and improve reliability vs generated code. - -## When to Split Files - -Split into separate files when: - -- SKILL.md exceeds 100 lines -- Content has distinct domains (finance vs sales schemas) -- Advanced features are rarely needed - -## Review Checklist - -After drafting, verify: - -- [ ] Description includes triggers ("Use when...") -- [ ] SKILL.md under 100 lines -- [ ] No time-sensitive info -- [ ] Consistent terminology -- [ ] Concrete examples included -- [ ] References one level deep diff --git a/.agents/skills/write-docs/SKILL.md b/.agents/skills/write-docs/SKILL.md new file mode 100644 index 0000000..0d2da2d --- /dev/null +++ b/.agents/skills/write-docs/SKILL.md @@ -0,0 +1,103 @@ +--- +name: write-docs +description: Write documentation for X, document this module, create docs for this feature. Use when the user wants to produce or update technical documentation derived from code, spec, or existing artifacts. Do NOT use when the user wants a PRD, ADR, decision doc, or skill file — those have dedicated skills. +version: "1.0" +updated: 2026-05-17 +when: invoked by explicit trigger ("write docs for X", "document this module", "create docs for this feature") or implicit request to produce technical documentation from code or spec +metadata: + category: implement +source: + - repo: anthropics/skills + commit: f458cee31a7577a47ba0c9a101976fa599385174 + files: + - skills/doc-coauthoring/SKILL.md # Reader Testing stage, surgical-edit constraint, gap-check step + updated: 2026-05-17 + - repo: mattpocock/skills + commit: e74f0061bb67222181640effa98c675bdb2fdaa7 + files: + - skills/productivity/write-a-skill/SKILL.md # trigger pattern, review checklist items + updated: 2026-05-17 + - repo: bmad-code-org/BMAD-METHOD + commit: 71136bc6af77cbf507d3768494311d5b6ca95cc5 + files: + - src/core-skills/bmad-advanced-elicitation/SKILL.md # confirmation gate before applying changes + updated: 2026-05-17 +--- + +## Role + +You are a technical writer that produces documentation by reading code and spec — you derive every claim from a source file or explicit user input and never invent behaviour. + +## When to use / When not to use + +**Use when:** +- User wants to document a module, class, function, feature, CLI flag, API endpoint, config file, or README section +- User says "write docs for X", "document this", "create docs for this feature", "write a README for this" + +**Do not use when:** +- User wants a PRD, decision doc, or architecture proposal → `to-prd` or `grill-me` +- User wants to document a skill file (skill files are self-describing) +- User wants marketing or blog copy +- Documentation requires tacit organisational knowledge that cannot be read from code or spec + +## Required inputs + +- Specific file(s) or module(s) to document, or enough description to propose candidates +- Target audience: developer / user / contributor / internal +- Documentation type: reference, guide, README section, inline comment, changelog entry + +## Constraints + +- Every claim must be traceable to a source file line, spec section, or explicit user statement — never invent behaviour +- User must approve specific files before the skill reads them; skill may propose candidates but waits for approval +- Stage skipping is allowed only with an explicit user request and a one-sentence logged reason +- Show the full revised section before each confirmation gate — never gate on output the user has not seen +- Never reprint the whole document; all edits are surgical +- Produce a one-line delta summary after each refinement round +- Reader Testing sub-agent receives only the finished doc and the question list — no source files +- Write summary and overview sections last, after all detail sections are stable + +## Process + +1. **Identify scope.** User names specific files or sections. If not provided, propose candidates based on the description — wait for explicit approval before reading. + +2. **Read and extract.** Read approved files. Extract: public API surface, described behaviour, visible constraints, non-obvious invariants. Note what the code does NOT explain (caller intent, error handling rationale, non-obvious side effects). + +3. **Gap check.** Present extracted behaviour to the user. Ask them to fill only the gaps — what the code does not explain. Log any explicitly deferred gaps. If the user requests to skip this step, log the reason and proceed. + +4. **Draft section by section.** For each section: state the proposed content and its source (code line / spec section / user input). Show; confirm before moving to the next section. + +5. **Confirmation gate.** Before finalising any section, show the full revised section. Wait for explicit confirmation or correction — never apply changes the user has not seen. + +6. **Delta summary.** After each round of revisions: "Round N: changed [sections], added [X], removed [Y]." + +7. **Reader Testing.** Predict 5–10 questions a target reader would ask. Spawn a scoped sub-agent that receives only the finished doc and the questions — no source files. Report its answers. If any answers fail, loop back to step 4. + +8. **Finalise.** Write summary and overview sections last. Prompt the user to review the complete document before committing. + +## Output format + +- Markdown artifact with section headers; produced one section at a time — never as a single large dump +- Delta summary after each refinement round: "Round N: [what changed]" +- Reader Testing report: numbered question list with sub-agent answers +- Final doc at the user-specified or conventionally appropriate path + +## Failure handling + +- Files not named and description too vague to propose candidates → ask for specific names before reading +- Stage skipped without a logged reason → flag and require the one-sentence log before continuing +- Code behaviour is undocumentable (internal implementation detail, no public spec) → note as out-of-scope in the doc; do not invent an explanation +- Reader Testing sub-agent fails on multiple questions → surface the failures, return to step 4; do not mark complete +- Requested output is a PRD, decision doc, or architecture proposal → redirect to `to-prd`, `grill-me`, or `grill-with-docs` + +## Self-check + +- [ ] All claims traceable to a source file or explicit user input +- [ ] No invented behaviour — unverifiable claims removed +- [ ] User approved specific files before reading +- [ ] Any stage skips logged with reason +- [ ] Full revised section shown before each confirmation gate +- [ ] Delta summary produced after each refinement round +- [ ] Reader Testing completed with scoped sub-agent (doc + questions only) +- [ ] Summary/overview written last +- [ ] User prompted to review before committing diff --git a/LESSONS.md b/LESSONS.md index 6e78f6a..25a5ee0 100644 --- a/LESSONS.md +++ b/LESSONS.md @@ -46,6 +46,14 @@ When a skill invokes another skill as part of its work (e.g. write-skill invokin When searching GitHub for agent skill upstreams, AGPL-3.0 repos (e.g. dceoy/speckit-agent-skills) appear alongside permissive-licensed ones without obvious visual distinction. AGPL imposes copyleft obligations on adopted content. Always run a licence check (GitHub API `/license` endpoint) before extracting any content from a new upstream. An AGPL finding is a hard exclude — record the repo, SHA, and licence in source review notes so future sessions don't re-review it. +## 2026-05-17 — Trigger description gate is not satisfied by embedding it in the section walk-through + +The per-skill workflow (and write-skill's own process step 4) requires testing the trigger description against 3 cases — explicit, implicit, negative — as a standalone gate with explicit PASS/FAIL markers before any body content is written. During write-docs (issue 0018 phase 2), the trigger description was included in the section walk-through (step 5b) rather than tested first as a named gate. The gate never had explicit pass/fail output, which means neither the human nor the agent confirmed the trigger was sound before section content was written. Fix: treat the trigger test as a numbered standalone step with per-case PASS/FAIL output before step 5b begins. A section walk-through that happens to include the description field is not a substitute. + +## 2026-05-17 — write-eval confirmation gate is bypassed when called via sub-agent with pre-designed cases + +write-eval's process requires presenting the full test plan and waiting for user confirmation before writing the file. When write-eval is invoked by passing pre-designed test cases directly to a write sub-agent, this gate is skipped — the file is written before the user sees the plan. This happened during write-docs (issue 0018 phase 2). Fix: when orchestrating write-eval as part of a larger workflow, split into two steps: (1) sub-agent proposes test cases and returns to the main conversation; (2) after user confirmation, sub-agent writes the file. Or: design cases in the main conversation, present them to the user, then spawn the write agent. The plan-then-write separation is the gate — collapsing it into a single sub-agent call silently removes it. + ## 2026-05-17 — HITL gap: agent delegates confirmation to permission system The agent-level HITL rule ("require explicit confirmation before irreversible shared-state operations") is being bypassed: the agent calls the tool and lets the permission dialog catch it. This means the rule is not firing in agent reasoning — it's the permission system acting as a safety net. If a user selects "don't ask again," the net disappears. Fix: the HITL rule needs to be framed as "do not call the tool" rather than "ask before proceeding" — the agent must ask first, then act only after explicit confirmation. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index f2d14d4..a6173ca 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -98,4 +98,4 @@ Items consciously not resolved — to be addressed in the relevant chunk PRD or - **Chunk 2 behavioral tests** — run and fully resolved 2026-05-17. 7/8 pass; scenario 4 (push confirmation) inconclusive — no remote in test environment, rule tightened but unverified. All fixable failures addressed: rule specificity in `providers/claude-code/CLAUDE.md`; context-loading guarantee via `@import CONTEXT.md` in repo CLAUDE.md; standing rule in CONTEXT.md to check `docs/adr/` and ROADMAP resolved entries before answering design questions. Chunk 2 ✅ complete. - **Governance Phase 1 behavioral tests** — run 2026-05-17. 3/4 testable scenarios pass. Secrets rule gap fixed (2026-05-17): extended to cover credential reproduction in response text and examples, with placeholder requirement added to `core/instructions/governance.md`. HITL scenario not testable in this environment (Nginx not installed); HITL gap evidenced by instructions test scenario 4 — push confirmation rule fix addresses the same root cause. Governance Phase 1 ✅ complete. - **AI ethics/security workstream** — `docs/notes/ai-ethics-security-principles.md` exploration note is superseded. Governance Phase 1 (`core/instructions/governance.md`) covers all planned scope: credentials, data classification, HITL, scope discipline, agent autonomy, transparency, and security code review. Tier-placement architectural question resolved by the `@import` always-on model. No separate workstream needed. -- **Chunk 3 grill complete** — 2026-05-17. PRD at `docs/prd/chunk-3-skills-library.md`. Key decisions: 42-skill target library, AGENTS.md refactor as prerequisite issue (both CLAUDE.md files become thin adapters), git-cliff for changelog, provider-agnostic issue tracker abstraction, grill-me/grill-lean design phase split, factory bootstrap order (write-eval → write-skill → write-docs phase 2 → write-adr → remaining factory → design → parallel category groups). ADRs written: 0011 (provider-agnostic issue tracker), 0012 (AGENTS.md governance entry point, partially supersedes ADR-0005). Upstream review cadence: per-skill + quarterly post-roadmap (per-chunk-start changed to per-skill by issue 0016 grill). **Issues created 0015–0028** — all HITL; ~~0015 (AGENTS.md refactor, prerequisite)~~ ✅, ~~0016 (skill workflow grill, produces conventions for 0017–0028)~~ ✅, ~~0017 (bootstrap skill: write-eval)~~ ⏳ HITL pending, ~~0018 phase 1 (write-skill)~~ ⏳ HITL pending, 0018 phase 2 (write-docs — first factory-authored skill), 0019 (remaining factory skills), 0020–0027 (design/implement/test/review/deploy/operate/iac/cross-cutting), 0028 (chunk closure). ~~Acceptance criteria for 0017–0028 to be refined after 0016 grill session.~~ ✅ Refined 2026-05-17 — see `docs/notes/skill-implementation-workflow.md`. +- **Chunk 3 grill complete** — 2026-05-17. PRD at `docs/prd/chunk-3-skills-library.md`. Key decisions: 42-skill target library, AGENTS.md refactor as prerequisite issue (both CLAUDE.md files become thin adapters), git-cliff for changelog, provider-agnostic issue tracker abstraction, grill-me/grill-lean design phase split, factory bootstrap order (write-eval → write-skill → write-docs phase 2 → write-adr → remaining factory → design → parallel category groups). ADRs written: 0011 (provider-agnostic issue tracker), 0012 (AGENTS.md governance entry point, partially supersedes ADR-0005). Upstream review cadence: per-skill + quarterly post-roadmap (per-chunk-start changed to per-skill by issue 0016 grill). **Issues created 0015–0028** — all HITL; ~~0015 (AGENTS.md refactor, prerequisite)~~ ✅, ~~0016 (skill workflow grill, produces conventions for 0017–0028)~~ ✅, ~~0017 (bootstrap skill: write-eval)~~ ⏳ HITL pending, ~~0018 phase 1 (write-skill)~~ ⏳ HITL pending, ~~0018 phase 2 (write-docs — first factory-authored skill)~~ ⏳ HITL pending, 0018 phase 3 (doc convention — grill first), 0019 (remaining factory skills), 0020–0027 (design/implement/test/review/deploy/operate/iac/cross-cutting), 0028 (chunk closure). ~~Acceptance criteria for 0017–0028 to be refined after 0016 grill session.~~ ✅ Refined 2026-05-17 — see `docs/notes/skill-implementation-workflow.md`. diff --git a/docs/issues/0018-factory-write-skill.md b/docs/issues/0018-factory-write-skill.md index 7fa756d..ccd1379 100644 --- a/docs/issues/0018-factory-write-skill.md +++ b/docs/issues/0018-factory-write-skill.md @@ -31,6 +31,26 @@ Build `write-docs` — the first skill authored via `write-skill` itself (the fa - Follow full per-skill workflow from `docs/notes/skill-implementation-workflow.md` (sub-agents for discovery, review, conflict check) - Derives from code and spec; never invents behaviour +### Phase 3: Documentation convention + +Define the canonical documentation convention for this repo — the missing input that `write-docs` currently defers to "user-specified or conventionally appropriate path." Without this, every `write-docs` invocation requires the user to re-decide where output goes. + +**Opening action:** `/grill-me` session to resolve the convention before writing anything. + +**Questions the grill must resolve:** +- What documentation types exist in this repo? (reference, guide, README section, inline comment, changelog entry, etc.) +- Where does each type live? (file paths, directory structure — e.g. does `docs/` own all prose, or do modules carry their own READMEs?) +- Global defaults vs. repo-specific overrides — what layer does the convention live at? +- What format standards apply per type? (required headers, prose vs structured, max length) +- Does `write-docs` need to be updated after the convention is defined, or does it reference it at runtime? + +**Expected outputs:** +- `docs/notes/doc-convention.md` — the convention document (file/folder/content structure, per-type rules, override model) +- Update to `write-docs` SKILL.md output format section — reference the convention instead of deferring to "conventionally appropriate path" +- Update to `CONTEXT.md` if the convention becomes a standing repo-level principle + +**No new SKILL.md for this phase** — this is a convention document, not a skill. If `write-docs` needs substantial changes after the grill, use `upgrade-skill`. + ## Implementation notes Follow the per-skill workflow defined in `docs/notes/skill-implementation-workflow.md` (produced by issue 0016). @@ -48,14 +68,18 @@ Follow the per-skill workflow defined in `docs/notes/skill-implementation-workfl - [x] `install.sh` deploys `write-skill` to `~/.agents/skills/` - [ ] **HITL:** human runs behavioral test: invoke "write a new skill for X" and verify the produced SKILL.md meets the authoring standard - [ ] **HITL:** human reviews SKILL.md and eval before committing -- [ ] Per-skill process followed for both phases (see `docs/notes/skill-implementation-workflow.md`) -- [ ] Trigger description for each skill tested against explicit, implicit, and negative queries before body written -- [ ] `when:` frontmatter field present in both SKILL.md files -- [ ] `source:` and `references:` fields correctly populated or absent -- [ ] eval.yaml for each skill contains all 5 required test types -- [ ] Body ≤500 lines for each skill -- [ ] Phase 2 (`write-docs`) is the first skill produced end-to-end by the factory -- [ ] `docs/spec/overview.md` updated to reflect both skills deployed +- [x] Per-skill process followed for both phases (see `docs/notes/skill-implementation-workflow.md`) +- [x] Trigger description for each skill tested against explicit, implicit, and negative queries before body written +- [x] `when:` frontmatter field present in both SKILL.md files +- [x] `source:` and `references:` fields correctly populated or absent +- [x] eval.yaml for each skill contains all 5 required test types +- [x] Body ≤500 lines for each skill +- [x] Phase 2 (`write-docs`) is the first skill produced end-to-end by the factory +- [x] `docs/spec/overview.md` updated to reflect both skills deployed +- [ ] **Phase 3:** `/grill-me` session completed; grill output committed +- [ ] **Phase 3:** `docs/notes/doc-convention.md` written and committed +- [ ] **Phase 3:** `write-docs` SKILL.md output format updated to reference the convention (via `upgrade-skill` if substantive) +- [ ] **Phase 3:** `CONTEXT.md` updated if convention becomes a standing principle ## Blocked by @@ -79,9 +103,34 @@ Follow the per-skill workflow defined in `docs/notes/skill-implementation-workfl - Role is self-contained (no reference to workflow doc) so it can be used standalone after chunk 3. **Open threads:** -- HITL behavioral test: open a fresh session, invoke "write a new skill for X" in this repo context, verify trigger is tested before body, per-section walk-through happens, write-eval is invoked, HITL prompt appears. -- Phase 2 (`write-docs`): first skill authored end-to-end by the factory. Follow full per-skill workflow from `docs/notes/skill-implementation-workflow.md`. +- HITL behavioral test for write-skill: open a fresh session, invoke "write a new skill for X" in this repo context, verify trigger is tested before body, per-section walk-through happens, write-eval is invoked, HITL prompt appears. +- Phase 2 HITL behavioral test: open a fresh session, invoke "write docs for X" or "document this module", verify file-approval gate fires before any reading, gap check step appears, full section shown before confirmation gate, Reader Testing step present. **Next session start:** -- Load: `CONTEXT.md`, `docs/notes/skill-implementation-workflow.md`, `docs/issues/0018-factory-write-skill.md` -- First action: HITL behavioral test for write-skill, then begin phase 2 (`write-docs`) using write-skill +- Load: `CONTEXT.md`, `docs/notes/skill-implementation-workflow.md`, `docs/issues/0019-factory-skills-remaining.md` +- First action: HITL behavioral tests for write-skill (phase 1) and write-docs (phase 2) if not yet done, then begin issue 0019 — start with `write-adr` (must be verified before design skills issue 0020 begins) + +--- + +## Handoff — Phase 2 + +**Status:** complete — pending HITL behavioral test + +**Files produced:** +- `.agents/skills/write-docs/SKILL.md` +- `.agents/evals/implement/write-docs/eval.yaml` + +**Key decisions:** +- File-approval gate before reading: user names specific files, or skill proposes candidates and waits for approval — enforces governance scope discipline. +- Gap check before drafting: presents extracted behaviour, asks user to fill only what code doesn't explain — prevents invented content. +- Stage skipping: allowed with explicit user request + one-sentence logged reason (hybrid per synthesis grill decision). +- Confirmation gate: shows full revised section before gate fires, not just the diff (per synthesis grill decision). +- Surgical edits only + per-round delta summary (no hard iteration cap, delta summary keeps cumulative change reviewable). +- Reader Testing: scoped sub-agent receives only finished doc + questions — no source files (minimum data exposure per governance conflict 1). +- Sources adopted: anthropics/skills doc-coauthoring (Reader Testing stage, surgical-edit constraint, gap-check), mattpocock/skills write-a-skill (trigger pattern, checklist items), bmad-code-org/BMAD-METHOD bmad-advanced-elicitation (confirmation gate). bmad infrastructure (CSV registry, party mode) explicitly excluded. +- Rejected mattpocock 100-line limit — project convention (500 lines) takes precedence; noted in inline source comment. +- Prompts-as-code governance obligation satisfied: SKILL.md committed to repo; version control is the enforcement mechanism. + +**Open threads:** +- Documentation convention: scoped to Phase 3 of this issue — see "What to build" above. `write-docs` output format section will be updated once the convention is defined. +- HITL behavioral test: see above. diff --git a/docs/notes/factory-integration-decisions.md b/docs/notes/factory-integration-decisions.md index 7908fd3..3d356bc 100644 --- a/docs/notes/factory-integration-decisions.md +++ b/docs/notes/factory-integration-decisions.md @@ -42,7 +42,7 @@ Categories are expressed via `metadata: category:` in SKILL.md frontmatter. Defi | `iac` | write-ansible-role, write-terraform-module, write-k8s-manifest, write-docker-compose, proxmox-vm-spec, iac-security-review, write-molecule-test — **global optional** | | `gitea` | setup-gitea-mcp, post-pr-review, create-issue — **global optional** | | `cross-cutting` | zoom-out, caveman, session-handoff, governance-check, git-guardrails, git-commit-message | -| `factory` | write-a-skill, write-adr, write-workflow, write-eval, validate-skill, upgrade-skill, write-issue-spec | +| `factory` | write-skill, write-adr, write-workflow, write-eval, validate-skill, upgrade-skill, write-issue-spec | | `roles` | architect, developer, reviewer, security, qa, ops — **Chunk 5** | ### 3. SKILL.md authoring standard (full) diff --git a/docs/notes/skill-implementation-workflow.md b/docs/notes/skill-implementation-workflow.md index 60c2a0f..29128d4 100644 --- a/docs/notes/skill-implementation-workflow.md +++ b/docs/notes/skill-implementation-workflow.md @@ -90,7 +90,7 @@ Write the `description:` frontmatter field first. Test it against three cases be 2. Implicit invocation — user describes the task without the trigger phrase 3. Negative case — adjacent task that must NOT activate this skill -Do not proceed to the body until all three pass. +For each case, output an explicit **PASS** or **FAIL** result. Do not proceed to step b until all three show PASS. Including the description inside the section walk-through (step b) does not satisfy this gate — it must be a standalone test-then-proceed step with per-case verdicts. If any case fails, revise the description and re-test before continuing. **b. Per-section options walk-through** Before writing anything, walk through each body section with the human. For each section: @@ -109,7 +109,11 @@ Populate after upstream review. Two distinct fields: - `references:` — general citations (research papers, documentation, standard specifications). Present only if the skill cites external research. **d. eval.yaml** (sub-agent) -Spawn an agent to produce the eval via `write-eval`. Co-located at `.agents/evals///eval.yaml`. Must contain all five required test types (see Eval schema below). +Invoke `write-eval` in two steps to preserve its confirmation gate: +1. Sub-agent proposes test cases and returns the plan to the main conversation. +2. Human confirms the plan; then sub-agent writes the file. + +Do not pass pre-designed test cases directly to a write agent — that collapses the plan-then-confirm gate into a single step, bypassing write-eval's own constraint. Co-located at `.agents/evals///eval.yaml`. Must contain all five required test types (see Eval schema below). **e. HITL behavioral test** Human opens a fresh Claude session, invokes the skill with its trigger phrase, and verifies output. Do not batch more than 2–3 skills before running behavioral tests — output volume must stay within genuine human review capacity. An approval that cannot be meaningfully evaluated is not an approval. diff --git a/docs/spec/overview.md b/docs/spec/overview.md index 43819d9..f9cb959 100644 --- a/docs/spec/overview.md +++ b/docs/spec/overview.md @@ -2,18 +2,19 @@ Current deployed state of this repo — what you get if you run `install.sh` today. Updated at the close of each chunk and in the same PR as any behavior change. -*Last updated: 2026-05-17 (issue 0018 phase 1)* +*Last updated: 2026-05-17 (issue 0018 phase 2)* ## What is deployed ### Skills -14 skills deployed to `~/.agents/skills/` via `install.sh`. Available as slash commands in Claude Code via `~/.claude/skills/ → ~/.agents/skills/` symlink. 12 are first-draft placeholders pending rebuild in Chunk 3; 2 are new Chunk 3 factory skills (`write-eval`, `write-skill`). +14 skills deployed to `~/.agents/skills/` via `install.sh`. Available as slash commands in Claude Code via `~/.claude/skills/ → ~/.agents/skills/` symlink. 11 are first-draft placeholders pending rebuild in Chunk 3; 3 are new Chunk 3 skills (`write-eval`, `write-skill`, `write-docs`). **Factory bootstrap (Chunk 3):** - `write-eval` — produces `eval.yaml` test files for skills. Hand-written (bootstrap). Eval at `.agents/evals/factory/write-eval/eval.yaml`. - `write-skill` — authors new SKILL.md files and converts placeholders to canonical format. Hand-written (bootstrap — cannot author itself before it exists). Eval at `.agents/evals/factory/write-skill/eval.yaml`. Invokes `write-eval` as part of its own process. +- `write-docs` — produces technical documentation derived from code and spec; never invents behaviour. **First factory-authored skill** (SKILL.md produced via `write-skill`, eval via `write-eval`). Eval at `.agents/evals/implement/write-docs/eval.yaml`. Sources: anthropics/skills `doc-coauthoring`, mattpocock/skills `write-a-skill`, bmad-code-org/BMAD-METHOD `bmad-advanced-elicitation`. -Current skills: `caveman`, `diagnose`, `grill-me`, `grill-with-docs`, `improve-codebase-architecture`, `prototype`, `tdd`, `to-issues`, `to-prd`, `triage`, `write-a-skill`, `write-eval`, `write-skill`, `zoom-out`. +Current skills: `caveman`, `diagnose`, `grill-me`, `grill-with-docs`, `improve-codebase-architecture`, `prototype`, `tdd`, `to-issues`, `to-prd`, `triage`, `write-docs`, `write-eval`, `write-skill`, `zoom-out`. **Chunk 3 target:** 42 skills across 9 categories. PRD: `docs/prd/chunk-3-skills-library.md`. Canonical build reference: `docs/research/ai-coding-factory/ai-coding-factory-skills-index.md` (delete once all skills exist). Skills stored flat (`skill-name/SKILL.md`) per ADR-0009; category in `metadata.category` frontmatter. Categories: design, factory, implement, test, review, deploy, operate, cross-cutting, iac (2 skills only — docker-compose + iac-security-review). Role skills (6) deferred to Chunk 5. Gitea skills moved to `providers/gitea/` provider adapter. @@ -46,6 +47,7 @@ For chunk planning and open questions, see `docs/ROADMAP.md`. ## Recent changes +- 2026-05-17 — Issue 0018 phase 2 complete: `write-docs` skill written and deployed. First skill produced end-to-end by the factory (SKILL.md via `write-skill`, eval via `write-eval`). Category: implement. Key decisions: file-approval gate before reading (user names files or approves proposals); gap check before drafting (user fills what code doesn't explain); stage skipping allowed with logged reason; full revised section shown before confirmation gate; surgical edits only with per-round delta summary; Reader Testing via scoped sub-agent (doc + questions only, no source files); summary/overview sections written last. Sources: anthropics/skills doc-coauthoring (Reader Testing stage, surgical-edit constraint), mattpocock/skills write-a-skill (trigger pattern), bmad-code-org/BMAD-METHOD bmad-advanced-elicitation (confirmation gate). Open follow-up: documentation convention (file/folder/content structure, global vs repo-specific) — not yet defined. - 2026-05-17 — Issue 0018 phase 1 complete: `write-skill` bootstrap skill written and deployed. Hand-written (factory bootstrap). Self-authored — no upstream content adopted; agentskills.io best-practices and optimizing-descriptions docs cited as references. Speckit excluded (AGPL-3.0). Key decisions: new-skill + placeholder-conversion scope only (upgrades → `upgrade-skill`); trigger description tested against 3 cases before body written; `write-eval` invoked as step 7 in process; HITL prompt as step 8. Eval at `.agents/evals/factory/write-skill/eval.yaml`. - 2026-05-17 — Issue 0017 complete: `write-eval` bootstrap skill written and deployed. Two sections schema (`trigger_tests` + `output_tests`), provider-agnostic string assertions, show-plan-then-merge-on-rerun behaviour, conflict flagging (B model). Sources: agentskills/agentskills, darkrishabh/agent-skills-eval, bmad-code-org/BMAD-METHOD, mattpocock/skills. Hand-written eval at `.agents/evals/factory/write-eval/eval.yaml`. - 2026-05-17 — Issue 0016 complete: skill implementation workflow grill completed. `docs/notes/skill-implementation-workflow.md` written. All issues 0017–0028 updated with specific acceptance criteria. Key conventions: sub-agents prescribed at each research/writing step; conflict check against constitution + factory principles before synthesis grill; `when:` and `references:` fields added to authoring standard; write-docs moved to issue 0018 phase 2 (first factory-authored skill).