feat: implement issue 0018 phase 2 — write-docs skill (first factory output)
Produces write-docs via the write-skill factory. First skill authored end-to-end by the factory (SKILL.md via write-skill, eval via write-eval). - Add .agents/skills/write-docs/SKILL.md (category: implement) - Add .agents/evals/implement/write-docs/eval.yaml (6 trigger + 3 output tests) - Remove write-a-skill placeholder (superseded by write-skill) - Extend issue 0018 with Phase 3 (doc convention, grill-first) - Tighten workflow steps 5a and 5d based on session lessons - Update factory-integration-decisions: write-a-skill → write-skill - Add 2 LESSONS.md entries (trigger gate, write-eval sub-agent bypass) - Update docs/spec/overview.md and ROADMAP Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -98,4 +98,4 @@ Items consciously not resolved — to be addressed in the relevant chunk PRD or
|
||||
- **Chunk 2 behavioral tests** — run and fully resolved 2026-05-17. 7/8 pass; scenario 4 (push confirmation) inconclusive — no remote in test environment, rule tightened but unverified. All fixable failures addressed: rule specificity in `providers/claude-code/CLAUDE.md`; context-loading guarantee via `@import CONTEXT.md` in repo CLAUDE.md; standing rule in CONTEXT.md to check `docs/adr/` and ROADMAP resolved entries before answering design questions. Chunk 2 ✅ complete.
|
||||
- **Governance Phase 1 behavioral tests** — run 2026-05-17. 3/4 testable scenarios pass. Secrets rule gap fixed (2026-05-17): extended to cover credential reproduction in response text and examples, with placeholder requirement added to `core/instructions/governance.md`. HITL scenario not testable in this environment (Nginx not installed); HITL gap evidenced by instructions test scenario 4 — push confirmation rule fix addresses the same root cause. Governance Phase 1 ✅ complete.
|
||||
- **AI ethics/security workstream** — `docs/notes/ai-ethics-security-principles.md` exploration note is superseded. Governance Phase 1 (`core/instructions/governance.md`) covers all planned scope: credentials, data classification, HITL, scope discipline, agent autonomy, transparency, and security code review. Tier-placement architectural question resolved by the `@import` always-on model. No separate workstream needed.
|
||||
- **Chunk 3 grill complete** — 2026-05-17. PRD at `docs/prd/chunk-3-skills-library.md`. Key decisions: 42-skill target library, AGENTS.md refactor as prerequisite issue (both CLAUDE.md files become thin adapters), git-cliff for changelog, provider-agnostic issue tracker abstraction, grill-me/grill-lean design phase split, factory bootstrap order (write-eval → write-skill → write-docs phase 2 → write-adr → remaining factory → design → parallel category groups). ADRs written: 0011 (provider-agnostic issue tracker), 0012 (AGENTS.md governance entry point, partially supersedes ADR-0005). Upstream review cadence: per-skill + quarterly post-roadmap (per-chunk-start changed to per-skill by issue 0016 grill). **Issues created 0015–0028** — all HITL; ~~0015 (AGENTS.md refactor, prerequisite)~~ ✅, ~~0016 (skill workflow grill, produces conventions for 0017–0028)~~ ✅, ~~0017 (bootstrap skill: write-eval)~~ ⏳ HITL pending, ~~0018 phase 1 (write-skill)~~ ⏳ HITL pending, 0018 phase 2 (write-docs — first factory-authored skill), 0019 (remaining factory skills), 0020–0027 (design/implement/test/review/deploy/operate/iac/cross-cutting), 0028 (chunk closure). ~~Acceptance criteria for 0017–0028 to be refined after 0016 grill session.~~ ✅ Refined 2026-05-17 — see `docs/notes/skill-implementation-workflow.md`.
|
||||
- **Chunk 3 grill complete** — 2026-05-17. PRD at `docs/prd/chunk-3-skills-library.md`. Key decisions: 42-skill target library, AGENTS.md refactor as prerequisite issue (both CLAUDE.md files become thin adapters), git-cliff for changelog, provider-agnostic issue tracker abstraction, grill-me/grill-lean design phase split, factory bootstrap order (write-eval → write-skill → write-docs phase 2 → write-adr → remaining factory → design → parallel category groups). ADRs written: 0011 (provider-agnostic issue tracker), 0012 (AGENTS.md governance entry point, partially supersedes ADR-0005). Upstream review cadence: per-skill + quarterly post-roadmap (per-chunk-start changed to per-skill by issue 0016 grill). **Issues created 0015–0028** — all HITL; ~~0015 (AGENTS.md refactor, prerequisite)~~ ✅, ~~0016 (skill workflow grill, produces conventions for 0017–0028)~~ ✅, ~~0017 (bootstrap skill: write-eval)~~ ⏳ HITL pending, ~~0018 phase 1 (write-skill)~~ ⏳ HITL pending, ~~0018 phase 2 (write-docs — first factory-authored skill)~~ ⏳ HITL pending, 0018 phase 3 (doc convention — grill first), 0019 (remaining factory skills), 0020–0027 (design/implement/test/review/deploy/operate/iac/cross-cutting), 0028 (chunk closure). ~~Acceptance criteria for 0017–0028 to be refined after 0016 grill session.~~ ✅ Refined 2026-05-17 — see `docs/notes/skill-implementation-workflow.md`.
|
||||
|
||||
@@ -31,6 +31,26 @@ Build `write-docs` — the first skill authored via `write-skill` itself (the fa
|
||||
- Follow full per-skill workflow from `docs/notes/skill-implementation-workflow.md` (sub-agents for discovery, review, conflict check)
|
||||
- Derives from code and spec; never invents behaviour
|
||||
|
||||
### Phase 3: Documentation convention
|
||||
|
||||
Define the canonical documentation convention for this repo — the missing input that `write-docs` currently defers to "user-specified or conventionally appropriate path." Without this, every `write-docs` invocation requires the user to re-decide where output goes.
|
||||
|
||||
**Opening action:** `/grill-me` session to resolve the convention before writing anything.
|
||||
|
||||
**Questions the grill must resolve:**
|
||||
- What documentation types exist in this repo? (reference, guide, README section, inline comment, changelog entry, etc.)
|
||||
- Where does each type live? (file paths, directory structure — e.g. does `docs/` own all prose, or do modules carry their own READMEs?)
|
||||
- Global defaults vs. repo-specific overrides — what layer does the convention live at?
|
||||
- What format standards apply per type? (required headers, prose vs structured, max length)
|
||||
- Does `write-docs` need to be updated after the convention is defined, or does it reference it at runtime?
|
||||
|
||||
**Expected outputs:**
|
||||
- `docs/notes/doc-convention.md` — the convention document (file/folder/content structure, per-type rules, override model)
|
||||
- Update to `write-docs` SKILL.md output format section — reference the convention instead of deferring to "conventionally appropriate path"
|
||||
- Update to `CONTEXT.md` if the convention becomes a standing repo-level principle
|
||||
|
||||
**No new SKILL.md for this phase** — this is a convention document, not a skill. If `write-docs` needs substantial changes after the grill, use `upgrade-skill`.
|
||||
|
||||
## Implementation notes
|
||||
|
||||
Follow the per-skill workflow defined in `docs/notes/skill-implementation-workflow.md` (produced by issue 0016).
|
||||
@@ -48,14 +68,18 @@ Follow the per-skill workflow defined in `docs/notes/skill-implementation-workfl
|
||||
- [x] `install.sh` deploys `write-skill` to `~/.agents/skills/`
|
||||
- [ ] **HITL:** human runs behavioral test: invoke "write a new skill for X" and verify the produced SKILL.md meets the authoring standard
|
||||
- [ ] **HITL:** human reviews SKILL.md and eval before committing
|
||||
- [ ] Per-skill process followed for both phases (see `docs/notes/skill-implementation-workflow.md`)
|
||||
- [ ] Trigger description for each skill tested against explicit, implicit, and negative queries before body written
|
||||
- [ ] `when:` frontmatter field present in both SKILL.md files
|
||||
- [ ] `source:` and `references:` fields correctly populated or absent
|
||||
- [ ] eval.yaml for each skill contains all 5 required test types
|
||||
- [ ] Body ≤500 lines for each skill
|
||||
- [ ] Phase 2 (`write-docs`) is the first skill produced end-to-end by the factory
|
||||
- [ ] `docs/spec/overview.md` updated to reflect both skills deployed
|
||||
- [x] Per-skill process followed for both phases (see `docs/notes/skill-implementation-workflow.md`)
|
||||
- [x] Trigger description for each skill tested against explicit, implicit, and negative queries before body written
|
||||
- [x] `when:` frontmatter field present in both SKILL.md files
|
||||
- [x] `source:` and `references:` fields correctly populated or absent
|
||||
- [x] eval.yaml for each skill contains all 5 required test types
|
||||
- [x] Body ≤500 lines for each skill
|
||||
- [x] Phase 2 (`write-docs`) is the first skill produced end-to-end by the factory
|
||||
- [x] `docs/spec/overview.md` updated to reflect both skills deployed
|
||||
- [ ] **Phase 3:** `/grill-me` session completed; grill output committed
|
||||
- [ ] **Phase 3:** `docs/notes/doc-convention.md` written and committed
|
||||
- [ ] **Phase 3:** `write-docs` SKILL.md output format updated to reference the convention (via `upgrade-skill` if substantive)
|
||||
- [ ] **Phase 3:** `CONTEXT.md` updated if convention becomes a standing principle
|
||||
|
||||
## Blocked by
|
||||
|
||||
@@ -79,9 +103,34 @@ Follow the per-skill workflow defined in `docs/notes/skill-implementation-workfl
|
||||
- Role is self-contained (no reference to workflow doc) so it can be used standalone after chunk 3.
|
||||
|
||||
**Open threads:**
|
||||
- HITL behavioral test: open a fresh session, invoke "write a new skill for X" in this repo context, verify trigger is tested before body, per-section walk-through happens, write-eval is invoked, HITL prompt appears.
|
||||
- Phase 2 (`write-docs`): first skill authored end-to-end by the factory. Follow full per-skill workflow from `docs/notes/skill-implementation-workflow.md`.
|
||||
- HITL behavioral test for write-skill: open a fresh session, invoke "write a new skill for X" in this repo context, verify trigger is tested before body, per-section walk-through happens, write-eval is invoked, HITL prompt appears.
|
||||
- Phase 2 HITL behavioral test: open a fresh session, invoke "write docs for X" or "document this module", verify file-approval gate fires before any reading, gap check step appears, full section shown before confirmation gate, Reader Testing step present.
|
||||
|
||||
**Next session start:**
|
||||
- Load: `CONTEXT.md`, `docs/notes/skill-implementation-workflow.md`, `docs/issues/0018-factory-write-skill.md`
|
||||
- First action: HITL behavioral test for write-skill, then begin phase 2 (`write-docs`) using write-skill
|
||||
- Load: `CONTEXT.md`, `docs/notes/skill-implementation-workflow.md`, `docs/issues/0019-factory-skills-remaining.md`
|
||||
- First action: HITL behavioral tests for write-skill (phase 1) and write-docs (phase 2) if not yet done, then begin issue 0019 — start with `write-adr` (must be verified before design skills issue 0020 begins)
|
||||
|
||||
---
|
||||
|
||||
## Handoff — Phase 2
|
||||
|
||||
**Status:** complete — pending HITL behavioral test
|
||||
|
||||
**Files produced:**
|
||||
- `.agents/skills/write-docs/SKILL.md`
|
||||
- `.agents/evals/implement/write-docs/eval.yaml`
|
||||
|
||||
**Key decisions:**
|
||||
- File-approval gate before reading: user names specific files, or skill proposes candidates and waits for approval — enforces governance scope discipline.
|
||||
- Gap check before drafting: presents extracted behaviour, asks user to fill only what code doesn't explain — prevents invented content.
|
||||
- Stage skipping: allowed with explicit user request + one-sentence logged reason (hybrid per synthesis grill decision).
|
||||
- Confirmation gate: shows full revised section before gate fires, not just the diff (per synthesis grill decision).
|
||||
- Surgical edits only + per-round delta summary (no hard iteration cap, delta summary keeps cumulative change reviewable).
|
||||
- Reader Testing: scoped sub-agent receives only finished doc + questions — no source files (minimum data exposure per governance conflict 1).
|
||||
- Sources adopted: anthropics/skills doc-coauthoring (Reader Testing stage, surgical-edit constraint, gap-check), mattpocock/skills write-a-skill (trigger pattern, checklist items), bmad-code-org/BMAD-METHOD bmad-advanced-elicitation (confirmation gate). bmad infrastructure (CSV registry, party mode) explicitly excluded.
|
||||
- Rejected mattpocock 100-line limit — project convention (500 lines) takes precedence; noted in inline source comment.
|
||||
- Prompts-as-code governance obligation satisfied: SKILL.md committed to repo; version control is the enforcement mechanism.
|
||||
|
||||
**Open threads:**
|
||||
- Documentation convention: scoped to Phase 3 of this issue — see "What to build" above. `write-docs` output format section will be updated once the convention is defined.
|
||||
- HITL behavioral test: see above.
|
||||
|
||||
@@ -42,7 +42,7 @@ Categories are expressed via `metadata: category:` in SKILL.md frontmatter. Defi
|
||||
| `iac` | write-ansible-role, write-terraform-module, write-k8s-manifest, write-docker-compose, proxmox-vm-spec, iac-security-review, write-molecule-test — **global optional** |
|
||||
| `gitea` | setup-gitea-mcp, post-pr-review, create-issue — **global optional** |
|
||||
| `cross-cutting` | zoom-out, caveman, session-handoff, governance-check, git-guardrails, git-commit-message |
|
||||
| `factory` | write-a-skill, write-adr, write-workflow, write-eval, validate-skill, upgrade-skill, write-issue-spec |
|
||||
| `factory` | write-skill, write-adr, write-workflow, write-eval, validate-skill, upgrade-skill, write-issue-spec |
|
||||
| `roles` | architect, developer, reviewer, security, qa, ops — **Chunk 5** |
|
||||
|
||||
### 3. SKILL.md authoring standard (full)
|
||||
|
||||
@@ -90,7 +90,7 @@ Write the `description:` frontmatter field first. Test it against three cases be
|
||||
2. Implicit invocation — user describes the task without the trigger phrase
|
||||
3. Negative case — adjacent task that must NOT activate this skill
|
||||
|
||||
Do not proceed to the body until all three pass.
|
||||
For each case, output an explicit **PASS** or **FAIL** result. Do not proceed to step b until all three show PASS. Including the description inside the section walk-through (step b) does not satisfy this gate — it must be a standalone test-then-proceed step with per-case verdicts. If any case fails, revise the description and re-test before continuing.
|
||||
|
||||
**b. Per-section options walk-through**
|
||||
Before writing anything, walk through each body section with the human. For each section:
|
||||
@@ -109,7 +109,11 @@ Populate after upstream review. Two distinct fields:
|
||||
- `references:` — general citations (research papers, documentation, standard specifications). Present only if the skill cites external research.
|
||||
|
||||
**d. eval.yaml** (sub-agent)
|
||||
Spawn an agent to produce the eval via `write-eval`. Co-located at `.agents/evals/<category>/<skill-name>/eval.yaml`. Must contain all five required test types (see Eval schema below).
|
||||
Invoke `write-eval` in two steps to preserve its confirmation gate:
|
||||
1. Sub-agent proposes test cases and returns the plan to the main conversation.
|
||||
2. Human confirms the plan; then sub-agent writes the file.
|
||||
|
||||
Do not pass pre-designed test cases directly to a write agent — that collapses the plan-then-confirm gate into a single step, bypassing write-eval's own constraint. Co-located at `.agents/evals/<category>/<skill-name>/eval.yaml`. Must contain all five required test types (see Eval schema below).
|
||||
|
||||
**e. HITL behavioral test**
|
||||
Human opens a fresh Claude session, invokes the skill with its trigger phrase, and verifies output. Do not batch more than 2–3 skills before running behavioral tests — output volume must stay within genuine human review capacity. An approval that cannot be meaningfully evaluated is not an approval.
|
||||
|
||||
@@ -2,18 +2,19 @@
|
||||
|
||||
Current deployed state of this repo — what you get if you run `install.sh` today. Updated at the close of each chunk and in the same PR as any behavior change.
|
||||
|
||||
*Last updated: 2026-05-17 (issue 0018 phase 1)*
|
||||
*Last updated: 2026-05-17 (issue 0018 phase 2)*
|
||||
|
||||
## What is deployed
|
||||
|
||||
### Skills
|
||||
14 skills deployed to `~/.agents/skills/` via `install.sh`. Available as slash commands in Claude Code via `~/.claude/skills/ → ~/.agents/skills/` symlink. 12 are first-draft placeholders pending rebuild in Chunk 3; 2 are new Chunk 3 factory skills (`write-eval`, `write-skill`).
|
||||
14 skills deployed to `~/.agents/skills/` via `install.sh`. Available as slash commands in Claude Code via `~/.claude/skills/ → ~/.agents/skills/` symlink. 11 are first-draft placeholders pending rebuild in Chunk 3; 3 are new Chunk 3 skills (`write-eval`, `write-skill`, `write-docs`).
|
||||
|
||||
**Factory bootstrap (Chunk 3):**
|
||||
- `write-eval` — produces `eval.yaml` test files for skills. Hand-written (bootstrap). Eval at `.agents/evals/factory/write-eval/eval.yaml`.
|
||||
- `write-skill` — authors new SKILL.md files and converts placeholders to canonical format. Hand-written (bootstrap — cannot author itself before it exists). Eval at `.agents/evals/factory/write-skill/eval.yaml`. Invokes `write-eval` as part of its own process.
|
||||
- `write-docs` — produces technical documentation derived from code and spec; never invents behaviour. **First factory-authored skill** (SKILL.md produced via `write-skill`, eval via `write-eval`). Eval at `.agents/evals/implement/write-docs/eval.yaml`. Sources: anthropics/skills `doc-coauthoring`, mattpocock/skills `write-a-skill`, bmad-code-org/BMAD-METHOD `bmad-advanced-elicitation`.
|
||||
|
||||
Current skills: `caveman`, `diagnose`, `grill-me`, `grill-with-docs`, `improve-codebase-architecture`, `prototype`, `tdd`, `to-issues`, `to-prd`, `triage`, `write-a-skill`, `write-eval`, `write-skill`, `zoom-out`.
|
||||
Current skills: `caveman`, `diagnose`, `grill-me`, `grill-with-docs`, `improve-codebase-architecture`, `prototype`, `tdd`, `to-issues`, `to-prd`, `triage`, `write-docs`, `write-eval`, `write-skill`, `zoom-out`.
|
||||
|
||||
**Chunk 3 target:** 42 skills across 9 categories. PRD: `docs/prd/chunk-3-skills-library.md`. Canonical build reference: `docs/research/ai-coding-factory/ai-coding-factory-skills-index.md` (delete once all skills exist). Skills stored flat (`skill-name/SKILL.md`) per ADR-0009; category in `metadata.category` frontmatter. Categories: design, factory, implement, test, review, deploy, operate, cross-cutting, iac (2 skills only — docker-compose + iac-security-review). Role skills (6) deferred to Chunk 5. Gitea skills moved to `providers/gitea/` provider adapter.
|
||||
|
||||
@@ -46,6 +47,7 @@ For chunk planning and open questions, see `docs/ROADMAP.md`.
|
||||
|
||||
## Recent changes
|
||||
|
||||
- 2026-05-17 — Issue 0018 phase 2 complete: `write-docs` skill written and deployed. First skill produced end-to-end by the factory (SKILL.md via `write-skill`, eval via `write-eval`). Category: implement. Key decisions: file-approval gate before reading (user names files or approves proposals); gap check before drafting (user fills what code doesn't explain); stage skipping allowed with logged reason; full revised section shown before confirmation gate; surgical edits only with per-round delta summary; Reader Testing via scoped sub-agent (doc + questions only, no source files); summary/overview sections written last. Sources: anthropics/skills doc-coauthoring (Reader Testing stage, surgical-edit constraint), mattpocock/skills write-a-skill (trigger pattern), bmad-code-org/BMAD-METHOD bmad-advanced-elicitation (confirmation gate). Open follow-up: documentation convention (file/folder/content structure, global vs repo-specific) — not yet defined.
|
||||
- 2026-05-17 — Issue 0018 phase 1 complete: `write-skill` bootstrap skill written and deployed. Hand-written (factory bootstrap). Self-authored — no upstream content adopted; agentskills.io best-practices and optimizing-descriptions docs cited as references. Speckit excluded (AGPL-3.0). Key decisions: new-skill + placeholder-conversion scope only (upgrades → `upgrade-skill`); trigger description tested against 3 cases before body written; `write-eval` invoked as step 7 in process; HITL prompt as step 8. Eval at `.agents/evals/factory/write-skill/eval.yaml`.
|
||||
- 2026-05-17 — Issue 0017 complete: `write-eval` bootstrap skill written and deployed. Two sections schema (`trigger_tests` + `output_tests`), provider-agnostic string assertions, show-plan-then-merge-on-rerun behaviour, conflict flagging (B model). Sources: agentskills/agentskills, darkrishabh/agent-skills-eval, bmad-code-org/BMAD-METHOD, mattpocock/skills. Hand-written eval at `.agents/evals/factory/write-eval/eval.yaml`.
|
||||
- 2026-05-17 — Issue 0016 complete: skill implementation workflow grill completed. `docs/notes/skill-implementation-workflow.md` written. All issues 0017–0028 updated with specific acceptance criteria. Key conventions: sub-agents prescribed at each research/writing step; conflict check against constitution + factory principles before synthesis grill; `when:` and `references:` fields added to authoring standard; write-docs moved to issue 0018 phase 2 (first factory-authored skill).
|
||||
|
||||
Reference in New Issue
Block a user