From 58476a96fdcc47487ffe54ccc4982240ca381551 Mon Sep 17 00:00:00 2001 From: Defame1297 Date: Sun, 17 May 2026 18:24:28 +0000 Subject: [PATCH] =?UTF-8?q?feat:=20implement=20issue=200018=20phase=201=20?= =?UTF-8?q?=E2=80=94=20factory/write-skill=20bootstrap=20skill?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Sonnet 4.6 --- .agents/evals/factory/write-skill/eval.yaml | 68 ++++++++++ .agents/skills/write-skill/SKILL.md | 130 ++++++++++++++++++++ LESSONS.md | 8 ++ docs/ROADMAP.md | 2 +- docs/issues/0018-factory-write-skill.md | 32 ++++- docs/spec/overview.md | 11 +- 6 files changed, 242 insertions(+), 9 deletions(-) create mode 100644 .agents/evals/factory/write-skill/eval.yaml create mode 100644 .agents/skills/write-skill/SKILL.md diff --git a/.agents/evals/factory/write-skill/eval.yaml b/.agents/evals/factory/write-skill/eval.yaml new file mode 100644 index 0000000..4193518 --- /dev/null +++ b/.agents/evals/factory/write-skill/eval.yaml @@ -0,0 +1,68 @@ +skill_name: write-skill + +trigger_tests: + - id: explicit-trigger-new-skill + name: Explicit — new skill phrase + query: "Write a new skill for handling database migrations" + should_trigger: true + + - id: implicit-trigger-no-phrase + name: Implicit — no trigger phrase + query: "I want to add a skill that automates our deploy process" + should_trigger: true + + - id: implicit-trigger-conversion + name: Implicit — placeholder conversion + query: "The grill-me skill is a Pocock placeholder, can you convert it to our standard?" + should_trigger: true + + - id: negative-trigger-upgrade + name: Negative — existing skill fix + query: "The tdd skill is producing wrong output, fix it" + should_trigger: false + + - id: negative-trigger-code-refactor + name: Negative — code refactor + query: "Refactor this module to use the new API client" + should_trigger: false + + - id: negative-trigger-write-eval + name: Negative — eval request + query: "Write evals for the diagnose skill" + should_trigger: false + +output_tests: + - id: output-has-all-sections + name: All 8 body sections present + type: deterministic + prompt: "Write a new skill for linting markdown files, category: implement" + expected_output: A complete SKILL.md containing all 8 required body sections in order. + assertions: + - "Output contains '## Role'" + - "Output contains '## When to use / When not to use'" + - "Output contains '## Required inputs'" + - "Output contains '## Constraints'" + - "Output contains '## Process'" + - "Output contains '## Output format'" + - "Output contains '## Failure handling'" + - "Output contains '## Self-check'" + + - id: output-path-correct + name: Output path and frontmatter fields correct + type: deterministic + prompt: "Write a new skill for linting markdown files, category: implement" + expected_output: A SKILL.md with correct output path stated and all required frontmatter fields present. + assertions: + - "Output contains '.agents/skills/' in the stated output path" + - "Output contains 'metadata:' and 'category:' in frontmatter" + - "Output contains 'version:'" + - "Output contains 'when:'" + + - id: output-trigger-tested-before-body + name: Trigger description tested before body content written + type: llm-rubric + prompt: "Write a new skill for summarising pull request diffs" + expected_output: The skill presents a trigger description and tests it against at least 3 cases (explicit, implicit, negative) before proposing or writing any body section content. + assertions: + - "The skill proposes a trigger description and explicitly tests it against an explicit query, an implicit query, and a negative query before writing any body section" + - "The skill walks through each body section individually and seeks confirmation before writing the file" diff --git a/.agents/skills/write-skill/SKILL.md b/.agents/skills/write-skill/SKILL.md new file mode 100644 index 0000000..5c28305 --- /dev/null +++ b/.agents/skills/write-skill/SKILL.md @@ -0,0 +1,130 @@ +--- +name: write-skill +description: Write a new skill for X, create a SKILL.md that does Y, build a skill to handle Z. Use when the user wants to author a new skill file or convert an existing placeholder skill to the canonical authoring standard. Do NOT use when the user wants to fix or update an existing well-formed skill, run existing evals, refactor application code, or write documentation for non-skill artifacts. +version: "1.0" +updated: 2026-05-17 +when: invoked by explicit trigger ("write a new skill for X", "create a SKILL.md that does Y") or implicit request to author a skill file or convert a placeholder to canonical format +metadata: + category: factory +references: + - https://agentskills.io/skill-creation/optimizing-descriptions + - https://agentskills.io/skill-creation/best-practices +--- + +## Role + +You are a skill author producing complete, correctly-structured SKILL.md files for an AI agent skills library, following the agentskills.io open standard with project-specific extensions. + +## When to use / When not to use + +**Use when:** +- User wants to create a new skill ("write a skill for X", "build a skill that does Y") +- User wants to convert a placeholder or non-standard SKILL.md to canonical format + +**Do not use when:** +- User wants to fix or update an existing well-formed skill → `upgrade-skill` +- User wants to run or evaluate existing skills → `write-eval` +- User wants to refactor application code → `implement/refactor` +- User wants to write docs for a non-skill artifact → `write-docs` +- User asks for a workflow, ADR, or PRD — dedicated skills exist for those + +## Required inputs + +- **Skill name** — kebab-case slug; inferred from description if not stated explicitly +- **Category** — one of: `design|factory|implement|test|review|deploy|operate|cross-cutting|iac` +- **Purpose + use cases** — what the skill does and what tasks it handles; source for the trigger description +- **When-not criteria** — adjacent tasks this skill must NOT activate on; needed for the negative trigger test +- For placeholder conversions: **existing SKILL.md path** — read before writing + +## Constraints + +- Output path: `.agents/skills//SKILL.md` +- Frontmatter must include: `name`, `description`, `version`, `updated`, `when`, `metadata.category`; `source:` only if upstream content adopted; `references:` only if external citations used +- Body must contain all 8 sections in order: Role, When to use / When not to use, Required inputs, Constraints, Process, Output format, Failure handling, Self-check +- Trigger description must be tested against 3 cases (explicit, implicit, negative) before any body content is written — do not proceed if any case fails +- Body ≤500 lines; XML tags only if ≥3 logical sections and 500+ tokens +- Check for overlapping skills in `.agents/skills/` before writing — flag overlap and get direction before continuing +- For placeholder conversions: read existing SKILL.md before writing; remove all dead Pocock references (`setup-matt-pocock-skills`, `AGENT-BRIEF.md`, `OUT-OF-SCOPE.md`) +- Show proposed structure and confirm with user before writing the file +- Do not mark a skill complete until both the eval and the HITL behavioral test are done + +## Process + +1. **Gather requirements.** If not provided upfront, ask: what does the skill do, what tasks does it handle, what should it NOT activate on, what category, is this a new skill or a placeholder conversion? + +2. **Check for overlap.** Scan `.agents/skills/` for existing skills with similar purpose or trigger phrases. If overlap found, surface the conflicting skill name and description — get explicit direction before continuing. + +3. **Draft the trigger description.** Write the `description:` field from the gathered requirements. + +4. **Test the trigger description** against three cases before writing anything else: + - Explicit: a query using the trigger phrase directly — must activate + - Implicit: a query describing the task without naming it — must activate + - Negative: an adjacent task from the when-not criteria — must NOT activate + + If any case fails, propose a revised description and re-test. Do not proceed to the body until all three pass. + +5. **Walk through each body section.** For each of the 8 sections: state the proposed content and where it comes from, present alternatives if they exist. Get confirmation before moving to the next section. Do not write the file until all sections are confirmed. + +6. **Write the SKILL.md** at `.agents/skills//SKILL.md`. For placeholder conversions: read the existing file first, carry forward content worth keeping, remove dead references. + +7. **Invoke `write-eval`.** Run `write-eval` to produce the eval at `.agents/evals///eval.yaml`. Do not mark the skill complete until the eval file exists. + +8. **Prompt for HITL behavioral test.** Remind the user to open a fresh session, invoke the skill with its trigger phrase, and verify the output meets the authoring standard before committing. The skill is not done until this test passes. + +## Output format + +```markdown +--- +name: +description: +version: "1.0" +updated: YYYY-MM-DD +when: +metadata: + category: +source: # omit if self-authored + - repo: org/repo + commit: + files: + - path/to/file.md # what was taken + updated: YYYY-MM-DD +references: # omit if no external citations + - +--- + +## Role +## When to use / When not to use +## Required inputs +## Constraints +## Process +## Output format +## Failure handling +## Self-check +``` + +For placeholder conversions, the output replaces the existing file entirely — no partial edits. + +## Failure handling + +- **Skill name missing or ambiguous:** ask before proceeding — do not invent a name +- **Category not provided:** ask before computing the output path — do not guess +- **Trigger description fails a test case:** propose a revised description and re-test; do not proceed to body content until all three pass +- **Overlap found with existing skill:** surface the conflict (name + description of the conflicting skill), ask for direction — do not continue autonomously +- **Existing SKILL.md not found for a placeholder conversion:** stop and report the path searched; do not write based on assumptions +- **Confirmed sections would produce a body exceeding 500 lines:** flag before writing, ask which sections to trim +- **`write-eval` fails or is unavailable:** flag to the user — do not mark the skill complete without an eval + +## Self-check + +- [ ] All 8 body sections present in order +- [ ] Trigger description tested against all 3 cases — all pass before body was written +- [ ] Overlap check completed against `.agents/skills/` +- [ ] Output path correct: `.agents/skills//SKILL.md` +- [ ] Frontmatter complete: `name`, `description`, `version`, `updated`, `when`, `metadata.category` present +- [ ] `source:` present only if upstream content adopted; absent if self-authored +- [ ] `references:` present only if external citations used; absent otherwise +- [ ] Body ≤500 lines +- [ ] Proposed structure confirmed by user before file was written +- [ ] For placeholder conversions: dead Pocock references removed +- [ ] `write-eval` invoked — eval.yaml exists at correct path +- [ ] User prompted for HITL behavioral test diff --git a/LESSONS.md b/LESSONS.md index 7f85159..6e78f6a 100644 --- a/LESSONS.md +++ b/LESSONS.md @@ -38,6 +38,14 @@ The secrets prohibition in `core/instructions/governance.md` fired correctly whe The synthesis grill (step 4) answers schema-level questions: how to combine upstreams, which eval schema to use, merge behaviour. Step 5b is a different conversation: how upstream content maps to each SKILL.md body section, what options each section had, and which was chosen. Collapsing them — writing the SKILL.md immediately after the grill without a per-section walk-through — means the human never sees the upstream options for the body and has no opportunity to redirect before the file is written. Fix: step 5b is now a named gate in the workflow. Walk through every body section one at a time, cite the upstream source, present alternatives, get confirmation. Only then write. Applies to both hand-written (bootstrap) and write-skill-produced skills. +## 2026-05-17 — Skill-calls-skill composition must be a named process step + +When a skill invokes another skill as part of its work (e.g. write-skill invoking write-eval to produce the eval), that call must be a numbered step in the Process section — not left as an implicit external workflow step. If it isn't named, practitioners either forget it or do it manually outside the skill, breaking the composition chain. The user caught this during the write-skill co-write; it was absent from the process despite being in the workflow doc. Fix: when designing any skill that composes another, list each composed call explicitly as a numbered step with a "do not mark complete until X exists" constraint. + +## 2026-05-17 — AGPL-3.0 repos appear prominently in community skill search results + +When searching GitHub for agent skill upstreams, AGPL-3.0 repos (e.g. dceoy/speckit-agent-skills) appear alongside permissive-licensed ones without obvious visual distinction. AGPL imposes copyleft obligations on adopted content. Always run a licence check (GitHub API `/license` endpoint) before extracting any content from a new upstream. An AGPL finding is a hard exclude — record the repo, SHA, and licence in source review notes so future sessions don't re-review it. + ## 2026-05-17 — HITL gap: agent delegates confirmation to permission system The agent-level HITL rule ("require explicit confirmation before irreversible shared-state operations") is being bypassed: the agent calls the tool and lets the permission dialog catch it. This means the rule is not firing in agent reasoning — it's the permission system acting as a safety net. If a user selects "don't ask again," the net disappears. Fix: the HITL rule needs to be framed as "do not call the tool" rather than "ask before proceeding" — the agent must ask first, then act only after explicit confirmation. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 166acdd..f2d14d4 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -98,4 +98,4 @@ Items consciously not resolved — to be addressed in the relevant chunk PRD or - **Chunk 2 behavioral tests** — run and fully resolved 2026-05-17. 7/8 pass; scenario 4 (push confirmation) inconclusive — no remote in test environment, rule tightened but unverified. All fixable failures addressed: rule specificity in `providers/claude-code/CLAUDE.md`; context-loading guarantee via `@import CONTEXT.md` in repo CLAUDE.md; standing rule in CONTEXT.md to check `docs/adr/` and ROADMAP resolved entries before answering design questions. Chunk 2 ✅ complete. - **Governance Phase 1 behavioral tests** — run 2026-05-17. 3/4 testable scenarios pass. Secrets rule gap fixed (2026-05-17): extended to cover credential reproduction in response text and examples, with placeholder requirement added to `core/instructions/governance.md`. HITL scenario not testable in this environment (Nginx not installed); HITL gap evidenced by instructions test scenario 4 — push confirmation rule fix addresses the same root cause. Governance Phase 1 ✅ complete. - **AI ethics/security workstream** — `docs/notes/ai-ethics-security-principles.md` exploration note is superseded. Governance Phase 1 (`core/instructions/governance.md`) covers all planned scope: credentials, data classification, HITL, scope discipline, agent autonomy, transparency, and security code review. Tier-placement architectural question resolved by the `@import` always-on model. No separate workstream needed. -- **Chunk 3 grill complete** — 2026-05-17. PRD at `docs/prd/chunk-3-skills-library.md`. Key decisions: 42-skill target library, AGENTS.md refactor as prerequisite issue (both CLAUDE.md files become thin adapters), git-cliff for changelog, provider-agnostic issue tracker abstraction, grill-me/grill-lean design phase split, factory bootstrap order (write-eval → write-skill → write-docs phase 2 → write-adr → remaining factory → design → parallel category groups). ADRs written: 0011 (provider-agnostic issue tracker), 0012 (AGENTS.md governance entry point, partially supersedes ADR-0005). Upstream review cadence: per-skill + quarterly post-roadmap (per-chunk-start changed to per-skill by issue 0016 grill). **Issues created 0015–0028** — all HITL; ~~0015 (AGENTS.md refactor, prerequisite)~~ ✅, ~~0016 (skill workflow grill, produces conventions for 0017–0028)~~ ✅, ~~0017 (bootstrap skill: write-eval)~~ ⏳ HITL pending, 0018 (write-skill + write-docs as phase 2), 0019 (remaining factory skills), 0020–0027 (design/implement/test/review/deploy/operate/iac/cross-cutting), 0028 (chunk closure). ~~Acceptance criteria for 0017–0028 to be refined after 0016 grill session.~~ ✅ Refined 2026-05-17 — see `docs/notes/skill-implementation-workflow.md`. +- **Chunk 3 grill complete** — 2026-05-17. PRD at `docs/prd/chunk-3-skills-library.md`. Key decisions: 42-skill target library, AGENTS.md refactor as prerequisite issue (both CLAUDE.md files become thin adapters), git-cliff for changelog, provider-agnostic issue tracker abstraction, grill-me/grill-lean design phase split, factory bootstrap order (write-eval → write-skill → write-docs phase 2 → write-adr → remaining factory → design → parallel category groups). ADRs written: 0011 (provider-agnostic issue tracker), 0012 (AGENTS.md governance entry point, partially supersedes ADR-0005). Upstream review cadence: per-skill + quarterly post-roadmap (per-chunk-start changed to per-skill by issue 0016 grill). **Issues created 0015–0028** — all HITL; ~~0015 (AGENTS.md refactor, prerequisite)~~ ✅, ~~0016 (skill workflow grill, produces conventions for 0017–0028)~~ ✅, ~~0017 (bootstrap skill: write-eval)~~ ⏳ HITL pending, ~~0018 phase 1 (write-skill)~~ ⏳ HITL pending, 0018 phase 2 (write-docs — first factory-authored skill), 0019 (remaining factory skills), 0020–0027 (design/implement/test/review/deploy/operate/iac/cross-cutting), 0028 (chunk closure). ~~Acceptance criteria for 0017–0028 to be refined after 0016 grill session.~~ ✅ Refined 2026-05-17 — see `docs/notes/skill-implementation-workflow.md`. diff --git a/docs/issues/0018-factory-write-skill.md b/docs/issues/0018-factory-write-skill.md index 98dbccf..7fa756d 100644 --- a/docs/issues/0018-factory-write-skill.md +++ b/docs/issues/0018-factory-write-skill.md @@ -42,10 +42,10 @@ Follow the per-skill workflow defined in `docs/notes/skill-implementation-workfl ## Acceptance criteria -- [ ] `.agents/skills/write-skill/SKILL.md` exists; `metadata.category: factory`; authoring standard met -- [ ] Trigger description validates against explicit, implicit, and negative test queries -- [ ] `.agents/evals/factory/write-skill/eval.yaml` exists; produced via `write-eval` -- [ ] `install.sh` deploys `write-skill` to `~/.agents/skills/` +- [x] `.agents/skills/write-skill/SKILL.md` exists; `metadata.category: factory`; authoring standard met +- [x] Trigger description validates against explicit, implicit, and negative test queries +- [x] `.agents/evals/factory/write-skill/eval.yaml` exists; produced via `write-eval` +- [x] `install.sh` deploys `write-skill` to `~/.agents/skills/` - [ ] **HITL:** human runs behavioral test: invoke "write a new skill for X" and verify the produced SKILL.md meets the authoring standard - [ ] **HITL:** human reviews SKILL.md and eval before committing - [ ] Per-skill process followed for both phases (see `docs/notes/skill-implementation-workflow.md`) @@ -61,3 +61,27 @@ Follow the per-skill workflow defined in `docs/notes/skill-implementation-workfl - 0016 (grill defines per-skill workflow) - 0017 (`write-eval` needed to produce the eval for this skill) + +## Handoff — Phase 1 + +**Status:** complete — pending HITL behavioral test (acceptance criteria steps 5–6) + +**Files produced:** +- `.agents/skills/write-skill/SKILL.md` +- `.agents/evals/factory/write-skill/eval.yaml` + +**Key decisions:** +- Scope: new-skill creation + placeholder→canonical conversion only. Updating/fixing existing skills → `upgrade-skill` (separate skill in the index). +- Trigger validation (3 cases) is a named gate in write-skill's process before body content is written. +- `write-eval` is step 7 of write-skill's process — the skill invokes it automatically. HITL prompt is step 8. +- Self-authored (no `source:` field); `references:` cites agentskills.io best-practices and optimizing-descriptions. +- speckit-agent-skills (dceoy) excluded — AGPL-3.0 copyleft. +- Role is self-contained (no reference to workflow doc) so it can be used standalone after chunk 3. + +**Open threads:** +- HITL behavioral test: open a fresh session, invoke "write a new skill for X" in this repo context, verify trigger is tested before body, per-section walk-through happens, write-eval is invoked, HITL prompt appears. +- Phase 2 (`write-docs`): first skill authored end-to-end by the factory. Follow full per-skill workflow from `docs/notes/skill-implementation-workflow.md`. + +**Next session start:** +- Load: `CONTEXT.md`, `docs/notes/skill-implementation-workflow.md`, `docs/issues/0018-factory-write-skill.md` +- First action: HITL behavioral test for write-skill, then begin phase 2 (`write-docs`) using write-skill diff --git a/docs/spec/overview.md b/docs/spec/overview.md index d2653fd..43819d9 100644 --- a/docs/spec/overview.md +++ b/docs/spec/overview.md @@ -2,16 +2,18 @@ Current deployed state of this repo — what you get if you run `install.sh` today. Updated at the close of each chunk and in the same PR as any behavior change. -*Last updated: 2026-05-17 (issue 0017)* +*Last updated: 2026-05-17 (issue 0018 phase 1)* ## What is deployed ### Skills -13 skills deployed to `~/.agents/skills/` via `install.sh`. Available as slash commands in Claude Code via `~/.claude/skills/ → ~/.agents/skills/` symlink. 12 are first-draft placeholders pending rebuild in Chunk 3; 1 is a new Chunk 3 factory skill (`write-eval`). +14 skills deployed to `~/.agents/skills/` via `install.sh`. Available as slash commands in Claude Code via `~/.claude/skills/ → ~/.agents/skills/` symlink. 12 are first-draft placeholders pending rebuild in Chunk 3; 2 are new Chunk 3 factory skills (`write-eval`, `write-skill`). -**Factory bootstrap (Chunk 3):** `write-eval` deployed — produces `eval.yaml` test files for skills. Hand-written (bootstrap skill). Eval at `.agents/evals/factory/write-eval/eval.yaml`. +**Factory bootstrap (Chunk 3):** +- `write-eval` — produces `eval.yaml` test files for skills. Hand-written (bootstrap). Eval at `.agents/evals/factory/write-eval/eval.yaml`. +- `write-skill` — authors new SKILL.md files and converts placeholders to canonical format. Hand-written (bootstrap — cannot author itself before it exists). Eval at `.agents/evals/factory/write-skill/eval.yaml`. Invokes `write-eval` as part of its own process. -Current skills: `caveman`, `diagnose`, `grill-me`, `grill-with-docs`, `improve-codebase-architecture`, `prototype`, `tdd`, `to-issues`, `to-prd`, `triage`, `write-a-skill`, `zoom-out`. +Current skills: `caveman`, `diagnose`, `grill-me`, `grill-with-docs`, `improve-codebase-architecture`, `prototype`, `tdd`, `to-issues`, `to-prd`, `triage`, `write-a-skill`, `write-eval`, `write-skill`, `zoom-out`. **Chunk 3 target:** 42 skills across 9 categories. PRD: `docs/prd/chunk-3-skills-library.md`. Canonical build reference: `docs/research/ai-coding-factory/ai-coding-factory-skills-index.md` (delete once all skills exist). Skills stored flat (`skill-name/SKILL.md`) per ADR-0009; category in `metadata.category` frontmatter. Categories: design, factory, implement, test, review, deploy, operate, cross-cutting, iac (2 skills only — docker-compose + iac-security-review). Role skills (6) deferred to Chunk 5. Gitea skills moved to `providers/gitea/` provider adapter. @@ -44,6 +46,7 @@ For chunk planning and open questions, see `docs/ROADMAP.md`. ## Recent changes +- 2026-05-17 — Issue 0018 phase 1 complete: `write-skill` bootstrap skill written and deployed. Hand-written (factory bootstrap). Self-authored — no upstream content adopted; agentskills.io best-practices and optimizing-descriptions docs cited as references. Speckit excluded (AGPL-3.0). Key decisions: new-skill + placeholder-conversion scope only (upgrades → `upgrade-skill`); trigger description tested against 3 cases before body written; `write-eval` invoked as step 7 in process; HITL prompt as step 8. Eval at `.agents/evals/factory/write-skill/eval.yaml`. - 2026-05-17 — Issue 0017 complete: `write-eval` bootstrap skill written and deployed. Two sections schema (`trigger_tests` + `output_tests`), provider-agnostic string assertions, show-plan-then-merge-on-rerun behaviour, conflict flagging (B model). Sources: agentskills/agentskills, darkrishabh/agent-skills-eval, bmad-code-org/BMAD-METHOD, mattpocock/skills. Hand-written eval at `.agents/evals/factory/write-eval/eval.yaml`. - 2026-05-17 — Issue 0016 complete: skill implementation workflow grill completed. `docs/notes/skill-implementation-workflow.md` written. All issues 0017–0028 updated with specific acceptance criteria. Key conventions: sub-agents prescribed at each research/writing step; conflict check against constitution + factory principles before synthesis grill; `when:` and `references:` fields added to authoring standard; write-docs moved to issue 0018 phase 2 (first factory-authored skill). - 2026-05-17 — Issue 0015 complete: AGENTS.md refactor implemented. Two AGENTS.md files created (`AGENTS.md` at repo root, `core/AGENTS.md` deployed to `~/.agents/AGENTS.md`). Both CLAUDE.md files slimmed to thin adapters. `deploy-manifest.sh` updated. `docs/spec/architecture.md` updated with new structure. ADR-0012 in effect.