Defame1297/holocron
0
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases 3 Wiki Activity

docs: issue 0016 — skill implementation workflow grill

Browse Source 
Produces docs/notes/skill-implementation-workflow.md with agreed conventions
for all Chunk 3 skill issues (0017–0028). Key decisions:

- Per-skill process: source discovery (sub-agent) → source review with
  licence/security check (sub-agent) → conflict check vs constitution +
  factory principles (sub-agent) → synthesis grill → co-write iteratively
- Bootstrap: write-eval (hand-written) → write-skill (hand-written) →
  write-docs (first factory-authored, phase 2 of 0018) → everything else
- Upstream review changed from per-chunk-start to per-skill
- `when:` and `references:` frontmatter fields added to authoring standard
- Sub-agent usage prescribed as named steps in the workflow
- HITL: human reviewed and approved conventions

Updates: PRD implementation decisions; issues 0016–0028 with specific
acceptance criteria; docs/spec/overview.md; ROADMAP Chunk 3 housekeeping note
(bootstrap order, cadence, acceptance criteria status); CONTEXT.md Source field
(per-skill cadence, references: companion field); LESSONS.md with three patterns
from the grill session.

Post-grill additions (same session): Step 6 (session handoff) added to the
workflow; handoff section appended to issue 0016; handoff checklist item added
to Chunk 3 closure issue (0028).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Defame1297
2026-05-17 16:21:38 +00:00
parent a1f1f9ded8
commit c705a38809
19 changed files with 393 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files

229
docs/notes/skill-implementation-workflow.md Normal file
View File

@@ -0,0 +1,229 @@
# Skill Implementation Workflow
**Produced by:** issue 0016 grill session, 2026-05-17
**Applies to:** all Chunk 3 skill issues (0017–0028)
---
## Bootstrap sequence
The first two skills are hand-authored because the factory cannot build itself before it exists. Everything from step 3 onward uses the factory.
| Step | Skill | SKILL.md | eval.yaml |
|---|---|---|---|
| 1 | `write-eval` | Hand-written | Hand-written |
| 2 | `write-skill` (phase 1 of issue 0018) | Hand-written | Via `write-eval` |
| 3 | `write-docs` (phase 2 of issue 0018) | Via `write-skill` | Via `write-eval` |
| 4+ | Everything else | Via `write-skill` | Via `write-eval` |
Step 3 (`write-docs`) is the first real factory output — `write-skill` authors its own SKILL.md.
---
## Per-skill process
Every skill — new or refactor — follows this sequence. Sub-agents are the default for research and writing steps to keep the main context lean.
### Step 1 — Source discovery (sub-agent)
Spawn an Explore agent to:
- Check the issue and skills index for listed upstream sources
- Web search for additional sources not already listed
- For refactors: retrieve the current SKILL.md as one of the inputs
Return: list of sources with repo slugs and current HEAD commit SHAs.
### Step 2 — Source review (sub-agent)
Spawn a research agent to read each source and extract relevant content. The agent must:
- Read relevant files from each upstream source
- Note the commit SHA at time of review
- Flag any changes since the last `updated:` date (for skills with an existing `source:` field)
- Apply the pull criteria below to recommend whether to incorporate each change
- Run a licence check: note the licence of each upstream repo; flag any copyleft (GPL, AGPL) material before adoption
- Flag any third-party skill files that appear unreviewed or have unusual structure (documented 36% prompt injection rate in community skill sets — treat unreviewed upstream skills as untrusted)
Return: extracted content per source, commit SHAs, licence notes, security flags.
**Pull criteria:**
| Change type | Action |
|---|---|
| Trigger description or process sections changed | Always pull |
| New constraints or failure-handling cases added | Evaluate — pull if it closes a real gap |
| Formatting, wording, minor structural changes | Skip |
| Any upstream reviewed (pull or skip) | Update `updated:` date in `source:` regardless |
### Step 3 — Conflict check (sub-agent)
Spawn an agent to cross-check the extracted upstream content against:
- `core/instructions/governance.md` — hard prohibitions, data classification, HITL requirements
- `docs/ai-constitution.md` — scope discipline, deterministic execution preference, licence obligations, output volume constraint, transparency requirements
The agent flags conflicts and tensions as numbered items for the synthesis grill. It does **not** resolve them — that is the grill's job.
Common conflicts to check:
- Does any upstream pattern require the agent to take irreversible action without HITL? Flag.
- Does any adopted content handle Restricted-tier data (credentials, health data)? Flag.
- Does any upstream skip licence scanning before code commit? Flag.
- Does any upstream encourage repeated AI inference for a deterministic task? Flag — note deterministic execution preference.
- Does the upstream skill body exceed 500 lines or load excessive context at startup? Flag — note token architecture constraint.
Return: numbered conflict/tension list, or "no conflicts found."
### Step 4 — Synthesis grill
Short grill session with the human covering:
- How to combine elements from the upstreams (what to take from each, what to leave)
- How to resolve any flagged conflicts from step 3
- Any open design questions specific to this skill
This is not a full grill-with-docs session — it is focused and bounded. If there are no conflicts and the combining decision is obvious, the grill can be a single confirmation exchange.
### Step 5 — Co-write iteratively
Work through the following in order, iterating with the human. Sub-agents handle writing tasks where context accumulation is a risk.
**a. Trigger description**
Write the `description:` frontmatter field first. Test it against three cases before writing the body:
1. Explicit invocation — user says the trigger phrase directly
2. Implicit invocation — user describes the task without the trigger phrase
3. Negative case — adjacent task that must NOT activate this skill
Do not proceed to the body until all three pass.
**b. SKILL.md** (sub-agent)
Spawn a write agent to produce the SKILL.md using `write-skill` (or hand-write for bootstrap skills). The agent receives: trigger description, synthesis grill decisions, upstream content to incorporate, authoring standard (see below).
**c. `source:` and `references:` fields**
Populate after upstream review. Two distinct fields:
- `source:` — upstream provenance tracking (repo slug, commit SHA, files adopted with inline comments, updated date). Present only if content was adopted. Absence = self-authored.
- `references:` — general citations (research papers, documentation, standard specifications). Present only if the skill cites external research.
**d. eval.yaml** (sub-agent)
Spawn an agent to produce the eval via `write-eval`. Co-located at `.agents/evals/<category>/<skill-name>/eval.yaml`. Must contain all five required test types (see Eval schema below).
**e. HITL behavioral test**
Human opens a fresh Claude session, invokes the skill with its trigger phrase, and verifies output. Do not batch more than 2–3 skills before running behavioral tests — output volume must stay within genuine human review capacity. An approval that cannot be meaningfully evaluated is not an approval.
### Step 6 — Session handoff
After the behavioral test passes, close the skill session by appending a `## Handoff` section to the skill's issue file. This is the entry point for the next session — write it assuming zero shared context.
```markdown
## Handoff
**Status:** complete
**Files produced:**
- `.agents/skills/<name>/SKILL.md`
- `.agents/evals/<category>/<name>/eval.yaml`
**Key decisions:**
- <any non-obvious choices made during synthesis or co-write — if obvious from the files, omit>
**Open threads:**
- <deferred items, follow-on issues, or unresolved tensions — if none, write "none">
**Next session start:**
- Load: `CONTEXT.md`, `docs/notes/skill-implementation-workflow.md`, next skill's issue file
- First action: Step 1 (source discovery) for `<next-skill-name>`
```
If this skill produced a cross-cutting observation — a pattern that appeared across two or more skills — add an entry to `LESSONS.md` now. Do not batch LESSONS.md updates to the end of the chunk; they lose precision.
---
## Authoring standard
### Frontmatter (required)
```yaml
---
name: skill-name
description: <trigger description — written and tested first>
version: "1.0"
updated: YYYY-MM-DD
when: <when this skill is invoked — e.g. "invoked by /skill-name trigger">
metadata:
category: <design|factory|implement|test|review|deploy|operate|cross-cutting|iac>
source: # omit entirely if self-authored
- repo: org/repo
commit: abc1234
files:
- path/to/file.md # inline comment: what was taken
updated: YYYY-MM-DD
references: # omit if no external citations
- <URL or citation>
---
```
`when:` is added to all Chunk 3 skills now; full bidirectional reference convention (reverse map, reference scanner) is deferred to Chunk 4 for refinement.
`allowed-tools:` — optional field to restrict which tools the skill may invoke. Reduces token cost of tool definitions. Add when the skill has a narrow, well-defined tool surface.
### Body sections (required, in this order)
1. **Role** — one sentence: who the agent is when this skill is active
2. **When to use / When not to use** — explicit inclusion and exclusion criteria; negative cases are as important as positive
3. **Required inputs** — what must be present before the skill proceeds
4. **Constraints** — non-negotiable rules; the highest-ROI section for preventing over-engineering
5. **Process** — numbered steps (not bullets)
6. **Output format** — exactly what the skill produces; structured if downstream workflow steps depend on it
7. **Failure handling** — what to return if inputs are missing or a step fails; never silent failure
8. **Self-check** — skill-specific correctness criteria the agent verifies before responding; must be concrete, not generic
**Size:** body ≤500 lines. Use XML tags only for skills with ≥3 logical sections and 500+ tokens; default to prose.
### Refactor conventions
When refactoring an existing Pocock placeholder skill:
- The existing SKILL.md is one input to step 2 (source review)
- Use `write-skill` to produce the new SKILL.md — do not edit the old one in place
- Remove all dead references: `setup-matt-pocock-skills`, `AGENT-BRIEF.md`, `OUT-OF-SCOPE.md`
- Populate `source:` with the Pocock repo entry plus any other upstreams adopted
- Flat skill name may change (e.g. `grill-with-docs` → `grill-me`) — delete old directory, create new
---
## Eval schema
**Location:** `.agents/evals/<category>/<skill-name>/eval.yaml` — committed to the repo.
**Enforcement:** CI gates are Chunk 6. The files document expected behaviour before then.
Every eval must contain all five required test types:
| Type | Minimum count | Purpose |
|---|---|---|
| Explicit trigger test | ≥1 | User says the exact trigger phrase — skill must activate |
| Implicit trigger test | ≥1 | User describes the task without trigger phrase — skill must activate |
| Negative trigger test | ≥1 | Adjacent task that must NOT activate this skill |
| Deterministic output test | ≥2 | Schema/contains/regex assertions — blocking gates |
| LLM-rubric quality test | ≥1 | Quality catches that deterministic assertions cannot capture |
Three eval categories (align language when writing evals):
- **Trigger evals** — does the description field activate correctly?
- **Output evals** — is the structured output correct and high-quality?
- **Regression evals** — did a model update break established behaviour? (run weekly in CI, Chunk 6)
When a human corrects a skill's output, that correction becomes a new eval test case. The eval suite grows from real failures.
---
## `source:` pull criteria (upstream review, per-skill)
Upstream review happens per-skill during step 2, not once at chunk start.
| Change type | Action |
|---|---|
| Trigger description or process sections changed | Always pull |
| New constraints or failure-handling cases added | Evaluate — pull if it closes a real gap |
| Formatting, wording, minor structural changes | Skip |
| Any upstream reviewed (pull or skip) | Update `updated:` date in `source:` |
---
## Open decisions carried forward
- **`when:` full resolution** — Chunk 4 (bidirectional reference convention, reference scanner tooling)
- **PRD/issue template scope** — refined during `write-prd` (0020) and `write-issue-spec` (0019) implementation
- **Merging `zoom-out` into architect role** — revisit at Chunk 5 grill