feat(factory-audit): audit hooks, instructions and prompts
factory-audit gains three Step 0 rows and flows for the apm primitives
that have no container of their own: a .json file under hooks/, a
*.instructions.md and a *.prompt.md. apm validates almost none of them
(invalid hook JSON is skipped silently, instruction validate() only
warns, input: names are never checked against ${input:x}), so the
deterministic checks live in a new scripts/lib-checks-primitive.sh,
wired into validate.sh's path-shape detection. Each check and tier
traces to the Authoring checklists in the microsoft-apm research docs.
- Hook: JSON/shape/event-list checks mirroring the Copilot payload
validator, never-firing event casing, missing/escaping/non-executable
scripts (FAIL); deprecated filename routing and ${CLAUDE_PLUGIN_ROOT}
(SUGGESTION).
- Instruction: location, frontmatter, description, body, stem clash
(FAIL); missing or list applyTo and unread keys (SUGGESTION).
- Prompt: location/name, frontmatter, description, input names, the
upstream `- name: x` docs bug, declared-vs-used ${input:x} (FAIL);
ADR-0029 description length and trigger clause, dropped keys,
camelCase aliases, argument-hint with input (SUGGESTION). Whether a
prompt carries procedure is judgment in prompt-flow.md, not a script
heuristic.
Vale now lints *.instructions.md and *.prompt.md with the Kyberforge
style; test-vale-wrap.sh gains their probe rows. New
tests/validate-primitive.bats (31 cases). kyberforge 2.0.1 -> 2.1.0 with
the executables.allow key, catalog 0.5.1 -> 0.5.2, marketplace.json
regenerated.
Refs #94
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KkT7RSDwDbmrM9T34b6sTi
This commit is contained in:
@@ -1,13 +1,13 @@
|
||||
---
|
||||
name: factory-audit
|
||||
description: >
|
||||
Use when the user wants a skill directory or agent definition audited,
|
||||
Use when a skill, agent, or apm hook, instruction or prompt needs auditing,
|
||||
including "is this ready to ship", or after hand-editing one outside its
|
||||
author skill. Not applying skill fixes -> skill-author. Not applying agent
|
||||
fixes -> agent-author.
|
||||
author skill. Not applying skill fixes -> skill-author.
|
||||
Not applying agent fixes -> agent-author.
|
||||
allowed-tools: Bash Read
|
||||
metadata:
|
||||
version: "1.0.5"
|
||||
version: "1.1.0"
|
||||
category: factory
|
||||
source_keys:
|
||||
- agentskills-home
|
||||
@@ -20,6 +20,8 @@ metadata:
|
||||
- claude-code-subagents-docs
|
||||
- context7-github-en-copilot
|
||||
- github-custom-agents-configuration
|
||||
- apm-cli-installed-source
|
||||
- apm-docs-llms-full
|
||||
---
|
||||
|
||||
## Gotchas
|
||||
@@ -30,21 +32,24 @@ metadata:
|
||||
|
||||
## Step 0 — Dispatch
|
||||
|
||||
Resolve the flow from the target path **before running anything**. The two flows run different validators over different dimension vocabularies, so dispatching after Step 1 means the wrong validator has already produced the wrong findings. The rows mirror the shapes `scripts/validate.sh` accepts; take the first that matches.
|
||||
Resolve the flow from the target path **before running anything**. The flows run different validators over different dimension vocabularies, so dispatching after Step 1 means the wrong validator has already produced the wrong findings. The rows mirror the shapes `scripts/validate.sh` accepts; take the first that matches.
|
||||
|
||||
| Target | Flow | Read |
|
||||
|---|---|---|
|
||||
| A directory containing `SKILL.md` | skill | `references/skill-flow.md` |
|
||||
| A file named `SKILL.md` — audit its parent directory | skill | `references/skill-flow.md` |
|
||||
| A file named `*.agent.md` | agent | `references/agent-flow.md` |
|
||||
| A file named `*.instructions.md` | instruction | `references/instruction-flow.md` |
|
||||
| A file named `*.prompt.md` | prompt | `references/prompt-flow.md` |
|
||||
| A `.md` file whose immediate parent directory is `agents/` (`.apm/agents`, `.claude/agents`, `.github/agents`, `.copilot/agents`) | agent | `references/agent-flow.md` |
|
||||
| A `.json` file whose immediate parent directory is `hooks/` (`.apm/hooks`, or a package's root `hooks/`) | hook | `references/hook-flow.md` |
|
||||
| Anything else — a missing path, a directory without `SKILL.md`, any other file | none | — |
|
||||
|
||||
Read only the file its row matched. Each carries Steps 1 to 3 — the deterministic checks, the read, and the qualitative audit — and is self-contained. Return here for Step 4.
|
||||
|
||||
On the last row, stop: run no validator and tell the user the two accepted shapes — a skill directory (or its `SKILL.md`), or an agent file (`*.agent.md`, or a `.md` directly under an `agents/` directory). Guessing a flow audits the path against the wrong spec.
|
||||
On the last row, stop: run no validator and tell the user the shapes the other rows accept. Guessing a flow audits the path against the wrong spec.
|
||||
|
||||
The scripts re-detect the flow from the path. If `validate.sh` reports on the other artifact type than your row, discard what you have and restart here — the flow file, not the script, picked your rubrics, coverage line and remediation line.
|
||||
The scripts re-detect the flow from the path. If `validate.sh` reports on a different artifact type than your row, discard what you have and restart here — the flow file, not the script, picked your rubrics, coverage line and remediation line.
|
||||
|
||||
## Step 4 — Report
|
||||
|
||||
@@ -64,6 +69,8 @@ Checked: structure · provider-safety · description · body · delegation · co
|
||||
|
||||
On the agent flow at plugin/APM scope, drop `pair-consistency` — there is no pair to check.
|
||||
|
||||
Hook, instruction and prompt flows: the line their flow file ends with.
|
||||
|
||||
Then output only the dimensions that have findings, grouped under H3 headings, FAILs before SUGGESTIONs within each. Omit clean dimensions — their absence is what confirms they passed.
|
||||
|
||||
Each finding:
|
||||
@@ -74,4 +81,4 @@ FAIL/SUGGESTION <finding> — file:line
|
||||
Fix: <exact change — quote before/after where applicable>
|
||||
```
|
||||
|
||||
Close with a `## Result` block holding one line: `PASS`, `PASS (N suggestions)`, or `FAIL (N fails · M suggestions)`, each optionally followed by ` · P info`. INFO findings are observational and never change PASS/FAIL; omit `· P info` when there are none. Add a second line whenever there is at least one finding — `Run skill-author to address findings.` on the skill flow, `Run agent-author to address findings.` on the agent flow. Do not apply fixes — report and propose only.
|
||||
Close with a `## Result` block holding one line: `PASS`, `PASS (N suggestions)`, or `FAIL (N fails · M suggestions)`, each optionally followed by ` · P info`. INFO findings are observational and never change PASS/FAIL; omit `· P info` when there are none. Add a second line whenever there is at least one finding — `Run skill-author to address findings.` on the skill flow, `Run agent-author to address findings.` on the agent flow, `Run primitive-author to address findings.` on the other three. Do not apply fixes — report and propose only.
|
||||
|
||||
Reference in New Issue
Block a user