feat(factory-audit): audit hooks, instructions and prompts
factory-audit gains three Step 0 rows and flows for the apm primitives
that have no container of their own: a .json file under hooks/, a
*.instructions.md and a *.prompt.md. apm validates almost none of them
(invalid hook JSON is skipped silently, instruction validate() only
warns, input: names are never checked against ${input:x}), so the
deterministic checks live in a new scripts/lib-checks-primitive.sh,
wired into validate.sh's path-shape detection. Each check and tier
traces to the Authoring checklists in the microsoft-apm research docs.
- Hook: JSON/shape/event-list checks mirroring the Copilot payload
validator, never-firing event casing, missing/escaping/non-executable
scripts (FAIL); deprecated filename routing and ${CLAUDE_PLUGIN_ROOT}
(SUGGESTION).
- Instruction: location, frontmatter, description, body, stem clash
(FAIL); missing or list applyTo and unread keys (SUGGESTION).
- Prompt: location/name, frontmatter, description, input names, the
upstream `- name: x` docs bug, declared-vs-used ${input:x} (FAIL);
ADR-0029 description length and trigger clause, dropped keys,
camelCase aliases, argument-hint with input (SUGGESTION). Whether a
prompt carries procedure is judgment in prompt-flow.md, not a script
heuristic.
Vale now lints *.instructions.md and *.prompt.md with the Kyberforge
style; test-vale-wrap.sh gains their probe rows. New
tests/validate-primitive.bats (31 cases). kyberforge 2.0.1 -> 2.1.0 with
the executables.allow key, catalog 0.5.1 -> 0.5.2, marketplace.json
regenerated.
Refs #94
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KkT7RSDwDbmrM9T34b6sTi
This commit is contained in:
@@ -0,0 +1,55 @@
|
||||
---
|
||||
source_keys:
|
||||
- apm-cli-installed-source
|
||||
- apm-docs-llms-full
|
||||
---
|
||||
|
||||
# Prompt Flow
|
||||
|
||||
Steps 1 to 3 for an apm prompt — the target Step 0 matched as a `*.prompt.md` file. Work them in
|
||||
order, then return to `SKILL.md` Step 4 to report.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- A prompt is judged against ADR-0029, not against apm's framing. apm calls a prompt "a callable program"; this repo holds it to a single-intent, user-triggered message that steers existing skills or agents by name and carries no procedure of its own.
|
||||
- A prompt's description is not a skill description. It is one plain user-facing sentence with no "Use when" trigger clause and no boundary clause — so never raise a missing trigger or boundary as a finding. A Vale `Kyberforge.DescriptionOpener` alert here means rewrite it as an imperative action ("Review the current PR with …"), not add a trigger.
|
||||
|
||||
## Step 1 — Deterministic checks
|
||||
|
||||
Resolve both paths against this skill's own directory. Run exactly:
|
||||
|
||||
```bash
|
||||
bash scripts/validate.sh <prompt-file>
|
||||
bash scripts/vale-wrap.sh <prompt-file>
|
||||
```
|
||||
|
||||
`validate.sh` findings become the `### Structure` dimension, FAILs and SUGGESTIONs both, at the tier the script assigned: path and name, frontmatter, `description` presence, length and trigger clause, keys Claude drops, `input:` names and shapes, and `${input:x}` references against `input:`. It exits **0** with no FAIL, **1** on real findings, **2** when it never ran — report that as `### Structure` unverified, quoting the stderr reason.
|
||||
|
||||
`vale-wrap.sh` applies the bundled `Kyberforge` style. Every alert is a FAIL under `### Prose`, cited by rule ID; do not re-derive it by judgment. `0 files` scanned means NOT RUN, not clean — say so and judge prose by reading.
|
||||
|
||||
There is no provenance step: a prompt carries no `source_keys`.
|
||||
|
||||
## Step 2 — Read the prompt and what it steers
|
||||
|
||||
Read the file end to end, then the description of every skill or agent its body names, and confirm each resolves in this repo or in a package the prompt's package declares.
|
||||
|
||||
## Step 3 — Qualitative audit
|
||||
|
||||
Cite file and line for every finding.
|
||||
|
||||
**role** — whether this is a prompt at all. Decide it by reading the body, not by its length or headings; there is no threshold.
|
||||
|
||||
- FAIL: the body clearly carries reusable procedure — steps, gotchas, domain know-how the agent could not act without — rather than steering skills or agents that hold it. Fix: move the procedure into a skill (new, or the one it belongs to) and reduce the prompt to the message that invokes it.
|
||||
- FAIL: the body names a skill or agent that does not resolve, or one carrying `disable-model-invocation: true`, which the model cannot invoke.
|
||||
- SUGGESTION: borderline — some how-to detail beyond steering, but not a full procedure.
|
||||
- SUGGESTION: more than one intent in one prompt.
|
||||
|
||||
**description**
|
||||
|
||||
- SUGGESTION: the description does not read as one user-facing action, or does not name the skills the prompt steers. On Claude the description is model-visible and apm drops `disable-model-invocation`, so naming the steered skills keeps the router pointed at the capability rather than the wrapper.
|
||||
|
||||
Then return to `SKILL.md` Step 4, opening the report with this coverage line:
|
||||
|
||||
```text
|
||||
Checked: structure · prose · role · description
|
||||
```
|
||||
Reference in New Issue
Block a user