Files

Defame1297 70210d6a7e feat(factory-audit): audit hooks, instructions and prompts

factory-audit gains three Step 0 rows and flows for the apm primitives
that have no container of their own: a .json file under hooks/, a
*.instructions.md and a *.prompt.md. apm validates almost none of them
(invalid hook JSON is skipped silently, instruction validate() only
warns, input: names are never checked against ${input:x}), so the
deterministic checks live in a new scripts/lib-checks-primitive.sh,
wired into validate.sh's path-shape detection. Each check and tier
traces to the Authoring checklists in the microsoft-apm research docs.

- Hook: JSON/shape/event-list checks mirroring the Copilot payload
  validator, never-firing event casing, missing/escaping/non-executable
  scripts (FAIL); deprecated filename routing and ${CLAUDE_PLUGIN_ROOT}
  (SUGGESTION).
- Instruction: location, frontmatter, description, body, stem clash
  (FAIL); missing or list applyTo and unread keys (SUGGESTION).
- Prompt: location/name, frontmatter, description, input names, the
  upstream `- name: x` docs bug, declared-vs-used ${input:x} (FAIL);
  ADR-0029 description length and trigger clause, dropped keys,
  camelCase aliases, argument-hint with input (SUGGESTION). Whether a
  prompt carries procedure is judgment in prompt-flow.md, not a script
  heuristic.

Vale now lints *.instructions.md and *.prompt.md with the Kyberforge
style; test-vale-wrap.sh gains their probe rows. New
tests/validate-primitive.bats (31 cases). kyberforge 2.0.1 -> 2.1.0 with
the executables.allow key, catalog 0.5.1 -> 0.5.2, marketplace.json
regenerated.

Refs #94

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KkT7RSDwDbmrM9T34b6sTi

2026-09-28 17:02:04 +00:00

3.2 KiB

Raw Blame History

source_keys

apm-cli-installed-source

apm-docs-llms-full

Prompt Flow

Steps 1 to 3 for an apm prompt — the target Step 0 matched as a *.prompt.md file. Work them in order, then return to SKILL.md Step 4 to report.

Gotchas

A prompt is judged against ADR-0029, not against apm's framing. apm calls a prompt "a callable program"; this repo holds it to a single-intent, user-triggered message that steers existing skills or agents by name and carries no procedure of its own.
A prompt's description is not a skill description. It is one plain user-facing sentence with no "Use when" trigger clause and no boundary clause — so never raise a missing trigger or boundary as a finding. A Vale Kyberforge.DescriptionOpener alert here means rewrite it as an imperative action ("Review the current PR with …"), not add a trigger.

Step 1 — Deterministic checks

Resolve both paths against this skill's own directory. Run exactly:

bash scripts/validate.sh <prompt-file>
bash scripts/vale-wrap.sh <prompt-file>

validate.sh findings become the ### Structure dimension, FAILs and SUGGESTIONs both, at the tier the script assigned: path and name, frontmatter, description presence, length and trigger clause, keys Claude drops, input: names and shapes, and ${input:x} references against input:. It exits 0 with no FAIL, 1 on real findings, 2 when it never ran — report that as ### Structure unverified, quoting the stderr reason.

vale-wrap.sh applies the bundled Kyberforge style. Every alert is a FAIL under ### Prose, cited by rule ID; do not re-derive it by judgment. 0 files scanned means NOT RUN, not clean — say so and judge prose by reading.

There is no provenance step: a prompt carries no source_keys.

Step 2 — Read the prompt and what it steers

Read the file end to end, then the description of every skill or agent its body names, and confirm each resolves in this repo or in a package the prompt's package declares.

Step 3 — Qualitative audit

Cite file and line for every finding.

role — whether this is a prompt at all. Decide it by reading the body, not by its length or headings; there is no threshold.

FAIL: the body clearly carries reusable procedure — steps, gotchas, domain know-how the agent could not act without — rather than steering skills or agents that hold it. Fix: move the procedure into a skill (new, or the one it belongs to) and reduce the prompt to the message that invokes it.
FAIL: the body names a skill or agent that does not resolve, or one carrying disable-model-invocation: true, which the model cannot invoke.
SUGGESTION: borderline — some how-to detail beyond steering, but not a full procedure.
SUGGESTION: more than one intent in one prompt.

description

SUGGESTION: the description does not read as one user-facing action, or does not name the skills the prompt steers. On Claude the description is model-visible and apm drops disable-model-invocation, so naming the steered skills keeps the router pointed at the capability rather than the wrapper.

Then return to SKILL.md Step 4, opening the report with this coverage line:

Checked: structure · prose · role · description

3.2 KiB Raw Blame History