Files

Defame1297 70210d6a7e feat(factory-audit): audit hooks, instructions and prompts

factory-audit gains three Step 0 rows and flows for the apm primitives
that have no container of their own: a .json file under hooks/, a
*.instructions.md and a *.prompt.md. apm validates almost none of them
(invalid hook JSON is skipped silently, instruction validate() only
warns, input: names are never checked against ${input:x}), so the
deterministic checks live in a new scripts/lib-checks-primitive.sh,
wired into validate.sh's path-shape detection. Each check and tier
traces to the Authoring checklists in the microsoft-apm research docs.

- Hook: JSON/shape/event-list checks mirroring the Copilot payload
  validator, never-firing event casing, missing/escaping/non-executable
  scripts (FAIL); deprecated filename routing and ${CLAUDE_PLUGIN_ROOT}
  (SUGGESTION).
- Instruction: location, frontmatter, description, body, stem clash
  (FAIL); missing or list applyTo and unread keys (SUGGESTION).
- Prompt: location/name, frontmatter, description, input names, the
  upstream `- name: x` docs bug, declared-vs-used ${input:x} (FAIL);
  ADR-0029 description length and trigger clause, dropped keys,
  camelCase aliases, argument-hint with input (SUGGESTION). Whether a
  prompt carries procedure is judgment in prompt-flow.md, not a script
  heuristic.

Vale now lints *.instructions.md and *.prompt.md with the Kyberforge
style; test-vale-wrap.sh gains their probe rows. New
tests/validate-primitive.bats (31 cases). kyberforge 2.0.1 -> 2.1.0 with
the executables.allow key, catalog 0.5.1 -> 0.5.2, marketplace.json
regenerated.

Refs #94

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KkT7RSDwDbmrM9T34b6sTi

2026-09-28 17:02:04 +00:00

3.4 KiB

Raw Blame History

source_keys

apm-cli-installed-source

apm-docs-llms-full

Hook Flow

Steps 1 to 3 for an apm hook — the target Step 0 matched as a .json file directly under a hooks/ directory. Work them in order, then return to SKILL.md Step 4 to report.

Gotchas

apm checks almost nothing here. Invalid JSON is skipped without a word, an all-lowercase event deploys and never fires, and a missing script only warns — so apm install exiting 0 says nothing about whether the hook works. Never cite a clean install as evidence against a finding.
Copilot receiving a Claude-shaped file is not a finding. apm renders one source for every target and documents that it owns the per-target shape; whether Copilot CLI honours a nested entry or matcher is unverified upstream, not a defect in the file.

Step 1 — Deterministic checks

Resolve the path against this skill's own directory. Run exactly:

bash scripts/validate.sh <hook-file>

Its findings become the ### Structure dimension, FAILs and SUGGESTIONs both, at the tier the script assigned: JSON validity, the wrapped-or-naked shape, event lists and nested handler lists (the checks whose failure makes the Copilot install fail), event names that never fire, referenced scripts that are missing, outside the package or not executable, deprecated filename routing, and ${CLAUDE_PLUGIN_ROOT} where ${PLUGIN_ROOT} would do. It exits 0 with no FAIL, 1 on real findings, 2 when it never ran — report that as ### Structure unverified, quoting the stderr reason.

There is no provenance and no Vale step: a hook carries no source_keys and no prose.

Step 2 — Read the hook and its scripts

Read the hook file, every script it references, and the package's apm.yml targets: — reach is narrowed there, never in the hook file.

Step 3 — Qualitative audit

Cite file and line for every finding.

purpose — apm's own rule is to reach for a skill, instruction or prompt first; a hook is for "this must always happen at this event".

FAIL: the script carries procedure the agent should follow — instructions printed to the model, a multi-step workflow — rather than a runtime callback. That is a skill.
SUGGESTION: the behaviour is harness-specific (a Claude-only event, a Claude-only matcher value) in a package whose targets: includes other harnesses, and nothing records that the other targets receiving it was accepted. The apm-native fix is a separate package with its own targets:, not a routing filename.

handlers — the research checklist's Should and audit-only items, which apm never checks:

SUGGESTION: a handler without "type": "command" or an explicit numeric timeout in seconds.
SUGGESTION: a tool event (PreToolUse, PostToolUse) or SessionStart with no matcher — Claude receives "*". A matcher on an event Claude ignores it for (Stop, UserPromptSubmit) is inert, not wrong.
SUGGESTION: a PascalCase event name that is not a real Claude Code event (a misspelling deploys verbatim and never fires; the script cannot tell a typo from an event it does not know).
SUGGESTION: bash/powershell/timeoutSec keys in a Claude-shaped file — they render, but leave stray keys in settings.json.
SUGGESTION: an unquoted script path that could contain spaces.

Then return to SKILL.md Step 4, opening the report with this coverage line:

Checked: structure · purpose · handlers

3.4 KiB Raw Blame History