Files

Defame1297 70210d6a7e feat(factory-audit): audit hooks, instructions and prompts

factory-audit gains three Step 0 rows and flows for the apm primitives
that have no container of their own: a .json file under hooks/, a
*.instructions.md and a *.prompt.md. apm validates almost none of them
(invalid hook JSON is skipped silently, instruction validate() only
warns, input: names are never checked against ${input:x}), so the
deterministic checks live in a new scripts/lib-checks-primitive.sh,
wired into validate.sh's path-shape detection. Each check and tier
traces to the Authoring checklists in the microsoft-apm research docs.

- Hook: JSON/shape/event-list checks mirroring the Copilot payload
  validator, never-firing event casing, missing/escaping/non-executable
  scripts (FAIL); deprecated filename routing and ${CLAUDE_PLUGIN_ROOT}
  (SUGGESTION).
- Instruction: location, frontmatter, description, body, stem clash
  (FAIL); missing or list applyTo and unread keys (SUGGESTION).
- Prompt: location/name, frontmatter, description, input names, the
  upstream `- name: x` docs bug, declared-vs-used ${input:x} (FAIL);
  ADR-0029 description length and trigger clause, dropped keys,
  camelCase aliases, argument-hint with input (SUGGESTION). Whether a
  prompt carries procedure is judgment in prompt-flow.md, not a script
  heuristic.

Vale now lints *.instructions.md and *.prompt.md with the Kyberforge
style; test-vale-wrap.sh gains their probe rows. New
tests/validate-primitive.bats (31 cases). kyberforge 2.0.1 -> 2.1.0 with
the executables.allow key, catalog 0.5.1 -> 0.5.2, marketplace.json
regenerated.

Refs #94

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KkT7RSDwDbmrM9T34b6sTi

2026-09-28 17:02:04 +00:00

4.4 KiB

Raw Blame History

tests/

Test files for scripts bundled with this skill.

When to add tests

Add tests here when the skill has scripts in scripts/ that are complex enough to break silently — validators, parsers, generators, anything with branching logic or edge cases. Test infrastructure (.bats, *_test.*, test_*.sh) belongs here, not in scripts/.

Dependencies

Tests require bats-support and bats-assert. The test files load helpers from the repo root's tests/test_helper/.

From the repo root:

git clone https://github.com/bats-core/bats-support tests/test_helper/bats-support
git clone https://github.com/bats-core/bats-assert tests/test_helper/bats-assert

Run all tests for this skill (from the repo root):

bats plugins/kyberforge/.apm/skills/factory-audit/tests/

Files

File	Purpose
`validate-skill.bats`	`scripts/validate.sh` against skill directories
`validate-agent.bats`	`scripts/validate.sh` against agent files
`validate-provenance-skill.bats`	`scripts/validate-provenance.sh` against skill directories
`validate-provenance-agent.bats`	`scripts/validate-provenance.sh` against agent files
`validate-primitive.bats`	`scripts/validate.sh` against apm hook, instruction and prompt files

Two scripts, five suites

factory-audit merges what were two skills — skill-audit and agent-audit — each of which shipped its own validate.sh and validate-provenance.sh. The merged skill has one of each. Every suite here invokes one of those two scripts; the four skill and agent files are two scripts × two artifact types, not four scripts. validate-primitive.bats is a fifth suite over the same scripts/validate.sh, for hooks, instructions and prompts, which have no provenance mode and so no provenance suite.

validate-skill.bats and validate-agent.bats run the same scripts/validate.sh and differ only in the fixtures they point it at. The two provenance suites stand in the same relation to scripts/validate-provenance.sh. Do not add a third script path here on the assumption that a differently named suite must mean a differently named script.

Auto-detection is pinned across the pair

Each entry point decides for itself what it was handed. ADR-0025 states the rule: a directory containing SKILL.md takes the skill flow; an .agent.md file, or a file under a directory named agents/, takes the agent flow. Anything else is rejected rather than guessed at. That behaviour is new with the merge — before it, each script was hard-wired to one artifact type and nothing about classification could be wrong — so it is asserted from both sides rather than in one place:

the skill-side suites pin the skill-directory classification and the neither-shape rejection,
the agent-side suites pin the two agent rules separately — .agent.md in a directory that is not agents/, and a plain .md under .apm/agents/ — so that a detector implementing only one of them cannot pass both. Plus a control asserting an agent file never picks up a skill-only gate.

Both skill-side suites additionally pin the SKILL.md file path, not just the directory: a pre-commit files: hook matches files, so every hook-driven invocation hands over a SKILL.md path. Each entry point resolves it to the directory, and the suites assert the two spellings produce the identical verdict rather than merely that the file spelling survives.

A misclassification is silent and total — the wrong rubric runs end to end and reports the artifact clean against gates that never applied to it — and no other fixture in these suites would notice, because every other fixture is already the shape its own suite expects.

The two provenance exit contracts are different on purpose

scripts/validate-provenance.sh does not behave identically in its two modes, and the difference is deliberate:

Mode	Successful run
skill	may exit 0 with output — INFO findings print, status stays 0
agent	exits 0 and prints nothing

Each half is asserted from its own side, by the exit contract: test in each provenance suite. Both are asserted on purpose: a merge that collapsed one contract into the other would still satisfy whichever side was left unasserted, so a single-sided pin would go green on exactly the defect it exists to catch.

4.4 KiB Raw Blame History Unescape Escape