fix(kyberforge): align primitive-author and factory-audit rule tiers
Second clean-context audit found author Must/Should and audit FAIL/SUGGESTION tiers drifting apart, and author Musts the audit never checked. - factory-audit: FAIL on absolute or bare relative hook script paths, an applyTo present but empty, and unbalanced braces/brackets in applyTo; judgment steps for dependency stem collisions, helper .json in hook dirs, unresolvable instruction links, prompt model slugs and second-person bodies; an unmatched glob drops to SUGGESTION; deliberate tier deviations recorded in hook-flow.md; validate.sh --help lists the three new modes; DescriptionOpener message no longer prescribes "Use when". - primitive-author: deprecated routing, extra prompt keys and the prompt description contract become Shoulds; hook Musts gain "contributes an entry", no bare relative paths, and executable-when-run-directly; prompt Must 1 covers hardlinks; Vale prose FAILs resolved at close. - forge: say "hook, instruction or prompt" rather than "apm primitive". Refs #94 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KkT7RSDwDbmrM9T34b6sTi
This commit is contained in:
@@ -1,5 +1,5 @@
|
||||
extends: existence
|
||||
message: "Description opens with '%s' — use an imperative 'Use when...' opener instead"
|
||||
message: "Description opens with '%s' — lead with the action or trigger, not 'This'"
|
||||
level: error
|
||||
scope: text.frontmatter.description
|
||||
ignorecase: true
|
||||
|
||||
@@ -22,10 +22,16 @@ Resolve the path against this skill's own directory. Run exactly:
|
||||
bash scripts/validate.sh <hook-file>
|
||||
```
|
||||
|
||||
Its findings become the `### Structure` dimension, FAILs and SUGGESTIONs both, at the tier the script assigned: JSON validity, the wrapped-or-naked shape, event lists and nested handler lists (the checks whose failure makes the Copilot install fail), event names that never fire, referenced scripts that are missing, outside the package or not executable, deprecated filename routing, and `${CLAUDE_PLUGIN_ROOT}` where `${PLUGIN_ROOT}` would do. It exits **0** with no FAIL, **1** on real findings, **2** when it never ran — report that as `### Structure` unverified, quoting the stderr reason.
|
||||
Its findings become the `### Structure` dimension, FAILs and SUGGESTIONs both, at the tier the script assigned: JSON validity, the wrapped-or-naked shape, event lists and nested handler lists (the checks whose failure makes the Copilot install fail), a file contributing no entries, event names that never fire, referenced scripts that are missing, outside the package, not executable when run directly, or referenced by an absolute or bare relative path apm will not bundle, deprecated filename routing, and `${CLAUDE_PLUGIN_ROOT}` where `${PLUGIN_ROOT}` would do. It exits **0** with no FAIL, **1** on real findings, **2** when it never ran — report that as `### Structure` unverified, quoting the stderr reason.
|
||||
|
||||
There is no provenance and no Vale step: a hook carries no `source_keys` and no prose.
|
||||
|
||||
Three tiers deliberately differ from `primitive-author`'s checklist or the research's. Do not re-tier them by judgment:
|
||||
|
||||
- A hook file directly under a package-root `hooks/` passes. apm discovers both `.apm/hooks/` and `hooks/`, and this audit may target a third-party package; `primitive-author` authors only in `.apm/hooks/`.
|
||||
- Deprecated filename routing is a SUGGESTION, matching the author's Should: the research allows it when deprecated routing is intended.
|
||||
- A non-executable script run as the command's first token is a FAIL, stricter than the research's Should, because it fails every time it fires.
|
||||
|
||||
## Step 2 — Read the hook and its scripts
|
||||
|
||||
Read the hook file, every script it references, and the package's `apm.yml` `targets:` — reach is narrowed there, never in the hook file.
|
||||
@@ -46,6 +52,7 @@ Cite file and line for every finding.
|
||||
- SUGGESTION: a PascalCase event name that is not a real Claude Code event (a misspelling deploys verbatim and never fires; the script cannot tell a typo from an event it does not know).
|
||||
- SUGGESTION: `bash`/`powershell`/`timeoutSec` keys in a Claude-shaped file — they render, but leave stray keys in `settings.json`.
|
||||
- SUGGESTION: an unquoted script path that could contain spaces.
|
||||
- SUGGESTION: a helper `.json` file in the hook directory without a `hooks` key — Copilot's loader scans the bundled scripts directory and rejects it. Keep helper configuration non-JSON.
|
||||
|
||||
Then return to `SKILL.md` Step 4, opening the report with this coverage line:
|
||||
|
||||
|
||||
@@ -23,7 +23,7 @@ bash scripts/validate.sh <instruction-file>
|
||||
bash scripts/vale-wrap.sh <instruction-file>
|
||||
```
|
||||
|
||||
`validate.sh` findings become the `### Structure` dimension, FAILs and SUGGESTIONs both, at the tier the script assigned. It exits **0** with no FAIL, **1** on real findings, **2** when it never ran — report that as `### Structure` unverified, quoting the stderr reason.
|
||||
`validate.sh` findings become the `### Structure` dimension, FAILs and SUGGESTIONs both, at the tier the script assigned: path, frontmatter, `description`, body, an `applyTo` that is present but empty or has unbalanced braces or brackets, a missing or list-form `applyTo`, extra keys, and a stem duplicated at the package root. It exits **0** with no FAIL, **1** on real findings, **2** when it never ran — report that as `### Structure` unverified, quoting the stderr reason.
|
||||
|
||||
`vale-wrap.sh` applies the bundled `Kyberforge` style. Every alert is a FAIL under `### Prose`, cited by rule ID; do not re-derive it by judgment. `0 files` scanned means NOT RUN, not clean — say so and judge prose by reading.
|
||||
|
||||
@@ -31,7 +31,7 @@ There is no provenance step: an instruction carries no `source_keys`.
|
||||
|
||||
## Step 2 — Read the instruction and its context
|
||||
|
||||
Read the file, the package's `apm.yml`, and the repo's root `AGENTS.md`. For a scoped file, list the tracked files its `applyTo` matches (`rtk git ls-files` filtered by the glob).
|
||||
Read the file, the package's `apm.yml`, and the repo's root `AGENTS.md`. For a scoped file, list the tracked files its `applyTo` matches (`rtk git ls-files` filtered by the glob). List the instruction stems the installed dependencies ship (`apm_modules/**/.apm/instructions/*.instructions.md`) — the script checks only the package root for a duplicate.
|
||||
|
||||
## Step 3 — Qualitative audit
|
||||
|
||||
@@ -40,13 +40,15 @@ Cite file and line for every finding.
|
||||
**scope** — an instruction applies when files matching `applyTo` are touched; with no `applyTo` it loads into every session of every repo that installs the package.
|
||||
|
||||
- FAIL: an always-on file whose content is a rule for this repo alone — it belongs in `AGENTS.md`, which is the repo's single always-on source, not in a package that ships it to every consumer.
|
||||
- FAIL: an `applyTo` glob that matches no tracked file in any repo the package plausibly targets, so the rule never loads.
|
||||
- FAIL: the stem matches an instruction an installed dependency ships — both deploy to `.claude/rules/<stem>.md`, and one silently overwrites the other.
|
||||
- SUGGESTION: an `applyTo` glob that matches no tracked file here. It is legitimate for files the package's consumers have and this repo does not, so name the mismatch rather than failing it.
|
||||
- SUGGESTION: an always-on file whose content is really file-type specific — narrow it with `applyTo`.
|
||||
- SUGGESTION: a glob much broader than the content (`**` for a rule about Python).
|
||||
|
||||
**description**
|
||||
|
||||
- SUGGESTION: the description does not say what the rule covers, or contradicts the body. Any rationale Claude readers need belongs in the body, because Claude drops the description.
|
||||
- SUGGESTION: a relative markdown link that does not resolve from the source file — apm rewrites links on deploy, and a broken one stays broken on every target.
|
||||
|
||||
Then return to `SKILL.md` Step 4, opening the report with this coverage line:
|
||||
|
||||
|
||||
@@ -23,7 +23,7 @@ bash scripts/validate.sh <prompt-file>
|
||||
bash scripts/vale-wrap.sh <prompt-file>
|
||||
```
|
||||
|
||||
`validate.sh` findings become the `### Structure` dimension, FAILs and SUGGESTIONs both, at the tier the script assigned: path and name, frontmatter, `description` presence, length and trigger clause, keys Claude drops, `input:` names and shapes, and `${input:x}` references against `input:`. It exits **0** with no FAIL, **1** on real findings, **2** when it never ran — report that as `### Structure` unverified, quoting the stderr reason.
|
||||
`validate.sh` findings become the `### Structure` dimension, FAILs and SUGGESTIONs both, at the tier the script assigned: path and name, frontmatter, `description` presence, length and trigger clause, keys Claude drops, `input:` names and shapes, and `${input:x}` references against `input:`. Keys Claude drops are a SUGGESTION, not a FAIL, on purpose: a Copilot-only key is legitimate when its Claude drop is intended, and only the author can say which. It exits **0** with no FAIL, **1** on real findings, **2** when it never ran — report that as `### Structure` unverified, quoting the stderr reason.
|
||||
|
||||
`vale-wrap.sh` applies the bundled `Kyberforge` style. Every alert is a FAIL under `### Prose`, cited by rule ID; do not re-derive it by judgment. `0 files` scanned means NOT RUN, not clean — say so and judge prose by reading.
|
||||
|
||||
@@ -43,6 +43,8 @@ Cite file and line for every finding.
|
||||
- FAIL: the body names a skill or agent that does not resolve, or one carrying `disable-model-invocation: true`, which the model cannot invoke.
|
||||
- SUGGESTION: borderline — some how-to detail beyond steering, but not a full procedure.
|
||||
- SUGGESTION: more than one intent in one prompt.
|
||||
- SUGGESTION: the body is not written as second-person instructions to the agent.
|
||||
- SUGGESTION: a `model` value that is not a model slug the package's Claude target accepts. Copilot ignores `model` and `allowed-tools`, so neither constrains a Copilot run.
|
||||
|
||||
**description**
|
||||
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
# lib-checks-primitive.sh — SOURCED, never executed.
|
||||
#
|
||||
# The structural check suite for the three apm primitives that have no author
|
||||
# skill of their own and no SKILL.md-shaped container: hooks
|
||||
# The structural check suite for the three apm primitives with no
|
||||
# SKILL.md-shaped container, all authored by primitive-author: hooks
|
||||
# (.apm/hooks/*.json), instructions (*.instructions.md) and prompts
|
||||
# (*.prompt.md). validate.sh detects which one it was handed from the path and
|
||||
# feeds $KYBERFORGE_PRIMITIVE_PY to python3 with the target as argv[1] and the
|
||||
@@ -13,9 +13,12 @@
|
||||
# description or body, and never validates a prompt's input: names against its
|
||||
# ${input:x} references — so `apm install` and `apm compile --validate` both exit
|
||||
# 0 on files that deploy nothing, or deploy something that never fires. The
|
||||
# checks and their tiers come from the Authoring checklists at the end of
|
||||
# checks follow the Authoring checklists at the end of
|
||||
# plugins/kyberforge/docs/research/docs/microsoft-apm/{hooks,instructions,prompt}-primitive-schema.md,
|
||||
# which trace each rule to the apm source that makes it matter.
|
||||
# which trace each rule to the apm source that makes it matter, except where
|
||||
# references/{hook,instruction,prompt}-flow.md documents a deliberate deviation
|
||||
# (a tier moved, or a check the research leaves audit-only). A Must in
|
||||
# primitive-author is a FAIL here, a Should a SUGGESTION.
|
||||
#
|
||||
# No boundary resolver and no word budgets: none of these files is routed on a
|
||||
# description the way a skill is. A prompt's description IS model-visible on
|
||||
@@ -204,6 +207,37 @@ def extract_script_refs(cmd):
|
||||
return refs
|
||||
|
||||
|
||||
SCRIPT_EXT_RE = re.compile(r'\.(?:sh|bash|zsh|py|js|mjs|cjs|ts|ps1|rb|pl)$', re.IGNORECASE)
|
||||
|
||||
|
||||
def first_token(cmd):
|
||||
m = re.match(r'\s*(["\']?)((?:\\.|[^\s"\'])*)\1', cmd)
|
||||
return m.group(2).replace('\\', '') if m else ''
|
||||
|
||||
|
||||
def check_unanchored_script(cmd, pkg_root, where):
|
||||
# apm rewrites and bundles only ${*_PLUGIN_ROOT}/... and ./... references;
|
||||
# a bare command (`npx foo`, `echo hi`) passes through untouched, which is
|
||||
# fine. An absolute script path, or a bare relative path to a file in the
|
||||
# package, also passes through untouched — so the script is not bundled
|
||||
# and the deployed hook points at a path that does not exist on the
|
||||
# consumer's machine.
|
||||
tok = first_token(cmd)
|
||||
if not tok or tok.startswith('./') or '$' in tok or tok.startswith('~'):
|
||||
return
|
||||
if tok.startswith('/'):
|
||||
real_root = os.path.realpath(pkg_root)
|
||||
inside = os.path.realpath(tok).startswith(real_root + os.sep)
|
||||
if inside or SCRIPT_EXT_RE.search(tok):
|
||||
fail(f"script '{tok}' is an absolute path — apm neither bundles nor rewrites it, so it breaks on every other machine; reference it as ${{PLUGIN_ROOT}}/<path> — {where}")
|
||||
return
|
||||
if '/' in tok:
|
||||
for base in (parent_dir, pkg_root):
|
||||
if os.path.isfile(os.path.join(base, tok)):
|
||||
fail(f"script '{tok}' is a bare relative path — apm bundles and rewrites only ${{PLUGIN_ROOT}}/... and ./... references, so this one deploys unbundled; prefix it with ${{PLUGIN_ROOT}}/ or ./ — {where}")
|
||||
return
|
||||
|
||||
|
||||
def check_script(kind_, rel, first, pkg_root, where):
|
||||
if not rel:
|
||||
return
|
||||
@@ -323,8 +357,11 @@ def audit_hook():
|
||||
continue
|
||||
if '${CLAUDE_PLUGIN_ROOT}' in cmd:
|
||||
uses_claude_token = True
|
||||
for kind_, rel, first in extract_script_refs(cmd):
|
||||
refs = extract_script_refs(cmd)
|
||||
for kind_, rel, first in refs:
|
||||
check_script(kind_, rel, first, pkg_root, where)
|
||||
if not any(first for _, _, first in refs):
|
||||
check_unanchored_script(cmd, pkg_root, where)
|
||||
|
||||
if uses_claude_token:
|
||||
suggest(f"uses ${{CLAUDE_PLUGIN_ROOT}} — apm documents the target-neutral ${{PLUGIN_ROOT}}, which it rewrites identically for every target — {fname}")
|
||||
@@ -337,6 +374,50 @@ def audit_hook():
|
||||
INSTRUCTION_KEYS = {'description', 'applyTo', 'author', 'version'}
|
||||
|
||||
|
||||
def split_top_level(value):
|
||||
# apm's parse_apply_to: split on commas outside {} (and not escaped \,),
|
||||
# strip each segment, drop empty ones.
|
||||
segs, cur, depth, i = [], '', 0, 0
|
||||
while i < len(value):
|
||||
c = value[i]
|
||||
if c == '\\' and i + 1 < len(value):
|
||||
cur += value[i:i + 2]
|
||||
i += 2
|
||||
continue
|
||||
if c == '{':
|
||||
depth += 1
|
||||
elif c == '}':
|
||||
depth -= 1
|
||||
if c == ',' and depth == 0:
|
||||
segs.append(cur)
|
||||
cur = ''
|
||||
else:
|
||||
cur += c
|
||||
i += 1
|
||||
segs.append(cur)
|
||||
return [s.strip() for s in segs if s.strip()]
|
||||
|
||||
|
||||
def check_apply_to(apply_to):
|
||||
if isinstance(apply_to, list):
|
||||
entries = [e for e in apply_to if e is not None and str(e).strip()]
|
||||
globs = [str(e).strip() for e in entries]
|
||||
elif isinstance(apply_to, str):
|
||||
globs = split_top_level(apply_to)
|
||||
else:
|
||||
fail(f"applyTo is neither a string nor a list — apm cannot read a glob from it — {fname}")
|
||||
return False
|
||||
if not globs:
|
||||
fail(f"applyTo is present but empty — remove the key for an intentionally always-on rule, or give it a glob — {fname}")
|
||||
return False
|
||||
ok = True
|
||||
for g in globs:
|
||||
if g.count('{') != g.count('}') or g.count('[') != g.count(']'):
|
||||
fail(f"applyTo glob '{g}' has unbalanced braces or brackets — it matches nothing, so the rule never fires — {fname}")
|
||||
ok = False
|
||||
return ok
|
||||
|
||||
|
||||
def audit_instruction():
|
||||
check_not_linked()
|
||||
stem = fname[:-len('.instructions.md')]
|
||||
@@ -359,9 +440,10 @@ def audit_instruction():
|
||||
fail(f"body is empty — apm deploys an empty rule without complaint — {fname}")
|
||||
|
||||
apply_to = fm.get('applyTo')
|
||||
if apply_to is None or apply_to == '' or apply_to == []:
|
||||
apply_to_ok = apply_to is not None and check_apply_to(apply_to)
|
||||
if apply_to is None:
|
||||
suggest(f"no applyTo — this loads into every session of every repo that installs the package; confirm always-on is intended, and that a rule for this repo alone is not really an AGENTS.md rule — {fname}")
|
||||
elif isinstance(apply_to, list):
|
||||
elif apply_to_ok and isinstance(apply_to, list):
|
||||
suggest(f"applyTo is a YAML list — Copilot receives the file verbatim and its handling of a list is unverified; use one comma-separated string — {fname}")
|
||||
|
||||
extra = sorted(k for k in fm if k not in INSTRUCTION_KEYS)
|
||||
|
||||
@@ -145,16 +145,20 @@ At project or user scope, <agent-file> is either half of a Claude Code .md /
|
||||
Copilot .agent.md pair.
|
||||
|
||||
Arguments:
|
||||
skill-dir Path to the skill directory containing SKILL.md.
|
||||
agent-file Path to the agent file (or either half of a project/user-scope pair).
|
||||
skill-dir Path to the skill directory containing SKILL.md.
|
||||
agent-file Path to the agent file (or either half of a project/user-scope pair).
|
||||
hook-file Path to a hook JSON file directly under .apm/hooks/ or hooks/.
|
||||
instruction-file Path to a *.instructions.md file.
|
||||
prompt-file Path to a *.prompt.md file.
|
||||
|
||||
Exit codes:
|
||||
0 All checks passed (may include SUGGESTIONs)
|
||||
1 One or more checks failed
|
||||
2 Nothing was audited (no argument, the target matches no shape, the
|
||||
target does not exist, an unrecognized file extension, a missing
|
||||
references/agent-field-inventory.md, or a missing or unreadable lib-*.sh
|
||||
beside this script)
|
||||
references/agent-field-inventory.md, a missing or unreadable lib-*.sh
|
||||
beside this script, or, for a hook, instruction or prompt, a missing
|
||||
python3 or PyYAML)
|
||||
EOF
|
||||
}
|
||||
|
||||
|
||||
@@ -173,6 +173,37 @@ teardown() {
|
||||
refute_output --partial "hardlink"
|
||||
}
|
||||
|
||||
@test "hook: an absolute script path is a FAIL; an absolute interpreter path is not" {
|
||||
write_hook hooks.json '{"hooks":{"Stop":[{"hooks":[{"type":"command","command":"/usr/local/bin/check.sh","timeout":5}]}]}}'
|
||||
run bash "$SCRIPT" "$PKG/.apm/hooks/hooks.json"
|
||||
assert_failure
|
||||
assert_output --partial "is an absolute path"
|
||||
|
||||
write_hook hooks.json '{"hooks":{"Stop":[{"hooks":[{"type":"command","command":"/usr/bin/env true","timeout":5}]}]}}'
|
||||
run bash "$SCRIPT" "$PKG/.apm/hooks/hooks.json"
|
||||
assert_success
|
||||
refute_output --partial "absolute path"
|
||||
}
|
||||
|
||||
@test "hook: a bare relative path to a package script is a FAIL; a bare command is not" {
|
||||
write_hook hooks.json '{"hooks":{"Stop":[{"hooks":[{"type":"command","command":".apm/hooks/scripts/check.sh","timeout":5}]}]}}'
|
||||
run bash "$SCRIPT" "$PKG/.apm/hooks/hooks.json"
|
||||
assert_failure
|
||||
assert_output --partial "is a bare relative path"
|
||||
|
||||
write_hook hooks.json '{"hooks":{"Stop":[{"hooks":[{"type":"command","command":"npx some-tool --check","timeout":5}]}]}}'
|
||||
run bash "$SCRIPT" "$PKG/.apm/hooks/hooks.json"
|
||||
assert_success
|
||||
refute_output --partial "bare relative path"
|
||||
}
|
||||
|
||||
@test "hook: a file contributing no entries is a FAIL" {
|
||||
write_hook hooks.json '{"hooks":{}}'
|
||||
run bash "$SCRIPT" "$PKG/.apm/hooks/hooks.json"
|
||||
assert_failure
|
||||
assert_output --partial "contributes no hook entries"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Instructions
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -226,6 +257,31 @@ name: python' 'body'
|
||||
assert_output --partial "frontmatter key(s) name"
|
||||
}
|
||||
|
||||
@test "instruction: an applyTo that is present but empty is a FAIL" {
|
||||
write_instruction empty 'description: x
|
||||
applyTo: ""' 'body'
|
||||
run bash "$SCRIPT" "$PKG/.apm/instructions/empty.instructions.md"
|
||||
assert_failure
|
||||
assert_output --partial "applyTo is present but empty"
|
||||
refute_output --partial "SUGGESTION no applyTo"
|
||||
}
|
||||
|
||||
@test "instruction: an applyTo glob with unbalanced braces is a FAIL" {
|
||||
write_instruction broken 'description: x
|
||||
applyTo: "**/*.{py"' 'body'
|
||||
run bash "$SCRIPT" "$PKG/.apm/instructions/broken.instructions.md"
|
||||
assert_failure
|
||||
assert_output --partial "unbalanced braces or brackets"
|
||||
}
|
||||
|
||||
@test "instruction: a top-level comma list with a brace group passes clean" {
|
||||
write_instruction multi 'description: x
|
||||
applyTo: "**/*.py, **/*.{pyi,pyx}"' 'body'
|
||||
run bash "$SCRIPT" "$PKG/.apm/instructions/multi.instructions.md"
|
||||
assert_success
|
||||
refute_output --partial "applyTo"
|
||||
}
|
||||
|
||||
@test "instruction: the same stem at the package root is a FAIL" {
|
||||
write_instruction python 'description: x
|
||||
applyTo: "**/*.py"' 'body'
|
||||
|
||||
Reference in New Issue
Block a user