fix(kyberforge): align the audit word ceiling and drop the fragile --config
Three divergences between what the audit skills claim and what the hooks enforce, each of which fails silently rather than loudly: - `skill-size-check.sh` blocked at 2900 words while `validate.sh` checked only the 500-line ceiling, so `/skill-audit` could report a skill ready to ship that the commit hook then rejected. `validate.sh` now checks the same pair on the same inclusive terms; the constants are duplicated with a comment naming the other file, because a plugin skill's scripts cannot read outside the plugin directory once installed to the cache. - Both audit skills' Step 1 passed `--config assets/vale/.vale.ini`, which is redundant (the wrapper self-locates its sibling config) and fragile: an agent that resolves the script path against the skill directory but not the config path gets E100, exit 2, which the surrounding fallback clause misreads as "vale unavailable" and downgrades to full LLM judgment with no signal. - The external-consumer test registered only the two Vale hooks, never the third shipped hook, so a lost executable bit would have broken every consumer while the local suite stayed green. Verified by mutation: `chmod 644` on the copied script now turns three passes into two failures. Also corrects the size hook's calibration comment, which claimed ~5.7-6.5 characters per word against a corpus whose measured median is 6.79 — the stated upper bound sat below the median, so the "calibrated with margin" claim was inverted for prose-dense files. MAX_WORDS is unchanged pending a decision; the comment is now explicit that the gate holds under 5,000 tokens for typical prose density, not for any file. Refs: #85
This commit is contained in:
@@ -36,12 +36,12 @@ metadata:
|
||||
```bash
|
||||
bash scripts/validate.sh <path-to-agent-file>
|
||||
bash scripts/validate-provenance.sh <path-to-agent-file>
|
||||
scripts/vale-wrap.sh --config assets/vale/.vale.ini <path-to-cc-file> <path-to-copilot-file>
|
||||
scripts/vale-wrap.sh <path-to-cc-file> <path-to-copilot-file>
|
||||
```
|
||||
|
||||
The script accepts either the CC file or the Copilot file. It detects provider from extension, derives the counterpart, and runs all structural checks. Note FAILs and SUGGESTIONs for the `### Structure` and `### Provider safety` report dimensions. Findings about missing fields, bad name format, empty body, or missing frontmatter → `### Structure`. Findings about CC-only fields in a Copilot file, Copilot-only fields in a CC file, plugin-silently-ignored fields, body length, or subagent-unavailable tools → `### Provider safety`. A missing counterpart file → `### Pair consistency`.
|
||||
|
||||
`vale-wrap.sh` and `.vale.ini` ship inside this skill's own `scripts/`/`assets/` — resolve them relative to this skill's directory the same way `scripts/validate.sh` is resolved above, so the invocation works whether this skill is running from this repo or from an installed plugin cache. Run it against both files of the pair (not just the one passed in). `Kyberforge` applies to both files; `KyberforgeCopilot` applies to the `.agent.md` file only, since its one rule (`Use proactively`) flags CC-specific phrasing that's meaningless in a Copilot description — there's nothing to flag in the CC file, so it isn't scoped there. Every Vale alert is a `FAIL` — all rules are graded `error` — so report each one in the `### Description` / `### Body` dimensions citing its rule ID (e.g. `KyberforgeCopilot.ProactivePhrase`). Skip and fall back to Step 2 judgment if vale or `.vale.ini` is unavailable. If Vale reports `0 files` scanned, treat the pass as NOT RUN — not as clean — and fall back to full Step 2 judgment for the dimensions it would have covered.
|
||||
`vale-wrap.sh` ships inside this skill's own `scripts/` — resolve it relative to this skill's directory the same way `scripts/validate.sh` is resolved above, so the invocation works whether this skill is running from this repo or from an installed plugin cache. Pass no `--config`: handed none, the wrapper loads its own sibling `assets/vale/.vale.ini`, located from the script's path rather than from the cwd. Adding an explicit relative `--config` breaks exactly the case the self-location covers — a resolved script path plus an unresolved config path yields `E100 Runtime error ... does not exist`, exit 2, which the fallback below then misreads as "vale unavailable". Run it against both files of the pair (not just the one passed in). `Kyberforge` applies to both files; `KyberforgeCopilot` applies to the `.agent.md` file only, since its one rule (`Use proactively`) flags CC-specific phrasing that's meaningless in a Copilot description — there's nothing to flag in the CC file, so it isn't scoped there. Every Vale alert is a `FAIL` — all rules are graded `error` — so report each one in the `### Description` / `### Body` dimensions citing its rule ID (e.g. `KyberforgeCopilot.ProactivePhrase`). Skip and fall back to Step 2 judgment if the `vale` binary is unavailable. If Vale reports `0 files` scanned, treat the pass as NOT RUN — not as clean — and fall back to full Step 2 judgment for the dimensions it would have covered.
|
||||
|
||||
`validate-provenance.sh` validates the provenance chain between the agent pair's `source_keys` and the plugin-scoped `sources.md` (plugin root — see ADR-0010). It exits 0 silently for non-plugin-scope agents and when no provenance data exists. Note FAILs from this script for the `### Provenance` dimension — surface them verbatim with Why and Fix.
|
||||
|
||||
|
||||
@@ -34,14 +34,14 @@ metadata:
|
||||
```bash
|
||||
bash scripts/validate.sh <skill-dir>
|
||||
bash scripts/validate-provenance.sh <skill-dir>
|
||||
scripts/vale-wrap.sh --config assets/vale/.vale.ini <skill-dir>/SKILL.md
|
||||
scripts/vale-wrap.sh <skill-dir>/SKILL.md
|
||||
```
|
||||
|
||||
Note any structural FAILs — they will appear in the report as a `### Structure` dimension. If the script cannot execute (python3 unavailable, Bash denied, or permission error), perform structural checks manually: name format, name matches directory, description length ≤1024 chars, SKILL.md ≤500 lines, no unfilled `FILL IN:` placeholders, scripts executable and free of interactive prompts.
|
||||
|
||||
Note any Provenance FAILs and INFO findings from `validate-provenance.sh` — they surface in the report as a `### Provenance` dimension (separate from `### Structure`). The script embeds full FAIL/INFO format with Why and Fix per finding; surface them verbatim.
|
||||
|
||||
`vale-wrap.sh` and `.vale.ini` ship inside this skill's own `scripts/`/`assets/` — resolve them relative to this skill's directory the same way `scripts/validate.sh` is resolved above, so the invocation works whether this skill is running from this repo or from an installed plugin cache. It applies `.vale.ini`'s `Kyberforge` style — a deterministic prefilter for a subset of the Description/Patterns/Body dimensions below, not a replacement for Step 3. Every Vale alert is a `FAIL` — all rules are graded `error` — so report each one citing its rule ID (e.g. `Kyberforge.DescriptionOpener`). Skip and fall back to Step 3 judgment if vale or `.vale.ini` is unavailable. If Vale reports `0 files` scanned, treat the pass as NOT RUN — not as clean — and fall back to full Step 3 judgment for the dimensions it would have covered.
|
||||
`vale-wrap.sh` ships inside this skill's own `scripts/` — resolve it relative to this skill's directory the same way `scripts/validate.sh` is resolved above, so the invocation works whether this skill is running from this repo or from an installed plugin cache. Pass no `--config`: handed none, the wrapper loads its own sibling `assets/vale/.vale.ini`, located from the script's path rather than from the cwd. Adding an explicit relative `--config` breaks exactly the case the self-location covers — a resolved script path plus an unresolved config path yields `E100 Runtime error ... does not exist`, exit 2, which the fallback below then misreads as "vale unavailable". It applies that config's `Kyberforge` style — a deterministic prefilter for a subset of the Description/Patterns/Body dimensions below, not a replacement for Step 3. Every Vale alert is a `FAIL` — all rules are graded `error` — so report each one citing its rule ID (e.g. `Kyberforge.DescriptionOpener`). Skip and fall back to Step 3 judgment if the `vale` binary is unavailable. If Vale reports `0 files` scanned, treat the pass as NOT RUN — not as clean — and fall back to full Step 3 judgment for the dimensions it would have covered.
|
||||
|
||||
## Step 2 — Read all skill files
|
||||
|
||||
|
||||
@@ -133,12 +133,28 @@ else:
|
||||
if desc:
|
||||
ok("description has no unfilled placeholders")
|
||||
|
||||
# SKILL.md line count
|
||||
# SKILL.md size ceilings (agentskills.io skill-authoring.md: 500 lines,
|
||||
# ~5,000 tokens). Both constants are DUPLICATED from the repo-root pre-commit
|
||||
# hook scripts/skill-size-check.sh — a plugin skill's scripts cannot read files
|
||||
# outside the plugin directory once the plugin is cache-installed, so there is
|
||||
# no single source to share. Keep the two in sync by hand: if they drift, this
|
||||
# audit will report a skill ready to ship that the commit hook then rejects.
|
||||
MAX_LINES = 500
|
||||
MAX_WORDS = 2900 # word-count proxy for the ~5,000-token ceiling
|
||||
|
||||
line_count = len(content.splitlines())
|
||||
if line_count <= 500:
|
||||
ok(f"SKILL.md line count {line_count} (limit: 500)")
|
||||
if line_count <= MAX_LINES:
|
||||
ok(f"SKILL.md line count {line_count} (limit: {MAX_LINES})")
|
||||
else:
|
||||
fail(f"SKILL.md line count {line_count} — exceeds 500-line limit")
|
||||
fail(f"SKILL.md line count {line_count} — exceeds {MAX_LINES}-line limit")
|
||||
|
||||
# str.split() with no argument splits on runs of whitespace, matching the
|
||||
# `wc -w` the hook uses, and counts the whole file including frontmatter.
|
||||
word_count = len(content.split())
|
||||
if word_count <= MAX_WORDS:
|
||||
ok(f"SKILL.md word count {word_count} (limit: {MAX_WORDS}, proxy for ~5,000 tokens)")
|
||||
else:
|
||||
fail(f"SKILL.md word count {word_count} — exceeds {MAX_WORDS}-word limit (proxy for ~5,000 tokens)")
|
||||
|
||||
# Body unfilled placeholders
|
||||
body = content[body_start:]
|
||||
|
||||
@@ -15,12 +15,23 @@ set -euo pipefail
|
||||
# audit and still be blocked by the commit hook.
|
||||
#
|
||||
# Token counts aren't computed exactly here — word count (`wc -w`) is used as
|
||||
# a proxy. This repo's own SKILL.md corpus measures ~5.7-6.5 characters per
|
||||
# word, which at the standard ~4-characters-per-token English approximation
|
||||
# works out to roughly 1.6-1.7 tokens per word. MAX_WORDS below is calibrated
|
||||
# from that measured ratio against the 5,000-token ceiling, with margin — it's
|
||||
# still a proxy, not exact BPE tokenization, but now grounded in actual repo
|
||||
# content rather than an unverified "conservative" assumption.
|
||||
# a proxy. Measured over this repo's 39 in-scope SKILL.md files, characters per
|
||||
# word runs min 5.97 / median 6.79 / mean 6.77 / max 7.22. At the standard
|
||||
# ~4-characters-per-token English approximation that is 1.49 / 1.70 / 1.69 /
|
||||
# 1.81 tokens per word.
|
||||
#
|
||||
# MAX_WORDS=2900 is therefore calibrated to the corpus MEDIAN, not to its worst
|
||||
# case, and carries no margin: 2900 x 1.70 = ~4,900 tokens for a median-density
|
||||
# file, but 2900 x 1.81 = ~5,240 tokens for the densest file in the corpus. So
|
||||
# what this gate actually guarantees is "under 5,000 tokens for a file of
|
||||
# typical prose density"; a prose-dense SKILL.md can sit at exactly MAX_WORDS
|
||||
# and still be a few hundred tokens over the agentskills.io ceiling. Holding
|
||||
# the worst observed ratio under 5,000 tokens would need MAX_WORDS ~2770.
|
||||
#
|
||||
# It is a one-sided proxy in the useful direction — nothing under the word
|
||||
# ceiling is wildly over the token ceiling — but it is not exact BPE
|
||||
# tokenization and does not replace one. Re-measure the corpus before treating
|
||||
# any of these numbers as still current.
|
||||
|
||||
MAX_LINES=500
|
||||
MAX_WORDS=2900
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regression test for scripts/skill-size-check.sh: enforces agentskills.io's
|
||||
# 500-line/5,000-word(proxy-for-token) SKILL.md size ceiling.
|
||||
# 500-line/5,000-token SKILL.md size ceiling. The token half is enforced via a
|
||||
# word-count proxy (MAX_WORDS, currently 2900) — 5,000 is the token ceiling,
|
||||
# 2,900 is the word budget the script derives from it.
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
@@ -55,9 +57,9 @@ echo ""
|
||||
echo "--- fails a file over the word-count limit ---"
|
||||
MANY_WORDS="$(make_fixture many-words 10 600)"
|
||||
if "$SCRIPT" "$MANY_WORDS" 2>/dev/null; then
|
||||
fail "file over the 5,000-word ceiling should have exited non-zero"
|
||||
fail "file over the word ceiling should have exited non-zero"
|
||||
else
|
||||
pass "file over the 5,000-word ceiling exits non-zero"
|
||||
pass "file over the word ceiling exits non-zero"
|
||||
fi
|
||||
|
||||
# Boundary-pair tests below read the script's current MAX_WORDS rather than
|
||||
|
||||
@@ -47,8 +47,11 @@ git -C "$HOOK_REPO" add -A
|
||||
git -C "$HOOK_REPO" -c user.email=test@example.invalid -c user.name=test commit -qm "hook repo"
|
||||
HOOK_REV="$(git -C "$HOOK_REPO" rev-parse HEAD)"
|
||||
|
||||
# The two Vale hooks scope by filename, so the consumer needs one file of each
|
||||
# shape: a hook with nothing to match reports `Skipped` and proves nothing.
|
||||
# Every hook scopes by filename, so the consumer needs one file of each shape:
|
||||
# a hook with nothing to match reports `Skipped` and proves nothing. All three
|
||||
# hooks .pre-commit-hooks.yaml ships are registered — an unregistered one would
|
||||
# let a regression (a lost `100755` bit, a bad entry path) reach every external
|
||||
# consumer while this repo's own `repo: local` runs stayed green.
|
||||
mkdir -p "$CONSUMER/skills/demo" "$CONSUMER/agents"
|
||||
git -C "$CONSUMER" init -q
|
||||
cat > "$CONSUMER/.pre-commit-config.yaml" <<EOF
|
||||
@@ -58,6 +61,7 @@ repos:
|
||||
hooks:
|
||||
- id: kyberforge-vale-audit-skill
|
||||
- id: kyberforge-vale-audit-agent
|
||||
- id: kyberforge-skill-size-check
|
||||
EOF
|
||||
|
||||
write_fixtures() {
|
||||
@@ -109,19 +113,49 @@ fi
|
||||
|
||||
# --- 2. Clean files pass — the hooks gate, they don't just always fail ---
|
||||
echo ""
|
||||
echo "--- both Vale hooks pass clean files in an external consumer repo ---"
|
||||
echo "--- all three hooks pass clean files in an external consumer repo ---"
|
||||
write_fixtures "of the packaged hook contract"
|
||||
set +e
|
||||
(cd "$CONSUMER" && pre-commit run --all-files > "$WORK/clean.log" 2>&1)
|
||||
RC_CLEAN=$?
|
||||
set -e
|
||||
if [[ $RC_CLEAN -eq 0 ]]; then
|
||||
pass "both hooks exit 0 on clean files"
|
||||
if grep -q "Skipped" "$WORK/clean.log"; then
|
||||
fail "a hook matched no files on the clean run, so it proved nothing"
|
||||
sed 's/^/ /' "$WORK/clean.log"
|
||||
elif [[ $RC_CLEAN -eq 0 ]]; then
|
||||
pass "all three hooks exit 0 on clean files"
|
||||
else
|
||||
fail "hooks failed on clean files (rc=$RC_CLEAN)"
|
||||
sed 's/^/ /' "$WORK/clean.log"
|
||||
fi
|
||||
|
||||
# --- 3. The size hook gates too. It ran clean above, which is what proves it
|
||||
# is executable and its entry path resolves; this half proves it still fails a
|
||||
# file that breaks the ceiling rather than passing everything. ---
|
||||
echo ""
|
||||
echo "--- kyberforge-skill-size-check fails an oversized SKILL.md in an external consumer repo ---"
|
||||
mkdir -p "$CONSUMER/skills/oversized"
|
||||
{
|
||||
echo "---"
|
||||
echo "name: oversized"
|
||||
echo "description: Use when the caller wants an oversized fixture."
|
||||
echo "---"
|
||||
for ((i = 1; i <= 600; i++)); do
|
||||
echo "word"
|
||||
done
|
||||
} > "$CONSUMER/skills/oversized/SKILL.md"
|
||||
git -C "$CONSUMER" add -A
|
||||
set +e
|
||||
(cd "$CONSUMER" && pre-commit run kyberforge-skill-size-check --all-files > "$WORK/size.log" 2>&1)
|
||||
RC_SIZE=$?
|
||||
set -e
|
||||
if [[ $RC_SIZE -ne 0 ]] && grep -q "500-line ceiling" "$WORK/size.log"; then
|
||||
pass "kyberforge-skill-size-check exits non-zero and names the ceiling it broke"
|
||||
else
|
||||
fail "kyberforge-skill-size-check did not gate an oversized SKILL.md (rc=$RC_SIZE)"
|
||||
sed 's/^/ /' "$WORK/size.log"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Results: $PASS passed, $FAIL failed"
|
||||
[[ $FAIL -eq 0 ]]
|
||||
|
||||
Reference in New Issue
Block a user