fix(kyberforge): align the audit word ceiling and drop the fragile --config

Three divergences between what the audit skills claim and what the hooks
enforce, each of which fails silently rather than loudly:

- `skill-size-check.sh` blocked at 2900 words while `validate.sh` checked only
  the 500-line ceiling, so `/skill-audit` could report a skill ready to ship
  that the commit hook then rejected. `validate.sh` now checks the same pair on
  the same inclusive terms; the constants are duplicated with a comment naming
  the other file, because a plugin skill's scripts cannot read outside the
  plugin directory once installed to the cache.
- Both audit skills' Step 1 passed `--config assets/vale/.vale.ini`, which is
  redundant (the wrapper self-locates its sibling config) and fragile: an agent
  that resolves the script path against the skill directory but not the config
  path gets E100, exit 2, which the surrounding fallback clause misreads as
  "vale unavailable" and downgrades to full LLM judgment with no signal.
- The external-consumer test registered only the two Vale hooks, never the
  third shipped hook, so a lost executable bit would have broken every consumer
  while the local suite stayed green. Verified by mutation: `chmod 644` on the
  copied script now turns three passes into two failures.

Also corrects the size hook's calibration comment, which claimed ~5.7-6.5
characters per word against a corpus whose measured median is 6.79 — the stated
upper bound sat below the median, so the "calibrated with margin" claim was
inverted for prose-dense files. MAX_WORDS is unchanged pending a decision; the
comment is now explicit that the gate holds under 5,000 tokens for typical
prose density, not for any file.

Refs: #85
This commit is contained in:
2026-08-09 15:44:16 +00:00
parent aa8cc22695
commit 4ae2429840
6 changed files with 85 additions and 22 deletions

View File

@@ -1,6 +1,8 @@
#!/usr/bin/env bash
# Regression test for scripts/skill-size-check.sh: enforces agentskills.io's
# 500-line/5,000-word(proxy-for-token) SKILL.md size ceiling.
# 500-line/5,000-token SKILL.md size ceiling. The token half is enforced via a
# word-count proxy (MAX_WORDS, currently 2900) — 5,000 is the token ceiling,
# 2,900 is the word budget the script derives from it.
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
@@ -55,9 +57,9 @@ echo ""
echo "--- fails a file over the word-count limit ---"
MANY_WORDS="$(make_fixture many-words 10 600)"
if "$SCRIPT" "$MANY_WORDS" 2>/dev/null; then
fail "file over the 5,000-word ceiling should have exited non-zero"
fail "file over the word ceiling should have exited non-zero"
else
pass "file over the 5,000-word ceiling exits non-zero"
pass "file over the word ceiling exits non-zero"
fi
# Boundary-pair tests below read the script's current MAX_WORDS rather than

View File

@@ -47,8 +47,11 @@ git -C "$HOOK_REPO" add -A
git -C "$HOOK_REPO" -c user.email=test@example.invalid -c user.name=test commit -qm "hook repo"
HOOK_REV="$(git -C "$HOOK_REPO" rev-parse HEAD)"
# The two Vale hooks scope by filename, so the consumer needs one file of each
# shape: a hook with nothing to match reports `Skipped` and proves nothing.
# Every hook scopes by filename, so the consumer needs one file of each shape:
# a hook with nothing to match reports `Skipped` and proves nothing. All three
# hooks .pre-commit-hooks.yaml ships are registered — an unregistered one would
# let a regression (a lost `100755` bit, a bad entry path) reach every external
# consumer while this repo's own `repo: local` runs stayed green.
mkdir -p "$CONSUMER/skills/demo" "$CONSUMER/agents"
git -C "$CONSUMER" init -q
cat > "$CONSUMER/.pre-commit-config.yaml" <<EOF
@@ -58,6 +61,7 @@ repos:
hooks:
- id: kyberforge-vale-audit-skill
- id: kyberforge-vale-audit-agent
- id: kyberforge-skill-size-check
EOF
write_fixtures() {
@@ -109,19 +113,49 @@ fi
# --- 2. Clean files pass — the hooks gate, they don't just always fail ---
echo ""
echo "--- both Vale hooks pass clean files in an external consumer repo ---"
echo "--- all three hooks pass clean files in an external consumer repo ---"
write_fixtures "of the packaged hook contract"
set +e
(cd "$CONSUMER" && pre-commit run --all-files > "$WORK/clean.log" 2>&1)
RC_CLEAN=$?
set -e
if [[ $RC_CLEAN -eq 0 ]]; then
pass "both hooks exit 0 on clean files"
if grep -q "Skipped" "$WORK/clean.log"; then
fail "a hook matched no files on the clean run, so it proved nothing"
sed 's/^/ /' "$WORK/clean.log"
elif [[ $RC_CLEAN -eq 0 ]]; then
pass "all three hooks exit 0 on clean files"
else
fail "hooks failed on clean files (rc=$RC_CLEAN)"
sed 's/^/ /' "$WORK/clean.log"
fi
# --- 3. The size hook gates too. It ran clean above, which is what proves it
# is executable and its entry path resolves; this half proves it still fails a
# file that breaks the ceiling rather than passing everything. ---
echo ""
echo "--- kyberforge-skill-size-check fails an oversized SKILL.md in an external consumer repo ---"
mkdir -p "$CONSUMER/skills/oversized"
{
echo "---"
echo "name: oversized"
echo "description: Use when the caller wants an oversized fixture."
echo "---"
for ((i = 1; i <= 600; i++)); do
echo "word"
done
} > "$CONSUMER/skills/oversized/SKILL.md"
git -C "$CONSUMER" add -A
set +e
(cd "$CONSUMER" && pre-commit run kyberforge-skill-size-check --all-files > "$WORK/size.log" 2>&1)
RC_SIZE=$?
set -e
if [[ $RC_SIZE -ne 0 ]] && grep -q "500-line ceiling" "$WORK/size.log"; then
pass "kyberforge-skill-size-check exits non-zero and names the ceiling it broke"
else
fail "kyberforge-skill-size-check did not gate an oversized SKILL.md (rc=$RC_SIZE)"
sed 's/^/ /' "$WORK/size.log"
fi
echo ""
echo "Results: $PASS passed, $FAIL failed"
[[ $FAIL -eq 0 ]]