fix(lint): attribute Vale alerts per hook and cover .vale.ini in the sync check

The external-consumer test asserted a combined alert count (>=2) across both
shipped Vale hooks, but the SKILL.md fixture alone raises two alerts — so one
working hook satisfied the threshold. Retargeting agent-audit's glob to match
nothing left the suite reporting "3 passed" under the message "both hooks
flatten and flag". The Skipped guard does not catch this: the hook still
matches the file, Vale lints nothing, reports 0 errors in 1 file and exits 0,
which pre-commit renders as Passed. An assertion aggregating over N subjects
proves nothing about any individual subject.

Each hook now runs individually and its alerts are attributed to the nearest
preceding path header, so an alert is checked by path rather than by presence
in the combined blob. The two fixtures carry distinct VagueWording tokens, so
one hook's alert cannot be credited to another.

Nothing in the repo read either .vale.ini — the sync check diffed only
vale-wrap.sh and styles/Kyberforge, so a one-line glob typo silently disabled
the prefilter for a whole file type. That was the enabling half of the same
defect. The check now asserts the shared lines both copies must carry
(StylesPath, a section naming Kyberforge as a whole word) without flagging
their intentional divergence, and probes each glob section by asking Vale
itself to lint a representative path. Regex-to-glob comparison was rejected as
it means reimplementing doublestar semantics in bash; a file-count dry-run was
rejected because a section whose glob matches but whose BasedOnStyles lost
Kyberforge reports "1 file" with no alerts and would pass it.

Every new assertion is bound to a failing case in both directions: breaking the
artifact fails the suite, and neutering the assertion fails exactly one case.
That reverse sweep exposed two assertions bound to no failing case at all, one
masked by a stronger check running first.

Refs: #85
This commit is contained in:
2026-08-09 17:23:10 +00:00
parent 57654c4b02
commit d25355077f
3 changed files with 395 additions and 34 deletions

View File

@@ -64,13 +64,18 @@ repos:
- id: kyberforge-skill-size-check
EOF
# The two fixtures carry DIFFERENT flagged tokens so an alert can never be
# credited to the hook that did not raise it. Both bodies land mid-sentence in a
# folded block scalar that still spans two physical lines, which is the
# flattening the wrapper exists to do.
write_fixtures() {
local body="$1"
local skill_body="$1"
local agent_body="${2:-$1}"
cat > "$CONSUMER/skills/demo/SKILL.md" <<EOF
---
name: demo
description: >
Use when the caller wants a demonstration skill $body across two
Use when the caller wants a demonstration skill $skill_body across two
physical lines of one folded block scalar.
---
@@ -80,7 +85,7 @@ EOF
---
name: demo
description: >
Use when the caller wants a demonstration agent $body across two
Use when the caller wants a demonstration agent $agent_body across two
physical lines of one folded block scalar.
---
@@ -89,27 +94,56 @@ EOF
git -C "$CONSUMER" add -A
}
run_hooks() {
(cd "$CONSUMER" && pre-commit run --all-files 2>&1) || true
# Vale prints each linted path as its own header line with that file's alerts
# indented beneath it, so an alert belongs to the nearest preceding path line.
# Reads a hook log on stdin and prints only the alert lines filed under `$1`.
# The `sed` strips vale's ANSI colouring, which it emits into pre-commit's pipe
# too, so the header lines compare as plain paths.
alerts_for() {
sed $'s/\033\\[[0-9;]*m//g' | awk -v want="$1" '
/^[^[:space:]].*\.md$/ { cur = $0; next }
/^[[:space:]]*[0-9]+:[0-9]+[[:space:]]/ { if (cur == want) print }
'
}
# --- 1. Both hooks resolve their config and actually gate on a bad file ---
# --- 1. Each Vale hook resolves its config and gates its own file shape ---
# Asserted per hook, against that hook's own fixture path and its own token. An
# aggregate alert count over both hooks' combined output does not prove this:
# one fixture description carries every flagged token, so ONE working hook
# already clears a `>= 2` threshold. And a hook whose .vale.ini globs match
# nothing reaches neither of the guards below — it still MATCHES the file via
# its `files:` regex, so pre-commit does not report `Skipped`; vale simply lints
# nothing, prints `0 errors ... in 1 file` and exits 0, and the hook shows
# `Passed`. Attribution is the only thing that catches it.
echo ""
echo "--- both Vale hooks run and fail a bad file in an external consumer repo ---"
write_fixtures "that helps with and utilize things"
OUT_BAD="$(run_hooks)"
if echo "$OUT_BAD" | grep -q "does not exist"; then
fail "hooks hard-errored on a path resolved against the consumer repo (E100) — the bug this test guards against"
echo "$OUT_BAD" | sed 's/^/ /'
elif echo "$OUT_BAD" | grep -q "Skipped"; then
fail "a hook matched no files, so it proved nothing"
echo "$OUT_BAD" | sed 's/^/ /'
elif [[ "$(echo "$OUT_BAD" | grep -c "VagueWording")" -ge 2 ]]; then
pass "both hooks flatten and flag the folded description in a consumer repo"
else
fail "hooks did not flag both fixtures"
echo "$OUT_BAD" | sed 's/^/ /'
fi
echo "--- each Vale hook flags its own fixture in an external consumer repo ---"
write_fixtures "that helps with things" "that will utilize things"
while IFS='|' read -r HOOK_ID FIXTURE TOKEN; do
[[ -n "$HOOK_ID" ]] || continue
LOG="$WORK/$HOOK_ID.log"
set +e
(cd "$CONSUMER" && pre-commit run "$HOOK_ID" --all-files > "$LOG" 2>&1)
RC_HOOK=$?
set -e
if grep -q "does not exist" "$LOG"; then
fail "$HOOK_ID hard-errored on a path resolved against the consumer repo (E100) — the bug this test guards against"
sed 's/^/ /' "$LOG"
elif grep -q "Skipped" "$LOG"; then
fail "$HOOK_ID matched no files, so it proved nothing"
sed 's/^/ /' "$LOG"
elif [[ $RC_HOOK -eq 0 ]]; then
fail "$HOOK_ID passed $FIXTURE despite its flagged '$TOKEN' — a .vale.ini glob matching nothing lints zero files and exits 0"
sed 's/^/ /' "$LOG"
elif alerts_for "$FIXTURE" < "$LOG" | grep -qF "'$TOKEN'"; then
pass "$HOOK_ID flattens $FIXTURE and flags its '$TOKEN' in a consumer repo"
else
fail "$HOOK_ID failed, but no alert quoting '$TOKEN' was filed under $FIXTURE"
sed 's/^/ /' "$LOG"
fi
done <<'EOF'
kyberforge-vale-audit-skill|skills/demo/SKILL.md|helps with
kyberforge-vale-audit-agent|agents/demo.md|utilize
EOF
# --- 2. Clean files pass — the hooks gate, they don't just always fail ---
echo ""