test: fail the gate when a suite is skipped or never reports

Three gaps left by the previous round, all the same shape: a gate reporting
success having verified less than it appears to.

run_bats() hard-failed on a missing or non-executable runner but never checked
that the runner produced anything. An empty, executable run-bats.sh exits 0,
and the dispatcher printed a green summary with 166 bats tests silently absent.
It now requires an "N tests, M failures" line with a non-zero count.

run-tests.sh's skip listing is swallowed by pre-commit on a pass, so on a
machine without vale three suites exited 77 and the pre-push gate went green
having run 14 of 17. The hook now invokes it as --strict, where a skip fails
and the error names each suite and the reason it skipped. An ad-hoc local run
still skips gracefully -- at pre-push a skip means a documented dependency is
missing, which is a setup error, not a legitimate state. Deliberately not wired
to the vale downgrade's env var: one flag must not disarm two gates.

BATS_FILE_FLOOR is replaced by an expectation derived from git ls-files. A
floor of 8 against a real count of 10 let two files and eleven tests disappear
green, and the number needed an edit whenever a plugin was added. The derived
set needs no number, and catches an addition as well as a removal -- a .bats
file staged into the index and deleted from disk is now demanded back.

The vale opt-out announced its downgrade to nobody: pre-commit prints nothing
for a passing hook, so the summary line AGENTS.md tells the reader to check was
unreachable in exactly the situation it exists for. The hook is now verbose.

Also corrects the PROBES_CHECKED guard, whose commit message described a state
that cannot occur -- the .vale.ini loop errs first. Its two reachable triggers,
a gutted probe heredoc and a probe row naming a missing directory, had no test;
they do now, each asserting the guard is the sole cause.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X7GvKuJfy2WrdBmUttV4DT
This commit is contained in:
2026-08-14 12:29:42 +00:00
parent 874bf06b18
commit aa15fc850c
7 changed files with 637 additions and 74 deletions

View File

@@ -77,6 +77,13 @@ fi
SKILL_INI="$SKILL_AUDIT/assets/vale/.vale.ini"
AGENT_INI="$AGENT_AUDIT/assets/vale/.vale.ini"
# Counted, not assumed. The summary line at the bottom used to hardcode `2
# .vale.ini file(s) checked` in both branches. That was true on any clean run --
# a missing or unreadable file errs below and the script never reaches the
# summary -- but the line's whole purpose is to say what this run actually
# inspected, and a constant says what the author expected. Nothing asserted it,
# so it would have survived becoming false.
INIS_CHECKED=0
for ini in "$SKILL_INI" "$AGENT_INI"; do
rel_ini="${ini#"$REPO_ROOT"/}"
# `-e`, not `-f`: a path that exists but is not a readable regular file (a
@@ -106,6 +113,9 @@ for ini in "$SKILL_INI" "$AGENT_INI"; do
err "$rel_ini exists but could not be read — none of its assertions could run, and an unreadable file cannot be distinguished from a clean one downstream"
continue
fi
# Counted here, past both `continue`s: the file exists and its bytes were
# readable, so every assertion below it really does run against it.
INIS_CHECKED=$((INIS_CHECKED + 1))
# StylesPath is resolved relative to the .vale.ini, which is the only reason
# the bundled styles are found from a consuming repo's clone prefix.
if ! grep -Eq '^[[:space:]]*StylesPath[[:space:]]*=[[:space:]]*styles[[:space:]]*$' "$ini"; then
@@ -348,13 +358,27 @@ agent-audit|.claude/agents/demo.md|hooks-only
agent-audit|copilot/demo.agent.md|hooks-only
EOF_PROBE
# Second floor, on the probe table rather than the directory paths: every probe
# `continue`s when its skill's `.vale.ini` is absent, so a relocation of
# `assets/vale/` alone — with both audit skill directories still in place, which
# the guard at the top of this file therefore cannot see — would skip the entire
# table and leave FAIL at 0. Zero probes checked is never a valid result.
# Second floor, on the probe TABLE rather than on the directory paths. Every row
# `continue`s when the `.vale.ini` of the skill its first column names is absent,
# so the table can verify nothing while FAIL stays 0. Two states do that, and no
# other assertion in this file sees either:
#
# * the `EOF_PROBE` heredoc gutted — a bad merge, a truncated edit, or a
# wholesale delete of the rows. The loop body never runs at all.
# * every row's skill column drifting away from the directory names on disk
# (`skill-audit|` -> `skill-auditX|`), which is what a skill rename plus a
# half-applied find/replace leaves behind.
#
# Both give a clean exit 0 from a section that checked nothing, which is why the
# guard is worth having. What it is NOT reachable by is a relocation of
# `assets/vale/`: PROBES_CHECKED only reaches 0 that way if BOTH `.vale.ini`
# files are gone, and the loop at the top of the `.vale.ini coverage` section
# errs on each of them first, so that state is already FAIL >= 2 and this guard
# is never the cause. The message therefore names the table, not the files —
# describing it as "every probe skill's .vale.ini is missing" misdiagnosed the
# one thing that can actually trigger it.
if [[ $PROBES_CHECKED -eq 0 ]]; then
err "no probe path was checked — every probe skill's assets/vale/.vale.ini is missing, so the glob-coverage section verified nothing at all"
err "no probe path was checked — the probe table is empty, or no row's first column names a skill directory under plugins/kyberforge/.apm/skills/ that has an assets/vale/.vale.ini, so the glob-coverage section verified nothing at all"
fi
if [[ $FAIL -gt 0 ]]; then
@@ -367,7 +391,7 @@ fi
# do any work against the real repo?" untestable from outside — the counts below
# are what tests/test-check-vale-style-sync.sh asserts a non-zero floor on.
if [[ "$VALE_AVAILABLE" == true ]]; then
echo "Vale style sync check passed: 2 .vale.ini file(s) checked, $PROBES_CHECKED glob probe(s) verified with vale."
echo "Vale style sync check passed: $INIS_CHECKED .vale.ini file(s) checked, $PROBES_CHECKED glob probe(s) verified with vale."
else
echo "Vale style sync check passed (text-level only, vale unavailable): 2 .vale.ini file(s) checked, 0 glob probe(s) verified."
echo "Vale style sync check passed (text-level only, vale unavailable): $INIS_CHECKED .vale.ini file(s) checked, 0 glob probe(s) verified."
fi