test: fail the gate when a suite is skipped or never reports
Three gaps left by the previous round, all the same shape: a gate reporting success having verified less than it appears to. run_bats() hard-failed on a missing or non-executable runner but never checked that the runner produced anything. An empty, executable run-bats.sh exits 0, and the dispatcher printed a green summary with 166 bats tests silently absent. It now requires an "N tests, M failures" line with a non-zero count. run-tests.sh's skip listing is swallowed by pre-commit on a pass, so on a machine without vale three suites exited 77 and the pre-push gate went green having run 14 of 17. The hook now invokes it as --strict, where a skip fails and the error names each suite and the reason it skipped. An ad-hoc local run still skips gracefully -- at pre-push a skip means a documented dependency is missing, which is a setup error, not a legitimate state. Deliberately not wired to the vale downgrade's env var: one flag must not disarm two gates. BATS_FILE_FLOOR is replaced by an expectation derived from git ls-files. A floor of 8 against a real count of 10 let two files and eleven tests disappear green, and the number needed an edit whenever a plugin was added. The derived set needs no number, and catches an addition as well as a removal -- a .bats file staged into the index and deleted from disk is now demanded back. The vale opt-out announced its downgrade to nobody: pre-commit prints nothing for a passing hook, so the summary line AGENTS.md tells the reader to check was unreachable in exactly the situation it exists for. The hook is now verbose. Also corrects the PROBES_CHECKED guard, whose commit message described a state that cannot occur -- the .vale.ini loop errs first. Its two reachable triggers, a gutted probe heredoc and a probe row naming a missing directory, had no test; they do now, each asserting the guard is the sole cause. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X7GvKuJfy2WrdBmUttV4DT
This commit is contained in:
@@ -77,6 +77,13 @@ fi
|
||||
SKILL_INI="$SKILL_AUDIT/assets/vale/.vale.ini"
|
||||
AGENT_INI="$AGENT_AUDIT/assets/vale/.vale.ini"
|
||||
|
||||
# Counted, not assumed. The summary line at the bottom used to hardcode `2
|
||||
# .vale.ini file(s) checked` in both branches. That was true on any clean run --
|
||||
# a missing or unreadable file errs below and the script never reaches the
|
||||
# summary -- but the line's whole purpose is to say what this run actually
|
||||
# inspected, and a constant says what the author expected. Nothing asserted it,
|
||||
# so it would have survived becoming false.
|
||||
INIS_CHECKED=0
|
||||
for ini in "$SKILL_INI" "$AGENT_INI"; do
|
||||
rel_ini="${ini#"$REPO_ROOT"/}"
|
||||
# `-e`, not `-f`: a path that exists but is not a readable regular file (a
|
||||
@@ -106,6 +113,9 @@ for ini in "$SKILL_INI" "$AGENT_INI"; do
|
||||
err "$rel_ini exists but could not be read — none of its assertions could run, and an unreadable file cannot be distinguished from a clean one downstream"
|
||||
continue
|
||||
fi
|
||||
# Counted here, past both `continue`s: the file exists and its bytes were
|
||||
# readable, so every assertion below it really does run against it.
|
||||
INIS_CHECKED=$((INIS_CHECKED + 1))
|
||||
# StylesPath is resolved relative to the .vale.ini, which is the only reason
|
||||
# the bundled styles are found from a consuming repo's clone prefix.
|
||||
if ! grep -Eq '^[[:space:]]*StylesPath[[:space:]]*=[[:space:]]*styles[[:space:]]*$' "$ini"; then
|
||||
@@ -348,13 +358,27 @@ agent-audit|.claude/agents/demo.md|hooks-only
|
||||
agent-audit|copilot/demo.agent.md|hooks-only
|
||||
EOF_PROBE
|
||||
|
||||
# Second floor, on the probe table rather than the directory paths: every probe
|
||||
# `continue`s when its skill's `.vale.ini` is absent, so a relocation of
|
||||
# `assets/vale/` alone — with both audit skill directories still in place, which
|
||||
# the guard at the top of this file therefore cannot see — would skip the entire
|
||||
# table and leave FAIL at 0. Zero probes checked is never a valid result.
|
||||
# Second floor, on the probe TABLE rather than on the directory paths. Every row
|
||||
# `continue`s when the `.vale.ini` of the skill its first column names is absent,
|
||||
# so the table can verify nothing while FAIL stays 0. Two states do that, and no
|
||||
# other assertion in this file sees either:
|
||||
#
|
||||
# * the `EOF_PROBE` heredoc gutted — a bad merge, a truncated edit, or a
|
||||
# wholesale delete of the rows. The loop body never runs at all.
|
||||
# * every row's skill column drifting away from the directory names on disk
|
||||
# (`skill-audit|` -> `skill-auditX|`), which is what a skill rename plus a
|
||||
# half-applied find/replace leaves behind.
|
||||
#
|
||||
# Both give a clean exit 0 from a section that checked nothing, which is why the
|
||||
# guard is worth having. What it is NOT reachable by is a relocation of
|
||||
# `assets/vale/`: PROBES_CHECKED only reaches 0 that way if BOTH `.vale.ini`
|
||||
# files are gone, and the loop at the top of the `.vale.ini coverage` section
|
||||
# errs on each of them first, so that state is already FAIL >= 2 and this guard
|
||||
# is never the cause. The message therefore names the table, not the files —
|
||||
# describing it as "every probe skill's .vale.ini is missing" misdiagnosed the
|
||||
# one thing that can actually trigger it.
|
||||
if [[ $PROBES_CHECKED -eq 0 ]]; then
|
||||
err "no probe path was checked — every probe skill's assets/vale/.vale.ini is missing, so the glob-coverage section verified nothing at all"
|
||||
err "no probe path was checked — the probe table is empty, or no row's first column names a skill directory under plugins/kyberforge/.apm/skills/ that has an assets/vale/.vale.ini, so the glob-coverage section verified nothing at all"
|
||||
fi
|
||||
|
||||
if [[ $FAIL -gt 0 ]]; then
|
||||
@@ -367,7 +391,7 @@ fi
|
||||
# do any work against the real repo?" untestable from outside — the counts below
|
||||
# are what tests/test-check-vale-style-sync.sh asserts a non-zero floor on.
|
||||
if [[ "$VALE_AVAILABLE" == true ]]; then
|
||||
echo "Vale style sync check passed: 2 .vale.ini file(s) checked, $PROBES_CHECKED glob probe(s) verified with vale."
|
||||
echo "Vale style sync check passed: $INIS_CHECKED .vale.ini file(s) checked, $PROBES_CHECKED glob probe(s) verified with vale."
|
||||
else
|
||||
echo "Vale style sync check passed (text-level only, vale unavailable): 2 .vale.ini file(s) checked, 0 glob probe(s) verified."
|
||||
echo "Vale style sync check passed (text-level only, vale unavailable): $INIS_CHECKED .vale.ini file(s) checked, 0 glob probe(s) verified."
|
||||
fi
|
||||
|
||||
Reference in New Issue
Block a user