test: fail the gate when a suite is skipped or never reports

Three gaps left by the previous round, all the same shape: a gate reporting
success having verified less than it appears to.

run_bats() hard-failed on a missing or non-executable runner but never checked
that the runner produced anything. An empty, executable run-bats.sh exits 0,
and the dispatcher printed a green summary with 166 bats tests silently absent.
It now requires an "N tests, M failures" line with a non-zero count.

run-tests.sh's skip listing is swallowed by pre-commit on a pass, so on a
machine without vale three suites exited 77 and the pre-push gate went green
having run 14 of 17. The hook now invokes it as --strict, where a skip fails
and the error names each suite and the reason it skipped. An ad-hoc local run
still skips gracefully -- at pre-push a skip means a documented dependency is
missing, which is a setup error, not a legitimate state. Deliberately not wired
to the vale downgrade's env var: one flag must not disarm two gates.

BATS_FILE_FLOOR is replaced by an expectation derived from git ls-files. A
floor of 8 against a real count of 10 let two files and eleven tests disappear
green, and the number needed an edit whenever a plugin was added. The derived
set needs no number, and catches an addition as well as a removal -- a .bats
file staged into the index and deleted from disk is now demanded back.

The vale opt-out announced its downgrade to nobody: pre-commit prints nothing
for a passing hook, so the summary line AGENTS.md tells the reader to check was
unreachable in exactly the situation it exists for. The hook is now verbose.

Also corrects the PROBES_CHECKED guard, whose commit message described a state
that cannot occur -- the .vale.ini loop errs first. Its two reachable triggers,
a gutted probe heredoc and a probe row naming a missing directory, had no test;
they do now, each asserting the guard is the sole cause.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X7GvKuJfy2WrdBmUttV4DT
This commit is contained in:
2026-08-14 12:29:42 +00:00
parent 874bf06b18
commit aa15fc850c
7 changed files with 637 additions and 74 deletions

View File

@@ -46,8 +46,8 @@ repos:
hooks:
- id: run-tests
name: Run test suite
description: Run all test-*.sh files and bats suite
entry: bash tests/run-tests.sh
description: Run all test-*.sh files and bats suite. --strict because a suite that exits 77 (SKIPPED) at pre-push means a documented dependency is missing on this machine, and pre-commit prints nothing for a passing hook -- without it the gate went green having verified 15 of 17 suites on a vale-less PATH, with the skip list swallowed. Ad-hoc `bash tests/run-tests.sh` still skips gracefully.
entry: bash tests/run-tests.sh --strict
language: system
stages: [pre-push]
pass_filenames: false
@@ -115,6 +115,16 @@ repos:
stages: [pre-push]
pass_filenames: false
always_run: true
# verbose so the DOWNGRADED run is audible. This hook can pass while
# having verified strictly less than its name claims:
# CHECK_VALE_STYLE_SYNC_ALLOW_MISSING_VALE=1 skips all six glob probes
# and says so on a `passed (text-level only, vale unavailable)` line.
# pre-commit prints nothing at all for a passing hook, so without this
# the opt-out reinstated exactly the silent vacuous pass the script was
# written to kill, one level up -- the run showed a bare `Passed` and
# AGENTS.md's instruction to read that summary line was impossible to
# follow in the one situation the opt-out exists for. The script's clean
# output is a single line, so this costs one line per push.
- id: check-scope-walkup-sync
name: Check scope walk-up implementations agree