test: stop the runners reporting green on suites that never ran

run_bats() was an if with no else, so a missing or non-executable run-bats.sh
made all 166 bats tests disappear with no diagnostic while the run printed a
green summary and exited 0. --bats-only became a total no-op. This is the same
defect the PR fixed one level down, left open in the dispatcher that pre-push
actually invokes -- and nothing tested run-tests.sh at all.

run-bats.sh's aggregation was asserted by nothing. Three separate mutations to
its failure-detection line all survived the existing suite, because real bats
emits both a nonzero exit and "not ok" lines, so each signal masked the other.
The new cases produce each signal without the other; all three mutants now die.

Also in this pass:
- zero discovered .bats files exited 0, so a widened path exclusion retired the
  suite silently. Replaced with a file-count floor: a collapse to one or two
  files is the same failure as a collapse to zero
- an existing-but-empty status file counted as a pass, because [[ "" -eq 0 ]]
  is arithmetic-true and the || echo 1 fallback only covered a missing file.
  The repro is deterministic: the stub truncates its own status file, then
  kill -9s its parent so the real exit-code write never happens

Adds tests/test-run-tests.sh, 9 cases pinning the exit 0 / 1 / 77 three-way
split against a single summary line.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X7GvKuJfy2WrdBmUttV4DT
This commit is contained in:
2026-08-14 11:04:22 +00:00
parent af085ed057
commit 52bbd62286
4 changed files with 531 additions and 13 deletions

View File

@@ -20,12 +20,22 @@ SKIPPED=()
PASSED=0
SKIP_EXIT=77
# A missing or non-executable run-bats.sh is a hard error, never a silent skip.
# This was `if [[ -x "$BATS" ]]; then ... fi` with no else and no assertion that
# bats ran at all, so renaming, moving, or dropping the executable bit off
# run-bats.sh made the entire bats suite vanish with zero diagnostic and the run
# still printed "Summary: N passed, 0 failed" and exited 0 -- and --bats-only
# degraded to a no-op that printed nothing and exited 0. That is the same
# green-either-way hole run-bats.sh's own zero-count guard closes one level down;
# this closes it in the dispatcher that pre-push actually invokes.
run_bats() {
if [[ -x "$BATS" ]]; then
echo "=== bats ==="
bash "$BATS"
echo ""
if [[ ! -x "$BATS" ]]; then
echo "Error: bats runner not found or not executable at $BATS — the bats suite cannot be skipped silently" >&2
exit 1
fi
echo "=== bats ==="
bash "$BATS"
echo ""
}
if $BATS_ONLY; then
@@ -89,9 +99,16 @@ for script in ${SCRIPTS[@]+"${SCRIPTS[@]}"}; do
echo "=== $rel ==="
cat "$SCRATCH_ROOT/$idx.log"
rc="$(cat "$SCRATCH_ROOT/$idx.status" 2>/dev/null || echo 1)"
if [[ $rc -eq 0 ]]; then
# String comparison, not `-eq`. `-eq` is arithmetic, and bash evaluates an
# empty string as 0 there -- `[[ "" -eq 0 ]]` is true -- so an *empty* status
# file counted as a pass. The `|| echo 1` fallback above only covers a
# *missing* file; a file that exists but is empty is what you get when the job
# is killed between the `>` truncating it and the `echo` completing, or on
# ENOSPC. Under `==` an empty status falls through to FAILED, which is the only
# safe reading of "the job did not report a result".
if [[ "$rc" == "0" ]]; then
PASSED=$((PASSED + 1))
elif [[ $rc -eq $SKIP_EXIT ]]; then
elif [[ "$rc" == "$SKIP_EXIT" ]]; then
SKIPPED+=("$rel")
else
FAILED+=("$rel")