test: stop the runners reporting green on suites that never ran

run_bats() was an if with no else, so a missing or non-executable run-bats.sh
made all 166 bats tests disappear with no diagnostic while the run printed a
green summary and exited 0. --bats-only became a total no-op. This is the same
defect the PR fixed one level down, left open in the dispatcher that pre-push
actually invokes -- and nothing tested run-tests.sh at all.

run-bats.sh's aggregation was asserted by nothing. Three separate mutations to
its failure-detection line all survived the existing suite, because real bats
emits both a nonzero exit and "not ok" lines, so each signal masked the other.
The new cases produce each signal without the other; all three mutants now die.

Also in this pass:
- zero discovered .bats files exited 0, so a widened path exclusion retired the
  suite silently. Replaced with a file-count floor: a collapse to one or two
  files is the same failure as a collapse to zero
- an existing-but-empty status file counted as a pass, because [[ "" -eq 0 ]]
  is arithmetic-true and the || echo 1 fallback only covered a missing file.
  The repro is deterministic: the stub truncates its own status file, then
  kill -9s its parent so the real exit-code write never happens

Adds tests/test-run-tests.sh, 9 cases pinning the exit 0 / 1 / 77 three-way
split against a single summary line.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X7GvKuJfy2WrdBmUttV4DT
This commit is contained in:
2026-08-14 11:04:22 +00:00
parent af085ed057
commit 52bbd62286
4 changed files with 531 additions and 13 deletions

View File

@@ -32,9 +32,24 @@ done < <(
| sort
)
if [[ ${#TEST_FILES[@]} -eq 0 ]]; then
echo "No .bats test files found." >&2
exit 0
# A floor on the discovered file count, not merely a zero check, and a hard error
# rather than the `exit 0` this used to be. Zero discovered files was the likelier
# of the two silent-green failures -- a moved tests/ tree, a renamed skill
# directory, or the `-not -path` exclusions above widening -- and it exited 0 with
# a note on stderr nobody reads, while the zero-*result* guard further down was
# already a hard error. A floor rather than `-gt 0` because the count collapsing
# to 1 or 2 is the same failure as collapsing to 0 and only a floor names it.
# Same reasoning, and the same "set it a little under the current count" rule, as
# the per-glob floors in tests/test-vale-wrap.sh -- ordinary file churn does not
# trip it, a broken or renamed path does.
#
# BATS_FILE_FLOOR overrides it. That override exists for the fixture repos in
# tests/test-run-bats.sh and tests/test-run-tests.sh, which hold one or two .bats
# files by design; it is not an escape hatch for a real run.
BATS_FILE_FLOOR="${BATS_FILE_FLOOR:-8}"
if [[ ${#TEST_FILES[@]} -lt $BATS_FILE_FLOOR ]]; then
echo "Error: found ${#TEST_FILES[@]} .bats file(s) under $REPO_ROOT, below the floor of $BATS_FILE_FLOOR — the search path is wrong or the suite has been gutted" >&2
exit 1
fi
# Each file gets its own `bats` process, run concurrently (bounded by core
@@ -83,17 +98,29 @@ for f in ${TEST_FILES[@]+"${TEST_FILES[@]}"}; do
# file bats really did run, so it has to be distinguishable from a file that
# produced nothing whatsoever.
file_plan="$(grep -c '^1\.\.[0-9]' "$SCRATCH_ROOT/$i.log" || true)"
# String-compared below, not `-ne`. `-ne` is arithmetic and bash evaluates an
# empty string as 0 there -- `[[ "" -ne 0 ]]` is false -- so an *empty* status
# file read as a clean exit. The `|| echo 1` fallback only covers a *missing*
# file; an existing-but-empty one is what a job killed between the `>` and the
# `echo` leaves behind, or what ENOSPC leaves behind.
status="$(cat "$SCRATCH_ROOT/$i.status" 2>/dev/null || echo 1)"
TOTAL_OK=$((TOTAL_OK + file_ok))
TOTAL_NOT_OK=$((TOTAL_NOT_OK + file_not_ok))
TOTAL_PLANS=$((TOTAL_PLANS + file_plan))
if [[ "$file_not_ok" -gt 0 || "$status" -ne 0 ]]; then
# Two independent failure signals, deliberately OR-ed: a file can report `not
# ok` lines while its process still exits 0 (a bats formatter or wrapper that
# swallows the status), and a file can exit non-zero having emitted no `not
# ok` at all (a crash, a timeout, an unbound variable in setup_file). Real
# bats normally emits both at once, so each signal masks the other and
# dropping either half is invisible without tests that produce one without
# the other -- tests/test-run-bats.sh has those.
if [[ "$file_not_ok" -gt 0 || "$status" != "0" ]]; then
FAIL=1
fi
done
# Zero counted tests is never a clean run: files were found (the empty-TEST_FILES
# case exits above), so nothing was executed. Without this, a `bats` that emits
# Zero counted tests is never a clean run: enough files were found (a count under
# the floor, zero included, exits non-zero above), so nothing was executed. Without this, a `bats` that emits
# nothing and exits 0 -- a broken binary, a formatter change, or a wholesale
# `@test` removal -- reports "0 tests, 0 failures" and exits green, silently
# turning a total harness failure into a pass.