test: stop the runners reporting green on suites that never ran
run_bats() was an if with no else, so a missing or non-executable run-bats.sh made all 166 bats tests disappear with no diagnostic while the run printed a green summary and exited 0. --bats-only became a total no-op. This is the same defect the PR fixed one level down, left open in the dispatcher that pre-push actually invokes -- and nothing tested run-tests.sh at all. run-bats.sh's aggregation was asserted by nothing. Three separate mutations to its failure-detection line all survived the existing suite, because real bats emits both a nonzero exit and "not ok" lines, so each signal masked the other. The new cases produce each signal without the other; all three mutants now die. Also in this pass: - zero discovered .bats files exited 0, so a widened path exclusion retired the suite silently. Replaced with a file-count floor: a collapse to one or two files is the same failure as a collapse to zero - an existing-but-empty status file counted as a pass, because [[ "" -eq 0 ]] is arithmetic-true and the || echo 1 fallback only covered a missing file. The repro is deterministic: the stub truncates its own status file, then kill -9s its parent so the real exit-code write never happens Adds tests/test-run-tests.sh, 9 cases pinning the exit 0 / 1 / 77 three-way split against a single summary line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X7GvKuJfy2WrdBmUttV4DT
This commit is contained in:
@@ -32,9 +32,24 @@ done < <(
|
||||
| sort
|
||||
)
|
||||
|
||||
if [[ ${#TEST_FILES[@]} -eq 0 ]]; then
|
||||
echo "No .bats test files found." >&2
|
||||
exit 0
|
||||
# A floor on the discovered file count, not merely a zero check, and a hard error
|
||||
# rather than the `exit 0` this used to be. Zero discovered files was the likelier
|
||||
# of the two silent-green failures -- a moved tests/ tree, a renamed skill
|
||||
# directory, or the `-not -path` exclusions above widening -- and it exited 0 with
|
||||
# a note on stderr nobody reads, while the zero-*result* guard further down was
|
||||
# already a hard error. A floor rather than `-gt 0` because the count collapsing
|
||||
# to 1 or 2 is the same failure as collapsing to 0 and only a floor names it.
|
||||
# Same reasoning, and the same "set it a little under the current count" rule, as
|
||||
# the per-glob floors in tests/test-vale-wrap.sh -- ordinary file churn does not
|
||||
# trip it, a broken or renamed path does.
|
||||
#
|
||||
# BATS_FILE_FLOOR overrides it. That override exists for the fixture repos in
|
||||
# tests/test-run-bats.sh and tests/test-run-tests.sh, which hold one or two .bats
|
||||
# files by design; it is not an escape hatch for a real run.
|
||||
BATS_FILE_FLOOR="${BATS_FILE_FLOOR:-8}"
|
||||
if [[ ${#TEST_FILES[@]} -lt $BATS_FILE_FLOOR ]]; then
|
||||
echo "Error: found ${#TEST_FILES[@]} .bats file(s) under $REPO_ROOT, below the floor of $BATS_FILE_FLOOR — the search path is wrong or the suite has been gutted" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Each file gets its own `bats` process, run concurrently (bounded by core
|
||||
@@ -83,17 +98,29 @@ for f in ${TEST_FILES[@]+"${TEST_FILES[@]}"}; do
|
||||
# file bats really did run, so it has to be distinguishable from a file that
|
||||
# produced nothing whatsoever.
|
||||
file_plan="$(grep -c '^1\.\.[0-9]' "$SCRATCH_ROOT/$i.log" || true)"
|
||||
# String-compared below, not `-ne`. `-ne` is arithmetic and bash evaluates an
|
||||
# empty string as 0 there -- `[[ "" -ne 0 ]]` is false -- so an *empty* status
|
||||
# file read as a clean exit. The `|| echo 1` fallback only covers a *missing*
|
||||
# file; an existing-but-empty one is what a job killed between the `>` and the
|
||||
# `echo` leaves behind, or what ENOSPC leaves behind.
|
||||
status="$(cat "$SCRATCH_ROOT/$i.status" 2>/dev/null || echo 1)"
|
||||
TOTAL_OK=$((TOTAL_OK + file_ok))
|
||||
TOTAL_NOT_OK=$((TOTAL_NOT_OK + file_not_ok))
|
||||
TOTAL_PLANS=$((TOTAL_PLANS + file_plan))
|
||||
if [[ "$file_not_ok" -gt 0 || "$status" -ne 0 ]]; then
|
||||
# Two independent failure signals, deliberately OR-ed: a file can report `not
|
||||
# ok` lines while its process still exits 0 (a bats formatter or wrapper that
|
||||
# swallows the status), and a file can exit non-zero having emitted no `not
|
||||
# ok` at all (a crash, a timeout, an unbound variable in setup_file). Real
|
||||
# bats normally emits both at once, so each signal masks the other and
|
||||
# dropping either half is invisible without tests that produce one without
|
||||
# the other -- tests/test-run-bats.sh has those.
|
||||
if [[ "$file_not_ok" -gt 0 || "$status" != "0" ]]; then
|
||||
FAIL=1
|
||||
fi
|
||||
done
|
||||
|
||||
# Zero counted tests is never a clean run: files were found (the empty-TEST_FILES
|
||||
# case exits above), so nothing was executed. Without this, a `bats` that emits
|
||||
# Zero counted tests is never a clean run: enough files were found (a count under
|
||||
# the floor, zero included, exits non-zero above), so nothing was executed. Without this, a `bats` that emits
|
||||
# nothing and exits 0 -- a broken binary, a formatter change, or a wholesale
|
||||
# `@test` removal -- reports "0 tests, 0 failures" and exits green, silently
|
||||
# turning a total harness failure into a pass.
|
||||
|
||||
Reference in New Issue
Block a user