test: stop the runners reporting green on suites that never ran
run_bats() was an if with no else, so a missing or non-executable run-bats.sh made all 166 bats tests disappear with no diagnostic while the run printed a green summary and exited 0. --bats-only became a total no-op. This is the same defect the PR fixed one level down, left open in the dispatcher that pre-push actually invokes -- and nothing tested run-tests.sh at all. run-bats.sh's aggregation was asserted by nothing. Three separate mutations to its failure-detection line all survived the existing suite, because real bats emits both a nonzero exit and "not ok" lines, so each signal masked the other. The new cases produce each signal without the other; all three mutants now die. Also in this pass: - zero discovered .bats files exited 0, so a widened path exclusion retired the suite silently. Replaced with a file-count floor: a collapse to one or two files is the same failure as a collapse to zero - an existing-but-empty status file counted as a pass, because [[ "" -eq 0 ]] is arithmetic-true and the || echo 1 fallback only covered a missing file. The repro is deterministic: the stub truncates its own status file, then kill -9s its parent so the real exit-code write never happens Adds tests/test-run-tests.sh, 9 cases pinning the exit 0 / 1 / 77 three-way split against a single summary line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X7GvKuJfy2WrdBmUttV4DT
This commit is contained in:
@@ -20,12 +20,22 @@ SKIPPED=()
|
||||
PASSED=0
|
||||
SKIP_EXIT=77
|
||||
|
||||
# A missing or non-executable run-bats.sh is a hard error, never a silent skip.
|
||||
# This was `if [[ -x "$BATS" ]]; then ... fi` with no else and no assertion that
|
||||
# bats ran at all, so renaming, moving, or dropping the executable bit off
|
||||
# run-bats.sh made the entire bats suite vanish with zero diagnostic and the run
|
||||
# still printed "Summary: N passed, 0 failed" and exited 0 -- and --bats-only
|
||||
# degraded to a no-op that printed nothing and exited 0. That is the same
|
||||
# green-either-way hole run-bats.sh's own zero-count guard closes one level down;
|
||||
# this closes it in the dispatcher that pre-push actually invokes.
|
||||
run_bats() {
|
||||
if [[ -x "$BATS" ]]; then
|
||||
echo "=== bats ==="
|
||||
bash "$BATS"
|
||||
echo ""
|
||||
if [[ ! -x "$BATS" ]]; then
|
||||
echo "Error: bats runner not found or not executable at $BATS — the bats suite cannot be skipped silently" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "=== bats ==="
|
||||
bash "$BATS"
|
||||
echo ""
|
||||
}
|
||||
|
||||
if $BATS_ONLY; then
|
||||
@@ -89,9 +99,16 @@ for script in ${SCRIPTS[@]+"${SCRIPTS[@]}"}; do
|
||||
echo "=== $rel ==="
|
||||
cat "$SCRATCH_ROOT/$idx.log"
|
||||
rc="$(cat "$SCRATCH_ROOT/$idx.status" 2>/dev/null || echo 1)"
|
||||
if [[ $rc -eq 0 ]]; then
|
||||
# String comparison, not `-eq`. `-eq` is arithmetic, and bash evaluates an
|
||||
# empty string as 0 there -- `[[ "" -eq 0 ]]` is true -- so an *empty* status
|
||||
# file counted as a pass. The `|| echo 1` fallback above only covers a
|
||||
# *missing* file; a file that exists but is empty is what you get when the job
|
||||
# is killed between the `>` truncating it and the `echo` completing, or on
|
||||
# ENOSPC. Under `==` an empty status falls through to FAILED, which is the only
|
||||
# safe reading of "the job did not report a result".
|
||||
if [[ "$rc" == "0" ]]; then
|
||||
PASSED=$((PASSED + 1))
|
||||
elif [[ $rc -eq $SKIP_EXIT ]]; then
|
||||
elif [[ "$rc" == "$SKIP_EXIT" ]]; then
|
||||
SKIPPED+=("$rel")
|
||||
else
|
||||
FAILED+=("$rel")
|
||||
|
||||
Reference in New Issue
Block a user