Defame1297
52bbd62286
run_bats() was an if with no else, so a missing or non-executable run-bats.sh made all 166 bats tests disappear with no diagnostic while the run printed a green summary and exited 0. --bats-only became a total no-op. This is the same defect the PR fixed one level down, left open in the dispatcher that pre-push actually invokes -- and nothing tested run-tests.sh at all. run-bats.sh's aggregation was asserted by nothing. Three separate mutations to its failure-detection line all survived the existing suite, because real bats emits both a nonzero exit and "not ok" lines, so each signal masked the other. The new cases produce each signal without the other; all three mutants now die. Also in this pass: - zero discovered .bats files exited 0, so a widened path exclusion retired the suite silently. Replaced with a file-count floor: a collapse to one or two files is the same failure as a collapse to zero - an existing-but-empty status file counted as a pass, because [[ "" -eq 0 ]] is arithmetic-true and the || echo 1 fallback only covered a missing file. The repro is deterministic: the stub truncates its own status file, then kill -9s its parent so the real exit-code write never happens Adds tests/test-run-tests.sh, 9 cases pinning the exit 0 / 1 / 77 three-way split against a single summary line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X7GvKuJfy2WrdBmUttV4DT
143 lines
6.5 KiB
Bash
Executable File
143 lines
6.5 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Run all bats test files in the repo.
|
|
# Usage: bash tests/run-bats.sh
|
|
set -euo pipefail
|
|
|
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
BATS="$REPO_ROOT/tests/bats/bin/bats"
|
|
|
|
if [[ ! -x "$BATS" ]]; then
|
|
echo "bats not found at $BATS — initializing submodules..." >&2
|
|
git -C "$REPO_ROOT" submodule update --init --recursive
|
|
fi
|
|
|
|
if [[ ! -x "$BATS" ]]; then
|
|
echo "Error: bats still not found at $BATS after submodule init" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# Collected with a `while read` loop rather than `mapfile` — macOS ships
|
|
# /bin/bash 3.2, which has no `mapfile`. Process substitution (not a pipe)
|
|
# keeps the loop in this shell so the appends survive. `sort` is still fed
|
|
# newline-delimited output, exactly as before. Same convention as
|
|
# tests/run-tests.sh.
|
|
TEST_FILES=()
|
|
while IFS= read -r f; do
|
|
TEST_FILES+=("$f")
|
|
done < <(
|
|
find "$REPO_ROOT" -name "*.bats" \
|
|
-not -path "*/tests/bats/*" \
|
|
-not -path "*/test_helper/*" \
|
|
-not -path "*/.claude/worktrees/*" \
|
|
| sort
|
|
)
|
|
|
|
# A floor on the discovered file count, not merely a zero check, and a hard error
|
|
# rather than the `exit 0` this used to be. Zero discovered files was the likelier
|
|
# of the two silent-green failures -- a moved tests/ tree, a renamed skill
|
|
# directory, or the `-not -path` exclusions above widening -- and it exited 0 with
|
|
# a note on stderr nobody reads, while the zero-*result* guard further down was
|
|
# already a hard error. A floor rather than `-gt 0` because the count collapsing
|
|
# to 1 or 2 is the same failure as collapsing to 0 and only a floor names it.
|
|
# Same reasoning, and the same "set it a little under the current count" rule, as
|
|
# the per-glob floors in tests/test-vale-wrap.sh -- ordinary file churn does not
|
|
# trip it, a broken or renamed path does.
|
|
#
|
|
# BATS_FILE_FLOOR overrides it. That override exists for the fixture repos in
|
|
# tests/test-run-bats.sh and tests/test-run-tests.sh, which hold one or two .bats
|
|
# files by design; it is not an escape hatch for a real run.
|
|
BATS_FILE_FLOOR="${BATS_FILE_FLOOR:-8}"
|
|
if [[ ${#TEST_FILES[@]} -lt $BATS_FILE_FLOOR ]]; then
|
|
echo "Error: found ${#TEST_FILES[@]} .bats file(s) under $REPO_ROOT, below the floor of $BATS_FILE_FLOOR — the search path is wrong or the suite has been gutted" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# Each file gets its own `bats` process, run concurrently (bounded by core
|
|
# count) instead of one `bats` invocation working through all files serially.
|
|
# A single test file is still serial internally -- this only overlaps the
|
|
# fixed per-process startup cost (git/apm subprocess spawns dominate several
|
|
# of these suites) across files, which is where the wall-clock actually goes.
|
|
# Output is buffered per file so concurrent TAP streams can't interleave, then
|
|
# flushed in stable sorted order once every job has finished.
|
|
#
|
|
# Dispatch and throttling is scripts/lib/batch-run.sh's batch_run -- shared
|
|
# with scripts/sync-plugin-content.sh and tests/run-tests.sh so a batching bug
|
|
# fix only needs to land once; see that file for why this is batched rather
|
|
# than a rolling `wait -n` pool.
|
|
SCRATCH_ROOT="$(mktemp -d)"
|
|
trap 'rm -rf "$SCRATCH_ROOT"' EXIT
|
|
# Repo-root-relative -- see tests/run-tests.sh for why `../scripts/...` does not
|
|
# resolve here despite looking right.
|
|
# shellcheck source=scripts/lib/batch-run.sh
|
|
source "$REPO_ROOT/scripts/lib/batch-run.sh"
|
|
|
|
declare -a batch_args=()
|
|
i=0
|
|
for f in ${TEST_FILES[@]+"${TEST_FILES[@]}"}; do
|
|
i=$((i + 1))
|
|
cmd="$(printf '%q %q; echo $? >%q' "$BATS" "$f" "$SCRATCH_ROOT/$i.status")"
|
|
batch_args+=("$i" "$cmd")
|
|
done
|
|
batch_run "$SCRATCH_ROOT" ${batch_args[@]+"${batch_args[@]}"}
|
|
|
|
FAIL=0
|
|
TOTAL_OK=0
|
|
TOTAL_NOT_OK=0
|
|
TOTAL_PLANS=0
|
|
i=0
|
|
for f in ${TEST_FILES[@]+"${TEST_FILES[@]}"}; do
|
|
i=$((i + 1))
|
|
rel="${f#"$REPO_ROOT"/}"
|
|
echo "=== $rel ==="
|
|
cat "$SCRATCH_ROOT/$i.log"
|
|
echo ""
|
|
file_ok="$(grep -c '^ok ' "$SCRATCH_ROOT/$i.log" || true)"
|
|
file_not_ok="$(grep -c '^not ok ' "$SCRATCH_ROOT/$i.log" || true)"
|
|
# The TAP plan line (`1..N`). Counted separately from the results because an
|
|
# empty-but-valid file emits `1..0` and no result lines at all -- that is a
|
|
# file bats really did run, so it has to be distinguishable from a file that
|
|
# produced nothing whatsoever.
|
|
file_plan="$(grep -c '^1\.\.[0-9]' "$SCRATCH_ROOT/$i.log" || true)"
|
|
# String-compared below, not `-ne`. `-ne` is arithmetic and bash evaluates an
|
|
# empty string as 0 there -- `[[ "" -ne 0 ]]` is false -- so an *empty* status
|
|
# file read as a clean exit. The `|| echo 1` fallback only covers a *missing*
|
|
# file; an existing-but-empty one is what a job killed between the `>` and the
|
|
# `echo` leaves behind, or what ENOSPC leaves behind.
|
|
status="$(cat "$SCRATCH_ROOT/$i.status" 2>/dev/null || echo 1)"
|
|
TOTAL_OK=$((TOTAL_OK + file_ok))
|
|
TOTAL_NOT_OK=$((TOTAL_NOT_OK + file_not_ok))
|
|
TOTAL_PLANS=$((TOTAL_PLANS + file_plan))
|
|
# Two independent failure signals, deliberately OR-ed: a file can report `not
|
|
# ok` lines while its process still exits 0 (a bats formatter or wrapper that
|
|
# swallows the status), and a file can exit non-zero having emitted no `not
|
|
# ok` at all (a crash, a timeout, an unbound variable in setup_file). Real
|
|
# bats normally emits both at once, so each signal masks the other and
|
|
# dropping either half is invisible without tests that produce one without
|
|
# the other -- tests/test-run-bats.sh has those.
|
|
if [[ "$file_not_ok" -gt 0 || "$status" != "0" ]]; then
|
|
FAIL=1
|
|
fi
|
|
done
|
|
|
|
# Zero counted tests is never a clean run: enough files were found (a count under
|
|
# the floor, zero included, exits non-zero above), so nothing was executed. Without this, a `bats` that emits
|
|
# nothing and exits 0 -- a broken binary, a formatter change, or a wholesale
|
|
# `@test` removal -- reports "0 tests, 0 failures" and exits green, silently
|
|
# turning a total harness failure into a pass.
|
|
#
|
|
# The two causes get different messages because they are different problems and
|
|
# `1..0` is itself valid TAP: no plan lines at all means bats produced no output
|
|
# to parse, while plans present with zero results means bats ran fine and the
|
|
# files genuinely declare no tests.
|
|
if [[ $((TOTAL_OK + TOTAL_NOT_OK)) -eq 0 ]]; then
|
|
if [[ "$TOTAL_PLANS" -eq 0 ]]; then
|
|
echo "Error: ${#TEST_FILES[@]} .bats file(s) ran but produced no TAP output at all — the bats harness is broken" >&2
|
|
else
|
|
echo "Error: ${#TEST_FILES[@]} .bats file(s) declared 0 tests — every @test appears to have been removed" >&2
|
|
fi
|
|
FAIL=1
|
|
fi
|
|
|
|
echo "$((TOTAL_OK + TOTAL_NOT_OK)) tests, $TOTAL_NOT_OK failures"
|
|
exit "$FAIL"
|