Defame1297
52bbd62286
run_bats() was an if with no else, so a missing or non-executable run-bats.sh made all 166 bats tests disappear with no diagnostic while the run printed a green summary and exited 0. --bats-only became a total no-op. This is the same defect the PR fixed one level down, left open in the dispatcher that pre-push actually invokes -- and nothing tested run-tests.sh at all. run-bats.sh's aggregation was asserted by nothing. Three separate mutations to its failure-detection line all survived the existing suite, because real bats emits both a nonzero exit and "not ok" lines, so each signal masked the other. The new cases produce each signal without the other; all three mutants now die. Also in this pass: - zero discovered .bats files exited 0, so a widened path exclusion retired the suite silently. Replaced with a file-count floor: a collapse to one or two files is the same failure as a collapse to zero - an existing-but-empty status file counted as a pass, because [[ "" -eq 0 ]] is arithmetic-true and the || echo 1 fallback only covered a missing file. The repro is deterministic: the stub truncates its own status file, then kill -9s its parent so the real exit-code write never happens Adds tests/test-run-tests.sh, 9 cases pinning the exit 0 / 1 / 77 three-way split against a single summary line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X7GvKuJfy2WrdBmUttV4DT
133 lines
4.9 KiB
Bash
Executable File
133 lines
4.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Run all test-*.sh files in the repo (including plugins) and the bats suite.
|
|
# Usage: bash tests/run-tests.sh [--bats-only]
|
|
#
|
|
# A script exiting 77 (the automake convention) is reported as SKIPPED, not
|
|
# passed — a suite that can't run for lack of a binary must not read as green.
|
|
#
|
|
# TEST_DIR — override root to search for test-*.sh (default: REPO_ROOT); used by tests.
|
|
set -euo pipefail
|
|
|
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
BATS="$REPO_ROOT/tests/run-bats.sh"
|
|
BATS_ONLY=false
|
|
[[ "${1:-}" == "--bats-only" ]] && BATS_ONLY=true
|
|
|
|
SEARCH_ROOT="${TEST_DIR:-$REPO_ROOT}"
|
|
|
|
FAILED=()
|
|
SKIPPED=()
|
|
PASSED=0
|
|
SKIP_EXIT=77
|
|
|
|
# A missing or non-executable run-bats.sh is a hard error, never a silent skip.
|
|
# This was `if [[ -x "$BATS" ]]; then ... fi` with no else and no assertion that
|
|
# bats ran at all, so renaming, moving, or dropping the executable bit off
|
|
# run-bats.sh made the entire bats suite vanish with zero diagnostic and the run
|
|
# still printed "Summary: N passed, 0 failed" and exited 0 -- and --bats-only
|
|
# degraded to a no-op that printed nothing and exited 0. That is the same
|
|
# green-either-way hole run-bats.sh's own zero-count guard closes one level down;
|
|
# this closes it in the dispatcher that pre-push actually invokes.
|
|
run_bats() {
|
|
if [[ ! -x "$BATS" ]]; then
|
|
echo "Error: bats runner not found or not executable at $BATS — the bats suite cannot be skipped silently" >&2
|
|
exit 1
|
|
fi
|
|
echo "=== bats ==="
|
|
bash "$BATS"
|
|
echo ""
|
|
}
|
|
|
|
if $BATS_ONLY; then
|
|
run_bats
|
|
exit 0
|
|
fi
|
|
|
|
run_bats
|
|
|
|
# Collected with a `while read` loop rather than `mapfile` — macOS ships
|
|
# /bin/bash 3.2, which has no `mapfile`. Process substitution (not a pipe)
|
|
# keeps the loop in this shell so the appends survive. `sort` is still fed
|
|
# newline-delimited output, exactly as before.
|
|
SCRIPTS=()
|
|
while IFS= read -r script; do
|
|
SCRIPTS+=("$script")
|
|
done < <(
|
|
find "$SEARCH_ROOT" -name "test-*.sh" \
|
|
-not -path "*/.git/*" \
|
|
-not -path "*/.claude/worktrees/*" \
|
|
| sort
|
|
)
|
|
|
|
# Each test-*.sh is independent (fixtures live under its own mktemp dir, none
|
|
# write back into the live repo tree -- verified before adding this), so they
|
|
# run concurrently in fixed-size batches instead of one at a time. Dispatch and
|
|
# throttling is scripts/lib/batch-run.sh's batch_run -- shared with
|
|
# scripts/sync-plugin-content.sh and tests/run-bats.sh so a batching bug fix
|
|
# only needs to land once; see that file for why this is batched rather than a
|
|
# rolling `wait -n` pool.
|
|
SCRATCH_ROOT="$(mktemp -d)"
|
|
trap 'rm -rf "$SCRATCH_ROOT"' EXIT
|
|
# The source= path below is repo-root-relative, matching scripts/install.sh:5 --
|
|
# NOT script-dir-relative. The source-path used to resolve it is the cwd
|
|
# pre-commit invokes the linter from, which is the repo root, so `lib/...`
|
|
# (resolving to a nonexistent tests/lib/) and `../scripts/...` (escaping the repo
|
|
# entirely) both fail. Both spellings were live until issue #97, and neither was
|
|
# visible: the resulting SC1091 is `info` while .pre-commit-config.yaml pins
|
|
# `--severity=warning`. A directive that does not resolve also blinds
|
|
# test-vale-wrap.sh's `sourced_files()` exemption, which reads these same
|
|
# directives to find array seeding that lives in the sourced file.
|
|
#
|
|
# Do not start a comment line here with the linter's name -- it is parsed as a
|
|
# directive and errors out (SC1073).
|
|
# shellcheck source=scripts/lib/batch-run.sh
|
|
source "$REPO_ROOT/scripts/lib/batch-run.sh"
|
|
|
|
declare -a batch_args=()
|
|
idx=0
|
|
for script in ${SCRIPTS[@]+"${SCRIPTS[@]}"}; do
|
|
idx=$((idx + 1))
|
|
cmd="$(printf 'rc=0; bash %q || rc=$?; echo "$rc" >%q' "$script" "$SCRATCH_ROOT/$idx.status")"
|
|
batch_args+=("$idx" "$cmd")
|
|
done
|
|
batch_run "$SCRATCH_ROOT" ${batch_args[@]+"${batch_args[@]}"}
|
|
|
|
idx=0
|
|
for script in ${SCRIPTS[@]+"${SCRIPTS[@]}"}; do
|
|
idx=$((idx + 1))
|
|
rel="${script#"$SEARCH_ROOT/"}"
|
|
echo "=== $rel ==="
|
|
cat "$SCRATCH_ROOT/$idx.log"
|
|
rc="$(cat "$SCRATCH_ROOT/$idx.status" 2>/dev/null || echo 1)"
|
|
# String comparison, not `-eq`. `-eq` is arithmetic, and bash evaluates an
|
|
# empty string as 0 there -- `[[ "" -eq 0 ]]` is true -- so an *empty* status
|
|
# file counted as a pass. The `|| echo 1` fallback above only covers a
|
|
# *missing* file; a file that exists but is empty is what you get when the job
|
|
# is killed between the `>` truncating it and the `echo` completing, or on
|
|
# ENOSPC. Under `==` an empty status falls through to FAILED, which is the only
|
|
# safe reading of "the job did not report a result".
|
|
if [[ "$rc" == "0" ]]; then
|
|
PASSED=$((PASSED + 1))
|
|
elif [[ "$rc" == "$SKIP_EXIT" ]]; then
|
|
SKIPPED+=("$rel")
|
|
else
|
|
FAILED+=("$rel")
|
|
fi
|
|
echo ""
|
|
done
|
|
|
|
echo "=== Summary: $PASSED passed, ${#SKIPPED[@]} skipped, ${#FAILED[@]} failed ==="
|
|
if [[ ${#SKIPPED[@]} -gt 0 ]]; then
|
|
echo "Skipped scripts:"
|
|
for s in ${SKIPPED[@]+"${SKIPPED[@]}"}; do
|
|
echo " $s"
|
|
done
|
|
fi
|
|
if [[ ${#FAILED[@]} -gt 0 ]]; then
|
|
echo "Failed scripts:"
|
|
for s in ${FAILED[@]+"${FAILED[@]}"}; do
|
|
echo " $s"
|
|
done
|
|
exit 1
|
|
fi
|