test: fail the gate when a suite is skipped or never reports

Three gaps left by the previous round, all the same shape: a gate reporting
success having verified less than it appears to.

run_bats() hard-failed on a missing or non-executable runner but never checked
that the runner produced anything. An empty, executable run-bats.sh exits 0,
and the dispatcher printed a green summary with 166 bats tests silently absent.
It now requires an "N tests, M failures" line with a non-zero count.

run-tests.sh's skip listing is swallowed by pre-commit on a pass, so on a
machine without vale three suites exited 77 and the pre-push gate went green
having run 14 of 17. The hook now invokes it as --strict, where a skip fails
and the error names each suite and the reason it skipped. An ad-hoc local run
still skips gracefully -- at pre-push a skip means a documented dependency is
missing, which is a setup error, not a legitimate state. Deliberately not wired
to the vale downgrade's env var: one flag must not disarm two gates.

BATS_FILE_FLOOR is replaced by an expectation derived from git ls-files. A
floor of 8 against a real count of 10 let two files and eleven tests disappear
green, and the number needed an edit whenever a plugin was added. The derived
set needs no number, and catches an addition as well as a removal -- a .bats
file staged into the index and deleted from disk is now demanded back.

The vale opt-out announced its downgrade to nobody: pre-commit prints nothing
for a passing hook, so the summary line AGENTS.md tells the reader to check was
unreachable in exactly the situation it exists for. The hook is now verbose.

Also corrects the PROBES_CHECKED guard, whose commit message described a state
that cannot occur -- the .vale.ini loop errs first. Its two reachable triggers,
a gutted probe heredoc and a probe row naming a missing directory, had no test;
they do now, each asserting the guard is the sole cause.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X7GvKuJfy2WrdBmUttV4DT

tests/test-run-tests.sh

@@ -8,6 +8,11 @@
 #     moving, or dropping the executable bit off run-bats.sh made the entire bats
 #     suite disappear with no diagnostic while the run printed a green summary and
 #     exited 0, and turned `--bats-only` into a no-op that printed nothing.
+#   * run_bats() then checked only that run-bats.sh was present and executable,
+#     never that it PRODUCED anything. `bash` on an empty run-bats.sh exits 0
+#     having printed nothing, so the dispatcher printed `=== bats ===` and a green
+#     summary. The runner's `N tests, M failures` line is now required, with a
+#     non-zero count.
 #   * The per-script status was compared with `-eq`, which is arithmetic, and bash
 #     evaluates an empty string as 0 there -- so a status file that existed but was
 #     empty counted as a pass.
@@ -191,6 +196,67 @@ else
   pass "a failing bats runner propagates out of run-tests.sh"
 fi
 
+# --- 5b. An EMPTY run-bats.sh is a hard error, not a green no-op ---
+# Present and executable was still not "it ran". `bash` on a zero-byte script
+# exits 0 having printed nothing, so the dispatcher printed `=== bats ===`, a
+# blank line, and `Summary: 1 passed, 0 skipped, 0 failed` with rc=0 -- the whole
+# bats suite gone, exactly the defect cases 2-4 close for the other spellings.
+# Truncation, a partial write, an editor saving an empty buffer, and a `set -e`
+# abort in a run-bats.sh preamble all produce this file.
+#
+# It only failed on the real repo incidentally, because tests/test-run-bats.sh
+# copies run-bats.sh into its own fixtures and blows up there; rename or retire
+# that file and the hole is live in the gate pre-push invokes. This asserts it
+# directly.
+echo ""
+echo "--- an empty run-bats.sh fails the run instead of passing silently ---"
+DIR5B="$(make_fake_repo)"
+FIXTURES+=("$DIR5B")
+: > "$DIR5B/tests/run-bats.sh"
+chmod +x "$DIR5B/tests/run-bats.sh"
+add_case "$DIR5B" test-ok.sh <<'EOF'
+#!/usr/bin/env bash
+echo "fine"
+EOF
+run_fake "$DIR5B"
+if echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$"; then
+  fail "an empty run-bats.sh produced a green summary — the bats suite vanished with no diagnostic"
+elif [[ $FAKE_RC -eq 0 ]]; then
+  fail "an empty run-bats.sh exited 0: $FAKE_OUT"
+elif echo "$FAKE_OUT" | grep -q "without reporting an 'N tests, M failures' summary"; then
+  pass "an empty run-bats.sh fails the run and says the bats suite was never verified"
+else
+  fail "the run failed but not with the no-summary message: $FAKE_OUT"
+fi
+
+# --- 5c. A runner reporting zero tests is a hard error too ---
+# The other half of "ran but produced nothing": the summary line is there and the
+# process exits 0, but it accounts for no tests. run-bats.sh has its own guard for
+# this one file down; asserting it here means the dispatcher does not depend on
+# that guard surviving, and it pins the count as the thing being read rather than
+# the mere presence of a line matching the pattern.
+echo ""
+echo "--- a bats runner reporting 0 tests fails the run ---"
+DIR5C="$(make_fake_repo)"
+FIXTURES+=("$DIR5C")
+install_stub_bats_runner "$DIR5C" <<'EOF'
+#!/usr/bin/env bash
+echo "0 tests, 0 failures"
+exit 0
+EOF
+add_case "$DIR5C" test-ok.sh <<'EOF'
+#!/usr/bin/env bash
+echo "fine"
+EOF
+run_fake "$DIR5C"
+if [[ $FAKE_RC -eq 0 ]]; then
+  fail "a bats runner reporting 0 tests exited 0 — a suite that executed nothing read as green: $FAKE_OUT"
+elif echo "$FAKE_OUT" | grep -q "reported 0 tests"; then
+  pass "a bats runner reporting 0 tests fails the run and says the suite executed nothing"
+else
+  fail "the run failed but not with the zero-tests message: $FAKE_OUT"
+fi
+
 # --- 6. An empty status file is FAILED, not a pass ---
 # The status is read back with `cat ... || echo 1`, which covers a *missing*
 # file. A file that exists but is empty is what a job killed between the `>`
@@ -311,6 +377,153 @@ else
   fail "the failed script was not named in the failure list: $FAKE_OUT"
 fi
 
+# --- 10. --strict turns a skip into a failure, and names the suite AND the reason ---
+# Graceful skipping is right for an ad-hoc run and wrong for a gate. At pre-push a
+# suite exiting 77 means a dependency AGENTS.md documents as required is missing
+# on the pushing machine -- and pre-commit prints nothing at all for a passing
+# hook, so the skip list this script writes to stdout was swallowed whole. A
+# vale-less PATH shipped a green gate having verified 15 of 17 suites.
+#
+# The reason is asserted, not just the name: "something was skipped" leaves the
+# reader with no idea which binary to install, which is most of why the swallowed
+# list was worth so little in the first place. Matched on the SIX-SPACE INDENT the
+# report writes, not on the reason text alone -- the suite's own log is echoed
+# back verbatim earlier in the same output, so a bare text match passes even with
+# the reason capture deleted. Verified: narrowing the capture to the `SKIP:`
+# prefix left the loose form green.
+echo ""
+echo "--- --strict fails the run on a skipped suite and names it with its reason ---"
+DIR10="$(make_fake_repo)"
+FIXTURES+=("$DIR10")
+install_healthy_bats_runner "$DIR10"
+add_case "$DIR10" test-needs-a-binary.sh <<'EOF'
+#!/usr/bin/env bash
+echo "SKIP: frobnicator is not installed — install it from https://example.invalid"
+exit 77
+EOF
+add_case "$DIR10" test-ok.sh <<'EOF'
+#!/usr/bin/env bash
+echo "fine"
+EOF
+run_fake "$DIR10" --strict
+if [[ $FAKE_RC -eq 0 ]]; then
+  fail "--strict passed with a skipped suite — the gate reports green having verified less than it ran: $FAKE_OUT"
+elif ! echo "$FAKE_OUT" | grep -q "a skip is a SETUP ERROR"; then
+  fail "--strict failed but never said a skip is a setup error: $FAKE_OUT"
+elif ! echo "$FAKE_OUT" | grep -q "test-needs-a-binary.sh"; then
+  fail "--strict failed without naming the skipped suite: $FAKE_OUT"
+elif echo "$FAKE_OUT" | grep -q "^      SKIP: frobnicator is not installed"; then
+  pass "--strict fails on a skip, names the suite, and carries through the reason it printed"
+else
+  fail "--strict named the suite but swallowed its skip reason: $FAKE_OUT"
+fi
+
+# --- 10b. RUN_TESTS_STRICT=1 is the same switch. The hook uses the flag because
+# it is self-documenting in .pre-commit-config.yaml; the env var exists for a CI
+# runner that cannot edit the command line. Both are asserted so one cannot rot.
+echo ""
+echo "--- RUN_TESTS_STRICT=1 fails the run on a skipped suite ---"
+STRICT_ENV_OUT=""
+STRICT_ENV_RC=0
+STRICT_ENV_PRIV="$(mktemp -d)"
+FIXTURES+=("$STRICT_ENV_PRIV")
+STRICT_ENV_OUT="$(TMPDIR="$STRICT_ENV_PRIV" TEST_DIR="$DIR10/cases" RUN_TESTS_STRICT=1 \
+  bash "$DIR10/tests/run-tests.sh" 2>&1)" || STRICT_ENV_RC=$?
+if [[ $STRICT_ENV_RC -eq 0 ]]; then
+  fail "RUN_TESTS_STRICT=1 passed with a skipped suite: $STRICT_ENV_OUT"
+elif echo "$STRICT_ENV_OUT" | grep -q "a skip is a SETUP ERROR"; then
+  pass "RUN_TESTS_STRICT=1 is the same gate as --strict"
+else
+  fail "RUN_TESTS_STRICT=1 failed for some other reason: $STRICT_ENV_OUT"
+fi
+
+# --- 10c. WITHOUT strict, the same fixture still skips gracefully and passes ---
+# The control for 10 and 10b, and the half the coordinator asked for explicitly:
+# an ad-hoc `bash tests/run-tests.sh` on a laptop missing a dev binary must not
+# go red. Without this, "fix the gate" could quietly mean "fail everywhere".
+echo ""
+echo "--- the same skipped suite passes, still SKIPPED, without strict ---"
+run_fake "$DIR10"
+if [[ $FAKE_RC -ne 0 ]]; then
+  fail "a skipped suite failed a non-strict run — graceful skipping is gone: $FAKE_OUT"
+elif ! echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 1 skipped, 0 failed ===$"; then
+  fail "a non-strict run miscounted the skip: $FAKE_OUT"
+elif echo "$FAKE_OUT" | grep -q "^      SKIP: frobnicator is not installed"; then
+  pass "without strict the suite is SKIPPED, the run passes, and the reason is still reported"
+else
+  fail "a non-strict run passed but dropped the skip reason: $FAKE_OUT"
+fi
+
+# --- 10d. --strict does not become a blanket failure ---
+# The case that proves 10 and 10b fail for their stated reason. A clean run with
+# nothing skipped must be just as green under --strict as without it, otherwise
+# the gate is not a gate, it is an outage.
+echo ""
+echo "--- --strict is still green when nothing skipped ---"
+DIR10D="$(make_fake_repo)"
+FIXTURES+=("$DIR10D")
+install_healthy_bats_runner "$DIR10D"
+add_case "$DIR10D" test-ok.sh <<'EOF'
+#!/usr/bin/env bash
+echo "fine"
+EOF
+run_fake "$DIR10D" --strict
+if [[ $FAKE_RC -ne 0 ]]; then
+  fail "--strict failed a run with nothing skipped — it fails unconditionally: $FAKE_OUT"
+elif echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$"; then
+  pass "--strict leaves a run with no skips green"
+else
+  fail "--strict passed with the wrong summary: $FAKE_OUT"
+fi
+
+# --- 10e. An unknown flag is rejected, not ignored ---
+# `--strict` reaching this script as a silently-ignored argument is the single
+# typo that turns the gate back off while every hook still reports Passed, so the
+# arg loop refuses anything it does not know rather than falling through.
+echo ""
+echo "--- an unrecognised flag fails with usage instead of being ignored ---"
+DIR10E="$(make_fake_repo)"
+FIXTURES+=("$DIR10E")
+install_healthy_bats_runner "$DIR10E"
+add_case "$DIR10E" test-ok.sh <<'EOF'
+#!/usr/bin/env bash
+echo "fine"
+EOF
+run_fake "$DIR10E" --strickt
+if [[ $FAKE_RC -eq 0 ]]; then
+  fail "a misspelled flag was ignored and the run passed — a typo silently disarms the gate: $FAKE_OUT"
+elif echo "$FAKE_OUT" | grep -q "Usage: .*--bats-only.*--strict"; then
+  pass "an unrecognised flag fails the run with usage"
+else
+  fail "an unrecognised flag failed but not with usage: $FAKE_OUT"
+fi
+
+# --- 10f. A skip reason printed on STDERR, with no `SKIP:` prefix, still lands ---
+# There is no house format: three suites print `SKIP: <reason>` on stdout and
+# tests/test-sync-plugin-content.sh prints `apm not installed -- skipping (...)`
+# on stderr. batch-run.sh folds stderr into the same log, so both are reachable,
+# but only a fallback chain finds the second one. Without this case the reason
+# extraction could be narrowed to the `SKIP:` prefix and the apm suite would fail
+# the gate with no indication of what to install.
+echo ""
+echo "--- a stderr skip reason with no SKIP: prefix is still reported ---"
+DIR10F="$(make_fake_repo)"
+FIXTURES+=("$DIR10F")
+install_healthy_bats_runner "$DIR10F"
+add_case "$DIR10F" test-stderr-skip.sh <<'EOF'
+#!/usr/bin/env bash
+echo "widgetizer not installed -- skipping (see docs)" >&2
+exit 77
+EOF
+run_fake "$DIR10F" --strict
+if [[ $FAKE_RC -eq 0 ]]; then
+  fail "--strict passed on a suite that skipped via stderr: $FAKE_OUT"
+elif echo "$FAKE_OUT" | grep -q "^      widgetizer not installed -- skipping"; then
+  pass "a skip reason printed to stderr without a SKIP: prefix is still carried into the failure"
+else
+  fail "--strict failed but lost the stderr skip reason: $FAKE_OUT"
+fi
+
 echo ""
 echo "Results: $PASS passed, $FAIL failed"
 [[ $FAIL -eq 0 ]]