From aa15fc850c053b18eb29f3241f5dabd1db6456b0 Mon Sep 17 00:00:00 2001 From: Defame1297 Date: Fri, 14 Aug 2026 12:29:42 +0000 Subject: [PATCH] test: fail the gate when a suite is skipped or never reports Three gaps left by the previous round, all the same shape: a gate reporting success having verified less than it appears to. run_bats() hard-failed on a missing or non-executable runner but never checked that the runner produced anything. An empty, executable run-bats.sh exits 0, and the dispatcher printed a green summary with 166 bats tests silently absent. It now requires an "N tests, M failures" line with a non-zero count. run-tests.sh's skip listing is swallowed by pre-commit on a pass, so on a machine without vale three suites exited 77 and the pre-push gate went green having run 14 of 17. The hook now invokes it as --strict, where a skip fails and the error names each suite and the reason it skipped. An ad-hoc local run still skips gracefully -- at pre-push a skip means a documented dependency is missing, which is a setup error, not a legitimate state. Deliberately not wired to the vale downgrade's env var: one flag must not disarm two gates. BATS_FILE_FLOOR is replaced by an expectation derived from git ls-files. A floor of 8 against a real count of 10 let two files and eleven tests disappear green, and the number needed an edit whenever a plugin was added. The derived set needs no number, and catches an addition as well as a removal -- a .bats file staged into the index and deleted from disk is now demanded back. The vale opt-out announced its downgrade to nobody: pre-commit prints nothing for a passing hook, so the summary line AGENTS.md tells the reader to check was unreachable in exactly the situation it exists for. The hook is now verbose. Also corrects the PROBES_CHECKED guard, whose commit message described a state that cannot occur -- the .vale.ini loop errs first. Its two reachable triggers, a gutted probe heredoc and a probe row naming a missing directory, had no test; they do now, each asserting the guard is the sole cause. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X7GvKuJfy2WrdBmUttV4DT --- .pre-commit-config.yaml | 14 +- scripts/check-vale-style-sync.sh | 40 ++++-- tests/run-bats.sh | 88 +++++++++--- tests/run-tests.sh | 128 ++++++++++++++++- tests/test-check-vale-style-sync.sh | 97 ++++++++++--- tests/test-run-bats.sh | 131 ++++++++++++++--- tests/test-run-tests.sh | 213 ++++++++++++++++++++++++++++ 7 files changed, 637 insertions(+), 74 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index ab25009..f012486 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -46,8 +46,8 @@ repos: hooks: - id: run-tests name: Run test suite - description: Run all test-*.sh files and bats suite - entry: bash tests/run-tests.sh + description: Run all test-*.sh files and bats suite. --strict because a suite that exits 77 (SKIPPED) at pre-push means a documented dependency is missing on this machine, and pre-commit prints nothing for a passing hook -- without it the gate went green having verified 15 of 17 suites on a vale-less PATH, with the skip list swallowed. Ad-hoc `bash tests/run-tests.sh` still skips gracefully. + entry: bash tests/run-tests.sh --strict language: system stages: [pre-push] pass_filenames: false @@ -115,6 +115,16 @@ repos: stages: [pre-push] pass_filenames: false always_run: true + # verbose so the DOWNGRADED run is audible. This hook can pass while + # having verified strictly less than its name claims: + # CHECK_VALE_STYLE_SYNC_ALLOW_MISSING_VALE=1 skips all six glob probes + # and says so on a `passed (text-level only, vale unavailable)` line. + # pre-commit prints nothing at all for a passing hook, so without this + # the opt-out reinstated exactly the silent vacuous pass the script was + # written to kill, one level up -- the run showed a bare `Passed` and + # AGENTS.md's instruction to read that summary line was impossible to + # follow in the one situation the opt-out exists for. The script's clean + # output is a single line, so this costs one line per push. - id: check-scope-walkup-sync name: Check scope walk-up implementations agree diff --git a/scripts/check-vale-style-sync.sh b/scripts/check-vale-style-sync.sh index 6eb50f4..f3b7804 100755 --- a/scripts/check-vale-style-sync.sh +++ b/scripts/check-vale-style-sync.sh @@ -77,6 +77,13 @@ fi SKILL_INI="$SKILL_AUDIT/assets/vale/.vale.ini" AGENT_INI="$AGENT_AUDIT/assets/vale/.vale.ini" +# Counted, not assumed. The summary line at the bottom used to hardcode `2 +# .vale.ini file(s) checked` in both branches. That was true on any clean run -- +# a missing or unreadable file errs below and the script never reaches the +# summary -- but the line's whole purpose is to say what this run actually +# inspected, and a constant says what the author expected. Nothing asserted it, +# so it would have survived becoming false. +INIS_CHECKED=0 for ini in "$SKILL_INI" "$AGENT_INI"; do rel_ini="${ini#"$REPO_ROOT"/}" # `-e`, not `-f`: a path that exists but is not a readable regular file (a @@ -106,6 +113,9 @@ for ini in "$SKILL_INI" "$AGENT_INI"; do err "$rel_ini exists but could not be read — none of its assertions could run, and an unreadable file cannot be distinguished from a clean one downstream" continue fi + # Counted here, past both `continue`s: the file exists and its bytes were + # readable, so every assertion below it really does run against it. + INIS_CHECKED=$((INIS_CHECKED + 1)) # StylesPath is resolved relative to the .vale.ini, which is the only reason # the bundled styles are found from a consuming repo's clone prefix. if ! grep -Eq '^[[:space:]]*StylesPath[[:space:]]*=[[:space:]]*styles[[:space:]]*$' "$ini"; then @@ -348,13 +358,27 @@ agent-audit|.claude/agents/demo.md|hooks-only agent-audit|copilot/demo.agent.md|hooks-only EOF_PROBE -# Second floor, on the probe table rather than the directory paths: every probe -# `continue`s when its skill's `.vale.ini` is absent, so a relocation of -# `assets/vale/` alone — with both audit skill directories still in place, which -# the guard at the top of this file therefore cannot see — would skip the entire -# table and leave FAIL at 0. Zero probes checked is never a valid result. +# Second floor, on the probe TABLE rather than on the directory paths. Every row +# `continue`s when the `.vale.ini` of the skill its first column names is absent, +# so the table can verify nothing while FAIL stays 0. Two states do that, and no +# other assertion in this file sees either: +# +# * the `EOF_PROBE` heredoc gutted — a bad merge, a truncated edit, or a +# wholesale delete of the rows. The loop body never runs at all. +# * every row's skill column drifting away from the directory names on disk +# (`skill-audit|` -> `skill-auditX|`), which is what a skill rename plus a +# half-applied find/replace leaves behind. +# +# Both give a clean exit 0 from a section that checked nothing, which is why the +# guard is worth having. What it is NOT reachable by is a relocation of +# `assets/vale/`: PROBES_CHECKED only reaches 0 that way if BOTH `.vale.ini` +# files are gone, and the loop at the top of the `.vale.ini coverage` section +# errs on each of them first, so that state is already FAIL >= 2 and this guard +# is never the cause. The message therefore names the table, not the files — +# describing it as "every probe skill's .vale.ini is missing" misdiagnosed the +# one thing that can actually trigger it. if [[ $PROBES_CHECKED -eq 0 ]]; then - err "no probe path was checked — every probe skill's assets/vale/.vale.ini is missing, so the glob-coverage section verified nothing at all" + err "no probe path was checked — the probe table is empty, or no row's first column names a skill directory under plugins/kyberforge/.apm/skills/ that has an assets/vale/.vale.ini, so the glob-coverage section verified nothing at all" fi if [[ $FAIL -gt 0 ]]; then @@ -367,7 +391,7 @@ fi # do any work against the real repo?" untestable from outside — the counts below # are what tests/test-check-vale-style-sync.sh asserts a non-zero floor on. if [[ "$VALE_AVAILABLE" == true ]]; then - echo "Vale style sync check passed: 2 .vale.ini file(s) checked, $PROBES_CHECKED glob probe(s) verified with vale." + echo "Vale style sync check passed: $INIS_CHECKED .vale.ini file(s) checked, $PROBES_CHECKED glob probe(s) verified with vale." else - echo "Vale style sync check passed (text-level only, vale unavailable): 2 .vale.ini file(s) checked, 0 glob probe(s) verified." + echo "Vale style sync check passed (text-level only, vale unavailable): $INIS_CHECKED .vale.ini file(s) checked, 0 glob probe(s) verified." fi diff --git a/tests/run-bats.sh b/tests/run-bats.sh index d26cb05..1020cf1 100755 --- a/tests/run-bats.sh +++ b/tests/run-bats.sh @@ -32,23 +32,74 @@ done < <( | sort ) -# A floor on the discovered file count, not merely a zero check, and a hard error -# rather than the `exit 0` this used to be. Zero discovered files was the likelier -# of the two silent-green failures -- a moved tests/ tree, a renamed skill -# directory, or the `-not -path` exclusions above widening -- and it exited 0 with -# a note on stderr nobody reads, while the zero-*result* guard further down was -# already a hard error. A floor rather than `-gt 0` because the count collapsing -# to 1 or 2 is the same failure as collapsing to 0 and only a floor names it. -# Same reasoning, and the same "set it a little under the current count" rule, as -# the per-glob floors in tests/test-vale-wrap.sh -- ordinary file churn does not -# trip it, a broken or renamed path does. +# The expected set of files is DERIVED from the index, not guessed at with a +# hardcoded floor. This was `BATS_FILE_FLOOR=8` against a real count of 10, and +# two files of slack is not a hypothetical margin -- deleting two .bats files +# (agentsmd-audit/tests/ alone holds three) left the run reporting +# "155 tests, 0 failures" and exiting 0 with 11 tests silently gone. # -# BATS_FILE_FLOOR overrides it. That override exists for the fixture repos in -# tests/test-run-bats.sh and tests/test-run-tests.sh, which hold one or two .bats -# files by design; it is not an escape hatch for a real run. -BATS_FILE_FLOOR="${BATS_FILE_FLOOR:-8}" -if [[ ${#TEST_FILES[@]} -lt $BATS_FILE_FLOOR ]]; then - echo "Error: found ${#TEST_FILES[@]} .bats file(s) under $REPO_ROOT, below the floor of $BATS_FILE_FLOOR — the search path is wrong or the suite has been gutted" >&2 +# `git ls-files` gives the exact set for free. It catches a *removal* (a tracked +# file gone from the worktree) and an *addition* the walk above missed (a tracked +# file the `find` exclusions or a moved search root no longer reach), it needs no +# magic number, and it needs no edit when a plugin is added or removed -- a newly +# `git add`ed .bats file joins the expectation immediately, where a floor only +# ever grows more slack as the suite grows. +# +# Direction matters: every tracked file must have been discovered, but a +# discovered file need NOT be tracked. An untracked, not-yet-committed .bats file +# is ordinary work in progress, and a file removed deliberately with `git rm` (or +# a staged deletion) leaves the index, so an intentional removal passes while an +# accidental disappearance fails. The same `-not -path` exclusions are reapplied +# to the index listing so the two sides are compared over the same universe. +# +# The exact-equality check on `--show-toplevel` is what keeps this off the +# fixture repos in tests/test-run-bats.sh and tests/test-run-tests.sh: those are +# mktemp trees holding one or two .bats files by design, and git resolves no +# worktree for them. That degradation is announced rather than silent, and the +# zero-file check below is unconditional, so a non-git checkout still cannot run +# on an empty set. +EXPECTED_FILES=() +DERIVED=false +GIT_TOPLEVEL="$(git -C "$REPO_ROOT" rev-parse --show-toplevel 2>/dev/null || true)" +if [[ -n "$GIT_TOPLEVEL" && "$GIT_TOPLEVEL" == "$REPO_ROOT" ]]; then + DERIVED=true + while IFS= read -r f; do + [[ -n "$f" ]] && EXPECTED_FILES+=("$REPO_ROOT/$f") + done < <( + git -C "$REPO_ROOT" ls-files -- '*.bats' \ + | grep -Ev '(^|/)tests/bats/|(^|/)test_helper/|(^|/)\.claude/worktrees/' \ + | sort || true + ) +else + echo "Note: $REPO_ROOT is not a git worktree root, so the expected .bats file set could not be derived from the index — only the zero-file check below applies" >&2 +fi + +if [[ "$DERIVED" == true && ${#EXPECTED_FILES[@]} -gt 0 ]]; then + MISSING=() + for expected in ${EXPECTED_FILES[@]+"${EXPECTED_FILES[@]}"}; do + found=false + for actual in ${TEST_FILES[@]+"${TEST_FILES[@]}"}; do + if [[ "$actual" == "$expected" ]]; then + found=true + break + fi + done + [[ "$found" == true ]] || MISSING+=("${expected#"$REPO_ROOT"/}") + done + if [[ ${#MISSING[@]} -gt 0 ]]; then + echo "Error: ${#MISSING[@]} of ${#EXPECTED_FILES[@]} tracked .bats file(s) were not discovered under $REPO_ROOT — they were deleted without being removed from the index, or the search path/exclusions above no longer reach them:" >&2 + for m in ${MISSING[@]+"${MISSING[@]}"}; do + echo " $m" >&2 + done + exit 1 + fi +fi + +# Unconditional, and separate from the derived check above: a tree with nothing +# tracked (a tarball export, a fresh scaffold) still must not run on an empty set +# and call it green. This was `exit 0` with a note on stderr nobody reads. +if [[ ${#TEST_FILES[@]} -eq 0 ]]; then + echo "Error: found 0 .bats file(s) under $REPO_ROOT — the search path is wrong or the suite has been gutted" >&2 exit 1 fi @@ -119,8 +170,9 @@ for f in ${TEST_FILES[@]+"${TEST_FILES[@]}"}; do fi done -# Zero counted tests is never a clean run: enough files were found (a count under -# the floor, zero included, exits non-zero above), so nothing was executed. Without this, a `bats` that emits +# Zero counted tests is never a clean run: files were found (zero discovered +# files, and any tracked file that went missing, exit non-zero above), so nothing +# was executed. Without this, a `bats` that emits # nothing and exits 0 -- a broken binary, a formatter change, or a wholesale # `@test` removal -- reports "0 tests, 0 failures" and exits green, silently # turning a total harness failure into a pass. diff --git a/tests/run-tests.sh b/tests/run-tests.sh index 1211ff2..4161ab7 100755 --- a/tests/run-tests.sh +++ b/tests/run-tests.sh @@ -1,22 +1,63 @@ #!/usr/bin/env bash # Run all test-*.sh files in the repo (including plugins) and the bats suite. -# Usage: bash tests/run-tests.sh [--bats-only] +# Usage: bash tests/run-tests.sh [--bats-only] [--strict] # # A script exiting 77 (the automake convention) is reported as SKIPPED, not # passed — a suite that can't run for lack of a binary must not read as green. # +# --strict (or RUN_TESTS_STRICT=1) additionally makes any skip FAIL the run. Two +# different readings of a skip are both correct, and which one applies depends on +# who is running: +# +# * ad-hoc, on a laptop: skipping gracefully is the point. You are missing a +# dev binary, the other 15 suites still tell you something, and turning that +# into a red run would just train people to ignore red. +# * as a GATE (the run-tests pre-push hook): a skip is a SETUP ERROR, not a +# legitimate state. AGENTS.md documents vale, apm and jq as required pre-push +# dependencies, so a suite that cannot run on the machine doing the pushing +# means the machine is misconfigured -- and pre-commit prints NOTHING for a +# passing hook, so the skip list below is swallowed entirely. On a vale-less +# PATH that silently shipped a green gate having verified 15 of 17 suites. +# Exactly the vacuous-pass class the rest of this file exists to close. +# +# Deliberately its own switch, NOT folded into +# CHECK_VALE_STYLE_SYNC_ALLOW_MISSING_VALE. That one governs whether +# check-vale-style-sync may downgrade itself; this one governs whether the test +# dispatcher tolerates an unrunnable suite. They are separate decisions and one +# flag disarming both gates is how an opt-out quietly grows blast radius. +# # TEST_DIR — override root to search for test-*.sh (default: REPO_ROOT); used by tests. set -euo pipefail REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" BATS="$REPO_ROOT/tests/run-bats.sh" BATS_ONLY=false -[[ "${1:-}" == "--bats-only" ]] && BATS_ONLY=true +STRICT=false +if [[ "${RUN_TESTS_STRICT:-}" == "1" ]]; then + STRICT=true +fi +# A loop rather than the `[[ "${1:-}" == --bats-only ]]` test this used to be, so +# the two flags compose and an unknown flag is rejected instead of ignored. A +# silently-ignored `--strict` is the one typo that would turn the gate back off. +for arg in ${@+"$@"}; do + case "$arg" in + --bats-only) BATS_ONLY=true ;; + --strict) STRICT=true ;; + *) + echo "Usage: $0 [--bats-only] [--strict]" >&2 + exit 2 + ;; + esac +done SEARCH_ROOT="${TEST_DIR:-$REPO_ROOT}" FAILED=() SKIPPED=() +# Parallel array, index-matched to SKIPPED. Not an associative array: bash 3.2 +# (macOS) has none, and tests/test-vale-wrap.sh's bash-3.2 scan rejects +# `declare -A` outright. +SKIP_REASONS=() PASSED=0 SKIP_EXIT=77 @@ -28,14 +69,43 @@ SKIP_EXIT=77 # degraded to a no-op that printed nothing and exited 0. That is the same # green-either-way hole run-bats.sh's own zero-count guard closes one level down; # this closes it in the dispatcher that pre-push actually invokes. +# +# Present and executable is still not "it ran". `bash "$BATS"` on an EMPTY +# run-bats.sh exits 0 having printed nothing, and the dispatcher printed +# `=== bats ===`, a blank line, and a green summary -- the same green-either-way +# defect one spelling over. Truncation, a partial write, an editor saving an +# empty buffer, and a `set -e` abort in a future run-bats.sh preamble all land +# there. So the runner's own summary line is required, and its count must be +# non-zero: that line is run-bats.sh's contract with this script, and it is only +# emitted after run-bats.sh's own zero-count guard has passed. +# +# Stdout is captured (the summary is on stdout) while stderr passes straight +# through, so a failing runner's diagnostics still reach the terminal live. The +# capture costs no streaming that was not already lost: run-bats.sh buffers its +# per-file output and flushes it at the end regardless. run_bats() { if [[ ! -x "$BATS" ]]; then echo "Error: bats runner not found or not executable at $BATS — the bats suite cannot be skipped silently" >&2 exit 1 fi echo "=== bats ===" - bash "$BATS" + local out rc=0 summary count + out="$(bash "$BATS")" || rc=$? + [[ -z "$out" ]] || printf '%s\n' "$out" echo "" + if [[ $rc -ne 0 ]]; then + exit "$rc" + fi + summary="$(printf '%s\n' "$out" | grep -E '^[0-9]+ tests, [0-9]+ failures$' | tail -n 1 || true)" + if [[ -z "$summary" ]]; then + echo "Error: $BATS exited 0 without reporting an 'N tests, M failures' summary — it ran but produced nothing, so the bats suite was not verified" >&2 + exit 1 + fi + count="${summary%% *}" + if [[ "$count" -eq 0 ]]; then + echo "Error: $BATS reported 0 tests — the bats suite executed nothing" >&2 + exit 1 + fi } if $BATS_ONLY; then @@ -110,6 +180,28 @@ for script in ${SCRIPTS[@]+"${SCRIPTS[@]}"}; do PASSED=$((PASSED + 1)) elif [[ "$rc" == "$SKIP_EXIT" ]]; then SKIPPED+=("$rel") + # Capture WHY, not just that. The reason is printed by the suite itself and + # is otherwise swallowed with the rest of its log, which leaves the reader + # knowing something was skipped but not which binary to install. There is no + # single house format for it -- three suites print `SKIP: ` on stdout + # and one prints `apm not installed -- skipping (...)` on stderr -- so this + # tries the shapes in decreasing order of confidence and falls back to the + # last thing the suite said before exiting 77, which for a guard that exits + # immediately is the reason by construction. batch-run.sh folds stderr into + # the same log, so the stderr spelling is reachable here. + reason="$(grep -E '^[[:space:]]*SKIP' "$SCRATCH_ROOT/$idx.log" 2>/dev/null | head -n 1 || true)" + if [[ -z "$reason" ]]; then + reason="$(grep -iE 'skip' "$SCRATCH_ROOT/$idx.log" 2>/dev/null | head -n 1 || true)" + fi + if [[ -z "$reason" ]]; then + reason="$(grep -vE '^[[:space:]]*$' "$SCRATCH_ROOT/$idx.log" 2>/dev/null | tail -n 1 || true)" + fi + if [[ -z "$reason" ]]; then + reason="(exited $SKIP_EXIT without printing a reason)" + fi + # Trimmed of leading whitespace so the reasons line up under their suite + # names regardless of how each suite indents its own message. + SKIP_REASONS+=("${reason#"${reason%%[![:space:]]*}"}") else FAILED+=("$rel") fi @@ -117,16 +209,42 @@ for script in ${SCRIPTS[@]+"${SCRIPTS[@]}"}; do done echo "=== Summary: $PASSED passed, ${#SKIPPED[@]} skipped, ${#FAILED[@]} failed ===" -if [[ ${#SKIPPED[@]} -gt 0 ]]; then +# Suppressed under --strict: the strict block below reports the same suites with +# the same reasons, and printing both left the reader scrolling past one list to +# reach an identical one. Under strict the failure block IS the list. +if [[ ${#SKIPPED[@]} -gt 0 && "$STRICT" != true ]]; then echo "Skipped scripts:" + sidx=0 for s in ${SKIPPED[@]+"${SKIPPED[@]}"}; do echo " $s" + echo " ${SKIP_REASONS[$sidx]}" + sidx=$((sidx + 1)) done fi + +RC=0 if [[ ${#FAILED[@]} -gt 0 ]]; then echo "Failed scripts:" for s in ${FAILED[@]+"${FAILED[@]}"}; do echo " $s" done - exit 1 + RC=1 fi + +# Strict mode turns every skip into a failure. Reported separately from FAILED +# above rather than folded into it: a skipped suite did not fail, the machine +# did, and a message that says so points at the fix. Named with reasons again +# here (not just referenced) because this block goes to stderr and is what a +# pre-push reader actually gets handed. +if [[ "$STRICT" == true && ${#SKIPPED[@]} -gt 0 ]]; then + echo "Error: --strict and ${#SKIPPED[@]} suite(s) skipped. Run as a gate, a skip is a SETUP ERROR on this machine, not a legitimate state: AGENTS.md documents vale, apm and jq as required pre-push dependencies, so every suite is expected to be runnable here. Install what each suite names below and re-run; do not skip the hook." >&2 + sidx=0 + for s in ${SKIPPED[@]+"${SKIPPED[@]}"}; do + echo " $s" >&2 + echo " ${SKIP_REASONS[$sidx]}" >&2 + sidx=$((sidx + 1)) + done + RC=1 +fi + +exit "$RC" diff --git a/tests/test-check-vale-style-sync.sh b/tests/test-check-vale-style-sync.sh index 2f431a7..69beb69 100755 --- a/tests/test-check-vale-style-sync.sh +++ b/tests/test-check-vale-style-sync.sh @@ -226,49 +226,108 @@ else pass "exits non-zero and reports a stale .apm/ path when plugins/kyberforge exists without it" fi -# --- 5d. Exits 1, saying so, when the probe table verifies nothing --- -# The directory floor above cannot see this one: both audit skill directories are -# still in place, only `assets/vale/` has moved. Every probe then `continue`s on -# its missing .vale.ini and the glob-coverage section checks zero paths. Asserted -# on the message because several other assertions also fire on this fixture. +# --- 5d/5d2. Exits 1, saying so, when the probe TABLE itself verifies nothing --- +# 5d used to relocate `assets/vale/` in both skills, on the belief that doing so +# skipped the whole probe table with FAIL still at 0. It does not. Run against +# the PRE-guard script that fixture already exited 1 with three errors: the +# `.vale.ini` loop errs on both missing files long before the probe loop, and +# PROBES_CHECKED can only reach 0 when both files are gone — which necessarily +# means FAIL >= 2. So it never exercised the guard as a cause, only checked that +# its message showed up beside unrelated failures. +# +# The guard is still worth having, but its real triggers live in the probe table, +# which is part of the script rather than the fixture — so these two cases mutate +# a COPY of the script and run that. Both assert `1 error(s)`, which is what makes +# them real: with the guard deleted each mutation exits 0, and with it present the +# guard is provably the only thing that failed the run. +assert_mutated() { + if diff -q "$SCRIPT" "$1" >/dev/null 2>&1; then + fail "the script mutation changed nothing — the probe table's shape has moved, so this case would pass vacuously" + return 1 + fi +} + echo "" -echo "--- exits 1 and says so when zero glob probes were checked ---" +echo "--- exits 1 and says so when every probe row names a directory that does not exist ---" FIXTURE5D="$(make_fixture)" FIXTURES+=("$FIXTURE5D") -mv "$FIXTURE5D/plugins/kyberforge/.apm/skills/skill-audit/assets/vale" \ - "$FIXTURE5D/plugins/kyberforge/.apm/skills/skill-audit/assets/vale-moved" -mv "$FIXTURE5D/plugins/kyberforge/.apm/skills/agent-audit/assets/vale" \ - "$FIXTURE5D/plugins/kyberforge/.apm/skills/agent-audit/assets/vale-moved" +SCRATCH5D="$(mktemp -d)" +FIXTURES+=("$SCRATCH5D") +sed 's/^skill-audit|/skill-auditX|/; s/^agent-audit|/agent-auditX|/' "$SCRIPT" > "$SCRATCH5D/drifted.sh" NOPROBE_OUT="" NOPROBE_RC=0 -NOPROBE_OUT="$(bash "$SCRIPT" "$FIXTURE5D" 2>&1)" || NOPROBE_RC=$? -if [[ $NOPROBE_RC -eq 0 ]]; then - fail "exited 0 when no glob probe could be checked — expected exit 1" -elif ! printf '%s\n' "$NOPROBE_OUT" | grep -q "no probe path was checked"; then - fail "did not report that zero probe paths were checked: $(printf '%s' "$NOPROBE_OUT" | tr '\n' ' ')" -else - pass "exits non-zero and reports that zero glob probes were checked" +if assert_mutated "$SCRATCH5D/drifted.sh"; then + NOPROBE_OUT="$(bash "$SCRATCH5D/drifted.sh" "$FIXTURE5D" 2>&1)" || NOPROBE_RC=$? + if [[ $NOPROBE_RC -eq 0 ]]; then + fail "a probe table naming no existing skill directory exited 0 — the glob-coverage section checked nothing and reported success" + elif ! printf '%s\n' "$NOPROBE_OUT" | grep -q "no probe path was checked"; then + fail "did not report that zero probe paths were checked: $(printf '%s' "$NOPROBE_OUT" | tr '\n' ' ')" + elif ! printf '%s\n' "$NOPROBE_OUT" | grep -q "failed: 1 error(s)"; then + fail "drifted probe rows failed for reasons beyond the empty probe table, so this guard is not provably what fired: $(printf '%s' "$NOPROBE_OUT" | tr '\n' ' ')" + else + pass "a probe table whose rows name no existing skill directory fails with that guard as the sole error" + fi +fi + +echo "" +echo "--- exits 1 and says so when the probe table is empty ---" +FIXTURE5D2="$(make_fixture)" +FIXTURES+=("$FIXTURE5D2") +SCRATCH5D2="$(mktemp -d)" +FIXTURES+=("$SCRATCH5D2") +# The other reachable trigger: the heredoc gutted outright by a bad merge or a +# truncated edit. `done <<'EOF_PROBE'` with no rows between the delimiters is +# valid bash — the loop body simply never runs. +awk ' + /^done <<.EOF_PROBE.$/ { print; inblk = 1; next } + inblk && /^EOF_PROBE$/ { print; inblk = 0; next } + inblk { next } + { print } +' "$SCRIPT" > "$SCRATCH5D2/gutted.sh" +EMPTYTBL_OUT="" +EMPTYTBL_RC=0 +if assert_mutated "$SCRATCH5D2/gutted.sh"; then + EMPTYTBL_OUT="$(bash "$SCRATCH5D2/gutted.sh" "$FIXTURE5D2" 2>&1)" || EMPTYTBL_RC=$? + if [[ $EMPTYTBL_RC -eq 0 ]]; then + fail "an empty probe table exited 0 — the glob-coverage section verified nothing and reported success" + elif ! printf '%s\n' "$EMPTYTBL_OUT" | grep -q "no probe path was checked"; then + fail "did not report that zero probe paths were checked: $(printf '%s' "$EMPTYTBL_OUT" | tr '\n' ' ')" + elif ! printf '%s\n' "$EMPTYTBL_OUT" | grep -q "failed: 1 error(s)"; then + fail "an empty probe table failed for reasons beyond the guard: $(printf '%s' "$EMPTYTBL_OUT" | tr '\n' ' ')" + else + pass "an emptied probe heredoc fails with that guard as the sole error" + fi fi # --- 5e. Positive: the check does real work against THIS repo --- # Every case above runs against a synthetic fixture, so the whole suite could be # green while the script inspected nothing at all in the repo it is wired into at # pre-push. The summary line carries the counts; assert they are non-zero. +# +# BOTH counts, not just the probe count. The `.vale.ini` half of that line was a +# hardcoded `2` in each branch of the summary — true on any clean run, since a +# missing or unreadable file errs out before the summary is reached, but a +# constant states what the author expected rather than what the run inspected, +# and extracting only the probe count left it asserted by nothing. It is computed +# now, so the count is worth reading and worth pinning. echo "" echo "--- reports a non-zero number of inspected targets against this repo ---" REAL_OUT="" REAL_RC=0 REAL_OUT="$(bash "$SCRIPT" "$REPO_ROOT" 2>&1)" || REAL_RC=$? REAL_PROBES="$(printf '%s\n' "$REAL_OUT" | sed -n 's/.*checked, \([0-9][0-9]*\) glob probe(s).*/\1/p')" +REAL_INIS="$(printf '%s\n' "$REAL_OUT" | sed -n 's/.*: \([0-9][0-9]*\) \.vale\.ini file(s) checked.*/\1/p')" if [[ $REAL_RC -ne 0 ]]; then fail "exited non-zero against this repo's real Vale copies" printf '%s\n' "$REAL_OUT" | sed 's/^/ /' -elif [[ -z "$REAL_PROBES" ]]; then +elif [[ -z "$REAL_PROBES" || -z "$REAL_INIS" ]]; then fail "a clean run against this repo reported no inspected-target counts, so 'it checked something' is unverifiable: $(printf '%s' "$REAL_OUT" | tr '\n' ' ')" elif [[ "$REAL_PROBES" -lt 1 ]]; then fail "a clean run against this repo verified $REAL_PROBES glob probes — a pass that inspected nothing" +elif [[ "$REAL_INIS" -lt 2 ]]; then + fail "a clean run against this repo reported $REAL_INIS .vale.ini file(s) checked — both copies' configs must be inspected" else - pass "inspects $REAL_PROBES glob probe(s) against this repo, and exits 0" + pass "inspects $REAL_INIS .vale.ini file(s) and $REAL_PROBES glob probe(s) against this repo, and exits 0" fi # --- 5b. Exits 1 when REPO_ROOT does not exist --- diff --git a/tests/test-run-bats.sh b/tests/test-run-bats.sh index c9bd38b..0763023 100644 --- a/tests/test-run-bats.sh +++ b/tests/test-run-bats.sh @@ -70,20 +70,33 @@ seed_bats_files() { # Runs the fixture's run-bats.sh, capturing output and exit code separately. # -# BATS_FILE_FLOOR is lowered to 1 for every case that is not specifically about -# the floor: these fixtures hold one or two .bats files by design, which is well -# under the real repo's floor. FAKE_FLOOR lets the floor cases opt back into the -# script's own default. TMPDIR is a private per-run directory so a stub can find -# the scratch dir run-bats.sh mktemp'd for itself -- see case 7. +# No file-count knob is passed any more, and none is needed: the expected file +# set is derived from `git ls-files`, and a mktemp fixture is not a git worktree +# root, so run-bats.sh announces that it could not derive an expectation and +# falls back to the unconditional zero-file check. Cases 8-8c drive the derived +# path deliberately by `git init`-ing their fixtures. +# +# TMPDIR is a private per-run directory so a stub can find the scratch dir +# run-bats.sh mktemp'd for itself -- see case 7. FAKE_OUT="" FAKE_RC=0 -FAKE_FLOOR="" run_fake() { local priv priv="$(mktemp -d)" FIXTURES+=("$priv") FAKE_RC=0 - FAKE_OUT="$(TMPDIR="$priv" BATS_FILE_FLOOR="${FAKE_FLOOR:-1}" bash "$1/tests/run-bats.sh" 2>&1)" || FAKE_RC=$? + FAKE_OUT="$(TMPDIR="$priv" bash "$1/tests/run-bats.sh" 2>&1)" || FAKE_RC=$? +} + +# A fixture whose root IS a git worktree root, so run-bats.sh derives its +# expected set from the index instead of degrading. Only `git add` is used -- +# `git ls-files` reads the index, so nothing needs committing and no user +# identity is required. +make_git_fake_repo() { + local dir + dir="$(make_fake_repo)" + git -C "$dir" init -q + echo "$dir" } # --- 1. A stub emitting nothing at all is a broken harness, not a clean run --- @@ -277,13 +290,15 @@ else pass "an empty status file fails the run despite a healthy TAP stream" fi -# --- 8. Too few discovered .bats files is a hard error, not a green run. Zero -# files used to print a note to stderr and `exit 0`; the floor makes both zero -# and a collapsed-but-nonzero count fail, because a moved tests/ tree or a -# widened `-not -path` exclusion produces exactly that and nothing else notices. +# --- 8. A tracked .bats file the walk did not discover is a hard error. This +# replaces a `BATS_FILE_FLOOR=8` guess against a real count of 10 -- two files of +# slack, which is not hypothetical: deleting two real .bats files left the suite +# reporting "155 tests, 0 failures" and exiting 0 with 11 tests silently gone. +# The expectation is now derived from `git ls-files`, so it is exact and needs no +# magic number. echo "" -echo "--- a discovered-file count under the floor fails the run ---" -DIR8="$(make_fake_repo)" +echo "--- a tracked .bats file missing from the walk fails the run and names it ---" +DIR8="$(make_git_fake_repo)" FIXTURES+=("$DIR8") seed_bats_files "$DIR8" install_stub_bats "$DIR8" <<'EOF' @@ -292,20 +307,92 @@ echo "1..1" echo "ok 1 first" exit 0 EOF -FAKE_FLOOR=8 +git -C "$DIR8" add tests/a.bats tests/b.bats +rm "$DIR8/tests/b.bats" run_fake "$DIR8" -FAKE_FLOOR="" if [[ $FAKE_RC -eq 0 ]]; then - fail "2 .bats files under a floor of 8 passed — a collapsed search path reads as green" -elif echo "$FAKE_OUT" | grep -q "below the floor of 8"; then - pass "a file count under the floor fails and names the floor it missed" + fail "a tracked .bats file gone from the worktree passed — a deleted suite reads as green" +elif ! echo "$FAKE_OUT" | grep -q "tracked .bats file(s) were not discovered"; then + fail "the run failed but not with the undiscovered-tracked-file message: $FAKE_OUT" +elif echo "$FAKE_OUT" | grep -q "^ tests/b.bats$"; then + pass "a tracked .bats file missing from the walk fails the run and names the file" else - fail "the run failed but not with the floor message: $FAKE_OUT" + fail "the run failed without naming the missing file: $FAKE_OUT" fi -# --- 9. Zero discovered .bats files is the same hard error. Kept separate from -# case 8 because it is the state the old `exit 0` branch handled by name, and it -# is the one a path change actually produces. +# --- 8b. An UNTRACKED .bats file is not a finding. The derived expectation runs +# one way only: every tracked file must have been discovered, but a discovered +# file need not be tracked. Without this the check would fail on ordinary +# not-yet-committed work, which is how a correct guard gets disabled. +echo "" +echo "--- an untracked new .bats file does not fail the run ---" +DIR8B="$(make_git_fake_repo)" +FIXTURES+=("$DIR8B") +seed_bats_files "$DIR8B" +install_stub_bats "$DIR8B" <<'EOF' +#!/usr/bin/env bash +echo "1..1" +echo "ok 1 first" +exit 0 +EOF +git -C "$DIR8B" add tests/a.bats +run_fake "$DIR8B" +if [[ $FAKE_RC -ne 0 ]]; then + fail "an untracked .bats file was reported as a finding: $FAKE_OUT" +elif echo "$FAKE_OUT" | grep -q "^2 tests, 0 failures$"; then + pass "an untracked .bats file is run without being demanded of the index" +else + fail "the untracked-file run passed with the wrong count: $FAKE_OUT" +fi + +# --- 8c. A newly added .bats file joins the expectation immediately. This is the +# half a floor can never have: adding files only ever widens a floor's slack, +# while `git add` alone makes the new file required from the next run on, with no +# edit to this script and no number to bump. +echo "" +echo "--- a newly git-added .bats file is required from the next run on ---" +git -C "$DIR8B" add tests/b.bats +rm "$DIR8B/tests/b.bats" +run_fake "$DIR8B" +if [[ $FAKE_RC -eq 0 ]]; then + fail "the .bats file added to the index a moment ago was not demanded back: $FAKE_OUT" +elif echo "$FAKE_OUT" | grep -q "^ tests/b.bats$"; then + pass "a file added to the index joins the expected set with no floor to bump" +else + fail "the run failed but did not name the newly tracked file: $FAKE_OUT" +fi + +# --- 8d. Outside a git worktree the run still works, and says the expectation +# could not be derived. That degradation is what every other fixture here relies +# on, and it must be announced rather than silent -- an unannounced fallback is +# how a derived check quietly becomes no check at all on a tarball export. +echo "" +echo "--- a non-git tree runs, and announces that no expectation could be derived ---" +DIR8D="$(make_fake_repo)" +FIXTURES+=("$DIR8D") +seed_bats_files "$DIR8D" +install_stub_bats "$DIR8D" <<'EOF' +#!/usr/bin/env bash +echo "1..1" +echo "ok 1 first" +exit 0 +EOF +run_fake "$DIR8D" +if [[ $FAKE_RC -ne 0 ]]; then + fail "a non-git tree failed the run: $FAKE_OUT" +elif ! echo "$FAKE_OUT" | grep -q "not a git worktree root"; then + fail "a non-git tree silently skipped the derived expectation with no note: $FAKE_OUT" +elif echo "$FAKE_OUT" | grep -q "^2 tests, 0 failures$"; then + pass "a non-git tree runs the suite and says the expected set could not be derived" +else + fail "the non-git run passed with the wrong count: $FAKE_OUT" +fi + +# --- 9. Zero discovered .bats files is a hard error regardless, and is checked +# unconditionally rather than through the derived set: a tree with nothing +# tracked at all (a tarball export, a fresh scaffold) must still not run on an +# empty set and call it green. It is the state the old `exit 0` branch handled by +# name, and the one a path change actually produces. echo "" echo "--- zero discovered .bats files fails the run ---" DIR9="$(make_fake_repo)" diff --git a/tests/test-run-tests.sh b/tests/test-run-tests.sh index 3b9f87c..d3e80ed 100644 --- a/tests/test-run-tests.sh +++ b/tests/test-run-tests.sh @@ -8,6 +8,11 @@ # moving, or dropping the executable bit off run-bats.sh made the entire bats # suite disappear with no diagnostic while the run printed a green summary and # exited 0, and turned `--bats-only` into a no-op that printed nothing. +# * run_bats() then checked only that run-bats.sh was present and executable, +# never that it PRODUCED anything. `bash` on an empty run-bats.sh exits 0 +# having printed nothing, so the dispatcher printed `=== bats ===` and a green +# summary. The runner's `N tests, M failures` line is now required, with a +# non-zero count. # * The per-script status was compared with `-eq`, which is arithmetic, and bash # evaluates an empty string as 0 there -- so a status file that existed but was # empty counted as a pass. @@ -191,6 +196,67 @@ else pass "a failing bats runner propagates out of run-tests.sh" fi +# --- 5b. An EMPTY run-bats.sh is a hard error, not a green no-op --- +# Present and executable was still not "it ran". `bash` on a zero-byte script +# exits 0 having printed nothing, so the dispatcher printed `=== bats ===`, a +# blank line, and `Summary: 1 passed, 0 skipped, 0 failed` with rc=0 -- the whole +# bats suite gone, exactly the defect cases 2-4 close for the other spellings. +# Truncation, a partial write, an editor saving an empty buffer, and a `set -e` +# abort in a run-bats.sh preamble all produce this file. +# +# It only failed on the real repo incidentally, because tests/test-run-bats.sh +# copies run-bats.sh into its own fixtures and blows up there; rename or retire +# that file and the hole is live in the gate pre-push invokes. This asserts it +# directly. +echo "" +echo "--- an empty run-bats.sh fails the run instead of passing silently ---" +DIR5B="$(make_fake_repo)" +FIXTURES+=("$DIR5B") +: > "$DIR5B/tests/run-bats.sh" +chmod +x "$DIR5B/tests/run-bats.sh" +add_case "$DIR5B" test-ok.sh <<'EOF' +#!/usr/bin/env bash +echo "fine" +EOF +run_fake "$DIR5B" +if echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$"; then + fail "an empty run-bats.sh produced a green summary — the bats suite vanished with no diagnostic" +elif [[ $FAKE_RC -eq 0 ]]; then + fail "an empty run-bats.sh exited 0: $FAKE_OUT" +elif echo "$FAKE_OUT" | grep -q "without reporting an 'N tests, M failures' summary"; then + pass "an empty run-bats.sh fails the run and says the bats suite was never verified" +else + fail "the run failed but not with the no-summary message: $FAKE_OUT" +fi + +# --- 5c. A runner reporting zero tests is a hard error too --- +# The other half of "ran but produced nothing": the summary line is there and the +# process exits 0, but it accounts for no tests. run-bats.sh has its own guard for +# this one file down; asserting it here means the dispatcher does not depend on +# that guard surviving, and it pins the count as the thing being read rather than +# the mere presence of a line matching the pattern. +echo "" +echo "--- a bats runner reporting 0 tests fails the run ---" +DIR5C="$(make_fake_repo)" +FIXTURES+=("$DIR5C") +install_stub_bats_runner "$DIR5C" <<'EOF' +#!/usr/bin/env bash +echo "0 tests, 0 failures" +exit 0 +EOF +add_case "$DIR5C" test-ok.sh <<'EOF' +#!/usr/bin/env bash +echo "fine" +EOF +run_fake "$DIR5C" +if [[ $FAKE_RC -eq 0 ]]; then + fail "a bats runner reporting 0 tests exited 0 — a suite that executed nothing read as green: $FAKE_OUT" +elif echo "$FAKE_OUT" | grep -q "reported 0 tests"; then + pass "a bats runner reporting 0 tests fails the run and says the suite executed nothing" +else + fail "the run failed but not with the zero-tests message: $FAKE_OUT" +fi + # --- 6. An empty status file is FAILED, not a pass --- # The status is read back with `cat ... || echo 1`, which covers a *missing* # file. A file that exists but is empty is what a job killed between the `>` @@ -311,6 +377,153 @@ else fail "the failed script was not named in the failure list: $FAKE_OUT" fi +# --- 10. --strict turns a skip into a failure, and names the suite AND the reason --- +# Graceful skipping is right for an ad-hoc run and wrong for a gate. At pre-push a +# suite exiting 77 means a dependency AGENTS.md documents as required is missing +# on the pushing machine -- and pre-commit prints nothing at all for a passing +# hook, so the skip list this script writes to stdout was swallowed whole. A +# vale-less PATH shipped a green gate having verified 15 of 17 suites. +# +# The reason is asserted, not just the name: "something was skipped" leaves the +# reader with no idea which binary to install, which is most of why the swallowed +# list was worth so little in the first place. Matched on the SIX-SPACE INDENT the +# report writes, not on the reason text alone -- the suite's own log is echoed +# back verbatim earlier in the same output, so a bare text match passes even with +# the reason capture deleted. Verified: narrowing the capture to the `SKIP:` +# prefix left the loose form green. +echo "" +echo "--- --strict fails the run on a skipped suite and names it with its reason ---" +DIR10="$(make_fake_repo)" +FIXTURES+=("$DIR10") +install_healthy_bats_runner "$DIR10" +add_case "$DIR10" test-needs-a-binary.sh <<'EOF' +#!/usr/bin/env bash +echo "SKIP: frobnicator is not installed — install it from https://example.invalid" +exit 77 +EOF +add_case "$DIR10" test-ok.sh <<'EOF' +#!/usr/bin/env bash +echo "fine" +EOF +run_fake "$DIR10" --strict +if [[ $FAKE_RC -eq 0 ]]; then + fail "--strict passed with a skipped suite — the gate reports green having verified less than it ran: $FAKE_OUT" +elif ! echo "$FAKE_OUT" | grep -q "a skip is a SETUP ERROR"; then + fail "--strict failed but never said a skip is a setup error: $FAKE_OUT" +elif ! echo "$FAKE_OUT" | grep -q "test-needs-a-binary.sh"; then + fail "--strict failed without naming the skipped suite: $FAKE_OUT" +elif echo "$FAKE_OUT" | grep -q "^ SKIP: frobnicator is not installed"; then + pass "--strict fails on a skip, names the suite, and carries through the reason it printed" +else + fail "--strict named the suite but swallowed its skip reason: $FAKE_OUT" +fi + +# --- 10b. RUN_TESTS_STRICT=1 is the same switch. The hook uses the flag because +# it is self-documenting in .pre-commit-config.yaml; the env var exists for a CI +# runner that cannot edit the command line. Both are asserted so one cannot rot. +echo "" +echo "--- RUN_TESTS_STRICT=1 fails the run on a skipped suite ---" +STRICT_ENV_OUT="" +STRICT_ENV_RC=0 +STRICT_ENV_PRIV="$(mktemp -d)" +FIXTURES+=("$STRICT_ENV_PRIV") +STRICT_ENV_OUT="$(TMPDIR="$STRICT_ENV_PRIV" TEST_DIR="$DIR10/cases" RUN_TESTS_STRICT=1 \ + bash "$DIR10/tests/run-tests.sh" 2>&1)" || STRICT_ENV_RC=$? +if [[ $STRICT_ENV_RC -eq 0 ]]; then + fail "RUN_TESTS_STRICT=1 passed with a skipped suite: $STRICT_ENV_OUT" +elif echo "$STRICT_ENV_OUT" | grep -q "a skip is a SETUP ERROR"; then + pass "RUN_TESTS_STRICT=1 is the same gate as --strict" +else + fail "RUN_TESTS_STRICT=1 failed for some other reason: $STRICT_ENV_OUT" +fi + +# --- 10c. WITHOUT strict, the same fixture still skips gracefully and passes --- +# The control for 10 and 10b, and the half the coordinator asked for explicitly: +# an ad-hoc `bash tests/run-tests.sh` on a laptop missing a dev binary must not +# go red. Without this, "fix the gate" could quietly mean "fail everywhere". +echo "" +echo "--- the same skipped suite passes, still SKIPPED, without strict ---" +run_fake "$DIR10" +if [[ $FAKE_RC -ne 0 ]]; then + fail "a skipped suite failed a non-strict run — graceful skipping is gone: $FAKE_OUT" +elif ! echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 1 skipped, 0 failed ===$"; then + fail "a non-strict run miscounted the skip: $FAKE_OUT" +elif echo "$FAKE_OUT" | grep -q "^ SKIP: frobnicator is not installed"; then + pass "without strict the suite is SKIPPED, the run passes, and the reason is still reported" +else + fail "a non-strict run passed but dropped the skip reason: $FAKE_OUT" +fi + +# --- 10d. --strict does not become a blanket failure --- +# The case that proves 10 and 10b fail for their stated reason. A clean run with +# nothing skipped must be just as green under --strict as without it, otherwise +# the gate is not a gate, it is an outage. +echo "" +echo "--- --strict is still green when nothing skipped ---" +DIR10D="$(make_fake_repo)" +FIXTURES+=("$DIR10D") +install_healthy_bats_runner "$DIR10D" +add_case "$DIR10D" test-ok.sh <<'EOF' +#!/usr/bin/env bash +echo "fine" +EOF +run_fake "$DIR10D" --strict +if [[ $FAKE_RC -ne 0 ]]; then + fail "--strict failed a run with nothing skipped — it fails unconditionally: $FAKE_OUT" +elif echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$"; then + pass "--strict leaves a run with no skips green" +else + fail "--strict passed with the wrong summary: $FAKE_OUT" +fi + +# --- 10e. An unknown flag is rejected, not ignored --- +# `--strict` reaching this script as a silently-ignored argument is the single +# typo that turns the gate back off while every hook still reports Passed, so the +# arg loop refuses anything it does not know rather than falling through. +echo "" +echo "--- an unrecognised flag fails with usage instead of being ignored ---" +DIR10E="$(make_fake_repo)" +FIXTURES+=("$DIR10E") +install_healthy_bats_runner "$DIR10E" +add_case "$DIR10E" test-ok.sh <<'EOF' +#!/usr/bin/env bash +echo "fine" +EOF +run_fake "$DIR10E" --strickt +if [[ $FAKE_RC -eq 0 ]]; then + fail "a misspelled flag was ignored and the run passed — a typo silently disarms the gate: $FAKE_OUT" +elif echo "$FAKE_OUT" | grep -q "Usage: .*--bats-only.*--strict"; then + pass "an unrecognised flag fails the run with usage" +else + fail "an unrecognised flag failed but not with usage: $FAKE_OUT" +fi + +# --- 10f. A skip reason printed on STDERR, with no `SKIP:` prefix, still lands --- +# There is no house format: three suites print `SKIP: ` on stdout and +# tests/test-sync-plugin-content.sh prints `apm not installed -- skipping (...)` +# on stderr. batch-run.sh folds stderr into the same log, so both are reachable, +# but only a fallback chain finds the second one. Without this case the reason +# extraction could be narrowed to the `SKIP:` prefix and the apm suite would fail +# the gate with no indication of what to install. +echo "" +echo "--- a stderr skip reason with no SKIP: prefix is still reported ---" +DIR10F="$(make_fake_repo)" +FIXTURES+=("$DIR10F") +install_healthy_bats_runner "$DIR10F" +add_case "$DIR10F" test-stderr-skip.sh <<'EOF' +#!/usr/bin/env bash +echo "widgetizer not installed -- skipping (see docs)" >&2 +exit 77 +EOF +run_fake "$DIR10F" --strict +if [[ $FAKE_RC -eq 0 ]]; then + fail "--strict passed on a suite that skipped via stderr: $FAKE_OUT" +elif echo "$FAKE_OUT" | grep -q "^ widgetizer not installed -- skipping"; then + pass "a skip reason printed to stderr without a SKIP: prefix is still carried into the failure" +else + fail "--strict failed but lost the stderr skip reason: $FAKE_OUT" +fi + echo "" echo "Results: $PASS passed, $FAIL failed" [[ $FAIL -eq 0 ]]