Defame1297/holocron
0
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases 3 Wiki Activity

test: fail the gate when a suite is skipped or never reports

Browse Source 
Three gaps left by the previous round, all the same shape: a gate reporting
success having verified less than it appears to.

run_bats() hard-failed on a missing or non-executable runner but never checked
that the runner produced anything. An empty, executable run-bats.sh exits 0,
and the dispatcher printed a green summary with 166 bats tests silently absent.
It now requires an "N tests, M failures" line with a non-zero count.

run-tests.sh's skip listing is swallowed by pre-commit on a pass, so on a
machine without vale three suites exited 77 and the pre-push gate went green
having run 14 of 17. The hook now invokes it as --strict, where a skip fails
and the error names each suite and the reason it skipped. An ad-hoc local run
still skips gracefully -- at pre-push a skip means a documented dependency is
missing, which is a setup error, not a legitimate state. Deliberately not wired
to the vale downgrade's env var: one flag must not disarm two gates.

BATS_FILE_FLOOR is replaced by an expectation derived from git ls-files. A
floor of 8 against a real count of 10 let two files and eleven tests disappear
green, and the number needed an edit whenever a plugin was added. The derived
set needs no number, and catches an addition as well as a removal -- a .bats
file staged into the index and deleted from disk is now demanded back.

The vale opt-out announced its downgrade to nobody: pre-commit prints nothing
for a passing hook, so the summary line AGENTS.md tells the reader to check was
unreachable in exactly the situation it exists for. The hook is now verbose.

Also corrects the PROBES_CHECKED guard, whose commit message described a state
that cannot occur -- the .vale.ini loop errs first. Its two reachable triggers,
a gutted probe heredoc and a probe row naming a missing directory, had no test;
they do now, each asserting the guard is the sole cause.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X7GvKuJfy2WrdBmUttV4DT
This commit is contained in:
Defame1297
2026-08-14 12:29:42 +00:00
parent 874bf06b18
commit aa15fc850c
7 changed files with 637 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
.pre-commit-config.yaml
View File

@@ -46,8 +46,8 @@ repos:
hooks:
- id: run-tests
name: Run test suite
description: Run all test-*.sh files and bats suite
entry: bash tests/run-tests.sh
description: Run all test-*.sh files and bats suite. --strict because a suite that exits 77 (SKIPPED) at pre-push means a documented dependency is missing on this machine, and pre-commit prints nothing for a passing hook -- without it the gate went green having verified 15 of 17 suites on a vale-less PATH, with the skip list swallowed. Ad-hoc `bash tests/run-tests.sh` still skips gracefully.
entry: bash tests/run-tests.sh --strict
language: system
stages: [pre-push]
pass_filenames: false
@@ -115,6 +115,16 @@ repos:
stages: [pre-push]
pass_filenames: false
always_run: true
# verbose so the DOWNGRADED run is audible. This hook can pass while
# having verified strictly less than its name claims:
# CHECK_VALE_STYLE_SYNC_ALLOW_MISSING_VALE=1 skips all six glob probes
# and says so on a `passed (text-level only, vale unavailable)` line.
# pre-commit prints nothing at all for a passing hook, so without this
# the opt-out reinstated exactly the silent vacuous pass the script was
# written to kill, one level up -- the run showed a bare `Passed` and
# AGENTS.md's instruction to read that summary line was impossible to
# follow in the one situation the opt-out exists for. The script's clean
# output is a single line, so this costs one line per push.
- id: check-scope-walkup-sync
name: Check scope walk-up implementations agree

40
scripts/check-vale-style-sync.sh
View File

@@ -77,6 +77,13 @@ fi
SKILL_INI="$SKILL_AUDIT/assets/vale/.vale.ini"
AGENT_INI="$AGENT_AUDIT/assets/vale/.vale.ini"
# Counted, not assumed. The summary line at the bottom used to hardcode `2
# .vale.ini file(s) checked` in both branches. That was true on any clean run --
# a missing or unreadable file errs below and the script never reaches the
# summary -- but the line's whole purpose is to say what this run actually
# inspected, and a constant says what the author expected. Nothing asserted it,
# so it would have survived becoming false.
INIS_CHECKED=0
for ini in "$SKILL_INI" "$AGENT_INI"; do
rel_ini="${ini#"$REPO_ROOT"/}"
# `-e`, not `-f`: a path that exists but is not a readable regular file (a
@@ -106,6 +113,9 @@ for ini in "$SKILL_INI" "$AGENT_INI"; do
err "$rel_ini exists but could not be read — none of its assertions could run, and an unreadable file cannot be distinguished from a clean one downstream"
continue
fi
# Counted here, past both `continue`s: the file exists and its bytes were
# readable, so every assertion below it really does run against it.
INIS_CHECKED=$((INIS_CHECKED + 1))
# StylesPath is resolved relative to the .vale.ini, which is the only reason
# the bundled styles are found from a consuming repo's clone prefix.
if ! grep -Eq '^[[:space:]]*StylesPath[[:space:]]*=[[:space:]]*styles[[:space:]]*$' "$ini"; then
@@ -348,13 +358,27 @@ agent-audit|.claude/agents/demo.md|hooks-only
agent-audit|copilot/demo.agent.md|hooks-only
EOF_PROBE
# Second floor, on the probe table rather than the directory paths: every probe
# `continue`s when its skill's `.vale.ini` is absent, so a relocation of
# `assets/vale/` alone — with both audit skill directories still in place, which
# the guard at the top of this file therefore cannot see — would skip the entire
# table and leave FAIL at 0. Zero probes checked is never a valid result.
# Second floor, on the probe TABLE rather than on the directory paths. Every row
# `continue`s when the `.vale.ini` of the skill its first column names is absent,
# so the table can verify nothing while FAIL stays 0. Two states do that, and no
# other assertion in this file sees either:
#
# * the `EOF_PROBE` heredoc gutted — a bad merge, a truncated edit, or a
# wholesale delete of the rows. The loop body never runs at all.
# * every row's skill column drifting away from the directory names on disk
# (`skill-audit|` -> `skill-auditX|`), which is what a skill rename plus a
# half-applied find/replace leaves behind.
#
# Both give a clean exit 0 from a section that checked nothing, which is why the
# guard is worth having. What it is NOT reachable by is a relocation of
# `assets/vale/`: PROBES_CHECKED only reaches 0 that way if BOTH `.vale.ini`
# files are gone, and the loop at the top of the `.vale.ini coverage` section
# errs on each of them first, so that state is already FAIL >= 2 and this guard
# is never the cause. The message therefore names the table, not the files —
# describing it as "every probe skill's .vale.ini is missing" misdiagnosed the
# one thing that can actually trigger it.
if [[ $PROBES_CHECKED -eq 0 ]]; then
err "no probe path was checked — every probe skill's assets/vale/.vale.ini is missing, so the glob-coverage section verified nothing at all"
err "no probe path was checked — the probe table is empty, or no row's first column names a skill directory under plugins/kyberforge/.apm/skills/ that has an assets/vale/.vale.ini, so the glob-coverage section verified nothing at all"
fi
if [[ $FAIL -gt 0 ]]; then
@@ -367,7 +391,7 @@ fi
# do any work against the real repo?" untestable from outside — the counts below
# are what tests/test-check-vale-style-sync.sh asserts a non-zero floor on.
if [[ "$VALE_AVAILABLE" == true ]]; then
echo "Vale style sync check passed: 2 .vale.ini file(s) checked, $PROBES_CHECKED glob probe(s) verified with vale."
echo "Vale style sync check passed: $INIS_CHECKED .vale.ini file(s) checked, $PROBES_CHECKED glob probe(s) verified with vale."
else
echo "Vale style sync check passed (text-level only, vale unavailable): 2 .vale.ini file(s) checked, 0 glob probe(s) verified."
echo "Vale style sync check passed (text-level only, vale unavailable): $INIS_CHECKED .vale.ini file(s) checked, 0 glob probe(s) verified."
fi

88
tests/run-bats.sh
View File

@@ -32,23 +32,74 @@ done < <(
| sort
)
# A floor on the discovered file count, not merely a zero check, and a hard error
# rather than the `exit 0` this used to be. Zero discovered files was the likelier
# of the two silent-green failures -- a moved tests/ tree, a renamed skill
# directory, or the `-not -path` exclusions above widening -- and it exited 0 with
# a note on stderr nobody reads, while the zero-*result* guard further down was
# already a hard error. A floor rather than `-gt 0` because the count collapsing
# to 1 or 2 is the same failure as collapsing to 0 and only a floor names it.
# Same reasoning, and the same "set it a little under the current count" rule, as
# the per-glob floors in tests/test-vale-wrap.sh -- ordinary file churn does not
# trip it, a broken or renamed path does.
# The expected set of files is DERIVED from the index, not guessed at with a
# hardcoded floor. This was `BATS_FILE_FLOOR=8` against a real count of 10, and
# two files of slack is not a hypothetical margin -- deleting two .bats files
# (agentsmd-audit/tests/ alone holds three) left the run reporting
# "155 tests, 0 failures" and exiting 0 with 11 tests silently gone.
#
# BATS_FILE_FLOOR overrides it. That override exists for the fixture repos in
# tests/test-run-bats.sh and tests/test-run-tests.sh, which hold one or two .bats
# files by design; it is not an escape hatch for a real run.
BATS_FILE_FLOOR="${BATS_FILE_FLOOR:-8}"
if [[ ${#TEST_FILES[@]} -lt $BATS_FILE_FLOOR ]]; then
echo "Error: found ${#TEST_FILES[@]} .bats file(s) under $REPO_ROOT, below the floor of $BATS_FILE_FLOOR — the search path is wrong or the suite has been gutted" >&2
# `git ls-files` gives the exact set for free. It catches a *removal* (a tracked
# file gone from the worktree) and an *addition* the walk above missed (a tracked
# file the `find` exclusions or a moved search root no longer reach), it needs no
# magic number, and it needs no edit when a plugin is added or removed -- a newly
# `git add`ed .bats file joins the expectation immediately, where a floor only
# ever grows more slack as the suite grows.
#
# Direction matters: every tracked file must have been discovered, but a
# discovered file need NOT be tracked. An untracked, not-yet-committed .bats file
# is ordinary work in progress, and a file removed deliberately with `git rm` (or
# a staged deletion) leaves the index, so an intentional removal passes while an
# accidental disappearance fails. The same `-not -path` exclusions are reapplied
# to the index listing so the two sides are compared over the same universe.
#
# The exact-equality check on `--show-toplevel` is what keeps this off the
# fixture repos in tests/test-run-bats.sh and tests/test-run-tests.sh: those are
# mktemp trees holding one or two .bats files by design, and git resolves no
# worktree for them. That degradation is announced rather than silent, and the
# zero-file check below is unconditional, so a non-git checkout still cannot run
# on an empty set.
EXPECTED_FILES=()
DERIVED=false
GIT_TOPLEVEL="$(git -C "$REPO_ROOT" rev-parse --show-toplevel 2>/dev/null || true)"
if [[ -n "$GIT_TOPLEVEL" && "$GIT_TOPLEVEL" == "$REPO_ROOT" ]]; then
DERIVED=true
while IFS= read -r f; do
[[ -n "$f" ]] && EXPECTED_FILES+=("$REPO_ROOT/$f")
done < <(
git -C "$REPO_ROOT" ls-files -- '*.bats' \
| grep -Ev '(^|/)tests/bats/|(^|/)test_helper/|(^|/)\.claude/worktrees/' \
| sort || true
)
else
echo "Note: $REPO_ROOT is not a git worktree root, so the expected .bats file set could not be derived from the index — only the zero-file check below applies" >&2
fi
if [[ "$DERIVED" == true && ${#EXPECTED_FILES[@]} -gt 0 ]]; then
MISSING=()
for expected in ${EXPECTED_FILES[@]+"${EXPECTED_FILES[@]}"}; do
found=false
for actual in ${TEST_FILES[@]+"${TEST_FILES[@]}"}; do
if [[ "$actual" == "$expected" ]]; then
found=true
break
fi
done
[[ "$found" == true ]] || MISSING+=("${expected#"$REPO_ROOT"/}")
done
if [[ ${#MISSING[@]} -gt 0 ]]; then
echo "Error: ${#MISSING[@]} of ${#EXPECTED_FILES[@]} tracked .bats file(s) were not discovered under $REPO_ROOT — they were deleted without being removed from the index, or the search path/exclusions above no longer reach them:" >&2
for m in ${MISSING[@]+"${MISSING[@]}"}; do
echo " $m" >&2
done
exit 1
fi
fi
# Unconditional, and separate from the derived check above: a tree with nothing
# tracked (a tarball export, a fresh scaffold) still must not run on an empty set
# and call it green. This was `exit 0` with a note on stderr nobody reads.
if [[ ${#TEST_FILES[@]} -eq 0 ]]; then
echo "Error: found 0 .bats file(s) under $REPO_ROOT — the search path is wrong or the suite has been gutted" >&2
exit 1
fi
@@ -119,8 +170,9 @@ for f in ${TEST_FILES[@]+"${TEST_FILES[@]}"}; do
fi
done
# Zero counted tests is never a clean run: enough files were found (a count under
# the floor, zero included, exits non-zero above), so nothing was executed. Without this, a `bats` that emits
# Zero counted tests is never a clean run: files were found (zero discovered
# files, and any tracked file that went missing, exit non-zero above), so nothing
# was executed. Without this, a `bats` that emits
# nothing and exits 0 -- a broken binary, a formatter change, or a wholesale
# `@test` removal -- reports "0 tests, 0 failures" and exits green, silently
# turning a total harness failure into a pass.

128
tests/run-tests.sh
View File

@@ -1,22 +1,63 @@
#!/usr/bin/env bash
# Run all test-*.sh files in the repo (including plugins) and the bats suite.
# Usage: bash tests/run-tests.sh [--bats-only]
# Usage: bash tests/run-tests.sh [--bats-only] [--strict]
#
# A script exiting 77 (the automake convention) is reported as SKIPPED, not
# passed — a suite that can't run for lack of a binary must not read as green.
#
# --strict (or RUN_TESTS_STRICT=1) additionally makes any skip FAIL the run. Two
# different readings of a skip are both correct, and which one applies depends on
# who is running:
#
# * ad-hoc, on a laptop: skipping gracefully is the point. You are missing a
# dev binary, the other 15 suites still tell you something, and turning that
# into a red run would just train people to ignore red.
# * as a GATE (the run-tests pre-push hook): a skip is a SETUP ERROR, not a
# legitimate state. AGENTS.md documents vale, apm and jq as required pre-push
# dependencies, so a suite that cannot run on the machine doing the pushing
# means the machine is misconfigured -- and pre-commit prints NOTHING for a
# passing hook, so the skip list below is swallowed entirely. On a vale-less
# PATH that silently shipped a green gate having verified 15 of 17 suites.
# Exactly the vacuous-pass class the rest of this file exists to close.
#
# Deliberately its own switch, NOT folded into
# CHECK_VALE_STYLE_SYNC_ALLOW_MISSING_VALE. That one governs whether
# check-vale-style-sync may downgrade itself; this one governs whether the test
# dispatcher tolerates an unrunnable suite. They are separate decisions and one
# flag disarming both gates is how an opt-out quietly grows blast radius.
#
# TEST_DIR — override root to search for test-*.sh (default: REPO_ROOT); used by tests.
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
BATS="$REPO_ROOT/tests/run-bats.sh"
BATS_ONLY=false
[[ "${1:-}" == "--bats-only" ]] && BATS_ONLY=true
STRICT=false
if [[ "${RUN_TESTS_STRICT:-}" == "1" ]]; then
STRICT=true
fi
# A loop rather than the `[[ "${1:-}" == --bats-only ]]` test this used to be, so
# the two flags compose and an unknown flag is rejected instead of ignored. A
# silently-ignored `--strict` is the one typo that would turn the gate back off.
for arg in ${@+"$@"}; do
case "$arg" in
--bats-only) BATS_ONLY=true ;;
--strict) STRICT=true ;;
*)
echo "Usage: $0 [--bats-only] [--strict]" >&2
exit 2
;;
esac
done
SEARCH_ROOT="${TEST_DIR:-$REPO_ROOT}"
FAILED=()
SKIPPED=()
# Parallel array, index-matched to SKIPPED. Not an associative array: bash 3.2
# (macOS) has none, and tests/test-vale-wrap.sh's bash-3.2 scan rejects
# `declare -A` outright.
SKIP_REASONS=()
PASSED=0
SKIP_EXIT=77
@@ -28,14 +69,43 @@ SKIP_EXIT=77
# degraded to a no-op that printed nothing and exited 0. That is the same
# green-either-way hole run-bats.sh's own zero-count guard closes one level down;
# this closes it in the dispatcher that pre-push actually invokes.
#
# Present and executable is still not "it ran". `bash "$BATS"` on an EMPTY
# run-bats.sh exits 0 having printed nothing, and the dispatcher printed
# `=== bats ===`, a blank line, and a green summary -- the same green-either-way
# defect one spelling over. Truncation, a partial write, an editor saving an
# empty buffer, and a `set -e` abort in a future run-bats.sh preamble all land
# there. So the runner's own summary line is required, and its count must be
# non-zero: that line is run-bats.sh's contract with this script, and it is only
# emitted after run-bats.sh's own zero-count guard has passed.
#
# Stdout is captured (the summary is on stdout) while stderr passes straight
# through, so a failing runner's diagnostics still reach the terminal live. The
# capture costs no streaming that was not already lost: run-bats.sh buffers its
# per-file output and flushes it at the end regardless.
run_bats() {
if [[ ! -x "$BATS" ]]; then
echo "Error: bats runner not found or not executable at $BATS — the bats suite cannot be skipped silently" >&2
exit 1
fi
echo "=== bats ==="
bash "$BATS"
local out rc=0 summary count
out="$(bash "$BATS")" || rc=$?
[[ -z "$out" ]] || printf '%s\n' "$out"
echo ""
if [[ $rc -ne 0 ]]; then
exit "$rc"
fi
summary="$(printf '%s\n' "$out" | grep -E '^[0-9]+ tests, [0-9]+ failures$' | tail -n 1 || true)"
if [[ -z "$summary" ]]; then
echo "Error: $BATS exited 0 without reporting an 'N tests, M failures' summary — it ran but produced nothing, so the bats suite was not verified" >&2
exit 1
fi
count="${summary%% *}"
if [[ "$count" -eq 0 ]]; then
echo "Error: $BATS reported 0 tests — the bats suite executed nothing" >&2
exit 1
fi
}
if $BATS_ONLY; then
@@ -110,6 +180,28 @@ for script in ${SCRIPTS[@]+"${SCRIPTS[@]}"}; do
PASSED=$((PASSED + 1))
elif [[ "$rc" == "$SKIP_EXIT" ]]; then
SKIPPED+=("$rel")
# Capture WHY, not just that. The reason is printed by the suite itself and
# is otherwise swallowed with the rest of its log, which leaves the reader
# knowing something was skipped but not which binary to install. There is no
# single house format for it -- three suites print `SKIP: <reason>` on stdout
# and one prints `apm not installed -- skipping (...)` on stderr -- so this
# tries the shapes in decreasing order of confidence and falls back to the
# last thing the suite said before exiting 77, which for a guard that exits
# immediately is the reason by construction. batch-run.sh folds stderr into
# the same log, so the stderr spelling is reachable here.
reason="$(grep -E '^[[:space:]]*SKIP' "$SCRATCH_ROOT/$idx.log" 2>/dev/null | head -n 1 || true)"
if [[ -z "$reason" ]]; then
reason="$(grep -iE 'skip' "$SCRATCH_ROOT/$idx.log" 2>/dev/null | head -n 1 || true)"
fi
if [[ -z "$reason" ]]; then
reason="$(grep -vE '^[[:space:]]*$' "$SCRATCH_ROOT/$idx.log" 2>/dev/null | tail -n 1 || true)"
fi
if [[ -z "$reason" ]]; then
reason="(exited $SKIP_EXIT without printing a reason)"
fi
# Trimmed of leading whitespace so the reasons line up under their suite
# names regardless of how each suite indents its own message.
SKIP_REASONS+=("${reason#"${reason%%[![:space:]]*}"}")
else
FAILED+=("$rel")
fi
@@ -117,16 +209,42 @@ for script in ${SCRIPTS[@]+"${SCRIPTS[@]}"}; do
done
echo "=== Summary: $PASSED passed, ${#SKIPPED[@]} skipped, ${#FAILED[@]} failed ==="
if [[ ${#SKIPPED[@]} -gt 0 ]]; then
# Suppressed under --strict: the strict block below reports the same suites with
# the same reasons, and printing both left the reader scrolling past one list to
# reach an identical one. Under strict the failure block IS the list.
if [[ ${#SKIPPED[@]} -gt 0 && "$STRICT" != true ]]; then
echo "Skipped scripts:"
sidx=0
for s in ${SKIPPED[@]+"${SKIPPED[@]}"}; do
echo " $s"
echo " ${SKIP_REASONS[$sidx]}"
sidx=$((sidx + 1))
done
fi
RC=0
if [[ ${#FAILED[@]} -gt 0 ]]; then
echo "Failed scripts:"
for s in ${FAILED[@]+"${FAILED[@]}"}; do
echo " $s"
done
exit 1
RC=1
fi
# Strict mode turns every skip into a failure. Reported separately from FAILED
# above rather than folded into it: a skipped suite did not fail, the machine
# did, and a message that says so points at the fix. Named with reasons again
# here (not just referenced) because this block goes to stderr and is what a
# pre-push reader actually gets handed.
if [[ "$STRICT" == true && ${#SKIPPED[@]} -gt 0 ]]; then
echo "Error: --strict and ${#SKIPPED[@]} suite(s) skipped. Run as a gate, a skip is a SETUP ERROR on this machine, not a legitimate state: AGENTS.md documents vale, apm and jq as required pre-push dependencies, so every suite is expected to be runnable here. Install what each suite names below and re-run; do not skip the hook." >&2
sidx=0
for s in ${SKIPPED[@]+"${SKIPPED[@]}"}; do
echo " $s" >&2
echo " ${SKIP_REASONS[$sidx]}" >&2
sidx=$((sidx + 1))
done
RC=1
fi
exit "$RC"

97
tests/test-check-vale-style-sync.sh
View File

@@ -226,49 +226,108 @@ else
pass "exits non-zero and reports a stale .apm/ path when plugins/kyberforge exists without it"
fi
# --- 5d. Exits 1, saying so, when the probe table verifies nothing ---
# The directory floor above cannot see this one: both audit skill directories are
# still in place, only `assets/vale/` has moved. Every probe then `continue`s on
# its missing .vale.ini and the glob-coverage section checks zero paths. Asserted
# on the message because several other assertions also fire on this fixture.
# --- 5d/5d2. Exits 1, saying so, when the probe TABLE itself verifies nothing ---
# 5d used to relocate `assets/vale/` in both skills, on the belief that doing so
# skipped the whole probe table with FAIL still at 0. It does not. Run against
# the PRE-guard script that fixture already exited 1 with three errors: the
# `.vale.ini` loop errs on both missing files long before the probe loop, and
# PROBES_CHECKED can only reach 0 when both files are gone — which necessarily
# means FAIL >= 2. So it never exercised the guard as a cause, only checked that
# its message showed up beside unrelated failures.
#
# The guard is still worth having, but its real triggers live in the probe table,
# which is part of the script rather than the fixture — so these two cases mutate
# a COPY of the script and run that. Both assert `1 error(s)`, which is what makes
# them real: with the guard deleted each mutation exits 0, and with it present the
# guard is provably the only thing that failed the run.
assert_mutated() {
if diff -q "$SCRIPT" "$1" >/dev/null 2>&1; then
fail "the script mutation changed nothing — the probe table's shape has moved, so this case would pass vacuously"
return 1
fi
}
echo ""
echo "--- exits 1 and says so when zero glob probes were checked ---"
echo "--- exits 1 and says so when every probe row names a directory that does not exist ---"
FIXTURE5D="$(make_fixture)"
FIXTURES+=("$FIXTURE5D")
mv "$FIXTURE5D/plugins/kyberforge/.apm/skills/skill-audit/assets/vale" \
"$FIXTURE5D/plugins/kyberforge/.apm/skills/skill-audit/assets/vale-moved"
mv "$FIXTURE5D/plugins/kyberforge/.apm/skills/agent-audit/assets/vale" \
"$FIXTURE5D/plugins/kyberforge/.apm/skills/agent-audit/assets/vale-moved"
SCRATCH5D="$(mktemp -d)"
FIXTURES+=("$SCRATCH5D")
sed 's/^skill-audit|/skill-auditX|/; s/^agent-audit|/agent-auditX|/' "$SCRIPT" > "$SCRATCH5D/drifted.sh"
NOPROBE_OUT=""
NOPROBE_RC=0
NOPROBE_OUT="$(bash "$SCRIPT" "$FIXTURE5D" 2>&1)" || NOPROBE_RC=$?
if [[ $NOPROBE_RC -eq 0 ]]; then
fail "exited 0 when no glob probe could be checked — expected exit 1"
elif ! printf '%s\n' "$NOPROBE_OUT" | grep -q "no probe path was checked"; then
fail "did not report that zero probe paths were checked: $(printf '%s' "$NOPROBE_OUT" | tr '\n' ' ')"
else
pass "exits non-zero and reports that zero glob probes were checked"
if assert_mutated "$SCRATCH5D/drifted.sh"; then
NOPROBE_OUT="$(bash "$SCRATCH5D/drifted.sh" "$FIXTURE5D" 2>&1)" || NOPROBE_RC=$?
if [[ $NOPROBE_RC -eq 0 ]]; then
fail "a probe table naming no existing skill directory exited 0 — the glob-coverage section checked nothing and reported success"
elif ! printf '%s\n' "$NOPROBE_OUT" | grep -q "no probe path was checked"; then
fail "did not report that zero probe paths were checked: $(printf '%s' "$NOPROBE_OUT" | tr '\n' ' ')"
elif ! printf '%s\n' "$NOPROBE_OUT" | grep -q "failed: 1 error(s)"; then
fail "drifted probe rows failed for reasons beyond the empty probe table, so this guard is not provably what fired: $(printf '%s' "$NOPROBE_OUT" | tr '\n' ' ')"
else
pass "a probe table whose rows name no existing skill directory fails with that guard as the sole error"
fi
fi
echo ""
echo "--- exits 1 and says so when the probe table is empty ---"
FIXTURE5D2="$(make_fixture)"
FIXTURES+=("$FIXTURE5D2")
SCRATCH5D2="$(mktemp -d)"
FIXTURES+=("$SCRATCH5D2")
# The other reachable trigger: the heredoc gutted outright by a bad merge or a
# truncated edit. `done <<'EOF_PROBE'` with no rows between the delimiters is
# valid bash — the loop body simply never runs.
awk '
/^done <<.EOF_PROBE.$/ { print; inblk = 1; next }
inblk && /^EOF_PROBE$/ { print; inblk = 0; next }
inblk { next }
{ print }
' "$SCRIPT" > "$SCRATCH5D2/gutted.sh"
EMPTYTBL_OUT=""
EMPTYTBL_RC=0
if assert_mutated "$SCRATCH5D2/gutted.sh"; then
EMPTYTBL_OUT="$(bash "$SCRATCH5D2/gutted.sh" "$FIXTURE5D2" 2>&1)" || EMPTYTBL_RC=$?
if [[ $EMPTYTBL_RC -eq 0 ]]; then
fail "an empty probe table exited 0 — the glob-coverage section verified nothing and reported success"
elif ! printf '%s\n' "$EMPTYTBL_OUT" | grep -q "no probe path was checked"; then
fail "did not report that zero probe paths were checked: $(printf '%s' "$EMPTYTBL_OUT" | tr '\n' ' ')"
elif ! printf '%s\n' "$EMPTYTBL_OUT" | grep -q "failed: 1 error(s)"; then
fail "an empty probe table failed for reasons beyond the guard: $(printf '%s' "$EMPTYTBL_OUT" | tr '\n' ' ')"
else
pass "an emptied probe heredoc fails with that guard as the sole error"
fi
fi
# --- 5e. Positive: the check does real work against THIS repo ---
# Every case above runs against a synthetic fixture, so the whole suite could be
# green while the script inspected nothing at all in the repo it is wired into at
# pre-push. The summary line carries the counts; assert they are non-zero.
#
# BOTH counts, not just the probe count. The `.vale.ini` half of that line was a
# hardcoded `2` in each branch of the summary — true on any clean run, since a
# missing or unreadable file errs out before the summary is reached, but a
# constant states what the author expected rather than what the run inspected,
# and extracting only the probe count left it asserted by nothing. It is computed
# now, so the count is worth reading and worth pinning.
echo ""
echo "--- reports a non-zero number of inspected targets against this repo ---"
REAL_OUT=""
REAL_RC=0
REAL_OUT="$(bash "$SCRIPT" "$REPO_ROOT" 2>&1)" || REAL_RC=$?
REAL_PROBES="$(printf '%s\n' "$REAL_OUT" | sed -n 's/.*checked, \([0-9][0-9]*\) glob probe(s).*/\1/p')"
REAL_INIS="$(printf '%s\n' "$REAL_OUT" | sed -n 's/.*: \([0-9][0-9]*\) \.vale\.ini file(s) checked.*/\1/p')"
if [[ $REAL_RC -ne 0 ]]; then
fail "exited non-zero against this repo's real Vale copies"
printf '%s\n' "$REAL_OUT" | sed 's/^/ /'
elif [[ -z "$REAL_PROBES" ]]; then
elif [[ -z "$REAL_PROBES" || -z "$REAL_INIS" ]]; then
fail "a clean run against this repo reported no inspected-target counts, so 'it checked something' is unverifiable: $(printf '%s' "$REAL_OUT" | tr '\n' ' ')"
elif [[ "$REAL_PROBES" -lt 1 ]]; then
fail "a clean run against this repo verified $REAL_PROBES glob probes — a pass that inspected nothing"
elif [[ "$REAL_INIS" -lt 2 ]]; then
fail "a clean run against this repo reported $REAL_INIS .vale.ini file(s) checked — both copies' configs must be inspected"
else
pass "inspects $REAL_PROBES glob probe(s) against this repo, and exits 0"
pass "inspects $REAL_INIS .vale.ini file(s) and $REAL_PROBES glob probe(s) against this repo, and exits 0"
fi
# --- 5b. Exits 1 when REPO_ROOT does not exist ---

131
tests/test-run-bats.sh
View File

@@ -70,20 +70,33 @@ seed_bats_files() {
# Runs the fixture's run-bats.sh, capturing output and exit code separately.
#
# BATS_FILE_FLOOR is lowered to 1 for every case that is not specifically about
# the floor: these fixtures hold one or two .bats files by design, which is well
# under the real repo's floor. FAKE_FLOOR lets the floor cases opt back into the
# script's own default. TMPDIR is a private per-run directory so a stub can find
# the scratch dir run-bats.sh mktemp'd for itself -- see case 7.
# No file-count knob is passed any more, and none is needed: the expected file
# set is derived from `git ls-files`, and a mktemp fixture is not a git worktree
# root, so run-bats.sh announces that it could not derive an expectation and
# falls back to the unconditional zero-file check. Cases 8-8c drive the derived
# path deliberately by `git init`-ing their fixtures.
#
# TMPDIR is a private per-run directory so a stub can find the scratch dir
# run-bats.sh mktemp'd for itself -- see case 7.
FAKE_OUT=""
FAKE_RC=0
FAKE_FLOOR=""
run_fake() {
local priv
priv="$(mktemp -d)"
FIXTURES+=("$priv")
FAKE_RC=0
FAKE_OUT="$(TMPDIR="$priv" BATS_FILE_FLOOR="${FAKE_FLOOR:-1}" bash "$1/tests/run-bats.sh" 2>&1)" || FAKE_RC=$?
FAKE_OUT="$(TMPDIR="$priv" bash "$1/tests/run-bats.sh" 2>&1)" || FAKE_RC=$?
}
# A fixture whose root IS a git worktree root, so run-bats.sh derives its
# expected set from the index instead of degrading. Only `git add` is used --
# `git ls-files` reads the index, so nothing needs committing and no user
# identity is required.
make_git_fake_repo() {
local dir
dir="$(make_fake_repo)"
git -C "$dir" init -q
echo "$dir"
}
# --- 1. A stub emitting nothing at all is a broken harness, not a clean run ---
@@ -277,13 +290,15 @@ else
pass "an empty status file fails the run despite a healthy TAP stream"
fi
# --- 8. Too few discovered .bats files is a hard error, not a green run. Zero
# files used to print a note to stderr and `exit 0`; the floor makes both zero
# and a collapsed-but-nonzero count fail, because a moved tests/ tree or a
# widened `-not -path` exclusion produces exactly that and nothing else notices.
# --- 8. A tracked .bats file the walk did not discover is a hard error. This
# replaces a `BATS_FILE_FLOOR=8` guess against a real count of 10 -- two files of
# slack, which is not hypothetical: deleting two real .bats files left the suite
# reporting "155 tests, 0 failures" and exiting 0 with 11 tests silently gone.
# The expectation is now derived from `git ls-files`, so it is exact and needs no
# magic number.
echo ""
echo "--- a discovered-file count under the floor fails the run ---"
DIR8="$(make_fake_repo)"
echo "--- a tracked .bats file missing from the walk fails the run and names it ---"
DIR8="$(make_git_fake_repo)"
FIXTURES+=("$DIR8")
seed_bats_files "$DIR8"
install_stub_bats "$DIR8" <<'EOF'
@@ -292,20 +307,92 @@ echo "1..1"
echo "ok 1 first"
exit 0
EOF
FAKE_FLOOR=8
git -C "$DIR8" add tests/a.bats tests/b.bats
rm "$DIR8/tests/b.bats"
run_fake "$DIR8"
FAKE_FLOOR=""
if [[ $FAKE_RC -eq 0 ]]; then
fail "2 .bats files under a floor of 8 passed — a collapsed search path reads as green"
elif echo "$FAKE_OUT" | grep -q "below the floor of 8"; then
pass "a file count under the floor fails and names the floor it missed"
fail "a tracked .bats file gone from the worktree passed — a deleted suite reads as green"
elif ! echo "$FAKE_OUT" | grep -q "tracked .bats file(s) were not discovered"; then
fail "the run failed but not with the undiscovered-tracked-file message: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^ tests/b.bats$"; then
pass "a tracked .bats file missing from the walk fails the run and names the file"
else
fail "the run failed but not with the floor message: $FAKE_OUT"
fail "the run failed without naming the missing file: $FAKE_OUT"
fi
# --- 9. Zero discovered .bats files is the same hard error. Kept separate from
# case 8 because it is the state the old `exit 0` branch handled by name, and it
# is the one a path change actually produces.
# --- 8b. An UNTRACKED .bats file is not a finding. The derived expectation runs
# one way only: every tracked file must have been discovered, but a discovered
# file need not be tracked. Without this the check would fail on ordinary
# not-yet-committed work, which is how a correct guard gets disabled.
echo ""
echo "--- an untracked new .bats file does not fail the run ---"
DIR8B="$(make_git_fake_repo)"
FIXTURES+=("$DIR8B")
seed_bats_files "$DIR8B"
install_stub_bats "$DIR8B" <<'EOF'
#!/usr/bin/env bash
echo "1..1"
echo "ok 1 first"
exit 0
EOF
git -C "$DIR8B" add tests/a.bats
run_fake "$DIR8B"
if [[ $FAKE_RC -ne 0 ]]; then
fail "an untracked .bats file was reported as a finding: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^2 tests, 0 failures$"; then
pass "an untracked .bats file is run without being demanded of the index"
else
fail "the untracked-file run passed with the wrong count: $FAKE_OUT"
fi
# --- 8c. A newly added .bats file joins the expectation immediately. This is the
# half a floor can never have: adding files only ever widens a floor's slack,
# while `git add` alone makes the new file required from the next run on, with no
# edit to this script and no number to bump.
echo ""
echo "--- a newly git-added .bats file is required from the next run on ---"
git -C "$DIR8B" add tests/b.bats
rm "$DIR8B/tests/b.bats"
run_fake "$DIR8B"
if [[ $FAKE_RC -eq 0 ]]; then
fail "the .bats file added to the index a moment ago was not demanded back: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^ tests/b.bats$"; then
pass "a file added to the index joins the expected set with no floor to bump"
else
fail "the run failed but did not name the newly tracked file: $FAKE_OUT"
fi
# --- 8d. Outside a git worktree the run still works, and says the expectation
# could not be derived. That degradation is what every other fixture here relies
# on, and it must be announced rather than silent -- an unannounced fallback is
# how a derived check quietly becomes no check at all on a tarball export.
echo ""
echo "--- a non-git tree runs, and announces that no expectation could be derived ---"
DIR8D="$(make_fake_repo)"
FIXTURES+=("$DIR8D")
seed_bats_files "$DIR8D"
install_stub_bats "$DIR8D" <<'EOF'
#!/usr/bin/env bash
echo "1..1"
echo "ok 1 first"
exit 0
EOF
run_fake "$DIR8D"
if [[ $FAKE_RC -ne 0 ]]; then
fail "a non-git tree failed the run: $FAKE_OUT"
elif ! echo "$FAKE_OUT" | grep -q "not a git worktree root"; then
fail "a non-git tree silently skipped the derived expectation with no note: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^2 tests, 0 failures$"; then
pass "a non-git tree runs the suite and says the expected set could not be derived"
else
fail "the non-git run passed with the wrong count: $FAKE_OUT"
fi
# --- 9. Zero discovered .bats files is a hard error regardless, and is checked
# unconditionally rather than through the derived set: a tree with nothing
# tracked at all (a tarball export, a fresh scaffold) must still not run on an
# empty set and call it green. It is the state the old `exit 0` branch handled by
# name, and the one a path change actually produces.
echo ""
echo "--- zero discovered .bats files fails the run ---"
DIR9="$(make_fake_repo)"

213
tests/test-run-tests.sh
View File

@@ -8,6 +8,11 @@
# moving, or dropping the executable bit off run-bats.sh made the entire bats
# suite disappear with no diagnostic while the run printed a green summary and
# exited 0, and turned `--bats-only` into a no-op that printed nothing.
# * run_bats() then checked only that run-bats.sh was present and executable,
# never that it PRODUCED anything. `bash` on an empty run-bats.sh exits 0
# having printed nothing, so the dispatcher printed `=== bats ===` and a green
# summary. The runner's `N tests, M failures` line is now required, with a
# non-zero count.
# * The per-script status was compared with `-eq`, which is arithmetic, and bash
# evaluates an empty string as 0 there -- so a status file that existed but was
# empty counted as a pass.
@@ -191,6 +196,67 @@ else
pass "a failing bats runner propagates out of run-tests.sh"
fi
# --- 5b. An EMPTY run-bats.sh is a hard error, not a green no-op ---
# Present and executable was still not "it ran". `bash` on a zero-byte script
# exits 0 having printed nothing, so the dispatcher printed `=== bats ===`, a
# blank line, and `Summary: 1 passed, 0 skipped, 0 failed` with rc=0 -- the whole
# bats suite gone, exactly the defect cases 2-4 close for the other spellings.
# Truncation, a partial write, an editor saving an empty buffer, and a `set -e`
# abort in a run-bats.sh preamble all produce this file.
#
# It only failed on the real repo incidentally, because tests/test-run-bats.sh
# copies run-bats.sh into its own fixtures and blows up there; rename or retire
# that file and the hole is live in the gate pre-push invokes. This asserts it
# directly.
echo ""
echo "--- an empty run-bats.sh fails the run instead of passing silently ---"
DIR5B="$(make_fake_repo)"
FIXTURES+=("$DIR5B")
: > "$DIR5B/tests/run-bats.sh"
chmod +x "$DIR5B/tests/run-bats.sh"
add_case "$DIR5B" test-ok.sh <<'EOF'
#!/usr/bin/env bash
echo "fine"
EOF
run_fake "$DIR5B"
if echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$"; then
fail "an empty run-bats.sh produced a green summary — the bats suite vanished with no diagnostic"
elif [[ $FAKE_RC -eq 0 ]]; then
fail "an empty run-bats.sh exited 0: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "without reporting an 'N tests, M failures' summary"; then
pass "an empty run-bats.sh fails the run and says the bats suite was never verified"
else
fail "the run failed but not with the no-summary message: $FAKE_OUT"
fi
# --- 5c. A runner reporting zero tests is a hard error too ---
# The other half of "ran but produced nothing": the summary line is there and the
# process exits 0, but it accounts for no tests. run-bats.sh has its own guard for
# this one file down; asserting it here means the dispatcher does not depend on
# that guard surviving, and it pins the count as the thing being read rather than
# the mere presence of a line matching the pattern.
echo ""
echo "--- a bats runner reporting 0 tests fails the run ---"
DIR5C="$(make_fake_repo)"
FIXTURES+=("$DIR5C")
install_stub_bats_runner "$DIR5C" <<'EOF'
#!/usr/bin/env bash
echo "0 tests, 0 failures"
exit 0
EOF
add_case "$DIR5C" test-ok.sh <<'EOF'
#!/usr/bin/env bash
echo "fine"
EOF
run_fake "$DIR5C"
if [[ $FAKE_RC -eq 0 ]]; then
fail "a bats runner reporting 0 tests exited 0 — a suite that executed nothing read as green: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "reported 0 tests"; then
pass "a bats runner reporting 0 tests fails the run and says the suite executed nothing"
else
fail "the run failed but not with the zero-tests message: $FAKE_OUT"
fi
# --- 6. An empty status file is FAILED, not a pass ---
# The status is read back with `cat ... || echo 1`, which covers a *missing*
# file. A file that exists but is empty is what a job killed between the `>`
@@ -311,6 +377,153 @@ else
fail "the failed script was not named in the failure list: $FAKE_OUT"
fi
# --- 10. --strict turns a skip into a failure, and names the suite AND the reason ---
# Graceful skipping is right for an ad-hoc run and wrong for a gate. At pre-push a
# suite exiting 77 means a dependency AGENTS.md documents as required is missing
# on the pushing machine -- and pre-commit prints nothing at all for a passing
# hook, so the skip list this script writes to stdout was swallowed whole. A
# vale-less PATH shipped a green gate having verified 15 of 17 suites.
#
# The reason is asserted, not just the name: "something was skipped" leaves the
# reader with no idea which binary to install, which is most of why the swallowed
# list was worth so little in the first place. Matched on the SIX-SPACE INDENT the
# report writes, not on the reason text alone -- the suite's own log is echoed
# back verbatim earlier in the same output, so a bare text match passes even with
# the reason capture deleted. Verified: narrowing the capture to the `SKIP:`
# prefix left the loose form green.
echo ""
echo "--- --strict fails the run on a skipped suite and names it with its reason ---"
DIR10="$(make_fake_repo)"
FIXTURES+=("$DIR10")
install_healthy_bats_runner "$DIR10"
add_case "$DIR10" test-needs-a-binary.sh <<'EOF'
#!/usr/bin/env bash
echo "SKIP: frobnicator is not installed — install it from https://example.invalid"
exit 77
EOF
add_case "$DIR10" test-ok.sh <<'EOF'
#!/usr/bin/env bash
echo "fine"
EOF
run_fake "$DIR10" --strict
if [[ $FAKE_RC -eq 0 ]]; then
fail "--strict passed with a skipped suite — the gate reports green having verified less than it ran: $FAKE_OUT"
elif ! echo "$FAKE_OUT" | grep -q "a skip is a SETUP ERROR"; then
fail "--strict failed but never said a skip is a setup error: $FAKE_OUT"
elif ! echo "$FAKE_OUT" | grep -q "test-needs-a-binary.sh"; then
fail "--strict failed without naming the skipped suite: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^ SKIP: frobnicator is not installed"; then
pass "--strict fails on a skip, names the suite, and carries through the reason it printed"
else
fail "--strict named the suite but swallowed its skip reason: $FAKE_OUT"
fi
# --- 10b. RUN_TESTS_STRICT=1 is the same switch. The hook uses the flag because
# it is self-documenting in .pre-commit-config.yaml; the env var exists for a CI
# runner that cannot edit the command line. Both are asserted so one cannot rot.
echo ""
echo "--- RUN_TESTS_STRICT=1 fails the run on a skipped suite ---"
STRICT_ENV_OUT=""
STRICT_ENV_RC=0
STRICT_ENV_PRIV="$(mktemp -d)"
FIXTURES+=("$STRICT_ENV_PRIV")
STRICT_ENV_OUT="$(TMPDIR="$STRICT_ENV_PRIV" TEST_DIR="$DIR10/cases" RUN_TESTS_STRICT=1 \
bash "$DIR10/tests/run-tests.sh" 2>&1)" || STRICT_ENV_RC=$?
if [[ $STRICT_ENV_RC -eq 0 ]]; then
fail "RUN_TESTS_STRICT=1 passed with a skipped suite: $STRICT_ENV_OUT"
elif echo "$STRICT_ENV_OUT" | grep -q "a skip is a SETUP ERROR"; then
pass "RUN_TESTS_STRICT=1 is the same gate as --strict"
else
fail "RUN_TESTS_STRICT=1 failed for some other reason: $STRICT_ENV_OUT"
fi
# --- 10c. WITHOUT strict, the same fixture still skips gracefully and passes ---
# The control for 10 and 10b, and the half the coordinator asked for explicitly:
# an ad-hoc `bash tests/run-tests.sh` on a laptop missing a dev binary must not
# go red. Without this, "fix the gate" could quietly mean "fail everywhere".
echo ""
echo "--- the same skipped suite passes, still SKIPPED, without strict ---"
run_fake "$DIR10"
if [[ $FAKE_RC -ne 0 ]]; then
fail "a skipped suite failed a non-strict run — graceful skipping is gone: $FAKE_OUT"
elif ! echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 1 skipped, 0 failed ===$"; then
fail "a non-strict run miscounted the skip: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^ SKIP: frobnicator is not installed"; then
pass "without strict the suite is SKIPPED, the run passes, and the reason is still reported"
else
fail "a non-strict run passed but dropped the skip reason: $FAKE_OUT"
fi
# --- 10d. --strict does not become a blanket failure ---
# The case that proves 10 and 10b fail for their stated reason. A clean run with
# nothing skipped must be just as green under --strict as without it, otherwise
# the gate is not a gate, it is an outage.
echo ""
echo "--- --strict is still green when nothing skipped ---"
DIR10D="$(make_fake_repo)"
FIXTURES+=("$DIR10D")
install_healthy_bats_runner "$DIR10D"
add_case "$DIR10D" test-ok.sh <<'EOF'
#!/usr/bin/env bash
echo "fine"
EOF
run_fake "$DIR10D" --strict
if [[ $FAKE_RC -ne 0 ]]; then
fail "--strict failed a run with nothing skipped — it fails unconditionally: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$"; then
pass "--strict leaves a run with no skips green"
else
fail "--strict passed with the wrong summary: $FAKE_OUT"
fi
# --- 10e. An unknown flag is rejected, not ignored ---
# `--strict` reaching this script as a silently-ignored argument is the single
# typo that turns the gate back off while every hook still reports Passed, so the
# arg loop refuses anything it does not know rather than falling through.
echo ""
echo "--- an unrecognised flag fails with usage instead of being ignored ---"
DIR10E="$(make_fake_repo)"
FIXTURES+=("$DIR10E")
install_healthy_bats_runner "$DIR10E"
add_case "$DIR10E" test-ok.sh <<'EOF'
#!/usr/bin/env bash
echo "fine"
EOF
run_fake "$DIR10E" --strickt
if [[ $FAKE_RC -eq 0 ]]; then
fail "a misspelled flag was ignored and the run passed — a typo silently disarms the gate: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "Usage: .*--bats-only.*--strict"; then
pass "an unrecognised flag fails the run with usage"
else
fail "an unrecognised flag failed but not with usage: $FAKE_OUT"
fi
# --- 10f. A skip reason printed on STDERR, with no `SKIP:` prefix, still lands ---
# There is no house format: three suites print `SKIP: <reason>` on stdout and
# tests/test-sync-plugin-content.sh prints `apm not installed -- skipping (...)`
# on stderr. batch-run.sh folds stderr into the same log, so both are reachable,
# but only a fallback chain finds the second one. Without this case the reason
# extraction could be narrowed to the `SKIP:` prefix and the apm suite would fail
# the gate with no indication of what to install.
echo ""
echo "--- a stderr skip reason with no SKIP: prefix is still reported ---"
DIR10F="$(make_fake_repo)"
FIXTURES+=("$DIR10F")
install_healthy_bats_runner "$DIR10F"
add_case "$DIR10F" test-stderr-skip.sh <<'EOF'
#!/usr/bin/env bash
echo "widgetizer not installed -- skipping (see docs)" >&2
exit 77
EOF
run_fake "$DIR10F" --strict
if [[ $FAKE_RC -eq 0 ]]; then
fail "--strict passed on a suite that skipped via stderr: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^ widgetizer not installed -- skipping"; then
pass "a skip reason printed to stderr without a SKIP: prefix is still carried into the failure"
else
fail "--strict failed but lost the stderr skip reason: $FAKE_OUT"
fi
echo ""
echo "Results: $PASS passed, $FAIL failed"
[[ $FAIL -eq 0 ]]