test: fail the gate when a suite is skipped or never reports
Three gaps left by the previous round, all the same shape: a gate reporting success having verified less than it appears to. run_bats() hard-failed on a missing or non-executable runner but never checked that the runner produced anything. An empty, executable run-bats.sh exits 0, and the dispatcher printed a green summary with 166 bats tests silently absent. It now requires an "N tests, M failures" line with a non-zero count. run-tests.sh's skip listing is swallowed by pre-commit on a pass, so on a machine without vale three suites exited 77 and the pre-push gate went green having run 14 of 17. The hook now invokes it as --strict, where a skip fails and the error names each suite and the reason it skipped. An ad-hoc local run still skips gracefully -- at pre-push a skip means a documented dependency is missing, which is a setup error, not a legitimate state. Deliberately not wired to the vale downgrade's env var: one flag must not disarm two gates. BATS_FILE_FLOOR is replaced by an expectation derived from git ls-files. A floor of 8 against a real count of 10 let two files and eleven tests disappear green, and the number needed an edit whenever a plugin was added. The derived set needs no number, and catches an addition as well as a removal -- a .bats file staged into the index and deleted from disk is now demanded back. The vale opt-out announced its downgrade to nobody: pre-commit prints nothing for a passing hook, so the summary line AGENTS.md tells the reader to check was unreachable in exactly the situation it exists for. The hook is now verbose. Also corrects the PROBES_CHECKED guard, whose commit message described a state that cannot occur -- the .vale.ini loop errs first. Its two reachable triggers, a gutted probe heredoc and a probe row naming a missing directory, had no test; they do now, each asserting the guard is the sole cause. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X7GvKuJfy2WrdBmUttV4DT
This commit is contained in:
@@ -226,49 +226,108 @@ else
|
||||
pass "exits non-zero and reports a stale .apm/ path when plugins/kyberforge exists without it"
|
||||
fi
|
||||
|
||||
# --- 5d. Exits 1, saying so, when the probe table verifies nothing ---
|
||||
# The directory floor above cannot see this one: both audit skill directories are
|
||||
# still in place, only `assets/vale/` has moved. Every probe then `continue`s on
|
||||
# its missing .vale.ini and the glob-coverage section checks zero paths. Asserted
|
||||
# on the message because several other assertions also fire on this fixture.
|
||||
# --- 5d/5d2. Exits 1, saying so, when the probe TABLE itself verifies nothing ---
|
||||
# 5d used to relocate `assets/vale/` in both skills, on the belief that doing so
|
||||
# skipped the whole probe table with FAIL still at 0. It does not. Run against
|
||||
# the PRE-guard script that fixture already exited 1 with three errors: the
|
||||
# `.vale.ini` loop errs on both missing files long before the probe loop, and
|
||||
# PROBES_CHECKED can only reach 0 when both files are gone — which necessarily
|
||||
# means FAIL >= 2. So it never exercised the guard as a cause, only checked that
|
||||
# its message showed up beside unrelated failures.
|
||||
#
|
||||
# The guard is still worth having, but its real triggers live in the probe table,
|
||||
# which is part of the script rather than the fixture — so these two cases mutate
|
||||
# a COPY of the script and run that. Both assert `1 error(s)`, which is what makes
|
||||
# them real: with the guard deleted each mutation exits 0, and with it present the
|
||||
# guard is provably the only thing that failed the run.
|
||||
assert_mutated() {
|
||||
if diff -q "$SCRIPT" "$1" >/dev/null 2>&1; then
|
||||
fail "the script mutation changed nothing — the probe table's shape has moved, so this case would pass vacuously"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
echo ""
|
||||
echo "--- exits 1 and says so when zero glob probes were checked ---"
|
||||
echo "--- exits 1 and says so when every probe row names a directory that does not exist ---"
|
||||
FIXTURE5D="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE5D")
|
||||
mv "$FIXTURE5D/plugins/kyberforge/.apm/skills/skill-audit/assets/vale" \
|
||||
"$FIXTURE5D/plugins/kyberforge/.apm/skills/skill-audit/assets/vale-moved"
|
||||
mv "$FIXTURE5D/plugins/kyberforge/.apm/skills/agent-audit/assets/vale" \
|
||||
"$FIXTURE5D/plugins/kyberforge/.apm/skills/agent-audit/assets/vale-moved"
|
||||
SCRATCH5D="$(mktemp -d)"
|
||||
FIXTURES+=("$SCRATCH5D")
|
||||
sed 's/^skill-audit|/skill-auditX|/; s/^agent-audit|/agent-auditX|/' "$SCRIPT" > "$SCRATCH5D/drifted.sh"
|
||||
NOPROBE_OUT=""
|
||||
NOPROBE_RC=0
|
||||
NOPROBE_OUT="$(bash "$SCRIPT" "$FIXTURE5D" 2>&1)" || NOPROBE_RC=$?
|
||||
if [[ $NOPROBE_RC -eq 0 ]]; then
|
||||
fail "exited 0 when no glob probe could be checked — expected exit 1"
|
||||
elif ! printf '%s\n' "$NOPROBE_OUT" | grep -q "no probe path was checked"; then
|
||||
fail "did not report that zero probe paths were checked: $(printf '%s' "$NOPROBE_OUT" | tr '\n' ' ')"
|
||||
else
|
||||
pass "exits non-zero and reports that zero glob probes were checked"
|
||||
if assert_mutated "$SCRATCH5D/drifted.sh"; then
|
||||
NOPROBE_OUT="$(bash "$SCRATCH5D/drifted.sh" "$FIXTURE5D" 2>&1)" || NOPROBE_RC=$?
|
||||
if [[ $NOPROBE_RC -eq 0 ]]; then
|
||||
fail "a probe table naming no existing skill directory exited 0 — the glob-coverage section checked nothing and reported success"
|
||||
elif ! printf '%s\n' "$NOPROBE_OUT" | grep -q "no probe path was checked"; then
|
||||
fail "did not report that zero probe paths were checked: $(printf '%s' "$NOPROBE_OUT" | tr '\n' ' ')"
|
||||
elif ! printf '%s\n' "$NOPROBE_OUT" | grep -q "failed: 1 error(s)"; then
|
||||
fail "drifted probe rows failed for reasons beyond the empty probe table, so this guard is not provably what fired: $(printf '%s' "$NOPROBE_OUT" | tr '\n' ' ')"
|
||||
else
|
||||
pass "a probe table whose rows name no existing skill directory fails with that guard as the sole error"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "--- exits 1 and says so when the probe table is empty ---"
|
||||
FIXTURE5D2="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE5D2")
|
||||
SCRATCH5D2="$(mktemp -d)"
|
||||
FIXTURES+=("$SCRATCH5D2")
|
||||
# The other reachable trigger: the heredoc gutted outright by a bad merge or a
|
||||
# truncated edit. `done <<'EOF_PROBE'` with no rows between the delimiters is
|
||||
# valid bash — the loop body simply never runs.
|
||||
awk '
|
||||
/^done <<.EOF_PROBE.$/ { print; inblk = 1; next }
|
||||
inblk && /^EOF_PROBE$/ { print; inblk = 0; next }
|
||||
inblk { next }
|
||||
{ print }
|
||||
' "$SCRIPT" > "$SCRATCH5D2/gutted.sh"
|
||||
EMPTYTBL_OUT=""
|
||||
EMPTYTBL_RC=0
|
||||
if assert_mutated "$SCRATCH5D2/gutted.sh"; then
|
||||
EMPTYTBL_OUT="$(bash "$SCRATCH5D2/gutted.sh" "$FIXTURE5D2" 2>&1)" || EMPTYTBL_RC=$?
|
||||
if [[ $EMPTYTBL_RC -eq 0 ]]; then
|
||||
fail "an empty probe table exited 0 — the glob-coverage section verified nothing and reported success"
|
||||
elif ! printf '%s\n' "$EMPTYTBL_OUT" | grep -q "no probe path was checked"; then
|
||||
fail "did not report that zero probe paths were checked: $(printf '%s' "$EMPTYTBL_OUT" | tr '\n' ' ')"
|
||||
elif ! printf '%s\n' "$EMPTYTBL_OUT" | grep -q "failed: 1 error(s)"; then
|
||||
fail "an empty probe table failed for reasons beyond the guard: $(printf '%s' "$EMPTYTBL_OUT" | tr '\n' ' ')"
|
||||
else
|
||||
pass "an emptied probe heredoc fails with that guard as the sole error"
|
||||
fi
|
||||
fi
|
||||
|
||||
# --- 5e. Positive: the check does real work against THIS repo ---
|
||||
# Every case above runs against a synthetic fixture, so the whole suite could be
|
||||
# green while the script inspected nothing at all in the repo it is wired into at
|
||||
# pre-push. The summary line carries the counts; assert they are non-zero.
|
||||
#
|
||||
# BOTH counts, not just the probe count. The `.vale.ini` half of that line was a
|
||||
# hardcoded `2` in each branch of the summary — true on any clean run, since a
|
||||
# missing or unreadable file errs out before the summary is reached, but a
|
||||
# constant states what the author expected rather than what the run inspected,
|
||||
# and extracting only the probe count left it asserted by nothing. It is computed
|
||||
# now, so the count is worth reading and worth pinning.
|
||||
echo ""
|
||||
echo "--- reports a non-zero number of inspected targets against this repo ---"
|
||||
REAL_OUT=""
|
||||
REAL_RC=0
|
||||
REAL_OUT="$(bash "$SCRIPT" "$REPO_ROOT" 2>&1)" || REAL_RC=$?
|
||||
REAL_PROBES="$(printf '%s\n' "$REAL_OUT" | sed -n 's/.*checked, \([0-9][0-9]*\) glob probe(s).*/\1/p')"
|
||||
REAL_INIS="$(printf '%s\n' "$REAL_OUT" | sed -n 's/.*: \([0-9][0-9]*\) \.vale\.ini file(s) checked.*/\1/p')"
|
||||
if [[ $REAL_RC -ne 0 ]]; then
|
||||
fail "exited non-zero against this repo's real Vale copies"
|
||||
printf '%s\n' "$REAL_OUT" | sed 's/^/ /'
|
||||
elif [[ -z "$REAL_PROBES" ]]; then
|
||||
elif [[ -z "$REAL_PROBES" || -z "$REAL_INIS" ]]; then
|
||||
fail "a clean run against this repo reported no inspected-target counts, so 'it checked something' is unverifiable: $(printf '%s' "$REAL_OUT" | tr '\n' ' ')"
|
||||
elif [[ "$REAL_PROBES" -lt 1 ]]; then
|
||||
fail "a clean run against this repo verified $REAL_PROBES glob probes — a pass that inspected nothing"
|
||||
elif [[ "$REAL_INIS" -lt 2 ]]; then
|
||||
fail "a clean run against this repo reported $REAL_INIS .vale.ini file(s) checked — both copies' configs must be inspected"
|
||||
else
|
||||
pass "inspects $REAL_PROBES glob probe(s) against this repo, and exits 0"
|
||||
pass "inspects $REAL_INIS .vale.ini file(s) and $REAL_PROBES glob probe(s) against this repo, and exits 0"
|
||||
fi
|
||||
|
||||
# --- 5b. Exits 1 when REPO_ROOT does not exist ---
|
||||
|
||||
Reference in New Issue
Block a user