fix(scripts): fail the vale and scope gates when they cannot verify
check-vale-style-sync.sh's glob-coverage probe silently self-disabled when vale was absent from PATH, exiting 0 on the one-character glob typo it exists to catch. pre-commit swallows a passing hook's output, so the pre-push hook reported Passed. The script already hard-fails on a bad REPO_ROOT for exactly this reason -- "a clean exit 0 here would read as 'checked, in sync' when nothing ran at all" -- and six of its assertions are vale invocations. Absence now fails; the opt-out is an env var that must be set deliberately, and it downgrades the run to text-level assertions while saying so in the summary. Neither script had a floor on its rewritten .apm/ paths, so relocating .apm/ made both exit 0 -- and this PR's whole change to them was a path rewrite, the exact edit that failure mode survives. A third gap the directory check could not see: relocating only assets/vale/ left both audit skill directories in place while every probe continued past its missing .vale.ini, skipping the whole table with FAIL=0. A zero-probe run is now an error. Both test suites encoded the vacuous pass as a passing case. Those cases are now scoped to "no plugins/kyberforge at all" and assert the fixture really lacks it, with new counterparts covering the drift shape and new positive cases requiring each script to report a non-zero inspected-target count. Also removes the HOOK_REGEX_CACHE memoization: every call site was a command substitution, so the writes happened in a subshell and the lookup always missed. Measured at 14ms of an ~870ms run, all of which is the six vale invocations. Deleted rather than repaired -- 35 lines claiming a benefit they never delivered is worse than no cache -- with a comment recording why. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X7GvKuJfy2WrdBmUttV4DT
This commit is contained in:
@@ -20,40 +20,10 @@ trap cleanup EXIT
|
||||
RUN_TMP="$(mktemp -d)"
|
||||
FIXTURES+=("$RUN_TMP")
|
||||
|
||||
# --- 1. Exits 0 against this repo's own (fixed) scripts ---
|
||||
echo ""
|
||||
echo "--- exits 0 against this repo's real scripts ---"
|
||||
if bash "$SCRIPT" "$REPO_ROOT" > "$RUN_TMP/clean.out" 2>&1; then
|
||||
pass "exits 0 against this repo's real scope walk-up scripts"
|
||||
else
|
||||
fail "exited non-zero against this repo's real (already-fixed) scripts"
|
||||
sed 's/^/ /' "$RUN_TMP/clean.out"
|
||||
fi
|
||||
|
||||
# --- 2. Exits 0 as a no-op when the kyberforge skills aren't present ---
|
||||
echo ""
|
||||
echo "--- exits 0 (no-op) when the target scripts don't exist ---"
|
||||
FIXTURE_EMPTY="$(mktemp -d)"
|
||||
FIXTURES+=("$FIXTURE_EMPTY")
|
||||
if bash "$SCRIPT" "$FIXTURE_EMPTY" > /dev/null 2>&1; then
|
||||
pass "exits 0 as a no-op when agent-author/agent-audit/skill-author aren't present"
|
||||
else
|
||||
fail "exited non-zero when the kyberforge skills are simply absent"
|
||||
fi
|
||||
|
||||
# --- 3. Exits 1 against a REPO_ROOT that doesn't exist ---
|
||||
echo ""
|
||||
echo "--- exits 1 when REPO_ROOT does not exist ---"
|
||||
if bash "$SCRIPT" "/nonexistent/path/$(date +%s)-$$" > /dev/null 2>&1; then
|
||||
fail "exited 0 for a nonexistent REPO_ROOT — expected exit 1"
|
||||
else
|
||||
pass "exits non-zero for a nonexistent REPO_ROOT"
|
||||
fi
|
||||
|
||||
# --- 4. Regression guard: reintroducing the $HOME-collapse bug into
|
||||
# validate.sh's detect_scope must make the check fail. Builds a minimal
|
||||
# REPO_ROOT (just the four scripts, at their real relative paths) so this
|
||||
# doesn't depend on — or risk mutating — the real repo tree.
|
||||
# Builds a minimal REPO_ROOT (just the four scripts, at their real relative
|
||||
# paths) so the mutation cases below don't depend on — or risk mutating — the
|
||||
# real repo tree. Defined up here rather than beside its first mutation case
|
||||
# because case 2b's stale-.apm/ fixture is built from it too.
|
||||
make_minimal_repo_root() {
|
||||
local dir
|
||||
dir="$(mktemp -d)"
|
||||
@@ -75,6 +45,85 @@ make_minimal_repo_root() {
|
||||
echo "$dir"
|
||||
}
|
||||
|
||||
# --- 1. Exits 0 against this repo's own (fixed) scripts ---
|
||||
echo ""
|
||||
echo "--- exits 0 against this repo's real scripts ---"
|
||||
if bash "$SCRIPT" "$REPO_ROOT" > "$RUN_TMP/clean.out" 2>&1; then
|
||||
pass "exits 0 against this repo's real scope walk-up scripts"
|
||||
else
|
||||
fail "exited non-zero against this repo's real (already-fixed) scripts"
|
||||
sed 's/^/ /' "$RUN_TMP/clean.out"
|
||||
fi
|
||||
|
||||
# --- 1b. Positive: that exit 0 was earned, not vacuous ---
|
||||
# Every other case here runs against a synthetic fixture, and exit 0 is also
|
||||
# what the script produces when it finds nothing to check at all. So the
|
||||
# assertion above passes just as happily on a run that executed zero fixtures.
|
||||
# The `ok:` lines are the record of work actually done; each of the seven
|
||||
# fixtures emits at least one, so a floor of 7 catches a whole fixture going
|
||||
# dark as well as the all-or-nothing case (13 at the time of writing — the floor
|
||||
# is deliberately below that so adding assertions to a fixture doesn't churn it).
|
||||
CLEAN_OKS="$(grep -c '^ ok:' "$RUN_TMP/clean.out" || true)"
|
||||
if [[ "$CLEAN_OKS" -ge 7 ]]; then
|
||||
pass "the clean run against this repo actually exercised its fixtures ($CLEAN_OKS ok assertions)"
|
||||
else
|
||||
fail "the clean run against this repo reported only $CLEAN_OKS ok assertions (expected at least one per fixture) — exit 0 without the fixtures having run means nothing was checked"
|
||||
sed 's/^/ /' "$RUN_TMP/clean.out"
|
||||
fi
|
||||
|
||||
# --- 2. Exits 0 as a no-op ONLY when there is no kyberforge plugin at all ---
|
||||
# The no-op is scoped to a repo that never installed kyberforge. Case 2b below is
|
||||
# its counterpart and the one that matters.
|
||||
echo ""
|
||||
echo "--- exits 0 (no-op) when there is no plugins/kyberforge at all ---"
|
||||
FIXTURE_EMPTY="$(mktemp -d)"
|
||||
FIXTURES+=("$FIXTURE_EMPTY")
|
||||
if [[ -e "$FIXTURE_EMPTY/plugins/kyberforge" ]]; then
|
||||
fail "the empty fixture unexpectedly has a plugins/kyberforge, so it does not exercise the no-kyberforge no-op"
|
||||
elif bash "$SCRIPT" "$FIXTURE_EMPTY" > /dev/null 2>&1; then
|
||||
pass "exits 0 as a no-op when the repo has no kyberforge plugin"
|
||||
else
|
||||
fail "exited non-zero when the repo simply has no kyberforge plugin"
|
||||
fi
|
||||
|
||||
# --- 2b. Exits 1, saying so, when plugins/kyberforge exists but the .apm/
|
||||
# scripts under it do not ---
|
||||
# The four target paths are hardcoded as plugins/kyberforge/.apm/skills/... with
|
||||
# no floor under them: `mv plugins/kyberforge/.apm plugins/kyberforge/.apm2` hit
|
||||
# the "not present, nothing to check" branch and exited 0, indistinguishable
|
||||
# from "all four implementations agree" and swallowed by pre-commit as `Passed`.
|
||||
# A path rewrite is exactly the edit that produces this, and it is what this PR
|
||||
# did to these paths.
|
||||
#
|
||||
# Asserted on the MESSAGE, not just the code: this script exits 1 for any fixture
|
||||
# disagreement too, so the code alone would not tell a stale path from a genuine
|
||||
# walk-up regression — and those call for opposite fixes.
|
||||
echo ""
|
||||
echo "--- exits 1 and says so when plugins/kyberforge exists but .apm/ does not ---"
|
||||
FIXTURE_STALE="$(make_minimal_repo_root)"
|
||||
FIXTURES+=("$FIXTURE_STALE")
|
||||
mv "$FIXTURE_STALE/plugins/kyberforge/.apm" "$FIXTURE_STALE/plugins/kyberforge/.apm2"
|
||||
STALE_RC=0
|
||||
bash "$SCRIPT" "$FIXTURE_STALE" > "$RUN_TMP/stale.out" 2>&1 || STALE_RC=$?
|
||||
if [[ $STALE_RC -eq 0 ]]; then
|
||||
fail "exited 0 when plugins/kyberforge exists but its .apm/ scripts are gone — expected exit 1"
|
||||
elif ! grep -q "\.apm/ paths have gone stale" "$RUN_TMP/stale.out"; then
|
||||
fail "failed for the wrong reason on a stale .apm/ path: $(tr '\n' ' ' < "$RUN_TMP/stale.out")"
|
||||
else
|
||||
pass "exits non-zero and reports a stale .apm/ path when plugins/kyberforge exists without it"
|
||||
fi
|
||||
|
||||
# --- 3. Exits 1 against a REPO_ROOT that doesn't exist ---
|
||||
echo ""
|
||||
echo "--- exits 1 when REPO_ROOT does not exist ---"
|
||||
if bash "$SCRIPT" "/nonexistent/path/$(date +%s)-$$" > /dev/null 2>&1; then
|
||||
fail "exited 0 for a nonexistent REPO_ROOT — expected exit 1"
|
||||
else
|
||||
pass "exits non-zero for a nonexistent REPO_ROOT"
|
||||
fi
|
||||
|
||||
# --- 4. Regression guard: reintroducing the $HOME-collapse bug into
|
||||
# validate.sh's detect_scope must make the check fail.
|
||||
echo ""
|
||||
echo "--- exits 1 when validate.sh's detect_scope collapses back to the \$HOME-walk-up bug ---"
|
||||
FIXTURE_BUG="$(make_minimal_repo_root)"
|
||||
|
||||
Reference in New Issue
Block a user