Why Two suites failed intermittently — tests/test-vale-wrap.sh case 21 and tests/test-check-release-needed.sh cases 4 and 15 — on correct output, and never when run alone. The cause is the `echo "$OUT" | grep -q P` idiom under `set -o pipefail`: grep -q exits as soon as it has an answer, bash's echo can hand a multi-line value to the pipe one line at a time, and a write after the reader is gone kills echo with SIGPIPE. pipefail then reports the writer's death, so output that DID match reads as "no match". Every observed failure had lines after its match; case 15's match is on line 1 of 6, the widest window in that file. Forced with a pause before the writer's last line, the pipe form failed 50 of 50 runs; a here-string, a match on the last line, and the same pipe without pipefail each passed 50 of 50. Unforced the rate is about 1 per 670 suite runs, which is why it read as a flaky gate rather than a bug. The failures at review time are consistent with this, but were not proven to be it: the suite was running while agents edited live config files in place, and a brief change to .vale.ini or .pre-commit-hooks.yaml would produce the same two failures. The race is real and fixed either way. Implementation Notes `grep -q P <<< "$VAR"` has no separate writer process, so there is nothing to race. It is not a retry or a sleep. 121 sites converted across 9 files, three of them scripts rather than tests: new-agent.sh, new-skill.sh and check-executables-allow-sync.sh. None ships via .pre-commit-hooks.yaml, so no external consumer pins them, and all three are single-pipeline checks whose verdict cannot change. Left alone deliberately: 14 sites whose writer is a command, not a shell builtin — they either absorb the writer's status with `|| true` or are python3 and awk, which write once at exit — and one file with no pipefail. `printf '%s'` sites differ from a here-string only by a trailing newline, which no -q verdict on a non-empty pattern depends on. tests/test-no-pipefail-early-exit-grep.sh is a static guard against new occurrences, discovered automatically by run-tests.sh. It only scans files that set pipefail, joins continuation lines, skips comments, and flags only echo/printf writers. Its first case proves the scanner can fail before its second trusts a clean verdict on the tree. A guard covers exactly the spellings its regex models, so the miss surface was measured rather than assumed. Four were found and closed: pipefail declared as `set -o errexit -o pipefail` (where the old pattern required pipefail to follow the FIRST -o, and a file-level miss skips every site in that file); a writer separated from grep by an intermediate stage; a pipeline wrapped on a trailing `|` rather than a backslash; and readers spelled egrep, fgrep, /bin/grep, `command grep` or with an env-var prefix. Segment characters exclude a bare `&` so `echo ok && other | grep -q x`, whose writer is `other`, does not false-fire. Widening surfaced 5 live sites invisible to the original scanner, all in tests/test-apm-current-hook.sh, all `echo "$out" | json_field ... | grep -q`; they are safe today only because json_field is python3, which reads to EOF and writes once. Fixtures go 4 to 12 vulnerable spellings plus near-miss negatives. Two `grep ... | head -1` sites (test-vale-wrap.sh) are the same race with a different early-exiting reader, and are fixed by absorbing the writer. The scanner deliberately does not model `head`, `sed -n 1p` or a bare `read`: most legitimate uses in this tree are already absorbed with `|| true` and the scanner cannot see absorption from pipeline text, so a high false-positive rate would be how this guard gets weakened. Heredoc bodies are scanned as code; none in the tree trips it today. Impact The bug predates the factory-audit merge: every converted site in check-release-needed and case 21 dates to4d018afandaa8cc22(2026-08-09). Test suites go 19 to 20. `run-tests.sh --strict` passes 20/20 with 0 skipped, four consecutive runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YR2CjVumUbEGWcMikcoXBD
450 lines
22 KiB
Bash
Executable File
450 lines
22 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
|
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
SCRIPT="$REPO_ROOT/scripts/check-release-needed.sh"
|
|
PASS=0
|
|
FAIL=0
|
|
|
|
pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
|
|
fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
|
|
|
|
# Output assertions are `grep -q PATTERN <<< "$OUT"`, never `echo "$OUT" | grep -q`.
|
|
# Under pipefail the pipe form is scheduling-dependent: bash's echo writes a
|
|
# multi-line value one line at a time, `grep -q` exits on its first match, and a
|
|
# later line then hits a closed pipe. echo dies of SIGPIPE, pipefail reports the
|
|
# pipeline as failed, and a correct output reads as a missing match. The
|
|
# here-string has no writer process to race.
|
|
|
|
# Both entry shapes the real .pre-commit-hooks.yaml ships: a bare script with no
|
|
# bundled data, and a bare script whose sibling assets/ tree it self-locates at
|
|
# runtime. Neither carries arguments — pre-commit only rewrites entry[0] to the
|
|
# hook-repo clone path, so an argument path would resolve against the consuming
|
|
# repo. RELEASE_PATHS is derived from the manifest rather than hand-maintained,
|
|
# so it has to cope with both.
|
|
HOOK_DIR="plugins/demo/skills/demo-audit"
|
|
|
|
write_manifest() {
|
|
local dir="$1"
|
|
cat > "$dir/.pre-commit-hooks.yaml" <<EOF
|
|
- id: fake-size-check
|
|
entry: scripts/skill-size-check.sh
|
|
language: script
|
|
- id: fake-vale-check
|
|
entry: $HOOK_DIR/scripts/vale-wrap.sh
|
|
language: script
|
|
EOF
|
|
}
|
|
|
|
# Helper: writes the files both manifest entries expose — the two hook scripts
|
|
# plus the bundled Vale config and style rule the second one self-locates.
|
|
write_release_paths() {
|
|
local dir="$1"
|
|
mkdir -p "$dir/scripts" "$dir/$HOOK_DIR/scripts" "$dir/$HOOK_DIR/assets/vale/styles/Kyberforge"
|
|
echo "v1" > "$dir/scripts/skill-size-check.sh"
|
|
echo "v1" > "$dir/$HOOK_DIR/scripts/vale-wrap.sh"
|
|
echo "cfg" > "$dir/$HOOK_DIR/assets/vale/.vale.ini"
|
|
echo "rule: v1" > "$dir/$HOOK_DIR/assets/vale/styles/Kyberforge/DemoRule.yml"
|
|
}
|
|
|
|
# Helper: a fixture repo with a manifest and every release-relevant path it
|
|
# exposes, committed and tagged v1.0.0.
|
|
make_tagged_fixture() {
|
|
local dir
|
|
dir="$(mktemp -d)"
|
|
(cd "$dir" && git init -q && git config user.email t@t.t && git config user.name t)
|
|
write_manifest "$dir"
|
|
write_release_paths "$dir"
|
|
(cd "$dir" && git add -A && git commit -q -m "initial" && git tag v1.0.0)
|
|
echo "$dir"
|
|
}
|
|
|
|
# Helper: a fixture whose manifest carries one malformed entry: at the tag *and*
|
|
# at HEAD, plus a post-tag change to the file that entry was meant to cover.
|
|
# Committing the bad entry before the tag is what makes the assertion sharp — an
|
|
# edited manifest is itself release-relevant, so the gate would fail for the
|
|
# wrong reason and hide a parser that degrades silently.
|
|
make_malformed_fixture() {
|
|
local entry="$1" dir
|
|
dir="$(mktemp -d)"
|
|
(cd "$dir" && git init -q && git config user.email t@t.t && git config user.name t)
|
|
write_release_paths "$dir"
|
|
cat > "$dir/.pre-commit-hooks.yaml" <<EOF
|
|
- id: fake-size-check
|
|
entry: $entry
|
|
language: script
|
|
EOF
|
|
(cd "$dir" && git add -A && git commit -q -m "initial" && git tag v1.0.0)
|
|
echo "v2" > "$dir/scripts/skill-size-check.sh"
|
|
(cd "$dir" && git add -A && git commit -q -m "change the file the malformed entry should cover")
|
|
echo "$dir"
|
|
}
|
|
|
|
# $3 is optional: pre-commit's PRE_COMMIT_TO_REF, the local sha being pushed.
|
|
# Left off entirely, the variable stays unset and the script falls back to HEAD,
|
|
# exactly as a plain `git push <remote> <current-branch>` behaves.
|
|
# The fixture repo is the subject under test, so every PRE_COMMIT_* input must
|
|
# come from this function and nowhere else. Any such variable already in the
|
|
# environment belongs to the *caller's* repo: run under the pre-push hook this
|
|
# suite guards, PRE_COMMIT_TO_REF holds a sha of the real repo, which does not
|
|
# exist in the fixture, and the script resolves against the wrong rev. Clearing
|
|
# them is what makes a standalone run and a pre-push run the same test — this
|
|
# suite passed everywhere except under the hook it exists to protect.
|
|
run_check() {
|
|
local dir="$1" branch="$2"
|
|
if [[ $# -ge 3 ]]; then
|
|
(cd "$dir" && unset PRE_COMMIT_FROM_REF \
|
|
&& PRE_COMMIT_REMOTE_BRANCH="$branch" PRE_COMMIT_TO_REF="$3" bash "$SCRIPT" 2>&1)
|
|
else
|
|
(cd "$dir" && unset PRE_COMMIT_FROM_REF PRE_COMMIT_TO_REF \
|
|
&& PRE_COMMIT_REMOTE_BRANCH="$branch" bash "$SCRIPT" 2>&1)
|
|
fi
|
|
}
|
|
|
|
CLEANUP_DIRS=()
|
|
trap 'rm -rf ${CLEANUP_DIRS[@]+"${CLEANUP_DIRS[@]}"}' EXIT
|
|
track() { CLEANUP_DIRS+=("$1"); }
|
|
|
|
# --- 1. Not targeting main: silent no-op regardless of state ---
|
|
echo ""
|
|
echo "--- exits 0 when not pushing to main, even with no tags ---"
|
|
FIXTURE1="$(mktemp -d)"; track "$FIXTURE1"
|
|
(cd "$FIXTURE1" && git init -q)
|
|
if run_check "$FIXTURE1" "refs/heads/feature-branch" > /dev/null; then
|
|
pass "exits 0 when target branch isn't main"
|
|
else
|
|
fail "exited non-zero on a non-main target branch"
|
|
fi
|
|
|
|
# --- 2. Targeting main, no tag exists at all: hard fail ---
|
|
echo ""
|
|
echo "--- exits 1 when targeting main and no tag exists ---"
|
|
FIXTURE2="$(mktemp -d)"; track "$FIXTURE2"
|
|
(cd "$FIXTURE2" && git init -q && git config user.email t@t.t && git config user.name t)
|
|
write_manifest "$FIXTURE2"
|
|
write_release_paths "$FIXTURE2"
|
|
(cd "$FIXTURE2" && git add -A && git commit -q -m "initial")
|
|
if run_check "$FIXTURE2" "refs/heads/main" > /dev/null; then
|
|
fail "exited 0 when targeting main with no tag — expected exit 1"
|
|
else
|
|
pass "exits non-zero when targeting main and no tag exists yet"
|
|
fi
|
|
|
|
# --- 3. Targeting main, tag exists, no release-relevant changes since: passes ---
|
|
echo ""
|
|
echo "--- exits 0 when targeting main and nothing release-relevant changed since the tag ---"
|
|
FIXTURE3="$(make_tagged_fixture)"; track "$FIXTURE3"
|
|
echo "unrelated" > "$FIXTURE3/README.md"
|
|
(cd "$FIXTURE3" && git add -A && git commit -q -m "unrelated change")
|
|
if run_check "$FIXTURE3" "refs/heads/main" > /dev/null; then
|
|
pass "exits 0 when only unrelated files changed since the tag"
|
|
else
|
|
fail "exited non-zero despite no release-relevant changes since the tag"
|
|
fi
|
|
|
|
# --- 4. Targeting main, tag exists, a release-relevant file changed since: hard fail ---
|
|
echo ""
|
|
echo "--- exits 1 when a release-relevant file changed since the tag ---"
|
|
FIXTURE4="$(make_tagged_fixture)"; track "$FIXTURE4"
|
|
echo "v2" > "$FIXTURE4/scripts/skill-size-check.sh"
|
|
(cd "$FIXTURE4" && git add -A && git commit -q -m "update release-relevant script")
|
|
OUT4=$(run_check "$FIXTURE4" "refs/heads/main" || true)
|
|
if grep -q "skill-size-check.sh" <<< "$OUT4"; then
|
|
pass "exits non-zero and names the changed file when a release-relevant path changed since the tag"
|
|
else
|
|
fail "did not flag the release-relevant file that changed since the tag"
|
|
fi
|
|
|
|
# --- 5. Not targeting main even with release-relevant changes and a tag: still a no-op ---
|
|
echo ""
|
|
echo "--- exits 0 on a feature branch even with release-relevant changes since the tag ---"
|
|
FIXTURE5="$(make_tagged_fixture)"; track "$FIXTURE5"
|
|
echo "v2" > "$FIXTURE5/scripts/skill-size-check.sh"
|
|
(cd "$FIXTURE5" && git add -A && git commit -q -m "update release-relevant script")
|
|
if run_check "$FIXTURE5" "refs/heads/some-feature" > /dev/null; then
|
|
pass "exits 0 on a feature branch regardless of un-tagged release-relevant changes"
|
|
else
|
|
fail "hard-failed on a feature branch — should only ever fail when targeting main"
|
|
fi
|
|
|
|
# --- 6. A release-relevant path deleted since the tag is still flagged ---
|
|
echo ""
|
|
echo "--- exits 1 when a release-relevant path was deleted since the tag, not just modified ---"
|
|
FIXTURE6="$(make_tagged_fixture)"; track "$FIXTURE6"
|
|
rm -f "$FIXTURE6/$HOOK_DIR/assets/vale/.vale.ini"
|
|
(cd "$FIXTURE6" && git add -A && git commit -q -m "delete the bundled vale config")
|
|
OUT6=$(run_check "$FIXTURE6" "refs/heads/main" || true)
|
|
if grep -q "assets/vale/.vale.ini" <<< "$OUT6"; then
|
|
pass "flags a deleted release-relevant path instead of silently dropping it from the diff"
|
|
else
|
|
fail "did not flag deletion of a release-relevant path since the tag"
|
|
fi
|
|
|
|
# --- 7. A git diff failure hard-fails instead of reading as a clean pass ---
|
|
echo ""
|
|
echo "--- exits 1 (not a silent pass) when the underlying git diff errors out ---"
|
|
FIXTURE7="$(make_tagged_fixture)"; track "$FIXTURE7"
|
|
TAG_TREE="$(cd "$FIXTURE7" && git rev-parse 'v1.0.0^{tree}')"
|
|
echo "v2" > "$FIXTURE7/scripts/skill-size-check.sh"
|
|
(cd "$FIXTURE7" && git add -A && git commit -q -m "advance past the tag")
|
|
rm -f "$FIXTURE7/.git/objects/${TAG_TREE:0:2}/${TAG_TREE:2}"
|
|
if run_check "$FIXTURE7" "refs/heads/main" > /dev/null; then
|
|
fail "silently exited 0 when the underlying git diff failed"
|
|
else
|
|
pass "hard-fails instead of silently passing when git diff can't be computed"
|
|
fi
|
|
|
|
# --- 8. A non-version tag reachable from HEAD does not become the diff baseline ---
|
|
echo ""
|
|
echo "--- ignores a non-vX.Y.Z tag and still flags a change since the real release tag ---"
|
|
FIXTURE8="$(make_tagged_fixture)"; track "$FIXTURE8"
|
|
echo "checkpoint" > "$FIXTURE8/scripts/skill-size-check.sh"
|
|
(cd "$FIXTURE8" && git add -A && git commit -q -m "checkpoint work" && git tag checkpoint-1)
|
|
echo "v2" > "$FIXTURE8/scripts/skill-size-check.sh"
|
|
(cd "$FIXTURE8" && git add -A && git commit -q -m "real release-relevant change")
|
|
OUT8=$(run_check "$FIXTURE8" "refs/heads/main" || true)
|
|
if grep -q "skill-size-check.sh" <<< "$OUT8"; then
|
|
pass "still flags the release-relevant change since v1.0.0, ignoring the non-version checkpoint tag"
|
|
else
|
|
fail "an incidental non-version tag shifted the baseline and hid a real release-relevant change"
|
|
fi
|
|
|
|
# --- 9. A file outside every manifest entry does not trigger a fail ---
|
|
echo ""
|
|
echo "--- exits 0 when a changed file sits near, but isn't referenced by, a manifest entry ---"
|
|
FIXTURE9="$(make_tagged_fixture)"; track "$FIXTURE9"
|
|
echo "irrelevant" > "$FIXTURE9/scripts/unrelated-helper.sh"
|
|
(cd "$FIXTURE9" && git add -A && git commit -q -m "add an unrelated script alongside the exposed one")
|
|
if run_check "$FIXTURE9" "refs/heads/main" > /dev/null; then
|
|
pass "exits 0 for a file that lives alongside, but isn't referenced by, any manifest entry"
|
|
else
|
|
fail "flagged a file that no .pre-commit-hooks.yaml entry actually exposes"
|
|
fi
|
|
|
|
# --- 10. A change confined to a hook's bundled styles/ tree is release-relevant ---
|
|
# The manifest entry names only the wrapper script; the Vale rules it enforces
|
|
# live in the sibling assets/ tree it self-locates at runtime. If that tree is
|
|
# not covered, editing a rule and landing it on main demands no new tag, and a
|
|
# consumer pinned to the old rev keeps the stale rules forever.
|
|
echo ""
|
|
echo "--- exits 1 when only a bundled Vale style rule changed since the tag ---"
|
|
FIXTURE10="$(make_tagged_fixture)"; track "$FIXTURE10"
|
|
echo "rule: v2" > "$FIXTURE10/$HOOK_DIR/assets/vale/styles/Kyberforge/DemoRule.yml"
|
|
(cd "$FIXTURE10" && git add -A && git commit -q -m "tighten a vale rule")
|
|
OUT10=$(run_check "$FIXTURE10" "refs/heads/main" || true)
|
|
if grep -q "assets/vale/styles/Kyberforge/DemoRule.yml" <<< "$OUT10"; then
|
|
pass "flags a change confined to a hook's bundled assets/vale/styles/ tree"
|
|
else
|
|
fail "a bundled Vale style rule changed since the tag without demanding a release"
|
|
fi
|
|
|
|
# --- 11. The assets/ derivation must not invent a path for a bundle-less hook ---
|
|
# scripts/skill-size-check.sh has no sibling assets/ tree, so its derived
|
|
# candidate normalises to a bare top-level assets/ — a directory this repo does
|
|
# not ship. Adding it unconditionally would make any unrelated repo-root
|
|
# assets/ file falsely demand a release.
|
|
echo ""
|
|
echo "--- exits 0 when a top-level assets/ file changed but no hook bundles one ---"
|
|
FIXTURE11="$(make_tagged_fixture)"; track "$FIXTURE11"
|
|
mkdir -p "$FIXTURE11/assets"
|
|
echo "unrelated" > "$FIXTURE11/assets/logo.txt"
|
|
(cd "$FIXTURE11" && git add -A && git commit -q -m "add an unrelated top-level assets file")
|
|
if run_check "$FIXTURE11" "refs/heads/main" > /dev/null; then
|
|
pass "exits 0 for a top-level assets/ file that no manifest entry bundles"
|
|
else
|
|
fail "invented a bogus assets/ path for a hook script with no bundled tree"
|
|
fi
|
|
|
|
# --- 12. Deleting a hook's entire bundled assets/ tree is release-relevant ---
|
|
# The worktree-only derivation guarded the assets/ path on the directory still
|
|
# existing, so wiping the whole tree removed the path from the pathspec instead
|
|
# of diffing it: the single most consumer-breaking change possible diffed clean.
|
|
# The path list therefore has to be unioned with what $LAST_TAG exposed.
|
|
echo ""
|
|
echo "--- exits 1 when a hook's entire bundled assets/ tree was deleted since the tag ---"
|
|
FIXTURE12="$(make_tagged_fixture)"; track "$FIXTURE12"
|
|
rm -rf "${FIXTURE12:?}/$HOOK_DIR/assets"
|
|
(cd "$FIXTURE12" && git add -A && git commit -q -m "delete the whole bundled assets tree")
|
|
OUT12=$(run_check "$FIXTURE12" "refs/heads/main" || true)
|
|
if grep -q "assets/vale/.vale.ini" <<< "$OUT12"; then
|
|
pass "flags a wholesale deletion of a hook's bundled assets/ tree"
|
|
else
|
|
fail "a hook's entire bundled assets/ tree vanished since the tag without demanding a release"
|
|
fi
|
|
|
|
# --- 13. A hook script deleted while its manifest entry survives is flagged ---
|
|
# Characterisation test, not a bug fix: tokens[0] is added to the pathspec
|
|
# unconditionally (no existence guard), so this case was already covered. It is
|
|
# pinned here so the tagged-tree union can't accidentally introduce an existence
|
|
# guard on tokens[0] and reopen the hole its assets/ sibling had.
|
|
echo ""
|
|
echo "--- exits 1 when a hook script was deleted but its manifest entry remains ---"
|
|
FIXTURE13="$(make_tagged_fixture)"; track "$FIXTURE13"
|
|
rm -f "$FIXTURE13/$HOOK_DIR/scripts/vale-wrap.sh"
|
|
(cd "$FIXTURE13" && git add -A && git commit -q -m "delete a hook script, keep its manifest entry")
|
|
OUT13=$(run_check "$FIXTURE13" "refs/heads/main" || true)
|
|
if grep -q "vale-wrap.sh" <<< "$OUT13"; then
|
|
pass "flags a hook script deleted out from under a surviving manifest entry"
|
|
else
|
|
fail "a manifest entry's script vanished since the tag without demanding a release"
|
|
fi
|
|
|
|
# --- 14. Retiring a whole hook names what the tag exposed, not just the manifest ---
|
|
# Removing the entry and everything it shipped changes $HOOKS_MANIFEST, so the
|
|
# gate fires either way — but a derivation that only reads the current manifest
|
|
# can no longer name the retired script or its assets, and the failure message
|
|
# understates the breakage to consumers pinned at the old rev. The tagged
|
|
# manifest is what makes those paths reportable.
|
|
echo ""
|
|
echo "--- names the retired hook's own paths when an entry and its files are removed together ---"
|
|
FIXTURE14="$(make_tagged_fixture)"; track "$FIXTURE14"
|
|
cat > "$FIXTURE14/.pre-commit-hooks.yaml" <<'EOF'
|
|
- id: fake-size-check
|
|
entry: scripts/skill-size-check.sh
|
|
language: script
|
|
EOF
|
|
rm -rf "${FIXTURE14:?}/$HOOK_DIR"
|
|
(cd "$FIXTURE14" && git add -A && git commit -q -m "retire the vale hook entirely")
|
|
OUT14=$(run_check "$FIXTURE14" "refs/heads/main" || true)
|
|
if grep -q "vale-wrap.sh" <<< "$OUT14" && grep -q "assets/vale/.vale.ini" <<< "$OUT14"; then
|
|
pass "names the retired hook's script and bundled assets, not just the manifest edit"
|
|
else
|
|
fail "reported only the manifest change and hid which shipped paths the retirement removed"
|
|
fi
|
|
|
|
# --- 15. A multi-token entry: is rejected loudly, not silently mis-parsed ---
|
|
# ADR-0014 binds entries to a bare script path, but nothing enforced it, and the
|
|
# sibling .pre-commit-config.yaml already ships `entry: bash <script>`. Under the
|
|
# old parser tokens[0] became "bash": a pathspec matching nothing (which git diff
|
|
# accepts in silence) and a bundle root of "." (skipped), so the hook's whole
|
|
# surface dropped out of the gate and the post-tag change below diffed clean.
|
|
echo ""
|
|
echo "--- exits 1 naming the hook when an entry: carries more than one token ---"
|
|
# The entry is quoted back verbatim, not just its first token: that is what makes
|
|
# the diagnostic point at the argument the author has to remove, and what
|
|
# distinguishes this from the unresolvable-path rejection test 16 covers.
|
|
FIXTURE15="$(make_malformed_fixture "bash scripts/skill-size-check.sh")"; track "$FIXTURE15"
|
|
OUT15=$(run_check "$FIXTURE15" "refs/heads/main" || true)
|
|
if run_check "$FIXTURE15" "refs/heads/main" > /dev/null; then
|
|
fail "silently exited 0 on a multi-token entry, dropping that hook's paths from the gate"
|
|
elif grep -q "fake-size-check" <<< "$OUT15" \
|
|
&& grep -q "bash scripts/skill-size-check.sh" <<< "$OUT15" \
|
|
&& grep -q "ADR-0014" <<< "$OUT15"; then
|
|
pass "rejects a multi-token entry, quoting it back and naming the hook and ADR-0014"
|
|
else
|
|
fail "rejected the multi-token entry without naming the hook, the entry, and ADR-0014"
|
|
fi
|
|
|
|
# --- 16. An entry naming no file this repo ships is rejected loudly ---
|
|
# The token-count guard alone still lets a single bare command name (`entry:
|
|
# vale`, valid for language: system) through as a pathspec matching nothing.
|
|
# Existence is checked against the union of the worktree and $LAST_TAG, so this
|
|
# cannot misfire on the deletion cases tests 12-14 pin.
|
|
echo ""
|
|
echo "--- exits 1 naming the hook when an entry: names no file in the worktree or at the tag ---"
|
|
FIXTURE16="$(make_malformed_fixture "vale")"; track "$FIXTURE16"
|
|
OUT16=$(run_check "$FIXTURE16" "refs/heads/main" || true)
|
|
if run_check "$FIXTURE16" "refs/heads/main" > /dev/null; then
|
|
fail "silently exited 0 on an entry that names no shipped file"
|
|
elif grep -q "fake-size-check" <<< "$OUT16" && grep -q "ADR-0014" <<< "$OUT16"; then
|
|
pass "rejects an entry that resolves to no file, naming the hook and the ADR-0014 constraint"
|
|
else
|
|
fail "rejected the unresolvable entry without naming the hook and the ADR-0014 constraint"
|
|
fi
|
|
|
|
# --- 17. The pushed ref, not HEAD, is what gets gated ---
|
|
# pre-commit exports the local sha of each pushed ref as PRE_COMMIT_TO_REF.
|
|
# `git push <remote> pushed-tip:main` from a checkout sitting on an older commit
|
|
# is the false-negative direction: HEAD is still at the tag and diffs clean while
|
|
# the branch actually landing on main carries an untagged, release-relevant
|
|
# change. HEAD is reset back to the tag so the two genuinely differ.
|
|
echo ""
|
|
echo "--- exits 1 on a release-relevant change reachable only from PRE_COMMIT_TO_REF ---"
|
|
FIXTURE17="$(make_tagged_fixture)"; track "$FIXTURE17"
|
|
echo "v2" > "$FIXTURE17/scripts/skill-size-check.sh"
|
|
(cd "$FIXTURE17" && git add -A && git commit -q -m "release-relevant change" \
|
|
&& git branch pushed-tip && git reset -q --hard v1.0.0)
|
|
OUT17=$(run_check "$FIXTURE17" "refs/heads/main" "pushed-tip" || true)
|
|
if grep -q "skill-size-check.sh" <<< "$OUT17"; then
|
|
pass "gates the pushed ref's tip, not HEAD, when HEAD is behind it"
|
|
else
|
|
fail "diffed HEAD instead of PRE_COMMIT_TO_REF and missed a release-relevant change"
|
|
fi
|
|
|
|
# --- 18. Neither the diff tip nor the tag baseline may come from a newer HEAD ---
|
|
# The false-positive direction: HEAD has moved past a v2.0.0 that the pushed ref
|
|
# never saw. Reading either end of the diff off HEAD fails a push that is clean
|
|
# since its own baseline — diffing v2.0.0..HEAD flags HEAD's untagged commit, and
|
|
# resolving the tag from HEAD while diffing pushed-tip flags v2.0.0's change.
|
|
echo ""
|
|
echo "--- exits 0 when the pushed ref is clean since its own tag but HEAD has moved on ---"
|
|
FIXTURE18="$(make_tagged_fixture)"; track "$FIXTURE18"
|
|
(cd "$FIXTURE18" && git branch pushed-tip)
|
|
echo "v2" > "$FIXTURE18/scripts/skill-size-check.sh"
|
|
(cd "$FIXTURE18" && git add -A && git commit -q -m "released change" && git tag v2.0.0)
|
|
echo "v3" > "$FIXTURE18/scripts/skill-size-check.sh"
|
|
(cd "$FIXTURE18" && git add -A && git commit -q -m "unreleased change on HEAD's line")
|
|
if run_check "$FIXTURE18" "refs/heads/main" "pushed-tip" > /dev/null; then
|
|
pass "exits 0 for a pushed ref clean since the tag reachable from it, ignoring HEAD's line"
|
|
else
|
|
fail "gated HEAD's tag or tip and falsely demanded a release for a clean pushed ref"
|
|
fi
|
|
|
|
# --- 19. A branch deletion is a no-op, not a confusing git failure ---
|
|
# pre-commit sets PRE_COMMIT_TO_REF to an all-zeros sha when the push deletes a
|
|
# branch. Nothing is being shipped, and the sha resolves to nothing, so without
|
|
# an explicit guard the gate reports "could not diff" on an unrelated operation.
|
|
echo ""
|
|
echo "--- exits 0 when PRE_COMMIT_TO_REF is the all-zeros branch-deletion sha ---"
|
|
FIXTURE19="$(make_tagged_fixture)"; track "$FIXTURE19"
|
|
echo "v2" > "$FIXTURE19/scripts/skill-size-check.sh"
|
|
(cd "$FIXTURE19" && git add -A && git commit -q -m "release-relevant change")
|
|
if run_check "$FIXTURE19" "refs/heads/main" "0000000000000000000000000000000000000000" > /dev/null; then
|
|
pass "treats an all-zeros PRE_COMMIT_TO_REF as a branch deletion and exits 0"
|
|
else
|
|
fail "turned a branch deletion into a failure instead of a no-op"
|
|
fi
|
|
|
|
# --- 20. The repo's own .pre-commit-hooks.yaml satisfies the entry constraints ---
|
|
# The parser guards above are only safe to ship if the manifest actually in tree
|
|
# passes them. It is replayed into a fixture (with the paths its entries name
|
|
# created) rather than run against the real repo, which has no release tag yet.
|
|
echo ""
|
|
echo "--- accepts the real .pre-commit-hooks.yaml this repo ships ---"
|
|
FIXTURE20="$(mktemp -d)"; track "$FIXTURE20"
|
|
(cd "$FIXTURE20" && git init -q && git config user.email t@t.t && git config user.name t)
|
|
cp "$REPO_ROOT/.pre-commit-hooks.yaml" "$FIXTURE20/.pre-commit-hooks.yaml"
|
|
while IFS= read -r real_entry; do
|
|
mkdir -p "$FIXTURE20/$(dirname "$real_entry")"
|
|
echo "v1" > "$FIXTURE20/$real_entry"
|
|
done < <(sed -n 's/^[[:space:]]*entry:[[:space:]]*//p' "$REPO_ROOT/.pre-commit-hooks.yaml")
|
|
(cd "$FIXTURE20" && git add -A && git commit -q -m "initial" && git tag v1.0.0)
|
|
OUT20=$(run_check "$FIXTURE20" "refs/heads/main" || true)
|
|
if [[ -z "$OUT20" ]]; then
|
|
pass "parses every entry in the repo's real .pre-commit-hooks.yaml without complaint"
|
|
else
|
|
fail "the repo's own .pre-commit-hooks.yaml no longer satisfies the entry constraints: $OUT20"
|
|
fi
|
|
|
|
# --- 21. A vX.Y.Z-suffixed checkpoint tag must not satisfy the release gate ---
|
|
# git describe --match uses shell-glob semantics, not regex: the trailing `*` in
|
|
# 'v[0-9]*.[0-9]*.[0-9]*' matches any suffix, so a pre-release/checkpoint tag like
|
|
# v1.0.1-checkpoint also satisfies the glob and can be picked as LAST_TAG instead
|
|
# of the true last release tag — hiding a real release-relevant change that landed
|
|
# before the checkpoint tag from the diff.
|
|
echo ""
|
|
echo "--- ignores a vX.Y.Z-checkpoint tag and still flags the change since the real release tag ---"
|
|
FIXTURE21="$(make_tagged_fixture)"; track "$FIXTURE21"
|
|
echo "v2" > "$FIXTURE21/scripts/skill-size-check.sh"
|
|
(cd "$FIXTURE21" && git add -A && git commit -q -m "real release-relevant change" && git tag v1.0.1-checkpoint)
|
|
OUT21=$(run_check "$FIXTURE21" "refs/heads/main" || true)
|
|
if grep -q "skill-size-check.sh" <<< "$OUT21"; then
|
|
pass "still flags the release-relevant change since v1.0.0, ignoring the vX.Y.Z-checkpoint tag"
|
|
else
|
|
fail "a vX.Y.Z-checkpoint tag satisfied the glob and hid a real release-relevant change"
|
|
fi
|
|
|
|
echo ""
|
|
echo "Results: $PASS passed, $FAIL failed"
|
|
[[ $FAIL -eq 0 ]]
|