fix(tests): replace pipefail-racy echo | grep -q with here-strings

Why

Two suites failed intermittently — tests/test-vale-wrap.sh case 21 and
tests/test-check-release-needed.sh cases 4 and 15 — on correct output, and never
when run alone. The cause is the `echo "$OUT" | grep -q P` idiom under
`set -o pipefail`: grep -q exits as soon as it has an answer, bash's echo can
hand a multi-line value to the pipe one line at a time, and a write after the
reader is gone kills echo with SIGPIPE. pipefail then reports the writer's
death, so output that DID match reads as "no match". Every observed failure had
lines after its match; case 15's match is on line 1 of 6, the widest window in
that file.

Forced with a pause before the writer's last line, the pipe form failed 50 of 50
runs; a here-string, a match on the last line, and the same pipe without
pipefail each passed 50 of 50. Unforced the rate is about 1 per 670 suite runs,
which is why it read as a flaky gate rather than a bug.

The failures at review time are consistent with this, but were not proven to be
it: the suite was running while agents edited live config files in place, and a
brief change to .vale.ini or .pre-commit-hooks.yaml would produce the same two
failures. The race is real and fixed either way.

Implementation Notes

`grep -q P <<< "$VAR"` has no separate writer process, so there is nothing to
race. It is not a retry or a sleep. 121 sites converted across 9 files, three of
them scripts rather than tests: new-agent.sh, new-skill.sh and
check-executables-allow-sync.sh. None ships via .pre-commit-hooks.yaml, so no
external consumer pins them, and all three are single-pipeline checks whose
verdict cannot change.

Left alone deliberately: 14 sites whose writer is a command, not a shell
builtin — they either absorb the writer's status with `|| true` or are python3
and awk, which write once at exit — and one file with no pipefail. `printf '%s'`
sites differ from a here-string only by a trailing newline, which no -q verdict
on a non-empty pattern depends on.

tests/test-no-pipefail-early-exit-grep.sh is a static guard against new
occurrences, discovered automatically by run-tests.sh. It only scans files that
set pipefail, joins continuation lines, skips comments, and flags only
echo/printf writers. Its first case proves the scanner can fail before its
second trusts a clean verdict on the tree.

A guard covers exactly the spellings its regex models, so the miss surface was
measured rather than assumed. Four were found and closed: pipefail declared as
`set -o errexit -o pipefail` (where the old pattern required pipefail to follow
the FIRST -o, and a file-level miss skips every site in that file); a writer
separated from grep by an intermediate stage; a pipeline wrapped on a trailing
`|` rather than a backslash; and readers spelled egrep, fgrep, /bin/grep,
`command grep` or with an env-var prefix. Segment characters exclude a bare `&`
so `echo ok && other | grep -q x`, whose writer is `other`, does not false-fire.
Widening surfaced 5 live sites invisible to the original scanner, all in
tests/test-apm-current-hook.sh, all `echo "$out" | json_field ... | grep -q`;
they are safe today only because json_field is python3, which reads to EOF and
writes once. Fixtures go 4 to 12 vulnerable spellings plus near-miss negatives.

Two `grep ... | head -1` sites (test-vale-wrap.sh) are the same race with a
different early-exiting reader, and are fixed by absorbing the writer. The
scanner deliberately does not model `head`, `sed -n 1p` or a bare `read`: most
legitimate uses in this tree are already absorbed with `|| true` and the scanner
cannot see absorption from pipeline text, so a high false-positive rate would be
how this guard gets weakened. Heredoc bodies are scanned as code; none in the
tree trips it today.

Impact

The bug predates the factory-audit merge: every converted site in
check-release-needed and case 21 dates to 4d018af and aa8cc22 (2026-08-09).

Test suites go 19 to 20. `run-tests.sh --strict` passes 20/20 with 0 skipped,
four consecutive runs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YR2CjVumUbEGWcMikcoXBD
This commit is contained in:
2026-09-15 21:08:13 +00:00
parent 620f20b0fd
commit ffcbed6c41
13 changed files with 365 additions and 121 deletions

View File

@@ -133,3 +133,7 @@ Widening a description-opener rule to also catch mid-sentence text looked like a
## 2026-08-16 — A rule reversed inside a retrofit leaves no trace unless someone writes it down ## 2026-08-16 — A rule reversed inside a retrofit leaves no trace unless someone writes it down
A retrofit replaced "keep reference chains one level deep" with "two hops, never three" — the opposite rule, needed because the new dispatch pattern requires `SKILL.md` → `improve.md` → `retrofit.md`. The ADR never mentioned chain depth, so the reversal was carried entirely by the diff with no sign a contradicting rule ever existed. Fix: when a change inverts a standing rule, record the inversion where the rule's rationale lives, or it reads as forgotten rather than overturned. A retrofit replaced "keep reference chains one level deep" with "two hops, never three" — the opposite rule, needed because the new dispatch pattern requires `SKILL.md` → `improve.md` → `retrofit.md`. The ADR never mentioned chain depth, so the reversal was carried entirely by the diff with no sign a contradicting rule ever existed. Fix: when a change inverts a standing rule, record the inversion where the rule's rationale lives, or it reads as forgotten rather than overturned.
## 2026-09-15 — A rare flake in a pipefail suite is a race until proven otherwise
The pre-push `run-tests` failed 2 of 3 full runs, on a different suite each time, and neither failure reproduced alone, so it was treated as noise. The cause was `echo "$OUT" | grep -q P` under `set -o pipefail`, at 116 sites. `grep -q` exits on its first match, `echo` takes SIGPIPE on its next write, and pipefail reports correct output as "no match". Unforced it failed about once in 670 runs; with a pause forced before the last line, 50 of 50. Fix: use `grep -q P <<< "$OUT"` (a here-string has no writer process to race), and add a static guard (`tests/test-no-pipefail-early-exit-grep.sh`) instead of relying on convention.

View File

@@ -83,7 +83,7 @@ Where the 276 s goes (each suite run alone, sequential):
Five suites account for 215 s of 276 s. Three of those five (sync-plugin-content, vale-style-sync, adr0020-differential) test tooling that findings 2, 7, and 14 propose to delete or shrink, so the fastest path to a quick pre-push is removing the duplication those tests guard rather than optimising the tests. Five suites account for 215 s of 276 s. Three of those five (sync-plugin-content, vale-style-sync, adr0020-differential) test tooling that findings 2, 7, and 14 propose to delete or shrink, so the fastest path to a quick pre-push is removing the duplication those tests guard rather than optimising the tests.
> **Also struck (2026-09-15):** `test-check-vale-style-sync.sh` (25 s) went with the `check-vale-style-sync` hook in finding 14's merge (ADR-0025). Measured at HEAD: `tests/` holds **19** `test-*.sh` suites and `tests/run-tests.sh` reports `19 passed, 0 skipped, 0 failed`. Same basis as the note below — arithmetic on the 2026-09-10 baseline minus the struck rows, not a fresh timing run. > **Also struck (2026-09-15):** `test-check-vale-style-sync.sh` (25 s) went with the `check-vale-style-sync` hook in finding 14's merge (ADR-0025). Measured at HEAD: `tests/` holds **19** `test-*.sh` suites and `tests/run-tests.sh` reports `19 passed, 0 skipped, 0 failed`. (Later the same day, the pipefail-race fix added `tests/test-no-pipefail-early-exit-grep.sh`, making it **20**. That suite is a static scan and runs in well under a second, so the timing arithmetic here is unaffected.) Same basis as the note below — arithmetic on the 2026-09-10 baseline minus the struck rows, not a fresh timing run.
> **Done (2026-09-14):** see commit `718c79a` on `docs/simplification-audit`. The three struck-through rows are gone: `test-sync-plugin-content.sh` (83 s, 1,289 lines, 92 cases), `check-plugin-content-sync` (4.5 s) and `validate-plugins` (4.9 s). Expected, not re-measured: roughly 92 s comes off every push (83 + 4.5 + 4.9 = 92.4 s) (~83 s of it out of `run-tests`, which loses its single slowest suite), on the arithmetic of the 2026-09-10 figures alone. The remaining rows have not been re-timed since, so treat every number in this section as the 2026-09-10 baseline minus those three, not as a fresh measurement. > **Done (2026-09-14):** see commit `718c79a` on `docs/simplification-audit`. The three struck-through rows are gone: `test-sync-plugin-content.sh` (83 s, 1,289 lines, 92 cases), `check-plugin-content-sync` (4.5 s) and `validate-plugins` (4.9 s). Expected, not re-measured: roughly 92 s comes off every push (83 + 4.5 + 4.9 = 92.4 s) (~83 s of it out of `run-tests`, which loses its single slowest suite), on the arithmetic of the 2026-09-10 figures alone. The remaining rows have not been re-timed since, so treat every number in this section as the 2026-09-10 baseline minus those three, not as a fresh measurement.

View File

@@ -862,6 +862,16 @@ vale-less PATH, with the skip list swallowed. Without vale, two suites skip —
what to install. (It was three until `test-check-vale-style-sync.sh` was deleted with its hook; see what to install. (It was three until `test-check-vale-style-sync.sh` was deleted with its hook; see
[One copy, one config](#one-copy-one-config).) [One copy, one config](#one-copy-one-config).)
**Output assertions use a here-string, never a pipe.** Write `grep -q PATTERN <<< "$OUT"`, not
`echo "$OUT" | grep -q PATTERN`. Under `set -o pipefail` the pipe form fails depending on timing:
`grep -q` exits on its first match, `echo` takes SIGPIPE on its next write, and pipefail reports that
as the pipeline failing, so output that matched reads as "no match". It showed up as a push gate that
failed about once in 670 runs, on a different suite each time. `tests/test-no-pipefail-early-exit-grep.sh`
scans every tracked shell file that sets pipefail and fails on the pipe form. It does this for `echo`
or `printf` piped into `grep` with `-q`, `-m`, `-l`, `-L`, `--quiet` or `--silent`. It checks its own
scanner against fixtures before trusting a clean result. Pipes from other commands are out of scope.
In practice they either absorb the writer's exit status with `|| true` or write only once, at exit.
`tests/run-bats.sh` derives the set of `.bats` files it expects from `git ls-files`, so a `.bats` `tests/run-bats.sh` derives the set of `.bats` files it expects from `git ls-files`, so a `.bats`
file deleted from the worktree but still tracked in the index fails the run rather than silently file deleted from the worktree but still tracked in the index fails the run rather than silently
shrinking the suite. Remove one with `git rm` (or stage the deletion) when intentional; an untracked shrinking the suite. Remove one with `git rm` (or stage the deletion) when intentional; an untracked

View File

@@ -61,7 +61,7 @@ AGENT_NAME="$1"
ROOT="$2" ROOT="$2"
# Validate agent name format # Validate agent name format
if ! echo "$AGENT_NAME" | grep -qE '^[a-z0-9]+(-[a-z0-9]+)*$'; then if ! grep -qE '^[a-z0-9]+(-[a-z0-9]+)*$' <<< "$AGENT_NAME"; then
echo "Error: agent-name must use lowercase letters, numbers, and hyphens only." >&2 echo "Error: agent-name must use lowercase letters, numbers, and hyphens only." >&2
echo " No leading, trailing, or consecutive hyphens." >&2 echo " No leading, trailing, or consecutive hyphens." >&2
echo " Received: '$AGENT_NAME'" >&2 echo " Received: '$AGENT_NAME'" >&2

View File

@@ -61,7 +61,7 @@ SKILL_NAME="$1"
TARGET_INPUT="$2" TARGET_INPUT="$2"
# Validate skill name format # Validate skill name format
if ! echo "$SKILL_NAME" | grep -qE '^[a-z0-9]+(-[a-z0-9]+)*$'; then if ! grep -qE '^[a-z0-9]+(-[a-z0-9]+)*$' <<< "$SKILL_NAME"; then
echo "Error: skill-name must use lowercase letters, numbers, and hyphens only." >&2 echo "Error: skill-name must use lowercase letters, numbers, and hyphens only." >&2
echo " No leading, trailing, or consecutive hyphens." >&2 echo " No leading, trailing, or consecutive hyphens." >&2
echo " Received: '$SKILL_NAME'" >&2 echo " Received: '$SKILL_NAME'" >&2

View File

@@ -202,7 +202,7 @@ if [[ "$ALLOW_STATE" != "present" ]]; then
exit 1 exit 1
fi fi
if printf '%s\n' "$ALLOW_KEYS" | grep -qxF "$EXPECTED_KEY"; then if grep -qxF "$EXPECTED_KEY" <<< "$ALLOW_KEYS"; then
exit 0 exit 0
fi fi

View File

@@ -114,11 +114,11 @@ if echo "$out" | python3 -m json.tool > /dev/null 2>&1; then
&& pass "declares hookEventName SessionStart" || fail "wrong hookEventName" && pass "declares hookEventName SessionStart" || fail "wrong hookEventName"
[[ "$(echo "$out" | json_field reloadSkills)" == "True" ]] \ [[ "$(echo "$out" | json_field reloadSkills)" == "True" ]] \
&& pass "asks the host to reload skills after a successful refresh" || fail "reloadSkills should be true" && pass "asks the host to reload skills after a successful refresh" || fail "reloadSkills should be true"
echo "$out" | json_field additionalContext | grep -q "6 package" \ grep -q "6 package" <<< "$(json_field additionalContext <<< "$out")" \
&& pass "reports the stale package count" || fail "should report the count" && pass "reports the stale package count" || fail "should report the count"
# The lockfile rewrite is the surprising part of auto-updating; the notice has # The lockfile rewrite is the surprising part of auto-updating; the notice has
# to say so or a dirty worktree looks like something else went wrong. # to say so or a dirty worktree looks like something else went wrong.
echo "$out" | json_field additionalContext | grep -q "apm.lock.yaml" \ grep -q "apm.lock.yaml" <<< "$(json_field additionalContext <<< "$out")" \
&& pass "warns that apm.lock.yaml was rewritten" || fail "should warn about the lockfile rewrite" && pass "warns that apm.lock.yaml was rewritten" || fail "should warn about the lockfile rewrite"
else else
fail "emits valid JSON" fail "emits valid JSON"
@@ -136,7 +136,7 @@ if echo "$out" | python3 -m json.tool > /dev/null 2>&1; then
pass "emits valid JSON on failure" pass "emits valid JSON on failure"
[[ "$(echo "$out" | json_field reloadSkills)" == "False" ]] \ [[ "$(echo "$out" | json_field reloadSkills)" == "False" ]] \
&& pass "does not ask for a skill reload when nothing was deployed" || fail "reloadSkills should be false" && pass "does not ask for a skill reload when nothing was deployed" || fail "reloadSkills should be false"
echo "$out" | json_field additionalContext | grep -q "apm update" \ grep -q "apm update" <<< "$(json_field additionalContext <<< "$out")" \
&& pass "tells the reader how to refresh by hand" || fail "should name the manual command" && pass "tells the reader how to refresh by hand" || fail "should name the manual command"
else else
fail "emits valid JSON on failure" fail "emits valid JSON on failure"
@@ -175,7 +175,7 @@ out="$(run_hook_in "$ELSEWHERE" "$WORK")"
[[ "$(cat "$WORK/apm-cwd" 2>/dev/null)" == "$WORK" ]] \ [[ "$(cat "$WORK/apm-cwd" 2>/dev/null)" == "$WORK" ]] \
&& pass "runs apm in the directory the guard checked, not the cwd" \ && pass "runs apm in the directory the guard checked, not the cwd" \
|| fail "apm ran in '$(cat "$WORK/apm-cwd" 2>/dev/null)' — must run in the resolved project directory" || fail "apm ran in '$(cat "$WORK/apm-cwd" 2>/dev/null)' — must run in the resolved project directory"
echo "$out" | json_field additionalContext | grep -q "6 package" \ grep -q "6 package" <<< "$(json_field additionalContext <<< "$out")" \
&& pass "reports the count found via CLAUDE_PROJECT_DIR" || fail "should report the count" && pass "reports the count found via CLAUDE_PROJECT_DIR" || fail "should report the count"
# The fallback is not cosmetic: a host that installed this plugin natively sets # The fallback is not cosmetic: a host that installed this plugin natively sets
@@ -348,7 +348,7 @@ EOF
chmod +x "$FAKE_BIN/apm" chmod +x "$FAKE_BIN/apm"
rm -f "$WORK/update-was-called" rm -f "$WORK/update-was-called"
out="$(run_hook)" out="$(run_hook)"
if [[ -n "$out" ]] && echo "$out" | json_field additionalContext 2>/dev/null | grep -q "$want package"; then if [[ -n "$out" ]] && grep -q "$want package" <<< "$(json_field additionalContext 2>/dev/null <<< "$out")"; then
pass "detects staleness in real \`apm outdated\` output and counts $want package(s)" pass "detects staleness in real \`apm outdated\` output and counts $want package(s)"
else else
fail "real apm reported $want outdated dependency/dependencies but the hook did not act on it — apm reworded its summary line. Real output: $(tr '\n' ' ' < "$PROBE/genuine-$want.txt" | tail -c 120)" fail "real apm reported $want outdated dependency/dependencies but the hook did not act on it — apm reworded its summary line. Real output: $(tr '\n' ' ' < "$PROBE/genuine-$want.txt" | tail -c 120)"

View File

@@ -9,6 +9,13 @@ FAIL=0
pass() { echo " PASS: $1"; PASS=$((PASS + 1)); } pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); } fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
# Output assertions are `grep -q PATTERN <<< "$OUT"`, never `echo "$OUT" | grep -q`.
# Under pipefail the pipe form is scheduling-dependent: bash's echo writes a
# multi-line value one line at a time, `grep -q` exits on its first match, and a
# later line then hits a closed pipe. echo dies of SIGPIPE, pipefail reports the
# pipeline as failed, and a correct output reads as a missing match. The
# here-string has no writer process to race.
# Both entry shapes the real .pre-commit-hooks.yaml ships: a bare script with no # Both entry shapes the real .pre-commit-hooks.yaml ships: a bare script with no
# bundled data, and a bare script whose sibling assets/ tree it self-locates at # bundled data, and a bare script whose sibling assets/ tree it self-locates at
# runtime. Neither carries arguments — pre-commit only rewrites entry[0] to the # runtime. Neither carries arguments — pre-commit only rewrites entry[0] to the
@@ -142,7 +149,7 @@ FIXTURE4="$(make_tagged_fixture)"; track "$FIXTURE4"
echo "v2" > "$FIXTURE4/scripts/skill-size-check.sh" echo "v2" > "$FIXTURE4/scripts/skill-size-check.sh"
(cd "$FIXTURE4" && git add -A && git commit -q -m "update release-relevant script") (cd "$FIXTURE4" && git add -A && git commit -q -m "update release-relevant script")
OUT4=$(run_check "$FIXTURE4" "refs/heads/main" || true) OUT4=$(run_check "$FIXTURE4" "refs/heads/main" || true)
if echo "$OUT4" | grep -q "skill-size-check.sh"; then if grep -q "skill-size-check.sh" <<< "$OUT4"; then
pass "exits non-zero and names the changed file when a release-relevant path changed since the tag" pass "exits non-zero and names the changed file when a release-relevant path changed since the tag"
else else
fail "did not flag the release-relevant file that changed since the tag" fail "did not flag the release-relevant file that changed since the tag"
@@ -167,7 +174,7 @@ FIXTURE6="$(make_tagged_fixture)"; track "$FIXTURE6"
rm -f "$FIXTURE6/$HOOK_DIR/assets/vale/.vale.ini" rm -f "$FIXTURE6/$HOOK_DIR/assets/vale/.vale.ini"
(cd "$FIXTURE6" && git add -A && git commit -q -m "delete the bundled vale config") (cd "$FIXTURE6" && git add -A && git commit -q -m "delete the bundled vale config")
OUT6=$(run_check "$FIXTURE6" "refs/heads/main" || true) OUT6=$(run_check "$FIXTURE6" "refs/heads/main" || true)
if echo "$OUT6" | grep -q "assets/vale/.vale.ini"; then if grep -q "assets/vale/.vale.ini" <<< "$OUT6"; then
pass "flags a deleted release-relevant path instead of silently dropping it from the diff" pass "flags a deleted release-relevant path instead of silently dropping it from the diff"
else else
fail "did not flag deletion of a release-relevant path since the tag" fail "did not flag deletion of a release-relevant path since the tag"
@@ -196,7 +203,7 @@ echo "checkpoint" > "$FIXTURE8/scripts/skill-size-check.sh"
echo "v2" > "$FIXTURE8/scripts/skill-size-check.sh" echo "v2" > "$FIXTURE8/scripts/skill-size-check.sh"
(cd "$FIXTURE8" && git add -A && git commit -q -m "real release-relevant change") (cd "$FIXTURE8" && git add -A && git commit -q -m "real release-relevant change")
OUT8=$(run_check "$FIXTURE8" "refs/heads/main" || true) OUT8=$(run_check "$FIXTURE8" "refs/heads/main" || true)
if echo "$OUT8" | grep -q "skill-size-check.sh"; then if grep -q "skill-size-check.sh" <<< "$OUT8"; then
pass "still flags the release-relevant change since v1.0.0, ignoring the non-version checkpoint tag" pass "still flags the release-relevant change since v1.0.0, ignoring the non-version checkpoint tag"
else else
fail "an incidental non-version tag shifted the baseline and hid a real release-relevant change" fail "an incidental non-version tag shifted the baseline and hid a real release-relevant change"
@@ -225,7 +232,7 @@ FIXTURE10="$(make_tagged_fixture)"; track "$FIXTURE10"
echo "rule: v2" > "$FIXTURE10/$HOOK_DIR/assets/vale/styles/Kyberforge/DemoRule.yml" echo "rule: v2" > "$FIXTURE10/$HOOK_DIR/assets/vale/styles/Kyberforge/DemoRule.yml"
(cd "$FIXTURE10" && git add -A && git commit -q -m "tighten a vale rule") (cd "$FIXTURE10" && git add -A && git commit -q -m "tighten a vale rule")
OUT10=$(run_check "$FIXTURE10" "refs/heads/main" || true) OUT10=$(run_check "$FIXTURE10" "refs/heads/main" || true)
if echo "$OUT10" | grep -q "assets/vale/styles/Kyberforge/DemoRule.yml"; then if grep -q "assets/vale/styles/Kyberforge/DemoRule.yml" <<< "$OUT10"; then
pass "flags a change confined to a hook's bundled assets/vale/styles/ tree" pass "flags a change confined to a hook's bundled assets/vale/styles/ tree"
else else
fail "a bundled Vale style rule changed since the tag without demanding a release" fail "a bundled Vale style rule changed since the tag without demanding a release"
@@ -259,7 +266,7 @@ FIXTURE12="$(make_tagged_fixture)"; track "$FIXTURE12"
rm -rf "${FIXTURE12:?}/$HOOK_DIR/assets" rm -rf "${FIXTURE12:?}/$HOOK_DIR/assets"
(cd "$FIXTURE12" && git add -A && git commit -q -m "delete the whole bundled assets tree") (cd "$FIXTURE12" && git add -A && git commit -q -m "delete the whole bundled assets tree")
OUT12=$(run_check "$FIXTURE12" "refs/heads/main" || true) OUT12=$(run_check "$FIXTURE12" "refs/heads/main" || true)
if echo "$OUT12" | grep -q "assets/vale/.vale.ini"; then if grep -q "assets/vale/.vale.ini" <<< "$OUT12"; then
pass "flags a wholesale deletion of a hook's bundled assets/ tree" pass "flags a wholesale deletion of a hook's bundled assets/ tree"
else else
fail "a hook's entire bundled assets/ tree vanished since the tag without demanding a release" fail "a hook's entire bundled assets/ tree vanished since the tag without demanding a release"
@@ -276,7 +283,7 @@ FIXTURE13="$(make_tagged_fixture)"; track "$FIXTURE13"
rm -f "$FIXTURE13/$HOOK_DIR/scripts/vale-wrap.sh" rm -f "$FIXTURE13/$HOOK_DIR/scripts/vale-wrap.sh"
(cd "$FIXTURE13" && git add -A && git commit -q -m "delete a hook script, keep its manifest entry") (cd "$FIXTURE13" && git add -A && git commit -q -m "delete a hook script, keep its manifest entry")
OUT13=$(run_check "$FIXTURE13" "refs/heads/main" || true) OUT13=$(run_check "$FIXTURE13" "refs/heads/main" || true)
if echo "$OUT13" | grep -q "vale-wrap.sh"; then if grep -q "vale-wrap.sh" <<< "$OUT13"; then
pass "flags a hook script deleted out from under a surviving manifest entry" pass "flags a hook script deleted out from under a surviving manifest entry"
else else
fail "a manifest entry's script vanished since the tag without demanding a release" fail "a manifest entry's script vanished since the tag without demanding a release"
@@ -299,7 +306,7 @@ EOF
rm -rf "${FIXTURE14:?}/$HOOK_DIR" rm -rf "${FIXTURE14:?}/$HOOK_DIR"
(cd "$FIXTURE14" && git add -A && git commit -q -m "retire the vale hook entirely") (cd "$FIXTURE14" && git add -A && git commit -q -m "retire the vale hook entirely")
OUT14=$(run_check "$FIXTURE14" "refs/heads/main" || true) OUT14=$(run_check "$FIXTURE14" "refs/heads/main" || true)
if echo "$OUT14" | grep -q "vale-wrap.sh" && echo "$OUT14" | grep -q "assets/vale/.vale.ini"; then if grep -q "vale-wrap.sh" <<< "$OUT14" && grep -q "assets/vale/.vale.ini" <<< "$OUT14"; then
pass "names the retired hook's script and bundled assets, not just the manifest edit" pass "names the retired hook's script and bundled assets, not just the manifest edit"
else else
fail "reported only the manifest change and hid which shipped paths the retirement removed" fail "reported only the manifest change and hid which shipped paths the retirement removed"
@@ -320,9 +327,9 @@ FIXTURE15="$(make_malformed_fixture "bash scripts/skill-size-check.sh")"; track
OUT15=$(run_check "$FIXTURE15" "refs/heads/main" || true) OUT15=$(run_check "$FIXTURE15" "refs/heads/main" || true)
if run_check "$FIXTURE15" "refs/heads/main" > /dev/null; then if run_check "$FIXTURE15" "refs/heads/main" > /dev/null; then
fail "silently exited 0 on a multi-token entry, dropping that hook's paths from the gate" fail "silently exited 0 on a multi-token entry, dropping that hook's paths from the gate"
elif echo "$OUT15" | grep -q "fake-size-check" \ elif grep -q "fake-size-check" <<< "$OUT15" \
&& echo "$OUT15" | grep -q "bash scripts/skill-size-check.sh" \ && grep -q "bash scripts/skill-size-check.sh" <<< "$OUT15" \
&& echo "$OUT15" | grep -q "ADR-0014"; then && grep -q "ADR-0014" <<< "$OUT15"; then
pass "rejects a multi-token entry, quoting it back and naming the hook and ADR-0014" pass "rejects a multi-token entry, quoting it back and naming the hook and ADR-0014"
else else
fail "rejected the multi-token entry without naming the hook, the entry, and ADR-0014" fail "rejected the multi-token entry without naming the hook, the entry, and ADR-0014"
@@ -339,7 +346,7 @@ FIXTURE16="$(make_malformed_fixture "vale")"; track "$FIXTURE16"
OUT16=$(run_check "$FIXTURE16" "refs/heads/main" || true) OUT16=$(run_check "$FIXTURE16" "refs/heads/main" || true)
if run_check "$FIXTURE16" "refs/heads/main" > /dev/null; then if run_check "$FIXTURE16" "refs/heads/main" > /dev/null; then
fail "silently exited 0 on an entry that names no shipped file" fail "silently exited 0 on an entry that names no shipped file"
elif echo "$OUT16" | grep -q "fake-size-check" && echo "$OUT16" | grep -q "ADR-0014"; then elif grep -q "fake-size-check" <<< "$OUT16" && grep -q "ADR-0014" <<< "$OUT16"; then
pass "rejects an entry that resolves to no file, naming the hook and the ADR-0014 constraint" pass "rejects an entry that resolves to no file, naming the hook and the ADR-0014 constraint"
else else
fail "rejected the unresolvable entry without naming the hook and the ADR-0014 constraint" fail "rejected the unresolvable entry without naming the hook and the ADR-0014 constraint"
@@ -358,7 +365,7 @@ echo "v2" > "$FIXTURE17/scripts/skill-size-check.sh"
(cd "$FIXTURE17" && git add -A && git commit -q -m "release-relevant change" \ (cd "$FIXTURE17" && git add -A && git commit -q -m "release-relevant change" \
&& git branch pushed-tip && git reset -q --hard v1.0.0) && git branch pushed-tip && git reset -q --hard v1.0.0)
OUT17=$(run_check "$FIXTURE17" "refs/heads/main" "pushed-tip" || true) OUT17=$(run_check "$FIXTURE17" "refs/heads/main" "pushed-tip" || true)
if echo "$OUT17" | grep -q "skill-size-check.sh"; then if grep -q "skill-size-check.sh" <<< "$OUT17"; then
pass "gates the pushed ref's tip, not HEAD, when HEAD is behind it" pass "gates the pushed ref's tip, not HEAD, when HEAD is behind it"
else else
fail "diffed HEAD instead of PRE_COMMIT_TO_REF and missed a release-relevant change" fail "diffed HEAD instead of PRE_COMMIT_TO_REF and missed a release-relevant change"
@@ -431,7 +438,7 @@ FIXTURE21="$(make_tagged_fixture)"; track "$FIXTURE21"
echo "v2" > "$FIXTURE21/scripts/skill-size-check.sh" echo "v2" > "$FIXTURE21/scripts/skill-size-check.sh"
(cd "$FIXTURE21" && git add -A && git commit -q -m "real release-relevant change" && git tag v1.0.1-checkpoint) (cd "$FIXTURE21" && git add -A && git commit -q -m "real release-relevant change" && git tag v1.0.1-checkpoint)
OUT21=$(run_check "$FIXTURE21" "refs/heads/main" || true) OUT21=$(run_check "$FIXTURE21" "refs/heads/main" || true)
if echo "$OUT21" | grep -q "skill-size-check.sh"; then if grep -q "skill-size-check.sh" <<< "$OUT21"; then
pass "still flags the release-relevant change since v1.0.0, ignoring the vX.Y.Z-checkpoint tag" pass "still flags the release-relevant change since v1.0.0, ignoring the vX.Y.Z-checkpoint tag"
else else
fail "a vX.Y.Z-checkpoint tag satisfied the glob and hid a real release-relevant change" fail "a vX.Y.Z-checkpoint tag satisfied the glob and hid a real release-relevant change"

View File

@@ -0,0 +1,221 @@
#!/usr/bin/env bash
set -euo pipefail
# Static guard: no `echo|printf ... | grep -q` pipeline in a script that runs
# under `set -o pipefail`.
#
# The defect. grep -q (and -m, -l, -L, --quiet, --silent) exits as soon as it
# has an answer. If the writer on the left of the pipe still has output to
# write, that next write hits a closed pipe and the writer dies of SIGPIPE.
# Under pipefail the pipeline then reports the WRITER's failure, so output that
# matched reads as "no match". Whether it happens depends on scheduling: bash's
# echo can hand a multi-line value to the pipe a line at a time, and a reader
# that matched on an early line exits before the later lines arrive. Forced with
# a pause before the last line, the pipe form failed 50 of 50 runs; each of a
# here-string, a match on the last line, and the same pipe without pipefail
# passed 50 of 50. Unforced it surfaced about once per 670 suite runs, which is
# why it read as a flaky gate rather than as a bug.
#
# The fix is `grep -q PATTERN <<< "$VAR"`: a here-string has no separate writer
# process, so there is nothing to race. `printf '%s\n' "$X" | grep` and
# `<<< "$X"` feed grep the same bytes; `printf '%s'` and `echo` differ only by a
# trailing newline, which no -q verdict on a non-empty pattern depends on.
#
# Scope. Only echo/printf writers are flagged: they are the shell builtins that
# can split a write, and they are always replaceable by a here-string. A
# command writer (`run_wrap ... | grep -q`) is not flagged; those sites either
# absorb the writer's status (`|| true`) or write once at exit. Files without
# pipefail are not flagged, because without it the pipeline's status is grep's.
# Comment-only lines are skipped so the pattern can be named in prose.
#
# Remit. This scanner models early-exiting GREP readers only — grep, egrep and
# fgrep, however they are spelled (a path prefix, a `command` prefix, env-var
# assignments), anywhere in the pipeline. They are not the only readers that
# exit early: `head`, `sed -n 1p` and a bare `read` do too, and an echo/printf
# writer feeding any of them is the same race. Those sites are guarded by
# convention instead — absorb the writer's status with `|| true` (or take the
# verdict from a here-string) — and deliberately not by this test, because most
# legitimate uses of them in this tree are already absorbed and the scanner
# cannot see the absorption from the pipeline text alone. Flagging them would be
# noise; the grep readers are flagged because a here-string is always available.
#
# Known limitation: heredoc bodies are scanned as code, so a `cat <<'EOF'` body
# containing a vulnerable-looking line reads as a real site. There are none in
# the tree today.
#
# Case 1 proves the scanner can fail before case 2 trusts its clean verdict on
# the live tree.
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
SELF="$REPO_ROOT/tests/$(basename "${BASH_SOURCE[0]}")"
PASS=0
FAIL=0
pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
RUN_TMP="$(mktemp -d)"
trap 'rm -rf "$RUN_TMP"' EXIT
# A pipeline-safe run of characters: anything but `|`, `;` or `&`, except that a
# `&` immediately followed by a digit is kept so `2>&1` does not end a segment.
# Excluding a bare `&` is what stops `echo ok && other | grep -q x` — a pipeline
# whose writer is `other`, not the echo — from being read as one site.
SAFE='([^|;&]|&[[:digit:]])'
# Writer, then any number of intermediate stages, then an early-exiting grep.
# The reader may be spelled `grep`, `egrep` or `fgrep`, behind a path prefix
# (`/bin/grep`), a `command` prefix, or env-var assignments (`LC_ALL=C grep`).
SITE_RE="(^|[^[:alnum:]_])(echo|printf)[[:space:]]${SAFE}*[|][[:space:]]*"
SITE_RE+="([^|;&[:space:]]${SAFE}*[|][[:space:]]*)*"
SITE_RE+="((command|[A-Za-z_][A-Za-z0-9_]*=[^[:space:]]*)[[:space:]]+)*"
SITE_RE+="([^[:space:];&|]*/)?(grep|egrep|fgrep)[[:space:]]"
SITE_RE+="(${SAFE}*[[:space:]])?(-[[:alpha:]]*[qmlL][[:alnum:]]*|--quiet|--silent|--max-count|--files-with)"
# `pipefail` need not follow the first `-o`: `set -o errexit -o pipefail` and
# `set -o posix -o pipefail` are the same setting. A file-level miss here skips
# every site in the file, so this is deliberately loose — the cost of scanning a
# file that does not set pipefail is nil, the cost of skipping one is total.
PIPEFAIL_RE='(^|[^[:alnum:]_-])set[[:space:]]+-[^;#]*pipefail'
# Prints `path:line: text` for every vulnerable pipeline in the given files.
# Continued lines are joined first — both a trailing backslash and a trailing
# `|`, which is equally legal as a line continuation in a pipeline — so a reader
# on the next physical line is still seen; the reported line is where the
# command starts.
scan() {
local f
for f in "$@"; do
grep -Eq "$PIPEFAIL_RE" "$f" || continue
awk -v file="$f" -v re="$SITE_RE" '
function flush( b) {
if (buf == "") return
if (buf !~ /^[[:space:]]*#/ && buf ~ re) print file ":" start ": " buf
buf = ""
}
{
if (buf == "") start = NR
line = $0
if (line ~ /\\$/) { buf = buf substr(line, 1, length(line) - 1) " "; next }
if (line ~ /[|][[:space:]]*$/) { buf = buf line " "; next }
buf = buf line
flush()
}
END { flush() }
' "$f"
done
}
# ---------------------------------------------------------------------------
echo "--- the scanner flags each vulnerable spelling and nothing else ---"
# ---------------------------------------------------------------------------
BAD="$RUN_TMP/bad.sh"
cat > "$BAD" <<'EOF_BAD'
#!/usr/bin/env bash
set -euo pipefail
if echo "$OUT" | grep -q "needle"; then :; fi
! printf '%s\n' "$OUT" | grep -qF "needle" && :
if printf '%s' "$OUT" | grep -E -q "a|b"; then :; fi
if printf '%s\n' "$OUT" \
| grep -m1 "needle"; then :; fi
if echo "$OUT" | tr -d ' ' | grep -q "needle"; then :; fi
echo "$OUT" |
grep -q "needle" && :
if echo "$OUT" | egrep -q "needle"; then :; fi
if printf '%s\n' "$OUT" | fgrep -q "needle"; then :; fi
if echo "$OUT" | /bin/grep -q "needle"; then :; fi
if echo "$OUT" | command grep -q "needle"; then :; fi
if echo "$OUT" | LC_ALL=C grep -q "needle"; then :; fi
if echo "$OUT" | sed -n '1,$p' | command /usr/bin/grep --quiet "needle"; then :; fi
EOF_BAD
ALTSET="$RUN_TMP/alt-pipefail.sh"
cat > "$ALTSET" <<'EOF_ALT'
#!/usr/bin/env bash
set -o errexit -o pipefail
if echo "$OUT" | grep -q "needle"; then :; fi
EOF_ALT
GOOD="$RUN_TMP/good.sh"
cat > "$GOOD" <<'EOF_GOOD'
#!/usr/bin/env bash
set -euo pipefail
# A comment naming the bad form: echo "$OUT" | grep -q needle
if grep -q "needle" <<< "$OUT"; then :; fi
COUNT="$(echo "$OUT" | grep -c "needle" || true)"
echo "$OUT" || grep -q "needle" <<< "$OUT"
run_wrap "$DIR" file.md | grep -q "needle" || :
printf '%s\n' "$OUT" | grep -c "needle"
echo "$OUT" | tail -n 1
echo "$OUT" | tr -d ' ' | tail -n 1
echo "$OUT" | mygrep -q "needle"
echo "$OUT" | command tail -n 1
echo "$OUT" | LC_ALL=C sort
echo "$OUT" |
tail -n 1
echo ok && run_wrap "$DIR" | grep -q "needle"
EOF_GOOD
NOPIPEFAIL="$RUN_TMP/no-pipefail.sh"
cat > "$NOPIPEFAIL" <<'EOF_NOPF'
#!/usr/bin/env bash
set -eu
if echo "$OUT" | grep -q "needle"; then :; fi
EOF_NOPF
BAD_HITS="$(scan "$BAD")"
BAD_COUNT="$(grep -c . <<< "$BAD_HITS" || true)"
if [[ "$BAD_COUNT" -eq 12 ]]; then
pass "all 12 vulnerable spellings are flagged (echo, negated printf, split options, both continuations, an intermediate stage, egrep/fgrep, and path/command/env-assignment reader prefixes)"
else
fail "expected 12 hits in the bad fixture, got $BAD_COUNT: $BAD_HITS"
fi
if grep -q "^$BAD:6: " <<< "$BAD_HITS"; then
pass "a backslash-continued pipe is reported at the line the command starts"
else
fail "the backslash-continued site was not reported at line 6: $BAD_HITS"
fi
if grep -q "^$BAD:9: " <<< "$BAD_HITS"; then
pass "a pipeline wrapped after a trailing '|' is reported at the line the command starts"
else
fail "the trailing-pipe continuation site was not reported at line 9: $BAD_HITS"
fi
ALT_HITS="$(scan "$ALTSET")"
if [[ -n "$ALT_HITS" ]]; then
pass "pipefail set as 'set -o errexit -o pipefail' is recognised, so its sites are scanned"
else
fail "a file setting pipefail after another -o option was skipped entirely"
fi
GOOD_HITS="$(scan "$GOOD")"
if [[ -z "$GOOD_HITS" ]]; then
pass "here-strings, comments, grep -c, '||', '&&', command writers and non-grep readers are left alone"
else
fail "the clean fixture was flagged: $GOOD_HITS"
fi
if [[ -z "$(scan "$NOPIPEFAIL")" ]]; then
pass "a file without pipefail is not flagged — grep's own status is the pipeline's"
else
fail "a file without pipefail was flagged"
fi
# ---------------------------------------------------------------------------
echo "--- no tracked shell script carries the race ---"
# ---------------------------------------------------------------------------
FILES=()
while IFS= read -r rel; do
[[ "$REPO_ROOT/$rel" == "$SELF" ]] && continue
[[ -f "$REPO_ROOT/$rel" ]] && FILES+=("$REPO_ROOT/$rel")
done < <(git -C "$REPO_ROOT" ls-files -- '*.sh' '*.bats' '*.bash')
if [[ "${#FILES[@]}" -lt 20 ]]; then
fail "only ${#FILES[@]} tracked shell files found — the scan is looking in the wrong place"
else
LIVE_HITS="$(scan ${FILES[@]+"${FILES[@]}"})"
if [[ -z "$LIVE_HITS" ]]; then
pass "none of ${#FILES[@]} tracked shell files pipes echo/printf into an early-exit grep under pipefail"
else
fail "use \`grep -q PATTERN <<< \"\$VAR\"\` instead at:"
echo "${LIVE_HITS//"$REPO_ROOT/"/ }"
fi
fi
echo ""
echo "Results: $PASS passed, $FAIL failed"
[[ "$FAIL" -eq 0 ]]

View File

@@ -112,7 +112,7 @@ EOF
run_fake "$DIR1" run_fake "$DIR1"
if [[ $FAKE_RC -eq 0 ]]; then if [[ $FAKE_RC -eq 0 ]]; then
fail "a bats emitting nothing exited 0 — a total harness failure reported as a pass" fail "a bats emitting nothing exited 0 — a total harness failure reported as a pass"
elif echo "$FAKE_OUT" | grep -q "no TAP output at all"; then elif grep -q "no TAP output at all" <<< "$FAKE_OUT"; then
pass "an empty TAP stream fails the run and names the broken harness" pass "an empty TAP stream fails the run and names the broken harness"
else else
fail "the run failed but not with the broken-harness message: $FAKE_OUT" fail "the run failed but not with the broken-harness message: $FAKE_OUT"
@@ -134,9 +134,9 @@ EOF
run_fake "$DIR2" run_fake "$DIR2"
if [[ $FAKE_RC -eq 0 ]]; then if [[ $FAKE_RC -eq 0 ]]; then
fail "every test declaring zero tests exited 0 — wholesale @test removal reported as a pass" fail "every test declaring zero tests exited 0 — wholesale @test removal reported as a pass"
elif echo "$FAKE_OUT" | grep -q "declared 0 tests"; then elif grep -q "declared 0 tests" <<< "$FAKE_OUT"; then
pass "a plan-only TAP stream fails the run and names the removed tests" pass "a plan-only TAP stream fails the run and names the removed tests"
elif echo "$FAKE_OUT" | grep -q "no TAP output at all"; then elif grep -q "no TAP output at all" <<< "$FAKE_OUT"; then
fail "a valid '1..0' plan was misreported as a broken harness — the two branches are not distinguished" fail "a valid '1..0' plan was misreported as a broken harness — the two branches are not distinguished"
else else
fail "the run failed but not with the no-tests-declared message: $FAKE_OUT" fail "the run failed but not with the no-tests-declared message: $FAKE_OUT"
@@ -160,7 +160,7 @@ EOF
run_fake "$DIR3" run_fake "$DIR3"
if [[ $FAKE_RC -ne 0 ]]; then if [[ $FAKE_RC -ne 0 ]]; then
fail "a healthy TAP stream was failed by the zero-count guard: $FAKE_OUT" fail "a healthy TAP stream was failed by the zero-count guard: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^4 tests, 0 failures$"; then elif grep -q "^4 tests, 0 failures$" <<< "$FAKE_OUT"; then
pass "two files reporting two passing tests each aggregate to 4 tests, 0 failures" pass "two files reporting two passing tests each aggregate to 4 tests, 0 failures"
else else
fail "a healthy TAP stream produced the wrong count: $FAKE_OUT" fail "a healthy TAP stream produced the wrong count: $FAKE_OUT"
@@ -187,9 +187,9 @@ else
run_fake "$DIR4" run_fake "$DIR4"
if [[ $FAKE_RC -ne 0 ]]; then if [[ $FAKE_RC -ne 0 ]]; then
fail "an empty .bats file beside a real one failed the run: $FAKE_OUT" fail "an empty .bats file beside a real one failed the run: $FAKE_OUT"
elif ! echo "$FAKE_OUT" | grep -q "^1\.\.0$"; then elif ! grep -q "^1\.\.0$" <<< "$FAKE_OUT"; then
fail "real bats did not emit '1..0' for an empty file, so the case-2 stub no longer matches it: $FAKE_OUT" fail "real bats did not emit '1..0' for an empty file, so the case-2 stub no longer matches it: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^1 tests, 0 failures$"; then elif grep -q "^1 tests, 0 failures$" <<< "$FAKE_OUT"; then
pass "an empty .bats file contributes a '1..0' plan and the suite still passes" pass "an empty .bats file contributes a '1..0' plan and the suite still passes"
else else
fail "the real-bats run passed with an unexpected count: $FAKE_OUT" fail "the real-bats run passed with an unexpected count: $FAKE_OUT"
@@ -215,7 +215,7 @@ EOF
run_fake "$DIR5" run_fake "$DIR5"
if [[ $FAKE_RC -eq 0 ]]; then if [[ $FAKE_RC -eq 0 ]]; then
fail "a 'not ok' TAP result exited 0 — a failing test reported as a pass because only the process status was consulted" fail "a 'not ok' TAP result exited 0 — a failing test reported as a pass because only the process status was consulted"
elif echo "$FAKE_OUT" | grep -q "^2 tests, 2 failures$"; then elif grep -q "^2 tests, 2 failures$" <<< "$FAKE_OUT"; then
pass "'not ok' lines fail the run even when every bats process exits 0" pass "'not ok' lines fail the run even when every bats process exits 0"
else else
fail "the run failed but with the wrong count: $FAKE_OUT" fail "the run failed but with the wrong count: $FAKE_OUT"
@@ -242,7 +242,7 @@ EOF
run_fake "$DIR6" run_fake "$DIR6"
if [[ $FAKE_RC -eq 0 ]]; then if [[ $FAKE_RC -eq 0 ]]; then
fail "a bats process exiting 1 was reported as a pass because only the TAP text was consulted" fail "a bats process exiting 1 was reported as a pass because only the TAP text was consulted"
elif echo "$FAKE_OUT" | grep -q "^2 tests, 0 failures$"; then elif grep -q "^2 tests, 0 failures$" <<< "$FAKE_OUT"; then
pass "a non-zero bats exit fails the run even with no 'not ok' line to find" pass "a non-zero bats exit fails the run even with no 'not ok' line to find"
else else
fail "the run failed but with the wrong count: $FAKE_OUT" fail "the run failed but with the wrong count: $FAKE_OUT"
@@ -282,7 +282,7 @@ run_fake "$DIR7"
# the stated reason: the TAP stream the stub emitted is healthy, so "1 tests, 0 # the stated reason: the TAP stream the stub emitted is healthy, so "1 tests, 0
# failures" proves the zero-count guard did not fire and the empty status is the # failures" proves the zero-count guard did not fire and the empty status is the
# only thing left that can have failed the run. # only thing left that can have failed the run.
if ! echo "$FAKE_OUT" | grep -q "^1 tests, 0 failures$"; then if ! grep -q "^1 tests, 0 failures$" <<< "$FAKE_OUT"; then
fail "the killed job did not leave the healthy TAP stream the case needs: $FAKE_OUT" fail "the killed job did not leave the healthy TAP stream the case needs: $FAKE_OUT"
elif [[ $FAKE_RC -eq 0 ]]; then elif [[ $FAKE_RC -eq 0 ]]; then
fail "an empty status file was counted as a clean exit — a killed job reported as a pass" fail "an empty status file was counted as a clean exit — a killed job reported as a pass"
@@ -312,9 +312,9 @@ rm "$DIR8/tests/b.bats"
run_fake "$DIR8" run_fake "$DIR8"
if [[ $FAKE_RC -eq 0 ]]; then if [[ $FAKE_RC -eq 0 ]]; then
fail "a tracked .bats file gone from the worktree passed — a deleted suite reads as green" fail "a tracked .bats file gone from the worktree passed — a deleted suite reads as green"
elif ! echo "$FAKE_OUT" | grep -q "tracked .bats file(s) were not discovered"; then elif ! grep -q "tracked .bats file(s) were not discovered" <<< "$FAKE_OUT"; then
fail "the run failed but not with the undiscovered-tracked-file message: $FAKE_OUT" fail "the run failed but not with the undiscovered-tracked-file message: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^ tests/b.bats$"; then elif grep -q "^ tests/b.bats$" <<< "$FAKE_OUT"; then
pass "a tracked .bats file missing from the walk fails the run and names the file" pass "a tracked .bats file missing from the walk fails the run and names the file"
else else
fail "the run failed without naming the missing file: $FAKE_OUT" fail "the run failed without naming the missing file: $FAKE_OUT"
@@ -339,7 +339,7 @@ git -C "$DIR8B" add tests/a.bats
run_fake "$DIR8B" run_fake "$DIR8B"
if [[ $FAKE_RC -ne 0 ]]; then if [[ $FAKE_RC -ne 0 ]]; then
fail "an untracked .bats file was reported as a finding: $FAKE_OUT" fail "an untracked .bats file was reported as a finding: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^2 tests, 0 failures$"; then elif grep -q "^2 tests, 0 failures$" <<< "$FAKE_OUT"; then
pass "an untracked .bats file is run without being demanded of the index" pass "an untracked .bats file is run without being demanded of the index"
else else
fail "the untracked-file run passed with the wrong count: $FAKE_OUT" fail "the untracked-file run passed with the wrong count: $FAKE_OUT"
@@ -356,7 +356,7 @@ rm "$DIR8B/tests/b.bats"
run_fake "$DIR8B" run_fake "$DIR8B"
if [[ $FAKE_RC -eq 0 ]]; then if [[ $FAKE_RC -eq 0 ]]; then
fail "the .bats file added to the index a moment ago was not demanded back: $FAKE_OUT" fail "the .bats file added to the index a moment ago was not demanded back: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^ tests/b.bats$"; then elif grep -q "^ tests/b.bats$" <<< "$FAKE_OUT"; then
pass "a file added to the index joins the expected set with no floor to bump" pass "a file added to the index joins the expected set with no floor to bump"
else else
fail "the run failed but did not name the newly tracked file: $FAKE_OUT" fail "the run failed but did not name the newly tracked file: $FAKE_OUT"
@@ -380,9 +380,9 @@ EOF
run_fake "$DIR8D" run_fake "$DIR8D"
if [[ $FAKE_RC -ne 0 ]]; then if [[ $FAKE_RC -ne 0 ]]; then
fail "a non-git tree failed the run: $FAKE_OUT" fail "a non-git tree failed the run: $FAKE_OUT"
elif ! echo "$FAKE_OUT" | grep -q "not a git worktree root"; then elif ! grep -q "not a git worktree root" <<< "$FAKE_OUT"; then
fail "a non-git tree silently skipped the derived expectation with no note: $FAKE_OUT" fail "a non-git tree silently skipped the derived expectation with no note: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^2 tests, 0 failures$"; then elif grep -q "^2 tests, 0 failures$" <<< "$FAKE_OUT"; then
pass "a non-git tree runs the suite and says the expected set could not be derived" pass "a non-git tree runs the suite and says the expected set could not be derived"
else else
fail "the non-git run passed with the wrong count: $FAKE_OUT" fail "the non-git run passed with the wrong count: $FAKE_OUT"
@@ -404,7 +404,7 @@ EOF
run_fake "$DIR9" run_fake "$DIR9"
if [[ $FAKE_RC -eq 0 ]]; then if [[ $FAKE_RC -eq 0 ]]; then
fail "finding no .bats files at all exited 0 — the whole suite can vanish and the run stays green" fail "finding no .bats files at all exited 0 — the whole suite can vanish and the run stays green"
elif echo "$FAKE_OUT" | grep -q "found 0 .bats file"; then elif grep -q "found 0 .bats file" <<< "$FAKE_OUT"; then
pass "finding no .bats files fails the run and says so" pass "finding no .bats files fails the run and says so"
else else
fail "the run failed but not with the zero-files message: $FAKE_OUT" fail "the run failed but not with the zero-files message: $FAKE_OUT"

View File

@@ -115,9 +115,9 @@ EOF
run_fake "$DIR1" run_fake "$DIR1"
if [[ $FAKE_RC -ne 0 ]]; then if [[ $FAKE_RC -ne 0 ]]; then
fail "a healthy fixture failed: $FAKE_OUT" fail "a healthy fixture failed: $FAKE_OUT"
elif ! echo "$FAKE_OUT" | grep -q "^=== bats ===$"; then elif ! grep -q "^=== bats ===$" <<< "$FAKE_OUT"; then
fail "a healthy run never announced the bats leg: $FAKE_OUT" fail "a healthy run never announced the bats leg: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$"; then elif grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$" <<< "$FAKE_OUT"; then
pass "a passing case script and a healthy bats runner report 1 passed, 0 failed" pass "a passing case script and a healthy bats runner report 1 passed, 0 failed"
else else
fail "a healthy run produced the wrong summary: $FAKE_OUT" fail "a healthy run produced the wrong summary: $FAKE_OUT"
@@ -140,7 +140,7 @@ EOF
run_fake "$DIR2" run_fake "$DIR2"
if [[ $FAKE_RC -eq 0 ]]; then if [[ $FAKE_RC -eq 0 ]]; then
fail "a non-executable run-bats.sh exited 0 — the whole bats suite can vanish silently" fail "a non-executable run-bats.sh exited 0 — the whole bats suite can vanish silently"
elif echo "$FAKE_OUT" | grep -q "bats runner not found or not executable"; then elif grep -q "bats runner not found or not executable" <<< "$FAKE_OUT"; then
pass "a non-executable run-bats.sh fails the run and names what is missing" pass "a non-executable run-bats.sh fails the run and names what is missing"
else else
fail "the run failed but not with the missing-runner message: $FAKE_OUT" fail "the run failed but not with the missing-runner message: $FAKE_OUT"
@@ -160,7 +160,7 @@ EOF
run_fake "$DIR3" run_fake "$DIR3"
if [[ $FAKE_RC -eq 0 ]]; then if [[ $FAKE_RC -eq 0 ]]; then
fail "a missing run-bats.sh exited 0 — a rename deletes the bats suite from the run with no diagnostic" fail "a missing run-bats.sh exited 0 — a rename deletes the bats suite from the run with no diagnostic"
elif echo "$FAKE_OUT" | grep -q "bats runner not found or not executable"; then elif grep -q "bats runner not found or not executable" <<< "$FAKE_OUT"; then
pass "a missing run-bats.sh fails the run and names what is missing" pass "a missing run-bats.sh fails the run and names what is missing"
else else
fail "the run failed but not with the missing-runner message: $FAKE_OUT" fail "the run failed but not with the missing-runner message: $FAKE_OUT"
@@ -180,7 +180,7 @@ EOF
run_fake "$DIR4" --bats-only run_fake "$DIR4" --bats-only
if [[ $FAKE_RC -eq 0 ]]; then if [[ $FAKE_RC -eq 0 ]]; then
fail "--bats-only with no runner exited 0 having printed nothing — a total no-op reported as a pass" fail "--bats-only with no runner exited 0 having printed nothing — a total no-op reported as a pass"
elif echo "$FAKE_OUT" | grep -q "bats runner not found or not executable"; then elif grep -q "bats runner not found or not executable" <<< "$FAKE_OUT"; then
pass "--bats-only fails when the runner is missing rather than doing nothing quietly" pass "--bats-only fails when the runner is missing rather than doing nothing quietly"
else else
fail "--bats-only failed but not with the missing-runner message: $FAKE_OUT" fail "--bats-only failed but not with the missing-runner message: $FAKE_OUT"
@@ -232,11 +232,11 @@ add_case "$DIR5B" test-ok.sh <<'EOF'
echo "fine" echo "fine"
EOF EOF
run_fake "$DIR5B" run_fake "$DIR5B"
if echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$"; then if grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$" <<< "$FAKE_OUT"; then
fail "an empty run-bats.sh produced a green summary — the bats suite vanished with no diagnostic" fail "an empty run-bats.sh produced a green summary — the bats suite vanished with no diagnostic"
elif [[ $FAKE_RC -eq 0 ]]; then elif [[ $FAKE_RC -eq 0 ]]; then
fail "an empty run-bats.sh exited 0: $FAKE_OUT" fail "an empty run-bats.sh exited 0: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "without reporting an 'N tests, M failures' summary"; then elif grep -q "without reporting an 'N tests, M failures' summary" <<< "$FAKE_OUT"; then
pass "an empty run-bats.sh fails the run and says the bats suite was never verified" pass "an empty run-bats.sh fails the run and says the bats suite was never verified"
else else
fail "the run failed but not with the no-summary message: $FAKE_OUT" fail "the run failed but not with the no-summary message: $FAKE_OUT"
@@ -264,7 +264,7 @@ EOF
run_fake "$DIR5C" run_fake "$DIR5C"
if [[ $FAKE_RC -eq 0 ]]; then if [[ $FAKE_RC -eq 0 ]]; then
fail "a bats runner reporting 0 tests exited 0 — a suite that executed nothing read as green: $FAKE_OUT" fail "a bats runner reporting 0 tests exited 0 — a suite that executed nothing read as green: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "reported 0 tests"; then elif grep -q "reported 0 tests" <<< "$FAKE_OUT"; then
pass "a bats runner reporting 0 tests fails the run and says the suite executed nothing" pass "a bats runner reporting 0 tests fails the run and says the suite executed nothing"
else else
fail "the run failed but not with the zero-tests message: $FAKE_OUT" fail "the run failed but not with the zero-tests message: $FAKE_OUT"
@@ -298,11 +298,11 @@ kill -9 $PPID
sleep 5 sleep 5
EOF EOF
run_fake "$DIR6" run_fake "$DIR6"
if echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$"; then if grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$" <<< "$FAKE_OUT"; then
fail "an empty status file counted as a pass — a killed job reads as green" fail "an empty status file counted as a pass — a killed job reads as green"
elif [[ $FAKE_RC -eq 0 ]]; then elif [[ $FAKE_RC -eq 0 ]]; then
fail "an empty status file did not fail the run: $FAKE_OUT" fail "an empty status file did not fail the run: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^=== Summary: 0 passed, 0 skipped, 1 failed ===$"; then elif grep -q "^=== Summary: 0 passed, 0 skipped, 1 failed ===$" <<< "$FAKE_OUT"; then
pass "an empty status file is counted as FAILED" pass "an empty status file is counted as FAILED"
else else
fail "an empty status file failed the run with the wrong summary: $FAKE_OUT" fail "an empty status file failed the run with the wrong summary: $FAKE_OUT"
@@ -326,7 +326,7 @@ EOF
run_fake "$DIR7" run_fake "$DIR7"
if [[ $FAKE_RC -eq 0 ]]; then if [[ $FAKE_RC -eq 0 ]]; then
fail "a SIGKILLed job did not fail the run: $FAKE_OUT" fail "a SIGKILLed job did not fail the run: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^=== Summary: 0 passed, 0 skipped, 1 failed ===$"; then elif grep -q "^=== Summary: 0 passed, 0 skipped, 1 failed ===$" <<< "$FAKE_OUT"; then
pass "a job killed with no status file written is counted as FAILED" pass "a job killed with no status file written is counted as FAILED"
else else
fail "a SIGKILLed job failed the run with the wrong summary: $FAKE_OUT" fail "a SIGKILLed job failed the run with the wrong summary: $FAKE_OUT"
@@ -348,7 +348,7 @@ EOF
run_fake "$DIR8" run_fake "$DIR8"
if [[ $FAKE_RC -eq 0 ]]; then if [[ $FAKE_RC -eq 0 ]]; then
fail "a case script with a syntax error did not fail the run: $FAKE_OUT" fail "a case script with a syntax error did not fail the run: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^=== Summary: 0 passed, 0 skipped, 1 failed ===$"; then elif grep -q "^=== Summary: 0 passed, 0 skipped, 1 failed ===$" <<< "$FAKE_OUT"; then
pass "a case script that fails to parse is counted as FAILED" pass "a case script that fails to parse is counted as FAILED"
else else
fail "a syntax error failed the run with the wrong summary: $FAKE_OUT" fail "a syntax error failed the run with the wrong summary: $FAKE_OUT"
@@ -380,11 +380,11 @@ EOF
run_fake "$DIR9" run_fake "$DIR9"
if [[ $FAKE_RC -eq 0 ]]; then if [[ $FAKE_RC -eq 0 ]]; then
fail "a case script exiting 1 did not fail the run: $FAKE_OUT" fail "a case script exiting 1 did not fail the run: $FAKE_OUT"
elif ! echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 1 skipped, 1 failed ===$"; then elif ! grep -q "^=== Summary: 1 passed, 1 skipped, 1 failed ===$" <<< "$FAKE_OUT"; then
fail "the pass/skip/fail split was miscounted: $FAKE_OUT" fail "the pass/skip/fail split was miscounted: $FAKE_OUT"
elif ! echo "$FAKE_OUT" | grep -q "^ test-b-skips.sh$"; then elif ! grep -q "^ test-b-skips.sh$" <<< "$FAKE_OUT"; then
fail "the skipped script was not named in the skip list: $FAKE_OUT" fail "the skipped script was not named in the skip list: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^ test-a-fails.sh$"; then elif grep -q "^ test-a-fails.sh$" <<< "$FAKE_OUT"; then
pass "exit 1 is FAILED, exit 77 is SKIPPED, and both are named in their lists" pass "exit 1 is FAILED, exit 77 is SKIPPED, and both are named in their lists"
else else
fail "the failed script was not named in the failure list: $FAKE_OUT" fail "the failed script was not named in the failure list: $FAKE_OUT"
@@ -422,11 +422,11 @@ EOF
run_fake "$DIR10" --strict run_fake "$DIR10" --strict
if [[ $FAKE_RC -eq 0 ]]; then if [[ $FAKE_RC -eq 0 ]]; then
fail "--strict passed with a skipped suite — the gate reports green having verified less than it ran: $FAKE_OUT" fail "--strict passed with a skipped suite — the gate reports green having verified less than it ran: $FAKE_OUT"
elif ! echo "$FAKE_OUT" | grep -q "a skip is a SETUP ERROR"; then elif ! grep -q "a skip is a SETUP ERROR" <<< "$FAKE_OUT"; then
fail "--strict failed but never said a skip is a setup error: $FAKE_OUT" fail "--strict failed but never said a skip is a setup error: $FAKE_OUT"
elif ! echo "$FAKE_OUT" | grep -q "test-needs-a-binary.sh"; then elif ! grep -q "test-needs-a-binary.sh" <<< "$FAKE_OUT"; then
fail "--strict failed without naming the skipped suite: $FAKE_OUT" fail "--strict failed without naming the skipped suite: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^ SKIP: frobnicator is not installed"; then elif grep -q "^ SKIP: frobnicator is not installed" <<< "$FAKE_OUT"; then
pass "--strict fails on a skip, names the suite, and carries through the reason it printed" pass "--strict fails on a skip, names the suite, and carries through the reason it printed"
else else
fail "--strict named the suite but swallowed its skip reason: $FAKE_OUT" fail "--strict named the suite but swallowed its skip reason: $FAKE_OUT"
@@ -445,7 +445,7 @@ STRICT_ENV_OUT="$(TMPDIR="$STRICT_ENV_PRIV" TEST_DIR="$DIR10/cases" RUN_TESTS_ST
bash "$DIR10/tests/run-tests.sh" 2>&1)" || STRICT_ENV_RC=$? bash "$DIR10/tests/run-tests.sh" 2>&1)" || STRICT_ENV_RC=$?
if [[ $STRICT_ENV_RC -eq 0 ]]; then if [[ $STRICT_ENV_RC -eq 0 ]]; then
fail "RUN_TESTS_STRICT=1 passed with a skipped suite: $STRICT_ENV_OUT" fail "RUN_TESTS_STRICT=1 passed with a skipped suite: $STRICT_ENV_OUT"
elif echo "$STRICT_ENV_OUT" | grep -q "a skip is a SETUP ERROR"; then elif grep -q "a skip is a SETUP ERROR" <<< "$STRICT_ENV_OUT"; then
pass "RUN_TESTS_STRICT=1 is the same gate as --strict" pass "RUN_TESTS_STRICT=1 is the same gate as --strict"
else else
fail "RUN_TESTS_STRICT=1 failed for some other reason: $STRICT_ENV_OUT" fail "RUN_TESTS_STRICT=1 failed for some other reason: $STRICT_ENV_OUT"
@@ -460,9 +460,9 @@ echo "--- the same skipped suite passes, still SKIPPED, without strict ---"
run_fake "$DIR10" run_fake "$DIR10"
if [[ $FAKE_RC -ne 0 ]]; then if [[ $FAKE_RC -ne 0 ]]; then
fail "a skipped suite failed a non-strict run — graceful skipping is gone: $FAKE_OUT" fail "a skipped suite failed a non-strict run — graceful skipping is gone: $FAKE_OUT"
elif ! echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 1 skipped, 0 failed ===$"; then elif ! grep -q "^=== Summary: 1 passed, 1 skipped, 0 failed ===$" <<< "$FAKE_OUT"; then
fail "a non-strict run miscounted the skip: $FAKE_OUT" fail "a non-strict run miscounted the skip: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^ SKIP: frobnicator is not installed"; then elif grep -q "^ SKIP: frobnicator is not installed" <<< "$FAKE_OUT"; then
pass "without strict the suite is SKIPPED, the run passes, and the reason is still reported" pass "without strict the suite is SKIPPED, the run passes, and the reason is still reported"
else else
fail "a non-strict run passed but dropped the skip reason: $FAKE_OUT" fail "a non-strict run passed but dropped the skip reason: $FAKE_OUT"
@@ -484,7 +484,7 @@ EOF
run_fake "$DIR10D" --strict run_fake "$DIR10D" --strict
if [[ $FAKE_RC -ne 0 ]]; then if [[ $FAKE_RC -ne 0 ]]; then
fail "--strict failed a run with nothing skipped — it fails unconditionally: $FAKE_OUT" fail "--strict failed a run with nothing skipped — it fails unconditionally: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$"; then elif grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$" <<< "$FAKE_OUT"; then
pass "--strict leaves a run with no skips green" pass "--strict leaves a run with no skips green"
else else
fail "--strict passed with the wrong summary: $FAKE_OUT" fail "--strict passed with the wrong summary: $FAKE_OUT"
@@ -506,7 +506,7 @@ EOF
run_fake "$DIR10E" --strickt run_fake "$DIR10E" --strickt
if [[ $FAKE_RC -eq 0 ]]; then if [[ $FAKE_RC -eq 0 ]]; then
fail "a misspelled flag was ignored and the run passed — a typo silently disarms the gate: $FAKE_OUT" fail "a misspelled flag was ignored and the run passed — a typo silently disarms the gate: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "Usage: .*--bats-only.*--strict"; then elif grep -q "Usage: .*--bats-only.*--strict" <<< "$FAKE_OUT"; then
pass "an unrecognised flag fails the run with usage" pass "an unrecognised flag fails the run with usage"
else else
fail "an unrecognised flag failed but not with usage: $FAKE_OUT" fail "an unrecognised flag failed but not with usage: $FAKE_OUT"
@@ -532,7 +532,7 @@ EOF
run_fake "$DIR10F" --strict run_fake "$DIR10F" --strict
if [[ $FAKE_RC -eq 0 ]]; then if [[ $FAKE_RC -eq 0 ]]; then
fail "--strict passed on a suite that skipped via stderr: $FAKE_OUT" fail "--strict passed on a suite that skipped via stderr: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^ widgetizer not installed -- skipping"; then elif grep -q "^ widgetizer not installed -- skipping" <<< "$FAKE_OUT"; then
pass "a skip reason printed to stderr without a SKIP: prefix is still carried into the failure" pass "a skip reason printed to stderr without a SKIP: prefix is still carried into the failure"
else else
fail "--strict failed but lost the stderr skip reason: $FAKE_OUT" fail "--strict failed but lost the stderr skip reason: $FAKE_OUT"
@@ -570,9 +570,9 @@ run_fake "$DIR10"
unset RUN_TESTS_STRICT unset RUN_TESTS_STRICT
if [[ $FAKE_RC -ne 0 ]]; then if [[ $FAKE_RC -ne 0 ]]; then
fail "an inherited RUN_TESTS_STRICT=1 turned a non-strict fixture strict — this suite's own result depends on how it was launched: $FAKE_OUT" fail "an inherited RUN_TESTS_STRICT=1 turned a non-strict fixture strict — this suite's own result depends on how it was launched: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "a skip is a SETUP ERROR"; then elif grep -q "a skip is a SETUP ERROR" <<< "$FAKE_OUT"; then
fail "the fixture ran strict despite not asking for it: $FAKE_OUT" fail "the fixture ran strict despite not asking for it: $FAKE_OUT"
elif echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 1 skipped, 0 failed ===$"; then elif grep -q "^=== Summary: 1 passed, 1 skipped, 0 failed ===$" <<< "$FAKE_OUT"; then
pass "an ambient RUN_TESTS_STRICT=1 is scrubbed from fixtures that did not ask for strict" pass "an ambient RUN_TESTS_STRICT=1 is scrubbed from fixtures that did not ask for strict"
else else
fail "the scrubbed run produced the wrong summary: $FAKE_OUT" fail "the scrubbed run produced the wrong summary: $FAKE_OUT"

View File

@@ -22,32 +22,32 @@ echo "--- assembly ---"
out=$(run "$FULL") out=$(run "$FULL")
echo "$out" | grep -q "myproject" && pass "shows basename of working directory" || fail "dir missing — got: $out" grep -q "myproject" <<< "$out" && pass "shows basename of working directory" || fail "dir missing — got: $out"
echo "$out" | grep -q " · " && pass "segments joined with ' · '" || fail "separator missing — got: $out" grep -q " · " <<< "$out" && pass "segments joined with ' · '" || fail "separator missing — got: $out"
echo "$out" | grep -q "Sonnet 4.6" && pass "shows model name" || fail "model name missing — got: $out" grep -q "Sonnet 4.6" <<< "$out" && pass "shows model name" || fail "model name missing — got: $out"
echo "$out" | grep -q "context 20%" && pass "shows context % with 'context' label" || fail "context label wrong — got: $out" grep -q "context 20%" <<< "$out" && pass "shows context % with 'context' label" || fail "context label wrong — got: $out"
echo "" echo ""
echo "--- token formatting ---" echo "--- token formatting ---"
# 15k + 5k = 20k → shown as "20k tokens" # 15k + 5k = 20k → shown as "20k tokens"
echo "$out" | grep -q "20k tokens" && pass "formats total tokens as Xk when >= 1000" || fail "Xk format wrong — got: $out" grep -q "20k tokens" <<< "$out" && pass "formats total tokens as Xk when >= 1000" || fail "Xk format wrong — got: $out"
# Sub-1000 total: 500 + 300 = 800 → shown as "800 tokens" # Sub-1000 total: 500 + 300 = 800 → shown as "800 tokens"
LOW='{"workspace":{"current_dir":"/x"},"context_window":{"total_input_tokens":500,"total_output_tokens":300}}' LOW='{"workspace":{"current_dir":"/x"},"context_window":{"total_input_tokens":500,"total_output_tokens":300}}'
out_low=$(run "$LOW") out_low=$(run "$LOW")
echo "$out_low" | grep -q "800 tokens" && pass "shows raw count when total tokens < 1000" || fail "sub-1000 format wrong — got: $out_low" grep -q "800 tokens" <<< "$out_low" && pass "shows raw count when total tokens < 1000" || fail "sub-1000 format wrong — got: $out_low"
echo "" echo ""
echo "--- cost formatting ---" echo "--- cost formatting ---"
# 0.25 → 25¢ # 0.25 → 25¢
echo "$out" | grep -q "25¢" && pass "formats sub-dollar cost as cents (¢)" || fail "cents format wrong — got: $out" grep -q "25¢" <<< "$out" && pass "formats sub-dollar cost as cents (¢)" || fail "cents format wrong — got: $out"
# 1.71 → $1.71 # 1.71 → $1.71
DOLLAR='{"workspace":{"current_dir":"/x"},"cost":{"total_cost_usd":1.71},"context_window":{"total_input_tokens":0,"total_output_tokens":0}}' DOLLAR='{"workspace":{"current_dir":"/x"},"cost":{"total_cost_usd":1.71},"context_window":{"total_input_tokens":0,"total_output_tokens":0}}'
out_dollar=$(run "$DOLLAR") out_dollar=$(run "$DOLLAR")
echo "$out_dollar" | grep -qF '$1.71' && pass "formats dollar-plus cost as \$X.XX" || fail "dollar format wrong — got: $out_dollar" grep -qF '$1.71' <<< "$out_dollar" && pass "formats dollar-plus cost as \$X.XX" || fail "dollar format wrong — got: $out_dollar"
echo "" echo ""
echo "--- missing fields omitted ---" echo "--- missing fields omitted ---"
@@ -55,12 +55,12 @@ echo "--- missing fields omitted ---"
# No cost field → no ¢ or $ in output # No cost field → no ¢ or $ in output
NO_COST='{"workspace":{"current_dir":"/x"},"context_window":{"total_input_tokens":0,"total_output_tokens":0}}' NO_COST='{"workspace":{"current_dir":"/x"},"context_window":{"total_input_tokens":0,"total_output_tokens":0}}'
out_nocost=$(run "$NO_COST") out_nocost=$(run "$NO_COST")
! echo "$out_nocost" | grep -qE '[¢$]' && pass "omits cost segment when cost absent" || fail "cost shown unexpectedly — got: $out_nocost" ! grep -qE '[¢$]' <<< "$out_nocost" && pass "omits cost segment when cost absent" || fail "cost shown unexpectedly — got: $out_nocost"
# Zero tokens → no token segment # Zero tokens → no token segment
ZERO_TOK='{"workspace":{"current_dir":"/x"},"context_window":{"total_input_tokens":0,"total_output_tokens":0}}' ZERO_TOK='{"workspace":{"current_dir":"/x"},"context_window":{"total_input_tokens":0,"total_output_tokens":0}}'
out_zerotok=$(run "$ZERO_TOK") out_zerotok=$(run "$ZERO_TOK")
! echo "$out_zerotok" | grep -q "tokens" && pass "omits token segment when total is zero" || fail "token segment shown unexpectedly — got: $out_zerotok" ! grep -q "tokens" <<< "$out_zerotok" && pass "omits token segment when total is zero" || fail "token segment shown unexpectedly — got: $out_zerotok"
echo "" echo ""
echo "--- executable bit ---" echo "--- executable bit ---"

View File

@@ -246,12 +246,12 @@ EOF
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5"' EXIT trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5"' EXIT
OUT5=$(run_wrap "$FIXTURE5" --config "$VALE_CONFIG" \ OUT5=$(run_wrap "$FIXTURE5" --config "$VALE_CONFIG" \
plugins/testplugin/skills/zzzskill/SKILL.md) plugins/testplugin/skills/zzzskill/SKILL.md)
if echo "$OUT5" | grep -q "VagueWording"; then if grep -q "VagueWording" <<< "$OUT5"; then
pass "flags vague wording when the description contains an apostrophe" pass "flags vague wording when the description contains an apostrophe"
else else
fail "silently missed vague wording in a description containing an apostrophe" fail "silently missed vague wording in a description containing an apostrophe"
fi fi
if echo "$OUT5" | grep -qi "yaml:"; then if grep -qi "yaml:" <<< "$OUT5"; then
fail "flattened copy with an apostrophe produced a YAML parse error" fail "flattened copy with an apostrophe produced a YAML parse error"
else else
pass "flattened copy with an apostrophe is valid YAML (no parse error)" pass "flattened copy with an apostrophe is valid YAML (no parse error)"
@@ -297,9 +297,9 @@ EOF
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7"' EXIT trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7"' EXIT
OUT7=$(run_wrap "$FIXTURE7" --config "$VALE_CONFIG" \ OUT7=$(run_wrap "$FIXTURE7" --config "$VALE_CONFIG" \
plugins/testplugin/skills/zzzskill/SKILL.md) plugins/testplugin/skills/zzzskill/SKILL.md)
if echo "$OUT7" | grep -q "Traceback"; then if grep -q "Traceback" <<< "$OUT7"; then
fail "crashed while flattening a description with a blank line between paragraphs" fail "crashed while flattening a description with a blank line between paragraphs"
elif echo "$OUT7" | grep -q "VagueWording"; then elif grep -q "VagueWording" <<< "$OUT7"; then
pass "still flags vague wording in the second paragraph after a blank line" pass "still flags vague wording in the second paragraph after a blank line"
else else
fail "silently missed vague wording in the second paragraph after a blank line — the bug this test guards against" fail "silently missed vague wording in the second paragraph after a blank line — the bug this test guards against"
@@ -337,7 +337,7 @@ REL_CFG8="../../../../.vale.ini"
REL_FILE8="SKILL.md" REL_FILE8="SKILL.md"
OUT_EQ=$(run_wrap "$SUBDIR8" "--config=$REL_CFG8" "$REL_FILE8") OUT_EQ=$(run_wrap "$SUBDIR8" "--config=$REL_CFG8" "$REL_FILE8")
OUT_TWO=$(run_wrap "$SUBDIR8" --config "$REL_CFG8" "$REL_FILE8") OUT_TWO=$(run_wrap "$SUBDIR8" --config "$REL_CFG8" "$REL_FILE8")
if echo "$OUT_EQ" | grep -q "VagueWording" && [[ "$OUT_EQ" == "$OUT_TWO" ]]; then if grep -q "VagueWording" <<< "$OUT_EQ" && [[ "$OUT_EQ" == "$OUT_TWO" ]]; then
pass "cwd-relative --config resolves from a subdirectory in both argv forms" pass "cwd-relative --config resolves from a subdirectory in both argv forms"
else else
fail "cwd-relative --config did not resolve from a subdirectory (equals form vs two-argv form)" fail "cwd-relative --config did not resolve from a subdirectory (equals form vs two-argv form)"
@@ -355,7 +355,7 @@ RC_REL_CFG=$?
OUT_ABS_CFG=$(cd "$SUBDIR8" && bash "$SCRIPT" --config "$FIXTURE8/.vale.ini" "$REL_FILE8" 2>&1) OUT_ABS_CFG=$(cd "$SUBDIR8" && bash "$SCRIPT" --config "$FIXTURE8/.vale.ini" "$REL_FILE8" 2>&1)
RC_ABS_CFG=$? RC_ABS_CFG=$?
set -e set -e
if echo "$OUT_REL_CFG" | grep -qi "does not exist"; then if grep -qi "does not exist" <<< "$OUT_REL_CFG"; then
fail "cwd-relative --config hard-errored ('does not exist') — the bug this test guards against" fail "cwd-relative --config hard-errored ('does not exist') — the bug this test guards against"
elif [[ "$OUT_REL_CFG" == "$OUT_ABS_CFG" && "$RC_REL_CFG" -eq "$RC_ABS_CFG" ]]; then elif [[ "$OUT_REL_CFG" == "$OUT_ABS_CFG" && "$RC_REL_CFG" -eq "$RC_ABS_CFG" ]]; then
pass "cwd-relative --config matches the absolute-path invocation (output and exit code)" pass "cwd-relative --config matches the absolute-path invocation (output and exit code)"
@@ -373,9 +373,9 @@ echo ""
echo "--- flattens a cwd-relative file argument passed from a subdirectory ---" echo "--- flattens a cwd-relative file argument passed from a subdirectory ---"
WRAPPED_REL=$(run_wrap "$SUBDIR8" --config "$FIXTURE8/.vale.ini" "$REL_FILE8") WRAPPED_REL=$(run_wrap "$SUBDIR8" --config "$FIXTURE8/.vale.ini" "$REL_FILE8")
BARE_REL=$(cd "$SUBDIR8" && vale --config "$FIXTURE8/.vale.ini" "$REL_FILE8" 2>&1 || true) BARE_REL=$(cd "$SUBDIR8" && vale --config "$FIXTURE8/.vale.ini" "$REL_FILE8" 2>&1 || true)
if ! echo "$WRAPPED_REL" | grep -q "VagueWording"; then if ! grep -q "VagueWording" <<< "$WRAPPED_REL"; then
fail "cwd-relative file argument produced no alert — flattening was silently skipped, the bug this test guards against" fail "cwd-relative file argument produced no alert — flattening was silently skipped, the bug this test guards against"
elif echo "$BARE_REL" | grep -q "VagueWording"; then elif grep -q "VagueWording" <<< "$BARE_REL"; then
fail "bare vale already flags this fixture, so the test can't detect a silently-skipped flattening" fail "bare vale already flags this fixture, so the test can't detect a silently-skipped flattening"
else else
pass "cwd-relative file argument is flattened and flagged where bare vale reports nothing" pass "cwd-relative file argument is flattened and flagged where bare vale reports nothing"
@@ -443,7 +443,7 @@ trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTU
REL11="plugins/testplugin/skills/zzzskill/SKILL.md" REL11="plugins/testplugin/skills/zzzskill/SKILL.md"
WRAPPED_OUT=$(run_wrap "$FIXTURE11" --config "$VALE_CONFIG" "$REL11") WRAPPED_OUT=$(run_wrap "$FIXTURE11" --config "$VALE_CONFIG" "$REL11")
BARE_OUT=$(cd "$FIXTURE11" && vale --config "$VALE_CONFIG" "$REL11" 2>&1 || true) BARE_OUT=$(cd "$FIXTURE11" && vale --config "$VALE_CONFIG" "$REL11" 2>&1 || true)
if ! echo "$BARE_OUT" | grep -q "VagueWording"; then if ! grep -q "VagueWording" <<< "$BARE_OUT"; then
fail "bare vale reports nothing for a literal (|) block scalar — the 'literal blocks are not broken' premise is wrong" fail "bare vale reports nothing for a literal (|) block scalar — the 'literal blocks are not broken' premise is wrong"
elif [[ "$WRAPPED_OUT" == "$BARE_OUT" ]]; then elif [[ "$WRAPPED_OUT" == "$BARE_OUT" ]]; then
pass "literal (|) block scalar output matches bare vale exactly — untouched by flattening" pass "literal (|) block scalar output matches bare vale exactly — untouched by flattening"
@@ -461,7 +461,7 @@ echo "--- defaults --config to the wrapper's own sibling assets/vale/.vale.ini -
FIXTURE12="$(make_fixture 2)" FIXTURE12="$(make_fixture 2)"
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7" "$FIXTURE8" "$FIXTURE10" "$FIXTURE11" "$FIXTURE12"' EXIT trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7" "$FIXTURE8" "$FIXTURE10" "$FIXTURE11" "$FIXTURE12"' EXIT
OUT12=$(run_wrap "$FIXTURE12" plugins/testplugin/skills/zzzskill/SKILL.md) OUT12=$(run_wrap "$FIXTURE12" plugins/testplugin/skills/zzzskill/SKILL.md)
if echo "$OUT12" | grep -q "VagueWording"; then if grep -q "VagueWording" <<< "$OUT12"; then
pass "a --config-less invocation uses the wrapper's bundled config" pass "a --config-less invocation uses the wrapper's bundled config"
else else
fail "a --config-less invocation found no config — external pre-commit consumers get E100, the bug this test guards against" fail "a --config-less invocation found no config — external pre-commit consumers get E100, the bug this test guards against"
@@ -490,9 +490,9 @@ EOF
chmod +x "$STUB13/realpath" chmod +x "$STUB13/realpath"
OUT13=$(cd "$FIXTURE12" && PATH="$STUB13:$PATH" bash "$SCRIPT" --config "$VALE_CONFIG" \ OUT13=$(cd "$FIXTURE12" && PATH="$STUB13:$PATH" bash "$SCRIPT" --config "$VALE_CONFIG" \
plugins/testplugin/skills/zzzskill/SKILL.md 2>&1 || true) plugins/testplugin/skills/zzzskill/SKILL.md 2>&1 || true)
if echo "$OUT13" | grep -q "illegal option"; then if grep -q "illegal option" <<< "$OUT13"; then
fail "invoked realpath -m — fails on macOS's BSD realpath, the bug this test guards against" fail "invoked realpath -m — fails on macOS's BSD realpath, the bug this test guards against"
elif echo "$OUT13" | grep -q "VagueWording"; then elif grep -q "VagueWording" <<< "$OUT13"; then
pass "flattens and flags with no GNU realpath available" pass "flattens and flags with no GNU realpath available"
else else
fail "produced no alert under a BSD-style realpath: $OUT13" fail "produced no alert under a BSD-style realpath: $OUT13"
@@ -509,9 +509,9 @@ FIXTURE14="$(make_fixture 2)"
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7" "$FIXTURE8" "$FIXTURE10" "$FIXTURE11" "$FIXTURE12" "$STUB13" "$FIXTURE14"' EXIT trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7" "$FIXTURE8" "$FIXTURE10" "$FIXTURE11" "$FIXTURE12" "$STUB13" "$FIXTURE14"' EXIT
WRAPPED_DIR=$(run_wrap "$FIXTURE14" --config "$VALE_CONFIG" plugins) WRAPPED_DIR=$(run_wrap "$FIXTURE14" --config "$VALE_CONFIG" plugins)
BARE_DIR=$(cd "$FIXTURE14" && vale --config "$VALE_CONFIG" plugins 2>&1 || true) BARE_DIR=$(cd "$FIXTURE14" && vale --config "$VALE_CONFIG" plugins 2>&1 || true)
if ! echo "$WRAPPED_DIR" | grep -q "VagueWording"; then if ! grep -q "VagueWording" <<< "$WRAPPED_DIR"; then
fail "a directory argument produced no alert — flattening was silently skipped, the bug this test guards against" fail "a directory argument produced no alert — flattening was silently skipped, the bug this test guards against"
elif echo "$BARE_DIR" | grep -q "VagueWording"; then elif grep -q "VagueWording" <<< "$BARE_DIR"; then
fail "bare vale already flags this fixture, so the test can't detect a silently-skipped flattening" fail "bare vale already flags this fixture, so the test can't detect a silently-skipped flattening"
else else
pass "a directory argument is walked and its files flattened" pass "a directory argument is walked and its files flattened"
@@ -525,7 +525,7 @@ mkdir -p "$SPACED15"
cp "$FIXTURE14/plugins/testplugin/skills/zzzskill/SKILL.md" "$SPACED15/SKILL.md" cp "$FIXTURE14/plugins/testplugin/skills/zzzskill/SKILL.md" "$SPACED15/SKILL.md"
rm -rf "$FIXTURE14/plugins/testplugin/skills/zzzskill" rm -rf "$FIXTURE14/plugins/testplugin/skills/zzzskill"
OUT15=$(run_wrap "$FIXTURE14" --config "$VALE_CONFIG" plugins/testplugin/skills) OUT15=$(run_wrap "$FIXTURE14" --config "$VALE_CONFIG" plugins/testplugin/skills)
if echo "$OUT15" | grep -q "zzz skill" && echo "$OUT15" | grep -q "VagueWording"; then if grep -q "zzz skill" <<< "$OUT15" && grep -q "VagueWording" <<< "$OUT15"; then
pass "a file under a directory whose name contains a space is walked and flattened" pass "a file under a directory whose name contains a space is walked and flattened"
else else
fail "a path with a space was dropped from the directory walk" fail "a path with a space was dropped from the directory walk"
@@ -626,7 +626,7 @@ unguarded_expansions() {
done < <(sourced_files "$file") done < <(sourced_files "$file")
while IFS= read -r hit; do while IFS= read -r hit; do
name="$(printf '%s\n' "$hit" \ name="$(printf '%s\n' "$hit" \
| grep -oE '\$\{[A-Za-z_][A-Za-z0-9_]*\[@\]\}' | head -1 \ | { grep -oE '\$\{[A-Za-z_][A-Za-z0-9_]*\[@\]\}' || true; } | head -1 \
| sed -E 's/^\$\{//; s/\[@\]\}$//')" | sed -E 's/^\$\{//; s/\[@\]\}$//')"
# Shell-maintained arrays are never seeded by a `NAME=(...)` line, so the # Shell-maintained arrays are never seeded by a `NAME=(...)` line, so the
# seeding exemption below can never clear them: without this case they are # seeding exemption below can never clear them: without this case they are
@@ -653,8 +653,7 @@ unguarded_expansions() {
PIPESTATUS|BASH_SOURCE|BASH_LINENO|BASH_VERSINFO|GROUPS|DIRSTACK) continue ;; PIPESTATUS|BASH_SOURCE|BASH_LINENO|BASH_VERSINFO|GROUPS|DIRSTACK) continue ;;
esac esac
# Same-line emptiness short-circuit: the expansion cannot be reached empty. # Same-line emptiness short-circuit: the expansion cannot be reached empty.
if printf '%s\n' "$hit" \ if grep -qE "\\\$\{#$name\[@\]\}[[:space:]]*-(eq|lt)[[:space:]]*[01][^|]*\|\|" <<< "$hit"; then
| grep -qE "\\\$\{#$name\[@\]\}[[:space:]]*-(eq|lt)[[:space:]]*[01][^|]*\|\|"; then
continue continue
fi fi
for seed_file in ${seed_files[@]+"${seed_files[@]}"}; do for seed_file in ${seed_files[@]+"${seed_files[@]}"}; do
@@ -809,7 +808,7 @@ for ARGS17 in "" "--config $VALE_CONFIG" "--config $VALE_CONFIG emptydir"; do
# shellcheck disable=SC2086 # deliberate word splitting of the argv fixture # shellcheck disable=SC2086 # deliberate word splitting of the argv fixture
OUT17+="$( (cd "$FIXTURE17" && "$OLD_BASH" "$SCRIPT" $ARGS17 </dev/null 2>&1) || true)" OUT17+="$( (cd "$FIXTURE17" && "$OLD_BASH" "$SCRIPT" $ARGS17 </dev/null 2>&1) || true)"
done done
if echo "$OUT17" | grep -q "unbound variable"; then if grep -q "unbound variable" <<< "$OUT17"; then
fail "aborted with 'unbound variable' on bash $OLD_BASH_VER — the bug this test guards against" fail "aborted with 'unbound variable' on bash $OLD_BASH_VER — the bug this test guards against"
else else
pass "degenerate invocations run clean under bash $OLD_BASH_VER ($OLD_BASH)" pass "degenerate invocations run clean under bash $OLD_BASH_VER ($OLD_BASH)"
@@ -828,7 +827,7 @@ SPACED18="$FIXTURE18/plugins/testplugin/skills/zzz skill dir"
mkdir -p "$SPACED18" mkdir -p "$SPACED18"
mv "$FIXTURE18/plugins/testplugin/skills/zzzskill/SKILL.md" "$SPACED18/SKILL.md" mv "$FIXTURE18/plugins/testplugin/skills/zzzskill/SKILL.md" "$SPACED18/SKILL.md"
OUT18=$(run_wrap "$FIXTURE18" --config "$VALE_CONFIG" "plugins/testplugin/skills/zzz skill dir/SKILL.md") OUT18=$(run_wrap "$FIXTURE18" --config "$VALE_CONFIG" "plugins/testplugin/skills/zzz skill dir/SKILL.md")
if echo "$OUT18" | grep -q "zzz skill dir/SKILL.md" && echo "$OUT18" | grep -q "VagueWording"; then if grep -q "zzz skill dir/SKILL.md" <<< "$OUT18" && grep -q "VagueWording" <<< "$OUT18"; then
pass "a path argument with a space is passed to vale as one word" pass "a path argument with a space is passed to vale as one word"
else else
fail "a path argument with a space was split by the array expansion: $OUT18" fail "a path argument with a space was split by the array expansion: $OUT18"
@@ -934,7 +933,7 @@ else
new_fixture "$DIR19" new_fixture "$DIR19"
BARE19="$(cd "$DIR19" && vale --config "$VALE_CONFIG" "$REL_SKILL19" 2>&1 || true)" BARE19="$(cd "$DIR19" && vale --config "$VALE_CONFIG" "$REL_SKILL19" 2>&1 || true)"
GOT19="$(alert_text "$(run_wrap "$DIR19" --config "$VALE_CONFIG" "$REL_SKILL19")")" GOT19="$(alert_text "$(run_wrap "$DIR19" --config "$VALE_CONFIG" "$REL_SKILL19")")"
if echo "$BARE19" | grep -q "VagueWording"; then if grep -q "VagueWording" <<< "$BARE19"; then
fail "bare vale already flags the $FORM19 form, so this case can't detect a silently-skipped flattening" fail "bare vale already flags the $FORM19 form, so this case can't detect a silently-skipped flattening"
elif [[ "$GOT19" == "$BASELINE19" ]]; then elif [[ "$GOT19" == "$BASELINE19" ]]; then
pass "a $FORM19 multi-line description reports the same alerts as its single-line form" pass "a $FORM19 multi-line description reports the same alerts as its single-line form"
@@ -1044,7 +1043,7 @@ done
echo "" echo ""
echo "--- the |- literal-block fallback lints normally and preserves line numbers ---" echo "--- the |- literal-block fallback lints normally and preserves line numbers ---"
OUT20B=$(run_wrap "$FIXTURE20_BLOCK" --config "$VALE_CONFIG" "$REL_SKILL19") OUT20B=$(run_wrap "$FIXTURE20_BLOCK" --config "$VALE_CONFIG" "$REL_SKILL19")
if echo "$OUT20B" | grep -q "VagueWording"; then if grep -q "VagueWording" <<< "$OUT20B"; then
pass "a description needing quotes with both an apostrophe and a double quote is still linted" pass "a description needing quotes with both an apostrophe and a double quote is still linted"
else else
fail "a description needing quotes with both an apostrophe and a double quote produced no alerts" fail "a description needing quotes with both an apostrophe and a double quote produced no alerts"
@@ -1053,7 +1052,7 @@ WANT20B_LINE="$(grep -n 'flattening marker phrase' "$FIXTURE20_BLOCK/$REL_SKILL1
# `--output line` prints `file:line:col:Rule:message`, so the line number reads # `--output line` prints `file:line:col:Rule:message`, so the line number reads
# back without any wrapping or colour to strip. # back without any wrapping or colour to strip.
GOT20B_LINE="$(run_wrap "$FIXTURE20_BLOCK" --config "$APOS_STYLE/.vale.ini" --output line "$REL_SKILL19" \ GOT20B_LINE="$(run_wrap "$FIXTURE20_BLOCK" --config "$APOS_STYLE/.vale.ini" --output line "$REL_SKILL19" \
| grep 'Apostrophe.Body' | head -1 | cut -d: -f2)" | { grep 'Apostrophe.Body' || true; } | head -1 | cut -d: -f2)"
if [[ "$GOT20B_LINE" == "$WANT20B_LINE" ]]; then if [[ "$GOT20B_LINE" == "$WANT20B_LINE" ]]; then
pass "a body line after a |- flattened description keeps its original line number ($WANT20B_LINE)" pass "a body line after a |- flattened description keeps its original line number ($WANT20B_LINE)"
else else
@@ -1079,7 +1078,10 @@ if [[ "$BARE21_FILES" != "in 1 file" ]]; then
fail "bare vale did not lint the symlinked file ($BARE21_FILES), so this case can't detect the walk dropping it" fail "bare vale did not lint the symlinked file ($BARE21_FILES), so this case can't detect the walk dropping it"
elif [[ "$WRAPPED21_FILES" != "$BARE21_FILES" ]]; then elif [[ "$WRAPPED21_FILES" != "$BARE21_FILES" ]]; then
fail "the directory walk dropped a symlinked file: wrapper saw '$WRAPPED21_FILES', bare vale '$BARE21_FILES'" fail "the directory walk dropped a symlinked file: wrapper saw '$WRAPPED21_FILES', bare vale '$BARE21_FILES'"
elif echo "$WRAPPED21" | grep -q "VagueWording"; then # A here-string, not `echo "$WRAPPED21" | grep -q`: the match sits on line 3 of
# 8, and under pipefail grep -q exiting early can SIGPIPE echo mid-write and
# fail this branch on correct output (see tests/test-check-release-needed.sh).
elif grep -q "VagueWording" <<< "$WRAPPED21"; then
pass "a symlinked file under a directory argument is mirrored, flattened and flagged" pass "a symlinked file under a directory argument is mirrored, flattened and flagged"
else else
fail "a symlinked file was mirrored but not flattened — no alert came back" fail "a symlinked file was mirrored but not flattened — no alert came back"
@@ -1117,9 +1119,9 @@ RC23=$?
set -e set -e
if [[ $RC23 -eq 0 ]]; then if [[ $RC23 -eq 0 ]]; then
fail "a typo'd path exited 0 — indistinguishable from a clean run, the bug this test guards against" fail "a typo'd path exited 0 — indistinguishable from a clean run, the bug this test guards against"
elif echo "$OUT23" | grep -q "in stdin"; then elif grep -q "in stdin" <<< "$OUT23"; then
fail "a typo'd path fell back to reading stdin and reported 'in stdin' instead of erroring" fail "a typo'd path fell back to reading stdin and reported 'in stdin' instead of erroring"
elif echo "$OUT23" | grep -q "SKILLL.md"; then elif grep -q "SKILLL.md" <<< "$OUT23"; then
pass "a typo'd path exits nonzero with a message naming the path" pass "a typo'd path exits nonzero with a message naming the path"
else else
fail "a typo'd path exited $RC23 but the message does not name it: $OUT23" fail "a typo'd path exited $RC23 but the message does not name it: $OUT23"
@@ -1141,9 +1143,9 @@ mkdir -p "$FIXTURE24/line"
for FORM24 in "--output line" "--output=line" "--output JSON" "--output=JSON"; do for FORM24 in "--output line" "--output=line" "--output JSON" "--output=JSON"; do
# shellcheck disable=SC2086 # deliberate word splitting of the argv fixture # shellcheck disable=SC2086 # deliberate word splitting of the argv fixture
OUT24="$(run_wrap "$FIXTURE24" --config "$VALE_CONFIG" $FORM24 "$REL_SKILL19")" OUT24="$(run_wrap "$FIXTURE24" --config "$VALE_CONFIG" $FORM24 "$REL_SKILL19")"
if echo "$OUT24" | grep -q "E100"; then if grep -q "E100" <<< "$OUT24"; then
fail "'$FORM24' was rewritten to a cwd path and vale flipped into template mode — the bug this test guards against" fail "'$FORM24' was rewritten to a cwd path and vale flipped into template mode — the bug this test guards against"
elif echo "$OUT24" | grep -q "VagueWording"; then elif grep -q "VagueWording" <<< "$OUT24"; then
pass "'$FORM24' is passed through as a built-in style name" pass "'$FORM24' is passed through as a built-in style name"
else else
fail "'$FORM24' produced no alert: $OUT24" fail "'$FORM24' produced no alert: $OUT24"
@@ -1215,12 +1217,12 @@ FOUND26="$(unguarded_expansions "$FIXTURE26/probe.sh")"
MISSING26="" MISSING26=""
LEAKED26="" LEAKED26=""
for ARR26 in $EXEMPT26; do for ARR26 in $EXEMPT26; do
if echo "$FOUND26" | grep -q "{$ARR26\[@\]}"; then if grep -q "{$ARR26\[@\]}" <<< "$FOUND26"; then
LEAKED26+="$ARR26 " LEAKED26+="$ARR26 "
fi fi
done done
for ARR26 in $FLAGGED26; do for ARR26 in $FLAGGED26; do
if ! echo "$FOUND26" | grep -q "{$ARR26\[@\]}"; then if ! grep -q "{$ARR26\[@\]}" <<< "$FOUND26"; then
MISSING26+="$ARR26 " MISSING26+="$ARR26 "
fi fi
done done
@@ -1465,7 +1467,7 @@ matches_any_regex28() {
[[ -n "$regexes" ]] || return 1 [[ -n "$regexes" ]] || return 1
while IFS= read -r re; do while IFS= read -r re; do
[[ -n "$re" ]] || continue [[ -n "$re" ]] || continue
if printf '%s\n' "$rel" | grep -Eq "$re"; then if grep -Eq "$re" <<< "$rel"; then
return 0 return 0
fi fi
done <<EOF_RE28 done <<EOF_RE28
@@ -1499,7 +1501,7 @@ elif [[ -z "$SECTIONS28" ]]; then
else else
while IFS= read -r SEC28; do while IFS= read -r SEC28; do
[[ -n "$SEC28" ]] || continue [[ -n "$SEC28" ]] || continue
if ! printf '%s\n' "$PROBE_TABLE28" | grep -qF "|$SEC28|isolating"; then if ! grep -qF "|$SEC28|isolating" <<< "$PROBE_TABLE28"; then
UNPROBED28+="$SEC28 " UNPROBED28+="$SEC28 "
fi fi
done <<EOF_SEC28 done <<EOF_SEC28
@@ -1508,7 +1510,7 @@ EOF_SEC28
STALE_SECTIONS28="" STALE_SECTIONS28=""
while IFS= read -r TSEC28; do while IFS= read -r TSEC28; do
[[ -n "$TSEC28" ]] || continue [[ -n "$TSEC28" ]] || continue
printf '%s\n' "$SECTIONS28" | grep -qF -- "$TSEC28" || STALE_SECTIONS28+="$TSEC28 " grep -qF -- "$TSEC28" <<< "$SECTIONS28" || STALE_SECTIONS28+="$TSEC28 "
done <<EOF_TSEC28 done <<EOF_TSEC28
$TABLE_SECTIONS28 $TABLE_SECTIONS28
EOF_TSEC28 EOF_TSEC28
@@ -1548,7 +1550,7 @@ probe_coverage28() {
echo "FAIL|$rel|vale printed no summary line for $rel, so it is not known whether anything was scanned: ${report:-<empty>}" echo "FAIL|$rel|vale printed no summary line for $rel, so it is not known whether anything was scanned: ${report:-<empty>}"
elif [[ "$count" -eq 0 ]]; then elif [[ "$count" -eq 0 ]]; then
echo "FAIL|$rel|$sec scanned 0 files for $rel — the section's glob covers no path of that shape, so vale exits 0 and every gate reads it as a pass" echo "FAIL|$rel|$sec scanned 0 files for $rel — the section's glob covers no path of that shape, so vale exits 0 and every gate reads it as a pass"
elif ! printf '%s\n' "$report" | grep -qF "Kyberforge.VagueWording"; then elif ! grep -qF "Kyberforge.VagueWording" <<< "$report"; then
echo "FAIL|$rel|$sec scanned $rel but raised no Kyberforge alert — the glob matches but the style is not loaded, which lints the file and reports nothing" echo "FAIL|$rel|$sec scanned $rel but raised no Kyberforge alert — the glob matches but the style is not loaded, which lints the file and reports nothing"
else else
echo "PASS|$rel|$sec scans $rel ($count file) and raises a Kyberforge alert" echo "PASS|$rel|$sec scans $rel ($count file) and raises a Kyberforge alert"
@@ -1622,7 +1624,7 @@ if [[ "$VALE_READY" == true && "$PART_A_CLEAN28" == true ]]; then
[[ -n "$REL28" && "$ISO28" == "isolating" ]] || continue [[ -n "$REL28" && "$ISO28" == "isolating" ]] || continue
LINE28="$(printf '%s\n' "$RESULTS28" | { grep -F "|$REL28|" || true; } | head -1)" LINE28="$(printf '%s\n' "$RESULTS28" | { grep -F "|$REL28|" || true; } | head -1)"
if [[ "$ROWSEC28" == "$MSEC28" ]]; then if [[ "$ROWSEC28" == "$MSEC28" ]]; then
printf '%s\n' "$LINE28" | grep -qF "but raised no Kyberforge alert" \ grep -qF "but raised no Kyberforge alert" <<< "$LINE28" \
|| STYLE_MUT_FAILS28+="[$MSEC28 lost Kyberforge but $REL28 did not fail as style-not-loaded: ${LINE28:-<no result>}] " || STYLE_MUT_FAILS28+="[$MSEC28 lost Kyberforge but $REL28 did not fail as style-not-loaded: ${LINE28:-<no result>}] "
else else
[[ "$LINE28" == PASS\|* ]] \ [[ "$LINE28" == PASS\|* ]] \
@@ -1719,7 +1721,7 @@ copilot_scope_failures30() {
[[ -n "$rel" ]] || continue [[ -n "$rel" ]] || continue
report="$(vale_report28 "$cfg" "$TREE28" "$rel")" report="$(vale_report28 "$cfg" "$TREE28" "$rel")"
has=false has=false
if printf '%s\n' "$report" | grep -qF "KyberforgeCopilot.ProactivePhrase"; then has=true; fi if grep -qF "KyberforgeCopilot.ProactivePhrase" <<< "$report"; then has=true; fi
case "$rel" in case "$rel" in
*.agent.md) *.agent.md)
[[ "$has" == true ]] || bad+="[$rel is an agent file but raised no ProactivePhrase alert — the Copilot style is shipped but never loaded for it, so its rules lint nothing] " [[ "$has" == true ]] || bad+="[$rel is an agent file but raised no ProactivePhrase alert — the Copilot style is shipped but never loaded for it, so its rules lint nothing] "
@@ -1728,7 +1730,7 @@ copilot_scope_failures30() {
# Non-vacuity guard: `absent` only means `scoped out` if the file was # Non-vacuity guard: `absent` only means `scoped out` if the file was
# scanned at all. Without the Kyberforge half, a glob that stopped # scanned at all. Without the Kyberforge half, a glob that stopped
# matching this path entirely would read as correct scoping. # matching this path entirely would read as correct scoping.
if ! printf '%s\n' "$report" | grep -qF "Kyberforge.VagueWording"; then if ! grep -qF "Kyberforge.VagueWording" <<< "$report"; then
bad+="[$rel raised no Kyberforge alert either, so its missing ProactivePhrase proves nothing about scoping] " bad+="[$rel raised no Kyberforge alert either, so its missing ProactivePhrase proves nothing about scoping] "
elif [[ "$has" == true ]]; then elif [[ "$has" == true ]]; then
bad+="[$rel is not an .agent.md file but raised a ProactivePhrase alert — the Copilot style has leaked past the scope ADR-0013 fixes it to] " bad+="[$rel is not an .agent.md file but raised a ProactivePhrase alert — the Copilot style has leaked past the scope ADR-0013 fixes it to] "
@@ -1789,9 +1791,9 @@ UNLOAD_FAILS30="$(copilot_scope_failures30 "$UNLOAD30/.vale.ini")"
LEAK_FAILS30="$(copilot_scope_failures30 "$LEAK30/.vale.ini")" LEAK_FAILS30="$(copilot_scope_failures30 "$LEAK30/.vale.ini")"
if cmp -s "$VALE_ASSETS28/.vale.ini" "$UNLOAD30/.vale.ini" || cmp -s "$VALE_ASSETS28/.vale.ini" "$LEAK30/.vale.ini"; then if cmp -s "$VALE_ASSETS28/.vale.ini" "$UNLOAD30/.vale.ini" || cmp -s "$VALE_ASSETS28/.vale.ini" "$LEAK30/.vale.ini"; then
fail "a Copilot-scope mutation left the copied config unchanged, so Part B mutated nothing and proves nothing about Part A" fail "a Copilot-scope mutation left the copied config unchanged, so Part B mutated nothing and proves nothing about Part A"
elif ! printf '%s' "$UNLOAD_FAILS30" | grep -qF "[copilot/demo.agent.md is an agent file but raised no ProactivePhrase alert"; then elif ! grep -qF "[copilot/demo.agent.md is an agent file but raised no ProactivePhrase alert" <<< "$UNLOAD_FAILS30"; then
fail "dropping KyberforgeCopilot from [**/*.agent.md] did not fail Part A, so an unloaded Copilot style would pass silently again: ${UNLOAD_FAILS30:-<no failure>}" fail "dropping KyberforgeCopilot from [**/*.agent.md] did not fail Part A, so an unloaded Copilot style would pass silently again: ${UNLOAD_FAILS30:-<no failure>}"
elif ! printf '%s' "$LEAK_FAILS30" | grep -qF "[plugins/demo/.apm/skills/demo/SKILL.md is not an .agent.md file but raised a ProactivePhrase alert"; then elif ! grep -qF "[plugins/demo/.apm/skills/demo/SKILL.md is not an .agent.md file but raised a ProactivePhrase alert" <<< "$LEAK_FAILS30"; then
fail "adding KyberforgeCopilot to [**/SKILL.md] did not fail Part A, so the style could leak past ADR-0013's scope unnoticed: ${LEAK_FAILS30:-<no failure>}" fail "adding KyberforgeCopilot to [**/SKILL.md] did not fail Part A, so the style could leak past ADR-0013's scope unnoticed: ${LEAK_FAILS30:-<no failure>}"
else else
pass "unloading KyberforgeCopilot from .agent.md files and leaking it onto SKILL.md files are each caught by Part A" pass "unloading KyberforgeCopilot from .agent.md files and leaking it onto SKILL.md files are each caught by Part A"
@@ -1972,7 +1974,7 @@ SILENCED31="$(vale_report28 "$OVERRIDE_DIR31/.vale.ini" "$TREE28" "copilot/demo.
SILENCED_COUNT31="$(files_scanned28 "$SILENCED31")" SILENCED_COUNT31="$(files_scanned28 "$SILENCED31")"
if [[ -z "$SILENCED_COUNT31" || "$SILENCED_COUNT31" -eq 0 ]]; then if [[ -z "$SILENCED_COUNT31" || "$SILENCED_COUNT31" -eq 0 ]]; then
fail "the override fixture scanned no file at all, so the disappearance of the VagueWording alert proves nothing about overrides" fail "the override fixture scanned no file at all, so the disappearance of the VagueWording alert proves nothing about overrides"
elif printf '%s\n' "$SILENCED31" | grep -qF "Kyberforge.VagueWording"; then elif grep -qF "Kyberforge.VagueWording" <<< "$SILENCED31"; then
# Reported as a FAIL, not a pass. It is not a defect in the config, but it # Reported as a FAIL, not a pass. It is not a defect in the config, but it
# means Parts A and B are guarding a failure mode this vale build no longer # means Parts A and B are guarding a failure mode this vale build no longer
# has — and a guard that guards nothing while reporting PASS is the same # has — and a guard that guards nothing while reporting PASS is the same
@@ -2115,13 +2117,13 @@ awk '/^[ \t]*files:/ { sub(/\^/, "^zzz-no-such-path/") } { print }' \
"$PC_CONFIG32" > "$MUT32/.pre-commit-config.yaml" "$PC_CONFIG32" > "$MUT32/.pre-commit-config.yaml"
MUT_RECORDS32="$(hook_records28 "$MUT32/.pre-commit-config.yaml")" MUT_RECORDS32="$(hook_records28 "$MUT32/.pre-commit-config.yaml")"
MUT_FAILS32="$(prefilter_scope_failures32 "$MUT32/.pre-commit-config.yaml" "$REPO_FILES32")" MUT_FAILS32="$(prefilter_scope_failures32 "$MUT32/.pre-commit-config.yaml" "$REPO_FILES32")"
if ! printf '%s\n' "$MUT_RECORDS32" | grep -q 'zzz-no-such-path'; then if ! grep -q 'zzz-no-such-path' <<< "$MUT_RECORDS32"; then
fail "the narrowed regexes never reached the copied config, so Part B narrowed nothing and proves nothing about Part A" fail "the narrowed regexes never reached the copied config, so Part B narrowed nothing and proves nothing about Part A"
elif [[ "$(printf '%s\n' "$MUT_RECORDS32" | grep -c .)" -ne 2 ]]; then elif [[ "$(printf '%s\n' "$MUT_RECORDS32" | grep -c .)" -ne 2 ]]; then
fail "the mutated config did not parse back as two prefilter hooks, so any failure below would come from the parser, not from the narrowing" fail "the mutated config did not parse back as two prefilter hooks, so any failure below would come from the parser, not from the narrowing"
elif ! printf '%s' "$MUT_FAILS32" | grep -qF "vale-audit-prefilter-skill: 'files: ^zzz-no-such-path/"; then elif ! grep -qF "vale-audit-prefilter-skill: 'files: ^zzz-no-such-path/" <<< "$MUT_FAILS32"; then
fail "narrowing the skill hook's regex to match zero files did not fail this check, so Part A cannot detect a prefilter that has been silently switched off for SKILL.md files" fail "narrowing the skill hook's regex to match zero files did not fail this check, so Part A cannot detect a prefilter that has been silently switched off for SKILL.md files"
elif ! printf '%s' "$MUT_FAILS32" | grep -qF "vale-audit-prefilter-agent: 'files: ^zzz-no-such-path/"; then elif ! grep -qF "vale-audit-prefilter-agent: 'files: ^zzz-no-such-path/" <<< "$MUT_FAILS32"; then
fail "narrowing the agent hook's regex to match zero files did not fail this check, so Part A cannot detect a prefilter that has been silently switched off for agent files" fail "narrowing the agent hook's regex to match zero files did not fail this check, so Part A cannot detect a prefilter that has been silently switched off for agent files"
else else
pass "narrowing either hook's 'files:' regex to match zero files is caught by Part A, which is what makes its pass mean something" pass "narrowing either hook's 'files:' regex to match zero files is caught by Part A, which is what makes its pass mean something"
@@ -2314,15 +2316,15 @@ if ! grep -qF '^plugins/kyberforge/\.apm/skills/' "$MUT33/skill.yaml" \
|| ! grep -qF '^plugins/kyberforge/\.apm/agents/' "$MUT33/agent.yaml" \ || ! grep -qF '^plugins/kyberforge/\.apm/agents/' "$MUT33/agent.yaml" \
|| ! grep -qF 'vale-audit-prefilter-skill-renamed' "$MUT33/id.yaml"; then || ! grep -qF 'vale-audit-prefilter-skill-renamed' "$MUT33/id.yaml"; then
fail "a mutation never reached its copied config, so Part B mutated nothing and proves nothing about Part A" fail "a mutation never reached its copied config, so Part B mutated nothing and proves nothing about Part A"
elif ! printf '%s' "$SKILL_FAILS33" | grep -qF "[plugins/demo/.apm/skills/demo/SKILL.md is in scope of kyberforge-vale-audit-skill"; then elif ! grep -qF "[plugins/demo/.apm/skills/demo/SKILL.md is in scope of kyberforge-vale-audit-skill" <<< "$SKILL_FAILS33"; then
fail "narrowing the local skill hook to ^plugins/kyberforge/ did not fail Part A, so the prefilter can drop every other plugin's skills with every gate green: ${SKILL_FAILS33:-<no failure>}" fail "narrowing the local skill hook to ^plugins/kyberforge/ did not fail Part A, so the prefilter can drop every other plugin's skills with every gate green: ${SKILL_FAILS33:-<no failure>}"
elif [[ -z "$AGREE_FAILS33" ]] && printf '%s' "$SKILL_FAILS33" | grep -qF "kyberforge-vale-audit-agent"; then elif [[ -z "$AGREE_FAILS33" ]] && grep -qF "kyberforge-vale-audit-agent" <<< "$SKILL_FAILS33"; then
# Only meaningful against a clean base: when Part A already failed, the copy # Only meaningful against a clean base: when Part A already failed, the copy
# inherits that defect, and reporting it again here would be one defect twice. # inherits that defect, and reporting it again here would be one defect twice.
fail "narrowing only the skill hook also reported an agent-class defect, so the comparison is not confined to its class: $SKILL_FAILS33" fail "narrowing only the skill hook also reported an agent-class defect, so the comparison is not confined to its class: $SKILL_FAILS33"
elif ! printf '%s' "$AGENT_FAILS33" | grep -qF "[plugins/demo/.apm/agents/demo.agent.md is in scope of kyberforge-vale-audit-agent"; then elif ! grep -qF "[plugins/demo/.apm/agents/demo.agent.md is in scope of kyberforge-vale-audit-agent" <<< "$AGENT_FAILS33"; then
fail "narrowing the local agent hook to ^plugins/kyberforge/ did not fail Part A: ${AGENT_FAILS33:-<no failure>}" fail "narrowing the local agent hook to ^plugins/kyberforge/ did not fail Part A: ${AGENT_FAILS33:-<no failure>}"
elif ! printf '%s' "$ID_FAILS33" | grep -qF "[no hook with id 'vale-audit-prefilter-skill' and a files: regex in id.yaml"; then elif ! grep -qF "[no hook with id 'vale-audit-prefilter-skill' and a files: regex in id.yaml" <<< "$ID_FAILS33"; then
fail "renaming the local skill hook's id did not fail Part A by name, so the skill class could fall out of the comparison silently: ${ID_FAILS33:-<no failure>}" fail "renaming the local skill hook's id did not fail Part A by name, so the skill class could fall out of the comparison silently: ${ID_FAILS33:-<no failure>}"
else else
pass "narrowing either local hook to one plugin, or renaming one, is caught by Part A" pass "narrowing either local hook to one plugin, or renaming one, is caught by Part A"