fix(tests): replace pipefail-racy echo | grep -q with here-strings
Why Two suites failed intermittently — tests/test-vale-wrap.sh case 21 and tests/test-check-release-needed.sh cases 4 and 15 — on correct output, and never when run alone. The cause is the `echo "$OUT" | grep -q P` idiom under `set -o pipefail`: grep -q exits as soon as it has an answer, bash's echo can hand a multi-line value to the pipe one line at a time, and a write after the reader is gone kills echo with SIGPIPE. pipefail then reports the writer's death, so output that DID match reads as "no match". Every observed failure had lines after its match; case 15's match is on line 1 of 6, the widest window in that file. Forced with a pause before the writer's last line, the pipe form failed 50 of 50 runs; a here-string, a match on the last line, and the same pipe without pipefail each passed 50 of 50. Unforced the rate is about 1 per 670 suite runs, which is why it read as a flaky gate rather than a bug. The failures at review time are consistent with this, but were not proven to be it: the suite was running while agents edited live config files in place, and a brief change to .vale.ini or .pre-commit-hooks.yaml would produce the same two failures. The race is real and fixed either way. Implementation Notes `grep -q P <<< "$VAR"` has no separate writer process, so there is nothing to race. It is not a retry or a sleep. 121 sites converted across 9 files, three of them scripts rather than tests: new-agent.sh, new-skill.sh and check-executables-allow-sync.sh. None ships via .pre-commit-hooks.yaml, so no external consumer pins them, and all three are single-pipeline checks whose verdict cannot change. Left alone deliberately: 14 sites whose writer is a command, not a shell builtin — they either absorb the writer's status with `|| true` or are python3 and awk, which write once at exit — and one file with no pipefail. `printf '%s'` sites differ from a here-string only by a trailing newline, which no -q verdict on a non-empty pattern depends on. tests/test-no-pipefail-early-exit-grep.sh is a static guard against new occurrences, discovered automatically by run-tests.sh. It only scans files that set pipefail, joins continuation lines, skips comments, and flags only echo/printf writers. Its first case proves the scanner can fail before its second trusts a clean verdict on the tree. A guard covers exactly the spellings its regex models, so the miss surface was measured rather than assumed. Four were found and closed: pipefail declared as `set -o errexit -o pipefail` (where the old pattern required pipefail to follow the FIRST -o, and a file-level miss skips every site in that file); a writer separated from grep by an intermediate stage; a pipeline wrapped on a trailing `|` rather than a backslash; and readers spelled egrep, fgrep, /bin/grep, `command grep` or with an env-var prefix. Segment characters exclude a bare `&` so `echo ok && other | grep -q x`, whose writer is `other`, does not false-fire. Widening surfaced 5 live sites invisible to the original scanner, all in tests/test-apm-current-hook.sh, all `echo "$out" | json_field ... | grep -q`; they are safe today only because json_field is python3, which reads to EOF and writes once. Fixtures go 4 to 12 vulnerable spellings plus near-miss negatives. Two `grep ... | head -1` sites (test-vale-wrap.sh) are the same race with a different early-exiting reader, and are fixed by absorbing the writer. The scanner deliberately does not model `head`, `sed -n 1p` or a bare `read`: most legitimate uses in this tree are already absorbed with `|| true` and the scanner cannot see absorption from pipeline text, so a high false-positive rate would be how this guard gets weakened. Heredoc bodies are scanned as code; none in the tree trips it today. Impact The bug predates the factory-audit merge: every converted site in check-release-needed and case 21 dates to4d018afandaa8cc22(2026-08-09). Test suites go 19 to 20. `run-tests.sh --strict` passes 20/20 with 0 skipped, four consecutive runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YR2CjVumUbEGWcMikcoXBD
This commit is contained in:
@@ -133,3 +133,7 @@ Widening a description-opener rule to also catch mid-sentence text looked like a
|
|||||||
## 2026-08-16 — A rule reversed inside a retrofit leaves no trace unless someone writes it down
|
## 2026-08-16 — A rule reversed inside a retrofit leaves no trace unless someone writes it down
|
||||||
|
|
||||||
A retrofit replaced "keep reference chains one level deep" with "two hops, never three" — the opposite rule, needed because the new dispatch pattern requires `SKILL.md` → `improve.md` → `retrofit.md`. The ADR never mentioned chain depth, so the reversal was carried entirely by the diff with no sign a contradicting rule ever existed. Fix: when a change inverts a standing rule, record the inversion where the rule's rationale lives, or it reads as forgotten rather than overturned.
|
A retrofit replaced "keep reference chains one level deep" with "two hops, never three" — the opposite rule, needed because the new dispatch pattern requires `SKILL.md` → `improve.md` → `retrofit.md`. The ADR never mentioned chain depth, so the reversal was carried entirely by the diff with no sign a contradicting rule ever existed. Fix: when a change inverts a standing rule, record the inversion where the rule's rationale lives, or it reads as forgotten rather than overturned.
|
||||||
|
|
||||||
|
## 2026-09-15 — A rare flake in a pipefail suite is a race until proven otherwise
|
||||||
|
|
||||||
|
The pre-push `run-tests` failed 2 of 3 full runs, on a different suite each time, and neither failure reproduced alone, so it was treated as noise. The cause was `echo "$OUT" | grep -q P` under `set -o pipefail`, at 116 sites. `grep -q` exits on its first match, `echo` takes SIGPIPE on its next write, and pipefail reports correct output as "no match". Unforced it failed about once in 670 runs; with a pause forced before the last line, 50 of 50. Fix: use `grep -q P <<< "$OUT"` (a here-string has no writer process to race), and add a static guard (`tests/test-no-pipefail-early-exit-grep.sh`) instead of relying on convention.
|
||||||
|
|||||||
@@ -83,7 +83,7 @@ Where the 276 s goes (each suite run alone, sequential):
|
|||||||
|
|
||||||
Five suites account for 215 s of 276 s. Three of those five (sync-plugin-content, vale-style-sync, adr0020-differential) test tooling that findings 2, 7, and 14 propose to delete or shrink, so the fastest path to a quick pre-push is removing the duplication those tests guard rather than optimising the tests.
|
Five suites account for 215 s of 276 s. Three of those five (sync-plugin-content, vale-style-sync, adr0020-differential) test tooling that findings 2, 7, and 14 propose to delete or shrink, so the fastest path to a quick pre-push is removing the duplication those tests guard rather than optimising the tests.
|
||||||
|
|
||||||
> **Also struck (2026-09-15):** `test-check-vale-style-sync.sh` (25 s) went with the `check-vale-style-sync` hook in finding 14's merge (ADR-0025). Measured at HEAD: `tests/` holds **19** `test-*.sh` suites and `tests/run-tests.sh` reports `19 passed, 0 skipped, 0 failed`. Same basis as the note below — arithmetic on the 2026-09-10 baseline minus the struck rows, not a fresh timing run.
|
> **Also struck (2026-09-15):** `test-check-vale-style-sync.sh` (25 s) went with the `check-vale-style-sync` hook in finding 14's merge (ADR-0025). Measured at HEAD: `tests/` holds **19** `test-*.sh` suites and `tests/run-tests.sh` reports `19 passed, 0 skipped, 0 failed`. (Later the same day, the pipefail-race fix added `tests/test-no-pipefail-early-exit-grep.sh`, making it **20**. That suite is a static scan and runs in well under a second, so the timing arithmetic here is unaffected.) Same basis as the note below — arithmetic on the 2026-09-10 baseline minus the struck rows, not a fresh timing run.
|
||||||
|
|
||||||
> **Done (2026-09-14):** see commit `718c79a` on `docs/simplification-audit`. The three struck-through rows are gone: `test-sync-plugin-content.sh` (83 s, 1,289 lines, 92 cases), `check-plugin-content-sync` (4.5 s) and `validate-plugins` (4.9 s). Expected, not re-measured: roughly 92 s comes off every push (83 + 4.5 + 4.9 = 92.4 s) (~83 s of it out of `run-tests`, which loses its single slowest suite), on the arithmetic of the 2026-09-10 figures alone. The remaining rows have not been re-timed since, so treat every number in this section as the 2026-09-10 baseline minus those three, not as a fresh measurement.
|
> **Done (2026-09-14):** see commit `718c79a` on `docs/simplification-audit`. The three struck-through rows are gone: `test-sync-plugin-content.sh` (83 s, 1,289 lines, 92 cases), `check-plugin-content-sync` (4.5 s) and `validate-plugins` (4.9 s). Expected, not re-measured: roughly 92 s comes off every push (83 + 4.5 + 4.9 = 92.4 s) (~83 s of it out of `run-tests`, which loses its single slowest suite), on the arithmetic of the 2026-09-10 figures alone. The remaining rows have not been re-timed since, so treat every number in this section as the 2026-09-10 baseline minus those three, not as a fresh measurement.
|
||||||
|
|
||||||
|
|||||||
@@ -862,6 +862,16 @@ vale-less PATH, with the skip list swallowed. Without vale, two suites skip —
|
|||||||
what to install. (It was three until `test-check-vale-style-sync.sh` was deleted with its hook; see
|
what to install. (It was three until `test-check-vale-style-sync.sh` was deleted with its hook; see
|
||||||
[One copy, one config](#one-copy-one-config).)
|
[One copy, one config](#one-copy-one-config).)
|
||||||
|
|
||||||
|
**Output assertions use a here-string, never a pipe.** Write `grep -q PATTERN <<< "$OUT"`, not
|
||||||
|
`echo "$OUT" | grep -q PATTERN`. Under `set -o pipefail` the pipe form fails depending on timing:
|
||||||
|
`grep -q` exits on its first match, `echo` takes SIGPIPE on its next write, and pipefail reports that
|
||||||
|
as the pipeline failing, so output that matched reads as "no match". It showed up as a push gate that
|
||||||
|
failed about once in 670 runs, on a different suite each time. `tests/test-no-pipefail-early-exit-grep.sh`
|
||||||
|
scans every tracked shell file that sets pipefail and fails on the pipe form. It does this for `echo`
|
||||||
|
or `printf` piped into `grep` with `-q`, `-m`, `-l`, `-L`, `--quiet` or `--silent`. It checks its own
|
||||||
|
scanner against fixtures before trusting a clean result. Pipes from other commands are out of scope.
|
||||||
|
In practice they either absorb the writer's exit status with `|| true` or write only once, at exit.
|
||||||
|
|
||||||
`tests/run-bats.sh` derives the set of `.bats` files it expects from `git ls-files`, so a `.bats`
|
`tests/run-bats.sh` derives the set of `.bats` files it expects from `git ls-files`, so a `.bats`
|
||||||
file deleted from the worktree but still tracked in the index fails the run rather than silently
|
file deleted from the worktree but still tracked in the index fails the run rather than silently
|
||||||
shrinking the suite. Remove one with `git rm` (or stage the deletion) when intentional; an untracked
|
shrinking the suite. Remove one with `git rm` (or stage the deletion) when intentional; an untracked
|
||||||
|
|||||||
@@ -61,7 +61,7 @@ AGENT_NAME="$1"
|
|||||||
ROOT="$2"
|
ROOT="$2"
|
||||||
|
|
||||||
# Validate agent name format
|
# Validate agent name format
|
||||||
if ! echo "$AGENT_NAME" | grep -qE '^[a-z0-9]+(-[a-z0-9]+)*$'; then
|
if ! grep -qE '^[a-z0-9]+(-[a-z0-9]+)*$' <<< "$AGENT_NAME"; then
|
||||||
echo "Error: agent-name must use lowercase letters, numbers, and hyphens only." >&2
|
echo "Error: agent-name must use lowercase letters, numbers, and hyphens only." >&2
|
||||||
echo " No leading, trailing, or consecutive hyphens." >&2
|
echo " No leading, trailing, or consecutive hyphens." >&2
|
||||||
echo " Received: '$AGENT_NAME'" >&2
|
echo " Received: '$AGENT_NAME'" >&2
|
||||||
|
|||||||
@@ -61,7 +61,7 @@ SKILL_NAME="$1"
|
|||||||
TARGET_INPUT="$2"
|
TARGET_INPUT="$2"
|
||||||
|
|
||||||
# Validate skill name format
|
# Validate skill name format
|
||||||
if ! echo "$SKILL_NAME" | grep -qE '^[a-z0-9]+(-[a-z0-9]+)*$'; then
|
if ! grep -qE '^[a-z0-9]+(-[a-z0-9]+)*$' <<< "$SKILL_NAME"; then
|
||||||
echo "Error: skill-name must use lowercase letters, numbers, and hyphens only." >&2
|
echo "Error: skill-name must use lowercase letters, numbers, and hyphens only." >&2
|
||||||
echo " No leading, trailing, or consecutive hyphens." >&2
|
echo " No leading, trailing, or consecutive hyphens." >&2
|
||||||
echo " Received: '$SKILL_NAME'" >&2
|
echo " Received: '$SKILL_NAME'" >&2
|
||||||
|
|||||||
@@ -202,7 +202,7 @@ if [[ "$ALLOW_STATE" != "present" ]]; then
|
|||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if printf '%s\n' "$ALLOW_KEYS" | grep -qxF "$EXPECTED_KEY"; then
|
if grep -qxF "$EXPECTED_KEY" <<< "$ALLOW_KEYS"; then
|
||||||
exit 0
|
exit 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
|||||||
@@ -114,11 +114,11 @@ if echo "$out" | python3 -m json.tool > /dev/null 2>&1; then
|
|||||||
&& pass "declares hookEventName SessionStart" || fail "wrong hookEventName"
|
&& pass "declares hookEventName SessionStart" || fail "wrong hookEventName"
|
||||||
[[ "$(echo "$out" | json_field reloadSkills)" == "True" ]] \
|
[[ "$(echo "$out" | json_field reloadSkills)" == "True" ]] \
|
||||||
&& pass "asks the host to reload skills after a successful refresh" || fail "reloadSkills should be true"
|
&& pass "asks the host to reload skills after a successful refresh" || fail "reloadSkills should be true"
|
||||||
echo "$out" | json_field additionalContext | grep -q "6 package" \
|
grep -q "6 package" <<< "$(json_field additionalContext <<< "$out")" \
|
||||||
&& pass "reports the stale package count" || fail "should report the count"
|
&& pass "reports the stale package count" || fail "should report the count"
|
||||||
# The lockfile rewrite is the surprising part of auto-updating; the notice has
|
# The lockfile rewrite is the surprising part of auto-updating; the notice has
|
||||||
# to say so or a dirty worktree looks like something else went wrong.
|
# to say so or a dirty worktree looks like something else went wrong.
|
||||||
echo "$out" | json_field additionalContext | grep -q "apm.lock.yaml" \
|
grep -q "apm.lock.yaml" <<< "$(json_field additionalContext <<< "$out")" \
|
||||||
&& pass "warns that apm.lock.yaml was rewritten" || fail "should warn about the lockfile rewrite"
|
&& pass "warns that apm.lock.yaml was rewritten" || fail "should warn about the lockfile rewrite"
|
||||||
else
|
else
|
||||||
fail "emits valid JSON"
|
fail "emits valid JSON"
|
||||||
@@ -136,7 +136,7 @@ if echo "$out" | python3 -m json.tool > /dev/null 2>&1; then
|
|||||||
pass "emits valid JSON on failure"
|
pass "emits valid JSON on failure"
|
||||||
[[ "$(echo "$out" | json_field reloadSkills)" == "False" ]] \
|
[[ "$(echo "$out" | json_field reloadSkills)" == "False" ]] \
|
||||||
&& pass "does not ask for a skill reload when nothing was deployed" || fail "reloadSkills should be false"
|
&& pass "does not ask for a skill reload when nothing was deployed" || fail "reloadSkills should be false"
|
||||||
echo "$out" | json_field additionalContext | grep -q "apm update" \
|
grep -q "apm update" <<< "$(json_field additionalContext <<< "$out")" \
|
||||||
&& pass "tells the reader how to refresh by hand" || fail "should name the manual command"
|
&& pass "tells the reader how to refresh by hand" || fail "should name the manual command"
|
||||||
else
|
else
|
||||||
fail "emits valid JSON on failure"
|
fail "emits valid JSON on failure"
|
||||||
@@ -175,7 +175,7 @@ out="$(run_hook_in "$ELSEWHERE" "$WORK")"
|
|||||||
[[ "$(cat "$WORK/apm-cwd" 2>/dev/null)" == "$WORK" ]] \
|
[[ "$(cat "$WORK/apm-cwd" 2>/dev/null)" == "$WORK" ]] \
|
||||||
&& pass "runs apm in the directory the guard checked, not the cwd" \
|
&& pass "runs apm in the directory the guard checked, not the cwd" \
|
||||||
|| fail "apm ran in '$(cat "$WORK/apm-cwd" 2>/dev/null)' — must run in the resolved project directory"
|
|| fail "apm ran in '$(cat "$WORK/apm-cwd" 2>/dev/null)' — must run in the resolved project directory"
|
||||||
echo "$out" | json_field additionalContext | grep -q "6 package" \
|
grep -q "6 package" <<< "$(json_field additionalContext <<< "$out")" \
|
||||||
&& pass "reports the count found via CLAUDE_PROJECT_DIR" || fail "should report the count"
|
&& pass "reports the count found via CLAUDE_PROJECT_DIR" || fail "should report the count"
|
||||||
|
|
||||||
# The fallback is not cosmetic: a host that installed this plugin natively sets
|
# The fallback is not cosmetic: a host that installed this plugin natively sets
|
||||||
@@ -348,7 +348,7 @@ EOF
|
|||||||
chmod +x "$FAKE_BIN/apm"
|
chmod +x "$FAKE_BIN/apm"
|
||||||
rm -f "$WORK/update-was-called"
|
rm -f "$WORK/update-was-called"
|
||||||
out="$(run_hook)"
|
out="$(run_hook)"
|
||||||
if [[ -n "$out" ]] && echo "$out" | json_field additionalContext 2>/dev/null | grep -q "$want package"; then
|
if [[ -n "$out" ]] && grep -q "$want package" <<< "$(json_field additionalContext 2>/dev/null <<< "$out")"; then
|
||||||
pass "detects staleness in real \`apm outdated\` output and counts $want package(s)"
|
pass "detects staleness in real \`apm outdated\` output and counts $want package(s)"
|
||||||
else
|
else
|
||||||
fail "real apm reported $want outdated dependency/dependencies but the hook did not act on it — apm reworded its summary line. Real output: $(tr '\n' ' ' < "$PROBE/genuine-$want.txt" | tail -c 120)"
|
fail "real apm reported $want outdated dependency/dependencies but the hook did not act on it — apm reworded its summary line. Real output: $(tr '\n' ' ' < "$PROBE/genuine-$want.txt" | tail -c 120)"
|
||||||
|
|||||||
@@ -9,6 +9,13 @@ FAIL=0
|
|||||||
pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
|
pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
|
||||||
fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
|
fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
|
||||||
|
|
||||||
|
# Output assertions are `grep -q PATTERN <<< "$OUT"`, never `echo "$OUT" | grep -q`.
|
||||||
|
# Under pipefail the pipe form is scheduling-dependent: bash's echo writes a
|
||||||
|
# multi-line value one line at a time, `grep -q` exits on its first match, and a
|
||||||
|
# later line then hits a closed pipe. echo dies of SIGPIPE, pipefail reports the
|
||||||
|
# pipeline as failed, and a correct output reads as a missing match. The
|
||||||
|
# here-string has no writer process to race.
|
||||||
|
|
||||||
# Both entry shapes the real .pre-commit-hooks.yaml ships: a bare script with no
|
# Both entry shapes the real .pre-commit-hooks.yaml ships: a bare script with no
|
||||||
# bundled data, and a bare script whose sibling assets/ tree it self-locates at
|
# bundled data, and a bare script whose sibling assets/ tree it self-locates at
|
||||||
# runtime. Neither carries arguments — pre-commit only rewrites entry[0] to the
|
# runtime. Neither carries arguments — pre-commit only rewrites entry[0] to the
|
||||||
@@ -142,7 +149,7 @@ FIXTURE4="$(make_tagged_fixture)"; track "$FIXTURE4"
|
|||||||
echo "v2" > "$FIXTURE4/scripts/skill-size-check.sh"
|
echo "v2" > "$FIXTURE4/scripts/skill-size-check.sh"
|
||||||
(cd "$FIXTURE4" && git add -A && git commit -q -m "update release-relevant script")
|
(cd "$FIXTURE4" && git add -A && git commit -q -m "update release-relevant script")
|
||||||
OUT4=$(run_check "$FIXTURE4" "refs/heads/main" || true)
|
OUT4=$(run_check "$FIXTURE4" "refs/heads/main" || true)
|
||||||
if echo "$OUT4" | grep -q "skill-size-check.sh"; then
|
if grep -q "skill-size-check.sh" <<< "$OUT4"; then
|
||||||
pass "exits non-zero and names the changed file when a release-relevant path changed since the tag"
|
pass "exits non-zero and names the changed file when a release-relevant path changed since the tag"
|
||||||
else
|
else
|
||||||
fail "did not flag the release-relevant file that changed since the tag"
|
fail "did not flag the release-relevant file that changed since the tag"
|
||||||
@@ -167,7 +174,7 @@ FIXTURE6="$(make_tagged_fixture)"; track "$FIXTURE6"
|
|||||||
rm -f "$FIXTURE6/$HOOK_DIR/assets/vale/.vale.ini"
|
rm -f "$FIXTURE6/$HOOK_DIR/assets/vale/.vale.ini"
|
||||||
(cd "$FIXTURE6" && git add -A && git commit -q -m "delete the bundled vale config")
|
(cd "$FIXTURE6" && git add -A && git commit -q -m "delete the bundled vale config")
|
||||||
OUT6=$(run_check "$FIXTURE6" "refs/heads/main" || true)
|
OUT6=$(run_check "$FIXTURE6" "refs/heads/main" || true)
|
||||||
if echo "$OUT6" | grep -q "assets/vale/.vale.ini"; then
|
if grep -q "assets/vale/.vale.ini" <<< "$OUT6"; then
|
||||||
pass "flags a deleted release-relevant path instead of silently dropping it from the diff"
|
pass "flags a deleted release-relevant path instead of silently dropping it from the diff"
|
||||||
else
|
else
|
||||||
fail "did not flag deletion of a release-relevant path since the tag"
|
fail "did not flag deletion of a release-relevant path since the tag"
|
||||||
@@ -196,7 +203,7 @@ echo "checkpoint" > "$FIXTURE8/scripts/skill-size-check.sh"
|
|||||||
echo "v2" > "$FIXTURE8/scripts/skill-size-check.sh"
|
echo "v2" > "$FIXTURE8/scripts/skill-size-check.sh"
|
||||||
(cd "$FIXTURE8" && git add -A && git commit -q -m "real release-relevant change")
|
(cd "$FIXTURE8" && git add -A && git commit -q -m "real release-relevant change")
|
||||||
OUT8=$(run_check "$FIXTURE8" "refs/heads/main" || true)
|
OUT8=$(run_check "$FIXTURE8" "refs/heads/main" || true)
|
||||||
if echo "$OUT8" | grep -q "skill-size-check.sh"; then
|
if grep -q "skill-size-check.sh" <<< "$OUT8"; then
|
||||||
pass "still flags the release-relevant change since v1.0.0, ignoring the non-version checkpoint tag"
|
pass "still flags the release-relevant change since v1.0.0, ignoring the non-version checkpoint tag"
|
||||||
else
|
else
|
||||||
fail "an incidental non-version tag shifted the baseline and hid a real release-relevant change"
|
fail "an incidental non-version tag shifted the baseline and hid a real release-relevant change"
|
||||||
@@ -225,7 +232,7 @@ FIXTURE10="$(make_tagged_fixture)"; track "$FIXTURE10"
|
|||||||
echo "rule: v2" > "$FIXTURE10/$HOOK_DIR/assets/vale/styles/Kyberforge/DemoRule.yml"
|
echo "rule: v2" > "$FIXTURE10/$HOOK_DIR/assets/vale/styles/Kyberforge/DemoRule.yml"
|
||||||
(cd "$FIXTURE10" && git add -A && git commit -q -m "tighten a vale rule")
|
(cd "$FIXTURE10" && git add -A && git commit -q -m "tighten a vale rule")
|
||||||
OUT10=$(run_check "$FIXTURE10" "refs/heads/main" || true)
|
OUT10=$(run_check "$FIXTURE10" "refs/heads/main" || true)
|
||||||
if echo "$OUT10" | grep -q "assets/vale/styles/Kyberforge/DemoRule.yml"; then
|
if grep -q "assets/vale/styles/Kyberforge/DemoRule.yml" <<< "$OUT10"; then
|
||||||
pass "flags a change confined to a hook's bundled assets/vale/styles/ tree"
|
pass "flags a change confined to a hook's bundled assets/vale/styles/ tree"
|
||||||
else
|
else
|
||||||
fail "a bundled Vale style rule changed since the tag without demanding a release"
|
fail "a bundled Vale style rule changed since the tag without demanding a release"
|
||||||
@@ -259,7 +266,7 @@ FIXTURE12="$(make_tagged_fixture)"; track "$FIXTURE12"
|
|||||||
rm -rf "${FIXTURE12:?}/$HOOK_DIR/assets"
|
rm -rf "${FIXTURE12:?}/$HOOK_DIR/assets"
|
||||||
(cd "$FIXTURE12" && git add -A && git commit -q -m "delete the whole bundled assets tree")
|
(cd "$FIXTURE12" && git add -A && git commit -q -m "delete the whole bundled assets tree")
|
||||||
OUT12=$(run_check "$FIXTURE12" "refs/heads/main" || true)
|
OUT12=$(run_check "$FIXTURE12" "refs/heads/main" || true)
|
||||||
if echo "$OUT12" | grep -q "assets/vale/.vale.ini"; then
|
if grep -q "assets/vale/.vale.ini" <<< "$OUT12"; then
|
||||||
pass "flags a wholesale deletion of a hook's bundled assets/ tree"
|
pass "flags a wholesale deletion of a hook's bundled assets/ tree"
|
||||||
else
|
else
|
||||||
fail "a hook's entire bundled assets/ tree vanished since the tag without demanding a release"
|
fail "a hook's entire bundled assets/ tree vanished since the tag without demanding a release"
|
||||||
@@ -276,7 +283,7 @@ FIXTURE13="$(make_tagged_fixture)"; track "$FIXTURE13"
|
|||||||
rm -f "$FIXTURE13/$HOOK_DIR/scripts/vale-wrap.sh"
|
rm -f "$FIXTURE13/$HOOK_DIR/scripts/vale-wrap.sh"
|
||||||
(cd "$FIXTURE13" && git add -A && git commit -q -m "delete a hook script, keep its manifest entry")
|
(cd "$FIXTURE13" && git add -A && git commit -q -m "delete a hook script, keep its manifest entry")
|
||||||
OUT13=$(run_check "$FIXTURE13" "refs/heads/main" || true)
|
OUT13=$(run_check "$FIXTURE13" "refs/heads/main" || true)
|
||||||
if echo "$OUT13" | grep -q "vale-wrap.sh"; then
|
if grep -q "vale-wrap.sh" <<< "$OUT13"; then
|
||||||
pass "flags a hook script deleted out from under a surviving manifest entry"
|
pass "flags a hook script deleted out from under a surviving manifest entry"
|
||||||
else
|
else
|
||||||
fail "a manifest entry's script vanished since the tag without demanding a release"
|
fail "a manifest entry's script vanished since the tag without demanding a release"
|
||||||
@@ -299,7 +306,7 @@ EOF
|
|||||||
rm -rf "${FIXTURE14:?}/$HOOK_DIR"
|
rm -rf "${FIXTURE14:?}/$HOOK_DIR"
|
||||||
(cd "$FIXTURE14" && git add -A && git commit -q -m "retire the vale hook entirely")
|
(cd "$FIXTURE14" && git add -A && git commit -q -m "retire the vale hook entirely")
|
||||||
OUT14=$(run_check "$FIXTURE14" "refs/heads/main" || true)
|
OUT14=$(run_check "$FIXTURE14" "refs/heads/main" || true)
|
||||||
if echo "$OUT14" | grep -q "vale-wrap.sh" && echo "$OUT14" | grep -q "assets/vale/.vale.ini"; then
|
if grep -q "vale-wrap.sh" <<< "$OUT14" && grep -q "assets/vale/.vale.ini" <<< "$OUT14"; then
|
||||||
pass "names the retired hook's script and bundled assets, not just the manifest edit"
|
pass "names the retired hook's script and bundled assets, not just the manifest edit"
|
||||||
else
|
else
|
||||||
fail "reported only the manifest change and hid which shipped paths the retirement removed"
|
fail "reported only the manifest change and hid which shipped paths the retirement removed"
|
||||||
@@ -320,9 +327,9 @@ FIXTURE15="$(make_malformed_fixture "bash scripts/skill-size-check.sh")"; track
|
|||||||
OUT15=$(run_check "$FIXTURE15" "refs/heads/main" || true)
|
OUT15=$(run_check "$FIXTURE15" "refs/heads/main" || true)
|
||||||
if run_check "$FIXTURE15" "refs/heads/main" > /dev/null; then
|
if run_check "$FIXTURE15" "refs/heads/main" > /dev/null; then
|
||||||
fail "silently exited 0 on a multi-token entry, dropping that hook's paths from the gate"
|
fail "silently exited 0 on a multi-token entry, dropping that hook's paths from the gate"
|
||||||
elif echo "$OUT15" | grep -q "fake-size-check" \
|
elif grep -q "fake-size-check" <<< "$OUT15" \
|
||||||
&& echo "$OUT15" | grep -q "bash scripts/skill-size-check.sh" \
|
&& grep -q "bash scripts/skill-size-check.sh" <<< "$OUT15" \
|
||||||
&& echo "$OUT15" | grep -q "ADR-0014"; then
|
&& grep -q "ADR-0014" <<< "$OUT15"; then
|
||||||
pass "rejects a multi-token entry, quoting it back and naming the hook and ADR-0014"
|
pass "rejects a multi-token entry, quoting it back and naming the hook and ADR-0014"
|
||||||
else
|
else
|
||||||
fail "rejected the multi-token entry without naming the hook, the entry, and ADR-0014"
|
fail "rejected the multi-token entry without naming the hook, the entry, and ADR-0014"
|
||||||
@@ -339,7 +346,7 @@ FIXTURE16="$(make_malformed_fixture "vale")"; track "$FIXTURE16"
|
|||||||
OUT16=$(run_check "$FIXTURE16" "refs/heads/main" || true)
|
OUT16=$(run_check "$FIXTURE16" "refs/heads/main" || true)
|
||||||
if run_check "$FIXTURE16" "refs/heads/main" > /dev/null; then
|
if run_check "$FIXTURE16" "refs/heads/main" > /dev/null; then
|
||||||
fail "silently exited 0 on an entry that names no shipped file"
|
fail "silently exited 0 on an entry that names no shipped file"
|
||||||
elif echo "$OUT16" | grep -q "fake-size-check" && echo "$OUT16" | grep -q "ADR-0014"; then
|
elif grep -q "fake-size-check" <<< "$OUT16" && grep -q "ADR-0014" <<< "$OUT16"; then
|
||||||
pass "rejects an entry that resolves to no file, naming the hook and the ADR-0014 constraint"
|
pass "rejects an entry that resolves to no file, naming the hook and the ADR-0014 constraint"
|
||||||
else
|
else
|
||||||
fail "rejected the unresolvable entry without naming the hook and the ADR-0014 constraint"
|
fail "rejected the unresolvable entry without naming the hook and the ADR-0014 constraint"
|
||||||
@@ -358,7 +365,7 @@ echo "v2" > "$FIXTURE17/scripts/skill-size-check.sh"
|
|||||||
(cd "$FIXTURE17" && git add -A && git commit -q -m "release-relevant change" \
|
(cd "$FIXTURE17" && git add -A && git commit -q -m "release-relevant change" \
|
||||||
&& git branch pushed-tip && git reset -q --hard v1.0.0)
|
&& git branch pushed-tip && git reset -q --hard v1.0.0)
|
||||||
OUT17=$(run_check "$FIXTURE17" "refs/heads/main" "pushed-tip" || true)
|
OUT17=$(run_check "$FIXTURE17" "refs/heads/main" "pushed-tip" || true)
|
||||||
if echo "$OUT17" | grep -q "skill-size-check.sh"; then
|
if grep -q "skill-size-check.sh" <<< "$OUT17"; then
|
||||||
pass "gates the pushed ref's tip, not HEAD, when HEAD is behind it"
|
pass "gates the pushed ref's tip, not HEAD, when HEAD is behind it"
|
||||||
else
|
else
|
||||||
fail "diffed HEAD instead of PRE_COMMIT_TO_REF and missed a release-relevant change"
|
fail "diffed HEAD instead of PRE_COMMIT_TO_REF and missed a release-relevant change"
|
||||||
@@ -431,7 +438,7 @@ FIXTURE21="$(make_tagged_fixture)"; track "$FIXTURE21"
|
|||||||
echo "v2" > "$FIXTURE21/scripts/skill-size-check.sh"
|
echo "v2" > "$FIXTURE21/scripts/skill-size-check.sh"
|
||||||
(cd "$FIXTURE21" && git add -A && git commit -q -m "real release-relevant change" && git tag v1.0.1-checkpoint)
|
(cd "$FIXTURE21" && git add -A && git commit -q -m "real release-relevant change" && git tag v1.0.1-checkpoint)
|
||||||
OUT21=$(run_check "$FIXTURE21" "refs/heads/main" || true)
|
OUT21=$(run_check "$FIXTURE21" "refs/heads/main" || true)
|
||||||
if echo "$OUT21" | grep -q "skill-size-check.sh"; then
|
if grep -q "skill-size-check.sh" <<< "$OUT21"; then
|
||||||
pass "still flags the release-relevant change since v1.0.0, ignoring the vX.Y.Z-checkpoint tag"
|
pass "still flags the release-relevant change since v1.0.0, ignoring the vX.Y.Z-checkpoint tag"
|
||||||
else
|
else
|
||||||
fail "a vX.Y.Z-checkpoint tag satisfied the glob and hid a real release-relevant change"
|
fail "a vX.Y.Z-checkpoint tag satisfied the glob and hid a real release-relevant change"
|
||||||
|
|||||||
221
tests/test-no-pipefail-early-exit-grep.sh
Executable file
221
tests/test-no-pipefail-early-exit-grep.sh
Executable file
@@ -0,0 +1,221 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# Static guard: no `echo|printf ... | grep -q` pipeline in a script that runs
|
||||||
|
# under `set -o pipefail`.
|
||||||
|
#
|
||||||
|
# The defect. grep -q (and -m, -l, -L, --quiet, --silent) exits as soon as it
|
||||||
|
# has an answer. If the writer on the left of the pipe still has output to
|
||||||
|
# write, that next write hits a closed pipe and the writer dies of SIGPIPE.
|
||||||
|
# Under pipefail the pipeline then reports the WRITER's failure, so output that
|
||||||
|
# matched reads as "no match". Whether it happens depends on scheduling: bash's
|
||||||
|
# echo can hand a multi-line value to the pipe a line at a time, and a reader
|
||||||
|
# that matched on an early line exits before the later lines arrive. Forced with
|
||||||
|
# a pause before the last line, the pipe form failed 50 of 50 runs; each of a
|
||||||
|
# here-string, a match on the last line, and the same pipe without pipefail
|
||||||
|
# passed 50 of 50. Unforced it surfaced about once per 670 suite runs, which is
|
||||||
|
# why it read as a flaky gate rather than as a bug.
|
||||||
|
#
|
||||||
|
# The fix is `grep -q PATTERN <<< "$VAR"`: a here-string has no separate writer
|
||||||
|
# process, so there is nothing to race. `printf '%s\n' "$X" | grep` and
|
||||||
|
# `<<< "$X"` feed grep the same bytes; `printf '%s'` and `echo` differ only by a
|
||||||
|
# trailing newline, which no -q verdict on a non-empty pattern depends on.
|
||||||
|
#
|
||||||
|
# Scope. Only echo/printf writers are flagged: they are the shell builtins that
|
||||||
|
# can split a write, and they are always replaceable by a here-string. A
|
||||||
|
# command writer (`run_wrap ... | grep -q`) is not flagged; those sites either
|
||||||
|
# absorb the writer's status (`|| true`) or write once at exit. Files without
|
||||||
|
# pipefail are not flagged, because without it the pipeline's status is grep's.
|
||||||
|
# Comment-only lines are skipped so the pattern can be named in prose.
|
||||||
|
#
|
||||||
|
# Remit. This scanner models early-exiting GREP readers only — grep, egrep and
|
||||||
|
# fgrep, however they are spelled (a path prefix, a `command` prefix, env-var
|
||||||
|
# assignments), anywhere in the pipeline. They are not the only readers that
|
||||||
|
# exit early: `head`, `sed -n 1p` and a bare `read` do too, and an echo/printf
|
||||||
|
# writer feeding any of them is the same race. Those sites are guarded by
|
||||||
|
# convention instead — absorb the writer's status with `|| true` (or take the
|
||||||
|
# verdict from a here-string) — and deliberately not by this test, because most
|
||||||
|
# legitimate uses of them in this tree are already absorbed and the scanner
|
||||||
|
# cannot see the absorption from the pipeline text alone. Flagging them would be
|
||||||
|
# noise; the grep readers are flagged because a here-string is always available.
|
||||||
|
#
|
||||||
|
# Known limitation: heredoc bodies are scanned as code, so a `cat <<'EOF'` body
|
||||||
|
# containing a vulnerable-looking line reads as a real site. There are none in
|
||||||
|
# the tree today.
|
||||||
|
#
|
||||||
|
# Case 1 proves the scanner can fail before case 2 trusts its clean verdict on
|
||||||
|
# the live tree.
|
||||||
|
|
||||||
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
SELF="$REPO_ROOT/tests/$(basename "${BASH_SOURCE[0]}")"
|
||||||
|
PASS=0
|
||||||
|
FAIL=0
|
||||||
|
|
||||||
|
pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
|
||||||
|
fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
|
||||||
|
|
||||||
|
RUN_TMP="$(mktemp -d)"
|
||||||
|
trap 'rm -rf "$RUN_TMP"' EXIT
|
||||||
|
|
||||||
|
# A pipeline-safe run of characters: anything but `|`, `;` or `&`, except that a
|
||||||
|
# `&` immediately followed by a digit is kept so `2>&1` does not end a segment.
|
||||||
|
# Excluding a bare `&` is what stops `echo ok && other | grep -q x` — a pipeline
|
||||||
|
# whose writer is `other`, not the echo — from being read as one site.
|
||||||
|
SAFE='([^|;&]|&[[:digit:]])'
|
||||||
|
|
||||||
|
# Writer, then any number of intermediate stages, then an early-exiting grep.
|
||||||
|
# The reader may be spelled `grep`, `egrep` or `fgrep`, behind a path prefix
|
||||||
|
# (`/bin/grep`), a `command` prefix, or env-var assignments (`LC_ALL=C grep`).
|
||||||
|
SITE_RE="(^|[^[:alnum:]_])(echo|printf)[[:space:]]${SAFE}*[|][[:space:]]*"
|
||||||
|
SITE_RE+="([^|;&[:space:]]${SAFE}*[|][[:space:]]*)*"
|
||||||
|
SITE_RE+="((command|[A-Za-z_][A-Za-z0-9_]*=[^[:space:]]*)[[:space:]]+)*"
|
||||||
|
SITE_RE+="([^[:space:];&|]*/)?(grep|egrep|fgrep)[[:space:]]"
|
||||||
|
SITE_RE+="(${SAFE}*[[:space:]])?(-[[:alpha:]]*[qmlL][[:alnum:]]*|--quiet|--silent|--max-count|--files-with)"
|
||||||
|
|
||||||
|
# `pipefail` need not follow the first `-o`: `set -o errexit -o pipefail` and
|
||||||
|
# `set -o posix -o pipefail` are the same setting. A file-level miss here skips
|
||||||
|
# every site in the file, so this is deliberately loose — the cost of scanning a
|
||||||
|
# file that does not set pipefail is nil, the cost of skipping one is total.
|
||||||
|
PIPEFAIL_RE='(^|[^[:alnum:]_-])set[[:space:]]+-[^;#]*pipefail'
|
||||||
|
|
||||||
|
# Prints `path:line: text` for every vulnerable pipeline in the given files.
|
||||||
|
# Continued lines are joined first — both a trailing backslash and a trailing
|
||||||
|
# `|`, which is equally legal as a line continuation in a pipeline — so a reader
|
||||||
|
# on the next physical line is still seen; the reported line is where the
|
||||||
|
# command starts.
|
||||||
|
scan() {
|
||||||
|
local f
|
||||||
|
for f in "$@"; do
|
||||||
|
grep -Eq "$PIPEFAIL_RE" "$f" || continue
|
||||||
|
awk -v file="$f" -v re="$SITE_RE" '
|
||||||
|
function flush( b) {
|
||||||
|
if (buf == "") return
|
||||||
|
if (buf !~ /^[[:space:]]*#/ && buf ~ re) print file ":" start ": " buf
|
||||||
|
buf = ""
|
||||||
|
}
|
||||||
|
{
|
||||||
|
if (buf == "") start = NR
|
||||||
|
line = $0
|
||||||
|
if (line ~ /\\$/) { buf = buf substr(line, 1, length(line) - 1) " "; next }
|
||||||
|
if (line ~ /[|][[:space:]]*$/) { buf = buf line " "; next }
|
||||||
|
buf = buf line
|
||||||
|
flush()
|
||||||
|
}
|
||||||
|
END { flush() }
|
||||||
|
' "$f"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
echo "--- the scanner flags each vulnerable spelling and nothing else ---"
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
BAD="$RUN_TMP/bad.sh"
|
||||||
|
cat > "$BAD" <<'EOF_BAD'
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
if echo "$OUT" | grep -q "needle"; then :; fi
|
||||||
|
! printf '%s\n' "$OUT" | grep -qF "needle" && :
|
||||||
|
if printf '%s' "$OUT" | grep -E -q "a|b"; then :; fi
|
||||||
|
if printf '%s\n' "$OUT" \
|
||||||
|
| grep -m1 "needle"; then :; fi
|
||||||
|
if echo "$OUT" | tr -d ' ' | grep -q "needle"; then :; fi
|
||||||
|
echo "$OUT" |
|
||||||
|
grep -q "needle" && :
|
||||||
|
if echo "$OUT" | egrep -q "needle"; then :; fi
|
||||||
|
if printf '%s\n' "$OUT" | fgrep -q "needle"; then :; fi
|
||||||
|
if echo "$OUT" | /bin/grep -q "needle"; then :; fi
|
||||||
|
if echo "$OUT" | command grep -q "needle"; then :; fi
|
||||||
|
if echo "$OUT" | LC_ALL=C grep -q "needle"; then :; fi
|
||||||
|
if echo "$OUT" | sed -n '1,$p' | command /usr/bin/grep --quiet "needle"; then :; fi
|
||||||
|
EOF_BAD
|
||||||
|
ALTSET="$RUN_TMP/alt-pipefail.sh"
|
||||||
|
cat > "$ALTSET" <<'EOF_ALT'
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
set -o errexit -o pipefail
|
||||||
|
if echo "$OUT" | grep -q "needle"; then :; fi
|
||||||
|
EOF_ALT
|
||||||
|
GOOD="$RUN_TMP/good.sh"
|
||||||
|
cat > "$GOOD" <<'EOF_GOOD'
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
# A comment naming the bad form: echo "$OUT" | grep -q needle
|
||||||
|
if grep -q "needle" <<< "$OUT"; then :; fi
|
||||||
|
COUNT="$(echo "$OUT" | grep -c "needle" || true)"
|
||||||
|
echo "$OUT" || grep -q "needle" <<< "$OUT"
|
||||||
|
run_wrap "$DIR" file.md | grep -q "needle" || :
|
||||||
|
printf '%s\n' "$OUT" | grep -c "needle"
|
||||||
|
echo "$OUT" | tail -n 1
|
||||||
|
echo "$OUT" | tr -d ' ' | tail -n 1
|
||||||
|
echo "$OUT" | mygrep -q "needle"
|
||||||
|
echo "$OUT" | command tail -n 1
|
||||||
|
echo "$OUT" | LC_ALL=C sort
|
||||||
|
echo "$OUT" |
|
||||||
|
tail -n 1
|
||||||
|
echo ok && run_wrap "$DIR" | grep -q "needle"
|
||||||
|
EOF_GOOD
|
||||||
|
NOPIPEFAIL="$RUN_TMP/no-pipefail.sh"
|
||||||
|
cat > "$NOPIPEFAIL" <<'EOF_NOPF'
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
set -eu
|
||||||
|
if echo "$OUT" | grep -q "needle"; then :; fi
|
||||||
|
EOF_NOPF
|
||||||
|
|
||||||
|
BAD_HITS="$(scan "$BAD")"
|
||||||
|
BAD_COUNT="$(grep -c . <<< "$BAD_HITS" || true)"
|
||||||
|
if [[ "$BAD_COUNT" -eq 12 ]]; then
|
||||||
|
pass "all 12 vulnerable spellings are flagged (echo, negated printf, split options, both continuations, an intermediate stage, egrep/fgrep, and path/command/env-assignment reader prefixes)"
|
||||||
|
else
|
||||||
|
fail "expected 12 hits in the bad fixture, got $BAD_COUNT: $BAD_HITS"
|
||||||
|
fi
|
||||||
|
if grep -q "^$BAD:6: " <<< "$BAD_HITS"; then
|
||||||
|
pass "a backslash-continued pipe is reported at the line the command starts"
|
||||||
|
else
|
||||||
|
fail "the backslash-continued site was not reported at line 6: $BAD_HITS"
|
||||||
|
fi
|
||||||
|
if grep -q "^$BAD:9: " <<< "$BAD_HITS"; then
|
||||||
|
pass "a pipeline wrapped after a trailing '|' is reported at the line the command starts"
|
||||||
|
else
|
||||||
|
fail "the trailing-pipe continuation site was not reported at line 9: $BAD_HITS"
|
||||||
|
fi
|
||||||
|
ALT_HITS="$(scan "$ALTSET")"
|
||||||
|
if [[ -n "$ALT_HITS" ]]; then
|
||||||
|
pass "pipefail set as 'set -o errexit -o pipefail' is recognised, so its sites are scanned"
|
||||||
|
else
|
||||||
|
fail "a file setting pipefail after another -o option was skipped entirely"
|
||||||
|
fi
|
||||||
|
GOOD_HITS="$(scan "$GOOD")"
|
||||||
|
if [[ -z "$GOOD_HITS" ]]; then
|
||||||
|
pass "here-strings, comments, grep -c, '||', '&&', command writers and non-grep readers are left alone"
|
||||||
|
else
|
||||||
|
fail "the clean fixture was flagged: $GOOD_HITS"
|
||||||
|
fi
|
||||||
|
if [[ -z "$(scan "$NOPIPEFAIL")" ]]; then
|
||||||
|
pass "a file without pipefail is not flagged — grep's own status is the pipeline's"
|
||||||
|
else
|
||||||
|
fail "a file without pipefail was flagged"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
echo "--- no tracked shell script carries the race ---"
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
FILES=()
|
||||||
|
while IFS= read -r rel; do
|
||||||
|
[[ "$REPO_ROOT/$rel" == "$SELF" ]] && continue
|
||||||
|
[[ -f "$REPO_ROOT/$rel" ]] && FILES+=("$REPO_ROOT/$rel")
|
||||||
|
done < <(git -C "$REPO_ROOT" ls-files -- '*.sh' '*.bats' '*.bash')
|
||||||
|
|
||||||
|
if [[ "${#FILES[@]}" -lt 20 ]]; then
|
||||||
|
fail "only ${#FILES[@]} tracked shell files found — the scan is looking in the wrong place"
|
||||||
|
else
|
||||||
|
LIVE_HITS="$(scan ${FILES[@]+"${FILES[@]}"})"
|
||||||
|
if [[ -z "$LIVE_HITS" ]]; then
|
||||||
|
pass "none of ${#FILES[@]} tracked shell files pipes echo/printf into an early-exit grep under pipefail"
|
||||||
|
else
|
||||||
|
fail "use \`grep -q PATTERN <<< \"\$VAR\"\` instead at:"
|
||||||
|
echo "${LIVE_HITS//"$REPO_ROOT/"/ }"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Results: $PASS passed, $FAIL failed"
|
||||||
|
[[ "$FAIL" -eq 0 ]]
|
||||||
@@ -112,7 +112,7 @@ EOF
|
|||||||
run_fake "$DIR1"
|
run_fake "$DIR1"
|
||||||
if [[ $FAKE_RC -eq 0 ]]; then
|
if [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "a bats emitting nothing exited 0 — a total harness failure reported as a pass"
|
fail "a bats emitting nothing exited 0 — a total harness failure reported as a pass"
|
||||||
elif echo "$FAKE_OUT" | grep -q "no TAP output at all"; then
|
elif grep -q "no TAP output at all" <<< "$FAKE_OUT"; then
|
||||||
pass "an empty TAP stream fails the run and names the broken harness"
|
pass "an empty TAP stream fails the run and names the broken harness"
|
||||||
else
|
else
|
||||||
fail "the run failed but not with the broken-harness message: $FAKE_OUT"
|
fail "the run failed but not with the broken-harness message: $FAKE_OUT"
|
||||||
@@ -134,9 +134,9 @@ EOF
|
|||||||
run_fake "$DIR2"
|
run_fake "$DIR2"
|
||||||
if [[ $FAKE_RC -eq 0 ]]; then
|
if [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "every test declaring zero tests exited 0 — wholesale @test removal reported as a pass"
|
fail "every test declaring zero tests exited 0 — wholesale @test removal reported as a pass"
|
||||||
elif echo "$FAKE_OUT" | grep -q "declared 0 tests"; then
|
elif grep -q "declared 0 tests" <<< "$FAKE_OUT"; then
|
||||||
pass "a plan-only TAP stream fails the run and names the removed tests"
|
pass "a plan-only TAP stream fails the run and names the removed tests"
|
||||||
elif echo "$FAKE_OUT" | grep -q "no TAP output at all"; then
|
elif grep -q "no TAP output at all" <<< "$FAKE_OUT"; then
|
||||||
fail "a valid '1..0' plan was misreported as a broken harness — the two branches are not distinguished"
|
fail "a valid '1..0' plan was misreported as a broken harness — the two branches are not distinguished"
|
||||||
else
|
else
|
||||||
fail "the run failed but not with the no-tests-declared message: $FAKE_OUT"
|
fail "the run failed but not with the no-tests-declared message: $FAKE_OUT"
|
||||||
@@ -160,7 +160,7 @@ EOF
|
|||||||
run_fake "$DIR3"
|
run_fake "$DIR3"
|
||||||
if [[ $FAKE_RC -ne 0 ]]; then
|
if [[ $FAKE_RC -ne 0 ]]; then
|
||||||
fail "a healthy TAP stream was failed by the zero-count guard: $FAKE_OUT"
|
fail "a healthy TAP stream was failed by the zero-count guard: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "^4 tests, 0 failures$"; then
|
elif grep -q "^4 tests, 0 failures$" <<< "$FAKE_OUT"; then
|
||||||
pass "two files reporting two passing tests each aggregate to 4 tests, 0 failures"
|
pass "two files reporting two passing tests each aggregate to 4 tests, 0 failures"
|
||||||
else
|
else
|
||||||
fail "a healthy TAP stream produced the wrong count: $FAKE_OUT"
|
fail "a healthy TAP stream produced the wrong count: $FAKE_OUT"
|
||||||
@@ -187,9 +187,9 @@ else
|
|||||||
run_fake "$DIR4"
|
run_fake "$DIR4"
|
||||||
if [[ $FAKE_RC -ne 0 ]]; then
|
if [[ $FAKE_RC -ne 0 ]]; then
|
||||||
fail "an empty .bats file beside a real one failed the run: $FAKE_OUT"
|
fail "an empty .bats file beside a real one failed the run: $FAKE_OUT"
|
||||||
elif ! echo "$FAKE_OUT" | grep -q "^1\.\.0$"; then
|
elif ! grep -q "^1\.\.0$" <<< "$FAKE_OUT"; then
|
||||||
fail "real bats did not emit '1..0' for an empty file, so the case-2 stub no longer matches it: $FAKE_OUT"
|
fail "real bats did not emit '1..0' for an empty file, so the case-2 stub no longer matches it: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "^1 tests, 0 failures$"; then
|
elif grep -q "^1 tests, 0 failures$" <<< "$FAKE_OUT"; then
|
||||||
pass "an empty .bats file contributes a '1..0' plan and the suite still passes"
|
pass "an empty .bats file contributes a '1..0' plan and the suite still passes"
|
||||||
else
|
else
|
||||||
fail "the real-bats run passed with an unexpected count: $FAKE_OUT"
|
fail "the real-bats run passed with an unexpected count: $FAKE_OUT"
|
||||||
@@ -215,7 +215,7 @@ EOF
|
|||||||
run_fake "$DIR5"
|
run_fake "$DIR5"
|
||||||
if [[ $FAKE_RC -eq 0 ]]; then
|
if [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "a 'not ok' TAP result exited 0 — a failing test reported as a pass because only the process status was consulted"
|
fail "a 'not ok' TAP result exited 0 — a failing test reported as a pass because only the process status was consulted"
|
||||||
elif echo "$FAKE_OUT" | grep -q "^2 tests, 2 failures$"; then
|
elif grep -q "^2 tests, 2 failures$" <<< "$FAKE_OUT"; then
|
||||||
pass "'not ok' lines fail the run even when every bats process exits 0"
|
pass "'not ok' lines fail the run even when every bats process exits 0"
|
||||||
else
|
else
|
||||||
fail "the run failed but with the wrong count: $FAKE_OUT"
|
fail "the run failed but with the wrong count: $FAKE_OUT"
|
||||||
@@ -242,7 +242,7 @@ EOF
|
|||||||
run_fake "$DIR6"
|
run_fake "$DIR6"
|
||||||
if [[ $FAKE_RC -eq 0 ]]; then
|
if [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "a bats process exiting 1 was reported as a pass because only the TAP text was consulted"
|
fail "a bats process exiting 1 was reported as a pass because only the TAP text was consulted"
|
||||||
elif echo "$FAKE_OUT" | grep -q "^2 tests, 0 failures$"; then
|
elif grep -q "^2 tests, 0 failures$" <<< "$FAKE_OUT"; then
|
||||||
pass "a non-zero bats exit fails the run even with no 'not ok' line to find"
|
pass "a non-zero bats exit fails the run even with no 'not ok' line to find"
|
||||||
else
|
else
|
||||||
fail "the run failed but with the wrong count: $FAKE_OUT"
|
fail "the run failed but with the wrong count: $FAKE_OUT"
|
||||||
@@ -282,7 +282,7 @@ run_fake "$DIR7"
|
|||||||
# the stated reason: the TAP stream the stub emitted is healthy, so "1 tests, 0
|
# the stated reason: the TAP stream the stub emitted is healthy, so "1 tests, 0
|
||||||
# failures" proves the zero-count guard did not fire and the empty status is the
|
# failures" proves the zero-count guard did not fire and the empty status is the
|
||||||
# only thing left that can have failed the run.
|
# only thing left that can have failed the run.
|
||||||
if ! echo "$FAKE_OUT" | grep -q "^1 tests, 0 failures$"; then
|
if ! grep -q "^1 tests, 0 failures$" <<< "$FAKE_OUT"; then
|
||||||
fail "the killed job did not leave the healthy TAP stream the case needs: $FAKE_OUT"
|
fail "the killed job did not leave the healthy TAP stream the case needs: $FAKE_OUT"
|
||||||
elif [[ $FAKE_RC -eq 0 ]]; then
|
elif [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "an empty status file was counted as a clean exit — a killed job reported as a pass"
|
fail "an empty status file was counted as a clean exit — a killed job reported as a pass"
|
||||||
@@ -312,9 +312,9 @@ rm "$DIR8/tests/b.bats"
|
|||||||
run_fake "$DIR8"
|
run_fake "$DIR8"
|
||||||
if [[ $FAKE_RC -eq 0 ]]; then
|
if [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "a tracked .bats file gone from the worktree passed — a deleted suite reads as green"
|
fail "a tracked .bats file gone from the worktree passed — a deleted suite reads as green"
|
||||||
elif ! echo "$FAKE_OUT" | grep -q "tracked .bats file(s) were not discovered"; then
|
elif ! grep -q "tracked .bats file(s) were not discovered" <<< "$FAKE_OUT"; then
|
||||||
fail "the run failed but not with the undiscovered-tracked-file message: $FAKE_OUT"
|
fail "the run failed but not with the undiscovered-tracked-file message: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "^ tests/b.bats$"; then
|
elif grep -q "^ tests/b.bats$" <<< "$FAKE_OUT"; then
|
||||||
pass "a tracked .bats file missing from the walk fails the run and names the file"
|
pass "a tracked .bats file missing from the walk fails the run and names the file"
|
||||||
else
|
else
|
||||||
fail "the run failed without naming the missing file: $FAKE_OUT"
|
fail "the run failed without naming the missing file: $FAKE_OUT"
|
||||||
@@ -339,7 +339,7 @@ git -C "$DIR8B" add tests/a.bats
|
|||||||
run_fake "$DIR8B"
|
run_fake "$DIR8B"
|
||||||
if [[ $FAKE_RC -ne 0 ]]; then
|
if [[ $FAKE_RC -ne 0 ]]; then
|
||||||
fail "an untracked .bats file was reported as a finding: $FAKE_OUT"
|
fail "an untracked .bats file was reported as a finding: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "^2 tests, 0 failures$"; then
|
elif grep -q "^2 tests, 0 failures$" <<< "$FAKE_OUT"; then
|
||||||
pass "an untracked .bats file is run without being demanded of the index"
|
pass "an untracked .bats file is run without being demanded of the index"
|
||||||
else
|
else
|
||||||
fail "the untracked-file run passed with the wrong count: $FAKE_OUT"
|
fail "the untracked-file run passed with the wrong count: $FAKE_OUT"
|
||||||
@@ -356,7 +356,7 @@ rm "$DIR8B/tests/b.bats"
|
|||||||
run_fake "$DIR8B"
|
run_fake "$DIR8B"
|
||||||
if [[ $FAKE_RC -eq 0 ]]; then
|
if [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "the .bats file added to the index a moment ago was not demanded back: $FAKE_OUT"
|
fail "the .bats file added to the index a moment ago was not demanded back: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "^ tests/b.bats$"; then
|
elif grep -q "^ tests/b.bats$" <<< "$FAKE_OUT"; then
|
||||||
pass "a file added to the index joins the expected set with no floor to bump"
|
pass "a file added to the index joins the expected set with no floor to bump"
|
||||||
else
|
else
|
||||||
fail "the run failed but did not name the newly tracked file: $FAKE_OUT"
|
fail "the run failed but did not name the newly tracked file: $FAKE_OUT"
|
||||||
@@ -380,9 +380,9 @@ EOF
|
|||||||
run_fake "$DIR8D"
|
run_fake "$DIR8D"
|
||||||
if [[ $FAKE_RC -ne 0 ]]; then
|
if [[ $FAKE_RC -ne 0 ]]; then
|
||||||
fail "a non-git tree failed the run: $FAKE_OUT"
|
fail "a non-git tree failed the run: $FAKE_OUT"
|
||||||
elif ! echo "$FAKE_OUT" | grep -q "not a git worktree root"; then
|
elif ! grep -q "not a git worktree root" <<< "$FAKE_OUT"; then
|
||||||
fail "a non-git tree silently skipped the derived expectation with no note: $FAKE_OUT"
|
fail "a non-git tree silently skipped the derived expectation with no note: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "^2 tests, 0 failures$"; then
|
elif grep -q "^2 tests, 0 failures$" <<< "$FAKE_OUT"; then
|
||||||
pass "a non-git tree runs the suite and says the expected set could not be derived"
|
pass "a non-git tree runs the suite and says the expected set could not be derived"
|
||||||
else
|
else
|
||||||
fail "the non-git run passed with the wrong count: $FAKE_OUT"
|
fail "the non-git run passed with the wrong count: $FAKE_OUT"
|
||||||
@@ -404,7 +404,7 @@ EOF
|
|||||||
run_fake "$DIR9"
|
run_fake "$DIR9"
|
||||||
if [[ $FAKE_RC -eq 0 ]]; then
|
if [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "finding no .bats files at all exited 0 — the whole suite can vanish and the run stays green"
|
fail "finding no .bats files at all exited 0 — the whole suite can vanish and the run stays green"
|
||||||
elif echo "$FAKE_OUT" | grep -q "found 0 .bats file"; then
|
elif grep -q "found 0 .bats file" <<< "$FAKE_OUT"; then
|
||||||
pass "finding no .bats files fails the run and says so"
|
pass "finding no .bats files fails the run and says so"
|
||||||
else
|
else
|
||||||
fail "the run failed but not with the zero-files message: $FAKE_OUT"
|
fail "the run failed but not with the zero-files message: $FAKE_OUT"
|
||||||
|
|||||||
@@ -115,9 +115,9 @@ EOF
|
|||||||
run_fake "$DIR1"
|
run_fake "$DIR1"
|
||||||
if [[ $FAKE_RC -ne 0 ]]; then
|
if [[ $FAKE_RC -ne 0 ]]; then
|
||||||
fail "a healthy fixture failed: $FAKE_OUT"
|
fail "a healthy fixture failed: $FAKE_OUT"
|
||||||
elif ! echo "$FAKE_OUT" | grep -q "^=== bats ===$"; then
|
elif ! grep -q "^=== bats ===$" <<< "$FAKE_OUT"; then
|
||||||
fail "a healthy run never announced the bats leg: $FAKE_OUT"
|
fail "a healthy run never announced the bats leg: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$"; then
|
elif grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$" <<< "$FAKE_OUT"; then
|
||||||
pass "a passing case script and a healthy bats runner report 1 passed, 0 failed"
|
pass "a passing case script and a healthy bats runner report 1 passed, 0 failed"
|
||||||
else
|
else
|
||||||
fail "a healthy run produced the wrong summary: $FAKE_OUT"
|
fail "a healthy run produced the wrong summary: $FAKE_OUT"
|
||||||
@@ -140,7 +140,7 @@ EOF
|
|||||||
run_fake "$DIR2"
|
run_fake "$DIR2"
|
||||||
if [[ $FAKE_RC -eq 0 ]]; then
|
if [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "a non-executable run-bats.sh exited 0 — the whole bats suite can vanish silently"
|
fail "a non-executable run-bats.sh exited 0 — the whole bats suite can vanish silently"
|
||||||
elif echo "$FAKE_OUT" | grep -q "bats runner not found or not executable"; then
|
elif grep -q "bats runner not found or not executable" <<< "$FAKE_OUT"; then
|
||||||
pass "a non-executable run-bats.sh fails the run and names what is missing"
|
pass "a non-executable run-bats.sh fails the run and names what is missing"
|
||||||
else
|
else
|
||||||
fail "the run failed but not with the missing-runner message: $FAKE_OUT"
|
fail "the run failed but not with the missing-runner message: $FAKE_OUT"
|
||||||
@@ -160,7 +160,7 @@ EOF
|
|||||||
run_fake "$DIR3"
|
run_fake "$DIR3"
|
||||||
if [[ $FAKE_RC -eq 0 ]]; then
|
if [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "a missing run-bats.sh exited 0 — a rename deletes the bats suite from the run with no diagnostic"
|
fail "a missing run-bats.sh exited 0 — a rename deletes the bats suite from the run with no diagnostic"
|
||||||
elif echo "$FAKE_OUT" | grep -q "bats runner not found or not executable"; then
|
elif grep -q "bats runner not found or not executable" <<< "$FAKE_OUT"; then
|
||||||
pass "a missing run-bats.sh fails the run and names what is missing"
|
pass "a missing run-bats.sh fails the run and names what is missing"
|
||||||
else
|
else
|
||||||
fail "the run failed but not with the missing-runner message: $FAKE_OUT"
|
fail "the run failed but not with the missing-runner message: $FAKE_OUT"
|
||||||
@@ -180,7 +180,7 @@ EOF
|
|||||||
run_fake "$DIR4" --bats-only
|
run_fake "$DIR4" --bats-only
|
||||||
if [[ $FAKE_RC -eq 0 ]]; then
|
if [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "--bats-only with no runner exited 0 having printed nothing — a total no-op reported as a pass"
|
fail "--bats-only with no runner exited 0 having printed nothing — a total no-op reported as a pass"
|
||||||
elif echo "$FAKE_OUT" | grep -q "bats runner not found or not executable"; then
|
elif grep -q "bats runner not found or not executable" <<< "$FAKE_OUT"; then
|
||||||
pass "--bats-only fails when the runner is missing rather than doing nothing quietly"
|
pass "--bats-only fails when the runner is missing rather than doing nothing quietly"
|
||||||
else
|
else
|
||||||
fail "--bats-only failed but not with the missing-runner message: $FAKE_OUT"
|
fail "--bats-only failed but not with the missing-runner message: $FAKE_OUT"
|
||||||
@@ -232,11 +232,11 @@ add_case "$DIR5B" test-ok.sh <<'EOF'
|
|||||||
echo "fine"
|
echo "fine"
|
||||||
EOF
|
EOF
|
||||||
run_fake "$DIR5B"
|
run_fake "$DIR5B"
|
||||||
if echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$"; then
|
if grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$" <<< "$FAKE_OUT"; then
|
||||||
fail "an empty run-bats.sh produced a green summary — the bats suite vanished with no diagnostic"
|
fail "an empty run-bats.sh produced a green summary — the bats suite vanished with no diagnostic"
|
||||||
elif [[ $FAKE_RC -eq 0 ]]; then
|
elif [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "an empty run-bats.sh exited 0: $FAKE_OUT"
|
fail "an empty run-bats.sh exited 0: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "without reporting an 'N tests, M failures' summary"; then
|
elif grep -q "without reporting an 'N tests, M failures' summary" <<< "$FAKE_OUT"; then
|
||||||
pass "an empty run-bats.sh fails the run and says the bats suite was never verified"
|
pass "an empty run-bats.sh fails the run and says the bats suite was never verified"
|
||||||
else
|
else
|
||||||
fail "the run failed but not with the no-summary message: $FAKE_OUT"
|
fail "the run failed but not with the no-summary message: $FAKE_OUT"
|
||||||
@@ -264,7 +264,7 @@ EOF
|
|||||||
run_fake "$DIR5C"
|
run_fake "$DIR5C"
|
||||||
if [[ $FAKE_RC -eq 0 ]]; then
|
if [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "a bats runner reporting 0 tests exited 0 — a suite that executed nothing read as green: $FAKE_OUT"
|
fail "a bats runner reporting 0 tests exited 0 — a suite that executed nothing read as green: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "reported 0 tests"; then
|
elif grep -q "reported 0 tests" <<< "$FAKE_OUT"; then
|
||||||
pass "a bats runner reporting 0 tests fails the run and says the suite executed nothing"
|
pass "a bats runner reporting 0 tests fails the run and says the suite executed nothing"
|
||||||
else
|
else
|
||||||
fail "the run failed but not with the zero-tests message: $FAKE_OUT"
|
fail "the run failed but not with the zero-tests message: $FAKE_OUT"
|
||||||
@@ -298,11 +298,11 @@ kill -9 $PPID
|
|||||||
sleep 5
|
sleep 5
|
||||||
EOF
|
EOF
|
||||||
run_fake "$DIR6"
|
run_fake "$DIR6"
|
||||||
if echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$"; then
|
if grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$" <<< "$FAKE_OUT"; then
|
||||||
fail "an empty status file counted as a pass — a killed job reads as green"
|
fail "an empty status file counted as a pass — a killed job reads as green"
|
||||||
elif [[ $FAKE_RC -eq 0 ]]; then
|
elif [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "an empty status file did not fail the run: $FAKE_OUT"
|
fail "an empty status file did not fail the run: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "^=== Summary: 0 passed, 0 skipped, 1 failed ===$"; then
|
elif grep -q "^=== Summary: 0 passed, 0 skipped, 1 failed ===$" <<< "$FAKE_OUT"; then
|
||||||
pass "an empty status file is counted as FAILED"
|
pass "an empty status file is counted as FAILED"
|
||||||
else
|
else
|
||||||
fail "an empty status file failed the run with the wrong summary: $FAKE_OUT"
|
fail "an empty status file failed the run with the wrong summary: $FAKE_OUT"
|
||||||
@@ -326,7 +326,7 @@ EOF
|
|||||||
run_fake "$DIR7"
|
run_fake "$DIR7"
|
||||||
if [[ $FAKE_RC -eq 0 ]]; then
|
if [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "a SIGKILLed job did not fail the run: $FAKE_OUT"
|
fail "a SIGKILLed job did not fail the run: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "^=== Summary: 0 passed, 0 skipped, 1 failed ===$"; then
|
elif grep -q "^=== Summary: 0 passed, 0 skipped, 1 failed ===$" <<< "$FAKE_OUT"; then
|
||||||
pass "a job killed with no status file written is counted as FAILED"
|
pass "a job killed with no status file written is counted as FAILED"
|
||||||
else
|
else
|
||||||
fail "a SIGKILLed job failed the run with the wrong summary: $FAKE_OUT"
|
fail "a SIGKILLed job failed the run with the wrong summary: $FAKE_OUT"
|
||||||
@@ -348,7 +348,7 @@ EOF
|
|||||||
run_fake "$DIR8"
|
run_fake "$DIR8"
|
||||||
if [[ $FAKE_RC -eq 0 ]]; then
|
if [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "a case script with a syntax error did not fail the run: $FAKE_OUT"
|
fail "a case script with a syntax error did not fail the run: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "^=== Summary: 0 passed, 0 skipped, 1 failed ===$"; then
|
elif grep -q "^=== Summary: 0 passed, 0 skipped, 1 failed ===$" <<< "$FAKE_OUT"; then
|
||||||
pass "a case script that fails to parse is counted as FAILED"
|
pass "a case script that fails to parse is counted as FAILED"
|
||||||
else
|
else
|
||||||
fail "a syntax error failed the run with the wrong summary: $FAKE_OUT"
|
fail "a syntax error failed the run with the wrong summary: $FAKE_OUT"
|
||||||
@@ -380,11 +380,11 @@ EOF
|
|||||||
run_fake "$DIR9"
|
run_fake "$DIR9"
|
||||||
if [[ $FAKE_RC -eq 0 ]]; then
|
if [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "a case script exiting 1 did not fail the run: $FAKE_OUT"
|
fail "a case script exiting 1 did not fail the run: $FAKE_OUT"
|
||||||
elif ! echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 1 skipped, 1 failed ===$"; then
|
elif ! grep -q "^=== Summary: 1 passed, 1 skipped, 1 failed ===$" <<< "$FAKE_OUT"; then
|
||||||
fail "the pass/skip/fail split was miscounted: $FAKE_OUT"
|
fail "the pass/skip/fail split was miscounted: $FAKE_OUT"
|
||||||
elif ! echo "$FAKE_OUT" | grep -q "^ test-b-skips.sh$"; then
|
elif ! grep -q "^ test-b-skips.sh$" <<< "$FAKE_OUT"; then
|
||||||
fail "the skipped script was not named in the skip list: $FAKE_OUT"
|
fail "the skipped script was not named in the skip list: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "^ test-a-fails.sh$"; then
|
elif grep -q "^ test-a-fails.sh$" <<< "$FAKE_OUT"; then
|
||||||
pass "exit 1 is FAILED, exit 77 is SKIPPED, and both are named in their lists"
|
pass "exit 1 is FAILED, exit 77 is SKIPPED, and both are named in their lists"
|
||||||
else
|
else
|
||||||
fail "the failed script was not named in the failure list: $FAKE_OUT"
|
fail "the failed script was not named in the failure list: $FAKE_OUT"
|
||||||
@@ -422,11 +422,11 @@ EOF
|
|||||||
run_fake "$DIR10" --strict
|
run_fake "$DIR10" --strict
|
||||||
if [[ $FAKE_RC -eq 0 ]]; then
|
if [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "--strict passed with a skipped suite — the gate reports green having verified less than it ran: $FAKE_OUT"
|
fail "--strict passed with a skipped suite — the gate reports green having verified less than it ran: $FAKE_OUT"
|
||||||
elif ! echo "$FAKE_OUT" | grep -q "a skip is a SETUP ERROR"; then
|
elif ! grep -q "a skip is a SETUP ERROR" <<< "$FAKE_OUT"; then
|
||||||
fail "--strict failed but never said a skip is a setup error: $FAKE_OUT"
|
fail "--strict failed but never said a skip is a setup error: $FAKE_OUT"
|
||||||
elif ! echo "$FAKE_OUT" | grep -q "test-needs-a-binary.sh"; then
|
elif ! grep -q "test-needs-a-binary.sh" <<< "$FAKE_OUT"; then
|
||||||
fail "--strict failed without naming the skipped suite: $FAKE_OUT"
|
fail "--strict failed without naming the skipped suite: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "^ SKIP: frobnicator is not installed"; then
|
elif grep -q "^ SKIP: frobnicator is not installed" <<< "$FAKE_OUT"; then
|
||||||
pass "--strict fails on a skip, names the suite, and carries through the reason it printed"
|
pass "--strict fails on a skip, names the suite, and carries through the reason it printed"
|
||||||
else
|
else
|
||||||
fail "--strict named the suite but swallowed its skip reason: $FAKE_OUT"
|
fail "--strict named the suite but swallowed its skip reason: $FAKE_OUT"
|
||||||
@@ -445,7 +445,7 @@ STRICT_ENV_OUT="$(TMPDIR="$STRICT_ENV_PRIV" TEST_DIR="$DIR10/cases" RUN_TESTS_ST
|
|||||||
bash "$DIR10/tests/run-tests.sh" 2>&1)" || STRICT_ENV_RC=$?
|
bash "$DIR10/tests/run-tests.sh" 2>&1)" || STRICT_ENV_RC=$?
|
||||||
if [[ $STRICT_ENV_RC -eq 0 ]]; then
|
if [[ $STRICT_ENV_RC -eq 0 ]]; then
|
||||||
fail "RUN_TESTS_STRICT=1 passed with a skipped suite: $STRICT_ENV_OUT"
|
fail "RUN_TESTS_STRICT=1 passed with a skipped suite: $STRICT_ENV_OUT"
|
||||||
elif echo "$STRICT_ENV_OUT" | grep -q "a skip is a SETUP ERROR"; then
|
elif grep -q "a skip is a SETUP ERROR" <<< "$STRICT_ENV_OUT"; then
|
||||||
pass "RUN_TESTS_STRICT=1 is the same gate as --strict"
|
pass "RUN_TESTS_STRICT=1 is the same gate as --strict"
|
||||||
else
|
else
|
||||||
fail "RUN_TESTS_STRICT=1 failed for some other reason: $STRICT_ENV_OUT"
|
fail "RUN_TESTS_STRICT=1 failed for some other reason: $STRICT_ENV_OUT"
|
||||||
@@ -460,9 +460,9 @@ echo "--- the same skipped suite passes, still SKIPPED, without strict ---"
|
|||||||
run_fake "$DIR10"
|
run_fake "$DIR10"
|
||||||
if [[ $FAKE_RC -ne 0 ]]; then
|
if [[ $FAKE_RC -ne 0 ]]; then
|
||||||
fail "a skipped suite failed a non-strict run — graceful skipping is gone: $FAKE_OUT"
|
fail "a skipped suite failed a non-strict run — graceful skipping is gone: $FAKE_OUT"
|
||||||
elif ! echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 1 skipped, 0 failed ===$"; then
|
elif ! grep -q "^=== Summary: 1 passed, 1 skipped, 0 failed ===$" <<< "$FAKE_OUT"; then
|
||||||
fail "a non-strict run miscounted the skip: $FAKE_OUT"
|
fail "a non-strict run miscounted the skip: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "^ SKIP: frobnicator is not installed"; then
|
elif grep -q "^ SKIP: frobnicator is not installed" <<< "$FAKE_OUT"; then
|
||||||
pass "without strict the suite is SKIPPED, the run passes, and the reason is still reported"
|
pass "without strict the suite is SKIPPED, the run passes, and the reason is still reported"
|
||||||
else
|
else
|
||||||
fail "a non-strict run passed but dropped the skip reason: $FAKE_OUT"
|
fail "a non-strict run passed but dropped the skip reason: $FAKE_OUT"
|
||||||
@@ -484,7 +484,7 @@ EOF
|
|||||||
run_fake "$DIR10D" --strict
|
run_fake "$DIR10D" --strict
|
||||||
if [[ $FAKE_RC -ne 0 ]]; then
|
if [[ $FAKE_RC -ne 0 ]]; then
|
||||||
fail "--strict failed a run with nothing skipped — it fails unconditionally: $FAKE_OUT"
|
fail "--strict failed a run with nothing skipped — it fails unconditionally: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$"; then
|
elif grep -q "^=== Summary: 1 passed, 0 skipped, 0 failed ===$" <<< "$FAKE_OUT"; then
|
||||||
pass "--strict leaves a run with no skips green"
|
pass "--strict leaves a run with no skips green"
|
||||||
else
|
else
|
||||||
fail "--strict passed with the wrong summary: $FAKE_OUT"
|
fail "--strict passed with the wrong summary: $FAKE_OUT"
|
||||||
@@ -506,7 +506,7 @@ EOF
|
|||||||
run_fake "$DIR10E" --strickt
|
run_fake "$DIR10E" --strickt
|
||||||
if [[ $FAKE_RC -eq 0 ]]; then
|
if [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "a misspelled flag was ignored and the run passed — a typo silently disarms the gate: $FAKE_OUT"
|
fail "a misspelled flag was ignored and the run passed — a typo silently disarms the gate: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "Usage: .*--bats-only.*--strict"; then
|
elif grep -q "Usage: .*--bats-only.*--strict" <<< "$FAKE_OUT"; then
|
||||||
pass "an unrecognised flag fails the run with usage"
|
pass "an unrecognised flag fails the run with usage"
|
||||||
else
|
else
|
||||||
fail "an unrecognised flag failed but not with usage: $FAKE_OUT"
|
fail "an unrecognised flag failed but not with usage: $FAKE_OUT"
|
||||||
@@ -532,7 +532,7 @@ EOF
|
|||||||
run_fake "$DIR10F" --strict
|
run_fake "$DIR10F" --strict
|
||||||
if [[ $FAKE_RC -eq 0 ]]; then
|
if [[ $FAKE_RC -eq 0 ]]; then
|
||||||
fail "--strict passed on a suite that skipped via stderr: $FAKE_OUT"
|
fail "--strict passed on a suite that skipped via stderr: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "^ widgetizer not installed -- skipping"; then
|
elif grep -q "^ widgetizer not installed -- skipping" <<< "$FAKE_OUT"; then
|
||||||
pass "a skip reason printed to stderr without a SKIP: prefix is still carried into the failure"
|
pass "a skip reason printed to stderr without a SKIP: prefix is still carried into the failure"
|
||||||
else
|
else
|
||||||
fail "--strict failed but lost the stderr skip reason: $FAKE_OUT"
|
fail "--strict failed but lost the stderr skip reason: $FAKE_OUT"
|
||||||
@@ -570,9 +570,9 @@ run_fake "$DIR10"
|
|||||||
unset RUN_TESTS_STRICT
|
unset RUN_TESTS_STRICT
|
||||||
if [[ $FAKE_RC -ne 0 ]]; then
|
if [[ $FAKE_RC -ne 0 ]]; then
|
||||||
fail "an inherited RUN_TESTS_STRICT=1 turned a non-strict fixture strict — this suite's own result depends on how it was launched: $FAKE_OUT"
|
fail "an inherited RUN_TESTS_STRICT=1 turned a non-strict fixture strict — this suite's own result depends on how it was launched: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "a skip is a SETUP ERROR"; then
|
elif grep -q "a skip is a SETUP ERROR" <<< "$FAKE_OUT"; then
|
||||||
fail "the fixture ran strict despite not asking for it: $FAKE_OUT"
|
fail "the fixture ran strict despite not asking for it: $FAKE_OUT"
|
||||||
elif echo "$FAKE_OUT" | grep -q "^=== Summary: 1 passed, 1 skipped, 0 failed ===$"; then
|
elif grep -q "^=== Summary: 1 passed, 1 skipped, 0 failed ===$" <<< "$FAKE_OUT"; then
|
||||||
pass "an ambient RUN_TESTS_STRICT=1 is scrubbed from fixtures that did not ask for strict"
|
pass "an ambient RUN_TESTS_STRICT=1 is scrubbed from fixtures that did not ask for strict"
|
||||||
else
|
else
|
||||||
fail "the scrubbed run produced the wrong summary: $FAKE_OUT"
|
fail "the scrubbed run produced the wrong summary: $FAKE_OUT"
|
||||||
|
|||||||
@@ -22,32 +22,32 @@ echo "--- assembly ---"
|
|||||||
|
|
||||||
out=$(run "$FULL")
|
out=$(run "$FULL")
|
||||||
|
|
||||||
echo "$out" | grep -q "myproject" && pass "shows basename of working directory" || fail "dir missing — got: $out"
|
grep -q "myproject" <<< "$out" && pass "shows basename of working directory" || fail "dir missing — got: $out"
|
||||||
echo "$out" | grep -q " · " && pass "segments joined with ' · '" || fail "separator missing — got: $out"
|
grep -q " · " <<< "$out" && pass "segments joined with ' · '" || fail "separator missing — got: $out"
|
||||||
echo "$out" | grep -q "Sonnet 4.6" && pass "shows model name" || fail "model name missing — got: $out"
|
grep -q "Sonnet 4.6" <<< "$out" && pass "shows model name" || fail "model name missing — got: $out"
|
||||||
echo "$out" | grep -q "context 20%" && pass "shows context % with 'context' label" || fail "context label wrong — got: $out"
|
grep -q "context 20%" <<< "$out" && pass "shows context % with 'context' label" || fail "context label wrong — got: $out"
|
||||||
|
|
||||||
echo ""
|
echo ""
|
||||||
echo "--- token formatting ---"
|
echo "--- token formatting ---"
|
||||||
|
|
||||||
# 15k + 5k = 20k → shown as "20k tokens"
|
# 15k + 5k = 20k → shown as "20k tokens"
|
||||||
echo "$out" | grep -q "20k tokens" && pass "formats total tokens as Xk when >= 1000" || fail "Xk format wrong — got: $out"
|
grep -q "20k tokens" <<< "$out" && pass "formats total tokens as Xk when >= 1000" || fail "Xk format wrong — got: $out"
|
||||||
|
|
||||||
# Sub-1000 total: 500 + 300 = 800 → shown as "800 tokens"
|
# Sub-1000 total: 500 + 300 = 800 → shown as "800 tokens"
|
||||||
LOW='{"workspace":{"current_dir":"/x"},"context_window":{"total_input_tokens":500,"total_output_tokens":300}}'
|
LOW='{"workspace":{"current_dir":"/x"},"context_window":{"total_input_tokens":500,"total_output_tokens":300}}'
|
||||||
out_low=$(run "$LOW")
|
out_low=$(run "$LOW")
|
||||||
echo "$out_low" | grep -q "800 tokens" && pass "shows raw count when total tokens < 1000" || fail "sub-1000 format wrong — got: $out_low"
|
grep -q "800 tokens" <<< "$out_low" && pass "shows raw count when total tokens < 1000" || fail "sub-1000 format wrong — got: $out_low"
|
||||||
|
|
||||||
echo ""
|
echo ""
|
||||||
echo "--- cost formatting ---"
|
echo "--- cost formatting ---"
|
||||||
|
|
||||||
# 0.25 → 25¢
|
# 0.25 → 25¢
|
||||||
echo "$out" | grep -q "25¢" && pass "formats sub-dollar cost as cents (¢)" || fail "cents format wrong — got: $out"
|
grep -q "25¢" <<< "$out" && pass "formats sub-dollar cost as cents (¢)" || fail "cents format wrong — got: $out"
|
||||||
|
|
||||||
# 1.71 → $1.71
|
# 1.71 → $1.71
|
||||||
DOLLAR='{"workspace":{"current_dir":"/x"},"cost":{"total_cost_usd":1.71},"context_window":{"total_input_tokens":0,"total_output_tokens":0}}'
|
DOLLAR='{"workspace":{"current_dir":"/x"},"cost":{"total_cost_usd":1.71},"context_window":{"total_input_tokens":0,"total_output_tokens":0}}'
|
||||||
out_dollar=$(run "$DOLLAR")
|
out_dollar=$(run "$DOLLAR")
|
||||||
echo "$out_dollar" | grep -qF '$1.71' && pass "formats dollar-plus cost as \$X.XX" || fail "dollar format wrong — got: $out_dollar"
|
grep -qF '$1.71' <<< "$out_dollar" && pass "formats dollar-plus cost as \$X.XX" || fail "dollar format wrong — got: $out_dollar"
|
||||||
|
|
||||||
echo ""
|
echo ""
|
||||||
echo "--- missing fields omitted ---"
|
echo "--- missing fields omitted ---"
|
||||||
@@ -55,12 +55,12 @@ echo "--- missing fields omitted ---"
|
|||||||
# No cost field → no ¢ or $ in output
|
# No cost field → no ¢ or $ in output
|
||||||
NO_COST='{"workspace":{"current_dir":"/x"},"context_window":{"total_input_tokens":0,"total_output_tokens":0}}'
|
NO_COST='{"workspace":{"current_dir":"/x"},"context_window":{"total_input_tokens":0,"total_output_tokens":0}}'
|
||||||
out_nocost=$(run "$NO_COST")
|
out_nocost=$(run "$NO_COST")
|
||||||
! echo "$out_nocost" | grep -qE '[¢$]' && pass "omits cost segment when cost absent" || fail "cost shown unexpectedly — got: $out_nocost"
|
! grep -qE '[¢$]' <<< "$out_nocost" && pass "omits cost segment when cost absent" || fail "cost shown unexpectedly — got: $out_nocost"
|
||||||
|
|
||||||
# Zero tokens → no token segment
|
# Zero tokens → no token segment
|
||||||
ZERO_TOK='{"workspace":{"current_dir":"/x"},"context_window":{"total_input_tokens":0,"total_output_tokens":0}}'
|
ZERO_TOK='{"workspace":{"current_dir":"/x"},"context_window":{"total_input_tokens":0,"total_output_tokens":0}}'
|
||||||
out_zerotok=$(run "$ZERO_TOK")
|
out_zerotok=$(run "$ZERO_TOK")
|
||||||
! echo "$out_zerotok" | grep -q "tokens" && pass "omits token segment when total is zero" || fail "token segment shown unexpectedly — got: $out_zerotok"
|
! grep -q "tokens" <<< "$out_zerotok" && pass "omits token segment when total is zero" || fail "token segment shown unexpectedly — got: $out_zerotok"
|
||||||
|
|
||||||
echo ""
|
echo ""
|
||||||
echo "--- executable bit ---"
|
echo "--- executable bit ---"
|
||||||
|
|||||||
@@ -246,12 +246,12 @@ EOF
|
|||||||
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5"' EXIT
|
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5"' EXIT
|
||||||
OUT5=$(run_wrap "$FIXTURE5" --config "$VALE_CONFIG" \
|
OUT5=$(run_wrap "$FIXTURE5" --config "$VALE_CONFIG" \
|
||||||
plugins/testplugin/skills/zzzskill/SKILL.md)
|
plugins/testplugin/skills/zzzskill/SKILL.md)
|
||||||
if echo "$OUT5" | grep -q "VagueWording"; then
|
if grep -q "VagueWording" <<< "$OUT5"; then
|
||||||
pass "flags vague wording when the description contains an apostrophe"
|
pass "flags vague wording when the description contains an apostrophe"
|
||||||
else
|
else
|
||||||
fail "silently missed vague wording in a description containing an apostrophe"
|
fail "silently missed vague wording in a description containing an apostrophe"
|
||||||
fi
|
fi
|
||||||
if echo "$OUT5" | grep -qi "yaml:"; then
|
if grep -qi "yaml:" <<< "$OUT5"; then
|
||||||
fail "flattened copy with an apostrophe produced a YAML parse error"
|
fail "flattened copy with an apostrophe produced a YAML parse error"
|
||||||
else
|
else
|
||||||
pass "flattened copy with an apostrophe is valid YAML (no parse error)"
|
pass "flattened copy with an apostrophe is valid YAML (no parse error)"
|
||||||
@@ -297,9 +297,9 @@ EOF
|
|||||||
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7"' EXIT
|
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7"' EXIT
|
||||||
OUT7=$(run_wrap "$FIXTURE7" --config "$VALE_CONFIG" \
|
OUT7=$(run_wrap "$FIXTURE7" --config "$VALE_CONFIG" \
|
||||||
plugins/testplugin/skills/zzzskill/SKILL.md)
|
plugins/testplugin/skills/zzzskill/SKILL.md)
|
||||||
if echo "$OUT7" | grep -q "Traceback"; then
|
if grep -q "Traceback" <<< "$OUT7"; then
|
||||||
fail "crashed while flattening a description with a blank line between paragraphs"
|
fail "crashed while flattening a description with a blank line between paragraphs"
|
||||||
elif echo "$OUT7" | grep -q "VagueWording"; then
|
elif grep -q "VagueWording" <<< "$OUT7"; then
|
||||||
pass "still flags vague wording in the second paragraph after a blank line"
|
pass "still flags vague wording in the second paragraph after a blank line"
|
||||||
else
|
else
|
||||||
fail "silently missed vague wording in the second paragraph after a blank line — the bug this test guards against"
|
fail "silently missed vague wording in the second paragraph after a blank line — the bug this test guards against"
|
||||||
@@ -337,7 +337,7 @@ REL_CFG8="../../../../.vale.ini"
|
|||||||
REL_FILE8="SKILL.md"
|
REL_FILE8="SKILL.md"
|
||||||
OUT_EQ=$(run_wrap "$SUBDIR8" "--config=$REL_CFG8" "$REL_FILE8")
|
OUT_EQ=$(run_wrap "$SUBDIR8" "--config=$REL_CFG8" "$REL_FILE8")
|
||||||
OUT_TWO=$(run_wrap "$SUBDIR8" --config "$REL_CFG8" "$REL_FILE8")
|
OUT_TWO=$(run_wrap "$SUBDIR8" --config "$REL_CFG8" "$REL_FILE8")
|
||||||
if echo "$OUT_EQ" | grep -q "VagueWording" && [[ "$OUT_EQ" == "$OUT_TWO" ]]; then
|
if grep -q "VagueWording" <<< "$OUT_EQ" && [[ "$OUT_EQ" == "$OUT_TWO" ]]; then
|
||||||
pass "cwd-relative --config resolves from a subdirectory in both argv forms"
|
pass "cwd-relative --config resolves from a subdirectory in both argv forms"
|
||||||
else
|
else
|
||||||
fail "cwd-relative --config did not resolve from a subdirectory (equals form vs two-argv form)"
|
fail "cwd-relative --config did not resolve from a subdirectory (equals form vs two-argv form)"
|
||||||
@@ -355,7 +355,7 @@ RC_REL_CFG=$?
|
|||||||
OUT_ABS_CFG=$(cd "$SUBDIR8" && bash "$SCRIPT" --config "$FIXTURE8/.vale.ini" "$REL_FILE8" 2>&1)
|
OUT_ABS_CFG=$(cd "$SUBDIR8" && bash "$SCRIPT" --config "$FIXTURE8/.vale.ini" "$REL_FILE8" 2>&1)
|
||||||
RC_ABS_CFG=$?
|
RC_ABS_CFG=$?
|
||||||
set -e
|
set -e
|
||||||
if echo "$OUT_REL_CFG" | grep -qi "does not exist"; then
|
if grep -qi "does not exist" <<< "$OUT_REL_CFG"; then
|
||||||
fail "cwd-relative --config hard-errored ('does not exist') — the bug this test guards against"
|
fail "cwd-relative --config hard-errored ('does not exist') — the bug this test guards against"
|
||||||
elif [[ "$OUT_REL_CFG" == "$OUT_ABS_CFG" && "$RC_REL_CFG" -eq "$RC_ABS_CFG" ]]; then
|
elif [[ "$OUT_REL_CFG" == "$OUT_ABS_CFG" && "$RC_REL_CFG" -eq "$RC_ABS_CFG" ]]; then
|
||||||
pass "cwd-relative --config matches the absolute-path invocation (output and exit code)"
|
pass "cwd-relative --config matches the absolute-path invocation (output and exit code)"
|
||||||
@@ -373,9 +373,9 @@ echo ""
|
|||||||
echo "--- flattens a cwd-relative file argument passed from a subdirectory ---"
|
echo "--- flattens a cwd-relative file argument passed from a subdirectory ---"
|
||||||
WRAPPED_REL=$(run_wrap "$SUBDIR8" --config "$FIXTURE8/.vale.ini" "$REL_FILE8")
|
WRAPPED_REL=$(run_wrap "$SUBDIR8" --config "$FIXTURE8/.vale.ini" "$REL_FILE8")
|
||||||
BARE_REL=$(cd "$SUBDIR8" && vale --config "$FIXTURE8/.vale.ini" "$REL_FILE8" 2>&1 || true)
|
BARE_REL=$(cd "$SUBDIR8" && vale --config "$FIXTURE8/.vale.ini" "$REL_FILE8" 2>&1 || true)
|
||||||
if ! echo "$WRAPPED_REL" | grep -q "VagueWording"; then
|
if ! grep -q "VagueWording" <<< "$WRAPPED_REL"; then
|
||||||
fail "cwd-relative file argument produced no alert — flattening was silently skipped, the bug this test guards against"
|
fail "cwd-relative file argument produced no alert — flattening was silently skipped, the bug this test guards against"
|
||||||
elif echo "$BARE_REL" | grep -q "VagueWording"; then
|
elif grep -q "VagueWording" <<< "$BARE_REL"; then
|
||||||
fail "bare vale already flags this fixture, so the test can't detect a silently-skipped flattening"
|
fail "bare vale already flags this fixture, so the test can't detect a silently-skipped flattening"
|
||||||
else
|
else
|
||||||
pass "cwd-relative file argument is flattened and flagged where bare vale reports nothing"
|
pass "cwd-relative file argument is flattened and flagged where bare vale reports nothing"
|
||||||
@@ -443,7 +443,7 @@ trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTU
|
|||||||
REL11="plugins/testplugin/skills/zzzskill/SKILL.md"
|
REL11="plugins/testplugin/skills/zzzskill/SKILL.md"
|
||||||
WRAPPED_OUT=$(run_wrap "$FIXTURE11" --config "$VALE_CONFIG" "$REL11")
|
WRAPPED_OUT=$(run_wrap "$FIXTURE11" --config "$VALE_CONFIG" "$REL11")
|
||||||
BARE_OUT=$(cd "$FIXTURE11" && vale --config "$VALE_CONFIG" "$REL11" 2>&1 || true)
|
BARE_OUT=$(cd "$FIXTURE11" && vale --config "$VALE_CONFIG" "$REL11" 2>&1 || true)
|
||||||
if ! echo "$BARE_OUT" | grep -q "VagueWording"; then
|
if ! grep -q "VagueWording" <<< "$BARE_OUT"; then
|
||||||
fail "bare vale reports nothing for a literal (|) block scalar — the 'literal blocks are not broken' premise is wrong"
|
fail "bare vale reports nothing for a literal (|) block scalar — the 'literal blocks are not broken' premise is wrong"
|
||||||
elif [[ "$WRAPPED_OUT" == "$BARE_OUT" ]]; then
|
elif [[ "$WRAPPED_OUT" == "$BARE_OUT" ]]; then
|
||||||
pass "literal (|) block scalar output matches bare vale exactly — untouched by flattening"
|
pass "literal (|) block scalar output matches bare vale exactly — untouched by flattening"
|
||||||
@@ -461,7 +461,7 @@ echo "--- defaults --config to the wrapper's own sibling assets/vale/.vale.ini -
|
|||||||
FIXTURE12="$(make_fixture 2)"
|
FIXTURE12="$(make_fixture 2)"
|
||||||
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7" "$FIXTURE8" "$FIXTURE10" "$FIXTURE11" "$FIXTURE12"' EXIT
|
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7" "$FIXTURE8" "$FIXTURE10" "$FIXTURE11" "$FIXTURE12"' EXIT
|
||||||
OUT12=$(run_wrap "$FIXTURE12" plugins/testplugin/skills/zzzskill/SKILL.md)
|
OUT12=$(run_wrap "$FIXTURE12" plugins/testplugin/skills/zzzskill/SKILL.md)
|
||||||
if echo "$OUT12" | grep -q "VagueWording"; then
|
if grep -q "VagueWording" <<< "$OUT12"; then
|
||||||
pass "a --config-less invocation uses the wrapper's bundled config"
|
pass "a --config-less invocation uses the wrapper's bundled config"
|
||||||
else
|
else
|
||||||
fail "a --config-less invocation found no config — external pre-commit consumers get E100, the bug this test guards against"
|
fail "a --config-less invocation found no config — external pre-commit consumers get E100, the bug this test guards against"
|
||||||
@@ -490,9 +490,9 @@ EOF
|
|||||||
chmod +x "$STUB13/realpath"
|
chmod +x "$STUB13/realpath"
|
||||||
OUT13=$(cd "$FIXTURE12" && PATH="$STUB13:$PATH" bash "$SCRIPT" --config "$VALE_CONFIG" \
|
OUT13=$(cd "$FIXTURE12" && PATH="$STUB13:$PATH" bash "$SCRIPT" --config "$VALE_CONFIG" \
|
||||||
plugins/testplugin/skills/zzzskill/SKILL.md 2>&1 || true)
|
plugins/testplugin/skills/zzzskill/SKILL.md 2>&1 || true)
|
||||||
if echo "$OUT13" | grep -q "illegal option"; then
|
if grep -q "illegal option" <<< "$OUT13"; then
|
||||||
fail "invoked realpath -m — fails on macOS's BSD realpath, the bug this test guards against"
|
fail "invoked realpath -m — fails on macOS's BSD realpath, the bug this test guards against"
|
||||||
elif echo "$OUT13" | grep -q "VagueWording"; then
|
elif grep -q "VagueWording" <<< "$OUT13"; then
|
||||||
pass "flattens and flags with no GNU realpath available"
|
pass "flattens and flags with no GNU realpath available"
|
||||||
else
|
else
|
||||||
fail "produced no alert under a BSD-style realpath: $OUT13"
|
fail "produced no alert under a BSD-style realpath: $OUT13"
|
||||||
@@ -509,9 +509,9 @@ FIXTURE14="$(make_fixture 2)"
|
|||||||
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7" "$FIXTURE8" "$FIXTURE10" "$FIXTURE11" "$FIXTURE12" "$STUB13" "$FIXTURE14"' EXIT
|
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7" "$FIXTURE8" "$FIXTURE10" "$FIXTURE11" "$FIXTURE12" "$STUB13" "$FIXTURE14"' EXIT
|
||||||
WRAPPED_DIR=$(run_wrap "$FIXTURE14" --config "$VALE_CONFIG" plugins)
|
WRAPPED_DIR=$(run_wrap "$FIXTURE14" --config "$VALE_CONFIG" plugins)
|
||||||
BARE_DIR=$(cd "$FIXTURE14" && vale --config "$VALE_CONFIG" plugins 2>&1 || true)
|
BARE_DIR=$(cd "$FIXTURE14" && vale --config "$VALE_CONFIG" plugins 2>&1 || true)
|
||||||
if ! echo "$WRAPPED_DIR" | grep -q "VagueWording"; then
|
if ! grep -q "VagueWording" <<< "$WRAPPED_DIR"; then
|
||||||
fail "a directory argument produced no alert — flattening was silently skipped, the bug this test guards against"
|
fail "a directory argument produced no alert — flattening was silently skipped, the bug this test guards against"
|
||||||
elif echo "$BARE_DIR" | grep -q "VagueWording"; then
|
elif grep -q "VagueWording" <<< "$BARE_DIR"; then
|
||||||
fail "bare vale already flags this fixture, so the test can't detect a silently-skipped flattening"
|
fail "bare vale already flags this fixture, so the test can't detect a silently-skipped flattening"
|
||||||
else
|
else
|
||||||
pass "a directory argument is walked and its files flattened"
|
pass "a directory argument is walked and its files flattened"
|
||||||
@@ -525,7 +525,7 @@ mkdir -p "$SPACED15"
|
|||||||
cp "$FIXTURE14/plugins/testplugin/skills/zzzskill/SKILL.md" "$SPACED15/SKILL.md"
|
cp "$FIXTURE14/plugins/testplugin/skills/zzzskill/SKILL.md" "$SPACED15/SKILL.md"
|
||||||
rm -rf "$FIXTURE14/plugins/testplugin/skills/zzzskill"
|
rm -rf "$FIXTURE14/plugins/testplugin/skills/zzzskill"
|
||||||
OUT15=$(run_wrap "$FIXTURE14" --config "$VALE_CONFIG" plugins/testplugin/skills)
|
OUT15=$(run_wrap "$FIXTURE14" --config "$VALE_CONFIG" plugins/testplugin/skills)
|
||||||
if echo "$OUT15" | grep -q "zzz skill" && echo "$OUT15" | grep -q "VagueWording"; then
|
if grep -q "zzz skill" <<< "$OUT15" && grep -q "VagueWording" <<< "$OUT15"; then
|
||||||
pass "a file under a directory whose name contains a space is walked and flattened"
|
pass "a file under a directory whose name contains a space is walked and flattened"
|
||||||
else
|
else
|
||||||
fail "a path with a space was dropped from the directory walk"
|
fail "a path with a space was dropped from the directory walk"
|
||||||
@@ -626,7 +626,7 @@ unguarded_expansions() {
|
|||||||
done < <(sourced_files "$file")
|
done < <(sourced_files "$file")
|
||||||
while IFS= read -r hit; do
|
while IFS= read -r hit; do
|
||||||
name="$(printf '%s\n' "$hit" \
|
name="$(printf '%s\n' "$hit" \
|
||||||
| grep -oE '\$\{[A-Za-z_][A-Za-z0-9_]*\[@\]\}' | head -1 \
|
| { grep -oE '\$\{[A-Za-z_][A-Za-z0-9_]*\[@\]\}' || true; } | head -1 \
|
||||||
| sed -E 's/^\$\{//; s/\[@\]\}$//')"
|
| sed -E 's/^\$\{//; s/\[@\]\}$//')"
|
||||||
# Shell-maintained arrays are never seeded by a `NAME=(...)` line, so the
|
# Shell-maintained arrays are never seeded by a `NAME=(...)` line, so the
|
||||||
# seeding exemption below can never clear them: without this case they are
|
# seeding exemption below can never clear them: without this case they are
|
||||||
@@ -653,8 +653,7 @@ unguarded_expansions() {
|
|||||||
PIPESTATUS|BASH_SOURCE|BASH_LINENO|BASH_VERSINFO|GROUPS|DIRSTACK) continue ;;
|
PIPESTATUS|BASH_SOURCE|BASH_LINENO|BASH_VERSINFO|GROUPS|DIRSTACK) continue ;;
|
||||||
esac
|
esac
|
||||||
# Same-line emptiness short-circuit: the expansion cannot be reached empty.
|
# Same-line emptiness short-circuit: the expansion cannot be reached empty.
|
||||||
if printf '%s\n' "$hit" \
|
if grep -qE "\\\$\{#$name\[@\]\}[[:space:]]*-(eq|lt)[[:space:]]*[01][^|]*\|\|" <<< "$hit"; then
|
||||||
| grep -qE "\\\$\{#$name\[@\]\}[[:space:]]*-(eq|lt)[[:space:]]*[01][^|]*\|\|"; then
|
|
||||||
continue
|
continue
|
||||||
fi
|
fi
|
||||||
for seed_file in ${seed_files[@]+"${seed_files[@]}"}; do
|
for seed_file in ${seed_files[@]+"${seed_files[@]}"}; do
|
||||||
@@ -809,7 +808,7 @@ for ARGS17 in "" "--config $VALE_CONFIG" "--config $VALE_CONFIG emptydir"; do
|
|||||||
# shellcheck disable=SC2086 # deliberate word splitting of the argv fixture
|
# shellcheck disable=SC2086 # deliberate word splitting of the argv fixture
|
||||||
OUT17+="$( (cd "$FIXTURE17" && "$OLD_BASH" "$SCRIPT" $ARGS17 </dev/null 2>&1) || true)"
|
OUT17+="$( (cd "$FIXTURE17" && "$OLD_BASH" "$SCRIPT" $ARGS17 </dev/null 2>&1) || true)"
|
||||||
done
|
done
|
||||||
if echo "$OUT17" | grep -q "unbound variable"; then
|
if grep -q "unbound variable" <<< "$OUT17"; then
|
||||||
fail "aborted with 'unbound variable' on bash $OLD_BASH_VER — the bug this test guards against"
|
fail "aborted with 'unbound variable' on bash $OLD_BASH_VER — the bug this test guards against"
|
||||||
else
|
else
|
||||||
pass "degenerate invocations run clean under bash $OLD_BASH_VER ($OLD_BASH)"
|
pass "degenerate invocations run clean under bash $OLD_BASH_VER ($OLD_BASH)"
|
||||||
@@ -828,7 +827,7 @@ SPACED18="$FIXTURE18/plugins/testplugin/skills/zzz skill dir"
|
|||||||
mkdir -p "$SPACED18"
|
mkdir -p "$SPACED18"
|
||||||
mv "$FIXTURE18/plugins/testplugin/skills/zzzskill/SKILL.md" "$SPACED18/SKILL.md"
|
mv "$FIXTURE18/plugins/testplugin/skills/zzzskill/SKILL.md" "$SPACED18/SKILL.md"
|
||||||
OUT18=$(run_wrap "$FIXTURE18" --config "$VALE_CONFIG" "plugins/testplugin/skills/zzz skill dir/SKILL.md")
|
OUT18=$(run_wrap "$FIXTURE18" --config "$VALE_CONFIG" "plugins/testplugin/skills/zzz skill dir/SKILL.md")
|
||||||
if echo "$OUT18" | grep -q "zzz skill dir/SKILL.md" && echo "$OUT18" | grep -q "VagueWording"; then
|
if grep -q "zzz skill dir/SKILL.md" <<< "$OUT18" && grep -q "VagueWording" <<< "$OUT18"; then
|
||||||
pass "a path argument with a space is passed to vale as one word"
|
pass "a path argument with a space is passed to vale as one word"
|
||||||
else
|
else
|
||||||
fail "a path argument with a space was split by the array expansion: $OUT18"
|
fail "a path argument with a space was split by the array expansion: $OUT18"
|
||||||
@@ -934,7 +933,7 @@ else
|
|||||||
new_fixture "$DIR19"
|
new_fixture "$DIR19"
|
||||||
BARE19="$(cd "$DIR19" && vale --config "$VALE_CONFIG" "$REL_SKILL19" 2>&1 || true)"
|
BARE19="$(cd "$DIR19" && vale --config "$VALE_CONFIG" "$REL_SKILL19" 2>&1 || true)"
|
||||||
GOT19="$(alert_text "$(run_wrap "$DIR19" --config "$VALE_CONFIG" "$REL_SKILL19")")"
|
GOT19="$(alert_text "$(run_wrap "$DIR19" --config "$VALE_CONFIG" "$REL_SKILL19")")"
|
||||||
if echo "$BARE19" | grep -q "VagueWording"; then
|
if grep -q "VagueWording" <<< "$BARE19"; then
|
||||||
fail "bare vale already flags the $FORM19 form, so this case can't detect a silently-skipped flattening"
|
fail "bare vale already flags the $FORM19 form, so this case can't detect a silently-skipped flattening"
|
||||||
elif [[ "$GOT19" == "$BASELINE19" ]]; then
|
elif [[ "$GOT19" == "$BASELINE19" ]]; then
|
||||||
pass "a $FORM19 multi-line description reports the same alerts as its single-line form"
|
pass "a $FORM19 multi-line description reports the same alerts as its single-line form"
|
||||||
@@ -1044,7 +1043,7 @@ done
|
|||||||
echo ""
|
echo ""
|
||||||
echo "--- the |- literal-block fallback lints normally and preserves line numbers ---"
|
echo "--- the |- literal-block fallback lints normally and preserves line numbers ---"
|
||||||
OUT20B=$(run_wrap "$FIXTURE20_BLOCK" --config "$VALE_CONFIG" "$REL_SKILL19")
|
OUT20B=$(run_wrap "$FIXTURE20_BLOCK" --config "$VALE_CONFIG" "$REL_SKILL19")
|
||||||
if echo "$OUT20B" | grep -q "VagueWording"; then
|
if grep -q "VagueWording" <<< "$OUT20B"; then
|
||||||
pass "a description needing quotes with both an apostrophe and a double quote is still linted"
|
pass "a description needing quotes with both an apostrophe and a double quote is still linted"
|
||||||
else
|
else
|
||||||
fail "a description needing quotes with both an apostrophe and a double quote produced no alerts"
|
fail "a description needing quotes with both an apostrophe and a double quote produced no alerts"
|
||||||
@@ -1053,7 +1052,7 @@ WANT20B_LINE="$(grep -n 'flattening marker phrase' "$FIXTURE20_BLOCK/$REL_SKILL1
|
|||||||
# `--output line` prints `file:line:col:Rule:message`, so the line number reads
|
# `--output line` prints `file:line:col:Rule:message`, so the line number reads
|
||||||
# back without any wrapping or colour to strip.
|
# back without any wrapping or colour to strip.
|
||||||
GOT20B_LINE="$(run_wrap "$FIXTURE20_BLOCK" --config "$APOS_STYLE/.vale.ini" --output line "$REL_SKILL19" \
|
GOT20B_LINE="$(run_wrap "$FIXTURE20_BLOCK" --config "$APOS_STYLE/.vale.ini" --output line "$REL_SKILL19" \
|
||||||
| grep 'Apostrophe.Body' | head -1 | cut -d: -f2)"
|
| { grep 'Apostrophe.Body' || true; } | head -1 | cut -d: -f2)"
|
||||||
if [[ "$GOT20B_LINE" == "$WANT20B_LINE" ]]; then
|
if [[ "$GOT20B_LINE" == "$WANT20B_LINE" ]]; then
|
||||||
pass "a body line after a |- flattened description keeps its original line number ($WANT20B_LINE)"
|
pass "a body line after a |- flattened description keeps its original line number ($WANT20B_LINE)"
|
||||||
else
|
else
|
||||||
@@ -1079,7 +1078,10 @@ if [[ "$BARE21_FILES" != "in 1 file" ]]; then
|
|||||||
fail "bare vale did not lint the symlinked file ($BARE21_FILES), so this case can't detect the walk dropping it"
|
fail "bare vale did not lint the symlinked file ($BARE21_FILES), so this case can't detect the walk dropping it"
|
||||||
elif [[ "$WRAPPED21_FILES" != "$BARE21_FILES" ]]; then
|
elif [[ "$WRAPPED21_FILES" != "$BARE21_FILES" ]]; then
|
||||||
fail "the directory walk dropped a symlinked file: wrapper saw '$WRAPPED21_FILES', bare vale '$BARE21_FILES'"
|
fail "the directory walk dropped a symlinked file: wrapper saw '$WRAPPED21_FILES', bare vale '$BARE21_FILES'"
|
||||||
elif echo "$WRAPPED21" | grep -q "VagueWording"; then
|
# A here-string, not `echo "$WRAPPED21" | grep -q`: the match sits on line 3 of
|
||||||
|
# 8, and under pipefail grep -q exiting early can SIGPIPE echo mid-write and
|
||||||
|
# fail this branch on correct output (see tests/test-check-release-needed.sh).
|
||||||
|
elif grep -q "VagueWording" <<< "$WRAPPED21"; then
|
||||||
pass "a symlinked file under a directory argument is mirrored, flattened and flagged"
|
pass "a symlinked file under a directory argument is mirrored, flattened and flagged"
|
||||||
else
|
else
|
||||||
fail "a symlinked file was mirrored but not flattened — no alert came back"
|
fail "a symlinked file was mirrored but not flattened — no alert came back"
|
||||||
@@ -1117,9 +1119,9 @@ RC23=$?
|
|||||||
set -e
|
set -e
|
||||||
if [[ $RC23 -eq 0 ]]; then
|
if [[ $RC23 -eq 0 ]]; then
|
||||||
fail "a typo'd path exited 0 — indistinguishable from a clean run, the bug this test guards against"
|
fail "a typo'd path exited 0 — indistinguishable from a clean run, the bug this test guards against"
|
||||||
elif echo "$OUT23" | grep -q "in stdin"; then
|
elif grep -q "in stdin" <<< "$OUT23"; then
|
||||||
fail "a typo'd path fell back to reading stdin and reported 'in stdin' instead of erroring"
|
fail "a typo'd path fell back to reading stdin and reported 'in stdin' instead of erroring"
|
||||||
elif echo "$OUT23" | grep -q "SKILLL.md"; then
|
elif grep -q "SKILLL.md" <<< "$OUT23"; then
|
||||||
pass "a typo'd path exits nonzero with a message naming the path"
|
pass "a typo'd path exits nonzero with a message naming the path"
|
||||||
else
|
else
|
||||||
fail "a typo'd path exited $RC23 but the message does not name it: $OUT23"
|
fail "a typo'd path exited $RC23 but the message does not name it: $OUT23"
|
||||||
@@ -1141,9 +1143,9 @@ mkdir -p "$FIXTURE24/line"
|
|||||||
for FORM24 in "--output line" "--output=line" "--output JSON" "--output=JSON"; do
|
for FORM24 in "--output line" "--output=line" "--output JSON" "--output=JSON"; do
|
||||||
# shellcheck disable=SC2086 # deliberate word splitting of the argv fixture
|
# shellcheck disable=SC2086 # deliberate word splitting of the argv fixture
|
||||||
OUT24="$(run_wrap "$FIXTURE24" --config "$VALE_CONFIG" $FORM24 "$REL_SKILL19")"
|
OUT24="$(run_wrap "$FIXTURE24" --config "$VALE_CONFIG" $FORM24 "$REL_SKILL19")"
|
||||||
if echo "$OUT24" | grep -q "E100"; then
|
if grep -q "E100" <<< "$OUT24"; then
|
||||||
fail "'$FORM24' was rewritten to a cwd path and vale flipped into template mode — the bug this test guards against"
|
fail "'$FORM24' was rewritten to a cwd path and vale flipped into template mode — the bug this test guards against"
|
||||||
elif echo "$OUT24" | grep -q "VagueWording"; then
|
elif grep -q "VagueWording" <<< "$OUT24"; then
|
||||||
pass "'$FORM24' is passed through as a built-in style name"
|
pass "'$FORM24' is passed through as a built-in style name"
|
||||||
else
|
else
|
||||||
fail "'$FORM24' produced no alert: $OUT24"
|
fail "'$FORM24' produced no alert: $OUT24"
|
||||||
@@ -1215,12 +1217,12 @@ FOUND26="$(unguarded_expansions "$FIXTURE26/probe.sh")"
|
|||||||
MISSING26=""
|
MISSING26=""
|
||||||
LEAKED26=""
|
LEAKED26=""
|
||||||
for ARR26 in $EXEMPT26; do
|
for ARR26 in $EXEMPT26; do
|
||||||
if echo "$FOUND26" | grep -q "{$ARR26\[@\]}"; then
|
if grep -q "{$ARR26\[@\]}" <<< "$FOUND26"; then
|
||||||
LEAKED26+="$ARR26 "
|
LEAKED26+="$ARR26 "
|
||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
for ARR26 in $FLAGGED26; do
|
for ARR26 in $FLAGGED26; do
|
||||||
if ! echo "$FOUND26" | grep -q "{$ARR26\[@\]}"; then
|
if ! grep -q "{$ARR26\[@\]}" <<< "$FOUND26"; then
|
||||||
MISSING26+="$ARR26 "
|
MISSING26+="$ARR26 "
|
||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
@@ -1465,7 +1467,7 @@ matches_any_regex28() {
|
|||||||
[[ -n "$regexes" ]] || return 1
|
[[ -n "$regexes" ]] || return 1
|
||||||
while IFS= read -r re; do
|
while IFS= read -r re; do
|
||||||
[[ -n "$re" ]] || continue
|
[[ -n "$re" ]] || continue
|
||||||
if printf '%s\n' "$rel" | grep -Eq "$re"; then
|
if grep -Eq "$re" <<< "$rel"; then
|
||||||
return 0
|
return 0
|
||||||
fi
|
fi
|
||||||
done <<EOF_RE28
|
done <<EOF_RE28
|
||||||
@@ -1499,7 +1501,7 @@ elif [[ -z "$SECTIONS28" ]]; then
|
|||||||
else
|
else
|
||||||
while IFS= read -r SEC28; do
|
while IFS= read -r SEC28; do
|
||||||
[[ -n "$SEC28" ]] || continue
|
[[ -n "$SEC28" ]] || continue
|
||||||
if ! printf '%s\n' "$PROBE_TABLE28" | grep -qF "|$SEC28|isolating"; then
|
if ! grep -qF "|$SEC28|isolating" <<< "$PROBE_TABLE28"; then
|
||||||
UNPROBED28+="$SEC28 "
|
UNPROBED28+="$SEC28 "
|
||||||
fi
|
fi
|
||||||
done <<EOF_SEC28
|
done <<EOF_SEC28
|
||||||
@@ -1508,7 +1510,7 @@ EOF_SEC28
|
|||||||
STALE_SECTIONS28=""
|
STALE_SECTIONS28=""
|
||||||
while IFS= read -r TSEC28; do
|
while IFS= read -r TSEC28; do
|
||||||
[[ -n "$TSEC28" ]] || continue
|
[[ -n "$TSEC28" ]] || continue
|
||||||
printf '%s\n' "$SECTIONS28" | grep -qF -- "$TSEC28" || STALE_SECTIONS28+="$TSEC28 "
|
grep -qF -- "$TSEC28" <<< "$SECTIONS28" || STALE_SECTIONS28+="$TSEC28 "
|
||||||
done <<EOF_TSEC28
|
done <<EOF_TSEC28
|
||||||
$TABLE_SECTIONS28
|
$TABLE_SECTIONS28
|
||||||
EOF_TSEC28
|
EOF_TSEC28
|
||||||
@@ -1548,7 +1550,7 @@ probe_coverage28() {
|
|||||||
echo "FAIL|$rel|vale printed no summary line for $rel, so it is not known whether anything was scanned: ${report:-<empty>}"
|
echo "FAIL|$rel|vale printed no summary line for $rel, so it is not known whether anything was scanned: ${report:-<empty>}"
|
||||||
elif [[ "$count" -eq 0 ]]; then
|
elif [[ "$count" -eq 0 ]]; then
|
||||||
echo "FAIL|$rel|$sec scanned 0 files for $rel — the section's glob covers no path of that shape, so vale exits 0 and every gate reads it as a pass"
|
echo "FAIL|$rel|$sec scanned 0 files for $rel — the section's glob covers no path of that shape, so vale exits 0 and every gate reads it as a pass"
|
||||||
elif ! printf '%s\n' "$report" | grep -qF "Kyberforge.VagueWording"; then
|
elif ! grep -qF "Kyberforge.VagueWording" <<< "$report"; then
|
||||||
echo "FAIL|$rel|$sec scanned $rel but raised no Kyberforge alert — the glob matches but the style is not loaded, which lints the file and reports nothing"
|
echo "FAIL|$rel|$sec scanned $rel but raised no Kyberforge alert — the glob matches but the style is not loaded, which lints the file and reports nothing"
|
||||||
else
|
else
|
||||||
echo "PASS|$rel|$sec scans $rel ($count file) and raises a Kyberforge alert"
|
echo "PASS|$rel|$sec scans $rel ($count file) and raises a Kyberforge alert"
|
||||||
@@ -1622,7 +1624,7 @@ if [[ "$VALE_READY" == true && "$PART_A_CLEAN28" == true ]]; then
|
|||||||
[[ -n "$REL28" && "$ISO28" == "isolating" ]] || continue
|
[[ -n "$REL28" && "$ISO28" == "isolating" ]] || continue
|
||||||
LINE28="$(printf '%s\n' "$RESULTS28" | { grep -F "|$REL28|" || true; } | head -1)"
|
LINE28="$(printf '%s\n' "$RESULTS28" | { grep -F "|$REL28|" || true; } | head -1)"
|
||||||
if [[ "$ROWSEC28" == "$MSEC28" ]]; then
|
if [[ "$ROWSEC28" == "$MSEC28" ]]; then
|
||||||
printf '%s\n' "$LINE28" | grep -qF "but raised no Kyberforge alert" \
|
grep -qF "but raised no Kyberforge alert" <<< "$LINE28" \
|
||||||
|| STYLE_MUT_FAILS28+="[$MSEC28 lost Kyberforge but $REL28 did not fail as style-not-loaded: ${LINE28:-<no result>}] "
|
|| STYLE_MUT_FAILS28+="[$MSEC28 lost Kyberforge but $REL28 did not fail as style-not-loaded: ${LINE28:-<no result>}] "
|
||||||
else
|
else
|
||||||
[[ "$LINE28" == PASS\|* ]] \
|
[[ "$LINE28" == PASS\|* ]] \
|
||||||
@@ -1719,7 +1721,7 @@ copilot_scope_failures30() {
|
|||||||
[[ -n "$rel" ]] || continue
|
[[ -n "$rel" ]] || continue
|
||||||
report="$(vale_report28 "$cfg" "$TREE28" "$rel")"
|
report="$(vale_report28 "$cfg" "$TREE28" "$rel")"
|
||||||
has=false
|
has=false
|
||||||
if printf '%s\n' "$report" | grep -qF "KyberforgeCopilot.ProactivePhrase"; then has=true; fi
|
if grep -qF "KyberforgeCopilot.ProactivePhrase" <<< "$report"; then has=true; fi
|
||||||
case "$rel" in
|
case "$rel" in
|
||||||
*.agent.md)
|
*.agent.md)
|
||||||
[[ "$has" == true ]] || bad+="[$rel is an agent file but raised no ProactivePhrase alert — the Copilot style is shipped but never loaded for it, so its rules lint nothing] "
|
[[ "$has" == true ]] || bad+="[$rel is an agent file but raised no ProactivePhrase alert — the Copilot style is shipped but never loaded for it, so its rules lint nothing] "
|
||||||
@@ -1728,7 +1730,7 @@ copilot_scope_failures30() {
|
|||||||
# Non-vacuity guard: `absent` only means `scoped out` if the file was
|
# Non-vacuity guard: `absent` only means `scoped out` if the file was
|
||||||
# scanned at all. Without the Kyberforge half, a glob that stopped
|
# scanned at all. Without the Kyberforge half, a glob that stopped
|
||||||
# matching this path entirely would read as correct scoping.
|
# matching this path entirely would read as correct scoping.
|
||||||
if ! printf '%s\n' "$report" | grep -qF "Kyberforge.VagueWording"; then
|
if ! grep -qF "Kyberforge.VagueWording" <<< "$report"; then
|
||||||
bad+="[$rel raised no Kyberforge alert either, so its missing ProactivePhrase proves nothing about scoping] "
|
bad+="[$rel raised no Kyberforge alert either, so its missing ProactivePhrase proves nothing about scoping] "
|
||||||
elif [[ "$has" == true ]]; then
|
elif [[ "$has" == true ]]; then
|
||||||
bad+="[$rel is not an .agent.md file but raised a ProactivePhrase alert — the Copilot style has leaked past the scope ADR-0013 fixes it to] "
|
bad+="[$rel is not an .agent.md file but raised a ProactivePhrase alert — the Copilot style has leaked past the scope ADR-0013 fixes it to] "
|
||||||
@@ -1789,9 +1791,9 @@ UNLOAD_FAILS30="$(copilot_scope_failures30 "$UNLOAD30/.vale.ini")"
|
|||||||
LEAK_FAILS30="$(copilot_scope_failures30 "$LEAK30/.vale.ini")"
|
LEAK_FAILS30="$(copilot_scope_failures30 "$LEAK30/.vale.ini")"
|
||||||
if cmp -s "$VALE_ASSETS28/.vale.ini" "$UNLOAD30/.vale.ini" || cmp -s "$VALE_ASSETS28/.vale.ini" "$LEAK30/.vale.ini"; then
|
if cmp -s "$VALE_ASSETS28/.vale.ini" "$UNLOAD30/.vale.ini" || cmp -s "$VALE_ASSETS28/.vale.ini" "$LEAK30/.vale.ini"; then
|
||||||
fail "a Copilot-scope mutation left the copied config unchanged, so Part B mutated nothing and proves nothing about Part A"
|
fail "a Copilot-scope mutation left the copied config unchanged, so Part B mutated nothing and proves nothing about Part A"
|
||||||
elif ! printf '%s' "$UNLOAD_FAILS30" | grep -qF "[copilot/demo.agent.md is an agent file but raised no ProactivePhrase alert"; then
|
elif ! grep -qF "[copilot/demo.agent.md is an agent file but raised no ProactivePhrase alert" <<< "$UNLOAD_FAILS30"; then
|
||||||
fail "dropping KyberforgeCopilot from [**/*.agent.md] did not fail Part A, so an unloaded Copilot style would pass silently again: ${UNLOAD_FAILS30:-<no failure>}"
|
fail "dropping KyberforgeCopilot from [**/*.agent.md] did not fail Part A, so an unloaded Copilot style would pass silently again: ${UNLOAD_FAILS30:-<no failure>}"
|
||||||
elif ! printf '%s' "$LEAK_FAILS30" | grep -qF "[plugins/demo/.apm/skills/demo/SKILL.md is not an .agent.md file but raised a ProactivePhrase alert"; then
|
elif ! grep -qF "[plugins/demo/.apm/skills/demo/SKILL.md is not an .agent.md file but raised a ProactivePhrase alert" <<< "$LEAK_FAILS30"; then
|
||||||
fail "adding KyberforgeCopilot to [**/SKILL.md] did not fail Part A, so the style could leak past ADR-0013's scope unnoticed: ${LEAK_FAILS30:-<no failure>}"
|
fail "adding KyberforgeCopilot to [**/SKILL.md] did not fail Part A, so the style could leak past ADR-0013's scope unnoticed: ${LEAK_FAILS30:-<no failure>}"
|
||||||
else
|
else
|
||||||
pass "unloading KyberforgeCopilot from .agent.md files and leaking it onto SKILL.md files are each caught by Part A"
|
pass "unloading KyberforgeCopilot from .agent.md files and leaking it onto SKILL.md files are each caught by Part A"
|
||||||
@@ -1972,7 +1974,7 @@ SILENCED31="$(vale_report28 "$OVERRIDE_DIR31/.vale.ini" "$TREE28" "copilot/demo.
|
|||||||
SILENCED_COUNT31="$(files_scanned28 "$SILENCED31")"
|
SILENCED_COUNT31="$(files_scanned28 "$SILENCED31")"
|
||||||
if [[ -z "$SILENCED_COUNT31" || "$SILENCED_COUNT31" -eq 0 ]]; then
|
if [[ -z "$SILENCED_COUNT31" || "$SILENCED_COUNT31" -eq 0 ]]; then
|
||||||
fail "the override fixture scanned no file at all, so the disappearance of the VagueWording alert proves nothing about overrides"
|
fail "the override fixture scanned no file at all, so the disappearance of the VagueWording alert proves nothing about overrides"
|
||||||
elif printf '%s\n' "$SILENCED31" | grep -qF "Kyberforge.VagueWording"; then
|
elif grep -qF "Kyberforge.VagueWording" <<< "$SILENCED31"; then
|
||||||
# Reported as a FAIL, not a pass. It is not a defect in the config, but it
|
# Reported as a FAIL, not a pass. It is not a defect in the config, but it
|
||||||
# means Parts A and B are guarding a failure mode this vale build no longer
|
# means Parts A and B are guarding a failure mode this vale build no longer
|
||||||
# has — and a guard that guards nothing while reporting PASS is the same
|
# has — and a guard that guards nothing while reporting PASS is the same
|
||||||
@@ -2115,13 +2117,13 @@ awk '/^[ \t]*files:/ { sub(/\^/, "^zzz-no-such-path/") } { print }' \
|
|||||||
"$PC_CONFIG32" > "$MUT32/.pre-commit-config.yaml"
|
"$PC_CONFIG32" > "$MUT32/.pre-commit-config.yaml"
|
||||||
MUT_RECORDS32="$(hook_records28 "$MUT32/.pre-commit-config.yaml")"
|
MUT_RECORDS32="$(hook_records28 "$MUT32/.pre-commit-config.yaml")"
|
||||||
MUT_FAILS32="$(prefilter_scope_failures32 "$MUT32/.pre-commit-config.yaml" "$REPO_FILES32")"
|
MUT_FAILS32="$(prefilter_scope_failures32 "$MUT32/.pre-commit-config.yaml" "$REPO_FILES32")"
|
||||||
if ! printf '%s\n' "$MUT_RECORDS32" | grep -q 'zzz-no-such-path'; then
|
if ! grep -q 'zzz-no-such-path' <<< "$MUT_RECORDS32"; then
|
||||||
fail "the narrowed regexes never reached the copied config, so Part B narrowed nothing and proves nothing about Part A"
|
fail "the narrowed regexes never reached the copied config, so Part B narrowed nothing and proves nothing about Part A"
|
||||||
elif [[ "$(printf '%s\n' "$MUT_RECORDS32" | grep -c .)" -ne 2 ]]; then
|
elif [[ "$(printf '%s\n' "$MUT_RECORDS32" | grep -c .)" -ne 2 ]]; then
|
||||||
fail "the mutated config did not parse back as two prefilter hooks, so any failure below would come from the parser, not from the narrowing"
|
fail "the mutated config did not parse back as two prefilter hooks, so any failure below would come from the parser, not from the narrowing"
|
||||||
elif ! printf '%s' "$MUT_FAILS32" | grep -qF "vale-audit-prefilter-skill: 'files: ^zzz-no-such-path/"; then
|
elif ! grep -qF "vale-audit-prefilter-skill: 'files: ^zzz-no-such-path/" <<< "$MUT_FAILS32"; then
|
||||||
fail "narrowing the skill hook's regex to match zero files did not fail this check, so Part A cannot detect a prefilter that has been silently switched off for SKILL.md files"
|
fail "narrowing the skill hook's regex to match zero files did not fail this check, so Part A cannot detect a prefilter that has been silently switched off for SKILL.md files"
|
||||||
elif ! printf '%s' "$MUT_FAILS32" | grep -qF "vale-audit-prefilter-agent: 'files: ^zzz-no-such-path/"; then
|
elif ! grep -qF "vale-audit-prefilter-agent: 'files: ^zzz-no-such-path/" <<< "$MUT_FAILS32"; then
|
||||||
fail "narrowing the agent hook's regex to match zero files did not fail this check, so Part A cannot detect a prefilter that has been silently switched off for agent files"
|
fail "narrowing the agent hook's regex to match zero files did not fail this check, so Part A cannot detect a prefilter that has been silently switched off for agent files"
|
||||||
else
|
else
|
||||||
pass "narrowing either hook's 'files:' regex to match zero files is caught by Part A, which is what makes its pass mean something"
|
pass "narrowing either hook's 'files:' regex to match zero files is caught by Part A, which is what makes its pass mean something"
|
||||||
@@ -2314,15 +2316,15 @@ if ! grep -qF '^plugins/kyberforge/\.apm/skills/' "$MUT33/skill.yaml" \
|
|||||||
|| ! grep -qF '^plugins/kyberforge/\.apm/agents/' "$MUT33/agent.yaml" \
|
|| ! grep -qF '^plugins/kyberforge/\.apm/agents/' "$MUT33/agent.yaml" \
|
||||||
|| ! grep -qF 'vale-audit-prefilter-skill-renamed' "$MUT33/id.yaml"; then
|
|| ! grep -qF 'vale-audit-prefilter-skill-renamed' "$MUT33/id.yaml"; then
|
||||||
fail "a mutation never reached its copied config, so Part B mutated nothing and proves nothing about Part A"
|
fail "a mutation never reached its copied config, so Part B mutated nothing and proves nothing about Part A"
|
||||||
elif ! printf '%s' "$SKILL_FAILS33" | grep -qF "[plugins/demo/.apm/skills/demo/SKILL.md is in scope of kyberforge-vale-audit-skill"; then
|
elif ! grep -qF "[plugins/demo/.apm/skills/demo/SKILL.md is in scope of kyberforge-vale-audit-skill" <<< "$SKILL_FAILS33"; then
|
||||||
fail "narrowing the local skill hook to ^plugins/kyberforge/ did not fail Part A, so the prefilter can drop every other plugin's skills with every gate green: ${SKILL_FAILS33:-<no failure>}"
|
fail "narrowing the local skill hook to ^plugins/kyberforge/ did not fail Part A, so the prefilter can drop every other plugin's skills with every gate green: ${SKILL_FAILS33:-<no failure>}"
|
||||||
elif [[ -z "$AGREE_FAILS33" ]] && printf '%s' "$SKILL_FAILS33" | grep -qF "kyberforge-vale-audit-agent"; then
|
elif [[ -z "$AGREE_FAILS33" ]] && grep -qF "kyberforge-vale-audit-agent" <<< "$SKILL_FAILS33"; then
|
||||||
# Only meaningful against a clean base: when Part A already failed, the copy
|
# Only meaningful against a clean base: when Part A already failed, the copy
|
||||||
# inherits that defect, and reporting it again here would be one defect twice.
|
# inherits that defect, and reporting it again here would be one defect twice.
|
||||||
fail "narrowing only the skill hook also reported an agent-class defect, so the comparison is not confined to its class: $SKILL_FAILS33"
|
fail "narrowing only the skill hook also reported an agent-class defect, so the comparison is not confined to its class: $SKILL_FAILS33"
|
||||||
elif ! printf '%s' "$AGENT_FAILS33" | grep -qF "[plugins/demo/.apm/agents/demo.agent.md is in scope of kyberforge-vale-audit-agent"; then
|
elif ! grep -qF "[plugins/demo/.apm/agents/demo.agent.md is in scope of kyberforge-vale-audit-agent" <<< "$AGENT_FAILS33"; then
|
||||||
fail "narrowing the local agent hook to ^plugins/kyberforge/ did not fail Part A: ${AGENT_FAILS33:-<no failure>}"
|
fail "narrowing the local agent hook to ^plugins/kyberforge/ did not fail Part A: ${AGENT_FAILS33:-<no failure>}"
|
||||||
elif ! printf '%s' "$ID_FAILS33" | grep -qF "[no hook with id 'vale-audit-prefilter-skill' and a files: regex in id.yaml"; then
|
elif ! grep -qF "[no hook with id 'vale-audit-prefilter-skill' and a files: regex in id.yaml" <<< "$ID_FAILS33"; then
|
||||||
fail "renaming the local skill hook's id did not fail Part A by name, so the skill class could fall out of the comparison silently: ${ID_FAILS33:-<no failure>}"
|
fail "renaming the local skill hook's id did not fail Part A by name, so the skill class could fall out of the comparison silently: ${ID_FAILS33:-<no failure>}"
|
||||||
else
|
else
|
||||||
pass "narrowing either local hook to one plugin, or renaming one, is caught by Part A"
|
pass "narrowing either local hook to one plugin, or renaming one, is caught by Part A"
|
||||||
|
|||||||
Reference in New Issue
Block a user