Defame1297
ffcbed6c41
Why Two suites failed intermittently — tests/test-vale-wrap.sh case 21 and tests/test-check-release-needed.sh cases 4 and 15 — on correct output, and never when run alone. The cause is the `echo "$OUT" | grep -q P` idiom under `set -o pipefail`: grep -q exits as soon as it has an answer, bash's echo can hand a multi-line value to the pipe one line at a time, and a write after the reader is gone kills echo with SIGPIPE. pipefail then reports the writer's death, so output that DID match reads as "no match". Every observed failure had lines after its match; case 15's match is on line 1 of 6, the widest window in that file. Forced with a pause before the writer's last line, the pipe form failed 50 of 50 runs; a here-string, a match on the last line, and the same pipe without pipefail each passed 50 of 50. Unforced the rate is about 1 per 670 suite runs, which is why it read as a flaky gate rather than a bug. The failures at review time are consistent with this, but were not proven to be it: the suite was running while agents edited live config files in place, and a brief change to .vale.ini or .pre-commit-hooks.yaml would produce the same two failures. The race is real and fixed either way. Implementation Notes `grep -q P <<< "$VAR"` has no separate writer process, so there is nothing to race. It is not a retry or a sleep. 121 sites converted across 9 files, three of them scripts rather than tests: new-agent.sh, new-skill.sh and check-executables-allow-sync.sh. None ships via .pre-commit-hooks.yaml, so no external consumer pins them, and all three are single-pipeline checks whose verdict cannot change. Left alone deliberately: 14 sites whose writer is a command, not a shell builtin — they either absorb the writer's status with `|| true` or are python3 and awk, which write once at exit — and one file with no pipefail. `printf '%s'` sites differ from a here-string only by a trailing newline, which no -q verdict on a non-empty pattern depends on. tests/test-no-pipefail-early-exit-grep.sh is a static guard against new occurrences, discovered automatically by run-tests.sh. It only scans files that set pipefail, joins continuation lines, skips comments, and flags only echo/printf writers. Its first case proves the scanner can fail before its second trusts a clean verdict on the tree. A guard covers exactly the spellings its regex models, so the miss surface was measured rather than assumed. Four were found and closed: pipefail declared as `set -o errexit -o pipefail` (where the old pattern required pipefail to follow the FIRST -o, and a file-level miss skips every site in that file); a writer separated from grep by an intermediate stage; a pipeline wrapped on a trailing `|` rather than a backslash; and readers spelled egrep, fgrep, /bin/grep, `command grep` or with an env-var prefix. Segment characters exclude a bare `&` so `echo ok && other | grep -q x`, whose writer is `other`, does not false-fire. Widening surfaced 5 live sites invisible to the original scanner, all in tests/test-apm-current-hook.sh, all `echo "$out" | json_field ... | grep -q`; they are safe today only because json_field is python3, which reads to EOF and writes once. Fixtures go 4 to 12 vulnerable spellings plus near-miss negatives. Two `grep ... | head -1` sites (test-vale-wrap.sh) are the same race with a different early-exiting reader, and are fixed by absorbing the writer. The scanner deliberately does not model `head`, `sed -n 1p` or a bare `read`: most legitimate uses in this tree are already absorbed with `|| true` and the scanner cannot see absorption from pipeline text, so a high false-positive rate would be how this guard gets weakened. Heredoc bodies are scanned as code; none in the tree trips it today. Impact The bug predates the factory-audit merge: every converted site in check-release-needed and case 21 dates to4d018afandaa8cc22(2026-08-09). Test suites go 19 to 20. `run-tests.sh --strict` passes 20/20 with 0 skipped, four consecutive runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YR2CjVumUbEGWcMikcoXBD
2521 lines
124 KiB
Bash
Executable File
2521 lines
124 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Regression test for scripts/vale-wrap.sh: Vale's `text.frontmatter.description`
|
|
# NLP scope silently stops matching when the description value is a YAML block
|
|
# scalar spanning 2+ physical lines. vale-wrap.sh flattens it to one line before
|
|
# handing off to the real vale binary — this asserts that actually happens.
|
|
set -euo pipefail
|
|
|
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
# ADR-0025 merged skill-audit and agent-audit, so there is now ONE vale-wrap.sh and
|
|
# ONE .vale.ini. Every fixture below is a SKILL.md, matched by the merged config's
|
|
# [**/SKILL.md] section — which is the same section the pre-merge skill-audit config
|
|
# carried, unchanged, so no fixture's expected verdict moves.
|
|
FACTORY_AUDIT="$REPO_ROOT/plugins/kyberforge/.apm/skills/factory-audit"
|
|
SCRIPT="$FACTORY_AUDIT/scripts/vale-wrap.sh"
|
|
VALE_CONFIG="$FACTORY_AUDIT/assets/vale/.vale.ini"
|
|
PASS=0
|
|
FAIL=0
|
|
|
|
pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
|
|
fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
|
|
|
|
# Vale absent skips the Vale-DEPENDENT cases, not the suite. An early `exit 77`
|
|
# here used to skip everything, including the checks that are plain greps and
|
|
# awk over the config and the two hook manifests (cases 16, 26-28's static
|
|
# halves, 31 Parts A/B, 32, 33) -- so a machine without vale reported a skip
|
|
# while never looking at a manifest it could have read. Those still run; the
|
|
# suite exits 77 at the end only if they all passed, so run-tests.sh keeps
|
|
# reporting SKIPPED and `--strict` keeps turning that skip into a failure. A
|
|
# static FAIL still exits 1, because a real defect is not a setup error.
|
|
VALE_AVAILABLE=true
|
|
if ! command -v vale &>/dev/null; then
|
|
VALE_AVAILABLE=false
|
|
echo "SKIP: vale is not installed — Vale-dependent cases skipped (matches factory-audit's own fallback behavior); the static cases still run"
|
|
fi
|
|
|
|
# The fixture handles the skippable regions below create. Seeded empty so the
|
|
# EXIT trap's `rm -rf` does not trip `set -u` on a region that never ran.
|
|
FIXTURE1="" FIXTURE2="" FIXTURE3="" FIXTURE4="" FIXTURE5="" FIXTURE6=""
|
|
FIXTURE7="" FIXTURE8="" FIXTURE10="" FIXTURE11="" FIXTURE12="" STUB13=""
|
|
FIXTURE14="" FIXTURE17="" FIXTURE18=""
|
|
|
|
# --- 0. The shipped Vale config can load at all ------------------------------
|
|
# A missing `.vale.ini`, a missing `StylesPath`, or a `BasedOnStyles` naming a
|
|
# style directory that is not there all stop vale before it lints anything
|
|
# (`path ... does not exist`, `style 'Kyberforge' does not exist on
|
|
# StylesPath`, rc 2). Every Vale-dependent case below then failed on its own
|
|
# generic symptom -- nine "vale printed no summary line" failures across cases
|
|
# 28-31 alone, none naming the cause. This names it once and holds the
|
|
# Vale-dependent cases back instead. Static on purpose: it needs no vale, so it
|
|
# runs on the machines that skip everything else.
|
|
#
|
|
# `BasedOnStyles =` left EMPTY is deliberately not a defect here: vale loads
|
|
# that config and lints the file with no style, which is the silent case 28
|
|
# exists to catch, and catching it here instead would leave 28's
|
|
# style-not-loaded branch untested.
|
|
vale_config_defects0() {
|
|
local cfg="$1" dir sp line names name bad=""
|
|
if [[ ! -f "$cfg" ]]; then
|
|
printf '%s' "[$cfg does not exist] "
|
|
return 0
|
|
fi
|
|
# Decided by ACTUALLY READING the file, not by `[[ -r ]]`. `-r` is access(2),
|
|
# which answers "would the permission bits allow it" -- and for uid 0 that is
|
|
# yes even on a mode-000 file. This repo's dev environment is root, so an
|
|
# `[[ ! -r ]]` guard could never fire in the one place it exists to fire: it
|
|
# was untestable because it was dead. A read attempt is also the stricter
|
|
# question, catching EISDIR and EIO, which access(2) reports on neither.
|
|
# `cat`, not a bare `< "$cfg"` redirect: opening a directory for reading
|
|
# succeeds, only the read fails. (scripts/check-vale-style-sync.sh carried
|
|
# this reasoning before ADR-0025 deleted it; the hazard did not go with it.)
|
|
if ! cat "$cfg" > /dev/null 2>&1; then
|
|
printf '%s' "[$cfg exists but could not be read] "
|
|
return 0
|
|
fi
|
|
dir="$(dirname "$cfg")"
|
|
sp="$({ grep -E '^[[:space:]]*StylesPath[[:space:]]*=' "$cfg" || true; } | tail -1)"
|
|
sp="${sp#*=}"
|
|
sp="${sp#"${sp%%[![:space:]]*}"}"
|
|
sp="${sp%"${sp##*[![:space:]]}"}"
|
|
if [[ -z "$sp" ]]; then
|
|
printf '%s' "[${cfg##*/} sets no StylesPath, so vale cannot find any style it names] "
|
|
return 0
|
|
fi
|
|
[[ "$sp" == /* ]] || sp="$dir/$sp"
|
|
if [[ ! -d "$sp" ]]; then
|
|
printf '%s' "[StylesPath resolves to $sp, which is not a directory] "
|
|
return 0
|
|
fi
|
|
while IFS= read -r line; do
|
|
names="${line#*=}"
|
|
while IFS= read -r name; do
|
|
name="${name#"${name%%[![:space:]]*}"}"
|
|
name="${name%"${name##*[![:space:]]}"}"
|
|
# `Vale` is vale's built-in style and has no directory.
|
|
[[ -n "$name" && "$name" != "Vale" ]] || continue
|
|
[[ -d "$sp/$name" ]] || bad+="[BasedOnStyles names '$name', but $sp/$name does not exist] "
|
|
done <<EOF_NAMES0
|
|
$(printf '%s\n' "$names" | tr ',' '\n')
|
|
EOF_NAMES0
|
|
done <<EOF_LINES0
|
|
$({ grep -E '^[[:space:]]*BasedOnStyles[[:space:]]*=' "$cfg" || true; })
|
|
EOF_LINES0
|
|
printf '%s' "$bad"
|
|
}
|
|
|
|
echo ""
|
|
echo "--- the shipped .vale.ini loads: it exists, sets StylesPath, and every style it names ships ---"
|
|
VALE_CONFIG_DEFECTS="$(vale_config_defects0 "$VALE_CONFIG")"
|
|
VALE_CONFIG_OK=true
|
|
if [[ -n "$VALE_CONFIG_DEFECTS" ]]; then
|
|
VALE_CONFIG_OK=false
|
|
fail "the shipped Vale config cannot load, so every Vale-dependent case below is held back rather than reported as a pile of generic 'no summary line' failures: $VALE_CONFIG_DEFECTS"
|
|
else
|
|
pass "the shipped .vale.ini exists, sets a StylesPath that is a directory, and every style it names ships under it"
|
|
fi
|
|
|
|
VALE_READY=false
|
|
[[ "$VALE_AVAILABLE" == true && "$VALE_CONFIG_OK" == true ]] && VALE_READY=true
|
|
|
|
make_fixture() {
|
|
local dir desc_lines file
|
|
dir="$(mktemp -d)"
|
|
(cd "$dir" && git init -q)
|
|
mkdir -p "$dir/plugins/testplugin/skills/zzzskill"
|
|
desc_lines="$1"
|
|
file="$dir/plugins/testplugin/skills/zzzskill/SKILL.md"
|
|
{
|
|
echo "---"
|
|
echo "name: zzzskill"
|
|
echo "description: >"
|
|
for ((i = 1; i <= desc_lines; i++)); do
|
|
echo " Line $i mentions helps with and utilize, plus a colon: like this."
|
|
done
|
|
echo "---"
|
|
echo ""
|
|
echo "Body."
|
|
} > "$file"
|
|
echo "$dir"
|
|
}
|
|
|
|
# Every Kyberforge rule is `level: error`, so vale exits non-zero whenever a
|
|
# fixture trips one — which is the expected outcome for nearly every case here.
|
|
# run_wrap therefore captures output and swallows the exit status; assertions
|
|
# are made on the report text. Cases that genuinely care about the exit code
|
|
# capture it explicitly instead.
|
|
run_wrap() {
|
|
local dir="$1"
|
|
shift
|
|
(cd "$dir" && bash "$SCRIPT" "$@" 2>&1) || true
|
|
}
|
|
|
|
# ===== BEGIN VALE-DEPENDENT REGION (cases 1-15) ==============================
|
|
# Not re-indented, so the case bodies stay diffable against their history. The
|
|
# matching `fi` is marked END with the same case range.
|
|
if [[ "$VALE_READY" == true ]]; then
|
|
|
|
# --- 1. A known-bad single-line description is caught (sanity check on Vale itself) ---
|
|
echo ""
|
|
echo "--- catches vague wording in a single-line description ---"
|
|
FIXTURE1="$(make_fixture 1)"
|
|
trap 'rm -rf "$FIXTURE1"' EXIT
|
|
if run_wrap "$FIXTURE1" --config "$VALE_CONFIG" \
|
|
plugins/testplugin/skills/zzzskill/SKILL.md | grep -q "VagueWording"; then
|
|
pass "flags vague wording when description is a single physical line"
|
|
else
|
|
fail "did not flag known-bad single-line description"
|
|
fi
|
|
|
|
# --- 2. The same known-bad wording across 2+ physical lines is still caught ---
|
|
echo ""
|
|
echo "--- catches vague wording in a multi-line folded description ---"
|
|
FIXTURE2="$(make_fixture 2)"
|
|
trap 'rm -rf "$FIXTURE1" "$FIXTURE2"' EXIT
|
|
if run_wrap "$FIXTURE2" --config "$VALE_CONFIG" \
|
|
plugins/testplugin/skills/zzzskill/SKILL.md | grep -q "VagueWording"; then
|
|
pass "flags vague wording when description spans 2+ physical lines"
|
|
else
|
|
fail "silently missed known-bad wording in a multi-line description — the bug this test guards against"
|
|
fi
|
|
|
|
# --- 3. Line count is preserved so unrelated report line numbers don't shift ---
|
|
echo ""
|
|
echo "--- preserves total line count when flattening ---"
|
|
FIXTURE3="$(make_fixture 3)"
|
|
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3"' EXIT
|
|
ORIG_LINES=$(wc -l < "$FIXTURE3/plugins/testplugin/skills/zzzskill/SKILL.md")
|
|
OUT=$(run_wrap "$FIXTURE3" --config "$VALE_CONFIG" \
|
|
plugins/testplugin/skills/zzzskill/SKILL.md)
|
|
MAX_LINE=$(echo "$OUT" | grep -oE '^[[:space:]]*[0-9]+:[0-9]+' | tr -d '[:space:]' | cut -d: -f1 | sort -n | tail -1)
|
|
if [[ -n "$MAX_LINE" ]] && (( MAX_LINE <= ORIG_LINES )); then
|
|
pass "reported line numbers stay within the original file's line count"
|
|
else
|
|
fail "reported line number ($MAX_LINE) exceeds original file line count ($ORIG_LINES)"
|
|
fi
|
|
|
|
# make_raw_fixture writes stdin verbatim to a fresh fixture's SKILL.md, for
|
|
# cases where the exact description body needs to be hand-crafted rather than
|
|
# generated from the desc_lines loop above.
|
|
make_raw_fixture() {
|
|
local dir
|
|
dir="$(mktemp -d)"
|
|
(cd "$dir" && git init -q)
|
|
mkdir -p "$dir/plugins/testplugin/skills/zzzskill"
|
|
cat > "$dir/plugins/testplugin/skills/zzzskill/SKILL.md"
|
|
echo "$dir"
|
|
}
|
|
|
|
# --- 4. A folded description containing a double quote is still caught ---
|
|
# This is the exact case that silently passed (zero alerts) before switching
|
|
# from json.dumps (double-quoted, backslash-escaped) to a single-quoted scalar.
|
|
echo ""
|
|
echo "--- catches vague wording when the folded description contains a double quote ---"
|
|
FIXTURE4="$(make_raw_fixture <<'EOF'
|
|
---
|
|
name: zzzskill
|
|
description: >
|
|
Use when the user says "audit this skill" and helps with and utilize things.
|
|
Second line continues the same folded scalar for flattening.
|
|
---
|
|
|
|
Body.
|
|
EOF
|
|
)"
|
|
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4"' EXIT
|
|
if run_wrap "$FIXTURE4" --config "$VALE_CONFIG" \
|
|
plugins/testplugin/skills/zzzskill/SKILL.md | grep -q "VagueWording"; then
|
|
pass "flags vague wording when the description contains a double quote"
|
|
else
|
|
fail "silently missed vague wording in a description containing a double quote — the bug this test guards against"
|
|
fi
|
|
|
|
# --- 5. A folded description containing an apostrophe fires and stays valid YAML ---
|
|
echo ""
|
|
echo "--- catches vague wording when the folded description contains an apostrophe ---"
|
|
FIXTURE5="$(make_raw_fixture <<'EOF'
|
|
---
|
|
name: zzzskill
|
|
description: >
|
|
Use when the user's task helps with and utilize things across two lines.
|
|
Second continuation line for the fold.
|
|
---
|
|
|
|
Body.
|
|
EOF
|
|
)"
|
|
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5"' EXIT
|
|
OUT5=$(run_wrap "$FIXTURE5" --config "$VALE_CONFIG" \
|
|
plugins/testplugin/skills/zzzskill/SKILL.md)
|
|
if grep -q "VagueWording" <<< "$OUT5"; then
|
|
pass "flags vague wording when the description contains an apostrophe"
|
|
else
|
|
fail "silently missed vague wording in a description containing an apostrophe"
|
|
fi
|
|
if grep -qi "yaml:" <<< "$OUT5"; then
|
|
fail "flattened copy with an apostrophe produced a YAML parse error"
|
|
else
|
|
pass "flattened copy with an apostrophe is valid YAML (no parse error)"
|
|
fi
|
|
|
|
# --- 6. A folded description with a backslash and a non-ASCII character ---
|
|
echo ""
|
|
echo "--- catches vague wording when the folded description has a backslash and non-ASCII text ---"
|
|
FIXTURE6="$(make_raw_fixture <<'EOF'
|
|
---
|
|
name: zzzskill
|
|
description: >
|
|
Use when the café résumé naïve thing helps with and utilize things here.
|
|
Path is C:\Users\test and this is the second continuation line.
|
|
---
|
|
|
|
Body.
|
|
EOF
|
|
)"
|
|
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6"' EXIT
|
|
if run_wrap "$FIXTURE6" --config "$VALE_CONFIG" \
|
|
plugins/testplugin/skills/zzzskill/SKILL.md | grep -q "VagueWording"; then
|
|
pass "flags vague wording when the description has a backslash and non-ASCII text"
|
|
else
|
|
fail "silently missed vague wording in a description with a backslash and non-ASCII text"
|
|
fi
|
|
|
|
# --- 7. A folded description with a blank line between two paragraphs ---
|
|
echo ""
|
|
echo "--- handles a blank line inside a folded description without crashing ---"
|
|
FIXTURE7="$(make_raw_fixture <<'EOF'
|
|
---
|
|
name: zzzskill
|
|
description: >
|
|
Use when the user needs a general helper.
|
|
|
|
Do not use when this helps with and utilize things instead.
|
|
---
|
|
|
|
Body.
|
|
EOF
|
|
)"
|
|
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7"' EXIT
|
|
OUT7=$(run_wrap "$FIXTURE7" --config "$VALE_CONFIG" \
|
|
plugins/testplugin/skills/zzzskill/SKILL.md)
|
|
if grep -q "Traceback" <<< "$OUT7"; then
|
|
fail "crashed while flattening a description with a blank line between paragraphs"
|
|
elif grep -q "VagueWording" <<< "$OUT7"; then
|
|
pass "still flags vague wording in the second paragraph after a blank line"
|
|
else
|
|
fail "silently missed vague wording in the second paragraph after a blank line — the bug this test guards against"
|
|
fi
|
|
|
|
# --- 8. Relative paths resolve against the caller's cwd, exactly as bare vale
|
|
# resolves them. Every path below is deliberately relative to $SUBDIR8, not to
|
|
# the fixture's repo root: an earlier version of the wrapper resolved relative
|
|
# paths against the git toplevel instead, which (a) hard-errored on a
|
|
# `--config ../../..` that bare vale accepts and (b) silently dropped file
|
|
# arguments that didn't resolve from the repo root, skipping the flattening the
|
|
# wrapper exists to perform. The old tests only ever passed repo-root-relative
|
|
# paths from a subdirectory, so neither failure mode was caught.
|
|
echo ""
|
|
echo "--- resolves a cwd-relative --config from a subdirectory (equals and two-argv forms) ---"
|
|
FIXTURE8="$(mktemp -d)"
|
|
(cd "$FIXTURE8" && git init -q)
|
|
cp "$VALE_CONFIG" "$FIXTURE8/.vale.ini"
|
|
cp -r "$FACTORY_AUDIT/assets/vale/styles" "$FIXTURE8/styles"
|
|
mkdir -p "$FIXTURE8/plugins/testplugin/skills/zzzskill"
|
|
{
|
|
echo "---"
|
|
echo "name: zzzskill"
|
|
echo "description: >"
|
|
echo " Line one mentions helps with and utilize, plus a colon: like this."
|
|
echo " Line two continues the same folded scalar for flattening."
|
|
echo "---"
|
|
echo ""
|
|
echo "Body."
|
|
} > "$FIXTURE8/plugins/testplugin/skills/zzzskill/SKILL.md"
|
|
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7" "$FIXTURE8"' EXIT
|
|
SUBDIR8="$FIXTURE8/plugins/testplugin/skills/zzzskill"
|
|
# Both paths are relative to $SUBDIR8 (four levels below the fixture root).
|
|
REL_CFG8="../../../../.vale.ini"
|
|
REL_FILE8="SKILL.md"
|
|
OUT_EQ=$(run_wrap "$SUBDIR8" "--config=$REL_CFG8" "$REL_FILE8")
|
|
OUT_TWO=$(run_wrap "$SUBDIR8" --config "$REL_CFG8" "$REL_FILE8")
|
|
if grep -q "VagueWording" <<< "$OUT_EQ" && [[ "$OUT_EQ" == "$OUT_TWO" ]]; then
|
|
pass "cwd-relative --config resolves from a subdirectory in both argv forms"
|
|
else
|
|
fail "cwd-relative --config did not resolve from a subdirectory (equals form vs two-argv form)"
|
|
fi
|
|
|
|
# --- 8b. A cwd-relative --config matches the equivalent absolute invocation ---
|
|
# Regression for failure mode (a): resolving --config against the repo root made
|
|
# `--config ../../../../.vale.ini` expand to a path above the toplevel, and vale
|
|
# hard-errored with "does not exist" (exit 2) on args bare vale handles fine.
|
|
echo ""
|
|
echo "--- a cwd-relative --config produces the same result as the absolute-path form ---"
|
|
set +e
|
|
OUT_REL_CFG=$(cd "$SUBDIR8" && bash "$SCRIPT" --config "$REL_CFG8" "$REL_FILE8" 2>&1)
|
|
RC_REL_CFG=$?
|
|
OUT_ABS_CFG=$(cd "$SUBDIR8" && bash "$SCRIPT" --config "$FIXTURE8/.vale.ini" "$REL_FILE8" 2>&1)
|
|
RC_ABS_CFG=$?
|
|
set -e
|
|
if grep -qi "does not exist" <<< "$OUT_REL_CFG"; then
|
|
fail "cwd-relative --config hard-errored ('does not exist') — the bug this test guards against"
|
|
elif [[ "$OUT_REL_CFG" == "$OUT_ABS_CFG" && "$RC_REL_CFG" -eq "$RC_ABS_CFG" ]]; then
|
|
pass "cwd-relative --config matches the absolute-path invocation (output and exit code)"
|
|
else
|
|
fail "cwd-relative --config (rc=$RC_REL_CFG) diverged from the absolute-path form (rc=$RC_ABS_CFG)"
|
|
fi
|
|
|
|
# --- 8c. A cwd-relative FILE argument is still flattened, not silently skipped ---
|
|
# Regression for failure mode (b): a relative file path that didn't resolve from
|
|
# the repo root failed the wrapper's file test, fell through to the vale flag
|
|
# list, and left the file list empty — so the wrapper exec'd bare vale and
|
|
# silently skipped the flattening. Bare vale reports nothing here, so asserting
|
|
# on the alert (not just the exit code) is what makes the silence detectable.
|
|
echo ""
|
|
echo "--- flattens a cwd-relative file argument passed from a subdirectory ---"
|
|
WRAPPED_REL=$(run_wrap "$SUBDIR8" --config "$FIXTURE8/.vale.ini" "$REL_FILE8")
|
|
BARE_REL=$(cd "$SUBDIR8" && vale --config "$FIXTURE8/.vale.ini" "$REL_FILE8" 2>&1 || true)
|
|
if ! grep -q "VagueWording" <<< "$WRAPPED_REL"; then
|
|
fail "cwd-relative file argument produced no alert — flattening was silently skipped, the bug this test guards against"
|
|
elif grep -q "VagueWording" <<< "$BARE_REL"; then
|
|
fail "bare vale already flags this fixture, so the test can't detect a silently-skipped flattening"
|
|
else
|
|
pass "cwd-relative file argument is flattened and flagged where bare vale reports nothing"
|
|
fi
|
|
|
|
# --- 9. Zero file args (or a file list that filters to nothing) exits promptly ---
|
|
echo ""
|
|
echo "--- exits promptly instead of hanging on stdin when no files are passed ---"
|
|
if timeout 5 bash "$SCRIPT" --config "$VALE_CONFIG" < <(sleep 100) >/dev/null 2>&1; then
|
|
pass "exits promptly with zero file args"
|
|
else
|
|
RC=$?
|
|
if [[ $RC -eq 124 ]]; then
|
|
fail "hung waiting on stdin with zero file args — the bug this test guards against"
|
|
else
|
|
pass "exits promptly (nonzero exit) with zero file args"
|
|
fi
|
|
fi
|
|
|
|
echo ""
|
|
echo "--- exits promptly when a file list filters down to nothing ---"
|
|
if timeout 5 bash "$SCRIPT" --config "$VALE_CONFIG" --no-such-flag < <(sleep 100) >/dev/null 2>&1; then
|
|
pass "exits promptly when no file-shaped args remain"
|
|
else
|
|
RC=$?
|
|
if [[ $RC -eq 124 ]]; then
|
|
fail "hung waiting on stdin when the file list filtered to nothing — the bug this test guards against"
|
|
else
|
|
pass "exits promptly (nonzero exit) when the file list filters to nothing"
|
|
fi
|
|
fi
|
|
|
|
# --- 10. An absolute path to the fixture SKILL.md is still linted, not skipped ---
|
|
echo ""
|
|
echo "--- lints an absolute path to a skill file instead of silently skipping it ---"
|
|
FIXTURE10="$(make_fixture 2)"
|
|
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7" "$FIXTURE8" "$FIXTURE10"' EXIT
|
|
ABS_FILE10="$FIXTURE10/plugins/testplugin/skills/zzzskill/SKILL.md"
|
|
if run_wrap "$FIXTURE10" --config "$VALE_CONFIG" "$ABS_FILE10" | grep -q "VagueWording"; then
|
|
pass "an absolute path is linted, not silently skipped"
|
|
else
|
|
fail "an absolute path was silently skipped — the bug this test guards against"
|
|
fi
|
|
|
|
# --- 11. A literal (|) block scalar passes through unflattened. Unlike every
|
|
# other multi-line form, `|` is not broken in Vale: its parsed value keeps the
|
|
# same line breaks the source has, so the description scope still matches. The
|
|
# second assertion pins that down — without it, a wrapper that broke `|` and a
|
|
# Vale that never matched `|` would agree on zero alerts and the comparison
|
|
# would pass vacuously.
|
|
echo ""
|
|
echo "--- leaves a literal (|) block scalar untouched (narrowed >-only scope) ---"
|
|
FIXTURE11="$(make_raw_fixture <<'EOF'
|
|
---
|
|
name: zzzskill
|
|
description: |
|
|
Line one mentions helps with and utilize things here.
|
|
Line two continues the literal block scalar for this test.
|
|
---
|
|
|
|
Body.
|
|
EOF
|
|
)"
|
|
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7" "$FIXTURE8" "$FIXTURE10" "$FIXTURE11"' EXIT
|
|
REL11="plugins/testplugin/skills/zzzskill/SKILL.md"
|
|
WRAPPED_OUT=$(run_wrap "$FIXTURE11" --config "$VALE_CONFIG" "$REL11")
|
|
BARE_OUT=$(cd "$FIXTURE11" && vale --config "$VALE_CONFIG" "$REL11" 2>&1 || true)
|
|
if ! grep -q "VagueWording" <<< "$BARE_OUT"; then
|
|
fail "bare vale reports nothing for a literal (|) block scalar — the 'literal blocks are not broken' premise is wrong"
|
|
elif [[ "$WRAPPED_OUT" == "$BARE_OUT" ]]; then
|
|
pass "literal (|) block scalar output matches bare vale exactly — untouched by flattening"
|
|
else
|
|
fail "wrapper altered output for a literal (|) block scalar description — should be left untouched"
|
|
fi
|
|
|
|
# --- 12. With no --config at all, the wrapper falls back to its own sibling
|
|
# assets/vale/.vale.ini. `.pre-commit-hooks.yaml` relies on this: pre-commit
|
|
# prefixes only entry[0] with the hook-repo clone path, so a --config argument
|
|
# there resolves against the consuming repo and hard-errors (E100) for every
|
|
# external consumer.
|
|
echo ""
|
|
echo "--- defaults --config to the wrapper's own sibling assets/vale/.vale.ini ---"
|
|
FIXTURE12="$(make_fixture 2)"
|
|
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7" "$FIXTURE8" "$FIXTURE10" "$FIXTURE11" "$FIXTURE12"' EXIT
|
|
OUT12=$(run_wrap "$FIXTURE12" plugins/testplugin/skills/zzzskill/SKILL.md)
|
|
if grep -q "VagueWording" <<< "$OUT12"; then
|
|
pass "a --config-less invocation uses the wrapper's bundled config"
|
|
else
|
|
fail "a --config-less invocation found no config — external pre-commit consumers get E100, the bug this test guards against"
|
|
fi
|
|
|
|
# --- 13. No GNU-only `realpath -m`. macOS ships the BSD realpath, which has no
|
|
# -m (canonicalize-missing) — and every scratch destination is a path that does
|
|
# not exist yet, so a plain `realpath` exits 1 and set -e aborts the hook.
|
|
echo ""
|
|
echo "--- runs with a BSD realpath that has no -m option ---"
|
|
STUB13="$(mktemp -d)"
|
|
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7" "$FIXTURE8" "$FIXTURE10" "$FIXTURE11" "$FIXTURE12" "$STUB13"' EXIT
|
|
REAL_REALPATH="$(command -v realpath || echo /bin/false)"
|
|
cat > "$STUB13/realpath" <<EOF
|
|
#!/usr/bin/env bash
|
|
for a in "\$@"; do
|
|
case "\$a" in
|
|
-m|--canonicalize-missing)
|
|
echo "realpath: illegal option -- m" >&2
|
|
exit 1
|
|
;;
|
|
esac
|
|
done
|
|
exec "$REAL_REALPATH" "\$@"
|
|
EOF
|
|
chmod +x "$STUB13/realpath"
|
|
OUT13=$(cd "$FIXTURE12" && PATH="$STUB13:$PATH" bash "$SCRIPT" --config "$VALE_CONFIG" \
|
|
plugins/testplugin/skills/zzzskill/SKILL.md 2>&1 || true)
|
|
if grep -q "illegal option" <<< "$OUT13"; then
|
|
fail "invoked realpath -m — fails on macOS's BSD realpath, the bug this test guards against"
|
|
elif grep -q "VagueWording" <<< "$OUT13"; then
|
|
pass "flattens and flags with no GNU realpath available"
|
|
else
|
|
fail "produced no alert under a BSD-style realpath: $OUT13"
|
|
fi
|
|
|
|
# --- 14. A directory argument is walked and its files flattened. The classifier
|
|
# used to accept only regular files, so a directory fell through to the vale
|
|
# flag list, left the file list empty, and exec'd bare vale — silently skipping
|
|
# the flattening. `lint`'s vale-run skill documents `vale <path-or-glob>` as
|
|
# normal usage, so this is a reachable path.
|
|
echo ""
|
|
echo "--- flattens files reached through a directory argument ---"
|
|
FIXTURE14="$(make_fixture 2)"
|
|
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7" "$FIXTURE8" "$FIXTURE10" "$FIXTURE11" "$FIXTURE12" "$STUB13" "$FIXTURE14"' EXIT
|
|
WRAPPED_DIR=$(run_wrap "$FIXTURE14" --config "$VALE_CONFIG" plugins)
|
|
BARE_DIR=$(cd "$FIXTURE14" && vale --config "$VALE_CONFIG" plugins 2>&1 || true)
|
|
if ! grep -q "VagueWording" <<< "$WRAPPED_DIR"; then
|
|
fail "a directory argument produced no alert — flattening was silently skipped, the bug this test guards against"
|
|
elif grep -q "VagueWording" <<< "$BARE_DIR"; then
|
|
fail "bare vale already flags this fixture, so the test can't detect a silently-skipped flattening"
|
|
else
|
|
pass "a directory argument is walked and its files flattened"
|
|
fi
|
|
|
|
# --- 15. Directory walking must survive paths with spaces ---
|
|
echo ""
|
|
echo "--- walks a directory containing a path with spaces ---"
|
|
SPACED15="$FIXTURE14/plugins/testplugin/skills/zzz skill"
|
|
mkdir -p "$SPACED15"
|
|
cp "$FIXTURE14/plugins/testplugin/skills/zzzskill/SKILL.md" "$SPACED15/SKILL.md"
|
|
rm -rf "$FIXTURE14/plugins/testplugin/skills/zzzskill"
|
|
OUT15=$(run_wrap "$FIXTURE14" --config "$VALE_CONFIG" plugins/testplugin/skills)
|
|
if grep -q "zzz skill" <<< "$OUT15" && grep -q "VagueWording" <<< "$OUT15"; then
|
|
pass "a file under a directory whose name contains a space is walked and flattened"
|
|
else
|
|
fail "a path with a space was dropped from the directory walk"
|
|
fi
|
|
|
|
fi
|
|
# ===== END VALE-DEPENDENT REGION (cases 1-15) ================================
|
|
|
|
# --- 16. No unguarded `"${arr[@]}"` expansion survives in any script that runs
|
|
# on macOS. bash before 4.4 — including the 3.2 that macOS still ships as
|
|
# /bin/bash — treats that form on an *empty* array as an unbound variable under
|
|
# `set -u` and aborts. The portable form is `${arr[@]+"${arr[@]}"}`. This is a
|
|
# static check because no bash 5 host can reproduce the abort at runtime: the
|
|
# construct is only fatal on the older shell, so absence of the construct is the
|
|
# property to assert. `${#arr[@]}` is deliberately not flagged — the count form
|
|
# is safe on 3.2. Neither is an array seeded with at least one element where it
|
|
# is declared and never reset to empty: it cannot be empty at any expansion
|
|
# site, so the construct is not a hazard there and demanding the guarded form
|
|
# would be a wrong test. Neither is an expansion whose own line first proves the
|
|
# array non-empty (`[[ ${#a[@]} -eq 0 ]] || rm -rf "${a[@]}"`) — the short-circuit
|
|
# means the expansion is unreachable when the array is empty.
|
|
#
|
|
# The scanned file list is DERIVED, not hand-maintained. A hardcoded list only
|
|
# guards the scripts someone remembered to add to it, and it silently fails to
|
|
# cover anything new: it omitted scripts/lib/batch-run.sh — the shared runner
|
|
# this branch introduced, whose own header (batch-run.sh:9-11) documents it as
|
|
# bash-3.2-safe — along with four other scripts/*.sh. Deriving the list means a
|
|
# new script is covered the moment it lands. The script headers name bash 3.2 as
|
|
# an explicit repo target -- scripts/lib/batch-run.sh:9 ("both callers are
|
|
# explicitly bash-3.2-safe") and providers/claude-code/statusline-command.sh:100
|
|
# ("macOS's system bash, and an explicit repo target") -- so the scope is four
|
|
# globs, each floor-asserted below:
|
|
# - scripts/**/*.sh — repo tooling and pre-commit hook scripts
|
|
# - tests/*.sh — the runners and every regression test
|
|
# - plugins/*/.apm/**/*.sh — the scripts plugins ship to users
|
|
# - providers/**/*.sh — the scripts install.sh deploys to user machines
|
|
# .apm/ is the only plugin content scanned, and it is the only plugin content
|
|
# there is: the generated flat plugins/*/skills/** mirror was deleted with
|
|
# native plugin-install support, so there is no second copy to double-report.
|
|
# Scanning .apm/ is what closed the gap where skill-audit's vale-wrap.sh was
|
|
# covered but agent-audit's byte-identical copy of it was not.
|
|
# providers/**/*.sh was excluded until issue #96: statusline-command.sh seeded
|
|
# `parts=()` empty and expanded it unguarded, a real latent hazard rather than a
|
|
# false positive, in a file this case did not own. That expansion is now guarded,
|
|
# so the glob is in the table below and the deployed provider scripts get the
|
|
# same coverage as scripts/, tests/ and plugins/*/.apm/. This is the glob most
|
|
# worth having: install.sh copies providers/ content onto every user machine, and
|
|
# it is also the glob most likely to look redundant to a future reader, because
|
|
# the hazard it catches is invisible on any modern bash — bash 4.4 stopped
|
|
# treating an empty-array expansion as unbound, so a bare "${a[@]}" under `set -u`
|
|
# runs clean on the maintainer's bash 5 and aborts only on macOS's bash 3.2.
|
|
# A runtime test cannot demonstrate that without a 3.2 binary; this static scan
|
|
# is the enforcement, which is why its floor must never be dropped to zero.
|
|
#
|
|
# Four constructs are checked, because the expansion scan cannot see any of the
|
|
# other three:
|
|
# - `mapfile`/`readarray` — bash 4.0+ builtins. run-tests.sh carried one until
|
|
# it was replaced with a `while read` loop.
|
|
# - `declare -A` — bash 4.0+ associative arrays. check-vale-style-sync.sh
|
|
# carried a pair until they became index-scanned plain arrays.
|
|
# - `wait -n` — bash 4.3+. A prior review round found this live in run-bats.sh.
|
|
# - `nproc` — GNU coreutils, absent on macOS entirely. Same review round, same
|
|
# file. `getconf _NPROCESSORS_ONLN` is the portable spelling batch-run.sh
|
|
# settled on.
|
|
# The last two had no static guard anywhere before this, and
|
|
# `shellcheck --severity=warning` (.pre-commit-config.yaml) pins no target
|
|
# version, so it does not catch them either — meaning the guard against the last
|
|
# regression would not have caught the last regression.
|
|
echo ""
|
|
echo "--- no bash-4-only construct remains in the macOS-facing scripts ---"
|
|
# Files a script pulls in via `source`, as named by its `# shellcheck source=`
|
|
# directives. Array seeding often lives in the sourced file (install.sh's
|
|
# DEPLOY_* come from deploy-manifest.sh), so the seeding check has to look
|
|
# there too or it reports a false hazard. shellcheck resolves these paths
|
|
# against either the repo root or the script's own directory depending on
|
|
# configuration, so both are tried and whichever exists is used.
|
|
sourced_files() {
|
|
local file="$1" rel cand
|
|
while IFS= read -r rel; do
|
|
for cand in "$REPO_ROOT/$rel" "$(dirname "$file")/$rel"; do
|
|
if [[ -f "$cand" ]]; then
|
|
printf '%s\n' "$cand"
|
|
break
|
|
fi
|
|
done
|
|
done < <(
|
|
grep -oE '^[[:space:]]*#[[:space:]]*shellcheck[[:space:]]+source=[^[:space:]]+' "$file" \
|
|
| sed -E 's/.*source=//' || true
|
|
)
|
|
return 0
|
|
}
|
|
unguarded_expansions() {
|
|
local file="$1" hit name seed_file
|
|
local seed_files
|
|
seed_files=("$file")
|
|
while IFS= read -r seed_file; do
|
|
seed_files+=("$seed_file")
|
|
done < <(sourced_files "$file")
|
|
while IFS= read -r hit; do
|
|
name="$(printf '%s\n' "$hit" \
|
|
| { grep -oE '\$\{[A-Za-z_][A-Za-z0-9_]*\[@\]\}' || true; } | head -1 \
|
|
| sed -E 's/^\$\{//; s/\[@\]\}$//')"
|
|
# Shell-maintained arrays are never seeded by a `NAME=(...)` line, so the
|
|
# seeding exemption below can never clear them: without this case they are
|
|
# permanent false positives. Exempted are the ones measured non-empty inside
|
|
# a running script -- element counts taken at a script's top level:
|
|
# PIPESTATUS >=1 once any command has run (0 only before the very first,
|
|
# where the variable is meaningless anyway)
|
|
# BASH_SOURCE 1 (one frame per sourced/executed file)
|
|
# BASH_LINENO 1 (maintained in parallel with BASH_SOURCE)
|
|
# BASH_VERSINFO 6 (always exactly six)
|
|
# GROUPS 1
|
|
# DIRSTACK 1 (always holds at least the current directory)
|
|
#
|
|
# FUNCNAME, BASH_ARGV, BASH_ARGC, BASH_REMATCH and COMP_WORDS are
|
|
# deliberately NOT exempted despite being shell-maintained, because they are
|
|
# genuinely empty in reachable states: FUNCNAME is 0 outside a function,
|
|
# BASH_ARGV is 0 without `shopt -s extdebug`, BASH_ARGC is 0 *inside a
|
|
# function* (it looks safe when measured at top level, where it is 1 -- it is
|
|
# not), BASH_REMATCH is 0 until a `=~` match succeeds, COMP_WORDS is 0
|
|
# outside completion. Expanding any of those bare really does abort on bash
|
|
# 3.2 under `set -u`, so flagging them is the correct answer rather than a
|
|
# false positive. Case 26 pins this split so neither half drifts.
|
|
case "$name" in
|
|
PIPESTATUS|BASH_SOURCE|BASH_LINENO|BASH_VERSINFO|GROUPS|DIRSTACK) continue ;;
|
|
esac
|
|
# Same-line emptiness short-circuit: the expansion cannot be reached empty.
|
|
if grep -qE "\\\$\{#$name\[@\]\}[[:space:]]*-(eq|lt)[[:space:]]*[01][^|]*\|\|" <<< "$hit"; then
|
|
continue
|
|
fi
|
|
for seed_file in ${seed_files[@]+"${seed_files[@]}"}; do
|
|
# `([^)]|$)` after the paren, not just `[^)]`: a multi-line declaration
|
|
# (`DEPLOY_FILES=(` with its elements on the following lines) ends the line
|
|
# right there, and requiring a character after the paren missed it. An
|
|
# empty `name=()` still does not match, which is what the check is for.
|
|
if grep -qE "^[[:space:]]*((local|declare|readonly)[[:space:]]+)?(-a[[:space:]]+)?$name=\(([^)]|$)" "$seed_file" \
|
|
&& ! grep -qE "^[[:space:]]*$name=\(\)" "$seed_file"; then
|
|
continue 2
|
|
fi
|
|
done
|
|
printf '%s:%s\n' "${file##*/}" "$hit"
|
|
done < <(
|
|
# Blank out whole-line comments (keeping line numbers), delete every
|
|
# correctly guarded expansion — in both its bare spelling and the
|
|
# backslash-escaped one that appears inside this file's own PASS message —
|
|
# then anything still matching is a candidate.
|
|
awk '{ if ($0 ~ /^[[:space:]]*#/) print ""; else print }' "$file" \
|
|
| sed -E 's/\\?\$\{([A-Za-z_][A-Za-z0-9_]*)\[@\]\+\\?"\\?\$\{\1\[@\]\}\\?"\}//g' \
|
|
| grep -nE '\$\{[A-Za-z_][A-Za-z0-9_]*\[@\]\}' || true
|
|
)
|
|
}
|
|
# Blanks WHOLE-LINE comments only (keeping line numbers so hits stay locatable),
|
|
# so prose naming a hazard on its own line is not a hit.
|
|
#
|
|
# KNOW THIS BEFORE YOU EDIT ANY SCANNED FILE. Nothing else is stripped — this is
|
|
# a line-based scanner, not a shell parser — so all of the following DO trip the
|
|
# case even though none is a real hazard:
|
|
# true # avoid nproc on macOS <- trailing comment naming a hazard
|
|
# echo "avoid mapfile in scripts" <- hazard named inside a string
|
|
# <<EOF ... nproc ... EOF <- hazard named in a heredoc body
|
|
# foo # see ${x[@]} <- trailing comment holding an expansion
|
|
# Teaching it to parse shell would cost far more than it returns, so the rule is:
|
|
# put the mention on its own comment line, or break the token with a one-character
|
|
# bracket class the way the `npro[c]` rule below does — `npro[c]` matches exactly
|
|
# what a bare spelling would while containing no bare spelling itself. This
|
|
# matters because a static check that cries wolf is a static check someone
|
|
# eventually deletes.
|
|
strip_comments() { awk '{ if ($0 ~ /^[[:space:]]*#/) print ""; else print }' "$1"; }
|
|
bash32_glob() {
|
|
case "$1" in
|
|
scripts) find "$REPO_ROOT/scripts" -name '*.sh' -type f ;;
|
|
tests) find "$REPO_ROOT/tests" -maxdepth 1 -name '*.sh' -type f ;;
|
|
plugins) find "$REPO_ROOT/plugins" -path '*/.apm/*' -name '*.sh' -type f ;;
|
|
providers) find "$REPO_ROOT/providers" -name '*.sh' -type f ;;
|
|
*) echo "bash32_glob: unknown glob '$1'" >&2; return 1 ;;
|
|
esac
|
|
}
|
|
# Floors are PER GLOB, not on the merged total. A single total floor cannot
|
|
# detect the failure this assertion exists to name: with 12 + 18 + 13 + 1 files,
|
|
# losing the whole `scripts` glob still leaves 32 and losing the whole `tests`
|
|
# glob still leaves 26, so any total floor low enough to survive normal churn
|
|
# is too low to notice an entire glob silently resolving to nothing. Each floor
|
|
# sits a little under its current count so ordinary file removal does not trip
|
|
# it, but a broken or renamed path does. Parallel arrays rather than an
|
|
# associative one — `declare -A` is bash 4.0+, which this very case forbids.
|
|
#
|
|
# `providers` is floored at 1 rather than "current count minus slack" because it
|
|
# holds exactly one file: any slack at all would be a floor of 0, which passes on
|
|
# a renamed or deleted directory and is precisely the silent-zero failure the
|
|
# per-glob floors exist to catch. A floor of 1 means removing the last provider
|
|
# script trips this assertion — correct, because at that point the glob is dead
|
|
# weight and should be deleted from the table deliberately, not left to pass
|
|
# vacuously.
|
|
#
|
|
# `scripts` dropped from 10 to 8 with ADR-0025: the merge retired
|
|
# scripts/check-vale-style-sync.sh and scripts/sync-vale-styles.sh, both of which
|
|
# existed only to keep two copies of the Vale config in step, leaving 9 files.
|
|
# The floor moves with the count on a deliberate deletion — it is a guard against
|
|
# a broken or renamed PATH resolving to nothing, never a headcount to maintain.
|
|
BASH32_GLOB_NAMES=(scripts tests plugins providers)
|
|
BASH32_GLOB_FLOORS=(8 14 10 1)
|
|
BASH32_SCRIPTS=()
|
|
BASH32_IDX=0
|
|
while [[ $BASH32_IDX -lt ${#BASH32_GLOB_NAMES[@]} ]]; do
|
|
BASH32_GLOB="${BASH32_GLOB_NAMES[$BASH32_IDX]}"
|
|
BASH32_FLOOR="${BASH32_GLOB_FLOORS[$BASH32_IDX]}"
|
|
BASH32_COUNT=0
|
|
while IFS= read -r BASH32_FOUND; do
|
|
BASH32_SCRIPTS+=("$BASH32_FOUND")
|
|
BASH32_COUNT=$((BASH32_COUNT + 1))
|
|
done < <(bash32_glob "$BASH32_GLOB" | sort)
|
|
if [[ $BASH32_COUNT -lt $BASH32_FLOOR ]]; then
|
|
fail "the bash-3.2 scan's '$BASH32_GLOB' glob derived $BASH32_COUNT file(s), under its floor of $BASH32_FLOOR — that path is wrong, so those scripts are silently unscanned"
|
|
fi
|
|
BASH32_IDX=$((BASH32_IDX + 1))
|
|
done
|
|
HAZARDS16=""
|
|
for BASH32_SCRIPT in ${BASH32_SCRIPTS[@]+"${BASH32_SCRIPTS[@]}"}; do
|
|
FOUND16="$(unguarded_expansions "$BASH32_SCRIPT")"
|
|
if [[ -n "$FOUND16" ]]; then
|
|
HAZARDS16+="$FOUND16 "
|
|
fi
|
|
# `mapfile`/`readarray`: bash 4.0+ builtins with no 3.2 fallback.
|
|
# `declare -A`: bash 4.0+ associative arrays. The flag cluster can carry other
|
|
# letters in any order (-Ag, -rA, ...); what matters is a literal uppercase A
|
|
# appearing in it, so match on that rather than the exact string "-A".
|
|
# `wait -n`: bash 4.3+. `nproc`: GNU coreutils, not present on macOS.
|
|
# The last two close on `[^[:alnum:]_]`, not `[[:space:]]`: the real spellings
|
|
# are `$(nproc)` and `wait -n;`, and a whitespace-only terminator misses both.
|
|
# `npro[c]` matches exactly what `nproc` would, but keeps the literal string
|
|
# "nproc" out of this file — the scan reads this file too, so a bare spelling
|
|
# here would report itself as a hazard.
|
|
for BASH32_RULE in \
|
|
'(^|[^[:alnum:]_])(mapfile|readarray)[[:space:]]' \
|
|
'(^|[^[:alnum:]_])declare[[:space:]]+-[a-zA-Z]*A[a-zA-Z]*([[:space:]]|$)' \
|
|
'(^|[^[:alnum:]_])wait[[:space:]]+-n([^[:alnum:]_]|$)' \
|
|
'(^|[^[:alnum:]_])npro[c]([^[:alnum:]_]|$)'; do
|
|
FOUND16B="$(strip_comments "$BASH32_SCRIPT" | grep -nE "$BASH32_RULE" || true)"
|
|
if [[ -n "$FOUND16B" ]]; then
|
|
HAZARDS16+="${BASH32_SCRIPT##*/}:$FOUND16B "
|
|
fi
|
|
done
|
|
done
|
|
if [[ -n "$HAZARDS16" ]]; then
|
|
fail "bash-4-only construct(s) found in ${#BASH32_SCRIPTS[@]} scanned script(s): $(echo "$HAZARDS16" | tr '\n' ' ')"
|
|
else
|
|
# This message deliberately names none of the four hazards in their literal
|
|
# spelling: the scan reads this file too, so a literal here is indistinguishable
|
|
# from a real occurrence and the case would fail on its own success message.
|
|
pass "all ${#BASH32_SCRIPTS[@]} scanned scripts are free of every bash-4-only construct this case checks for"
|
|
fi
|
|
|
|
# ===== BEGIN VALE-DEPENDENT REGION (cases 17-18) =============================
|
|
if [[ "$VALE_READY" == true ]]; then
|
|
|
|
# --- 17. The invocations whose arrays are closest to empty actually run. Under
|
|
# a bash older than 4.4 this is genuine macOS-shell coverage; on a modern bash it
|
|
# degrades to a smoke test, so the pass message names the shell that really ran.
|
|
# Point VALE_WRAP_TEST_BASH at a 3.2 build to get the real thing in CI.
|
|
echo ""
|
|
echo "--- degenerate invocations survive on the oldest available bash ---"
|
|
OLD_BASH="bash"
|
|
OLD_BASH_VER="$(bash -c 'echo "${BASH_VERSINFO[0]}.${BASH_VERSINFO[1]}"')"
|
|
for CAND in "${VALE_WRAP_TEST_BASH:-}" bash-3.2 bash3 /bin/bash /usr/local/bin/bash; do
|
|
[[ -n "$CAND" ]] && command -v "$CAND" >/dev/null 2>&1 || continue
|
|
CAND_VER="$("$CAND" -c 'echo "${BASH_VERSINFO[0]}.${BASH_VERSINFO[1]}"' 2>/dev/null)" || continue
|
|
[[ -n "$CAND_VER" ]] || continue
|
|
if (( ${CAND_VER%.*} * 100 + ${CAND_VER#*.} < ${OLD_BASH_VER%.*} * 100 + ${OLD_BASH_VER#*.} )); then
|
|
OLD_BASH="$CAND"
|
|
OLD_BASH_VER="$CAND_VER"
|
|
fi
|
|
done
|
|
FIXTURE17="$(make_fixture 2)"
|
|
mkdir -p "$FIXTURE17/emptydir"
|
|
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7" "$FIXTURE8" "$FIXTURE10" "$FIXTURE11" "$FIXTURE12" "$STUB13" "$FIXTURE14" "$FIXTURE17"' EXIT
|
|
# Zero args, flags with no path, and a directory that walks to nothing are the
|
|
# three shapes that leave vale_args/path_args/argv_paths at their emptiest.
|
|
OUT17=""
|
|
for ARGS17 in "" "--config $VALE_CONFIG" "--config $VALE_CONFIG emptydir"; do
|
|
# shellcheck disable=SC2086 # deliberate word splitting of the argv fixture
|
|
OUT17+="$( (cd "$FIXTURE17" && "$OLD_BASH" "$SCRIPT" $ARGS17 </dev/null 2>&1) || true)"
|
|
done
|
|
if grep -q "unbound variable" <<< "$OUT17"; then
|
|
fail "aborted with 'unbound variable' on bash $OLD_BASH_VER — the bug this test guards against"
|
|
else
|
|
pass "degenerate invocations run clean under bash $OLD_BASH_VER ($OLD_BASH)"
|
|
fi
|
|
|
|
# --- 18. The guarded expansion must keep argv word boundaries intact. Dropping
|
|
# the quotes (`${arr[@]}`) also silences the unbound-variable abort, so it is the
|
|
# tempting wrong fix — and it splits any path containing a space into two bogus
|
|
# arguments. Case 15 covers spaces found by the directory walk; this covers a
|
|
# space in the path argument itself, which is what argv_paths expands.
|
|
echo ""
|
|
echo "--- a path argument containing a space survives the guarded expansion ---"
|
|
FIXTURE18="$(make_fixture 2)"
|
|
trap 'rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" "$FIXTURE7" "$FIXTURE8" "$FIXTURE10" "$FIXTURE11" "$FIXTURE12" "$STUB13" "$FIXTURE14" "$FIXTURE17" "$FIXTURE18"' EXIT
|
|
SPACED18="$FIXTURE18/plugins/testplugin/skills/zzz skill dir"
|
|
mkdir -p "$SPACED18"
|
|
mv "$FIXTURE18/plugins/testplugin/skills/zzzskill/SKILL.md" "$SPACED18/SKILL.md"
|
|
OUT18=$(run_wrap "$FIXTURE18" --config "$VALE_CONFIG" "plugins/testplugin/skills/zzz skill dir/SKILL.md")
|
|
if grep -q "zzz skill dir/SKILL.md" <<< "$OUT18" && grep -q "VagueWording" <<< "$OUT18"; then
|
|
pass "a path argument with a space is passed to vale as one word"
|
|
else
|
|
fail "a path argument with a space was split by the array expansion: $OUT18"
|
|
fi
|
|
|
|
fi
|
|
# ===== END VALE-DEPENDENT REGION (cases 17-18) ===============================
|
|
|
|
# The cases below share one cleanup list. The per-case trap rebuilding above
|
|
# does not scale past the fixture count it already carries, and this trap is
|
|
# installed last, so it is the one that runs.
|
|
EXTRA_FIXTURES=()
|
|
new_fixture() { EXTRA_FIXTURES+=("$1"); }
|
|
cleanup_all() {
|
|
rm -rf "$FIXTURE1" "$FIXTURE2" "$FIXTURE3" "$FIXTURE4" "$FIXTURE5" "$FIXTURE6" \
|
|
"$FIXTURE7" "$FIXTURE8" "$FIXTURE10" "$FIXTURE11" "$FIXTURE12" "$STUB13" \
|
|
"$FIXTURE14" "$FIXTURE17" "$FIXTURE18" \
|
|
${EXTRA_FIXTURES[@]+"${EXTRA_FIXTURES[@]}"}
|
|
}
|
|
trap cleanup_all EXIT
|
|
|
|
# ===== BEGIN VALE-DEPENDENT REGION (cases 19-25) =============================
|
|
# Case 25 is static, but it audits the fixtures case 19 creates, so it cannot
|
|
# run without 19 and is held back with it.
|
|
if [[ "$VALE_READY" == true ]]; then
|
|
|
|
# --- 19. Every YAML form whose parsed value is joined back out of 2+ physical
|
|
# lines breaks the `text.frontmatter.description` scope identically, not just
|
|
# the `>` folded block the flattener originally handled: a plain scalar wrapped
|
|
# onto continuation lines, a double-quoted one, a single-quoted one, and a bare
|
|
# `description:` whose value starts on the next line all report zero alerts
|
|
# under bare vale. Each must come back with the same alerts as the single-line
|
|
# spelling of the same sentence. Line and column numbers legitimately move (the
|
|
# value lands on one physical line), so the comparison drops the `line:col`
|
|
# prefix and compares the alert text — message, matched token, and rule name.
|
|
REL_SKILL19="plugins/testplugin/skills/zzzskill/SKILL.md"
|
|
DESC19_A="Use when the caller helps with a specific job"
|
|
DESC19_B="and the second physical line will utilize the wrap"
|
|
|
|
# make_form_fixture spells the same two-clause description in one YAML scalar
|
|
# form: single, folded, plain, dquote, squote, or keyonly. It prints the fixture
|
|
# dir and does NOT call new_fixture itself: every caller invokes it inside `$( )`,
|
|
# so a registration made in here would land in the command substitution's
|
|
# subshell and never reach cleanup_all -- which leaked all six fixtures per run.
|
|
# Registration is therefore the caller's job, in the caller's shell.
|
|
make_form_fixture() {
|
|
local form="$1" dir
|
|
dir="$(mktemp -d)"
|
|
(cd "$dir" && git init -q)
|
|
mkdir -p "$dir/plugins/testplugin/skills/zzzskill"
|
|
{
|
|
echo "---"
|
|
echo "name: zzzskill"
|
|
case "$form" in
|
|
single) echo "description: $DESC19_A $DESC19_B" ;;
|
|
folded) echo "description: >"; echo " $DESC19_A"; echo " $DESC19_B" ;;
|
|
plain) echo "description: $DESC19_A"; echo " $DESC19_B" ;;
|
|
dquote) echo "description: \"$DESC19_A"; echo " $DESC19_B\"" ;;
|
|
squote) echo "description: '$DESC19_A"; echo " $DESC19_B'" ;;
|
|
keyonly) echo "description:"; echo " $DESC19_A"; echo " $DESC19_B" ;;
|
|
*) echo "make_form_fixture: unknown form '$form'" >&2; exit 1 ;;
|
|
esac
|
|
echo "---"
|
|
echo ""
|
|
echo "Body."
|
|
} > "$dir/plugins/testplugin/skills/zzzskill/SKILL.md"
|
|
echo "$dir"
|
|
}
|
|
|
|
# Alert text with the `line:col` prefix and ANSI colouring stripped, sorted.
|
|
# The `|| true` matters under this file's `set -o pipefail`: a report with no
|
|
# alerts at all makes grep exit 1, which would abort the whole run inside the
|
|
# command substitutions below — silently, before the empty-baseline guard could
|
|
# print anything. Returning empty output instead is what makes that guard
|
|
# reachable.
|
|
alert_text() {
|
|
echo "$1" \
|
|
| sed -E 's/\x1b\[[0-9;]*m//g' \
|
|
| { grep -oE '(error|warning|suggestion)[[:space:]]+.*' || true; } \
|
|
| sed -E 's/[[:space:]]+/ /g' \
|
|
| sort
|
|
}
|
|
|
|
echo ""
|
|
echo "--- every multi-line description form reports what its single-line form reports ---"
|
|
# FORM_FIXTURES19 is bookkeeping for case 25, which asserts every dir recorded
|
|
# here also reached EXTRA_FIXTURES. It is appended to independently of
|
|
# new_fixture so that dropping the new_fixture calls is detectable.
|
|
FORM_FIXTURES19=()
|
|
FIXTURE19_SINGLE="$(make_form_fixture single)"
|
|
FORM_FIXTURES19+=("$FIXTURE19_SINGLE")
|
|
new_fixture "$FIXTURE19_SINGLE"
|
|
BASELINE19="$(alert_text "$(run_wrap "$FIXTURE19_SINGLE" --config "$VALE_CONFIG" "$REL_SKILL19")")"
|
|
if [[ -z "$BASELINE19" ]]; then
|
|
# The loop below has to be skipped, not merely reported on: an empty baseline
|
|
# compares equal to five empty results, so it would print five vacuous PASSes
|
|
# alongside this one FAIL. The FAIL alone still fails the run at the end.
|
|
fail "the single-line baseline reported nothing — the comparisons below would be vacuous, so they are skipped"
|
|
else
|
|
for FORM19 in folded plain dquote squote keyonly; do
|
|
DIR19="$(make_form_fixture "$FORM19")"
|
|
FORM_FIXTURES19+=("$DIR19")
|
|
new_fixture "$DIR19"
|
|
BARE19="$(cd "$DIR19" && vale --config "$VALE_CONFIG" "$REL_SKILL19" 2>&1 || true)"
|
|
GOT19="$(alert_text "$(run_wrap "$DIR19" --config "$VALE_CONFIG" "$REL_SKILL19")")"
|
|
if grep -q "VagueWording" <<< "$BARE19"; then
|
|
fail "bare vale already flags the $FORM19 form, so this case can't detect a silently-skipped flattening"
|
|
elif [[ "$GOT19" == "$BASELINE19" ]]; then
|
|
pass "a $FORM19 multi-line description reports the same alerts as its single-line form"
|
|
else
|
|
fail "a $FORM19 multi-line description diverged from its single-line form: got [$GOT19]"
|
|
fi
|
|
done
|
|
fi
|
|
|
|
# --- 20. A style token containing an ASCII apostrophe matches inside a
|
|
# flattened description. The flattener used to substitute U+2019 for every `'`
|
|
# before writing the scratch copy, so no rule whose token carried an apostrophe
|
|
# could ever fire on a flattened description — a silent, rule-shaped blind spot.
|
|
# All three branches that can hold an apostrophe are exercised: a value that is
|
|
# safe unquoted; one that must be quoted (it contains `: `) and so lands in a
|
|
# double-quoted scalar, since a single-quoted one would need the `''` escape
|
|
# that kills the scope outright; and one that also holds a double quote, which
|
|
# no inline scalar can spell verbatim and which therefore lands in a `|-`
|
|
# literal block.
|
|
echo ""
|
|
echo "--- a style token containing an apostrophe matches in a flattened description ---"
|
|
APOS_STYLE="$(mktemp -d)"
|
|
new_fixture "$APOS_STYLE"
|
|
mkdir -p "$APOS_STYLE/styles/Apostrophe"
|
|
cat > "$APOS_STYLE/styles/Apostrophe/Token.yml" <<'EOF'
|
|
extends: existence
|
|
message: "apostrophe token: '%s'"
|
|
level: error
|
|
scope: text.frontmatter.description
|
|
ignorecase: true
|
|
tokens:
|
|
- "user's task"
|
|
EOF
|
|
# A body-scoped companion rule, used by case 20b to read back the line number of
|
|
# a line *after* the frontmatter — the only way to catch the blank-line pad
|
|
# being off in either direction.
|
|
cat > "$APOS_STYLE/styles/Apostrophe/Body.yml" <<'EOF'
|
|
extends: existence
|
|
message: "body token: '%s'"
|
|
level: error
|
|
scope: text
|
|
tokens:
|
|
- flattening marker phrase
|
|
EOF
|
|
cat > "$APOS_STYLE/.vale.ini" <<'EOF'
|
|
StylesPath = styles
|
|
|
|
[**/SKILL.md]
|
|
BasedOnStyles = Apostrophe
|
|
EOF
|
|
FIXTURE20_PLAIN="$(make_raw_fixture <<'EOF'
|
|
---
|
|
name: zzzskill
|
|
description: >
|
|
Use when the user's task needs handling, and a second physical
|
|
line continues the folded scalar.
|
|
---
|
|
|
|
Body.
|
|
EOF
|
|
)"
|
|
new_fixture "$FIXTURE20_PLAIN"
|
|
FIXTURE20_QUOTED="$(make_raw_fixture <<'EOF'
|
|
---
|
|
name: zzzskill
|
|
description: >
|
|
Triggers on: the user's task needing handling, and a second
|
|
physical line continues the folded scalar.
|
|
---
|
|
|
|
Body.
|
|
EOF
|
|
)"
|
|
new_fixture "$FIXTURE20_QUOTED"
|
|
# Needs quoting (`: `), holds an apostrophe AND a double quote — the one
|
|
# combination no inline scalar can carry, so this is the `|-` literal-block
|
|
# branch. The VagueWording tokens are there for case 20b, which reuses it.
|
|
FIXTURE20_BLOCK="$(make_raw_fixture <<'EOF'
|
|
---
|
|
name: zzzskill
|
|
description: >
|
|
Triggers on: the user's task and "audit this" phrasing, which helps
|
|
with and utilize things across a second physical line.
|
|
---
|
|
|
|
Body carrying a flattening marker phrase for the line-number check.
|
|
EOF
|
|
)"
|
|
new_fixture "$FIXTURE20_BLOCK"
|
|
for CASE20 in "unquoted:$FIXTURE20_PLAIN" "double-quoted:$FIXTURE20_QUOTED" \
|
|
"literal-block:$FIXTURE20_BLOCK"; do
|
|
if run_wrap "${CASE20#*:}" --config "$APOS_STYLE/.vale.ini" "$REL_SKILL19" \
|
|
| grep -q "Apostrophe.Token"; then
|
|
pass "an apostrophe-bearing token matches in a flattened ${CASE20%%:*} description"
|
|
else
|
|
fail "an apostrophe-bearing token was rewritten out of a flattened ${CASE20%%:*} description"
|
|
fi
|
|
done
|
|
|
|
# --- 20b. The `|-` literal-block branch that case 20 just proved lossless must
|
|
# also keep the rest of the scope working and keep the line accounting right.
|
|
# The block is 2 physical lines where every inline form is 1, so the blank-line
|
|
# pad that preserves later line numbers has to drop by one. The second
|
|
# assertion pins that arithmetic against the body line's true number: case 3's
|
|
# `<= original line count` bound would not, since a pad that is one line short
|
|
# shifts every later line *up*, staying inside the bound while still lying.
|
|
echo ""
|
|
echo "--- the |- literal-block fallback lints normally and preserves line numbers ---"
|
|
OUT20B=$(run_wrap "$FIXTURE20_BLOCK" --config "$VALE_CONFIG" "$REL_SKILL19")
|
|
if grep -q "VagueWording" <<< "$OUT20B"; then
|
|
pass "a description needing quotes with both an apostrophe and a double quote is still linted"
|
|
else
|
|
fail "a description needing quotes with both an apostrophe and a double quote produced no alerts"
|
|
fi
|
|
WANT20B_LINE="$(grep -n 'flattening marker phrase' "$FIXTURE20_BLOCK/$REL_SKILL19" | cut -d: -f1)"
|
|
# `--output line` prints `file:line:col:Rule:message`, so the line number reads
|
|
# back without any wrapping or colour to strip.
|
|
GOT20B_LINE="$(run_wrap "$FIXTURE20_BLOCK" --config "$APOS_STYLE/.vale.ini" --output line "$REL_SKILL19" \
|
|
| { grep 'Apostrophe.Body' || true; } | head -1 | cut -d: -f2)"
|
|
if [[ "$GOT20B_LINE" == "$WANT20B_LINE" ]]; then
|
|
pass "a body line after a |- flattened description keeps its original line number ($WANT20B_LINE)"
|
|
else
|
|
fail "the |- block's blank-line pad shifted the body: vale reported line $GOT20B_LINE, the file has it at $WANT20B_LINE"
|
|
fi
|
|
|
|
# --- 21. A symlinked file inside a directory argument is mirrored and linted.
|
|
# Vale follows symlinks (both a symlinked file and a file under a symlinked
|
|
# directory), so a `-type f` walk of the tree reported "0 files" where bare vale
|
|
# reports one — and the audit skills read a "0 files" report as NOT RUN.
|
|
echo ""
|
|
echo "--- mirrors a symlinked file reached through a directory argument ---"
|
|
FIXTURE21="$(make_fixture 2)"
|
|
new_fixture "$FIXTURE21"
|
|
mkdir -p "$FIXTURE21/real"
|
|
mv "$FIXTURE21/$REL_SKILL19" "$FIXTURE21/real/SKILL.md"
|
|
ln -s ../../../../real/SKILL.md "$FIXTURE21/$REL_SKILL19"
|
|
BARE21="$(cd "$FIXTURE21" && vale --config "$VALE_CONFIG" plugins 2>&1 || true)"
|
|
WRAPPED21="$(run_wrap "$FIXTURE21" --config "$VALE_CONFIG" plugins)"
|
|
BARE21_FILES="$(echo "$BARE21" | sed -E 's/\x1b\[[0-9;]*m//g' | grep -oE 'in [0-9]+ files?' | tail -1)"
|
|
WRAPPED21_FILES="$(echo "$WRAPPED21" | sed -E 's/\x1b\[[0-9;]*m//g' | grep -oE 'in [0-9]+ files?' | tail -1)"
|
|
if [[ "$BARE21_FILES" != "in 1 file" ]]; then
|
|
fail "bare vale did not lint the symlinked file ($BARE21_FILES), so this case can't detect the walk dropping it"
|
|
elif [[ "$WRAPPED21_FILES" != "$BARE21_FILES" ]]; then
|
|
fail "the directory walk dropped a symlinked file: wrapper saw '$WRAPPED21_FILES', bare vale '$BARE21_FILES'"
|
|
# A here-string, not `echo "$WRAPPED21" | grep -q`: the match sits on line 3 of
|
|
# 8, and under pipefail grep -q exiting early can SIGPIPE echo mid-write and
|
|
# fail this branch on correct output (see tests/test-check-release-needed.sh).
|
|
elif grep -q "VagueWording" <<< "$WRAPPED21"; then
|
|
pass "a symlinked file under a directory argument is mirrored, flattened and flagged"
|
|
else
|
|
fail "a symlinked file was mirrored but not flattened — no alert came back"
|
|
fi
|
|
|
|
# --- 22. The value of a separated two-argv flag is never treated as a lint
|
|
# target, however file-like it looks. `--output tmpl.tmpl` names a real
|
|
# template file: classifying it as input both linted the template and reordered
|
|
# argv, so vale received `--output --no-wrap` and died on `open :`.
|
|
echo ""
|
|
echo "--- a separated flag value that names a real file is not linted as a target ---"
|
|
FIXTURE22="$(make_fixture 1)"
|
|
new_fixture "$FIXTURE22"
|
|
printf 'TMPL{{range .Files}} {{.Path}}{{end}}\n' > "$FIXTURE22/tmpl.tmpl"
|
|
WRAPPED22="$(run_wrap "$FIXTURE22" --config "$VALE_CONFIG" --output tmpl.tmpl --no-wrap "$REL_SKILL19")"
|
|
BARE22="$(cd "$FIXTURE22" && vale --config "$VALE_CONFIG" --output tmpl.tmpl --no-wrap "$REL_SKILL19" 2>&1 || true)"
|
|
# The fixture's description is a single physical line, so flattening is a no-op
|
|
# and the two invocations must agree byte for byte.
|
|
if [[ "$WRAPPED22" == "$BARE22" ]]; then
|
|
pass "a separated --output value is passed through to vale, not linted"
|
|
else
|
|
fail "a separated --output value was misrouted: wrapper gave [$WRAPPED22], bare vale [$BARE22]"
|
|
fi
|
|
|
|
# --- 23. A path argument that does not exist is a hard error. Bare vale drops
|
|
# it, falls back to stdin and prints `0 errors ... in stdin` with exit 0, so a
|
|
# typo'd target is indistinguishable from a clean run — and the audit skills'
|
|
# NOT RUN guard string-matches `0 files`, which `in stdin` never produces. This
|
|
# is a deliberate divergence from bare vale, documented in the wrapper header.
|
|
echo ""
|
|
echo "--- a nonexistent path argument fails loudly instead of falling back to stdin ---"
|
|
set +e
|
|
OUT23="$(cd "$FIXTURE22" && bash "$SCRIPT" --config "$VALE_CONFIG" plugins/testplugin/skills/zzzskill/SKILLL.md 2>&1)"
|
|
RC23=$?
|
|
set -e
|
|
if [[ $RC23 -eq 0 ]]; then
|
|
fail "a typo'd path exited 0 — indistinguishable from a clean run, the bug this test guards against"
|
|
elif grep -q "in stdin" <<< "$OUT23"; then
|
|
fail "a typo'd path fell back to reading stdin and reported 'in stdin' instead of erroring"
|
|
elif grep -q "SKILLL.md" <<< "$OUT23"; then
|
|
pass "a typo'd path exits nonzero with a message naming the path"
|
|
else
|
|
fail "a typo'd path exited $RC23 but the message does not name it: $OUT23"
|
|
fi
|
|
|
|
# --- 24. `--output`'s built-in style names must not be path-absolutized. The
|
|
# wrapper rewrites path-valued flag values to absolute form so they still
|
|
# resolve after the `cd` into the scratch mirror, deciding with an `-e`
|
|
# existence test — but `line`, `JSON` and `CLI` are style names, not paths. With
|
|
# a file or directory of that name sitting in the caller's cwd the test hit, the
|
|
# built-in became `$cwd/line`, and vale flipped into template mode and died with
|
|
# `E100 [template] Runtime error` where bare vale prints a normal report.
|
|
echo ""
|
|
echo "--- a built-in --output style name survives a same-named entry in the cwd ---"
|
|
FIXTURE24="$(make_fixture 2)"
|
|
new_fixture "$FIXTURE24"
|
|
mkdir -p "$FIXTURE24/line"
|
|
: > "$FIXTURE24/JSON"
|
|
for FORM24 in "--output line" "--output=line" "--output JSON" "--output=JSON"; do
|
|
# shellcheck disable=SC2086 # deliberate word splitting of the argv fixture
|
|
OUT24="$(run_wrap "$FIXTURE24" --config "$VALE_CONFIG" $FORM24 "$REL_SKILL19")"
|
|
if grep -q "E100" <<< "$OUT24"; then
|
|
fail "'$FORM24' was rewritten to a cwd path and vale flipped into template mode — the bug this test guards against"
|
|
elif grep -q "VagueWording" <<< "$OUT24"; then
|
|
pass "'$FORM24' is passed through as a built-in style name"
|
|
else
|
|
fail "'$FORM24' produced no alert: $OUT24"
|
|
fi
|
|
done
|
|
|
|
# --- 25. Every fixture this run created is on the cleanup list. make_form_fixture
|
|
# used to call `new_fixture` itself, but every caller invokes it as
|
|
# `$(make_form_fixture ...)` — a command substitution — so the append landed in a
|
|
# subshell and was gone by the time the caller resumed. cleanup_all then removed
|
|
# five of the six form fixtures it never heard about, leaking one temp dir per
|
|
# YAML form per run (six in total, measured under a private TMPDIR).
|
|
#
|
|
# Nothing in the suite noticed: a leak fails no assertion, and the run reported
|
|
# "39 passed, 0 failed" with the bug present exactly as it does with the bug
|
|
# fixed. Leak-freeness was only ever observable by watching TMPDIR from outside,
|
|
# which is not a regression test. This case makes it one — it compares the dirs
|
|
# case 19 created against the dirs registered for cleanup, so re-introducing the
|
|
# subshell registration fails the run rather than quietly leaking again.
|
|
echo ""
|
|
echo "--- every fixture created by this run is registered for cleanup ---"
|
|
UNREGISTERED25=""
|
|
for DIR25 in ${FORM_FIXTURES19[@]+"${FORM_FIXTURES19[@]}"}; do
|
|
REGISTERED25=0
|
|
for KNOWN25 in ${EXTRA_FIXTURES[@]+"${EXTRA_FIXTURES[@]}"}; do
|
|
if [[ "$KNOWN25" == "$DIR25" ]]; then
|
|
REGISTERED25=1
|
|
break
|
|
fi
|
|
done
|
|
if [[ $REGISTERED25 -eq 0 ]]; then
|
|
UNREGISTERED25+="$DIR25 "
|
|
fi
|
|
done
|
|
if [[ ${#FORM_FIXTURES19[@]} -ne 6 ]]; then
|
|
fail "expected 6 form fixtures to have been created, saw ${#FORM_FIXTURES19[@]} — case 25 is not checking what it claims"
|
|
elif [[ -n "$UNREGISTERED25" ]]; then
|
|
fail "fixture(s) created but never registered for cleanup, so they leak: $UNREGISTERED25"
|
|
else
|
|
pass "all ${#FORM_FIXTURES19[@]} form fixtures are registered for cleanup"
|
|
fi
|
|
|
|
fi
|
|
# ===== END VALE-DEPENDENT REGION (cases 19-25) ===============================
|
|
|
|
# --- 26. Case 16's shell-special-array exemption is pinned to a fixture. No file
|
|
# the scan currently reads expands any of those arrays, so the exemption is inert
|
|
# in practice: it could be deleted, or quietly widened to cover an array that can
|
|
# genuinely be empty, and every existing case would still pass. This drives the
|
|
# real unguarded_expansions over a fixture holding one expansion of each, and
|
|
# asserts the split in BOTH directions -- exempt arrays absent from the findings,
|
|
# never-safe arrays present. The membership is not cosmetic: the exempt ones are
|
|
# shell-maintained and always non-empty, while the flagged ones are empty in
|
|
# reachable states (see the comment on the exemption for the measured counts), so
|
|
# widening the list to include one of the latter would suppress a real abort.
|
|
echo ""
|
|
echo "--- the shell-special-array exemption covers exactly the never-empty arrays ---"
|
|
FIXTURE26="$(mktemp -d)"
|
|
new_fixture "$FIXTURE26"
|
|
EXEMPT26="PIPESTATUS BASH_SOURCE BASH_LINENO BASH_VERSINFO GROUPS DIRSTACK"
|
|
FLAGGED26="FUNCNAME BASH_ARGV BASH_ARGC BASH_REMATCH COMP_WORDS"
|
|
{
|
|
echo "#!/usr/bin/env bash"
|
|
for ARR26 in $EXEMPT26 $FLAGGED26; do
|
|
echo "echo \"\${${ARR26}[@]}\""
|
|
done
|
|
} > "$FIXTURE26/probe.sh"
|
|
FOUND26="$(unguarded_expansions "$FIXTURE26/probe.sh")"
|
|
MISSING26=""
|
|
LEAKED26=""
|
|
for ARR26 in $EXEMPT26; do
|
|
if grep -q "{$ARR26\[@\]}" <<< "$FOUND26"; then
|
|
LEAKED26+="$ARR26 "
|
|
fi
|
|
done
|
|
for ARR26 in $FLAGGED26; do
|
|
if ! grep -q "{$ARR26\[@\]}" <<< "$FOUND26"; then
|
|
MISSING26+="$ARR26 "
|
|
fi
|
|
done
|
|
if [[ -n "$LEAKED26" ]]; then
|
|
fail "always-non-empty shell array(s) reported as hazards, so the exemption is not applying: $LEAKED26"
|
|
elif [[ -n "$MISSING26" ]]; then
|
|
fail "shell array(s) that CAN be empty were exempted, suppressing a real bash-3.2 abort: $MISSING26"
|
|
else
|
|
pass "all 6 never-empty shell arrays are exempt and all 5 sometimes-empty ones are still flagged"
|
|
fi
|
|
|
|
# Case 27 pins the `sourced_files()` exemption against the failure mode that
|
|
# actually happened (issue #97 item 2): a stale source-directive path.
|
|
#
|
|
# The exemption exists because array seeding often lives in the sourced file
|
|
# rather than the sourcing one -- install.sh's DEPLOY_* come from
|
|
# deploy-manifest.sh -- so without it those expansions read as hazards. It works
|
|
# by parsing each file's own source-directive comments. That makes it silently
|
|
# dependent on those paths resolving: a directive naming a file that is not
|
|
# there does not error, it just contributes no seed file, and the exemption
|
|
# quietly stops covering what it was written to cover.
|
|
#
|
|
# tests/run-tests.sh shipped exactly that for the length of PR #95 -- a directive
|
|
# resolving to neither the repo root nor the script's own directory. Nothing
|
|
# caught it: shellcheck's own SC1091 is `info`, and .pre-commit-config.yaml pins
|
|
# `--severity=warning`. The live consequence was nil only because both of
|
|
# run-tests.sh's expansions were independently guarded.
|
|
#
|
|
# Part C is the assertion that would have caught it, and it is the reason this
|
|
# case is not just fixture theatre: it holds every real directive in the scanned
|
|
# corpus to the resolution rule, so the next stale one fails here rather than
|
|
# lying dormant. Parts A and B pin the mechanism Part C depends on -- that
|
|
# resolution is what switches the exemption on, and non-resolution silently
|
|
# switches it off.
|
|
echo ""
|
|
echo "--- every shellcheck source directive resolves, so no seeding exemption is silently off ---"
|
|
FIXTURE27="$(mktemp -d)"
|
|
new_fixture "$FIXTURE27"
|
|
# The seeded array lives ONLY in the sourced file, never in the sourcing one --
|
|
# that separation is the whole point of the exemption.
|
|
{
|
|
echo "#!/usr/bin/env bash"
|
|
echo "SEEDED27=(a b c)"
|
|
} > "$FIXTURE27/seed-lib.sh"
|
|
|
|
# Both fixture bodies are emitted through printf with the closing `]` passed as
|
|
# an argument, never written literally. This file is itself inside the `tests`
|
|
# glob, so a literal bare expansion here would be scanned as a hazard in
|
|
# test-vale-wrap.sh's own source -- the trailing-comment/inside-a-string caveat
|
|
# documented above strip_comments(). Splitting the token means the emitted
|
|
# fixture holds the real text while this file holds no bare spelling, the same
|
|
# trick the `npro[c]` rule uses.
|
|
EXPANSION27="$(printf 'echo "${SEEDED27[@%s}"' ']')"
|
|
|
|
# Part A: a directive that resolves against the script's own directory exempts
|
|
# the expansion, so no hazard is reported.
|
|
{
|
|
echo "#!/usr/bin/env bash"
|
|
echo "# shellcheck source=seed-lib.sh"
|
|
echo 'source "$(dirname "$0")/seed-lib.sh"'
|
|
echo "$EXPANSION27"
|
|
} > "$FIXTURE27/resolves.sh"
|
|
|
|
# Part B: byte-identical except the directive names a file that is not there.
|
|
{
|
|
echo "#!/usr/bin/env bash"
|
|
echo "# shellcheck source=nonexistent/seed-lib.sh"
|
|
echo 'source "$(dirname "$0")/seed-lib.sh"'
|
|
echo "$EXPANSION27"
|
|
} > "$FIXTURE27/stale.sh"
|
|
|
|
RESOLVES27="$(unguarded_expansions "$FIXTURE27/resolves.sh")"
|
|
STALE27="$(unguarded_expansions "$FIXTURE27/stale.sh")"
|
|
|
|
# Part C: no real directive in the derived corpus may fail to resolve. Counted
|
|
# rather than diffed, because sourced_files() reports resolution only by
|
|
# omission -- an unresolvable directive produces no output line at all.
|
|
UNRESOLVED27=""
|
|
for BASH32_SCRIPT in ${BASH32_SCRIPTS[@]+"${BASH32_SCRIPTS[@]}"}; do
|
|
DECLARED27="$(grep -cE '^[[:space:]]*#[[:space:]]*shellcheck[[:space:]]+source=[^[:space:]]+' "$BASH32_SCRIPT" || true)"
|
|
[[ "$DECLARED27" -gt 0 ]] || continue
|
|
RESOLVED27="$(sourced_files "$BASH32_SCRIPT" | grep -c . || true)"
|
|
if [[ "$RESOLVED27" -lt "$DECLARED27" ]]; then
|
|
UNRESOLVED27+="${BASH32_SCRIPT#"$REPO_ROOT"/} ($RESOLVED27/$DECLARED27) "
|
|
fi
|
|
done
|
|
|
|
if [[ -n "$RESOLVES27" ]]; then
|
|
fail "an array seeded only in a resolvable sourced file was reported as a hazard, so the seeding exemption is not reading source directives at all"
|
|
elif [[ -z "$STALE27" ]]; then
|
|
fail "an array seeded only in an UNRESOLVABLE sourced file was still exempted, so this case cannot detect a stale directive and Part C proves nothing"
|
|
elif [[ -n "$UNRESOLVED27" ]]; then
|
|
fail "shellcheck source directive(s) resolve to nothing, silently disarming the seeding exemption for those files: $UNRESOLVED27"
|
|
else
|
|
pass "the seeding exemption switches on only for resolvable directives, and every directive in the scanned corpus resolves"
|
|
fi
|
|
|
|
# --- 28-30. `.vale.ini` glob coverage ---------------------------------------
|
|
#
|
|
# Rehomed from scripts/check-vale-style-sync.sh, deleted by ADR-0025. That
|
|
# script's text-level assertions diffed skill-audit's Vale copy against
|
|
# agent-audit's and went moot when the merge into factory-audit left one copy.
|
|
# Its GLOB-COVERAGE probes did not. Merging the copies does not make a glob typo
|
|
# impossible -- a typo in any one of the three sections of the surviving
|
|
# `.vale.ini` still skips that file shape -- and this file, which already owns
|
|
# the wrapper's behaviour against that same config, is where they belong.
|
|
#
|
|
# The original comment, preserved because it names the failure mode rather than
|
|
# describing the check: nothing in the repo read the `.vale.ini` at all, "and
|
|
# that is what let a one-character glob typo silently disable the prefilter for
|
|
# a whole file type: the hook still MATCHES the file via its `files:` regex, so
|
|
# pre-commit reports neither `Skipped` nor an error; vale lints zero files,
|
|
# prints `0 errors ... in 1 file` and exits 0, and the hook shows `Passed`."
|
|
# Measured again against the merged config while rehoming these: a mutated
|
|
# `[**/SKILL.md]` -> `[**/SKILLS.md]` reports `0 errors ... in 0 files` and
|
|
# exits 0. Every gate in this repo reads that as a pass.
|
|
#
|
|
# ADR-0014's reason for two hook IDs was this same problem, and .pre-commit-
|
|
# hooks.yaml still carries both IDs after the merge for that reason.
|
|
|
|
# One representative path per file shape the prefilter is supposed to cover,
|
|
# tagged with the `.vale.ini` section that is supposed to cover it and with
|
|
# whether that path is covered by that section ALONE.
|
|
#
|
|
# `isolating` is load-bearing, not decoration. `.apm/agents/demo.agent.md`
|
|
# matches BOTH `[**/agents/*.md]` and `[**/*.agent.md]`, so breaking either one
|
|
# leaves it linted by the other and a probe on it alone would prove nothing
|
|
# about which section is live. `copilot/demo.agent.md` -- a Copilot agent file
|
|
# outside any `agents/` directory -- is what pins `[**/*.agent.md]` on its own,
|
|
# and bare `demo.md` under `agents/` is what pins `[**/agents/*.md]`. Case 29
|
|
# asserts that separation instead of trusting this column.
|
|
#
|
|
# The two `.claude/`-prefixed rows carry the location-independence property
|
|
# (original comment, unchanged in substance): a `SKILL.md` outside `plugins/`
|
|
# still matches `[**/SKILL.md]` and gets linted normally -- the globs constrain
|
|
# filename shape, not location. Every other row starts with `plugins/`, so
|
|
# narrowing a glob to a `plugins/`-shaped path (`[**/SKILL.md]` ->
|
|
# `[**/.apm/skills/*/SKILL.md]`) left all of them matching while the
|
|
# project-scope shape started linting as `0 errors ... in 0 files`.
|
|
#
|
|
# `demo.md` (bare, no `.agent.md` suffix) exercises `[**/agents/*.md]` in
|
|
# isolation, not because any current `.apm/agents/*` file has that shape -- per
|
|
# ADR-0016 they are all `*.agent.md`. `.pre-commit-hooks.yaml`'s agent regex
|
|
# still covers the bare shape, which is what keeps the row honest.
|
|
PROBE_TABLE28="$(cat <<'EOF_PROBE28'
|
|
plugins/demo/.apm/skills/demo/SKILL.md|[**/SKILL.md]|isolating
|
|
.claude/skills/demo/SKILL.md|[**/SKILL.md]|isolating
|
|
plugins/demo/.apm/agents/demo.md|[**/agents/*.md]|isolating
|
|
.claude/agents/demo.md|[**/agents/*.md]|isolating
|
|
plugins/demo/.apm/agents/demo.agent.md|[**/*.agent.md]|overlapping
|
|
copilot/demo.agent.md|[**/*.agent.md]|isolating
|
|
EOF_PROBE28
|
|
)"
|
|
|
|
VALE_ASSETS28="$FACTORY_AUDIT/assets/vale"
|
|
|
|
# The probe file carries a description with a token Kyberforge.VagueWording
|
|
# flags, so a config whose glob matches but whose BasedOnStyles lost Kyberforge
|
|
# fails too: it would lint the file and report nothing. `Use proactively` is
|
|
# there for case 30 and is inert for the other two cases -- no Kyberforge rule
|
|
# matches it, only KyberforgeCopilot.ProactivePhrase does.
|
|
PROBE_DESC28="Use when the caller wants a probe that helps with things. Use proactively."
|
|
|
|
# One tree holding every probe path, reused by all three cases. The config is
|
|
# always passed as an absolute path from outside the tree, so one tree serves
|
|
# the real config and the mutated copies alike.
|
|
build_probe_tree28() {
|
|
local dir rel
|
|
dir="$(mktemp -d)"
|
|
while IFS='|' read -r rel _ _; do
|
|
[[ -n "$rel" ]] || continue
|
|
mkdir -p "$dir/$(dirname "$rel")"
|
|
{
|
|
echo "---"
|
|
echo "name: probe"
|
|
echo "description: $PROBE_DESC28"
|
|
echo "---"
|
|
echo ""
|
|
echo "Body."
|
|
} > "$dir/$rel"
|
|
done <<EOF_TREE28
|
|
$PROBE_TABLE28
|
|
EOF_TREE28
|
|
echo "$dir"
|
|
}
|
|
|
|
# vale's report with ANSI colouring stripped. `|| true` on the pipeline because
|
|
# vale exits non-zero merely for HAVING alerts, which is the expected outcome
|
|
# for every probe here, and this file runs under `set -o pipefail`.
|
|
vale_report28() {
|
|
local cfg="$1" dir="$2" rel="$3"
|
|
{ (cd "$dir" && vale --config "$cfg" "$rel" 2>&1) | sed -E 's/\x1b\[[0-9;]*m//g'; } || true
|
|
}
|
|
|
|
# The file count out of vale's own summary line, e.g. `... in 0 files.` Empty
|
|
# output means no summary line at all, which the callers treat as a failure
|
|
# rather than as zero -- vale not running and vale scanning nothing are
|
|
# different defects and must not report the same way.
|
|
files_scanned28() {
|
|
printf '%s\n' "$1" \
|
|
| { grep -oE 'in [0-9]+ files?\.' || true; } \
|
|
| { grep -oE '[0-9]+' || true; } \
|
|
| tail -1
|
|
}
|
|
|
|
# Prints `<id>|<files regex>` for every hook in the given pre-commit manifest
|
|
# whose entry is factory-audit's vale-wrap.sh -- either manifest, since the two
|
|
# carry the same two hooks in the same shape. Records are delimited by their
|
|
# `- id:` line, so this does not depend on `entry:` preceding `files:` within a
|
|
# record. Case 28 wants the regexes alone and case 32 needs to know which hook
|
|
# each belongs to, so the id is carried here and dropped by the wrapper below.
|
|
hook_records28() {
|
|
local manifest="$1" id raw
|
|
[[ -f "$manifest" ]] || return 0
|
|
awk '
|
|
function flush() {
|
|
if (entry ~ /factory-audit\/scripts\/vale-wrap\.sh/ && files != "") print id "|" files
|
|
id = ""; entry = ""; files = ""
|
|
}
|
|
/^[ \t]*-[ \t]*id:/ { flush(); id = $0; sub(/^[ \t]*-[ \t]*id:[ \t]*/, "", id) }
|
|
/^[ \t]*entry:/ { entry = $0 }
|
|
/^[ \t]*files:/ { files = $0; sub(/^[ \t]*files:[ \t]*/, "", files) }
|
|
END { flush() }
|
|
' "$manifest" | while IFS='|' read -r id raw; do
|
|
# Strip the surrounding YAML quotes; the regex itself never carries them.
|
|
# `read -r id raw` splits on the FIRST `|` only, so a regex containing an
|
|
# alternation survives intact in $raw.
|
|
raw="${raw%\'}"; raw="${raw#\'}"
|
|
raw="${raw%\"}"; raw="${raw#\"}"
|
|
printf '%s|%s\n' "$id" "$raw"
|
|
done
|
|
}
|
|
|
|
# Prints the `files:` regex of every hook in the given manifest whose entry is
|
|
# factory-audit's vale-wrap.sh.
|
|
hook_file_regexes28() {
|
|
hook_records28 "$1" | cut -d'|' -f2-
|
|
}
|
|
|
|
matches_any_regex28() {
|
|
local rel="$1" regexes="$2" re
|
|
[[ -n "$regexes" ]] || return 1
|
|
while IFS= read -r re; do
|
|
[[ -n "$re" ]] || continue
|
|
if grep -Eq "$re" <<< "$rel"; then
|
|
return 0
|
|
fi
|
|
done <<EOF_RE28
|
|
$regexes
|
|
EOF_RE28
|
|
return 1
|
|
}
|
|
|
|
TREE28="$(build_probe_tree28)"
|
|
new_fixture "$TREE28"
|
|
HOOK_REGEXES28="$(hook_file_regexes28 "$REPO_ROOT/.pre-commit-hooks.yaml")"
|
|
|
|
echo ""
|
|
echo "--- every .vale.ini glob section actually scans a real file shape ---"
|
|
|
|
# The floor the original carried as PROBES_CHECKED, moved off the probe count
|
|
# and onto the config itself, which is the stronger question. A gutted table or
|
|
# a table whose section column drifted away from the real headers verified
|
|
# nothing while FAIL stayed 0; so would a fourth section landing in `.vale.ini`
|
|
# with no probe behind it. Both now fail here.
|
|
SECTIONS28=""
|
|
[[ "$VALE_CONFIG_OK" != true ]] || SECTIONS28="$(grep -E '^\[.*\]$' "$VALE_ASSETS28/.vale.ini" || true)"
|
|
TABLE_SECTIONS28="$(printf '%s\n' "$PROBE_TABLE28" | cut -d'|' -f2 | sort -u)"
|
|
UNPROBED28=""
|
|
if [[ "$VALE_CONFIG_OK" != true ]]; then
|
|
# Case 0 already failed and named the cause; reading an unloadable config
|
|
# here would only add a second, vaguer failure for the same defect.
|
|
:
|
|
elif [[ -z "$SECTIONS28" ]]; then
|
|
fail "$VALE_ASSETS28/.vale.ini declares no [glob] section at all — vale would lint nothing, and every probe below would be vacuous"
|
|
else
|
|
while IFS= read -r SEC28; do
|
|
[[ -n "$SEC28" ]] || continue
|
|
if ! grep -qF "|$SEC28|isolating" <<< "$PROBE_TABLE28"; then
|
|
UNPROBED28+="$SEC28 "
|
|
fi
|
|
done <<EOF_SEC28
|
|
$SECTIONS28
|
|
EOF_SEC28
|
|
STALE_SECTIONS28=""
|
|
while IFS= read -r TSEC28; do
|
|
[[ -n "$TSEC28" ]] || continue
|
|
grep -qF -- "$TSEC28" <<< "$SECTIONS28" || STALE_SECTIONS28+="$TSEC28 "
|
|
done <<EOF_TSEC28
|
|
$TABLE_SECTIONS28
|
|
EOF_TSEC28
|
|
if [[ -n "$UNPROBED28" ]]; then
|
|
fail "glob section(s) of .vale.ini have no isolating probe row, so a typo in them would be invisible here: $UNPROBED28"
|
|
elif [[ -n "$STALE_SECTIONS28" ]]; then
|
|
fail "probe row(s) name a glob section that .vale.ini no longer declares, so those rows prove nothing about the live config: $STALE_SECTIONS28"
|
|
else
|
|
pass "every glob section in .vale.ini has an isolating probe row, and every probe row names a live section"
|
|
fi
|
|
fi
|
|
|
|
# One `PASS|<rel>|<message>` or `FAIL|<rel>|<message>` line per probe row, for
|
|
# the config at $1. A function rather than an inline loop so Part B can hold a
|
|
# mutated copy to this exact logic -- a second, "equivalent" loop for the
|
|
# fixture would prove nothing about the live check. With $2 = false only the
|
|
# hook-scope half runs: that half reads .pre-commit-hooks.yaml, not vale, so a
|
|
# machine without vale still gets it.
|
|
probe_coverage28() {
|
|
local cfg="$1" with_vale="$2" rel sec report count
|
|
while IFS='|' read -r rel sec _; do
|
|
[[ -n "$rel" ]] || continue
|
|
if ! matches_any_regex28 "$rel" "$HOOK_REGEXES28"; then
|
|
# Original wording: the probe path is stale, or the hook was rescoped away
|
|
# from a shape it still needs to lint. Either way the row below stops
|
|
# describing anything the push gate actually hands to vale.
|
|
echo "FAIL|$rel|$rel matches no 'files:' regex of any factory-audit vale hook in .pre-commit-hooks.yaml — the probe path is stale, or the hook was rescoped away from a shape it still needs to lint"
|
|
continue
|
|
fi
|
|
if [[ "$with_vale" != true ]]; then
|
|
echo "PASS|$rel|$rel is in scope of a published factory-audit vale hook (glob coverage not checked: Vale-dependent half held back)"
|
|
continue
|
|
fi
|
|
report="$(vale_report28 "$cfg" "$TREE28" "$rel")"
|
|
count="$(files_scanned28 "$report")"
|
|
if [[ -z "$count" ]]; then
|
|
echo "FAIL|$rel|vale printed no summary line for $rel, so it is not known whether anything was scanned: ${report:-<empty>}"
|
|
elif [[ "$count" -eq 0 ]]; then
|
|
echo "FAIL|$rel|$sec scanned 0 files for $rel — the section's glob covers no path of that shape, so vale exits 0 and every gate reads it as a pass"
|
|
elif ! grep -qF "Kyberforge.VagueWording" <<< "$report"; then
|
|
echo "FAIL|$rel|$sec scanned $rel but raised no Kyberforge alert — the glob matches but the style is not loaded, which lints the file and reports nothing"
|
|
else
|
|
echo "PASS|$rel|$sec scans $rel ($count file) and raises a Kyberforge alert"
|
|
fi
|
|
done <<EOF_C28
|
|
$PROBE_TABLE28
|
|
EOF_C28
|
|
}
|
|
|
|
# Part A: the live assertion, against the shipped config. Held back entirely
|
|
# when the config cannot load -- case 0 has already named that.
|
|
PART_A_CLEAN28=false
|
|
if [[ "$VALE_CONFIG_OK" == true ]]; then
|
|
PART_A_CLEAN28=true
|
|
while IFS='|' read -r VERDICT28 _ MSG28; do
|
|
[[ -n "$VERDICT28" ]] || continue
|
|
if [[ "$VERDICT28" == PASS ]]; then pass "$MSG28"; else fail "$MSG28"; PART_A_CLEAN28=false; fi
|
|
done <<EOF_A28
|
|
$(probe_coverage28 "$VALE_ASSETS28/.vale.ini" "$VALE_READY")
|
|
EOF_A28
|
|
fi
|
|
|
|
# Part B: proof that Part A's style-not-loaded branch can fail. Case 29 already
|
|
# breaks each GLOB and watches the file count drop; nothing broke the other half
|
|
# of a section, its `BasedOnStyles`, so the branch that reports "the glob
|
|
# matches but the style is not loaded" was never shown to fire. For every
|
|
# section, `Kyberforge` is dropped from that section's BasedOnStyles in a COPY
|
|
# of the shipped assets (never the real file), and the probes that section
|
|
# covers alone must fail on exactly that branch while every other section's
|
|
# isolating probes keep passing. Overlapping rows are not asserted either way:
|
|
# measured against vale 3.15.2, a later matching section's BasedOnStyles
|
|
# replaces an earlier one's, so which verdict an overlapping row gets depends on
|
|
# section order, not on whether the check works.
|
|
#
|
|
# Run only against a config Part A passed. Every copy inherits the base's
|
|
# defects, so on a broken base the "other sections keep passing" half would
|
|
# report Part A's failure a second time.
|
|
if [[ "$VALE_READY" == true && "$PART_A_CLEAN28" == true ]]; then
|
|
echo ""
|
|
echo "--- dropping Kyberforge from any one section's BasedOnStyles fails that section's probes ---"
|
|
STYLE_MUT_FAILS28=""
|
|
while IFS= read -r MSEC28; do
|
|
[[ -n "$MSEC28" ]] || continue
|
|
SDIR28="$(mktemp -d)"
|
|
new_fixture "$SDIR28"
|
|
cp -r "$VALE_ASSETS28/." "$SDIR28/"
|
|
# Removes the exact list element `Kyberforge` from that section's
|
|
# BasedOnStyles only, so `KyberforgeCopilot` and every other section are
|
|
# untouched and the mutation isolates one style load.
|
|
awk -v target="$MSEC28" '
|
|
/^\[.*\]$/ { sec = $0 }
|
|
sec == target && /^[ \t]*BasedOnStyles[ \t]*=/ {
|
|
rhs = $0; sub(/^[^=]*=/, "", rhs)
|
|
n = split(rhs, parts, ","); out = ""
|
|
for (i = 1; i <= n; i++) {
|
|
p = parts[i]; gsub(/^[ \t]+|[ \t]+$/, "", p)
|
|
if (p == "" || p == "Kyberforge") continue
|
|
out = (out == "" ? p : out ", " p)
|
|
}
|
|
print "BasedOnStyles = " out
|
|
next
|
|
}
|
|
{ print }
|
|
' "$VALE_ASSETS28/.vale.ini" > "$SDIR28/.vale.ini"
|
|
if cmp -s "$VALE_ASSETS28/.vale.ini" "$SDIR28/.vale.ini"; then
|
|
STYLE_MUT_FAILS28+="[$MSEC28: the mutation left the config unchanged, so nothing was tested] "
|
|
continue
|
|
fi
|
|
RESULTS28="$(probe_coverage28 "$SDIR28/.vale.ini" true)"
|
|
while IFS='|' read -r REL28 ROWSEC28 ISO28; do
|
|
[[ -n "$REL28" && "$ISO28" == "isolating" ]] || continue
|
|
LINE28="$(printf '%s\n' "$RESULTS28" | { grep -F "|$REL28|" || true; } | head -1)"
|
|
if [[ "$ROWSEC28" == "$MSEC28" ]]; then
|
|
grep -qF "but raised no Kyberforge alert" <<< "$LINE28" \
|
|
|| STYLE_MUT_FAILS28+="[$MSEC28 lost Kyberforge but $REL28 did not fail as style-not-loaded: ${LINE28:-<no result>}] "
|
|
else
|
|
[[ "$LINE28" == PASS\|* ]] \
|
|
|| STYLE_MUT_FAILS28+="[$MSEC28 lost Kyberforge and unrelated probe $REL28 stopped passing, so the mutation was not confined to one section: ${LINE28:-<no result>}] "
|
|
fi
|
|
done <<EOF_B28
|
|
$PROBE_TABLE28
|
|
EOF_B28
|
|
done <<EOF_MSEC28
|
|
$SECTIONS28
|
|
EOF_MSEC28
|
|
if [[ -n "$STYLE_MUT_FAILS28" ]]; then
|
|
fail "Part A's style-not-loaded branch does not catch a section that lost Kyberforge: $STYLE_MUT_FAILS28"
|
|
else
|
|
pass "each section, with Kyberforge dropped from its BasedOnStyles in a copy of the config, fails its isolating probes as style-not-loaded while the others keep passing"
|
|
fi
|
|
fi
|
|
|
|
# --- 29. A broken glob section must FAIL, not pass quietly -------------------
|
|
# The probes above all pass today; what makes them worth having is that they
|
|
# stop passing when a section breaks. Asserted, not assumed: each section is
|
|
# typo'd in a COPY of the shipped assets (never the real file), and its
|
|
# isolating probe paths must go to 0 files scanned while every other section's
|
|
# probes keep scanning. A section whose mutation nothing notices is a section
|
|
# this suite does not really cover, and it fails here.
|
|
if [[ "$VALE_READY" == true ]]; then
|
|
echo ""
|
|
echo "--- a typo in any one glob section is caught by that section's probes ---"
|
|
MUTATION_FAILS29=""
|
|
while IFS= read -r SEC29; do
|
|
[[ -n "$SEC29" ]] || continue
|
|
DIR29="$(mktemp -d)"
|
|
new_fixture "$DIR29"
|
|
cp -r "$VALE_ASSETS28/." "$DIR29/"
|
|
# Insert `ZZ` before the final `.` of the glob: `[**/SKILL.md]` ->
|
|
# `[**/SKILLZZ.md]`, the same one-token shape as the `SKILL.md`/`SKILLS.md`
|
|
# typo the original probes were written for. Applied to the header line only,
|
|
# so BasedOnStyles and StylesPath are untouched and the mutation isolates the
|
|
# glob.
|
|
MUTATED29="$(printf '%s\n' "$SEC29" | sed -E 's/\.([^.]*)\]$/ZZ.\1]/')"
|
|
if [[ "$MUTATED29" == "$SEC29" ]]; then
|
|
MUTATION_FAILS29+="[$SEC29: the mutation rule left the header unchanged, so nothing was tested] "
|
|
continue
|
|
fi
|
|
# Exact-line replacement via awk: the header is a bracket expression, so a
|
|
# sed pattern built from it would be read as a character class.
|
|
awk -v old="$SEC29" -v new="$MUTATED29" '$0 == old { print new; next } { print }' \
|
|
"$VALE_ASSETS28/.vale.ini" > "$DIR29/.vale.ini"
|
|
if ! grep -qF -- "$MUTATED29" "$DIR29/.vale.ini"; then
|
|
MUTATION_FAILS29+="[$SEC29: the mutated header never reached the copied config] "
|
|
continue
|
|
fi
|
|
while IFS='|' read -r REL29 ROWSEC29 ISO29; do
|
|
[[ -n "$REL29" ]] || continue
|
|
COUNT29="$(files_scanned28 "$(vale_report28 "$DIR29/.vale.ini" "$TREE28" "$REL29")")"
|
|
[[ -n "$COUNT29" ]] || { MUTATION_FAILS29+="[$SEC29 broken: no vale summary for $REL29] "; continue; }
|
|
if [[ "$ROWSEC29" == "$SEC29" && "$ISO29" == "isolating" ]]; then
|
|
[[ "$COUNT29" -eq 0 ]] || MUTATION_FAILS29+="[$SEC29 broken but $REL29 still scanned $COUNT29 file(s), so this section's probes cannot detect a typo in it] "
|
|
else
|
|
[[ "$COUNT29" -gt 0 ]] || MUTATION_FAILS29+="[$SEC29 broken and unrelated probe $REL29 stopped scanning, so the mutation was not confined to one section] "
|
|
fi
|
|
done <<EOF_C29
|
|
$PROBE_TABLE28
|
|
EOF_C29
|
|
done <<EOF_MUT29
|
|
$SECTIONS28
|
|
EOF_MUT29
|
|
if [[ -n "$MUTATION_FAILS29" ]]; then
|
|
fail "breaking a glob section did not fail its probes: $MUTATION_FAILS29"
|
|
else
|
|
pass "each glob section, typo'd in a copy of the config, drops its isolating probes to 0 files while the others keep scanning"
|
|
fi
|
|
fi
|
|
|
|
# --- 30. KyberforgeCopilot stays scoped to `.agent.md` -----------------------
|
|
# Per ADR-0013 the Copilot style covers `.agent.md` files ALONE, for the
|
|
# Copilot-only "`Use proactively` has no effect" check, and must not be extended
|
|
# to `.md` files. The merge preserved that scoping and nothing else asserts it.
|
|
#
|
|
# The original comment on the incident, which is why this is checked at all
|
|
# rather than assumed: dropping `, KyberforgeCopilot` from the `[**/*.agent.md]`
|
|
# section "unloaded the whole style silently: no glob broke, the styles/ diff
|
|
# above stayed clean (the style directory is still shipped, just never loaded),
|
|
# the two .vale.ini files are deliberately unequal so no equality check applies,
|
|
# and the probe below still passed because it keys on a Kyberforge alert."
|
|
# One config removes the diff and the equality argument; it does not remove the
|
|
# silence, so the assertion is kept and made behavioural.
|
|
# Prints one failure token per mis-scoped probe for the config at $1; empty
|
|
# output means the Copilot style reaches exactly the `.agent.md` probes. A
|
|
# function so Part B runs this exact logic against mutated copies.
|
|
copilot_scope_failures30() {
|
|
local cfg="$1" rel report has bad=""
|
|
while IFS='|' read -r rel _ _; do
|
|
[[ -n "$rel" ]] || continue
|
|
report="$(vale_report28 "$cfg" "$TREE28" "$rel")"
|
|
has=false
|
|
if grep -qF "KyberforgeCopilot.ProactivePhrase" <<< "$report"; then has=true; fi
|
|
case "$rel" in
|
|
*.agent.md)
|
|
[[ "$has" == true ]] || bad+="[$rel is an agent file but raised no ProactivePhrase alert — the Copilot style is shipped but never loaded for it, so its rules lint nothing] "
|
|
;;
|
|
*)
|
|
# Non-vacuity guard: `absent` only means `scoped out` if the file was
|
|
# scanned at all. Without the Kyberforge half, a glob that stopped
|
|
# matching this path entirely would read as correct scoping.
|
|
if ! grep -qF "Kyberforge.VagueWording" <<< "$report"; then
|
|
bad+="[$rel raised no Kyberforge alert either, so its missing ProactivePhrase proves nothing about scoping] "
|
|
elif [[ "$has" == true ]]; then
|
|
bad+="[$rel is not an .agent.md file but raised a ProactivePhrase alert — the Copilot style has leaked past the scope ADR-0013 fixes it to] "
|
|
fi
|
|
;;
|
|
esac
|
|
done <<EOF_C30
|
|
$PROBE_TABLE28
|
|
EOF_C30
|
|
printf '%s' "$bad"
|
|
}
|
|
|
|
# Rewrites the BasedOnStyles line of one section ($2) of config $1 to $3.
|
|
rewrite_styles30() {
|
|
awk -v target="$2" -v styles="$3" '
|
|
/^\[.*\]$/ { sec = $0 }
|
|
sec == target && /^[ \t]*BasedOnStyles[ \t]*=/ { print "BasedOnStyles = " styles; next }
|
|
{ print }
|
|
' "$1"
|
|
}
|
|
|
|
if [[ "$VALE_READY" == true ]]; then
|
|
echo ""
|
|
echo "--- KyberforgeCopilot reaches .agent.md files and nothing else ---"
|
|
|
|
# Part A: the live assertion, against the shipped config.
|
|
COPILOT_FAILS30="$(copilot_scope_failures30 "$VALE_ASSETS28/.vale.ini")"
|
|
if [[ -n "$COPILOT_FAILS30" ]]; then
|
|
fail "KyberforgeCopilot.ProactivePhrase is mis-scoped: $COPILOT_FAILS30"
|
|
else
|
|
pass "ProactivePhrase fires on every .agent.md probe and on no SKILL.md or agents/*.md probe"
|
|
fi
|
|
|
|
# Part B: proof that Part A can fail, in both directions it guards. Each
|
|
# mutation is written into a COPY of the shipped assets, never the real file.
|
|
#
|
|
# unload -- `, KyberforgeCopilot` dropped from `[**/*.agent.md]`: the exact
|
|
# edit the original comment above records as having passed silently.
|
|
# Part A must report the agent probes as never loading the style.
|
|
# leak -- `KyberforgeCopilot` added to `[**/SKILL.md]`, the extension to
|
|
# `.md` files ADR-0013 forbids. Part A must report the SKILL.md
|
|
# probes as leaked.
|
|
#
|
|
# Run only against a config Part A passed: on a base that already unloads or
|
|
# leaks the style, a mutation is a no-op or a duplicate of Part A's failure.
|
|
if [[ -z "$COPILOT_FAILS30" ]]; then
|
|
echo ""
|
|
echo "--- unloading or leaking KyberforgeCopilot in a copy of the config fails Part A ---"
|
|
UNLOAD30="$(mktemp -d)"
|
|
new_fixture "$UNLOAD30"
|
|
cp -r "$VALE_ASSETS28/." "$UNLOAD30/"
|
|
rewrite_styles30 "$VALE_ASSETS28/.vale.ini" "[**/*.agent.md]" "Kyberforge" > "$UNLOAD30/.vale.ini"
|
|
LEAK30="$(mktemp -d)"
|
|
new_fixture "$LEAK30"
|
|
cp -r "$VALE_ASSETS28/." "$LEAK30/"
|
|
rewrite_styles30 "$VALE_ASSETS28/.vale.ini" "[**/SKILL.md]" "Kyberforge, KyberforgeCopilot" > "$LEAK30/.vale.ini"
|
|
UNLOAD_FAILS30="$(copilot_scope_failures30 "$UNLOAD30/.vale.ini")"
|
|
LEAK_FAILS30="$(copilot_scope_failures30 "$LEAK30/.vale.ini")"
|
|
if cmp -s "$VALE_ASSETS28/.vale.ini" "$UNLOAD30/.vale.ini" || cmp -s "$VALE_ASSETS28/.vale.ini" "$LEAK30/.vale.ini"; then
|
|
fail "a Copilot-scope mutation left the copied config unchanged, so Part B mutated nothing and proves nothing about Part A"
|
|
elif ! grep -qF "[copilot/demo.agent.md is an agent file but raised no ProactivePhrase alert" <<< "$UNLOAD_FAILS30"; then
|
|
fail "dropping KyberforgeCopilot from [**/*.agent.md] did not fail Part A, so an unloaded Copilot style would pass silently again: ${UNLOAD_FAILS30:-<no failure>}"
|
|
elif ! grep -qF "[plugins/demo/.apm/skills/demo/SKILL.md is not an .agent.md file but raised a ProactivePhrase alert" <<< "$LEAK_FAILS30"; then
|
|
fail "adding KyberforgeCopilot to [**/SKILL.md] did not fail Part A, so the style could leak past ADR-0013's scope unnoticed: ${LEAK_FAILS30:-<no failure>}"
|
|
else
|
|
pass "unloading KyberforgeCopilot from .agent.md files and leaking it onto SKILL.md files are each caught by Part A"
|
|
fi
|
|
fi
|
|
fi
|
|
|
|
# --- 31. No Kyberforge rule is overridden out of its blocking level ----------
|
|
#
|
|
# Also rehomed from the deleted scripts/check-vale-style-sync.sh (ADR-0025).
|
|
# This one is not a glob probe and not a copy diff: it is a third class the
|
|
# merge took out with the script, and cases 28-30 cannot backstop it. They key
|
|
# on `Kyberforge.VagueWording` and `KyberforgeCopilot.ProactivePhrase`, so
|
|
# DescriptionOpener, PaddingPhrase, SentenceOpenerThereIs and CompositionNote
|
|
# can each be retired underneath a passing probe.
|
|
#
|
|
# The original's comments, verbatim:
|
|
#
|
|
# Per-rule overrides are the third way to retire a rule without touching a
|
|
# style file or a glob. Per ADR-0013, every rule is `level: error` and every
|
|
# alert is a FAIL — there is no ignorable tier. Vale's exit
|
|
# code keys on `error` alerts alone, so any override that leaves a rule at
|
|
# anything other than `error` still lints the file, still exits 0, and still
|
|
# shows `Passed` in pre-commit. The glob probe below cannot backstop this: it
|
|
# keys on one `Kyberforge.VagueWording` alert, so DescriptionOpener,
|
|
# PaddingPhrase, SentenceOpenerThereIs and ProactivePhrase can each be retired
|
|
# underneath a passing probe.
|
|
#
|
|
# Asserted as an ALLOWLIST, not a blocklist of `NO|warning|suggestion`, because
|
|
# that is vale 3.15.2's own semantic: only the exact tokens `YES` and `error`
|
|
# keep a rule blocking. `warning`/`suggestion` downgrade it (alert still
|
|
# printed, exit 0 — invisible, since pre-commit swallows a passing hook's
|
|
# output); every other value — `NO`, `false`, `0`, `off`, `n`, empty,
|
|
# `garbage`, and lowercase `yes`, `true`, `1`, `on` — silences the rule
|
|
# outright. Lowercase `yes` is the trap a blocklist cannot cover: it reads as
|
|
# "enabled" to a human and disables the rule. Verified by enumerating the
|
|
# value space against vale 3.15.2.
|
|
#
|
|
# The allowlist demands a BARE `YES`/`error` with nothing after it, which also
|
|
# rejects `error # note` and `error ; note`. Vale itself strips those — a
|
|
# whitespace-preceded `#` or `;` comment is removed and the rule stays live —
|
|
# so rejecting them is deliberately stricter than vale, not a workaround for
|
|
# it. Uniformity is worth more here than the ability to annotate a line that
|
|
# should not exist: no shipped `.vale.ini` has any override line at all, and
|
|
# the failure mode is a loud false positive rather than a silent pass. The
|
|
# genuine hazard is the no-space form — `error# note` and `error; note` are
|
|
# NOT stripped and silence the rule outright — and a rule that demands a bare
|
|
# token catches those without having to reimplement vale's comment parsing.
|
|
#
|
|
# `[A-Za-z0-9_-]` on both halves of the name, not `[A-Za-z]`: a rule named
|
|
# `Kyberforge.Vague2` is genuinely silenced by `= NO` (verified: 1 error ->
|
|
# 0 errors), so an alpha-only class would let a digit-bearing rule name slip
|
|
# past the gate. All five current rule names are pure alpha, so this is
|
|
# forward cover, not a live hole.
|
|
|
|
# The detector, with the original's two regexes unchanged. Part A runs it
|
|
# against the real config and Part B runs it against synthesised lines, so the
|
|
# thing Part B proves is the same thing Part A relies on — a second, "equivalent"
|
|
# copy of the regex for the fixtures would prove nothing about the live check.
|
|
#
|
|
# Original comment on the missing `-q`, kept because it names the trap: No
|
|
# `grep -q` in that pipeline on purpose: `-q` exits on its first match, and
|
|
# under `set -o pipefail` the resulting SIGPIPE on the upstream grep would make
|
|
# the whole pipeline report 141 and read as "no findings".
|
|
bad_overrides31() {
|
|
grep -E '^[[:space:]]*Kyberforge[A-Za-z0-9_-]*\.[A-Za-z0-9_-]+[[:space:]]*=' "$1" \
|
|
| grep -Ev '^[[:space:]]*Kyberforge[A-Za-z0-9_-]*\.[A-Za-z0-9_-]+[[:space:]]*=[[:space:]]*(YES|error)[[:space:]]*$' \
|
|
|| true
|
|
}
|
|
|
|
echo ""
|
|
echo "--- no .vale.ini override downgrades a Kyberforge rule out of its blocking level ---"
|
|
|
|
# Part A: the live assertion, against the shipped config. A plain grep, so it
|
|
# runs without vale; held back only when case 0 found the config missing or
|
|
# unreadable, where grepping it would read as "no bad override" and pass.
|
|
BAD31=""
|
|
[[ "$VALE_CONFIG_OK" != true ]] || BAD31="$(bad_overrides31 "$VALE_ASSETS28/.vale.ini")"
|
|
if [[ "$VALE_CONFIG_OK" != true ]]; then
|
|
:
|
|
elif [[ -n "$BAD31" ]]; then
|
|
fail ".vale.ini overrides a Kyberforge rule to something other than a bare YES or error (first: '${BAD31%%$'\n'*}') — every rule in this prefilter is level: error and every alert is a FAIL, and any other value downgrades or silences the rule while vale still exits 0. A trailing comment is rejected too: vale strips a spaced '# ...' but not 'error# ...', so this asks for the bare token rather than guessing which form you meant"
|
|
else
|
|
pass "the shipped .vale.ini carries no Kyberforge rule override outside the bare YES/error allowlist"
|
|
fi
|
|
|
|
# Part B: the detector is held to every rule that actually ships, enumerated
|
|
# from the style directories on disk rather than from a list in this file. A
|
|
# list would have to be remembered; the styles directory cannot be forgotten,
|
|
# because adding a rule IS adding a file to it. This is what makes Part A
|
|
# forward-covering: a rule added later under a style whose name the detector's
|
|
# `Kyberforge[A-Za-z0-9_-]*` class does not match fails here the day it lands,
|
|
# rather than being silently un-gated.
|
|
RULES31="$(
|
|
for RULE_FILE31 in "$VALE_ASSETS28"/styles/*/*.yml "$VALE_ASSETS28"/styles/*/*.yaml; do
|
|
[[ -f "$RULE_FILE31" ]] || continue
|
|
STYLE_DIR31="${RULE_FILE31%/*}"
|
|
RULE_BASE31="${RULE_FILE31##*/}"
|
|
RULE_BASE31="${RULE_BASE31%.yml}"
|
|
printf '%s.%s\n' "${STYLE_DIR31##*/}" "${RULE_BASE31%.yaml}"
|
|
done
|
|
)"
|
|
RULE_COUNT31="$(printf '%s\n' "$RULES31" | grep -c . || true)"
|
|
|
|
# Both halves are asserted. A detector that flags everything would pass the
|
|
# reject rows while making Part A a permanent false alarm; one that flags
|
|
# nothing would pass the accept rows while making Part A vacuous. Only holding
|
|
# it to both tells the two apart.
|
|
VARIANTS31="$(cat <<'EOF_VAR31'
|
|
reject|NO
|
|
reject|warning
|
|
reject|suggestion
|
|
reject|yes
|
|
reject|true
|
|
reject|
|
|
reject|error # note
|
|
reject|error# note
|
|
reject|error; note
|
|
accept|YES
|
|
accept|error
|
|
EOF_VAR31
|
|
)"
|
|
|
|
DETECTOR_FAILS31=""
|
|
SYNTH31="$(mktemp -d)"
|
|
new_fixture "$SYNTH31"
|
|
if [[ "$RULE_COUNT31" -eq 0 ]]; then
|
|
fail "no rule file was found under $VALE_ASSETS28/styles/*/ — the enumeration is empty, so Part B verifies nothing and Part A's forward cover is unproven"
|
|
else
|
|
while IFS= read -r RULE31; do
|
|
[[ -n "$RULE31" ]] || continue
|
|
while IFS='|' read -r VERDICT31 VALUE31; do
|
|
[[ -n "$VERDICT31" ]] || continue
|
|
# Written under a real section header so the fixture is a config a human
|
|
# could plausibly ship, not a bare fragment.
|
|
{
|
|
echo "StylesPath = styles"
|
|
echo ""
|
|
echo "[**/SKILL.md]"
|
|
echo "BasedOnStyles = Kyberforge"
|
|
echo "$RULE31 = $VALUE31"
|
|
} > "$SYNTH31/probe.ini"
|
|
GOT31="$(bad_overrides31 "$SYNTH31/probe.ini")"
|
|
if [[ "$VERDICT31" == "reject" && -z "$GOT31" ]]; then
|
|
DETECTOR_FAILS31+="[$RULE31 = '$VALUE31' was NOT flagged, so this value could silence or downgrade the rule with nothing noticing] "
|
|
elif [[ "$VERDICT31" == "accept" && -n "$GOT31" ]]; then
|
|
DETECTOR_FAILS31+="[$RULE31 = '$VALUE31' WAS flagged, so the allowlist rejects a value that keeps the rule blocking] "
|
|
fi
|
|
done <<EOF_VAR_INNER31
|
|
$VARIANTS31
|
|
EOF_VAR_INNER31
|
|
done <<EOF_RULES31
|
|
$RULES31
|
|
EOF_RULES31
|
|
if [[ -n "$DETECTOR_FAILS31" ]]; then
|
|
fail "the override allowlist does not hold for every shipped rule: $DETECTOR_FAILS31"
|
|
else
|
|
pass "all $RULE_COUNT31 rules enumerated from styles/ are covered by the allowlist, which accepts bare YES/error and rejects NO, warning, suggestion, lowercase yes, true, empty and every trailing-comment form"
|
|
fi
|
|
fi
|
|
|
|
# Part C: proof that the thing being gated is a real hazard and a SILENT one.
|
|
# The original recorded this as hand-verified ("1 error -> 0 errors"); asserting
|
|
# it here means the case cannot quietly degrade into guarding a failure mode
|
|
# vale no longer has. One rule is named deliberately rather than looping the
|
|
# enumeration: the mechanism under test is vale's override handling, which is
|
|
# not per-rule, and VagueWording is the rule the shared probe description
|
|
# already trips, so no second fixture shape is needed.
|
|
#
|
|
# `copilot/demo.agent.md` because the override is appended at the end of the
|
|
# file, which places it in the last section — `[**/*.agent.md]`.
|
|
if [[ "$VALE_READY" == true ]]; then
|
|
OVERRIDE_DIR31="$(mktemp -d)"
|
|
new_fixture "$OVERRIDE_DIR31"
|
|
cp -r "$VALE_ASSETS28/." "$OVERRIDE_DIR31/"
|
|
printf 'Kyberforge.VagueWording = NO\n' >> "$OVERRIDE_DIR31/.vale.ini"
|
|
SILENCED31="$(vale_report28 "$OVERRIDE_DIR31/.vale.ini" "$TREE28" "copilot/demo.agent.md")"
|
|
SILENCED_COUNT31="$(files_scanned28 "$SILENCED31")"
|
|
if [[ -z "$SILENCED_COUNT31" || "$SILENCED_COUNT31" -eq 0 ]]; then
|
|
fail "the override fixture scanned no file at all, so the disappearance of the VagueWording alert proves nothing about overrides"
|
|
elif grep -qF "Kyberforge.VagueWording" <<< "$SILENCED31"; then
|
|
# Reported as a FAIL, not a pass. It is not a defect in the config, but it
|
|
# means Parts A and B are guarding a failure mode this vale build no longer
|
|
# has — and a guard that guards nothing while reporting PASS is the same
|
|
# vacuous pass the whole case exists to close. Someone has to look.
|
|
fail "an '= NO' override no longer silences the rule in this vale build, so the allowlist in Parts A and B is guarding a failure mode that no longer exists — re-verify against this vale version before trusting or removing it"
|
|
else
|
|
pass "an '= NO' override silences a rule while vale still scans the file and exits 0, which is the silent downgrade the allowlist above exists to catch"
|
|
fi
|
|
fi
|
|
|
|
# --- 32. Every vale prefilter hook in .pre-commit-config.yaml still selects a
|
|
# live, correctly-classed corpus ----------------------------------------------
|
|
#
|
|
# The one assertion of the deleted scripts/check-vale-style-sync.sh (ADR-0025)
|
|
# that cases 28-31 did not rehome, re-scoped onto what is actually at risk. The
|
|
# original compared the `files:` regexes across the two manifests, selecting each
|
|
# hook by its `entry:` (`entry ~ skill "/scripts/vale-wrap.sh"`, old line 220) --
|
|
# which worked only because the two skills gave the two hooks two distinct entry
|
|
# paths. After the merge both hooks share one `entry:`, so that selector can no
|
|
# longer tell them apart and a faithful port would have to key on hook `id:`
|
|
# instead. Case 33 is that port; this case covers the separate question of
|
|
# whether each local hook selects a live corpus at all. The zero-match half of the hole stands on its own, and nothing
|
|
# else in the repo covers it: tests/test-vale-hooks-consumer.sh synthesises its
|
|
# own consumer config out of `.pre-commit-hooks.yaml` and never reads the local
|
|
# one, and cases 28-30 read `.pre-commit-hooks.yaml` too. This repo's OWN
|
|
# prefilter regexes -- `.pre-commit-config.yaml`'s vale-audit-prefilter-skill and
|
|
# vale-audit-prefilter-agent -- are therefore asserted by no test at all. Narrow
|
|
# either one to match zero files and every gate still passes: pre-commit does not
|
|
# error on a hook that matches nothing, it simply never runs it. That is the same
|
|
# silent-zero failure mode case 28 guards on the vale side of this pipeline, one
|
|
# layer up -- there the glob scans 0 files and exits 0, here the hook is handed 0
|
|
# files and never starts.
|
|
#
|
|
# Two properties, because matching SOMETHING is not the same as matching the
|
|
# right thing: a regex loosened to `^plugins/` would match hundreds of files and
|
|
# clear a bare non-emptiness check while handing vale a corpus it has no glob
|
|
# for. So each hook must also select only its own artifact class -- ADR-0014's
|
|
# reason for two hook IDs, carried across the merge by ADR-0025's comment in the
|
|
# config, is precisely that the two scopes stay independently addressable.
|
|
|
|
echo ""
|
|
echo "--- each .pre-commit-config.yaml vale prefilter hook matches a real, correctly-classed file ---"
|
|
|
|
PC_CONFIG32="$REPO_ROOT/.pre-commit-config.yaml"
|
|
# The corpus pre-commit itself draws from. A `files:` regex is matched against
|
|
# repository paths, so a regex matching no tracked path matches nothing the gate
|
|
# will ever hand the hook.
|
|
REPO_FILES32="$(cd "$REPO_ROOT" && git ls-files)"
|
|
|
|
# Prints one failure token per defect; empty output means every prefilter hook in
|
|
# $1 is scoped to a live corpus of its own artifact class. The config path is an
|
|
# argument so Part B can run this exact function against a mutated copy -- a
|
|
# second, "equivalent" implementation for the fixture would prove nothing about
|
|
# the live check.
|
|
prefilter_scope_failures32() {
|
|
local config="$1" files="$2"
|
|
local records id re class matched count offenders offending m
|
|
local seen_skill=false seen_agent=false bad=""
|
|
records="$(hook_records28 "$config")"
|
|
if [[ -z "$records" ]]; then
|
|
printf '%s' "[no hook whose entry is factory-audit's vale-wrap.sh was parsed out of ${config##*/} at all, so nothing below was checked] "
|
|
return 0
|
|
fi
|
|
while IFS='|' read -r id re; do
|
|
[[ -n "$id" ]] || continue
|
|
case "$id" in
|
|
*-skill) class="skill" ;;
|
|
*-agent) class="agent" ;;
|
|
*)
|
|
bad+="[$id is a vale prefilter hook whose id names neither artifact class, so this case cannot tell which corpus it is meant to select] "
|
|
continue
|
|
;;
|
|
esac
|
|
# Recorded as soon as the class is known, not at the end of the iteration:
|
|
# a hook that fails a check below is still a hook that EXISTS, and reporting
|
|
# it as missing as well would bury the real defect under a second message.
|
|
[[ "$class" != skill ]] || seen_skill=true
|
|
[[ "$class" != agent ]] || seen_agent=true
|
|
matched="$(printf '%s\n' "$files" | { grep -E "$re" || true; })"
|
|
count="$(printf '%s\n' "$matched" | grep -c . || true)"
|
|
if [[ "$count" -eq 0 ]]; then
|
|
bad+="[$id: 'files: $re' matches no tracked file in this repo, so pre-commit never runs it and the prefilter is off for the whole $class corpus while every gate still reports a pass] "
|
|
continue
|
|
fi
|
|
offenders=""
|
|
offending=0
|
|
while IFS= read -r m; do
|
|
[[ -n "$m" ]] || continue
|
|
# Per ADR-0016 every agent file is `<name>.agent.md` under an `agents/`
|
|
# directory; the skill corpus is SKILL.md files. Asserting the shape of
|
|
# every matched path is what rejects a regex loosened to a wider scope,
|
|
# and it makes the two corpora disjoint by construction.
|
|
if [[ "$class" == skill ]]; then
|
|
[[ "$m" != */SKILL.md ]] || continue
|
|
else
|
|
[[ "$m" != */agents/*.agent.md ]] || continue
|
|
fi
|
|
offending=$((offending + 1))
|
|
# A loosened regex can select hundreds of paths; three name the shape of
|
|
# the leak, and the count carries the scale.
|
|
[[ "$offending" -gt 3 ]] || offenders+="$m "
|
|
done <<EOF_MATCHED32
|
|
$matched
|
|
EOF_MATCHED32
|
|
if [[ "$offending" -gt 0 ]]; then
|
|
bad+="[$id: 'files: $re' selects $count file(s), $offending of them outside the $class artifact class, so the two prefilter scopes are no longer independently addressable and vale is handed files no glob in its config covers — first: $offenders] "
|
|
fi
|
|
done <<EOF_RECORDS32
|
|
$records
|
|
EOF_RECORDS32
|
|
# A config that simply dropped one of the two hooks would otherwise pass every
|
|
# check above by having nothing left to fail.
|
|
[[ "$seen_skill" == true ]] || bad+="[no vale prefilter hook for the skill corpus was found in ${config##*/}, so SKILL.md files are not prefiltered at all] "
|
|
[[ "$seen_agent" == true ]] || bad+="[no vale prefilter hook for the agent corpus was found in ${config##*/}, so agent files are not prefiltered at all] "
|
|
printf '%s' "$bad"
|
|
}
|
|
|
|
# Part A: the live assertion, against this repo's own config.
|
|
if [[ -z "$REPO_FILES32" ]]; then
|
|
fail "git ls-files reported no tracked file, so the corpus this case matches the hook regexes against is empty and every check below would be vacuous"
|
|
else
|
|
SCOPE_FAILS32="$(prefilter_scope_failures32 "$PC_CONFIG32" "$REPO_FILES32")"
|
|
if [[ -n "$SCOPE_FAILS32" ]]; then
|
|
fail ".pre-commit-config.yaml's vale prefilter hooks are mis-scoped: $SCOPE_FAILS32"
|
|
else
|
|
pass "both vale prefilter hooks in .pre-commit-config.yaml match at least one tracked file and select only their own artifact class"
|
|
fi
|
|
fi
|
|
|
|
# Part B: proof that Part A can fail. Its whole point is that a narrowed regex is
|
|
# SILENT -- pre-commit skips a hook that matches nothing without a word -- so a
|
|
# Part A that could not detect the narrowing would be exactly the vacuous pass it
|
|
# exists to close. Every `files:` regex in a COPY of the config (never the real
|
|
# file) is prefixed with a path segment no tracked file starts with, which is the
|
|
# minimal form of the narrowing: still a valid regex, still a hook pre-commit
|
|
# accepts, matching zero files.
|
|
MUT32="$(mktemp -d)"
|
|
new_fixture "$MUT32"
|
|
awk '/^[ \t]*files:/ { sub(/\^/, "^zzz-no-such-path/") } { print }' \
|
|
"$PC_CONFIG32" > "$MUT32/.pre-commit-config.yaml"
|
|
MUT_RECORDS32="$(hook_records28 "$MUT32/.pre-commit-config.yaml")"
|
|
MUT_FAILS32="$(prefilter_scope_failures32 "$MUT32/.pre-commit-config.yaml" "$REPO_FILES32")"
|
|
if ! grep -q 'zzz-no-such-path' <<< "$MUT_RECORDS32"; then
|
|
fail "the narrowed regexes never reached the copied config, so Part B narrowed nothing and proves nothing about Part A"
|
|
elif [[ "$(printf '%s\n' "$MUT_RECORDS32" | grep -c .)" -ne 2 ]]; then
|
|
fail "the mutated config did not parse back as two prefilter hooks, so any failure below would come from the parser, not from the narrowing"
|
|
elif ! grep -qF "vale-audit-prefilter-skill: 'files: ^zzz-no-such-path/" <<< "$MUT_FAILS32"; then
|
|
fail "narrowing the skill hook's regex to match zero files did not fail this check, so Part A cannot detect a prefilter that has been silently switched off for SKILL.md files"
|
|
elif ! grep -qF "vale-audit-prefilter-agent: 'files: ^zzz-no-such-path/" <<< "$MUT_FAILS32"; then
|
|
fail "narrowing the agent hook's regex to match zero files did not fail this check, so Part A cannot detect a prefilter that has been silently switched off for agent files"
|
|
else
|
|
pass "narrowing either hook's 'files:' regex to match zero files is caught by Part A, which is what makes its pass mean something"
|
|
fi
|
|
|
|
# --- 33. The local and published vale hooks agree on every shared file shape --
|
|
#
|
|
# The cross-manifest `files:` agreement check of the deleted
|
|
# scripts/check-vale-style-sync.sh (ADR-0025), ported. Case 32 does not cover
|
|
# it: a local regex narrowed from `^plugins/[^/]+/...` to
|
|
# `^plugins/kyberforge/...` still matches tracked files, all of them SKILL.md,
|
|
# so it clears both of 32's properties while silently dropping every other
|
|
# plugin's skills out of this repo's prefilter. Measured before this case
|
|
# existed: that exact narrowing left the whole suite green.
|
|
#
|
|
# The original's comment on why the two manifests are compared per hook rather
|
|
# than unioned, verbatim in substance: "A union here previously let a probe that
|
|
# matched only the older, looser .pre-commit-hooks.yaml pattern read as 'in
|
|
# scope' even after .pre-commit-config.yaml's copy of the same hook had been
|
|
# narrowed away from it -- silently masking exactly the kind of hook-rescoping
|
|
# drift this script exists to catch."
|
|
#
|
|
# What changed in the port is the selector and nothing else. The original found
|
|
# each manifest's hook by `entry ~ skill "/scripts/vale-wrap.sh"`, which told the
|
|
# two hooks apart only because two skills gave them two entry paths; after the
|
|
# merge both hooks share one entry. They are paired by `id:` instead, from an
|
|
# explicit table. The pairing is explicit rather than inferred from an id suffix
|
|
# so that renaming either id fails here by name instead of quietly dropping a
|
|
# class out of the comparison. The probe table and its shared/hooks-only scopes
|
|
# are the original's rows, unchanged:
|
|
#
|
|
# shared -- a shape this repo's own layout has, so both manifests must
|
|
# agree on it. This is what catches the narrowing above.
|
|
# hooks-only -- a shape only the layout-agnostic published manifest has to
|
|
# cover. `.pre-commit-config.yaml` pinning this repo's own
|
|
# `plugins/*/.apm/` layout is by design, not drift; bare
|
|
# `agents/demo.md` is hooks-only because per ADR-0016 every
|
|
# `.apm/agents/` file is `*.agent.md`.
|
|
#
|
|
# A probe in scope of neither manifest fails too, exactly as in the original:
|
|
# the probe path is stale, or both hooks were rescoped away from it.
|
|
|
|
HOOK_PAIRS33="$(cat <<'EOF_PAIRS33'
|
|
skill|kyberforge-vale-audit-skill|vale-audit-prefilter-skill
|
|
agent|kyberforge-vale-audit-agent|vale-audit-prefilter-agent
|
|
EOF_PAIRS33
|
|
)"
|
|
|
|
PROBES33="$(cat <<'EOF_PROBES33'
|
|
skill|plugins/demo/.apm/skills/demo/SKILL.md|shared
|
|
skill|.claude/skills/demo/SKILL.md|hooks-only
|
|
agent|plugins/demo/.apm/agents/demo.md|hooks-only
|
|
agent|plugins/demo/.apm/agents/demo.agent.md|shared
|
|
agent|.claude/agents/demo.md|hooks-only
|
|
agent|copilot/demo.agent.md|hooks-only
|
|
EOF_PROBES33
|
|
)"
|
|
|
|
# Prints the `files:` regex of the hook whose id is exactly $2 in manifest $1,
|
|
# or nothing. Records are delimited by their `- id:` line, as in
|
|
# hook_records28, but selected by id alone: after the merge `entry:` no longer
|
|
# distinguishes them.
|
|
hook_regex_by_id33() {
|
|
local manifest="$1" want="$2" raw
|
|
[[ -f "$manifest" ]] || return 0
|
|
raw="$(WANT="$want" awk '
|
|
function flush() {
|
|
if (id == ENVIRON["WANT"] && files != "") print files
|
|
id = ""; files = ""
|
|
}
|
|
/^[ \t]*-[ \t]*id:/ { flush(); id = $0; sub(/^[ \t]*-[ \t]*id:[ \t]*/, "", id); sub(/[ \t]+$/, "", id) }
|
|
/^[ \t]*files:/ { files = $0; sub(/^[ \t]*files:[ \t]*/, "", files) }
|
|
END { flush() }
|
|
' "$manifest" | head -1)"
|
|
raw="${raw%\'}"; raw="${raw#\'}"
|
|
raw="${raw%\"}"; raw="${raw#\"}"
|
|
printf '%s' "$raw"
|
|
}
|
|
|
|
# Prints one failure token per defect for published manifest $1 against local
|
|
# manifest $2; empty output means every probe is in scope of at least one of its
|
|
# class's hooks and every shared probe is in scope of both. Manifest paths are
|
|
# arguments so Part B runs this exact function against mutated copies.
|
|
cross_manifest_failures33() {
|
|
local published="$1" local_cfg="$2"
|
|
local class pub_id loc_id rel scope pub_re loc_re in_hooks in_config
|
|
local bad="" checked=0 shared_classes="" missing_classes=""
|
|
while IFS='|' read -r class pub_id loc_id; do
|
|
[[ -n "$class" ]] || continue
|
|
if [[ -z "$(hook_regex_by_id33 "$published" "$pub_id")" ]]; then
|
|
bad+="[no hook with id '$pub_id' and a files: regex in ${published##*/}, so the $class class is compared against nothing] "
|
|
missing_classes+="$class "
|
|
fi
|
|
if [[ -z "$(hook_regex_by_id33 "$local_cfg" "$loc_id")" ]]; then
|
|
bad+="[no hook with id '$loc_id' and a files: regex in ${local_cfg##*/}, so the $class class is compared against nothing] "
|
|
missing_classes+="$class "
|
|
fi
|
|
done <<EOF_PAIRCHK33
|
|
$HOOK_PAIRS33
|
|
EOF_PAIRCHK33
|
|
while IFS='|' read -r class rel scope; do
|
|
[[ -n "$class" ]] || continue
|
|
pub_id="$(printf '%s\n' "$HOOK_PAIRS33" | awk -F'|' -v c="$class" '$1 == c { print $2 }')"
|
|
loc_id="$(printf '%s\n' "$HOOK_PAIRS33" | awk -F'|' -v c="$class" '$1 == c { print $3 }')"
|
|
if [[ -z "$pub_id" || -z "$loc_id" ]]; then
|
|
bad+="[probe $rel names class '$class', which has no row in the hook pair table] "
|
|
continue
|
|
fi
|
|
pub_re="$(hook_regex_by_id33 "$published" "$pub_id")"
|
|
loc_re="$(hook_regex_by_id33 "$local_cfg" "$loc_id")"
|
|
# A missing hook was reported above; comparing against it would add a
|
|
# second message for the same defect.
|
|
[[ -n "$pub_re" && -n "$loc_re" ]] || continue
|
|
checked=$((checked + 1))
|
|
[[ "$scope" != shared ]] || shared_classes+="$class "
|
|
in_hooks=false
|
|
matches_any_regex28 "$rel" "$pub_re" && in_hooks=true
|
|
in_config=false
|
|
matches_any_regex28 "$rel" "$loc_re" && in_config=true
|
|
if [[ "$in_hooks" == false && "$in_config" == false ]]; then
|
|
bad+="[$rel matches neither $pub_id's nor $loc_id's 'files:' regex — the probe path is stale, or the hook was rescoped away from a shape it still needs to lint] "
|
|
elif [[ "$scope" == shared && "$in_hooks" != "$in_config" ]]; then
|
|
bad+="[$rel is in scope of $pub_id in ${published##*/} (hooks=$in_hooks) but not of $loc_id in ${local_cfg##*/} (config=$in_config), or vice versa — the local and published 'files:' regexes have drifted, and the narrower one silently stops prefiltering that shape] "
|
|
fi
|
|
done <<EOF_PROBECHK33
|
|
$PROBES33
|
|
EOF_PROBECHK33
|
|
# The original's PROBES_CHECKED floor, plus one it lacked: a class with no
|
|
# shared probe is never compared across manifests at all.
|
|
if [[ "$checked" -eq 0 && -z "$missing_classes" ]]; then
|
|
bad+="[no probe row was checked, so the agreement check verified nothing] "
|
|
fi
|
|
while IFS='|' read -r class _ _; do
|
|
[[ -n "$class" ]] || continue
|
|
# A class whose hook is missing was already reported, and its probes were
|
|
# skipped for that reason, not for want of a shared row.
|
|
[[ " $missing_classes" != *" $class "* ]] || continue
|
|
[[ " $shared_classes" == *" $class "* ]] \
|
|
|| bad+="[the $class class has no shared probe row, so its two hooks are never compared] "
|
|
done <<EOF_CLASSCHK33
|
|
$HOOK_PAIRS33
|
|
EOF_CLASSCHK33
|
|
printf '%s' "$bad"
|
|
}
|
|
|
|
echo ""
|
|
echo "--- the local and published vale hooks agree on every shared file shape ---"
|
|
|
|
PUBLISHED33="$REPO_ROOT/.pre-commit-hooks.yaml"
|
|
|
|
# Part A: the live assertion, against the two real manifests.
|
|
AGREE_FAILS33="$(cross_manifest_failures33 "$PUBLISHED33" "$PC_CONFIG32")"
|
|
if [[ -n "$AGREE_FAILS33" ]]; then
|
|
fail ".pre-commit-config.yaml and .pre-commit-hooks.yaml disagree on the vale hooks' scope: $AGREE_FAILS33"
|
|
else
|
|
pass "every probe is in scope of its class's hooks, and both manifests agree on every shared shape"
|
|
fi
|
|
|
|
# Part B: proof that Part A can fail. Each mutation lands in a COPY of
|
|
# .pre-commit-config.yaml, never the real file.
|
|
#
|
|
# skill -- the exact narrowing that slipped through before this case existed.
|
|
# agent -- the same narrowing applied to the agent hook.
|
|
# id -- the local skill hook renamed, which must fail by name rather than
|
|
# drop the skill class out of the comparison.
|
|
#
|
|
# Replacement is fixed-string, via ENVIRON, so the regex's backslashes reach awk
|
|
# literally; `awk -v` would process them as escape sequences.
|
|
narrow_config33() {
|
|
OLD33="$2" NEW33="$3" awk '
|
|
$0 ~ ENVIRON["LINE33"] {
|
|
i = index($0, ENVIRON["OLD33"])
|
|
if (i) $0 = substr($0, 1, i - 1) ENVIRON["NEW33"] substr($0, i + length(ENVIRON["OLD33"]))
|
|
}
|
|
{ print }
|
|
' "$1"
|
|
}
|
|
MUT33="$(mktemp -d)"
|
|
new_fixture "$MUT33"
|
|
LINE33='^[ \t]*files:' narrow_config33 "$PC_CONFIG32" \
|
|
'^plugins/[^/]+/\.apm/skills/' '^plugins/kyberforge/\.apm/skills/' > "$MUT33/skill.yaml"
|
|
LINE33='^[ \t]*files:' narrow_config33 "$PC_CONFIG32" \
|
|
'^plugins/[^/]+/\.apm/agents/' '^plugins/kyberforge/\.apm/agents/' > "$MUT33/agent.yaml"
|
|
LINE33='^[ \t]*-[ \t]*id:' narrow_config33 "$PC_CONFIG32" \
|
|
'vale-audit-prefilter-skill' 'vale-audit-prefilter-skill-renamed' > "$MUT33/id.yaml"
|
|
SKILL_FAILS33="$(cross_manifest_failures33 "$PUBLISHED33" "$MUT33/skill.yaml")"
|
|
AGENT_FAILS33="$(cross_manifest_failures33 "$PUBLISHED33" "$MUT33/agent.yaml")"
|
|
ID_FAILS33="$(cross_manifest_failures33 "$PUBLISHED33" "$MUT33/id.yaml")"
|
|
if ! grep -qF '^plugins/kyberforge/\.apm/skills/' "$MUT33/skill.yaml" \
|
|
|| ! grep -qF '^plugins/kyberforge/\.apm/agents/' "$MUT33/agent.yaml" \
|
|
|| ! grep -qF 'vale-audit-prefilter-skill-renamed' "$MUT33/id.yaml"; then
|
|
fail "a mutation never reached its copied config, so Part B mutated nothing and proves nothing about Part A"
|
|
elif ! grep -qF "[plugins/demo/.apm/skills/demo/SKILL.md is in scope of kyberforge-vale-audit-skill" <<< "$SKILL_FAILS33"; then
|
|
fail "narrowing the local skill hook to ^plugins/kyberforge/ did not fail Part A, so the prefilter can drop every other plugin's skills with every gate green: ${SKILL_FAILS33:-<no failure>}"
|
|
elif [[ -z "$AGREE_FAILS33" ]] && grep -qF "kyberforge-vale-audit-agent" <<< "$SKILL_FAILS33"; then
|
|
# Only meaningful against a clean base: when Part A already failed, the copy
|
|
# inherits that defect, and reporting it again here would be one defect twice.
|
|
fail "narrowing only the skill hook also reported an agent-class defect, so the comparison is not confined to its class: $SKILL_FAILS33"
|
|
elif ! grep -qF "[plugins/demo/.apm/agents/demo.agent.md is in scope of kyberforge-vale-audit-agent" <<< "$AGENT_FAILS33"; then
|
|
fail "narrowing the local agent hook to ^plugins/kyberforge/ did not fail Part A: ${AGENT_FAILS33:-<no failure>}"
|
|
elif ! grep -qF "[no hook with id 'vale-audit-prefilter-skill' and a files: regex in id.yaml" <<< "$ID_FAILS33"; then
|
|
fail "renaming the local skill hook's id did not fail Part A by name, so the skill class could fall out of the comparison silently: ${ID_FAILS33:-<no failure>}"
|
|
else
|
|
pass "narrowing either local hook to one plugin, or renaming one, is caught by Part A"
|
|
fi
|
|
|
|
# --- 34. Every glob section loads a real style, asserted without vale --------
|
|
#
|
|
# Case 28 asks this behaviourally -- it lints a probe through the real config
|
|
# and watches for a Kyberforge alert -- but every probe of it that can answer
|
|
# the question is behind `VALE_READY`. On a machine with no vale those probes
|
|
# do not run, the config is then read by case 0 alone, and case 0 deliberately
|
|
# exempts an EMPTY `BasedOnStyles` so that 28's style-not-loaded branch keeps
|
|
# something left to detect. Between the two, an emptied `BasedOnStyles` is
|
|
# caught by nothing in a non-strict run: vale loads that config, lints every
|
|
# file the section matches with NO rule, prints `0 errors ... in 1 file` and
|
|
# exits 0, and every gate in this repo reads that as a pass. That is the
|
|
# silent-pass class ADR-0013 exists to prevent, and `--strict` -- which a
|
|
# consumer's clone does not run -- is the only thing standing in front of it.
|
|
#
|
|
# So this case asks the same question of the config TEXT, with no dependency on
|
|
# vale being installed. It is a separate case rather than an addition to either
|
|
# neighbour: case 0's exemption and case 28's behavioural branch both stay
|
|
# exactly as documented, and no property is owned twice.
|
|
#
|
|
# The StylesPath half is the deleted scripts/check-vale-style-sync.sh's
|
|
# requirement (ADR-0025), restored. `vale_config_defects0` absolutizes with
|
|
# `[[ "$sp" == /* ]] || sp="$dir/$sp"`, which ACCEPTS an absolute StylesPath --
|
|
# a path that resolves on the machine that wrote it and on no other. Relative
|
|
# resolution against the config's own directory is the only reason the bundled
|
|
# styles are found under a CONSUMING repo's clone prefix, so an absolute one
|
|
# passes every check here and hard-fails every external consumer of
|
|
# .pre-commit-hooks.yaml. It is asserted here, not in case 0, to keep case 0's
|
|
# scope the one its comment describes.
|
|
|
|
VALE_ASSETS34="$FACTORY_AUDIT/assets/vale"
|
|
|
|
# `<section header>|<BasedOnStyles value>` for every `[glob]` section of $1, in
|
|
# file order. A section carrying no BasedOnStyles at all prints an empty value
|
|
# rather than no row: an absent key and an emptied one are the same defect and
|
|
# must report as one. Keys above the first header are vale's global scope, not
|
|
# a section, and are skipped -- a global BasedOnStyles does not make a section
|
|
# that overrides it with an empty one load anything.
|
|
section_styles34() {
|
|
awk '
|
|
function trim(s) { gsub(/^[ \t]+|[ \t]+$/, "", s); return s }
|
|
/^[ \t]*\[.*\][ \t]*$/ {
|
|
if (sec != "") print sec "|" val
|
|
sec = trim($0); val = ""; next
|
|
}
|
|
sec != "" && /^[ \t]*BasedOnStyles[ \t]*=/ {
|
|
val = $0; sub(/^[^=]*=/, "", val); val = trim(val)
|
|
}
|
|
END { if (sec != "") print sec "|" val }
|
|
' "$1"
|
|
}
|
|
|
|
# One failure token per defect in the config at $1; empty output means every
|
|
# section would load at least one real style for a consumer. The section floor
|
|
# is here for the same reason case 28 Part A and case 33 carry theirs: a config
|
|
# whose sections were all deleted lints nothing at all, and without a floor this
|
|
# function would report it clean.
|
|
style_load_defects34() {
|
|
local cfg="$1" dir sp sec names name bad="" sections=0
|
|
if [[ ! -f "$cfg" ]]; then
|
|
printf '%s' "[$cfg does not exist] "
|
|
return 0
|
|
fi
|
|
dir="$(dirname "$cfg")"
|
|
sp="$({ grep -E '^[[:space:]]*StylesPath[[:space:]]*=' "$cfg" || true; } | tail -1)"
|
|
sp="${sp#*=}"
|
|
sp="${sp#"${sp%%[![:space:]]*}"}"
|
|
sp="${sp%"${sp##*[![:space:]]}"}"
|
|
if [[ -z "$sp" ]]; then
|
|
printf '%s' "[${cfg##*/} sets no StylesPath, so no section can name a style that loads] "
|
|
return 0
|
|
fi
|
|
if [[ "$sp" == /* ]]; then
|
|
printf '%s' "[${cfg##*/} sets the ABSOLUTE StylesPath '$sp'; it resolves only on the machine that wrote it, and a consuming repo's clone -- which is the only reason .pre-commit-hooks.yaml ships these styles -- gets 'path does not exist'] "
|
|
return 0
|
|
fi
|
|
if [[ ! -d "$dir/$sp" ]]; then
|
|
printf '%s' "[StylesPath resolves to $dir/$sp, which is not a directory] "
|
|
return 0
|
|
fi
|
|
while IFS='|' read -r sec names; do
|
|
[[ -n "$sec" ]] || continue
|
|
sections=$((sections + 1))
|
|
if [[ -z "$names" ]]; then
|
|
bad+="[$sec declares no non-empty BasedOnStyles, so vale lints every file it matches with no rule loaded, prints '0 errors' and exits 0] "
|
|
continue
|
|
fi
|
|
while IFS= read -r name; do
|
|
name="${name#"${name%%[![:space:]]*}"}"
|
|
name="${name%"${name##*[![:space:]]}"}"
|
|
# `Vale` is vale's built-in style and has no directory, exactly as in
|
|
# vale_config_defects0.
|
|
[[ -n "$name" && "$name" != "Vale" ]] || continue
|
|
[[ -d "$dir/$sp/$name" ]] || bad+="[$sec names style '$name', but $dir/$sp/$name does not exist] "
|
|
done <<EOF_NAMES34
|
|
$(printf '%s\n' "$names" | tr ',' '\n')
|
|
EOF_NAMES34
|
|
done <<EOF_SEC34
|
|
$(section_styles34 "$cfg")
|
|
EOF_SEC34
|
|
if (( sections == 0 )); then
|
|
bad+="[${cfg##*/} declares no [glob] section at all, so vale matches no file and this check would otherwise pass vacuously] "
|
|
fi
|
|
printf '%s' "$bad"
|
|
}
|
|
|
|
echo ""
|
|
echo "--- every glob section of the shipped .vale.ini loads a real style (no vale needed) ---"
|
|
|
|
# Part A: the live assertion, against the shipped config.
|
|
STYLE_LOAD_DEFECTS34="$(style_load_defects34 "$VALE_ASSETS34/.vale.ini")"
|
|
if [[ -n "$STYLE_LOAD_DEFECTS34" ]]; then
|
|
fail "the shipped .vale.ini has a section vale would lint with no style loaded, or a StylesPath no consumer's clone can resolve: $STYLE_LOAD_DEFECTS34"
|
|
else
|
|
pass "every [glob] section of the shipped .vale.ini names at least one style, every name it lists resolves to a directory under a RELATIVE StylesPath, and at least one section exists"
|
|
fi
|
|
|
|
# Part B: proof that Part A can fail, against the same function Part A calls --
|
|
# a second, "equivalent" reader written for the fixture would prove nothing
|
|
# about the live check. Every mutation lands in a COPY of the shipped assets,
|
|
# never the real file, and each copy is guarded for having actually changed
|
|
# before its verdict is trusted.
|
|
#
|
|
# empty BasedOnStyles -- per section, the silent case described above.
|
|
# absolute StylesPath -- pointed at the copy's OWN styles directory, which
|
|
# exists, so absoluteness is the only property left
|
|
# failing and the branch cannot pass for the wrong
|
|
# reason.
|
|
#
|
|
# Run only against a config Part A passed: every copy inherits the base's
|
|
# defects, so on a broken base these would re-report Part A's failure.
|
|
if [[ -z "$STYLE_LOAD_DEFECTS34" ]]; then
|
|
echo ""
|
|
echo "--- emptying any one section's BasedOnStyles, or absolutizing StylesPath, fails that check ---"
|
|
MUT_FAILS34=""
|
|
while IFS='|' read -r MSEC34 _; do
|
|
[[ -n "$MSEC34" ]] || continue
|
|
MDIR34="$(mktemp -d)"
|
|
new_fixture "$MDIR34"
|
|
cp -r "$VALE_ASSETS34/." "$MDIR34/"
|
|
# The section header line and every other section are untouched, so the
|
|
# mutation isolates one section's style load.
|
|
awk -v target="$MSEC34" '
|
|
function trim(s) { gsub(/^[ \t]+|[ \t]+$/, "", s); return s }
|
|
/^[ \t]*\[.*\][ \t]*$/ { sec = trim($0) }
|
|
sec == target && /^[ \t]*BasedOnStyles[ \t]*=/ { print "BasedOnStyles ="; next }
|
|
{ print }
|
|
' "$VALE_ASSETS34/.vale.ini" > "$MDIR34/.vale.ini"
|
|
if cmp -s "$VALE_ASSETS34/.vale.ini" "$MDIR34/.vale.ini"; then
|
|
MUT_FAILS34+="[$MSEC34: the mutation left the config unchanged, so nothing was tested] "
|
|
continue
|
|
fi
|
|
MDEFECTS34="$(style_load_defects34 "$MDIR34/.vale.ini")"
|
|
grep -qF "[$MSEC34 declares no non-empty BasedOnStyles" <<< "$MDEFECTS34" \
|
|
|| MUT_FAILS34+="[$MSEC34 lost its BasedOnStyles and Part A did not fail on it: ${MDEFECTS34:-<no failure>}] "
|
|
done <<EOF_MUT34
|
|
$(section_styles34 "$VALE_ASSETS34/.vale.ini")
|
|
EOF_MUT34
|
|
ABS34="$(mktemp -d)"
|
|
new_fixture "$ABS34"
|
|
cp -r "$VALE_ASSETS34/." "$ABS34/"
|
|
# Written through ENVIRON, as in narrow_config33, so the path reaches awk
|
|
# literally rather than through `-v`'s escape processing.
|
|
ABS_SP34="$ABS34/styles" awk '
|
|
/^[ \t]*StylesPath[ \t]*=/ { print "StylesPath = " ENVIRON["ABS_SP34"]; next }
|
|
{ print }
|
|
' "$VALE_ASSETS34/.vale.ini" > "$ABS34/.vale.ini"
|
|
ABS_DEFECTS34="$(style_load_defects34 "$ABS34/.vale.ini")"
|
|
if ! grep -qF "StylesPath = $ABS34/styles" "$ABS34/.vale.ini"; then
|
|
fail "the absolute-StylesPath mutation never reached its copied config, so Part B proves nothing about that branch"
|
|
elif [[ ! -d "$ABS34/styles" ]]; then
|
|
fail "the absolute StylesPath mutation points at a directory that does not exist, so a failure below would not be about absoluteness at all"
|
|
elif ! grep -qF "sets the ABSOLUTE StylesPath" <<< "$ABS_DEFECTS34"; then
|
|
fail "an absolute StylesPath naming a REAL styles directory passed Part A, so the config can ship a path that resolves on this machine alone: ${ABS_DEFECTS34:-<no failure>}"
|
|
elif [[ -n "$MUT_FAILS34" ]]; then
|
|
fail "Part A does not catch a section whose BasedOnStyles was emptied, which is the silent case this whole case exists for: $MUT_FAILS34"
|
|
else
|
|
pass "each section, with its BasedOnStyles emptied in a copy of the config, fails Part A by name, and so does an absolute StylesPath pointing at a real styles directory"
|
|
fi
|
|
fi
|
|
|
|
echo ""
|
|
echo "Results: $PASS passed, $FAIL failed"
|
|
[[ $FAIL -eq 0 ]] || exit 1
|
|
if [[ "$VALE_AVAILABLE" != true ]]; then
|
|
# run-tests.sh records the first `SKIP` line printed above as the reason.
|
|
echo "Every static case passed, but vale is not installed and the Vale-dependent cases did not run — exiting 77 so run-tests.sh reports this suite as skipped, and --strict fails it."
|
|
exit 77
|
|
fi
|