Defame1297
b6e68e9a2b
Three ways the gates could report green having measured nothing. All three were
invisible to a passing test suite, because pre-commit prints nothing at all for a
hook that exits 0 — a gate that declines to check and a gate that checked and
passed produce the identical signal.
- A UTF-8 BOM, a leading blank line, a trailing space after a `---` marker or
CRLF line endings defeated the `^---\n` frontmatter matcher. Every ADR-0020
check was then skipped and the file passed: measured at the time, a
550-character description with a 1,000-word body exited 0 behind a BOM.
All four shapes are now tolerated, and frontmatter that genuinely cannot be
parsed is a hard ERROR rather than a silent skip.
- An agent file with a valueless `description:` followed by another key let a
line regex capture the *next* key, which looked non-empty, so the
missing-or-empty branch never fired and every gate below it early-returned on
the empty folded value — zero output, exit 0, on a blocking gate. The one
field this contract is entirely about was the one field a gate could fail to
notice was absent. Presence is now decided on the YAML-folded value and
nowhere else, and a missing or empty description is a hard FAIL in all three
validators.
- The hand-rolled frontmatter fallback disagreed with PyYAML across the FAIL
boundary on folded scalars, so which reader happened to be available decided
the verdict. A fallback that mis-parses a scalar shape reports a vacuous pass,
which is worse than not running, so it is deleted: python3 and PyYAML are hard
requirements that fail loudly with an install pointer.
Boundary-target resolution no longer derives its universe from its own location.
A `${BASH_SOURCE}`-relative repo root leaked this repo's 39-skill universe into
every consumer repo running the hook through pre-commit, so a consumer skill
routing to `skill-audit` resolved against a plugin it had never installed. The
interim form resolved through `.claude/` and `.agents/`, which are gitignored
`apm install` output — the same commit reported 2 dangling targets on a machine
that had run the install and 6 on a fresh clone. Resolution now walks up from the
file being checked to an authoring root (nearest ancestor holding
`plugins/*/.apm/{skills,agents}`, else the nearest `.git`, in two passes so a
nested `.git` cannot outrank a real monorepo root); the universe is every skill
and agent under `<root>/plugins/*/` plus the file's own apm package and that
package's declared `dependencies.apm`. Deployed trees are consulted only when no
authoring root exists at all — the consumer case. One commit now gets one verdict,
which a gate shipping hot with no baseline file has to.
Narrowed in the same pass: a routing target inferred from the prose boundary form
and corroborated by nothing else reports at SUGGESTION instead of blocking. A
blocking check with no escape hatch is the wrong trade when the inference from
prose is the weak part of it.
New deterministic checks, all previously untested or absent: every
`references/<file>.md` a body names must exist (ERROR — a broken pointer is not a
style opinion); a description with no boundary clause at all, a Gotchas section
over five entries, and a Gotchas section over 25% of the body are SUGGESTIONs.
Where no universe can be determined the target check prints `INFO ... DID NOT
RUN` rather than passing quietly. Each prose-scanning check needed its own
false-positive fix — a fenced example of a Gotchas section was being read as the
section itself — and those fixes are pinned rather than assumed.
The resolver is one block copied verbatim into all three scripts between
BEGIN/END markers, because a cache-installed plugin's scripts cannot read outside
their own plugin directory. Nothing asserted the copies were still identical; a
one-line edit to a single copy passed every constant-agreement assertion, since
constants are not what drifts.
Tests land here rather than in a later commit. The existing suites assert the old
behaviour and go red against these scripts, so splitting them would leave a commit
whose own `run-tests` pre-push gate fails in isolation.
Refs: ADR-0020
287 lines
13 KiB
Bash
Executable File
287 lines
13 KiB
Bash
Executable File
#!/usr/bin/env bash
|
||
# Regression test for the two ways an ADR-0020 gate can be made to check NOTHING
|
||
# while still exiting 0. Both were live defects, both were silent, and both sit
|
||
# in the shared resolver block that all three scripts embed verbatim — so every
|
||
# case below runs against all three.
|
||
#
|
||
# 1. THE FRONTMATTER BLOCKER. The frontmatter matcher used to be `^---\n`. A
|
||
# UTF-8 BOM, a leading blank line, a trailing space after either marker, or
|
||
# CRLF line endings all defeated it, and the miss was not reported: every
|
||
# ADR-0020 check was skipped and the file passed. Measured at the time: a
|
||
# 550-character description with a 1,000-word body exited 0 behind a BOM.
|
||
# So this file asserts two complementary things — that each of those four
|
||
# shapes is now TOLERATED (the findings actually fire), and that
|
||
# frontmatter which genuinely cannot be parsed is a hard ERROR rather than
|
||
# a quiet skip. A file that cannot be measured must never report green.
|
||
#
|
||
# 2. THE VALUELESS DESCRIPTION. `description:` with no value, followed by
|
||
# another key, let a line regex's `\s*` cross the newline and capture the
|
||
# NEXT key. The value then looked present (so "missing or empty" never
|
||
# fired) and was empty once folded (so every ADR-0020 gate early-returned).
|
||
# An agent file with one exited 0 with zero output through a BLOCKING
|
||
# pre-push gate. All five spellings of "no value" are pinned here.
|
||
#
|
||
# Both fixtures carry an over-ceiling description AND an over-ceiling body on
|
||
# purpose: asserting a non-zero exit alone would be satisfied by the "cannot
|
||
# parse" error itself, so the tolerated shapes are asserted on the CONTENT of
|
||
# the findings, not on the exit code.
|
||
set -euo pipefail
|
||
|
||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||
HOOK="$REPO_ROOT/scripts/skill-size-check.sh"
|
||
SKILL_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/skill-audit/scripts/validate.sh"
|
||
AGENT_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/agent-audit/scripts/validate.sh"
|
||
PASS=0
|
||
FAIL=0
|
||
|
||
pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
|
||
fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
|
||
|
||
TMPDIR_T="$(mktemp -d)"
|
||
trap 'rm -rf "$TMPDIR_T"' EXIT
|
||
|
||
DESC_CHARS=450
|
||
BODY_WORDS=1000
|
||
|
||
# write_fixture <kind> <path> <name> — one generator for both file shapes.
|
||
#
|
||
# Byte-level control is the point: BOM placement, line endings and trailing
|
||
# whitespace are exactly what is under test, so the file is emitted in binary
|
||
# mode rather than through a shell heredoc that would normalise them.
|
||
write_fixture() {
|
||
python3 - "$1" "$2" "$3" "$DESC_CHARS" "$BODY_WORDS" <<'PY'
|
||
import sys
|
||
|
||
kind, path, name, desc_chars, body_words = sys.argv[1:6]
|
||
desc = 'x' * int(desc_chars)
|
||
body = ' '.join(['word'] * int(body_words))
|
||
|
||
# The default, well-formed shape. Variants below mutate it.
|
||
open_marker = '---'
|
||
close_marker = '---'
|
||
prefix = ''
|
||
newline = '\n'
|
||
fm_lines = ['name: ' + name, 'description: ' + desc]
|
||
|
||
if kind == 'plain':
|
||
pass
|
||
elif kind == 'bom':
|
||
prefix = ''
|
||
elif kind == 'leading-blanks':
|
||
prefix = '\n\n \n'
|
||
elif kind == 'trailing-ws':
|
||
open_marker = '--- '
|
||
close_marker = '---\t '
|
||
elif kind == 'crlf':
|
||
newline = '\r\n'
|
||
elif kind == 'no-close':
|
||
close_marker = None
|
||
elif kind == 'yaml-list':
|
||
fm_lines = ['- one', '- two']
|
||
elif kind == 'yaml-string':
|
||
fm_lines = ['just a bare scalar, not a mapping']
|
||
elif kind == 'yaml-none':
|
||
fm_lines = []
|
||
elif kind == 'yaml-malformed':
|
||
fm_lines = ['name: ' + name, 'description: "unterminated', 'tabs:\t- a']
|
||
elif kind == 'desc-no-value':
|
||
# The exact shape that exited 0 with zero output: a line regex's `\s*`
|
||
# crosses the newline and captures `model: sonnet` as the description.
|
||
fm_lines = ['name: ' + name, 'description:', 'model: sonnet']
|
||
elif kind == 'desc-null':
|
||
fm_lines = ['name: ' + name, 'description: null']
|
||
elif kind == 'desc-single-quoted-empty':
|
||
fm_lines = ['name: ' + name, "description: ''"]
|
||
elif kind == 'desc-double-quoted-empty':
|
||
fm_lines = ['name: ' + name, 'description: ""']
|
||
elif kind == 'desc-empty-fold':
|
||
fm_lines = ['name: ' + name, 'description: >']
|
||
else:
|
||
raise SystemExit('unknown fixture kind: %s' % kind)
|
||
|
||
parts = [prefix, open_marker, newline]
|
||
for line in fm_lines:
|
||
parts.append(line)
|
||
parts.append(newline)
|
||
if close_marker is not None:
|
||
parts.append(close_marker)
|
||
parts.append(newline)
|
||
parts.append(newline)
|
||
parts.append(body)
|
||
parts.append(newline)
|
||
|
||
with open(path, 'wb') as fh:
|
||
fh.write(''.join(parts).encode('utf-8'))
|
||
PY
|
||
}
|
||
|
||
# Builds all three subjects for one fixture kind and echoes nothing; the paths
|
||
# are fixed by convention so the probes below can find them.
|
||
#
|
||
# skill-audit takes a DIRECTORY (SKILL.md inside it, name matching the dir);
|
||
# agent-audit takes a FILE inside an apm package. The hook takes the SKILL.md
|
||
# directly, so it and skill-audit share one file.
|
||
build_subjects() {
|
||
local kind="$1" base="$TMPDIR_T/$1"
|
||
rm -rf "$base"
|
||
mkdir -p "$base/skill/my-skill" "$base/agent/.apm/agents"
|
||
cat > "$base/agent/apm.yml" <<'EOF'
|
||
name: test-package
|
||
version: 0.1.0
|
||
type: skill
|
||
EOF
|
||
write_fixture "$kind" "$base/skill/my-skill/SKILL.md" my-skill
|
||
write_fixture "$kind" "$base/agent/.apm/agents/my-agent.agent.md" my-agent
|
||
}
|
||
|
||
# probe_all <label> <kind> <needle>... — runs all three scripts over the fixture
|
||
# and requires every one of them to exit non-zero AND report every needle. One
|
||
# assertion per script would let two of them drift apart while the suite stayed
|
||
# green; the whole point of the shared resolver block is that they cannot.
|
||
#
|
||
# A needle written `@skills:<text>` is asserted for the hook and skill-audit but
|
||
# NOT for agent-audit. There is exactly one such needle in this file — the body
|
||
# word ceiling — and the exemption is the ADR, not a workaround: ADR-0020 gives
|
||
# agents the description gates and deliberately NO body word gate, because an
|
||
# agent body becomes the system prompt of a fresh context rather than competing
|
||
# with the caller's live conversation. Demanding a body finding from agent-audit
|
||
# would be demanding the ADR be contradicted.
|
||
probe_all() {
|
||
local label="$1" kind="$2"
|
||
shift 2
|
||
local base="$TMPDIR_T/$kind"
|
||
local -a targets=(
|
||
"hook|$HOOK|$base/skill/my-skill/SKILL.md"
|
||
"skill-audit|$SKILL_VALIDATE|$base/skill/my-skill"
|
||
"agent-audit|$AGENT_VALIDATE|$base/agent/.apm/agents/my-agent.agent.md"
|
||
)
|
||
local problems=""
|
||
for target in "${targets[@]}"; do
|
||
local who="${target%%|*}" rest="${target#*|}"
|
||
local script="${rest%%|*}" arg="${rest#*|}"
|
||
local out status=0
|
||
set +e
|
||
out="$(bash "$script" "$arg" 2>&1)"
|
||
status=$?
|
||
set -e
|
||
if [[ $status -eq 0 ]]; then
|
||
problems="$problems [$who exited 0: ${out:-<no output>}]"
|
||
continue
|
||
fi
|
||
for needle in "$@"; do
|
||
if [[ "$needle" == @skills:* ]]; then
|
||
# Spelled as a full `if`, not `[[ ... ]] && continue`. Under `set -e` the
|
||
# short form's exit status is the test's when it is false, and relying on
|
||
# the &&-list exemption to keep that from aborting the run is a footgun
|
||
# one edit away from biting.
|
||
if [[ "$who" == agent-audit ]]; then
|
||
continue
|
||
fi
|
||
needle="${needle#@skills:}"
|
||
fi
|
||
if [[ "$out" != *"$needle"* ]]; then
|
||
problems="$problems [$who never said '$needle': $out]"
|
||
fi
|
||
done
|
||
done
|
||
if [[ -z "$problems" ]]; then
|
||
pass "$label"
|
||
else
|
||
fail "$label —$problems"
|
||
fi
|
||
}
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# 1a. Tolerated frontmatter shapes — the gates must RUN, not merely not-pass
|
||
# ---------------------------------------------------------------------------
|
||
# The needles are the FINDINGS, not the exit code. A script that rejected the BOM
|
||
# outright would exit non-zero too, and would still be skipping every ADR-0020
|
||
# measurement — which is the defect, one error message later.
|
||
echo ""
|
||
echo "--- a BOM, leading blanks, trailing marker whitespace and CRLF are all tolerated, and the gates still fire ---"
|
||
for kind in plain bom leading-blanks trailing-ws crlf; do
|
||
build_subjects "$kind"
|
||
done
|
||
probe_all "control: a well-formed over-ceiling file fails on BOTH the description and the body" \
|
||
plain "description is $DESC_CHARS char" "@skills:body is $BODY_WORDS words"
|
||
probe_all "a UTF-8 BOM does not hide an over-ceiling description or body" \
|
||
bom "description is $DESC_CHARS char" "@skills:body is $BODY_WORDS words"
|
||
probe_all "leading blank lines before the opening --- do not hide the findings" \
|
||
leading-blanks "description is $DESC_CHARS char" "@skills:body is $BODY_WORDS words"
|
||
probe_all "trailing whitespace after either --- marker does not hide the findings" \
|
||
trailing-ws "description is $DESC_CHARS char" "@skills:body is $BODY_WORDS words"
|
||
# Note on what this last one can and cannot detect. read_text() opens the file in
|
||
# TEXT mode, so Python's universal-newline translation turns \r\n into \n before
|
||
# the frontmatter matcher ever sees it — verified by mutation: reverting
|
||
# FRONTMATTER_RE to the old `^---\n(.*?)\n---` breaks the leading-blanks and
|
||
# trailing-whitespace cases above but NOT this one. So this case pins the
|
||
# end-to-end behaviour (a CRLF file is measured, not skipped) rather than the
|
||
# `\r?\n` alternations in the regex, and it would catch a future switch to binary
|
||
# reads or to a newline='' open. Kept for that reason, and labelled so nobody
|
||
# reads it as covering more than it does.
|
||
probe_all "CRLF line endings do not hide the findings" \
|
||
crlf "description is $DESC_CHARS char" "@skills:body is $BODY_WORDS words"
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# 1b. Unparseable frontmatter is a hard ERROR, never a quiet skip
|
||
# ---------------------------------------------------------------------------
|
||
echo ""
|
||
echo "--- genuinely unparseable frontmatter exits non-zero with a message, rather than passing quietly ---"
|
||
for kind in no-close yaml-list yaml-string yaml-none yaml-malformed; do
|
||
build_subjects "$kind"
|
||
done
|
||
probe_all "frontmatter with no closing --- is reported, not skipped" \
|
||
no-close "frontmatter"
|
||
probe_all "frontmatter that parses to a LIST is reported, not skipped" \
|
||
yaml-list "frontmatter"
|
||
probe_all "frontmatter that parses to a STRING is reported, not skipped" \
|
||
yaml-string "frontmatter"
|
||
probe_all "frontmatter that parses to None (empty block) is reported, not skipped" \
|
||
yaml-none "frontmatter"
|
||
probe_all "malformed YAML in the frontmatter is reported, not skipped" \
|
||
yaml-malformed "frontmatter"
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# 2. A valueless description is a hard FAIL in all three scripts
|
||
# ---------------------------------------------------------------------------
|
||
# All five spellings mean the same thing to a YAML parser — an empty value — and
|
||
# all five have to be decided on the FOLDED value rather than on a line regex.
|
||
# `description:` followed by `model: sonnet` is the one that shipped: it made the
|
||
# value look present, skipped the "missing or empty" failure, and then
|
||
# early-returned out of every ADR-0020 gate on the genuinely empty folded value.
|
||
echo ""
|
||
echo "--- every spelling of a valueless description hard-FAILs in all three scripts ---"
|
||
for kind in desc-no-value desc-null desc-single-quoted-empty desc-double-quoted-empty desc-empty-fold; do
|
||
build_subjects "$kind"
|
||
done
|
||
probe_all "'description:' with no value (next key not captured as the value) FAILs" \
|
||
desc-no-value "description field is missing or empty"
|
||
probe_all "'description: null' FAILs" \
|
||
desc-null "description field is missing or empty"
|
||
probe_all "\"description: ''\" FAILs" \
|
||
desc-single-quoted-empty "description field is missing or empty"
|
||
probe_all "'description: \"\"' FAILs" \
|
||
desc-double-quoted-empty "description field is missing or empty"
|
||
probe_all "'description: >' with nothing folded under it FAILs" \
|
||
desc-empty-fold "description field is missing or empty"
|
||
|
||
# The specific regression, spelled out: the valueless-description agent file must
|
||
# not merely fail — it must not be SILENT. Zero output on a blocking gate is what
|
||
# made this un-diagnosable, so the output is asserted non-empty independently.
|
||
echo ""
|
||
echo "--- the valueless-description agent file produces output, not silence ---"
|
||
build_subjects desc-no-value
|
||
set +e
|
||
SILENT_OUT="$(bash "$AGENT_VALIDATE" "$TMPDIR_T/desc-no-value/agent/.apm/agents/my-agent.agent.md" 2>&1)"
|
||
SILENT_RC=$?
|
||
set -e
|
||
if [[ $SILENT_RC -ne 0 && -n "$SILENT_OUT" ]]; then
|
||
pass "agent-audit reports a valueless description rather than exiting 0 with zero output"
|
||
else
|
||
fail "agent-audit exited $SILENT_RC with output '${SILENT_OUT:-<empty>}' — the original defect was exit 0 and total silence on a blocking pre-push gate"
|
||
fi
|
||
|
||
echo ""
|
||
echo "Results: $PASS passed, $FAIL failed"
|
||
[[ $FAIL -eq 0 ]]
|