Defame1297/holocron
0
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases 3 Wiki Activity

test: pin the nine ADR-0020 gate defects that shipped untested

Browse Source 
Every defect fixed in f7cc279 was reachable because nothing asserted against it.
The gate had 43 assertions and none of them covered a consumer repo, a non-string
description, an unclosed fence, or the two spec ceilings. Each case below fails
against the pre-fix code and passes against the current one; every one was proved
non-vacuous by mutating a scratch copy of the script and watching the test go red,
independently twice.

The two that mattered most had no fixture anywhere. A consumer repo WITH .git is
the shape the resolver exists to serve, and only the no-.git case had ever been
tested, which is exactly why the blocker was invisible. And ADR-0020 says the
walk-up runs in two passes specifically so a nested .git cannot beat a plugins/
root further up — no fixture had ever placed a .git inside a plugin.

test-adr0020-differential.sh loses _non_adr_hook_error(). It excluded MAX_LINES and
MAX_WORDS from the cross-script comparison on the untested assumption that awk and
splitlines() agree. They do not, and the divergence stayed invisible for exactly as
long as the exclusion stood. The ceilings are now compared like any other rule.

Two existing assertions were repairs, not additions. The skill-improve probe had
been fixed by this very branch, so its iteration permanently took an
assertion-free SKIP that still counted as a pass; both branches now fail loudly and
each names the other file's pin so the two stay in step. And the yaml-none fixture
emitted `---/---`, which never matched the frontmatter pattern at all — it passed on
the bare word "frontmatter", present in both messages, while never reaching the
branch it was named for. Needles throughout that file now name their branch.

Refs: #99
ADR: 0020

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015W3iwF9ncfRZddGBxsMCYi
This commit is contained in:
Defame1297
2026-08-16 19:49:15 +00:00
parent f7cc27908c
commit b0d6d08239
6 changed files with 673 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files

170
tests/test-adr0020-frontmatter.sh
View File

@@ -19,7 +19,22 @@
# NEXT key. The value then looked present (so "missing or empty" never
# fired) and was empty once folded (so every ADR-0020 gate early-returned).
# An agent file with one exited 0 with zero output through a BLOCKING
# pre-push gate. All five spellings of "no value" are pinned here.
# pre-push gate. All five spellings of "no value" are pinned here, plus the
# three shapes where the value is present but is not TEXT — a list, a
# mapping, a bool. Those used to be `str()`-coerced and then measured as a
# Python repr, so `description: true` was the four-character "True" and
# passed the 400-character gate.
#
# 3. THE INDENTED CLOSING MARKER. The mirror image of (1): content the pattern
# was too LOOSE to reject. `\r?\n[ \t]*---` matched an indented `---` inside
# a `>`-folded description, truncating the frontmatter mid-value — the
# description gate then measured a fragment and the body gate measured the
# discarded description text.
#
# Every needle names the specific branch or measurement the case is about. A
# needle loose enough to match two branches is how the yaml-none fixture spent
# its life asserting the wrong one: it emitted `---\n---\n`, which never matched
# the frontmatter pattern at all, and passed on the bare word "frontmatter".
#
# Both fixtures carry an over-ceiling description AND an over-ceiling body on
# purpose: asserting a non-zero exit alone would be satisfied by the "cannot
@@ -81,7 +96,48 @@ elif kind == 'yaml-list':
elif kind == 'yaml-string':
fm_lines = ['just a bare scalar, not a mapping']
elif kind == 'yaml-none':
# A comment-only block, NOT an empty one. `---\n---\n` does not match
# FRONTMATTER_RE at all (the pattern needs a `\n` between the markers), so
# it lands on the "no parseable frontmatter" branch and never reaches the
# `data is None` -> "not a YAML mapping" branch this fixture is named for.
# It passed anyway because the needle used to be the bare word
# "frontmatter", which both messages contain. A comment is real frontmatter
# text that yaml.safe_load() returns None for, which is the branch.
fm_lines = ['# nothing but a comment']
elif kind == 'yaml-empty-block':
# The shape the fixture above USED to have, kept as its own case so the
# "no parseable frontmatter block" branch is covered on purpose rather than
# by accident.
fm_lines = []
elif kind == 'desc-folded-indented':
# A `>`-folded description whose CONTENT contains an indented `---` line.
# YAML block-scalar content must be indented deeper than its key, so this is
# a value, not a document marker — but the closing pattern used to be
# `\r?\n[ \t]*---`, which matched it, truncated the frontmatter mid-value
# and silently reclassified the rest of the description as body. Both halves
# of that are vacuous greens: the description gate measured a fragment, and
# the body gate measured description text.
#
# The value is padded to exactly desc_chars AFTER folding, and the boundary
# clause naming a target sits in the part the truncation used to discard.
head = 'Use when doing the thing. '
tail = ' Do not use for improvements — use no-such-folded-target instead.'
span = int(desc_chars) - len(head) - len(tail) - len(' --- ')
if span < 2:
raise SystemExit('desc_chars too small for the folded fixture')
fm_lines = [
'name: ' + name,
'description: >',
' ' + head + 'x' * (span // 2),
' ---',
' ' + 'x' * (span - span // 2) + tail,
]
elif kind == 'desc-list':
fm_lines = ['name: ' + name, 'description:', ' - one', ' - two']
elif kind == 'desc-mapping':
fm_lines = ['name: ' + name, 'description:', ' text: a description']
elif kind == 'desc-bool':
fm_lines = ['name: ' + name, 'description: true']
elif kind == 'yaml-malformed':
fm_lines = ['name: ' + name, 'description: "unterminated', 'tabs:\t- a']
elif kind == 'desc-no-value':
@@ -222,24 +278,58 @@ probe_all "trailing whitespace after either --- marker does not hide the finding
probe_all "CRLF line endings do not hide the findings" \
crlf "description is $DESC_CHARS char" "@skills:body is $BODY_WORDS words"
# ---------------------------------------------------------------------------
# 1a-bis. An indented `---` inside a block scalar is CONTENT, not a marker
# ---------------------------------------------------------------------------
# The mirror image of the four shapes above. Those were markers the pattern was
# too strict to accept; this is content the pattern was too loose to reject. The
# closing marker used to be `\r?\n[ \t]*---`, so an indented `---` inside a
# `>`-folded description ended the frontmatter early: the description gate then
# measured a truncated fragment (under every ceiling, so silent) and the body
# gate measured the discarded description text as body. Measured on the fixture
# below, the old code exited 0 with nothing but a spurious "no boundary clause"
# SUGGESTION — the clause is in the half it threw away.
#
# The needle is the full-value length, so a script that merely rejected the file
# would not satisfy it.
echo ""
echo "--- an indented --- inside a >-folded description is content, not the end of the frontmatter ---"
build_subjects desc-folded-indented
probe_all "a folded description containing an indented '---' is measured whole" \
desc-folded-indented "description is $DESC_CHARS char"
# ---------------------------------------------------------------------------
# 1b. Unparseable frontmatter is a hard ERROR, never a quiet skip
# ---------------------------------------------------------------------------
echo ""
echo "--- genuinely unparseable frontmatter exits non-zero with a message, rather than passing quietly ---"
for kind in no-close yaml-list yaml-string yaml-none yaml-malformed; do
# Each needle names the BRANCH the fixture is supposed to reach, not the word
# "frontmatter" — which every one of these messages contains, and which is why
# the yaml-none fixture below passed for years while landing on the wrong branch
# entirely.
for kind in no-close yaml-list yaml-string yaml-none yaml-empty-block yaml-malformed; do
build_subjects "$kind"
done
probe_all "frontmatter with no closing --- is reported, not skipped" \
no-close "frontmatter"
no-close "parseable YAML frontmatter block"
probe_all "frontmatter that parses to a LIST is reported, not skipped" \
yaml-list "frontmatter"
yaml-list "frontmatter is not a YAML mapping"
probe_all "frontmatter that parses to a STRING is reported, not skipped" \
yaml-string "frontmatter"
probe_all "frontmatter that parses to None (empty block) is reported, not skipped" \
yaml-none "frontmatter"
yaml-string "frontmatter is not a YAML mapping"
probe_all "frontmatter that parses to None (a comment-only block) is reported, not skipped" \
yaml-none "frontmatter is not a YAML mapping"
probe_all "a completely empty '---/---' block is reported, not skipped" \
yaml-empty-block "parseable YAML frontmatter block"
# Two needles, both naming the SYNTAX branch specifically. "frontmatter is not
# valid YAML" is now exclusive to it — the wrong-typed-description failures reach
# the same wrapper and no longer borrow that phrase (see 2c below) — and the
# scanner context proves the parser's own diagnostic survives the wrapper rather
# than being replaced by a generic one. Do not needle the tail of PyYAML's
# message: an earlier attempt used "could not find expected", which PyYAML 6.0.3
# does not emit for this fixture at all, so the case failed on the assertion
# rather than on the behaviour.
probe_all "malformed YAML in the frontmatter is reported, not skipped" \
yaml-malformed "frontmatter"
yaml-malformed "frontmatter is not valid YAML" "while scanning a quoted scalar"
# ---------------------------------------------------------------------------
# 2. A valueless description is a hard FAIL in all three scripts
@@ -265,6 +355,70 @@ probe_all "'description: \"\"' FAILs" \
probe_all "'description: >' with nothing folded under it FAILs" \
desc-empty-fold "description field is missing or empty"
# ---------------------------------------------------------------------------
# 2b. A description that is not a STRING is a parse failure, not a measurement
# ---------------------------------------------------------------------------
# The other half of the same family, and the reason it belongs beside the five
# above: all eight shapes are "the description is not a description", and seven
# of them used to be handled while this one was silently coerced. A non-string
# value went through `str()` and was then measured as a Python repr —
# `description: true` became the four-character "True" and sailed through the
# 400-character gate, a list became "['one', 'two']", a mapping its dict repr.
# None of those is text a host can preload, so measuring one is a green verdict
# on a file that was never measured.
echo ""
echo "--- a description that is a list, a mapping or a bool hard-FAILs in all three scripts ---"
for kind in desc-list desc-mapping desc-bool; do
build_subjects "$kind"
done
probe_all "a LIST description FAILs rather than being measured as its repr" \
desc-list "description is a list, not a string"
probe_all "a MAPPING description FAILs rather than being measured as its repr" \
desc-mapping "description is a dict, not a string"
probe_all "a BOOL description FAILs rather than being measured as the 4-char 'True'" \
desc-bool "description is a bool, not a string"
# ---------------------------------------------------------------------------
# 2c. The FAILURE CLASS reported has to be the one that happened
# ---------------------------------------------------------------------------
# The three fixtures above reach the same wrapper as a genuine YAML syntax
# error, and that wrapper used to prefix a hard-coded "frontmatter is not valid
# YAML (...)" onto all of them. For a non-string description that is false: the
# block parses, only the field's TYPE is wrong. On a blocking gate with no
# baseline it sent the author hunting for a syntax error that is not there. The
# assertion runs in both directions, because fixing it by dropping the phrase
# everywhere would trade one wrong diagnosis for another.
echo ""
echo "--- 'not valid YAML' is said for a syntax error and NOT for a wrong-typed description ---"
YAML_CLASS_PROBLEMS=""
for spec in "yaml-malformed|yes" "desc-list|no" "desc-mapping|no" "desc-bool|no"; do
kind="${spec%%|*}"
want="${spec#*|}"
build_subjects "$kind"
for target in \
"hook|$HOOK|$TMPDIR_T/$kind/skill/my-skill/SKILL.md" \
"skill-audit|$SKILL_VALIDATE|$TMPDIR_T/$kind/skill/my-skill" \
"agent-audit|$AGENT_VALIDATE|$TMPDIR_T/$kind/agent/.apm/agents/my-agent.agent.md"
do
who="${target%%|*}"; rest="${target#*|}"
script="${rest%%|*}"; arg="${rest#*|}"
set +e
out="$(bash "$script" "$arg" 2>&1)"
set -e
if [[ "$want" == yes && "$out" != *"frontmatter is not valid YAML"* ]]; then
YAML_CLASS_PROBLEMS="$YAML_CLASS_PROBLEMS [$who did not call $kind a YAML syntax error: $out]"
fi
if [[ "$want" == no && "$out" == *"not valid YAML"* ]]; then
YAML_CLASS_PROBLEMS="$YAML_CLASS_PROBLEMS [$who called $kind invalid YAML, but the frontmatter parsed: $out]"
fi
done
done
if [[ -z "$YAML_CLASS_PROBLEMS" ]]; then
pass "a type error is reported as a type error and a syntax error as a syntax error"
else
fail "wrong failure class reported —$YAML_CLASS_PROBLEMS"
fi
# The specific regression, spelled out: the valueless-description agent file must
# not merely fail — it must not be SILENT. Zero output on a blocking gate is what
# made this un-diagnosable, so the output is asserted non-empty independently.