test: pin the nine ADR-0020 gate defects that shipped untested
Every defect fixed in f7cc279 was reachable because nothing asserted against it.
The gate had 43 assertions and none of them covered a consumer repo, a non-string
description, an unclosed fence, or the two spec ceilings. Each case below fails
against the pre-fix code and passes against the current one; every one was proved
non-vacuous by mutating a scratch copy of the script and watching the test go red,
independently twice.
The two that mattered most had no fixture anywhere. A consumer repo WITH .git is
the shape the resolver exists to serve, and only the no-.git case had ever been
tested, which is exactly why the blocker was invisible. And ADR-0020 says the
walk-up runs in two passes specifically so a nested .git cannot beat a plugins/
root further up — no fixture had ever placed a .git inside a plugin.
test-adr0020-differential.sh loses _non_adr_hook_error(). It excluded MAX_LINES and
MAX_WORDS from the cross-script comparison on the untested assumption that awk and
splitlines() agree. They do not, and the divergence stayed invisible for exactly as
long as the exclusion stood. The ceilings are now compared like any other rule.
Two existing assertions were repairs, not additions. The skill-improve probe had
been fixed by this very branch, so its iteration permanently took an
assertion-free SKIP that still counted as a pass; both branches now fail loudly and
each names the other file's pin so the two stay in step. And the yaml-none fixture
emitted `---/---`, which never matched the frontmatter pattern at all — it passed on
the bare word "frontmatter", present in both messages, while never reaching the
branch it was named for. Needles throughout that file now name their branch.
Refs: #99
ADR: 0020
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015W3iwF9ncfRZddGBxsMCYi
This commit is contained in:
@@ -535,19 +535,31 @@ expect_gate "a fixture with no authoring root reports DID NOT RUN and exits 0" \
|
||||
"Unchecked target(s): some-other-skill"
|
||||
|
||||
echo ""
|
||||
echo "--- the three live dangling routing targets are caught (issue #100) ---"
|
||||
# ADR-0020 records four broken routing targets and splits fixing them into its
|
||||
# own issue. Three are detectable from the description text alone; this asserts
|
||||
# the gate actually sees them rather than the check being vacuous in the corpus
|
||||
# it was written against.
|
||||
echo "--- the live dangling routing targets are caught (issue #100) ---"
|
||||
# ADR-0020 records the broken routing targets and splits fixing them into its own
|
||||
# issue. This asserts the gate actually sees them rather than the check being
|
||||
# vacuous in the corpus it was written against.
|
||||
#
|
||||
# There used to be a third probe here, for `skill-improve` in skill-audit's
|
||||
# description. It was already stale: that target was fixed, so the iteration
|
||||
# permanently took a `pass "SKIP: ..."` branch — an assertion-free result counted
|
||||
# in the totals, which is worse than no probe at all because it makes the suite
|
||||
# look one test stronger than it is. It also contradicted
|
||||
# tests/test-adr0020-targets.sh, which pins the live dangling set as EXACTLY
|
||||
# {gitea-labels, neuledge-context}; that file is the authority on the set, this
|
||||
# one only checks the two are individually detected.
|
||||
#
|
||||
# Both SKIP branches are gone with it, for the same reason. A probe whose fixture
|
||||
# has been retrofitted is not "still passing" — it is a pin that needs updating,
|
||||
# here and in the exact-set assertion in test-adr0020-targets.sh, and it should
|
||||
# say so out loud rather than quietly agreeing with whatever it finds.
|
||||
for probe in \
|
||||
"plugins/bin/.apm/skills/research/SKILL.md:neuledge-context" \
|
||||
"plugins/kyberforge/.apm/skills/skill-audit/SKILL.md:skill-improve" \
|
||||
"plugins/gitea/.apm/skills/gitea-issues/SKILL.md:gitea-labels"; do
|
||||
probe_file="$REPO_ROOT/${probe%%:*}"
|
||||
probe_name="${probe##*:}"
|
||||
if [[ ! -f "$probe_file" ]]; then
|
||||
pass "SKIP: ${probe%%:*} no longer exists (retrofitted)"
|
||||
fail "the probe fixture ${probe%%:*} no longer exists — this pin has become vacuous; update it and EXPECTED_DANGLING in tests/test-adr0020-targets.sh together"
|
||||
continue
|
||||
fi
|
||||
# Captured, not piped: the script exits non-zero on these files and
|
||||
@@ -558,10 +570,8 @@ for probe in \
|
||||
set -e
|
||||
if [[ "$probe_out" == *"routes to '$probe_name'"* ]]; then
|
||||
pass "detects the dangling '$probe_name' target in ${probe%%:*}"
|
||||
elif ! grep -q "$probe_name" "$probe_file"; then
|
||||
pass "SKIP: '$probe_name' no longer appears in ${probe%%:*} (fixed by issue #100)"
|
||||
else
|
||||
fail "did not detect the dangling '$probe_name' target in ${probe%%:*}"
|
||||
fail "did not detect the dangling '$probe_name' target in ${probe%%:*}. If issue #100 retrofitted it, drop this probe and update EXPECTED_DANGLING in tests/test-adr0020-targets.sh; if a false-positive fix took a true positive with it, that is the regression this asserts."
|
||||
fi
|
||||
done
|
||||
|
||||
|
||||
Reference in New Issue
Block a user