fix(gates): make the ADR-0020 boundary check parse what skills actually write

The routing-target check understood only a single-arrow clause naming a bare
skill, so most real boundary prose was silently skipped rather than verified.
Two of those silences were fail-open: an unrecognised token following a target
dropped that target from the check entirely, and a skill directory with no
SKILL.md still resolved as a valid routing target, so a broken route passed.

Multi-target arrow clauses now draw a SUGGESTION instead of being ignored,
hand-invocation phrasing is carved out so it is not read as a route, and a
dotted filename parses into a new `unparsed` status rather than disappearing.
Three test fixtures had been relying on the SKILL.md-less directory resolving
as a target; they are corrected alongside the check.

Addresses #107, #108, #110.
This commit is contained in:
2026-08-31 08:01:07 +00:00
parent 095929142f
commit db5a426416
8 changed files with 1688 additions and 90 deletions

View File

@@ -34,26 +34,44 @@ set -euo pipefail
# as a proxy (Python's str.split(), the same primitive
# skill-audit/scripts/validate.sh applies to these two constants; `wc -w`
# disagrees with it on Unicode separators, which is why the awk pass that used
# to live in the loop below is gone). Measured over this repo's 39 in-scope SKILL.md files, characters per
# word runs min 5.97 / median 6.79 / mean 6.77 / max 7.22. At the standard
# ~4-characters-per-token English approximation that is 1.49 / 1.70 / 1.69 /
# 1.81 tokens per word.
# to live in the loop below is gone).
#
# THE MEASUREMENT BASIS, stated because the previous re-measure drifted onto a
# different one and the numbers moved without the prose noticing: characters
# per word is len(text) / len(text.split()) over the WHOLE FILE, whitespace
# included, on plugins/*/.apm/skills/*/SKILL.md. Counting only non-whitespace
# characters gives a materially lower figure (4.90 / 5.52 / 5.54 / 6.19 today)
# and is not the basis MAX_WORDS is calibrated against.
#
# Measured over this repo's 39 in-scope SKILL.md files (2026-08-31, after the
# ADR-0020 retrofit), characters per word runs min 5.93 / median 6.67 / mean
# 6.63 / max 7.34. At the standard ~4-characters-per-token English
# approximation that is 1.48 / 1.67 / 1.66 / 1.84 tokens per word.
#
# MAX_WORDS=2770 is therefore calibrated to the corpus WORST case rather than
# its median: 2770 words at the densest observed 7.22 chars/word is ~20,000
# characters, or ~5,000 tokens at the 4-characters-per-token approximation. So
# what this gate guarantees is "under 5,000 tokens even for the densest prose
# its median: 2770 words at the densest observed 7.34 chars/word is ~20,300
# characters, or ~5,090 tokens at the 4-characters-per-token approximation. So
# what this gate guarantees is "about 5,000 tokens even for the densest prose
# the corpus has produced" — the earlier median-calibrated MAX_WORDS=2900 let
# such a file sit at exactly the ceiling and still spend ~5,240 tokens. A
# median-density file at 2770 words spends ~4,700 tokens, so typical prose
# gives up ~130 words of headroom to close that gap. The largest SKILL.md in
# the repo is 2,760 words whole-file (skill-author), twelve words under the
# ceiling — this is a gate two files have already grown into, not headroom.
# such a file sit at exactly the ceiling and spend ~5,320 tokens. A
# median-density file at 2770 words spends ~4,620 tokens, so typical prose
# gives up ~140 words of headroom to close that gap. Densest file today:
# git-commits at 7.34 chars/word.
#
# THE CORPUS IS NOWHERE NEAR THIS CEILING ANY MORE, and the note that used to
# stand here — "a gate two files have already grown into" — described the
# pre-retrofit corpus and is now wrong by a factor of three. The largest
# SKILL.md is write-docs at 914 whole-file words, then vale-run at 874;
# skill-author, the old high-water mark at 2,760, is down to 661. MAX_WORDS is
# a spec-conformance backstop with roughly 1,850 words of slack, and the gate
# that actually bites is ADR-0020's 900-word body budget below it. Do not read
# the two as redundant: they measure different spans, and a file can sit well
# inside one while failing the other.
#
# It is a one-sided proxy in the useful direction — nothing under the word
# ceiling is wildly over the token ceiling — but it is not exact BPE
# tokenization and does not replace one. Re-measure the corpus before treating
# any of these numbers as still current.
# tokenization and does not replace one. Re-measure the corpus, on the basis
# stated above, before treating any of these numbers as still current.
#
# python3 AND PyYAML are required for the ADR-0020 half, and both are hard
# dependencies rather than best-effort: python3 because pre-commit (which is how
@@ -263,7 +281,15 @@ def _collect_package(pkg_dir, names):
safe_dir = glob.escape(pkg_dir)
for sub in ('.apm/skills/*/', 'skills/*/'):
for path in glob.glob(os.path.join(safe_dir, sub)):
names.add(os.path.basename(path.rstrip('/')).lower())
# A directory is a skill only if it HOLDS a SKILL.md. An empty
# leftover — a deleted skill whose directory survived, a scaffolding
# stub, an editor's stray mkdir — is untracked by git, so it exists
# on the machine that made it and nowhere else. Counting it made a
# boundary target resolve locally and dangle in a fresh clone: the
# same install-dependence the deployed-tree rule above exists to
# remove, arriving through a different door.
if os.path.isfile(os.path.join(path, 'SKILL.md')):
names.add(os.path.basename(path.rstrip('/')).lower())
for sub in ('.apm/agents/*.md', 'agents/*.md'):
for path in glob.glob(os.path.join(safe_dir, sub)):
base = os.path.basename(path)
@@ -567,12 +593,33 @@ ROUTE_ANY = re.compile(r"\b%s\s+(?:the\s+|an?\s+)?%s" % (ROUTE_VERB, ANY_TARGET)
CONT_MARKED = re.compile(r"\s*(?:or|and|/|,)\s*%s" % MARKED_TARGET, re.I)
CONT_ANY = re.compile(r"\s*(?:or|and|/|,)\s*%s" % ANY_TARGET, re.I)
ARROW_MARKED = re.compile(r"(?:->|→)\s*%s" % MARKED_TARGET, re.I)
ARROW_BOUNDARY = re.compile(r"\bnot\b[^.;]*?(?:->|→)\s*(%s)\b" % NAME_HYPH, re.I)
# CLAUSE_BODY is what may sit between `Not` and the arrow, and it is NOT
# `[^.;]`. That class cannot cross a `.`, so every boundary clause naming a
# DOTTED FILENAME between the two — `.pre-commit-config.yaml`, `AGENTS.md`,
# `.vale.ini` — was invisible to both patterns below, and the two resulting
# failures were different sizes (issue #110):
# * with a BACKTICKED target the clause was MISDIAGNOSED. The backtick sweep
# still extracted the target, so the route was checked, but the gate
# reported "no boundary clause" on a clause that was present and working.
# Three authors in two retrofit waves reworded a correct clause to satisfy
# the regex, one of them stripping the very filename that discriminates the
# skill from its neighbour.
# * with a BARE target the clause was UNCHECKED. ARROW_BOUNDARY is the only
# extractor for a bare arrow target, so `Not AGENTS.md -> no-such-skill`
# produced no target, no dangling report and no missing-clause SUGGESTION.
# Silence, not noise — the worse of the two failure modes.
# A dot inside a filename is followed by a non-space; a sentence-ending dot is
# followed by whitespace or by end of string. So the class admits a `.` only
# when the next character is not whitespace, which crosses `AGENTS.md` and
# still stops at a real sentence end.
CLAUSE_BODY = r"(?:[^.;]|\.(?=\S))"
ARROW_BOUNDARY = re.compile(
r"\bnot\b%s*?(?:->|→)\s*(%s)\b" % (CLAUSE_BODY, NAME_HYPH), re.I)
BACKTICK = re.compile(r"`(%s)`" % NAME_HYPH, re.I)
# A boundary clause takes two shapes and BOTH count: the prose markers, and
# ADR-0020's compressed arrow form `Not <thing> -> <name>`.
BOUNDARY_MARKER = re.compile(r"\b(?:do\s+not|instead|rather\s+than|not\s+for)\b", re.I)
BOUNDARY_ARROW = re.compile(r"\bnot\b[^.;]*?(?:->|→)", re.I)
BOUNDARY_ARROW = re.compile(r"\bnot\b%s*?(?:->|→)" % CLAUSE_BODY, re.I)
# Sentence boundaries decide the CORROBORATION scope above, so getting one wrong
# is not cosmetic — it moves a target between SUGGESTION and blocking ERROR. Two
# shapes common in these descriptions defeat the naive "period, space, capital"
@@ -592,9 +639,17 @@ BOUNDARY_ARROW = re.compile(r"\bnot\b[^.;]*?(?:->|→)", re.I)
# a lowercase letter. Verified zero-delta on the current corpus (37 ERROR / 58
# SUGGESTION / 2 dangling before and after) — this protects the descriptions
# issue #99 is about to rewrite, not the ones already measured.
# re.I here too, and NOT as a tidy-up: this was the one pattern in the file
# built without it, contradicting the uniformity note on CONT_*/ARROW_* above.
# Without the flag `E.g.` and `I.e.` — the sentence-initial spellings, which is
# where an abbreviation most often lands — matched none of the lookbehinds, so
# the clause split at the abbreviation, the corroborating target was stranded on
# the far side of the cut, and a genuinely dangling target silently demoted from
# blocking ERROR to SUGGESTION. That is the OVER-SPLIT failure described
# directly above, still live for exactly the capitalised half of the input.
SENTENCE_SPLIT = re.compile(
u'(?<!\\be\\.g\\.)(?<!\\bi\\.e\\.)(?<!\\betc\\.)(?<!\\bvs\\.)(?<!\\bcf\\.)'
u'(?<=[.!?])\\s+(?=[A-Za-z`"“(])')
u'(?<=[.!?])\\s+(?=[A-Za-z`"“(])', re.I)
# The token that may follow a route target without turning it into a compound
# modifier: punctuation, end of sentence, a conjunction, a boundary word, or a
@@ -653,11 +708,26 @@ def _notation(text, start, arrow):
def _add(out, text, name, start, end, strict=None, arrow=False):
"""Record one target as (name, may_dangle, notation).
NOTATION IS DECIDED FIRST, and when it is set the follower test is skipped.
The header above promises that route notation "always blocks", and for the
`/name` form that was false: `-> name` reached this function with
strict=True from its two call sites, but `/name` did not, so it fell to
_terminal() and a follower outside FOLLOWER_OK set may_dangle=False. The
target then reached unresolved_targets() unblockable — and, before the
companion fix there, unreported as well. `... use /no-such-skill
afterwards.` exited 0 in total silence, on the one form ADR-0020 offers an
author who wants a route checked unconditionally.
"""
if not name:
return
notation = _notation(text, start, arrow)
if strict is None and notation:
strict = True
out.append((name,
_terminal(text, end) if strict is None else strict,
_notation(text, start, arrow)))
notation))
def _scan(text, route_re, cont_re, out):
@@ -716,6 +786,85 @@ def boundary_targets(description):
return sorted({name for name, _, _ in _extract(description)})
def _arrow_targets(description):
"""Names extracted from ARROW notation specifically.
Kept apart from boundary_targets() because the arrow form is the one shape
that ALWAYS names a target: ADR-0020's `Not <thing> -> <name>`. A clause
written that way from which nothing could be extracted is a parse failure
that deserves its own message, and telling it apart needs the arrow targets
alone rather than every target in the description.
"""
out = []
for sentence in SENTENCE_SPLIT.split(description):
for match in ARROW_MARKED.finditer(sentence):
name, _, _ = _first(match)
if name:
out.append(name)
for match in ARROW_BOUNDARY.finditer(sentence):
out.append(match.group(1))
return out
def boundary_clause_status(description):
"""'absent', 'unparsed' or 'present' — three outcomes, not two.
Issue #110's standing request: the gate must distinguish "no boundary
clause" from "boundary clause I could not parse". Reporting the first for
the second sends the author hunting for a problem that is not there, and
three of them reworded a correct clause to satisfy a regex instead.
'unparsed' is the narrow, certain case: an ADR-0020 arrow clause was
detected and NO target came out of it. The arrow form always names one, so
zero targets means the name is written in a shape the extractor cannot see
— a single-word bare target (`Not X -> forge`, which has to be written
`` `forge` `` or `/forge`) is the live example, since single-word names are
deliberately not matchable bare.
A PROSE clause yielding no target is NOT reported: "Do not use for anything
else" is a complete and legitimate boundary clause that names nowhere to go.
"""
if BOUNDARY_ARROW.search(description) and not _arrow_targets(description):
return 'unparsed'
if has_boundary_clause(description):
return 'present'
return 'absent'
def multi_target_arrow_clauses(description):
"""[(first, second)] for arrow clauses naming more than one target.
Issue #107: only the FIRST target after an arrow is resolved. The
conjunction continuation (CONT_*) is wired to the prose route verbs and
never to arrows, so `Not X -> a or b` resolved `a`, left `b` neither
resolved nor reported, and then printed "1 of 1 boundary target(s) resolve"
on a clause naming two — a gate under-reporting its own coverage, which is
the one failure mode ADR-0020 says a gate must not have.
The clause is REJECTED rather than the arrow scan extended. Extending it
would widen the resolver's deliberately conservative false-positive tuning
across every arrow in the corpus; rejecting costs nothing and makes the
one-arrow-per-target convention — already what every retrofitted gitea
skill does in practice — explicit instead of folkloric. The caller emits a
SUGGESTION telling the author to split.
"""
hits = []
for sentence in SENTENCE_SPLIT.split(description):
matches = (list(ARROW_MARKED.finditer(sentence))
+ list(ARROW_BOUNDARY.finditer(sentence)))
for match in matches:
first, _, _ = _first(match)
if not first:
continue
cont = CONT_ANY.match(sentence, match.end())
if not cont:
continue
second, _, _ = _first(cont)
if second:
hits.append((first, second))
return hits
def unresolved_targets(description, known):
"""Targets resolving to nothing, split into (blocking, reported).
@@ -732,6 +881,17 @@ def unresolved_targets(description, known):
Everything else is reported and left alone. `known` is the resolved
universe from known_targets(); passing an empty set is not meaningful —
callers check for that first and decline out loud instead.
A NON-TERMINAL target is reported, never dropped. FOLLOWER_OK is a closed
whitelist of maybe eighty words, so the follower rule says "this token is
outside a list I keep" and not "this is prose" — and the old `continue`
turned that into invisibility at every tier. The gate then failed OPEN on
its own unfamiliarity: any target followed by a word nobody thought to
enumerate was neither blocked nor mentioned, so the check that did not run
said nothing about not running. The follower rule may withdraw the power to
BLOCK a commit — that is what it was added for, and the ATTRIBUTIVE USE note
above is the argument for it — but it may not withdraw visibility, which is
the same rule the corroboration tier already follows.
"""
blocking, reported = set(), set()
for sentence in SENTENCE_SPLIT.split(description):
@@ -740,7 +900,10 @@ def unresolved_targets(description, known):
if normalize_target(name) in known}
for name, may_dangle, notation in found:
key = normalize_target(name)
if key in known or not may_dangle:
if key in known:
continue
if not may_dangle:
reported.add(name)
continue
if notation or (resolved - {key}):
blocking.add(name)
@@ -820,6 +983,47 @@ def description_value(fm_text):
return re.sub(r'\s+', ' ', value).strip()
def hand_invoked(fm_text):
"""True when the frontmatter marks this file as reached only by hand.
`disable-model-invocation: true` removes a skill from the model-visible
listing entirely — it is not preloaded, and the Skill tool refuses to call
it — so its description is never matched against user intent. ADR-0020 and
skill-author's contract give such a skill ONE plain human-facing sentence:
no trigger list, no boundary clause. No validator knew the field existed
(issue #108), so the boundary-clause SUGGESTION fired on exactly the shape
the contract mandates, and its remedy — "add a boundary clause so the router
knows where NOT to send this skill" — was addressed to a router that cannot
see the skill at all. An author who followed the advice made the file worse.
Only the ROUTING rules are lifted. The body word budget still applies: the
body is loaded on invocation like any other, and competes with the caller's
live conversation the same way. So does the 400-character description FAIL —
a hand-invoked description is not preloaded, but it is still the one line
the user reads when choosing from the `/` menu, and the ceiling is the
outlier stop rather than the style target.
A parse failure returns False rather than raising. This is a MODIFIER on
other checks, not a check of its own: the frontmatter's validity is decided,
and failed, by description_value() on the same text, and raising a second
exception here would report one broken file twice with two different
diagnoses.
"""
try:
data = yaml.safe_load(fm_text)
except Exception:
return False
if not isinstance(data, dict):
return False
value = data.get('disable-model-invocation')
if isinstance(value, str):
# PyYAML already resolves the unquoted YAML 1.1 booleans, so this only
# catches a QUOTED "true" — which a host reads as truthy and which no
# gate should treat as opting back in to the routing rules.
return value.strip().lower() in ('true', 'yes', 'on')
return value is True
# --- Body-shape checks (skills only; agents have no references/ dir) -------
# Deterministic and countable, so they are enforced here. Whether a given
# gotcha is WARRANTED is semantic and stays the auditor's judgment, which is why
@@ -1009,6 +1213,9 @@ for path in files:
body = content[fm_match.end():]
skill_dir = os.path.dirname(os.path.abspath(path))
# ADR-0020's hand-invocation carve-out. See hand_invoked() for what it lifts
# and, more importantly, what it does not (issue #108).
by_hand = hand_invoked(fm_match.group(1))
# An absent or empty description is an ERROR here too, not a silent skip.
# All three ADR-0020 scripts have to agree on this input: the description is
@@ -1030,7 +1237,12 @@ for path in files:
"enumeration, output-format detail, composition notes and implementation "
"detail to the body or README.md."
% (path, dlen, DESC_MAX_CHARS))
elif dlen > DESC_SUGGEST_CHARS:
elif dlen > DESC_SUGGEST_CHARS and not by_hand:
# The 250-character TARGET is a routing-quality budget: it exists to
# keep the preloaded listing small and the trigger clause sharp. A
# hand-invoked description is in no listing, so there is no budget to
# spend and no shape to enforce. The 400-character FAIL above still
# applies — see hand_invoked().
suggest("%s: description is %d characters, over the %d-character target "
"(ADR-0020, hard fail at %d)."
% (path, dlen, DESC_SUGGEST_CHARS, DESC_MAX_CHARS))
@@ -1073,15 +1285,40 @@ for path in files:
round(100.0 * section_words / body_words),
round(100.0 * GOTCHA_MAX_BODY_FRACTION)))
# Missing boundary clause. SUGGESTION, not ERROR: detecting the absence is
# Boundary clause. SUGGESTION, not ERROR: detecting the absence is
# deterministic, but whether this particular skill warrants one is the
# auditor's call. Both accepted shapes count — the prose markers and
# ADR-0020's compressed `Not <thing> -> <name>` arrow.
if desc and not has_boundary_clause(desc):
suggest("%s: description has no boundary clause (ADR-0020). Add the prose form "
"(\"Do not use for X — use `y` instead\") or the compressed form "
"(\"Not X -> y\") so the router knows where NOT to send this skill."
% path)
#
# THREE outcomes, not two. Reporting "no boundary clause" for a clause that
# is present and merely unparsed is a wrong finding, not a strict one, and
# it cost three authors a reworded clause before it was diagnosed (#110).
#
# Skipped entirely for a hand-invoked skill: the contract gives it one plain
# sentence with no boundary clause, so the finding is wrong and its remedy
# names a router that cannot see the skill (#108).
if desc and not by_hand:
status = boundary_clause_status(desc)
if status == 'absent':
suggest("%s: description has no boundary clause (ADR-0020). Add the prose form "
"(\"Do not use for X — use `y` instead\") or the compressed form "
"(\"Not X -> y\") so the router knows where NOT to send this skill."
% path)
elif status == 'unparsed':
suggest("%s: description has an arrow boundary clause (\"Not X -> y\") from which "
"no target could be read, so the dangling-target check did not run on it "
"(ADR-0020). The clause is present — this is a PARSE failure, not a "
"missing clause. Most often the target is a single word, which is "
"deliberately not matchable bare because `research`, `triage` and `forge` "
"are all ordinary English: write it as `name` or /name." % path)
# One arrow, one target. A second name after the arrow is resolved by
# nothing and reported by nothing, so the clause claims coverage it does
# not have (#107).
for first, second in multi_target_arrow_clauses(desc):
suggest("%s: an arrow boundary clause names more than one target ('%s', then "
"'%s'), and only the first is resolved — the second is checked by "
"nothing (ADR-0020). Split it into one arrow per target: "
"\"Not X -> %s. Not Y -> %s.\"" % (path, first, second, first, second))
targets = boundary_targets(desc)
if targets: