Defame1297/holocron
0
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases 3 Wiki Activity

refactor(skills): retrofit the corpus to the ADR-0020 context contract (#129)

Browse Source 
Retrofits all 39 skills to ADR-0020's description/body context contract, then fixes what six rounds of independent review found in that retrofit — including four ways the hot gate itself failed open.

Closes #99, #107, #108, #110, #111, #114, #115, #120.

## The retrofit (waves 1-5)

| | Start | Now |
|---|---|---|
| Description FAILs (>400 chars) | 26 | **0** |
| Body FAILs (>900 words, body-only) | 9 | **0** |
| Dangling routing targets | 2 | **0** |
| `Kyberforge.CompositionNote` | 10 | **0** |
| Preload tax | 21,005 chars | **~10,500** |

Under the 12,000-char success criterion. Per-wave detail is on #99.

## The review fixes

**The gate failed open four ways, three of them found after the retrofit shipped.** An unrecognised follower token made a dangling target vanish. A skill directory with no `SKILL.md` resolved as a valid target, so a commit could be green locally and red in a fresh clone — three existing fixtures were relying on that, one of which made the install-leak A/B pass vacuously. Then the free-standing `/name` sweep turned out to be gated on the sentence carrying a boundary marker, so route notation in any other sentence was invisible — not an ERROR, not a SUGGESTION, not an INFO — which left the documented "`/name` always blocks" promise false from a second direction. All four fixed and pinned.

**Two checks were silently not running.** `validate-provenance.sh` checks 7-8 were dead across nine skills. Waking them exposed a deeper problem: they assume `Research doc:` names a source index, but 30 of 121 entries point at topic content documents, so every new check-7 INFO was a false positive and check 8 was saved from a false-FAIL flood only by an *unannounced* skip. Checks 7/8 are now scoped to source indexes and every skip announces itself (#121).

**The retrofit's own anti-goal, four times.** ADR-0020 warns that a blunt gate gets satisfied by deleting content rather than relocating it. `diagnose` and `skill-audit` relocated prose and then read it unconditionally; `prototype` and `vale-config` deleted rules outright that survived nowhere. All four addressed.

## Verification

- `bash tests/run-tests.sh --strict` — 24 suites, 0 skipped, 0 failed
- `bash tests/run-bats.sh` — 325 tests, 0 failures
- `pre-commit run --all-files` — 17/17
- `pre-commit run --hook-stage pre-push --all-files` — 16/16, with `apm marketplace check` and `apm pack --check-clean` run against the remote, not skipped
- `scripts/skill-size-check.sh` over all 39 skills — rc 0, 0 ERROR/FAIL, SUGGESTION-only
- Preload tax measured at **10,498 chars**, max description 390 — both inside budget
- Every new test proven non-vacuous by a deliberate mutation of the behaviour it covers

**Per-commit sync, stated accurately:** the ten commits from the latest review round each pass `check-plugin-content-sync` in isolation, verified by checking each out in a detached worktree with a clean between. The earlier gitea window (`dfacf05..bedbd1d`, nine commits) does **not** — its mirror was regenerated in one batch at `bbc7300`. An earlier revision of this description claimed the property held for every commit; it does not, and a bisect through that window lands on a red commit. **Squash-merge** to collapse it, or accept that this range is not bisectable.

## Version bump

Six plugins and the catalog take a **patch**, not a minor. The branch is **89 commits — 40 `fix` / 30 `refactor` / 12 `docs` / 5 `chore` / 2 `test` — zero `feat`, zero `!`, zero `BREAKING CHANGE`** — and adds no skill, agent, command or hook. (Two earlier revisions of this section cited a stale histogram, most recently 78 commits; the figures above are measured at HEAD.) Both rules this repo ships (`forge/references/version-bump.md`, landing in this PR, and `git-commits/references/conventional-commits-spec.md`) make that a patch, and the catalog set is unchanged at 7 entries.

Not settled by that: four published files were removed from the installed tree, three moved, and `caveman` gained `disable-model-invocation`, retiring its old triggers. Under a strict reading those are major-class and currently ship under `refactor:` with no marker. Whether the deployed skill surface is a public contract is written down nowhere — worth deciding, but it outlives this PR.

## Deliberately not in scope

#112 (cherry-pick ownership, now resolved in favour of `git-commits`), #113 (`rtk git` normalisation), #116 (research fan-out), #101 (audit-skill merge), #122 (non-spec skill-root files), #123 (no PRD producer) stay open. #117 is the one worth reading: the contract's remedy is to move prose into `references/`, which is exactly where neither the size gate nor Vale looks — and the blind spot is wider than #117 currently records, since there is no root `.vale.ini` at all, so every ADR, `CONTEXT.md` and `README.md` is unlinted too.

That blind spot let this branch carry two `level: error` `Kyberforge.SentenceOpenerThereIs` violations into `references/` files it created — `provider-adapter-author/references/provider-matrix.md:31` and `agent-audit/references/finding-criteria.md:95`. Both are reworded in `afadaae`, confirmed by routing each file through the audit's own `vale-wrap.sh` (1 error each before, 0 after). Five further occurrences sit in `references/` files already on `main`; those are the pre-existing corpus and stay with #117, which is the real fix.

Also unfixed and not this PR's: `apm install` appends a duplicate `SessionStart` entry to `.claude/settings.json`, so a fresh clone cannot get pre-push green without an edit AGENTS.md warns against. Reproduces identically on `main`.

Co-authored-by: Defame1297 <gitea@rkdr.net>
Reviewed-on: https://git.dev.rkdr.net/Defame1297/holocron/pulls/129
Co-authored-by: Claude Code AI - Gitea MCP <claude@noreply.git.dev.rkdr.net>
Co-committed-by: Claude Code AI - Gitea MCP <claude@noreply.git.dev.rkdr.net>
This commit was merged in pull request #129.
This commit is contained in:
Claude Code AI - Gitea MCP
2026-09-01 13:47:46 +00:00
committed by Defame1297
parent 0e91a3ae66
commit 598a7c326a
420 changed files with 15303 additions and 4740 deletions
Download Patch File Download Diff File Expand all files Collapse all files

611
plugins/kyberforge/skills/skill-audit/scripts/validate-provenance.sh
View File

@@ -13,17 +13,37 @@ Arguments:
Exit codes:
0 All checks passed (or nothing to validate)
1 One or more checks failed
2 Usage error, or the argument is not a skill directory
An exit code of 2 is NOT a finding. SKILL.md tells the auditor to surface a
non-zero exit as findings, so a usage error leaving exit 1 with nothing on
stdout was indistinguishable from a clean-but-failing run. Environment and
argument problems exit 2; only real findings exit 1.
Checks performed:
0 source_keys present but references/sources.md absent
1 FILL IN: placeholders in sources.md
2 source_keys in SKILL.md → slug exists in sources.md
3 source_keys in references/*.md → slug exists in sources.md (INFO if no source_keys)
4 Contributing files listed in sources.md exist on disk
3 source_keys in references/*.md → slug exists in sources.md (INFO if no
source_keys; an explicit 'source_keys: []' declares the file house-authored
and passes silently)
4 Contributing files listed in sources.md exist on disk. An explicit
'(none)' skips silently; a Contributing files block this parser cannot
read is reported as an INFO saying checks 4 and 5 did not run, never
skipped silently.
5 Contributing files back-reference the parent slug in their source_keys
6 Research doc field present and not placeholder
7 Slug in sources.md present in upstream research doc (INFO only)
7 Slug in sources.md present in upstream research doc (INFO only). A section
annotation ('§ ...', '→ ...', '(...)') is stripped before the path is
resolved; a path that still does not resolve is reported as an INFO saying
checks 7 and 8 did not run, never skipped silently.
8 Extracted non-(none) slug in research doc present in sources.md
Checks 7 and 8 apply ONLY when the Research doc value names a research SOURCE
INDEX — a file whose basename is sources.md, whose H2 headings ARE source
slugs. A Research doc pointing at a topic document is reported as an INFO
saying the two checks are not applicable, and every other reason they do not
run is announced the same way.
EOF
}
@@ -32,11 +52,57 @@ if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then
exit 0
fi
# Usage and environment problems exit 2, findings exit 1. See the usage text
# above for why the two must not share a code. This is a deliberate divergence
# from validate.sh, which has no 2 tier: validate.sh always prints PASS lines,
# so a usage error there is visibly not a findings report. This script prints
# NOTHING on a clean run, so exit 1 plus empty stdout was the only signal a
# caller got either way.
if [[ $# -lt 1 ]]; then
echo "Error: skill-dir is required." >&2
echo "" >&2
usage >&2
exit 1
exit 2
fi
# Extra positional arguments were silently dropped, so a typo'd flag or a second
# path looked like it had been honoured.
if [[ $# -gt 1 ]]; then
echo "Error: expected exactly one argument, got $#: $*" >&2
echo "" >&2
usage >&2
exit 2
fi
# python3 is a HARD dependency. Without this preflight a missing interpreter
# produced 'line NN: python3: command not found' and exit 127 — an exit code no
# caller maps to anything, from a message that names this script's line number
# rather than the missing dependency.
if ! command -v python3 > /dev/null 2>&1; then
echo "Error: python3 is required but was not found on PATH." >&2
echo " Why: skipping the provenance checks entirely would be a vacuous pass." >&2
echo " Fix: install python3 (pre-commit itself is a Python application, so it is almost certainly already present)." >&2
exit 2
fi
# A path that is not a directory, or a directory that is not a skill, used to
# reach the Python body, find no sources.md and no source_keys, take the
# "nothing to validate" early exit and report exit 0 with no output — which
# references/validation-scripts.md explicitly told the auditor to read as a
# pass. A typo'd target was therefore indistinguishable from a clean skill.
# vale-wrap.sh hard-errors on a nonexistent path for exactly this reason.
if [[ ! -d "$1" ]]; then
echo "Error: not a directory: $1" >&2
echo " Why: a nonexistent target would otherwise report a silent pass." >&2
echo " Fix: pass the path of the skill directory to validate." >&2
exit 2
fi
if [[ ! -f "$1/SKILL.md" ]]; then
echo "Error: not a skill directory (no SKILL.md): $1" >&2
echo " Why: a directory with no SKILL.md has no provenance chain to validate, and reporting that as a pass hides the wrong-target mistake." >&2
echo " Fix: pass the skill directory itself, not its parent or its references/ subdirectory." >&2
exit 2
fi
python3 -u - "$1" <<'PYTHON'
@@ -44,13 +110,67 @@ import sys
import os
import re
# Output is UTF-8 for the same reason input is: under LC_ALL=C the streams
# default to ASCII, and every finding this script prints contains an em dash.
# Pinning only the reads moved the crash from the read to the write — a
# UnicodeEncodeError inside print_findings(), which loses the whole report
# after all the checks have already run.
for _stream in (sys.stdout, sys.stderr):
try:
_stream.reconfigure(encoding='utf-8')
except AttributeError: # pragma: no cover — Python < 3.7
pass
skill_dir = os.path.abspath(sys.argv[1])
sources_md_path = os.path.join(skill_dir, "references", "sources.md")
refs_dir = os.path.join(skill_dir, "references")
# --- Helpers ---
PLACEHOLDER_RE = re.compile(r'(?<!`)FILL IN:[^`\n]')
# The trailing character class used to be CONSUMING — `[^`\n]` — so a
# `FILL IN:` at end of line matched nothing and escaped checks 1 and 6
# entirely. `- **Description:** FILL IN:` is the most likely spelling of a
# half-written entry, and it was the one spelling the placeholder gate could
# not see. The exclusion it was really expressing is "not inside backticks",
# which a lookahead states without eating a character.
PLACEHOLDER_RE = re.compile(r'(?<!`)FILL IN:(?!`)')
# --- Input ----------------------------------------------------------------
# Ported from validate.sh, where the same two problems were already fixed.
#
# read_text() pins UTF-8 explicitly instead of inheriting
# locale.getpreferredencoding(), which is ASCII under LC_ALL=C — an ordinary em
# dash in a references file then aborted the run with a bare UnicodeDecodeError
# traceback, or, at the one call site that wrapped its read in `except
# Exception: return False`, reported the unreadable file as having no
# source_keys and therefore as clean. A file that genuinely is not UTF-8 still
# fails; it just says which file and why.
#
# strip_bom() runs on every read because a leading BOM defeats
# parse_frontmatter()'s `^---` anchor, which silently disabled check 2 on a
# BOM-prefixed SKILL.md: no frontmatter parsed means no source_keys parsed
# means nothing to validate.
class EncodingError(Exception):
pass
def strip_bom(text):
return text[1:] if text.startswith(u'\ufeff') else text
def read_text(path):
"""File contents as text, UTF-8 and BOM-free, with a diagnostic instead of a traceback."""
try:
with open(path, encoding='utf-8') as fh:
return strip_bom(fh.read())
except UnicodeDecodeError as exc:
raise EncodingError(
"not valid UTF-8 (%s at byte %d) — re-save the file as UTF-8; "
"this gate does not guess at other encodings"
% (exc.reason, exc.start))
def parse_frontmatter(content):
"""Return (frontmatter_str, body_str) or (None, content) if no frontmatter."""
@@ -89,13 +209,84 @@ def parse_source_keys(fm):
in_metadata = False
return keys
# An explicit `source_keys: []` — top-level or under metadata: — is a
# DECLARATION that the file is house-authored and has no external source.
# parse_source_keys() returns [] both for that and for a file with no
# source_keys key at all, so the two are indistinguishable downstream and
# check 3 emitted the same INFO for each (#111). That left no honest way to
# record "this file has no external source": the only ways to silence the INFO
# were to invent a slug or borrow an unrelated one, both false provenance
# claims that then have to be maintained in sources.md as well. A bare
# `source_keys:` with nothing after it is NOT accepted here — that reads as a
# truncated or half-written entry, not a decision.
#
# The indent is pinned to the two positions parse_source_keys() actually reads
# — column 0, or two spaces under `metadata:`. A permissive `^\s*` matched a
# `source_keys: []` nested at ANY depth under an unrelated key, which
# parse_source_keys() never reads, so a stray nested key silenced the check-3
# INFO for a file that had declared nothing.
EMPTY_SOURCE_KEYS_RE = re.compile(r'^(?: )?source_keys:\s*\[\s*\]\s*$')
def declares_empty_source_keys(fm):
"""True when frontmatter carries an explicit, empty `source_keys: []`."""
if fm is None:
return False
return any(EMPTY_SOURCE_KEYS_RE.match(line) for line in fm.splitlines())
def parse_h2_slugs(content):
"""Return list of H2 heading values from a markdown file."""
return re.findall(r'^## (.+)$', content, re.MULTILINE)
# ===== BEGIN SHARED CONTRIBUTING-FILES PARSER =====
# ONE parser, embedded VERBATIM in two scripts:
# plugins/kyberforge/.apm/skills/skill-audit/scripts/validate-provenance.sh
# plugins/kyberforge/.apm/skills/agent-audit/scripts/validate-provenance.sh
# The block between these markers must stay byte-identical in both. It is
# copied rather than imported because a cache-installed plugin's scripts cannot
# read files outside their own plugin directory, so there is no single file both
# can share — the same constraint that forces the ADR-0020 boundary resolver to
# be duplicated across three scripts. Edit one copy, then paste it over the
# other.
#
# tests/test-adr0020-contract.sh hashes both copies and fails on drift. Before
# it did, the agent-audit copy's docstring merely ASSERTED the two were
# "behaviourally identical" and nothing checked it — which is how the two
# already-diverged spellings of the bullet loop went unnoticed.
#
# Requires: re (imported by the host script).
def parse_contributing_files(content, slug):
"""Find the Contributing files value for a given slug H2 in content."""
# Find the H2 block for slug, then look for Contributing files line
"""Find the Contributing files for a given slug H2 in content.
Both authored forms are accepted, because both are in use across the
corpus and only recognising the first silently skipped the contributing-
file checks on every sources.md written the other way:
- **Contributing files:** SKILL.md, references/a.md
**Contributing files:**
- SKILL.md (what this source contributed)
- references/a.md (what this source contributed)
Returns a list of paths with any trailing parenthetical note stripped.
Note the bullet form's notes may themselves contain commas, so the list
is built per bullet rather than by splitting the joined value.
The three return values are NOT interchangeable, and callers depend on
the distinction:
[path, ...] the entry names contributing files
[] the entry EXPLICITLY records "(none)"
None the entry says nothing this parser can read
Only an explicit "(none)" yields []. A "Contributing files:" heading
followed by a numbered list, by `*` bullets, or by prose parses nothing
and returns None, never [] — a caller reads [] as a deliberate "no
contributing files" record and SKIPS its check on that basis, so a parse
failure returning [] would silently disable the check instead of leaving
the unreadable entry exposed to it.
"""
pattern = re.compile(
r'^## ' + re.escape(slug) + r'\s*\n(.*?)(?=^## |\Z)',
re.MULTILINE | re.DOTALL
@@ -104,28 +295,113 @@ def parse_contributing_files(content, slug):
if not m:
return None
block = m.group(1)
def strip_note(entry):
# "references/a.md (why)" -> "references/a.md"
return re.sub(r'\s*\(.*$', '', entry).strip()
# Inline form: value on the same line, comma-separated, no notes.
cf_m = re.search(r'^\- \*\*Contributing files:\*\* (.+)$', block, re.MULTILINE)
if cf_m:
value = cf_m.group(1).strip()
if value.startswith("(none"):
return []
return [p for p in (strip_note(x) for x in value.split(","))
if p] or None
# Bullet form: heading on its own line, one file per following bullet.
cf_m = re.search(r'^\*\*Contributing files:\*\*\s*$', block, re.MULTILINE)
if not cf_m:
return None
return cf_m.group(1).strip()
files = []
for line in block[cf_m.end():].splitlines():
line = line.strip()
if not line:
if files:
break
continue
if not line.startswith("- "):
break
entry = line[2:].strip()
if entry.startswith("(none"):
return []
entry = strip_note(entry)
if entry:
files.append(entry)
return files or None
# ===== END SHARED CONTRIBUTING-FILES PARSER =====
def parse_research_doc(content, slug):
"""Find the Research doc value for a given slug H2 in content."""
def parse_research_docs(content, slug):
"""Every Research doc value under a given slug H2, in document order.
The caller uses the first and reports the rest. Returning only the first —
what this did before — meant a second '- **Research doc:**' line in one
entry was silently ignored, so an author who added a doc rather than
replacing one got checks 7 and 8 run against the old path and no hint that
the new one was never looked at.
"""
pattern = re.compile(
r'^## ' + re.escape(slug) + r'\s*\n(.*?)(?=^## |\Z)',
re.MULTILINE | re.DOTALL
)
m = pattern.search(content)
if not m:
return None
return []
block = m.group(1)
rd_m = re.search(r'^\- \*\*Research doc:\*\* (.+)$', block, re.MULTILINE)
if not rd_m:
return None
return rd_m.group(1).strip()
return [v.strip() for v in
re.findall(r'^\- \*\*Research doc:\*\* (.+)$', block, re.MULTILINE)]
# A Research doc value is a path, and very often a path PLUS an annotation
# naming the section the slug came from:
#
# plugins/git/docs/research/docs/git/gitflow.md (whole-document reference)
# plugins/git/docs/research/docs/git/remotes.md → `## Pushing (`git push`)`
# .../pre-commit/hooks-reference.md § "pre-commit-hooks (official collection)"
#
# os.path.isfile() is false for every one of those strings, and checks 7 and 8
# used to skip SILENTLY whenever the path did not resolve. The effect was that
# both checks were dead on eight of the nine git skills — git-history, the one
# skill writing a bare path, was the only place they ran, which is why it was
# the only skill ever reporting a check-7 INFO. Strip the annotation before
# resolving, and report when the result still does not resolve: a check that
# quietly does not run is worse than one that fails.
RESEARCH_DOC_ANNOTATION_RE = re.compile(r'[§→(]')
def strip_research_doc_annotation(value):
"""Path part of a Research doc value, with any section annotation removed."""
return RESEARCH_DOC_ANNOTATION_RE.split(value, maxsplit=1)[0].strip()
def research_doc_is_none(value):
"""True when a Research doc value declares that no research doc backs the slug.
Both '(none)' and the bare 'none — org convention, ...' spelling are in
use; recognising only the parenthesised one would report the other as an
unresolvable path. Checked BEFORE the annotation strip, because '(none)'
is itself a parenthesis and would strip to the empty string.
"""
return re.match(r'\(?none\b', value.strip(), re.IGNORECASE) is not None
# The Status value is what gates check 8, so every spelling this parser fails
# to read is a check that does not run. Two were unreadable:
#
# - **Status:** `extracted` — partial fetch (a trailing note)
# **Status:** (the bullet form, the same
# - `extracted` shape parse_contributing_files
# already accepts)
#
# Both used to parse to a string that compared unequal to "`extracted`", and
# check 8 skipped on that inequality without a word. Returning the BACKTICKED
# TOKEN — not the whole line — is what makes the trailing note harmless, and it
# lets the caller name the actual status when it announces a skip.
STATUS_TOKEN_RE = re.compile(r'^`([^`]*)`')
def parse_status(content, slug):
"""Find the Status value for a given slug H2 in content."""
"""Find the Status value for a given slug H2 in content.
Returns the status with its backticks stripped ('extracted', 'referenced',
'no content extracted'), or None when the entry has no Status line.
"""
pattern = re.compile(
r'^## ' + re.escape(slug) + r'\s*\n(.*?)(?=^## |\Z)',
re.MULTILINE | re.DOTALL
@@ -134,10 +410,28 @@ def parse_status(content, slug):
if not m:
return None
block = m.group(1)
raw = None
st_m = re.search(r'^\- \*\*Status:\*\* (.+)$', block, re.MULTILINE)
if not st_m:
return None
return st_m.group(1).strip()
if st_m:
raw = st_m.group(1).strip()
else:
st_m = re.search(r'^\*\*Status:\*\*\s*$', block, re.MULTILINE)
if not st_m:
return None
for line in block[st_m.end():].splitlines():
line = line.strip()
if not line:
continue
if not line.startswith("- "):
break
raw = line[2:].strip()
break
if raw is None:
return None
token = STATUS_TOKEN_RE.match(raw)
return token.group(1).strip() if token else raw
def find_repo_root(start_dir):
"""Walk up from start_dir until we find a directory containing .git."""
@@ -153,13 +447,22 @@ def find_repo_root(start_dir):
findings = []
has_fail = False
# A finding identical in every field is the same finding, and the same file is
# now reached by more than one check — the walk that looks for source_keys and
# check 3 both read every references/*.md, so an unreadable one would otherwise
# be reported twice with the same words. Distinct findings about the same file
# still both appear.
def _record(entry):
if entry not in findings:
findings.append(entry)
def emit_fail(desc, fpath, why, fix):
global has_fail
has_fail = True
findings.append(("FAIL", desc, fpath, why, fix, None))
_record(("FAIL", desc, fpath, why, fix, None))
def emit_info(desc, fpath, note):
findings.append(("INFO", desc, fpath, None, None, note))
_record(("INFO", desc, fpath, None, None, note))
def print_findings():
for entry in findings:
@@ -181,11 +484,22 @@ def print_findings():
# --- Scan for any file with source_keys ---
def file_has_source_keys(fpath):
def emit_unreadable(rel, exc):
"""Report a file this script cannot decode. Never a silent skip."""
emit_fail(
f"File is {exc}",
rel,
f"'{rel}' cannot be decoded, so its frontmatter — and any source_keys in it — "
f"cannot be read. This used to be swallowed by a bare 'except Exception: return False', "
f"which reported the unreadable file as having no source_keys and therefore as clean.",
f"Re-save '{rel}' as UTF-8."
)
def file_has_source_keys(fpath, rel):
try:
with open(fpath) as f:
content = f.read()
except Exception:
content = read_text(fpath)
except EncodingError as exc:
emit_unreadable(rel, exc)
return False
fm, _ = parse_frontmatter(content)
if fm is None:
@@ -198,26 +512,33 @@ def find_files_with_source_keys():
for root, dirs, files in os.walk(skill_dir):
# Skip hidden dirs
dirs[:] = [d for d in dirs if not d.startswith('.')]
for fname in files:
for fname in sorted(files):
if fname.endswith('.md'):
abs_path = os.path.join(root, fname)
if file_has_source_keys(abs_path):
rel = os.path.relpath(abs_path, skill_dir)
rel = os.path.relpath(abs_path, skill_dir)
if file_has_source_keys(abs_path, rel):
results.append((rel, abs_path))
return results
sources_md_exists = os.path.isfile(sources_md_path)
files_with_source_keys = find_files_with_source_keys()
# Early exit: nothing to validate
# Early exit: nothing to validate. The scan itself can raise a finding — an
# unreadable file — so print before leaving; the clean case still prints
# nothing and exits 0.
if not sources_md_exists and not files_with_source_keys:
sys.exit(0)
print_findings()
sys.exit(1 if has_fail else 0)
# Load sources.md if it exists
sources_content = None
if sources_md_exists:
with open(sources_md_path) as f:
sources_content = f.read()
try:
sources_content = read_text(sources_md_path)
except EncodingError as exc:
emit_unreadable("references/sources.md", exc)
print_findings()
sys.exit(1)
sources_slugs = set(parse_h2_slugs(sources_content))
else:
sources_slugs = set()
@@ -248,8 +569,11 @@ for line in sources_content.splitlines():
# --- Check 2: source_keys in SKILL.md → slug exists in sources.md ---
skill_md_path = os.path.join(skill_dir, "SKILL.md")
if os.path.isfile(skill_md_path):
with open(skill_md_path) as f:
skill_content = f.read()
try:
skill_content = read_text(skill_md_path)
except EncodingError as exc:
emit_unreadable("SKILL.md", exc)
skill_content = ""
skill_fm, _ = parse_frontmatter(skill_content)
skill_source_keys = parse_source_keys(skill_fm)
for slug in skill_source_keys:
@@ -262,24 +586,43 @@ if os.path.isfile(skill_md_path):
)
# --- Check 3: source_keys in references/*.md → slug exists in sources.md (INFO if no source_keys) ---
# os.walk, not os.listdir: find_files_with_source_keys() above already walks
# references/ recursively, so a source_keys-bearing file in
# references/<subdir>/ was collected there — and then never validated here,
# because the flat listdir could not see it. The two halves of the same check
# disagreed about which files exist.
if os.path.isdir(refs_dir):
for fname in sorted(os.listdir(refs_dir)):
if not fname.endswith('.md'):
continue
if fname == "sources.md":
continue
fpath = os.path.join(refs_dir, fname)
ref_paths = []
for root, dirs, files in os.walk(refs_dir):
dirs[:] = sorted(d for d in dirs if not d.startswith('.'))
for fname in sorted(files):
if not fname.endswith('.md'):
continue
fpath = os.path.join(root, fname)
if os.path.relpath(fpath, refs_dir) == "sources.md":
continue
ref_paths.append(fpath)
for fpath in ref_paths:
rel = os.path.relpath(fpath, skill_dir)
with open(fpath) as f:
ref_content = f.read()
try:
ref_content = read_text(fpath)
except EncodingError as exc:
emit_unreadable(rel, exc)
continue
ref_fm, _ = parse_frontmatter(ref_content)
ref_keys = parse_source_keys(ref_fm)
if not ref_keys:
# An explicit `source_keys: []` is a deliberate declaration that
# the file is house-authored, and passes silently. The INFO is for
# files that never said either way.
if declares_empty_source_keys(ref_fm):
continue
emit_info(
f"No source_keys frontmatter",
rel,
"This references file has no source_keys — provenance cannot be verified. "
"Add source_keys frontmatter listing the slugs from references/sources.md that informed this file."
"Add source_keys frontmatter listing the slugs from references/sources.md that informed this file, "
"or declare an explicit 'source_keys: []' if the file is house-authored and has no external source."
)
else:
for slug in ref_keys:
@@ -296,14 +639,51 @@ if os.path.isdir(refs_dir):
repo_root = find_repo_root(skill_dir)
# Collect all research doc paths we'll check (for Check 8)
research_docs_seen = {} # abs_path → set of slugs in sources.md that reference it
research_docs_seen = {} # abs_path → (rel_path, slugs referencing it, content)
for slug in parse_h2_slugs(sources_content):
# Check 4: Contributing files exist
cf_value = parse_contributing_files(sources_content, slug)
if cf_value and not cf_value.startswith("(none"):
# Split by comma
cf_files = [p.strip() for p in cf_value.split(",") if p.strip()]
# Every per-slug parser below — parse_contributing_files, parse_research_docs,
# parse_status — locates its block with pattern.search(), so a slug written
# twice resolves to the FIRST block every time. Iterating the raw heading list
# therefore checked the first block's fields twice and the second block's
# never: a duplicated slug is half-validated, and looked fully validated. The
# duplicate is announced and the repeat visit dropped.
all_slugs = parse_h2_slugs(sources_content)
unique_slugs = []
for _slug in all_slugs:
if _slug in unique_slugs:
continue
unique_slugs.append(_slug)
_count = all_slugs.count(_slug)
if _count > 1:
emit_info(
f"Duplicate '## {_slug}' entry in sources.md — only the first block is checked",
f"references/sources.md (## {_slug})",
f"'## {_slug}' appears {_count} times. Every field parser here takes the first match, so the "
f"second and later blocks' Contributing files, Research doc and Status are never validated — "
f"checks 4, 5, 6, 7 and 8 did not run for them. "
f"Merge the blocks into one entry, or give each a distinct slug and reference it from source_keys."
)
for slug in unique_slugs:
# Checks 4 and 5: Contributing files exist, and back-reference the slug.
# `[]` and None are NOT the same answer here. `[]` is the author writing
# "(none)" — there is nothing to check and the skip is correct. None is a
# Contributing-files block this parser cannot read, and skipping THAT
# silently disables both checks on the one entry least likely to be right,
# which is the failure mode parse_contributing_files' own docstring warns
# about. Say so out loud instead, the same way an unresolvable Research doc
# value does.
cf_files = parse_contributing_files(sources_content, slug)
if cf_files is None:
emit_info(
f"Contributing-file checks skipped for '{slug}' — the Contributing files block could not be parsed",
f"references/sources.md (## {slug})",
f"The '## {slug}' entry has no Contributing files list this parser can read — a missing field, a bare heading, '*' bullets, a numbered list, or prose all read as unparsable rather than as an empty declaration. "
f"Checks 4 and 5 did not run for this slug, so nothing verified that its contributing files exist or name it back. "
f"Write the value as '- **Contributing files:** <comma-separated paths>', or as a '**Contributing files:**' heading followed by '- ' bullets — "
f"or record '(none)' if this source contributed no files."
)
elif cf_files:
for cf_rel in cf_files:
cf_abs = os.path.join(skill_dir, cf_rel)
if not os.path.isfile(cf_abs):
@@ -318,8 +698,11 @@ for slug in parse_h2_slugs(sources_content):
# Skip sources.md itself
if cf_rel == "references/sources.md":
continue
with open(cf_abs) as f:
cf_content = f.read()
try:
cf_content = read_text(cf_abs)
except EncodingError as exc:
emit_unreadable(cf_rel, exc)
continue
cf_fm, _ = parse_frontmatter(cf_content)
cf_keys = parse_source_keys(cf_fm)
if slug not in cf_keys:
@@ -331,7 +714,17 @@ for slug in parse_h2_slugs(sources_content):
)
# Check 6: Research doc field required
rd_value = parse_research_doc(sources_content, slug)
rd_values = parse_research_docs(sources_content, slug)
if len(rd_values) > 1:
emit_info(
f"Multiple '- **Research doc:**' lines for '{slug}' — only the first is used",
f"references/sources.md (## {slug})",
f"The '## {slug}' entry has {len(rd_values)} Research doc lines; checks 7 and 8 ran against the first "
f"('{rd_values[0]}') and never looked at the rest. "
f"Keep one Research doc line per entry — if a slug genuinely came from two documents, split it into two slugs, "
f"or name the extra document inside the first value's annotation where it is at least visible."
)
rd_value = rd_values[0] if rd_values else None
if rd_value is None:
emit_fail(
f"Research doc field missing",
@@ -346,39 +739,115 @@ for slug in parse_h2_slugs(sources_content):
f"The '## {slug}' entry has an unfilled Research doc value.",
f"Set '- **Research doc:**' to a real path relative to repo root, or '(none)' if not applicable."
)
else:
# Check 7: Upstream forward — slug should appear in research doc
if repo_root and not rd_value.startswith("(none"):
rd_abs = os.path.join(repo_root, rd_value)
if os.path.isfile(rd_abs):
with open(rd_abs) as f:
rd_content = f.read()
elif not research_doc_is_none(rd_value):
# Check 7: Upstream forward — slug should appear in research doc.
# Every path out of here that does NOT run the check says so out loud.
rd_path = strip_research_doc_annotation(rd_value)
if not repo_root:
emit_info(
f"Upstream checks skipped for '{slug}' — no repo root above the skill directory",
f"references/sources.md (## {slug})",
f"'{rd_value}' is a path relative to the repo root, but no ancestor of the skill directory contains a .git entry, "
f"so it cannot be resolved. Checks 7 and 8 did not run for this slug. "
f"Run this script against a skill inside a checkout."
)
elif not rd_path:
emit_info(
f"Upstream checks skipped for '{slug}' — Research doc value names no path",
f"references/sources.md (## {slug})",
f"The Research doc value '{rd_value}' is entirely annotation — stripping the section marker leaves no path. "
f"Checks 7 and 8 did not run for this slug. "
f"Give the value a file path relative to the repo root, or record '(none)' if no research doc backs this entry."
)
else:
rd_abs = os.path.join(repo_root, rd_path)
if not os.path.isfile(rd_abs):
emit_info(
f"Upstream checks skipped for '{slug}' — research doc '{rd_path}' does not exist",
f"references/sources.md (## {slug})",
f"'{rd_value}' resolves to '{rd_path}' relative to the repo root and no file is there. "
f"Checks 7 and 8 did not run for this slug, so nothing verified that the research doc still backs it. "
f"Point the value at one existing file — a brace expansion, a comma-separated list of paths, or a bare section title does not resolve — "
f"or record '(none)' if no research doc backs this entry."
)
elif os.path.basename(rd_path) != "sources.md":
# Checks 7 and 8 both assume the Research doc is a research
# SOURCE INDEX — a sources.md whose H2 headings ARE source
# slugs. 30 of the 121 corpus entries point instead at a TOPIC
# DOCUMENT (remotes.md, gitflow.md, api-reference.md), whose
# H2s are headings like '## Core Philosophy'. A slug can never
# match one, so check 7 reported all 30 as "slug not found" —
# every one a false positive — and check 8, aimed at documents
# that carry no '- **Status:**' line at all, was saved from a
# matching flood of false FAILs only by an UNANNOUNCED skip on
# that missing status. The premise, not the corpus, was wrong.
#
# A topic-document reference is a legitimate, useful value; it
# just is not something these two checks can verify. Say that
# once, out loud, instead of failing 30 entries for it.
emit_info(
f"Upstream checks not applicable for '{slug}' — research doc '{rd_path}' is a topic document, not a source index",
f"references/sources.md (## {slug})",
f"Checks 7 and 8 match slugs against the H2 headings of a research source index — a file named 'sources.md', "
f"where each H2 IS a source slug. '{os.path.basename(rd_path)}' is a topic document, so its H2s are section "
f"headings and no slug will ever match one. Checks 7 and 8 did not run for this slug. "
f"This needs no fix: point the value at the research corpus's own sources.md only if you want the "
f"provenance link machine-verified."
)
else:
try:
rd_content = read_text(rd_abs)
except EncodingError as exc:
emit_info(
f"Upstream checks skipped for '{slug}' — research doc '{rd_path}' is {exc}",
f"references/sources.md (## {slug})",
f"'{rd_path}' could not be decoded, so checks 7 and 8 did not run for this slug. "
f"Re-save the research doc as UTF-8."
)
continue
rd_slugs = set(parse_h2_slugs(rd_content))
if slug not in rd_slugs:
emit_info(
f"Slug '{slug}' not found as H2 in research doc '{rd_value}'",
f"Slug '{slug}' not found as H2 in research doc '{rd_path}'",
f"references/sources.md (## {slug})",
f"The research doc '{rd_value}' does not have a '## {slug}' heading. "
f"The research doc '{rd_path}' does not have a '## {slug}' heading. "
f"The provenance link may be imprecise — the slug name in sources.md may differ from the research doc's heading."
)
# Track for Check 8
# Track for Check 8. The content is carried with the entry so
# check 8 reuses this read rather than decoding the file a
# second time, with a second chance to fail differently.
if rd_abs not in research_docs_seen:
research_docs_seen[rd_abs] = (rd_value, set())
research_docs_seen[rd_abs] = (rd_path, set(), rd_content)
research_docs_seen[rd_abs][1].add(slug)
# --- Check 8: Upstream reverse ---
for rd_abs, (rd_rel, known_slugs) in research_docs_seen.items():
with open(rd_abs) as f:
rd_content = f.read()
for rd_abs, (rd_rel, known_slugs, rd_content) in research_docs_seen.items():
for rd_slug in parse_h2_slugs(rd_content):
# Parse this slug's Contributing files and Status in the research doc
rd_cf = parse_contributing_files(rd_content, rd_slug)
rd_status = parse_status(rd_content, rd_slug)
# Skip if contributing files start with (none
if rd_cf and rd_cf.startswith("(none"):
# Skip if the research doc explicitly records no contributing files
if rd_cf == []:
continue
# Skip if status is not `extracted`
if rd_status != "`extracted`":
# Skip if status is not `extracted` — and say so when the skip is what
# kept the slug out of the FAIL below. A status of `referenced` or
# `no content extracted` is a real reason not to demand the slug, but
# it was applied in silence, so an entry that should have been in
# sources.md and a status line nobody had updated produced the same
# output: nothing. Only a MATERIAL skip is announced; when the slug is
# already in sources.md the check passes either way and there is no
# fail-open to disclose.
if rd_status != "extracted":
if rd_slug not in sources_slugs:
shown = f"`{rd_status}`" if rd_status else "absent"
emit_info(
f"Check 8 skipped for research-doc slug '{rd_slug}' — its Status is {shown}, not `extracted`",
f"{rd_rel} (## {rd_slug})",
f"'{rd_rel}' has '## {rd_slug}' with contributing files but Status {shown}, and this skill's "
f"sources.md has no '## {rd_slug}' entry. Check 8 only demands an entry for an `extracted` slug, "
f"so it did not run here. If that status is stale — the content was extracted and the line was never "
f"updated — this skill is missing a source entry; if it is accurate, nothing needs doing."
)
continue
# This slug should be in sources.md
if rd_slug not in sources_slugs: