Defame1297/holocron
0
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases 3 Wiki Activity

feat(kyberforge): make Research doc name one Research registry

Browse Source 
validate-provenance.sh assumed `Research doc:` names a research
sources.md whose H2 headings are the source slugs, but 29 corpus entries
named topic docs and 6 values were not a single path, so checks 7 and 8
reported INFO for 36 entries and nothing ever failed.

`Research doc:` now takes exactly one path. An entry with no registry
writes `none` plus one `- **Basis:** <path>` bullet per path; each Basis
path is existence-checked unless annotated `(removed in <sha>)`.

- Check 7 FAILs when a resolved registry lacks the slug, when the value
  is a topic doc, or when it is a list. An unresolvable path stays INFO.
- Check 8 is retired: one registry serves many skills, so requiring
  every registry slug in each skill's sources.md is unsatisfiable.
- The Research doc and Basis parsers accept the inline, bullet and
  header-plus-bullets spellings, so a differently spelled field is no
  longer read as absent.

Refs: #121
ADR: 0028
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EGHFJextYtVQseaHPDDhxB
This commit is contained in:
Defame1297
2026-09-21 17:28:08 +00:00
parent af008b6d37
commit 5a52949c57
6 changed files with 525 additions and 590 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
plugins/kyberforge/.apm/skills/factory-audit/references/skill-file-structure.md
View File

@@ -50,11 +50,15 @@ on-disk check. Flag any other spelling of a cross-skill reference.
Two directories are exempt, and the exemptions are structural rather than discretionary: Two directories are exempt, and the exemptions are structural rather than discretionary:
- **`references/sources.md`.** Its `Research doc:` fields are development-time provenance pointers, - **`references/sources.md`.** Its `Research doc:` and `Basis:` fields are development-time
not runtime references. They are expected to be unresolvable after install, so provenance pointers, not runtime references. A `Research doc:` path that does not resolve after
`validate-provenance.sh` does not treat an absent path as a FAIL — it emits an INFO naming the install is expected, so `validate-provenance.sh` does not treat an absent path as a FAIL — it
slug and stating that checks 7 and 8 did not run for it. Flagging them as broken references emits an INFO naming the slug and stating that check 7 did not run for it. Flagging them as
would make every correctly-provenanced skill fail. broken references would make every correctly-provenanced skill fail. Where the path DOES
resolve, it is checked: `Research doc:` names exactly one Research registry (a `sources.md`
whose H2 headings are the source slugs), and a slug missing from it, a topic document in its
place, or a list of paths is a FAIL. An entry with no registry writes `Research doc: none` plus
`Basis:` repo paths, which are existence-checked unless annotated `(removed in <sha>)`.
- **`tests/`.** Test files are dev-only and may reference repo-level infrastructure such as a shared - **`tests/`.** Test files are dev-only and may reference repo-level infrastructure such as a shared
`tests/test_helper/`. The exemption is conditional on the dependency being declared: if `tests/` `tests/test_helper/`. The exemption is conditional on the dependency being declared: if `tests/`
exists and `tests/README.md` is absent or does not document it, that is a FAIL. exists and `tests/README.md` is absent or does not document it, that is a FAIL.

13
plugins/kyberforge/.apm/skills/factory-audit/scripts/lib-provenance-agent.sh
View File

@@ -77,12 +77,13 @@ Checks performed:
4 Contributing files back-reference the parent slug in their source_keys 4 Contributing files back-reference the parent slug in their source_keys
5 Research doc field present and not placeholder 5 Research doc field present and not placeholder
Agent mode has no counterpart to skill mode's checks 6, 7 and 8 (Research Agent mode has no counterpart to skill mode's checks 6 and 7 (Research doc
doc field / upstream forward / upstream reverse are numbered 6, 7, 8 there and field / slug in the Research registry are numbered 6 and 7 there, and the field
5 here): an agent at plugin scope is a single file with a plugin-root check is 5 here): an agent at plugin scope is a single file with a plugin-root
sources.md, so there is no references/ tree to walk and no upstream research sources.md, so there is no references/ tree to walk and no Research registry to
source index to cross-check. parse_status() and the sources.md-basename gate cross-check. The sources.md-basename gate and the Basis: check that those checks
that those checks need exist only in lib-provenance-skill.sh. need exist only in lib-provenance-skill.sh. Skill mode's check 8 is retired
(ADR-0028).
EOF EOF
} }

359
plugins/kyberforge/.apm/skills/factory-audit/scripts/lib-provenance-skill.sh
View File

@@ -63,12 +63,25 @@ Checks performed:
read is reported as an INFO saying checks 4 and 5 did not run, never read is reported as an INFO saying checks 4 and 5 did not run, never
skipped silently. skipped silently.
5 Contributing files back-reference the parent slug in their source_keys 5 Contributing files back-reference the parent slug in their source_keys
6 Research doc field present and not placeholder 6 Research doc field present and not a placeholder, and exactly ONE path — the Research registry, a plugin's
7 Slug in sources.md present in upstream research doc (INFO only). A section research sources.md whose H2 headings are the source slugs. A brace
expansion, a comma-separated list, a semicolon-separated pair and a
repeated '- **Research doc:**' line are each a FAIL. An entry with no
registry writes 'Research doc: none' (a trailing annotation after an em
dash is fine) and names what it was drawn from in '- **Basis:**', one
repo path per bullet; a missing Basis, or a Basis path that does not
exist, is a FAIL. A Basis bullet annotated '(removed in <sha>)' skips
the existence check.
7 Slug in sources.md present in the Research registry (FAIL). A section
annotation ('§ ...', '→ ...', '(...)') is stripped before the path is annotation ('§ ...', '→ ...', '(...)') is stripped before the path is
resolved; a path that still does not resolve is reported as an INFO saying resolved. A path that does not resolve, or no repo root above the skill
checks 7 and 8 did not run, never skipped silently. directory, is reported as an INFO saying check 7 did not run, never
8 Extracted non-(none) slug in research doc present in sources.md skipped silently. A Research doc that resolves to a file NOT named
sources.md (a topic document) is a FAIL.
8 (retired — #121) The reverse check, "every extracted slug in the research
doc appears in this skill's sources.md", could not be satisfied when one
registry serves many skills. The number is left vacant so check 9 keeps
the name the rest of the repo cites.
9 Description or Contributing files text changed since --base-ref (INFO 9 Description or Contributing files text changed since --base-ref (INFO
only — a bash script cannot verify the claim is still TRUE, only that it only — a bash script cannot verify the claim is still TRUE, only that it
changed; the auditor reads the named files to check that). Wrapped values changed; the auditor reads the named files to check that). Wrapped values
@@ -79,11 +92,10 @@ Checks performed:
or references/sources.md is not tracked under this path at that ref, this or references/sources.md is not tracked under this path at that ref, this
is announced as ONE INFO for the whole check, never a silent skip. is announced as ONE INFO for the whole check, never a silent skip.
Checks 7 and 8 apply ONLY when the Research doc value names a research SOURCE Check 7 applies to a Research doc that names a Research registry — a file
INDEX — a file whose basename is sources.md, whose H2 headings ARE source whose basename is sources.md, whose H2 headings ARE source slugs. A topic
slugs. A Research doc pointing at a topic document is reported as an INFO document is a FAIL, not a value the check skips, and every other reason it
saying the two checks are not applicable, and every other reason they do not does not run is announced as an INFO.
run is announced the same way.
EOF EOF
} }
@@ -344,25 +356,73 @@ KYBERFORGE_PROV_SKILL_PREAMBLE_PY="${KYBERFORGE_PROV_SKILL_PREAMBLE_PY%$'\n'}"
IFS='' read -r -d '' KYBERFORGE_PROV_SKILL_BODY_PY <<'KYBERFORGE_PROV_SKILL_BODY' || true IFS='' read -r -d '' KYBERFORGE_PROV_SKILL_BODY_PY <<'KYBERFORGE_PROV_SKILL_BODY' || true
def parse_research_docs(content, slug): def _entry_block(content, slug):
"""Every Research doc value under a given slug H2, in document order. """The text under a '## slug' heading, or None when there is no such entry."""
The caller uses the first and reports the rest. Returning only the first —
what this did before — meant a second '- **Research doc:**' line in one
entry was silently ignored, so an author who added a doc rather than
replacing one got checks 7 and 8 run against the old path and no hint that
the new one was never looked at.
"""
pattern = re.compile( pattern = re.compile(
r'^## ' + re.escape(slug) + r'\s*\n(.*?)(?=^## |\Z)', r'^## ' + re.escape(slug) + r'\s*\n(.*?)(?=^## |\Z)',
re.MULTILINE | re.DOTALL re.MULTILINE | re.DOTALL
) )
m = pattern.search(content) m = pattern.search(content)
if not m: return m.group(1) if m else None
def parse_field_values(content, slug, label):
"""Every value of a '**label:**' field under a slug H2, in document order.
The SPELLING of a field must not decide whether it is read. Three
spellings are in the corpus and all three are accepted here:
- **Label:** value (the documented form)
**Label:** value (no leading hyphen — gitea-releases writes Status so)
**Label:** (a header, then '- value' bullets)
- value
A field parsed by a regex that knew only the first form returned "nothing
found" for the other two, and every caller read that as "nothing declared"
(#121, second comment; the same failure shape as #111 and #118). A header's
bullets stop at the first line that is neither blank nor a bullet, and a
'- **Other:**' bullet is the NEXT field, not a value of this one.
"""
block = _entry_block(content, slug)
if block is None:
return [] return []
block = m.group(1) values = []
return [v.strip() for v in lines = block.splitlines()
re.findall(r'^\- \*\*Research doc:\*\* (.+)$', block, re.MULTILINE)] label_re = re.compile(r'^(?:- )?\*\*' + re.escape(label) + r':\*\*[ \t]*(.*)$')
i = 0
while i < len(lines):
m = label_re.match(lines[i])
i += 1
if not m:
continue
inline = m.group(1).strip()
if inline:
values.append(inline)
continue
while i < len(lines):
line = lines[i].strip()
if not line:
i += 1
continue
if not line.startswith('- ') or line.startswith('- **'):
break
values.append(line[2:].strip())
i += 1
return values
def parse_research_docs(content, slug):
"""Every Research doc value under a given slug H2, in document order.
Research doc takes exactly ONE path, so the caller FAILs on a second value
rather than using the first and announcing the rest — an author who added a
doc rather than replacing one otherwise got check 7 run against the
old path and a verdict that looked complete.
"""
return parse_field_values(content, slug, 'Research doc')
def parse_basis(content, slug):
"""Every Basis value under a slug H2 — the repo paths an entry with no
Research registry was actually drawn from, one per bullet."""
return parse_field_values(content, slug, 'Basis')
# A Research doc value is a path, and very often a path PLUS an annotation # A Research doc value is a path, and very often a path PLUS an annotation
# naming the section the slug came from: # naming the section the slug came from:
@@ -371,7 +431,7 @@ def parse_research_docs(content, slug):
# plugins/git/docs/research/docs/git/remotes.md → `## Pushing (`git push`)` # plugins/git/docs/research/docs/git/remotes.md → `## Pushing (`git push`)`
# .../pre-commit/hooks-reference.md § "pre-commit-hooks (official collection)" # .../pre-commit/hooks-reference.md § "pre-commit-hooks (official collection)"
# #
# os.path.isfile() is false for every one of those strings, and checks 7 and 8 # os.path.isfile() is false for every one of those strings, and check 7
# used to skip SILENTLY whenever the path did not resolve. The effect was that # used to skip SILENTLY whenever the path did not resolve. The effect was that
# both checks were dead on eight of the nine git skills — git-history, the one # both checks were dead on eight of the nine git skills — git-history, the one
# skill writing a bare path, was the only place they ran, which is why it was # skill writing a bare path, was the only place they ran, which is why it was
@@ -394,57 +454,24 @@ def research_doc_is_none(value):
""" """
return re.match(r'\(?none\b', value.strip(), re.IGNORECASE) is not None return re.match(r'\(?none\b', value.strip(), re.IGNORECASE) is not None
# The Status value is what gates check 8, so every spelling this parser fails # A Research doc or Basis value names ONE path. The three list spellings seen
# to read is a check that does not run. Two were unreadable: # in the corpus — a brace expansion, a comma-separated list and a
# # semicolon-separated pair — are humans writing "several documents" into a
# - **Status:** `extracted` — partial fetch (a trailing note) # single-path field. Nothing expands a brace in a markdown field, and the
# **Status:** (the bullet form, the same # annotation strip above discards everything after the first '(' or section
# - `extracted` shape parse_contributing_files # marker, so a second path parked after one was NEVER resolved and no check
# already accepts) # said so. Detected on the raw value, with commas and semicolons INSIDE the
# # annotation left alone: those are prose ('cross-cutting; no dedicated
# Both used to parse to a string that compared unequal to "`extracted`", and # section'), and only a second path-shaped token after a ';' is a list.
# check 8 skipped on that inequality without a word. Returning the BACKTICKED SECOND_PATH_AFTER_SEMICOLON_RE = re.compile(r';\s*[\w.\-]+/[\w./\-]*\.[A-Za-z]+')
# TOKEN — not the whole line — is what makes the trailing note harmless, and it
# lets the caller name the actual status when it announces a skip.
STATUS_TOKEN_RE = re.compile(r'^`([^`]*)`')
BASIS_REMOVED_RE = re.compile(r'\(removed in [0-9a-fA-F]{7,40}\b[^)]*\)')
def parse_status(content, slug): def names_more_than_one_path(value):
"""Find the Status value for a given slug H2 in content. path_part = strip_research_doc_annotation(value)
if '{' in path_part or '}' in path_part or ',' in path_part or ';' in path_part:
Returns the status with its backticks stripped ('extracted', 'referenced', return True
'no content extracted'), or None when the entry has no Status line. return SECOND_PATH_AFTER_SEMICOLON_RE.search(value) is not None
"""
pattern = re.compile(
r'^## ' + re.escape(slug) + r'\s*\n(.*?)(?=^## |\Z)',
re.MULTILINE | re.DOTALL
)
m = pattern.search(content)
if not m:
return None
block = m.group(1)
raw = None
st_m = re.search(r'^\- \*\*Status:\*\* (.+)$', block, re.MULTILINE)
if st_m:
raw = st_m.group(1).strip()
else:
st_m = re.search(r'^\*\*Status:\*\*\s*$', block, re.MULTILINE)
if not st_m:
return None
for line in block[st_m.end():].splitlines():
line = line.strip()
if not line:
continue
if not line.startswith("- "):
break
raw = line[2:].strip()
break
if raw is None:
return None
token = STATUS_TOKEN_RE.match(raw)
return token.group(1).strip() if token else raw
def find_repo_root(start_dir): def find_repo_root(start_dir):
"""Walk up from start_dir until we find a directory containing .git.""" """Walk up from start_dir until we find a directory containing .git."""
@@ -460,7 +487,7 @@ def find_repo_root(start_dir):
# --- Check 9 helpers --------------------------------------------------- # --- Check 9 helpers ---------------------------------------------------
# Check 9 needs a raw field VALUE (as text, to diff against an earlier # Check 9 needs a raw field VALUE (as text, to diff against an earlier
# version), not the parsed structure parse_contributing_files() and # version), not the parsed structure parse_contributing_files() and
# parse_status() return. The ONE normalization applied is whitespace # parse_field_values() return. The ONE normalization applied is whitespace
# collapsing, which is what makes a re-wrap or a re-indent invisible; nothing # collapsing, which is what makes a re-wrap or a re-indent invisible; nothing
# else is normalized away. # else is normalized away.
# #
@@ -532,7 +559,7 @@ def parse_field_raw(content, slug, field_name):
"""Raw text of a '**<field_name>:**' field under a slug H2, wrapping joined. """Raw text of a '**<field_name>:**' field under a slug H2, wrapping joined.
Mirrors the two authored shapes parse_contributing_files() and Mirrors the two authored shapes parse_contributing_files() and
parse_status() already handle (inline value on the same line, or a parse_field_values() already handle (inline value on the same line, or a
bare heading followed by '- ' bullets), but returns text rather than a bare heading followed by '- ' bullets), but returns text rather than a
parsed structure, because check 9 diffs wording, not semantics. parsed structure, because check 9 diffs wording, not semantics.
@@ -788,11 +815,8 @@ if os.path.isdir(refs_dir):
repo_root = find_repo_root(skill_dir) repo_root = find_repo_root(skill_dir)
# Collect all research doc paths we'll check (for Check 8)
research_docs_seen = {} # abs_path → (rel_path, slugs referencing it, content)
# Every per-slug parser below — parse_contributing_files, parse_research_docs, # Every per-slug parser below — parse_contributing_files, parse_research_docs,
# parse_status — locates its block with pattern.search(), so a slug written # parse_basis — locates its block with pattern.search(), so a slug written
# twice resolves to the FIRST block every time. Iterating the raw heading list # twice resolves to the FIRST block every time. Iterating the raw heading list
# therefore checked the first block's fields twice and the second block's # therefore checked the first block's fields twice and the second block's
# never: a duplicated slug is half-validated, and looked fully validated. The # never: a duplicated slug is half-validated, and looked fully validated. The
@@ -810,7 +834,7 @@ for _slug in all_slugs:
f"references/sources.md (## {_slug})", f"references/sources.md (## {_slug})",
f"'## {_slug}' appears {_count} times. Every field parser here takes the first match, so the " f"'## {_slug}' appears {_count} times. Every field parser here takes the first match, so the "
f"second and later blocks' Contributing files, Research doc and Status are never validated — " f"second and later blocks' Contributing files, Research doc and Status are never validated — "
f"checks 4, 5, 6, 7 and 8 did not run for them. " f"checks 4, 5, 6 and 7 did not run for them. "
f"Merge the blocks into one entry, or give each a distinct slug and reference it from source_keys." f"Merge the blocks into one entry, or give each a distinct slug and reference it from source_keys."
) )
@@ -866,13 +890,13 @@ for slug in unique_slugs:
# Check 6: Research doc field required # Check 6: Research doc field required
rd_values = parse_research_docs(sources_content, slug) rd_values = parse_research_docs(sources_content, slug)
if len(rd_values) > 1: if len(rd_values) > 1:
emit_info( emit_fail(
f"Multiple '- **Research doc:**' lines for '{slug}' — only the first is used", f"Multiple '- **Research doc:**' lines for '{slug}' — Research doc takes exactly one path",
f"references/sources.md (## {slug})", f"references/sources.md (## {slug})",
f"The '## {slug}' entry has {len(rd_values)} Research doc lines; checks 7 and 8 ran against the first " f"The '## {slug}' entry has {len(rd_values)} Research doc lines. Research doc names one Research registry, "
f"('{rd_values[0]}') and never looked at the rest. " f"so a second line is a list, and a list is not a grammar this field has.",
f"Keep one Research doc line per entry — if a slug genuinely came from two documents, split it into two slugs, " f"Keep one Research doc line, pointing at the plugin's research sources.md. If the entry has no registry, "
f"or name the extra document inside the first value's annotation where it is at least visible." f"write '- **Research doc:** none' and name what it was drawn from in '- **Basis:**', one repo path per bullet."
) )
rd_value = rd_values[0] if rd_values else None rd_value = rd_values[0] if rd_values else None
if rd_value is None: if rd_value is None:
@@ -889,7 +913,67 @@ for slug in unique_slugs:
f"The '## {slug}' entry has an unfilled Research doc value.", f"The '## {slug}' entry has an unfilled Research doc value.",
f"Set '- **Research doc:**' to a real path relative to repo root, or '(none)' if not applicable." f"Set '- **Research doc:**' to a real path relative to repo root, or '(none)' if not applicable."
) )
elif not research_doc_is_none(rd_value): elif research_doc_is_none(rd_value):
# An entry with no Research registry must still say what it WAS drawn
# from. Basis names repo paths, one per bullet, and each is checked to
# exist — the honest way to record an org convention, an ADR or a
# house-verified reproduction, none of which has a registry entry.
basis_values = parse_basis(sources_content, slug)
if not basis_values:
emit_fail(
f"Basis missing for '{slug}' — Research doc is 'none'",
f"references/sources.md (## {slug})",
f"The '## {slug}' entry declares no Research registry ('{rd_value}') and no '- **Basis:**' line, "
f"so nothing records what the entry was drawn from.",
f"Add '- **Basis:** <repo path>' to the '## {slug}' entry, one line per path, naming the ADR, "
f"convention file or reproduction the entry rests on."
)
for basis in basis_values:
basis_path = strip_research_doc_annotation(basis)
if PLACEHOLDER_RE.search(basis) or not basis_path:
emit_fail(
f"Basis is empty or placeholder for '{slug}'",
f"references/sources.md (## {slug})",
f"The '## {slug}' entry has an unfilled Basis value '{basis}'.",
f"Set '- **Basis:**' to one repo path."
)
elif names_more_than_one_path(basis):
emit_fail(
f"Basis value names more than one path for '{slug}'",
f"references/sources.md (## {slug})",
f"The Basis value '{basis}' is a brace expansion or a comma- or semicolon-separated list.",
f"Write one '- **Basis:** <repo path>' line per path."
)
elif not repo_root:
emit_info(
f"Basis check skipped for '{slug}' — no repo root above the skill directory",
f"references/sources.md (## {slug})",
f"'{basis}' is a path relative to the repo root, but no ancestor of the skill directory contains a .git entry, "
f"so it cannot be resolved. Run this script against a skill inside a checkout."
)
elif BASIS_REMOVED_RE.search(basis):
# A path the entry HISTORICALLY rested on, annotated
# '(removed in <sha>)', is a declaration that it is gone on
# purpose. The sha is not resolved: the annotation is the
# author saying "deleted, and here is where to look".
continue
elif not os.path.exists(os.path.join(repo_root, basis_path)):
emit_fail(
f"Basis path '{basis_path}' does not exist",
f"references/sources.md (## {slug})",
f"'{basis}' resolves to '{basis_path}' relative to the repo root and nothing is there.",
f"Correct the path, or remove the Basis line if the entry no longer rests on it."
)
elif names_more_than_one_path(rd_value):
emit_fail(
f"Research doc names more than one path for '{slug}'",
f"references/sources.md (## {slug})",
f"The Research doc value '{rd_value}' is a brace expansion or a comma- or semicolon-separated list. "
f"Research doc names exactly one Research registry.",
f"Point Research doc at the plugin's research sources.md. If the entry has no registry, write "
f"'- **Research doc:** none' and name what it was drawn from in '- **Basis:**', one repo path per bullet."
)
else:
# Check 7: Upstream forward — slug should appear in research doc. # Check 7: Upstream forward — slug should appear in research doc.
# Every path out of here that does NOT run the check says so out loud. # Every path out of here that does NOT run the check says so out loud.
rd_path = strip_research_doc_annotation(rd_value) rd_path = strip_research_doc_annotation(rd_value)
@@ -898,7 +982,7 @@ for slug in unique_slugs:
f"Upstream checks skipped for '{slug}' — no repo root above the skill directory", f"Upstream checks skipped for '{slug}' — no repo root above the skill directory",
f"references/sources.md (## {slug})", f"references/sources.md (## {slug})",
f"'{rd_value}' is a path relative to the repo root, but no ancestor of the skill directory contains a .git entry, " f"'{rd_value}' is a path relative to the repo root, but no ancestor of the skill directory contains a .git entry, "
f"so it cannot be resolved. Checks 7 and 8 did not run for this slug. " f"so it cannot be resolved. Check 7 did not run for this slug. "
f"Run this script against a skill inside a checkout." f"Run this script against a skill inside a checkout."
) )
elif not rd_path: elif not rd_path:
@@ -906,8 +990,8 @@ for slug in unique_slugs:
f"Upstream checks skipped for '{slug}' — Research doc value names no path", f"Upstream checks skipped for '{slug}' — Research doc value names no path",
f"references/sources.md (## {slug})", f"references/sources.md (## {slug})",
f"The Research doc value '{rd_value}' is entirely annotation — stripping the section marker leaves no path. " f"The Research doc value '{rd_value}' is entirely annotation — stripping the section marker leaves no path. "
f"Checks 7 and 8 did not run for this slug. " f"Check 7 did not run for this slug. "
f"Give the value a file path relative to the repo root, or record '(none)' if no research doc backs this entry." f"Give the value a file path relative to the repo root, or record 'none' plus a '- **Basis:**' if no registry backs this entry."
) )
else: else:
rd_abs = os.path.join(repo_root, rd_path) rd_abs = os.path.join(repo_root, rd_path)
@@ -916,33 +1000,26 @@ for slug in unique_slugs:
f"Upstream checks skipped for '{slug}' — research doc '{rd_path}' does not exist", f"Upstream checks skipped for '{slug}' — research doc '{rd_path}' does not exist",
f"references/sources.md (## {slug})", f"references/sources.md (## {slug})",
f"'{rd_value}' resolves to '{rd_path}' relative to the repo root and no file is there. " f"'{rd_value}' resolves to '{rd_path}' relative to the repo root and no file is there. "
f"Checks 7 and 8 did not run for this slug, so nothing verified that the research doc still backs it. " f"Check 7 did not run for this slug, so nothing verified that the research doc still backs it. "
f"Point the value at one existing file — a brace expansion, a comma-separated list of paths, or a bare section title does not resolve — " f"Point the value at the one existing Research registry (the plugin's research sources.md), "
f"or record '(none)' if no research doc backs this entry." f"or record 'none' plus a '- **Basis:**' if no registry backs this entry."
) )
elif os.path.basename(rd_path) != "sources.md": elif os.path.basename(rd_path) != "sources.md":
# Checks 7 and 8 both assume the Research doc is a research # Check 7 matches slugs against the H2 headings of a
# SOURCE INDEX — a sources.md whose H2 headings ARE source # Research registry — a sources.md whose H2s ARE source slugs.
# slugs. 30 of the 121 corpus entries point instead at a TOPIC # A topic document (remotes.md, gitflow.md) has section headings
# DOCUMENT (remotes.md, gitflow.md, api-reference.md), whose # for H2s, so no slug can ever match one. Research doc names the
# H2s are headings like '## Core Philosophy'. A slug can never # registry (#121), so a topic document there is the wrong file,
# match one, so check 7 reported all 30 as "slug not found" — # not a value these checks cannot verify. A pointer to the topic
# every one a false positive — and check 8, aimed at documents # document that digested the source belongs in the free-text
# that carry no '- **Status:**' line at all, was saved from a # annotation after the path, where it is not checked.
# matching flood of false FAILs only by an UNANNOUNCED skip on emit_fail(
# that missing status. The premise, not the corpus, was wrong. f"Research doc '{rd_path}' for '{slug}' is a topic document, not a Research registry",
#
# A topic-document reference is a legitimate, useful value; it
# just is not something these two checks can verify. Say that
# once, out loud, instead of failing 30 entries for it.
emit_info(
f"Upstream checks not applicable for '{slug}' — research doc '{rd_path}' is a topic document, not a source index",
f"references/sources.md (## {slug})", f"references/sources.md (## {slug})",
f"Checks 7 and 8 match slugs against the H2 headings of a research source index — a file named 'sources.md', " f"'{os.path.basename(rd_path)}' is not a sources.md, so its H2s are section headings and no slug can match one. "
f"where each H2 IS a source slug. '{os.path.basename(rd_path)}' is a topic document, so its H2s are section " f"Research doc names the plugin's Research registry — the sources.md whose H2s are source slugs.",
f"headings and no slug will ever match one. Checks 7 and 8 did not run for this slug. " f"Repoint '{slug}' at the sibling sources.md in '{os.path.dirname(rd_path)}/', and keep the topic document in the "
f"This needs no fix: point the value at the research corpus's own sources.md only if you want the " f"annotation, e.g. '<registry path> (digested in {os.path.basename(rd_path)})'."
f"provenance link machine-verified."
) )
else: else:
try: try:
@@ -951,63 +1028,19 @@ for slug in unique_slugs:
emit_info( emit_info(
f"Upstream checks skipped for '{slug}' — research doc '{rd_path}' is {exc}", f"Upstream checks skipped for '{slug}' — research doc '{rd_path}' is {exc}",
f"references/sources.md (## {slug})", f"references/sources.md (## {slug})",
f"'{rd_path}' could not be decoded, so checks 7 and 8 did not run for this slug. " f"'{rd_path}' could not be decoded, so check 7 did not run for this slug. "
f"Re-save the research doc as UTF-8." f"Re-save the research doc as UTF-8."
) )
continue continue
rd_slugs = set(parse_h2_slugs(rd_content)) rd_slugs = set(parse_h2_slugs(rd_content))
if slug not in rd_slugs: if slug not in rd_slugs:
emit_info( emit_fail(
f"Slug '{slug}' not found as H2 in research doc '{rd_path}'", f"Slug '{slug}' not found as H2 in research doc '{rd_path}'",
f"references/sources.md (## {slug})", f"references/sources.md (## {slug})",
f"The research doc '{rd_path}' does not have a '## {slug}' heading. " f"The Research registry '{rd_path}' does not have a '## {slug}' heading, so the entry's provenance "
f"The provenance link may be imprecise — the slug name in sources.md may differ from the research doc's heading." f"link resolves to nothing.",
f"Rename the slug to match a '## ' heading in '{rd_path}', or repoint Research doc at the registry that has it."
) )
# Track for Check 8. The content is carried with the entry so
# check 8 reuses this read rather than decoding the file a
# second time, with a second chance to fail differently.
if rd_abs not in research_docs_seen:
research_docs_seen[rd_abs] = (rd_path, set(), rd_content)
research_docs_seen[rd_abs][1].add(slug)
# --- Check 8: Upstream reverse ---
for rd_abs, (rd_rel, known_slugs, rd_content) in research_docs_seen.items():
for rd_slug in parse_h2_slugs(rd_content):
# Parse this slug's Contributing files and Status in the research doc
rd_cf = parse_contributing_files(rd_content, rd_slug)
rd_status = parse_status(rd_content, rd_slug)
# Skip if the research doc explicitly records no contributing files
if rd_cf == []:
continue
# Skip if status is not `extracted` — and say so when the skip is what
# kept the slug out of the FAIL below. A status of `referenced` or
# `no content extracted` is a real reason not to demand the slug, but
# it was applied in silence, so an entry that should have been in
# sources.md and a status line nobody had updated produced the same
# output: nothing. Only a MATERIAL skip is announced; when the slug is
# already in sources.md the check passes either way and there is no
# fail-open to disclose.
if rd_status != "extracted":
if rd_slug not in sources_slugs:
shown = f"`{rd_status}`" if rd_status else "absent"
emit_info(
f"Check 8 skipped for research-doc slug '{rd_slug}' — its Status is {shown}, not `extracted`",
f"{rd_rel} (## {rd_slug})",
f"'{rd_rel}' has '## {rd_slug}' with contributing files but Status {shown}, and this skill's "
f"sources.md has no '## {rd_slug}' entry. Check 8 only demands an entry for an `extracted` slug, "
f"so it did not run here. If that status is stale — the content was extracted and the line was never "
f"updated — this skill is missing a source entry; if it is accurate, nothing needs doing."
)
continue
# This slug should be in sources.md
if rd_slug not in sources_slugs:
emit_fail(
f"Research doc slug '{rd_slug}' missing from skill sources.md",
f"references/sources.md",
f"The research doc '{rd_rel}' has '## {rd_slug}' with status `extracted` and contributing files, "
f"but this skill's sources.md has no '## {rd_slug}' entry.",
f"Add '## {rd_slug}' to references/sources.md or mark it as '(none)' in the research doc's Contributing files."
)
# --- Check 9: Description / Contributing files changed since --base-ref --- # --- Check 9: Description / Contributing files changed since --base-ref ---
# A structural fact — the field's TEXT differs from an earlier revision — is # A structural fact — the field's TEXT differs from an earlier revision — is

704
plugins/kyberforge/.apm/skills/factory-audit/tests/validate-provenance-skill.bats
View File

@@ -53,6 +53,10 @@ EOF
local slug="${2:-my-source}" local slug="${2:-my-source}"
local contrib="${3:-SKILL.md}" local contrib="${3:-SKILL.md}"
local research="${4:-(none)}" local research="${4:-(none)}"
# A 'none' Research doc must name its Basis (#121); SKILL.md exists in
# every fixture skill and, once commit_as_base has run, is a repo path.
local basis_line=""
if [[ "$research" == *none* ]]; then basis_line="- **Basis:** SKILL.md"; fi
mkdir -p "$dir/references" mkdir -p "$dir/references"
cat > "$dir/references/sources.md" <<EOF cat > "$dir/references/sources.md" <<EOF
# Sources # Sources
@@ -63,6 +67,7 @@ EOF
- **Description:** A test source. - **Description:** A test source.
- **Contributing files:** ${contrib} - **Contributing files:** ${contrib}
- **Research doc:** ${research} - **Research doc:** ${research}
${basis_line}
- **Status:** \`extracted\` - **Status:** \`extracted\`
EOF EOF
} }
@@ -82,6 +87,7 @@ EOF
no forge-specific content drawn directly from it beyond that. no forge-specific content drawn directly from it beyond that.
- **Contributing files:** SKILL.md - **Contributing files:** SKILL.md
- **Research doc:** (none) - **Research doc:** (none)
- **Basis:** SKILL.md
- **Status:** `extracted` - **Status:** `extracted`
EOF EOF
} }
@@ -106,7 +112,7 @@ EOF
# Helper: create a fake repo (a real git repo, one commit, makes # Helper: create a fake repo (a real git repo, one commit, makes
# find_repo_root stop there) holding one skill whose single sources.md # find_repo_root stop there) holding one skill whose single sources.md
# slug points at the given Research doc value. Checks 7 and 8 only run # slug points at the given Research doc value. Check 7 only runs
# for a skill inside a checkout, so every upstream case needs this shape; # for a skill inside a checkout, so every upstream case needs this shape;
# the research doc itself is written per test into # the research doc itself is written per test into
# "$repo/docs/research/sources.md" — which check 9 does not examine, so # "$repo/docs/research/sources.md" — which check 9 does not examine, so
@@ -114,6 +120,8 @@ EOF
make_upstream_skill() { make_upstream_skill() {
local repo="$1" local repo="$1"
local research="${2:-docs/research/sources.md}" local research="${2:-docs/research/sources.md}"
local basis_line=""
if [[ "$research" == *none* ]]; then basis_line="- **Basis:** my-skill/SKILL.md"; fi
local skill="$repo/my-skill" local skill="$repo/my-skill"
mkdir -p "$skill/references" "$repo/docs/research" mkdir -p "$skill/references" "$repo/docs/research"
cat > "$skill/SKILL.md" <<EOF cat > "$skill/SKILL.md" <<EOF
@@ -138,6 +146,7 @@ EOF
- **Description:** A test source. - **Description:** A test source.
- **Contributing files:** SKILL.md - **Contributing files:** SKILL.md
- **Research doc:** ${research} - **Research doc:** ${research}
${basis_line}
- **Status:** \`extracted\` - **Status:** \`extracted\`
EOF EOF
commit_as_base "$repo" commit_as_base "$repo"
@@ -199,6 +208,7 @@ teardown() {
- **Description:** A test source. - **Description:** A test source.
- **Contributing files:** SKILL.md - **Contributing files:** SKILL.md
- **Research doc:** (none) - **Research doc:** (none)
- **Basis:** SKILL.md
- **Status:** \`extracted\` - **Status:** \`extracted\`
EOF EOF
run bash "$SCRIPT" "$skill" run bash "$SCRIPT" "$skill"
@@ -232,6 +242,7 @@ EOF
- **Description:** A different source. - **Description:** A different source.
- **Contributing files:** SKILL.md - **Contributing files:** SKILL.md
- **Research doc:** (none) - **Research doc:** (none)
- **Basis:** SKILL.md
- **Status:** \`extracted\` - **Status:** \`extracted\`
EOF EOF
run bash "$SCRIPT" "$skill" run bash "$SCRIPT" "$skill"
@@ -335,6 +346,7 @@ EOF
- **Description:** A test source. - **Description:** A test source.
- **Contributing files:** SKILL.md - **Contributing files:** SKILL.md
- **Research doc:** (none) - **Research doc:** (none)
- **Basis:** SKILL.md
- **Status:** \`extracted\` - **Status:** \`extracted\`
EOF EOF
# Now add my-source that references SKILL.md but SKILL.md doesn't back-reference it # Now add my-source that references SKILL.md but SKILL.md doesn't back-reference it
@@ -346,6 +358,7 @@ EOF
- **Description:** Another source. - **Description:** Another source.
- **Contributing files:** SKILL.md - **Contributing files:** SKILL.md
- **Research doc:** (none) - **Research doc:** (none)
- **Basis:** SKILL.md
- **Status:** \`extracted\` - **Status:** \`extracted\`
EOF EOF
run bash "$SCRIPT" "$skill" run bash "$SCRIPT" "$skill"
@@ -407,7 +420,7 @@ EOF
# Cycle 11 — Check 7: Upstream forward: slug in sources.md not in research doc → INFO # Cycle 11 — Check 7: Upstream forward: slug in sources.md not in research doc → INFO
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@test "INFO: slug in sources.md not found in research doc → INFO, exits 0" { @test "FAIL: slug in sources.md not found in the research registry → FAIL (was INFO before #121)" {
local skill="$TMPDIR/my-skill" local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill" make_skill_with_source_keys "$skill"
@@ -471,262 +484,9 @@ EOF
EOF EOF
run bash "$SCRIPT" "$skill2" run bash "$SCRIPT" "$skill2"
assert_success
assert_output --partial "INFO"
}
# ---------------------------------------------------------------------------
# Cycle 12 — Check 8: Upstream reverse: extracted slug in research doc not in sources.md → FAIL
# ---------------------------------------------------------------------------
@test "FAIL: extracted non-(none) slug in research doc missing from skill sources.md" {
local fake_repo="$TMPDIR/fakerepo"
mkdir -p "$fake_repo"
touch "$fake_repo/.git"
local skill="$fake_repo/my-skill"
mkdir -p "$skill"
cat > "$skill/SKILL.md" <<EOF
---
name: my-skill
description: A valid skill description.
metadata:
source_keys:
- my-source
---
## Step 1
Do the thing.
EOF
mkdir -p "$skill/references"
mkdir -p "$fake_repo/docs/research"
# Research doc has my-source (extracted, with a contributing file) AND extra-source (also extracted)
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
- **Contributing files:** some-skill/references/extra.md
- **Status:** \`extracted\`
EOF
cat > "$skill/references/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source.
- **Contributing files:** SKILL.md
- **Research doc:** docs/research/sources.md
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$skill"
assert_failure assert_failure
assert_output --partial "FAIL" assert_output --partial "FAIL"
} assert_output --partial "Slug 'my-source' not found as H2 in research doc 'docs/research/sources.md'"
@test "pass: extracted slug in research doc with (none) contributing files is not required in sources.md" {
local fake_repo="$TMPDIR/fakerepo"
mkdir -p "$fake_repo"
touch "$fake_repo/.git"
local skill="$fake_repo/my-skill"
mkdir -p "$skill"
cat > "$skill/SKILL.md" <<EOF
---
name: my-skill
description: A valid skill description.
metadata:
source_keys:
- my-source
---
## Step 1
Do the thing.
EOF
mkdir -p "$skill/references"
mkdir -p "$fake_repo/docs/research"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
- **Contributing files:** (none — not relevant)
- **Status:** \`extracted\`
EOF
cat > "$skill/references/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source.
- **Contributing files:** SKILL.md
- **Research doc:** docs/research/sources.md
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$skill"
assert_success
}
# ---------------------------------------------------------------------------
# Cycle 13 — parse_contributing_files: None ("could not parse") is NOT []
# ("explicitly (none)")
#
# Check 8 reads [] as "the research doc deliberately records no contributing
# files" and SKIPS the slug on that basis. A block the parser cannot read must
# therefore return None, so the slug stays exposed to check 8. Each case below
# asserts that CONSEQUENCE — check 8 firing on the unreadable entry — not the
# parser's return value, because returning [] is exactly the shape that makes
# the check silently do nothing while still exiting 0.
# ---------------------------------------------------------------------------
@test "check 8 runs: bullet form with '*' asterisk bullets is unparsable, not '(none)'" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
**Contributing files:**
* some-skill/references/extra.md
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_failure
assert_output --partial "FAIL"
assert_output --partial "extra-source"
}
@test "check 8 runs: bullet form with a numbered list is unparsable, not '(none)'" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
**Contributing files:**
1. some-skill/references/extra.md
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_failure
assert_output --partial "FAIL"
assert_output --partial "extra-source"
}
@test "check 8 runs: bullet form followed by prose is unparsable, not '(none)'" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
**Contributing files:**
See the table below for the file list.
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_failure
assert_output --partial "FAIL"
assert_output --partial "extra-source"
}
@test "check 8 runs: bullet form heading with a blank line and nothing after is unparsable, not '(none)'" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
- **Status:** \`extracted\`
**Contributing files:**
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_failure
assert_output --partial "FAIL"
assert_output --partial "extra-source"
}
@test "check 8 runs: inline form whose whole value is a parenthetical is unparsable, not '(none)'" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
- **Contributing files:** (see notes below)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_failure
assert_output --partial "FAIL"
assert_output --partial "extra-source"
} }
@test "check 4 runs: bullet form with '-' hyphen bullets still parses each path" { @test "check 4 runs: bullet form with '-' hyphen bullets still parses each path" {
@@ -746,6 +506,7 @@ EOF
- references/nonexistent.md (why this one matters) - references/nonexistent.md (why this one matters)
- **Research doc:** (none) - **Research doc:** (none)
- **Basis:** SKILL.md
- **Status:** \`extracted\` - **Status:** \`extracted\`
EOF EOF
run bash "$SCRIPT" "$skill" run bash "$SCRIPT" "$skill"
@@ -762,52 +523,6 @@ EOF
assert_output --partial "Contributing file 'references/nonexistent.md' does not exist" assert_output --partial "Contributing file 'references/nonexistent.md' does not exist"
} }
@test "check 8 skips: bullet form '- (none)' is an explicit declaration" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
**Contributing files:**
- (none — nothing was extracted from this section)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success
}
@test "check 8 skips: inline bare '(none)' is an explicit declaration" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
- **Contributing files:** (none)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success
}
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Cycle 14 — Check 3 (#111): an explicit 'source_keys: []' is a house-authored # Cycle 14 — Check 3 (#111): an explicit 'source_keys: []' is a house-authored
# declaration, a bare 'source_keys:' is truncation # declaration, a bare 'source_keys:' is truncation
@@ -894,7 +609,7 @@ EOF
} }
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Cycle 15 — Checks 7 and 8: Research doc annotation stripping, and the INFO # Cycle 15 — Check 7: Research doc annotation stripping, and the INFO
# that replaced the silent skip # that replaced the silent skip
# #
# A Research doc value is very often a path PLUS a section annotation, and # A Research doc value is very often a path PLUS a section annotation, and
@@ -917,7 +632,7 @@ EOF
EOF EOF
run bash "$SCRIPT" "$fake_repo/my-skill" run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success assert_failure
assert_output --partial "Slug 'my-source' not found as H2 in research doc 'docs/research/sources.md'" assert_output --partial "Slug 'my-source' not found as H2 in research doc 'docs/research/sources.md'"
refute_output --partial "§" refute_output --partial "§"
} }
@@ -936,7 +651,7 @@ EOF
EOF EOF
run bash "$SCRIPT" "$fake_repo/my-skill" run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success assert_failure
assert_output --partial "Slug 'my-source' not found as H2 in research doc 'docs/research/sources.md'" assert_output --partial "Slug 'my-source' not found as H2 in research doc 'docs/research/sources.md'"
refute_output --partial "→" refute_output --partial "→"
} }
@@ -955,7 +670,7 @@ EOF
EOF EOF
run bash "$SCRIPT" "$fake_repo/my-skill" run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success assert_failure
assert_output --partial "Slug 'my-source' not found as H2 in research doc 'docs/research/sources.md'" assert_output --partial "Slug 'my-source' not found as H2 in research doc 'docs/research/sources.md'"
refute_output --partial "whole-document reference" refute_output --partial "whole-document reference"
} }
@@ -978,7 +693,7 @@ EOF
assert_output "" assert_output ""
} }
@test "checks 7 and 8 skipped silently: Research doc '(none)' is recognised before the strip" { @test "check 7 skipped silently: Research doc '(none)' is recognised before the strip" {
local fake_repo="$TMPDIR/fakerepo" local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo" "(none)" make_upstream_skill "$fake_repo" "(none)"
@@ -987,7 +702,7 @@ EOF
assert_output "" assert_output ""
} }
@test "checks 7 and 8 skipped silently: bare 'none — reason' is recognised as a declaration" { @test "check 7 skipped silently: bare 'none — reason' is recognised as a declaration" {
local fake_repo="$TMPDIR/fakerepo" local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo" "none — org convention, no upstream research doc" make_upstream_skill "$fake_repo" "none — org convention, no upstream research doc"
@@ -1031,7 +746,7 @@ EOF
# Cycle 16 — Checks 4 and 5: None ("could not parse") is NOT [] ("explicitly # Cycle 16 — Checks 4 and 5: None ("could not parse") is NOT [] ("explicitly
# (none)"), on the sources.md side this time # (none)"), on the sources.md side this time
# #
# Cycle 13 pinned the distinction for check 8, which reads the parser's output # Cycle 13 pinned the distinction for check 8 (since retired), which read the parser's output
# against a RESEARCH doc. Checks 4 and 5 read it against the skill's own # against a RESEARCH doc. Checks 4 and 5 read it against the skill's own
# sources.md and honoured neither half: a truthiness test collapsed None into # sources.md and honoured neither half: a truthiness test collapsed None into
# [], so an unreadable Contributing files block disabled both checks and the # [], so an unreadable Contributing files block disabled both checks and the
@@ -1053,6 +768,7 @@ EOF
- **URL:** https://example.com/my-source - **URL:** https://example.com/my-source
- **Description:** A test source. - **Description:** A test source.
- **Research doc:** (none) - **Research doc:** (none)
- **Basis:** SKILL.md
**Contributing files:** **Contributing files:**
* references/ghost.md (asterisk bullets are not the bullet form) * references/ghost.md (asterisk bullets are not the bullet form)
- **Status:** \`extracted\` - **Status:** \`extracted\`
@@ -1075,6 +791,7 @@ EOF
- **URL:** https://example.com/my-source - **URL:** https://example.com/my-source
- **Description:** A test source. - **Description:** A test source.
- **Research doc:** (none) - **Research doc:** (none)
- **Basis:** SKILL.md
- **Status:** \`extracted\` - **Status:** \`extracted\`
EOF EOF
run bash "$SCRIPT" "$skill" run bash "$SCRIPT" "$skill"
@@ -1158,11 +875,11 @@ EOF
} }
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Cycle 20 — F: checks 7 and 8 apply only to a research SOURCE INDEX, and # Cycle 20 — F: check 7 applies only to a Research registry, and
# every skip announces itself # every skip announces itself
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@test "F: a topic-doc Research doc is reported as not applicable, not as a missing slug" { @test "F: a topic-doc Research doc is a FAIL naming the registry, not a missing slug (#121)" {
local fake_repo="$TMPDIR/fakerepo" local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo" "docs/research/remotes.md" make_upstream_skill "$fake_repo" "docs/research/remotes.md"
cat > "$fake_repo/docs/research/remotes.md" <<EOF cat > "$fake_repo/docs/research/remotes.md" <<EOF
@@ -1174,9 +891,8 @@ Prose about remotes.
EOF EOF
run bash "$SCRIPT" "$fake_repo/my-skill" run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success assert_failure
assert_output --partial "Upstream checks not applicable for 'my-source'" assert_output --partial "is a topic document, not a Research registry"
assert_output --partial "is a topic document, not a source index"
refute_output --partial "not found as H2 in research doc" refute_output --partial "not found as H2 in research doc"
} }
@@ -1193,102 +909,11 @@ EOF
EOF EOF
run bash "$SCRIPT" "$fake_repo/my-skill" run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success assert_failure
assert_output --partial "Slug 'my-source' not found as H2 in research doc 'docs/research/sources.md'" assert_output --partial "Slug 'my-source' not found as H2 in research doc 'docs/research/sources.md'"
refute_output --partial "not applicable" refute_output --partial "not applicable"
} }
@test "F: check 8 announces the slug it skipped for a non-extracted Status" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Sources
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
- **Contributing files:** some-skill/references/extra.md
- **Status:** \`referenced\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success
assert_output --partial "Check 8 skipped for research-doc slug 'extra-source'"
assert_output --partial "its Status is \`referenced\`, not \`extracted\`"
}
@test "F: a Status with a trailing note after the backticked value still reads as extracted" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Sources
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
- **Contributing files:** some-skill/references/extra.md
- **Status:** \`extracted\` — partial fetch, section 3 only
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_failure
assert_output --partial "Research doc slug 'extra-source' missing from skill sources.md"
}
@test "F: a bullet-form Status still reads as extracted" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Sources
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
- **Contributing files:** some-skill/references/extra.md
**Status:**
- \`extracted\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_failure
assert_output --partial "Research doc slug 'extra-source' missing from skill sources.md"
}
@test "F: a research-doc slug with no Status line at all is announced, not skipped silently" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Sources
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
- **Contributing files:** some-skill/references/extra.md
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success
assert_output --partial "Check 8 skipped for research-doc slug 'extra-source'"
assert_output --partial "its Status is absent, not \`extracted\`"
}
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Cycle 21 — G1: a bad target is a hard error, not a silent pass # Cycle 21 — G1: a bad target is a hard error, not a silent pass
# #
@@ -1361,6 +986,7 @@ EOF
- **Description:** A test source. - **Description:** A test source.
- **Contributing files:** references/sources.md - **Contributing files:** references/sources.md
- **Research doc:** (none) - **Research doc:** (none)
- **Basis:** SKILL.md
- **Status:** \`extracted\` - **Status:** \`extracted\`
EOF EOF
@@ -1446,6 +1072,7 @@ EOF
- **Description:** FILL IN: - **Description:** FILL IN:
- **Contributing files:** SKILL.md - **Contributing files:** SKILL.md
- **Research doc:** (none) - **Research doc:** (none)
- **Basis:** SKILL.md
- **Status:** \`extracted\` - **Status:** \`extracted\`
EOF EOF
@@ -1480,6 +1107,7 @@ EOF
- **Description:** A test source. - **Description:** A test source.
- **Contributing files:** SKILL.md - **Contributing files:** SKILL.md
- **Research doc:** (none) - **Research doc:** (none)
- **Basis:** SKILL.md
- **Status:** \`extracted\` - **Status:** \`extracted\`
## my-source ## my-source
@@ -1488,6 +1116,7 @@ EOF
- **Description:** The same slug a second time. - **Description:** The same slug a second time.
- **Contributing files:** references/nonexistent.md - **Contributing files:** references/nonexistent.md
- **Research doc:** (none) - **Research doc:** (none)
- **Basis:** SKILL.md
- **Status:** \`extracted\` - **Status:** \`extracted\`
EOF EOF
@@ -1497,7 +1126,7 @@ EOF
assert_output --partial "only the first block is checked" assert_output --partial "only the first block is checked"
} }
@test "G7: a second '- **Research doc:**' line in one entry is announced" { @test "G7: a second '- **Research doc:**' line in one entry FAILs (Research doc is single-valued, #121)" {
local skill="$TMPDIR/my-skill" local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill" make_skill_with_source_keys "$skill"
mkdir -p "$skill/references" mkdir -p "$skill/references"
@@ -1510,12 +1139,13 @@ EOF
- **Description:** A test source. - **Description:** A test source.
- **Contributing files:** SKILL.md - **Contributing files:** SKILL.md
- **Research doc:** (none) - **Research doc:** (none)
- **Basis:** SKILL.md
- **Research doc:** docs/research/sources.md - **Research doc:** docs/research/sources.md
- **Status:** \`extracted\` - **Status:** \`extracted\`
EOF EOF
run bash "$SCRIPT" "$skill" run bash "$SCRIPT" "$skill"
assert_success assert_failure
assert_output --partial "Multiple '- **Research doc:**' lines for 'my-source'" assert_output --partial "Multiple '- **Research doc:**' lines for 'my-source'"
} }
@@ -1664,6 +1294,7 @@ EOF
- **Description:** A brand-new source, never committed. - **Description:** A brand-new source, never committed.
- **Contributing files:** (none) - **Contributing files:** (none)
- **Research doc:** (none) - **Research doc:** (none)
- **Basis:** SKILL.md
- **Status:** \`extracted\` - **Status:** \`extracted\`
EOF EOF
@@ -1852,6 +1483,7 @@ PY
- references/other.md (the rubric) - references/other.md (the rubric)
- **Research doc:** (none) - **Research doc:** (none)
- **Basis:** SKILL.md
- **Status:** `extracted` - **Status:** `extracted`
EOF EOF
printf -- '---\nsource_keys:\n - my-source\n---\n\nnotes\n' > "$skill/references/other.md" printf -- '---\nsource_keys:\n - my-source\n---\n\nnotes\n' > "$skill/references/other.md"
@@ -1886,6 +1518,7 @@ EOF
- references/other.md (the rubric) - references/other.md (the rubric)
- **Research doc:** (none) - **Research doc:** (none)
- **Basis:** SKILL.md
- **Status:** `extracted` - **Status:** `extracted`
EOF EOF
printf -- '---\nsource_keys:\n - my-source\n---\n\nnotes\n' > "$skill/references/other.md" printf -- '---\nsource_keys:\n - my-source\n---\n\nnotes\n' > "$skill/references/other.md"
@@ -1924,6 +1557,7 @@ EOF
mkdir -p "$skill" mkdir -p "$skill"
make_skill_with_source_keys "$skill" make_skill_with_source_keys "$skill"
make_sources_md "$skill" make_sources_md "$skill"
sed -i 's#^- \*\*Basis:\*\* SKILL.md#- **Basis:** tracked-skill/SKILL.md#' "$skill/references/sources.md"
commit_as_base "$repo" commit_as_base "$repo"
# A copy of the same skill at a path git does not know — the everyday case # A copy of the same skill at a path git does not know — the everyday case
@@ -2060,3 +1694,257 @@ EOF
refute_output "" refute_output ""
assert_output --partial "$dir/notes.txt" assert_output --partial "$dir/notes.txt"
} }
# ---------------------------------------------------------------------------
# Cycle 28 — #121: `Research doc:` names exactly one Research registry;
# an entry with no registry writes `none` plus `Basis:`
#
# Helper: a fake repo holding one skill whose single entry is written verbatim
# from the fields passed in — make_upstream_skill only varies the Research doc
# value, and the Basis cases need a second field. The registry lives at
# docs/research/sources.md and a basis file at docs/basis.md.
# ---------------------------------------------------------------------------
make_entry_skill() {
local repo="$1"
local fields="$2"
local skill="$repo/my-skill"
mkdir -p "$skill/references" "$repo/docs/research"
cat > "$skill/SKILL.md" <<EOF
---
name: my-skill
description: A valid skill description.
metadata:
source_keys:
- my-source
---
## Step 1
Do the thing.
EOF
printf '%s\n' "# Basis" > "$repo/docs/basis.md"
printf '%s\n' "# Other" > "$repo/docs/other-basis.md"
{
printf '# Sources\n\n## my-source\n\n'
printf '%s\n' '- **URL:** https://example.com/my-source'
printf '%s\n' '- **Description:** A test source.'
printf '%s\n' '- **Contributing files:** SKILL.md'
printf '%s\n' "$fields"
printf '%s\n' '- **Status:** `extracted`'
} > "$skill/references/sources.md"
cat > "$repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** (none)
- **Status:** \`extracted\`
EOF
commit_as_base "$repo"
}
@test "#121 FAIL: a brace-expansion Research doc names more than one path" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" '- **Research doc:** docs/research/{sources,other}.md'
run bash "$SCRIPT" "$repo/my-skill"
assert_failure
assert_output --partial "Research doc names more than one path"
}
@test "#121 FAIL: a comma-separated Research doc names more than one path" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" '- **Research doc:** docs/research/sources.md, docs/research/other.md'
run bash "$SCRIPT" "$repo/my-skill"
assert_failure
assert_output --partial "Research doc names more than one path"
}
@test "#121 FAIL: a semicolon-separated pair of annotated paths names more than one path" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" '- **Research doc:** docs/research/sources.md (Releases section); docs/research/other.md (`delete_release` gotcha)'
run bash "$SCRIPT" "$repo/my-skill"
assert_failure
assert_output --partial "Research doc names more than one path"
}
@test "#121 pass: a comma or semicolon INSIDE the annotation is prose, not a list" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" '- **Research doc:** docs/research/sources.md (cross-cutting; no dedicated section, see notes)'
run bash "$SCRIPT" "$repo/my-skill"
assert_success
assert_output ""
}
@test "#121 FAIL: repeated Research doc lines are a list, not an INFO" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" "$(printf '%s\n%s' '- **Research doc:** docs/research/sources.md' '- **Research doc:** docs/research/other.md')"
run bash "$SCRIPT" "$repo/my-skill"
assert_failure
assert_output --partial "Multiple '- **Research doc:**' lines for 'my-source'"
}
@test "#121 pass: 'none' with one Basis bullet resolves and is clean" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" "$(printf '%s\n%s' '- **Research doc:** none' '- **Basis:** docs/basis.md')"
run bash "$SCRIPT" "$repo/my-skill"
assert_success
assert_output ""
}
@test "#121 pass: 'none — reason' carries a trailing annotation and still counts as none" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" "$(printf '%s\n%s' '- **Research doc:** none — org convention, not a research corpus entry' '- **Basis:** docs/basis.md')"
run bash "$SCRIPT" "$repo/my-skill"
assert_success
assert_output ""
}
@test "#121 pass: repeated single-path Basis bullets are each checked" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" "$(printf '%s\n%s\n%s' '- **Research doc:** none' '- **Basis:** docs/basis.md' '- **Basis:** docs/other-basis.md')"
run bash "$SCRIPT" "$repo/my-skill"
assert_success
assert_output ""
}
@test "#121 FAIL: a second Basis bullet naming a missing path fails even when the first resolves" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" "$(printf '%s\n%s\n%s' '- **Research doc:** none' '- **Basis:** docs/basis.md' '- **Basis:** docs/gone.md')"
run bash "$SCRIPT" "$repo/my-skill"
assert_failure
assert_output --partial "Basis path 'docs/gone.md' does not exist"
}
@test "#121 pass: a Basis header followed by '- ' path bullets is read too" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" "$(printf '%s\n%s\n%s\n%s' '- **Research doc:** none' '**Basis:**' '- docs/basis.md' '- docs/other-basis.md')"
run bash "$SCRIPT" "$repo/my-skill"
assert_success
assert_output ""
}
@test "#121 FAIL: 'none' with no Basis declares nothing" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" '- **Research doc:** none'
run bash "$SCRIPT" "$repo/my-skill"
assert_failure
assert_output --partial "Basis missing for 'my-source'"
}
@test "#121 FAIL: a Basis path that does not exist" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" "$(printf '%s\n%s' '- **Research doc:** none' '- **Basis:** docs/gone.md')"
run bash "$SCRIPT" "$repo/my-skill"
assert_failure
assert_output --partial "Basis path 'docs/gone.md' does not exist"
}
@test "#121 FAIL: one Basis value naming several paths is a list" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" "$(printf '%s\n%s' '- **Research doc:** none' '- **Basis:** docs/basis.md, docs/other-basis.md')"
run bash "$SCRIPT" "$repo/my-skill"
assert_failure
assert_output --partial "Basis value names more than one path"
}
@test "#121 pass: a '(removed in <sha>)' Basis skips the existence check" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" "$(printf '%s\n%s\n%s' '- **Research doc:** none' '- **Basis:** docs/basis.md' '- **Basis:** docs/deleted-adr.md (removed in 5b80f30)')"
run bash "$SCRIPT" "$repo/my-skill"
assert_success
assert_output ""
}
@test "#121 FAIL: a Basis annotated with something other than '(removed in <sha>)' is still existence-checked" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" "$(printf '%s\n%s' '- **Research doc:** none' '- **Basis:** docs/deleted-adr.md (removed later)')"
run bash "$SCRIPT" "$repo/my-skill"
assert_failure
assert_output --partial "Basis path 'docs/deleted-adr.md' does not exist"
}
@test "#121 pass: a Basis with a trailing annotation resolves its path" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" "$(printf '%s\n%s' '- **Research doc:** none' '- **Basis:** docs/basis.md (org convention file, not a corpus entry)')"
run bash "$SCRIPT" "$repo/my-skill"
assert_success
assert_output ""
}
@test "#121 FAIL: check 7 — a slug missing from a resolved registry is a FAIL, not an INFO" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" '- **Research doc:** docs/research/sources.md'
cat > "$repo/docs/research/sources.md" <<EOF
# Research
## different-slug
- **Contributing files:** (none)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$repo/my-skill"
assert_failure
assert_output --partial "FAIL"
assert_output --partial "Slug 'my-source' not found as H2 in research doc 'docs/research/sources.md'"
}
@test "#121 FAIL: a Research doc that resolves to a topic document is wrong, not 'not applicable'" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" '- **Research doc:** docs/research/remotes.md (whole-document reference)'
printf '# Remotes\n\n## Core Philosophy\n\nProse.\n' > "$repo/docs/research/remotes.md"
run bash "$SCRIPT" "$repo/my-skill"
assert_failure
assert_output --partial "is a topic document, not a Research registry"
refute_output --partial "not applicable"
}
@test "#121 INFO: an unresolvable Research doc path stays INFO and exits 0" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" '- **Research doc:** docs/research/gone/sources.md'
run bash "$SCRIPT" "$repo/my-skill"
assert_success
assert_output --partial "INFO"
assert_output --partial "does not exist"
}
@test "#121 INFO: a Basis cannot be existence-checked with no repo root, and says so" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
mkdir -p "$skill/references"
cat > "$skill/references/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source.
- **Contributing files:** SKILL.md
- **Research doc:** none
- **Basis:** docs/basis.md
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$skill"
assert_success
assert_output --partial "Basis check skipped for 'my-source'"
}
# --- Parser parity: the spelling of a field must not change what is read ----
@test "#121 parity: an inline Research doc with no leading hyphen is read, not reported missing" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" '**Research doc:** docs/research/sources.md'
run bash "$SCRIPT" "$repo/my-skill"
assert_success
refute_output --partial "Research doc field missing"
assert_output ""
}
@test "#121 parity: a Research doc header with one '- ' bullet is read" {
local repo="$TMPDIR/fakerepo"
make_entry_skill "$repo" "$(printf '%s\n%s' '**Research doc:**' '- docs/research/sources.md')"
run bash "$SCRIPT" "$repo/my-skill"
assert_success
refute_output --partial "Research doc field missing"
assert_output ""
}

6
plugins/kyberforge/.apm/skills/skill-author/references/contract.md
View File

@@ -252,6 +252,6 @@ inline that content directly into the skill (SKILL.md or a `references/` file) r
to the file's path. Plugins must be self-contained and portable — the org file may not exist to the file's path. Plugins must be self-contained and portable — the org file may not exist
wherever the plugin is installed, and in this repo such files are meant to be deleted once their wherever the plugin is installed, and in this repo such files are meant to be deleted once their
content is fully embedded downstream. Tag the inlined content with a `source_keys` entry using the content is fully embedded downstream. Tag the inlined content with a `source_keys` entry using the
same `references/sources.md` schema as the create flow's Step 6, noting in the `Research doc:` same `references/sources.md` schema as the create flow's Step 6: write `Research doc: none` and
field that the source is an org convention rather than a plugin research corpus entry, so name the org convention file in a `Basis:` line, so provenance survives after the source file is
provenance survives after the source file is gone. gone (annotate the Basis `(removed in <sha>)` once the file is deleted).

19
plugins/kyberforge/.apm/skills/skill-author/references/create.md
View File

@@ -171,11 +171,20 @@ If a research `sources.md` is present in the conversation context:
2. For each entry, determine which skill files it contributed to (SKILL.md and any files in 2. For each entry, determine which skill files it contributed to (SKILL.md and any files in
`references/` that drew from it). Update `Contributing files` accordingly — list skill files, `references/` that drew from it). Update `Contributing files` accordingly — list skill files,
not research topic files. not research topic files.
3. Write the updated content to `references/sources.md`. For each entry, include 3. Write the updated content to `references/sources.md`. Every entry carries exactly one
`- **Research doc:** <path>` where `<path>` is the relative path from the repo root to the `- **Research doc:** <path>` line. `<path>` is the relative path from the repo root to the
plugin-level research sources file this entry was drawn from (e.g. **Research registry** — the plugin-level research `sources.md` whose `## H2` headings are the
`plugins/myplugin/docs/research/docs/<topic>/sources.md`). This field is required on every source slugs (e.g. `plugins/myplugin/docs/research/docs/<topic>/sources.md`) — never a topic
entry — it makes the provenance chain explicit and is validated by `/factory-audit`. document, and never a list: no brace expansion, no comma- or semicolon-separated paths, no
second `Research doc:` line. A pointer to the topic document that digested the source goes in
an annotation after the path, e.g. `<registry path> (digested in remotes.md)`, where it is not
checked. `/factory-audit` fails a slug missing from the registry it names.
If the entry has no Research registry — an org convention, an ADR, a house-verified
reproduction — write `- **Research doc:** none` and name what it was drawn from with one
`- **Basis:** <repo path>` line per path. Each Basis path is checked to exist; annotate one
that has since been deleted `(removed in <sha>)` and the check is skipped. `none` with no Basis
is a FAIL.
4. Add `source_keys` to the frontmatter of `SKILL.md` (under `metadata`) listing the slugs of 4. Add `source_keys` to the frontmatter of `SKILL.md` (under `metadata`) listing the slugs of
sources that informed it. sources that informed it.
5. For each file in `references/` that was informed by research sources, add `source_keys` 5. For each file in `references/` that was informed by research sources, add `source_keys`