#!/usr/bin/env bash # lib-provenance-skill.sh — SOURCED, never executed. # # skill-audit's provenance suite: its validate-provenance.sh, minus the shared # Contributing-files parser (lib-contributing-files.sh holds the one copy) and # minus the --help dispatch that validate-provenance.sh now owns. The bash # argument handling, the preconditions and every exit code are lifted verbatim. # # The two provenance modes have DIFFERENT exit contracts and they are NOT # unified. Skill mode exits 0 with output whenever the only findings are INFO # — a check that could not run, announced rather than skipped silently — so a # caller must read exit 0 plus output as INFO-only findings, not as noise. # Agent mode (lib-provenance-agent.sh) prints nothing at all on a clean run and # additionally exits 0 SILENTLY when the scope walk-up finds no plugin package. # Skill mode has no such verdict: it has already hard-failed (exit 2) on a # directory that is not a skill before the interpreter starts. # # Skill mode also owns the --base-ref= flag (check 9) and the # VALIDATE_PROVENANCE_BASE_REF environment variable. Agent mode has no check 9 # and takes no flags, so a --base-ref passed to an agent target is still an # extra argument and is still rejected, exactly as before the merge. # # Consumed by: validate-provenance.sh, skill mode. # shellcheck shell=bash # shellcheck disable=SC2034 kyberforge_prov_skill_usage() { cat < [--base-ref=] Validate that a skill's sources provenance chain is complete and internally consistent. Arguments: skill-dir Path to the skill directory to validate. --base-ref=REF Git ref to diff references/sources.md against for check 9. Defaults to \`git merge-base HEAD origin/main\`. Override this when origin/main is not the right comparison point (a fork, a long-lived branch, a mirror with a different remote name). The VALIDATE_PROVENANCE_BASE_REF environment variable is an equivalent, lower-precedence way to set it — the flag wins if both are given, including when the flag is given empty (\`--base-ref=\`), which selects the default resolution. Exit codes: 0 All checks passed (or nothing to validate) 1 One or more checks failed 2 Usage error, or the argument is not a skill directory An exit code of 2 is NOT a finding. SKILL.md tells the auditor to surface a non-zero exit as findings, so a usage error leaving exit 1 with nothing on stdout was indistinguishable from a clean-but-failing run. Environment and argument problems exit 2; only real findings exit 1. Checks performed: 0 source_keys present but references/sources.md absent 1 FILL IN: placeholders in sources.md 2 source_keys in SKILL.md → slug exists in sources.md 3 source_keys in references/*.md → slug exists in sources.md (INFO if no source_keys; an explicit 'source_keys: []' declares the file house-authored and passes silently) 4 Contributing files listed in sources.md exist on disk. An explicit '(none)' skips silently; a Contributing files block this parser cannot read is reported as an INFO saying checks 4 and 5 did not run, never skipped silently. 5 Contributing files back-reference the parent slug in their source_keys 6 Research doc field present and not placeholder 7 Slug in sources.md present in upstream research doc (INFO only). A section annotation ('§ ...', '→ ...', '(...)') is stripped before the path is resolved; a path that still does not resolve is reported as an INFO saying checks 7 and 8 did not run, never skipped silently. 8 Extracted non-(none) slug in research doc present in sources.md 9 Description or Contributing files text changed since --base-ref (INFO only — a bash script cannot verify the claim is still TRUE, only that it changed; the auditor reads the named files to check that). Wrapped values are joined before comparison, so a re-wrap alone is not a change and a rewrite of any line of one is. A slug absent at the base ref is a creation, not a change, and is not flagged; a field that WAS there and is now gone is announced as a removal. When the base ref cannot be resolved, or references/sources.md is not tracked under this path at that ref, this is announced as ONE INFO for the whole check, never a silent skip. Checks 7 and 8 apply ONLY when the Research doc value names a research SOURCE INDEX — a file whose basename is sources.md, whose H2 headings ARE source slugs. A Research doc pointing at a topic document is reported as an INFO saying the two checks are not applicable, and every other reason they do not run is announced the same way. EOF } kyberforge_prov_skill_run() { local arg # --base-ref= is the only recognised flag, for check 9's base-ref # override. It is pulled out before the positional-count checks below so it # never counts against them — a caller passing it alongside skill-dir sees # the same argument-count behaviour as one who does not pass it at all, and a # genuinely extra positional argument is still rejected. # # BASE_REF_OVERRIDE is deliberately left UNSET here rather than initialised to # the empty string, and deliberately NOT declared `local`: `local X` with no # value still leaves X unset, but declaring it at all would scope it away from # a future caller that wants to inspect it. `--base-ref=` (given, but empty) # and "no flag at all" are different instructions — the first says "use the # default resolution, ignoring the environment", the second says "fall back to # the environment" — and an empty-string initialiser collapsed them: # `${BASE_REF_OVERRIDE:-$ENV}` treats an empty flag value as absent, so the # environment variable won and the usage text's "the flag wins if both are # given" was false for exactly that spelling. declare -a POSITIONAL_ARGS=() for arg in "$@"; do case "$arg" in --base-ref=*) BASE_REF_OVERRIDE="${arg#--base-ref=}" ;; *) POSITIONAL_ARGS+=("$arg") ;; esac done # Usage and environment problems exit 2, findings exit 1. See the usage text # above for why the two must not share a code. This is a deliberate divergence # from validate.sh, which has no 2 tier: validate.sh always prints PASS lines, # so a usage error there is visibly not a findings report. This script prints # NOTHING on a clean run, so exit 1 plus empty stdout was the only signal a # caller got either way. if [[ ${#POSITIONAL_ARGS[@]} -lt 1 ]]; then echo "Error: skill-dir is required." >&2 echo "" >&2 kyberforge_prov_skill_usage >&2 exit 2 fi # Extra positional arguments were silently dropped, so a typo'd flag or a second # path looked like it had been honoured. if [[ ${#POSITIONAL_ARGS[@]} -gt 1 ]]; then echo "Error: expected exactly one argument, got ${#POSITIONAL_ARGS[@]}: ${POSITIONAL_ARGS[*]}" >&2 echo "" >&2 kyberforge_prov_skill_usage >&2 exit 2 fi local SKILL_DIR_ARG="${POSITIONAL_ARGS[0]}" # The flag wins over the environment variable whenever the flag was GIVEN — # `+x` tests for presence, not for a non-empty value, which is the distinction # `:-` could not make. An empty result either way tells the Python body to fall # back to `git merge-base HEAD origin/main`. local BASE_REF if [[ -n "${BASE_REF_OVERRIDE+x}" ]]; then BASE_REF="$BASE_REF_OVERRIDE" else BASE_REF="${VALIDATE_PROVENANCE_BASE_REF:-}" fi # python3 is a HARD dependency. Without this preflight a missing interpreter # produced 'line NN: python3: command not found' and exit 127 — an exit code no # caller maps to anything, from a message that names this script's line number # rather than the missing dependency. if ! command -v python3 > /dev/null 2>&1; then echo "Error: python3 is required but was not found on PATH." >&2 echo " Why: skipping the provenance checks entirely would be a vacuous pass." >&2 echo " Fix: install python3 (pre-commit itself is a Python application, so it is almost certainly already present)." >&2 exit 2 fi # A path that is not a directory, or a directory that is not a skill, used to # reach the Python body, find no sources.md and no source_keys, take the # "nothing to validate" early exit and report exit 0 with no output — which # references/skill-validation-scripts.md explicitly told the auditor to read as a # pass. A typo'd target was therefore indistinguishable from a clean skill. # vale-wrap.sh hard-errors on a nonexistent path for exactly this reason. if [[ ! -d "$SKILL_DIR_ARG" ]]; then echo "Error: not a directory: $SKILL_DIR_ARG" >&2 echo " Why: a nonexistent target would otherwise report a silent pass." >&2 echo " Fix: pass the path of the skill directory to validate." >&2 exit 2 fi if [[ ! -f "$SKILL_DIR_ARG/SKILL.md" ]]; then echo "Error: not a skill directory (no SKILL.md): $SKILL_DIR_ARG" >&2 echo " Why: a directory with no SKILL.md has no provenance chain to validate, and reporting that as a pass hides the wrong-target mistake." >&2 echo " Fix: pass the skill directory itself, not its parent or its references/ subdirectory." >&2 exit 2 fi # The Python program, reassembled in the order the parser block sat in before # the merge: preamble, shared parser, body. local prog="$KYBERFORGE_PROV_SKILL_PREAMBLE_PY $KYBERFORGE_CONTRIBUTING_FILES_PY $KYBERFORGE_PROV_SKILL_BODY_PY" local rc=0 python3 -u - "$SKILL_DIR_ARG" "$BASE_REF" <<< "$prog" || rc=$? # The findings code travels in KYBERFORGE_PROV_RC and this function returns 0, # so the caller can invoke it UNTESTED. Testing a function's status (`f || RC=$?`) # disables errexit for its entire body, which would leave every command above # unguarded -- and no subshell or `set -e` inside can re-arm it once the call # sits in a condition context. Error paths above use `exit`, which is unaffected # either way; this keeps errexit armed for anything added later. KYBERFORGE_PROV_RC="$rc" return 0 } IFS='' read -r -d '' KYBERFORGE_PROV_SKILL_PREAMBLE_PY <<'KYBERFORGE_PROV_SKILL_PREAMBLE' || true import sys import os import re import subprocess # Output is UTF-8 for the same reason input is: under LC_ALL=C the streams # default to ASCII, and every finding this script prints contains an em dash. # Pinning only the reads moved the crash from the read to the write — a # UnicodeEncodeError inside print_findings(), which loses the whole report # after all the checks have already run. for _stream in (sys.stdout, sys.stderr): try: _stream.reconfigure(encoding='utf-8') except AttributeError: # pragma: no cover — Python < 3.7 pass skill_dir = os.path.abspath(sys.argv[1]) sources_md_path = os.path.join(skill_dir, "references", "sources.md") refs_dir = os.path.join(skill_dir, "references") # Empty string (the shell side passes "" when neither --base-ref nor # VALIDATE_PROVENANCE_BASE_REF was given) means: resolve the default via # `git merge-base HEAD origin/main` at check-9 time, below. base_ref_override = sys.argv[2] if len(sys.argv) > 2 else "" # --- Helpers --- # The trailing character class used to be CONSUMING — `[^`\n]` — so a # `FILL IN:` at end of line matched nothing and escaped checks 1 and 6 # entirely. `- **Description:** FILL IN:` is the most likely spelling of a # half-written entry, and it was the one spelling the placeholder gate could # not see. The exclusion it was really expressing is "not inside backticks", # which a lookahead states without eating a character. PLACEHOLDER_RE = re.compile(r'(?` in cwd. Returns (returncode, stdout, stderr) — never raises, so a missing git binary or an unexpected OSError is just another non-zero result the caller folds into "could not run", not a crash.""" try: result = subprocess.run( ["git"] + args, cwd=cwd, capture_output=True, text=True, encoding="utf-8", errors="replace" ) return result.returncode, result.stdout, result.stderr except OSError as exc: return 1, "", str(exc) def ref_exists(ref, cwd): """True when ref resolves to a commit in the repo at cwd.""" rc, _out, _err = run_git(["rev-parse", "--verify", "--quiet", ref + "^{commit}"], cwd) return rc == 0 def find_slug_block(content, slug): """The raw text of a '## ' entry's body, or None if no such H2.""" pattern = re.compile( r'^## ' + re.escape(slug) + r'\s*\n(.*?)(?=^## |\Z)', re.MULTILINE | re.DOTALL ) m = pattern.search(content) return m.group(1) if m else None # A field value ENDS at the next field, the next heading, or a blank line. # Every other non-blank line is a continuation of the value the author wrapped # across physical lines. # # This boundary is what the old `(.+)$` regex did not have. `.` does not cross # a newline, so only the FIRST physical line of a wrapped value was ever # compared — and a rewrite confined to a continuation line produced no finding # at all. That is verbatim the hedge-to-confident-claim regression #118 exists # to catch, invisible to the check written to catch it. The bullet branch had # the same defect one level down: a wrapped bullet's continuation does not # start with '- ', so the loop broke there and silently dropped every # remaining bullet. # # A continuation line that itself opens with bold text ('**note** — ...') is # read as a boundary and truncates the value. That is a known, narrow # false-negative, accepted because the alternative — no boundary at all — # is what produced the wide one above. FIELD_BOUNDARY_RE = re.compile(r'^(?:- )?\*\*|^#{1,6} ') def _is_field_boundary(stripped_line): """True when a stripped line starts a new field, bullet-less heading or H2.""" return bool(FIELD_BOUNDARY_RE.match(stripped_line)) def parse_field_raw(content, slug, field_name): """Raw text of a '**:**' field under a slug H2, wrapping joined. Mirrors the two authored shapes parse_contributing_files() and parse_status() already handle (inline value on the same line, or a bare heading followed by '- ' bullets), but returns text rather than a parsed structure, because check 9 diffs wording, not semantics. Continuation lines are joined into the value they belong to before the caller normalizes and compares, so a value wrapped across two lines and the same value on one line are the same text — and a change made on any line of a wrapped value is visible, not just one made on the first. Returns None when the H2 itself is absent (the slug did not exist at this content's revision) or the field is absent — both read as "no earlier claim to compare against" to the caller, which is deliberate: a field appearing for the first time is a creation, not a change. """ block = find_slug_block(content, slug) if block is None: return None lines = block.splitlines() inline_re = re.compile(r'^\- \*\*' + re.escape(field_name) + r':\*\*[ \t]*(.*)$') heading_re = re.compile(r'^\*\*' + re.escape(field_name) + r':\*\*[ \t]*$') for idx, line in enumerate(lines): im = inline_re.match(line) if im: parts = [im.group(1).strip()] for cont in lines[idx + 1:]: stripped = cont.strip() if not stripped or stripped.startswith("- ") or _is_field_boundary(stripped): break parts.append(stripped) joined = " ".join(p for p in parts if p).strip() return joined or None if heading_re.match(line): entries = [] for cont in lines[idx + 1:]: stripped = cont.strip() if not stripped: if entries: break continue if _is_field_boundary(stripped): break if stripped.startswith("- "): entries.append(stripped[2:].strip()) elif entries: # A wrapped bullet: fold it back into the bullet it # continues rather than ending the list here. entries[-1] = (entries[-1] + " " + stripped).strip() else: break return ", ".join(e for e in entries if e) or None return None def normalize_field_text(value): """Collapse whitespace so reformatting alone never registers as a change. True only because parse_field_raw() joins wrapped continuation lines first: collapsing whitespace inside a value that had already been truncated at its first newline normalized nothing a re-wrap could change. """ return re.sub(r'\s+', ' ', value).strip() findings = [] has_fail = False # A finding identical in every field is the same finding, and the same file is # now reached by more than one check — the walk that looks for source_keys and # check 3 both read every references/*.md, so an unreadable one would otherwise # be reported twice with the same words. Distinct findings about the same file # still both appear. def _record(entry): if entry not in findings: findings.append(entry) def emit_fail(desc, fpath, why, fix): global has_fail has_fail = True _record(("FAIL", desc, fpath, why, fix, None)) def emit_info(desc, fpath, note): _record(("INFO", desc, fpath, None, None, note)) def print_findings(): for entry in findings: kind = entry[0] desc = entry[1] fpath = entry[2] why = entry[3] fix = entry[4] note = entry[5] if kind == "FAIL": print(f"FAIL {desc} — {fpath}") print(f" Why: {why}") print(f" Fix: {fix}") print() else: print(f"INFO {desc} — {fpath}") print(f" Note: {note}") print() # --- Scan for any file with source_keys --- def emit_unreadable(rel, exc): """Report a file this script cannot decode. Never a silent skip.""" emit_fail( f"File is {exc}", rel, f"'{rel}' cannot be decoded, so its frontmatter — and any source_keys in it — " f"cannot be read. This used to be swallowed by a bare 'except Exception: return False', " f"which reported the unreadable file as having no source_keys and therefore as clean.", f"Re-save '{rel}' as UTF-8." ) def file_has_source_keys(fpath, rel): try: content = read_text(fpath) except EncodingError as exc: emit_unreadable(rel, exc) return False fm, _ = parse_frontmatter(content) if fm is None: return False return bool(parse_source_keys(fm)) def find_files_with_source_keys(): """Return list of (relative_path, abs_path) for all skill files with source_keys.""" results = [] for root, dirs, files in os.walk(skill_dir): # Skip hidden dirs dirs[:] = [d for d in dirs if not d.startswith('.')] for fname in sorted(files): if fname.endswith('.md'): abs_path = os.path.join(root, fname) rel = os.path.relpath(abs_path, skill_dir) if file_has_source_keys(abs_path, rel): results.append((rel, abs_path)) return results sources_md_exists = os.path.isfile(sources_md_path) files_with_source_keys = find_files_with_source_keys() # Early exit: nothing to validate. The scan itself can raise a finding — an # unreadable file — so print before leaving; the clean case still prints # nothing and exits 0. if not sources_md_exists and not files_with_source_keys: print_findings() sys.exit(1 if has_fail else 0) # Load sources.md if it exists sources_content = None if sources_md_exists: try: sources_content = read_text(sources_md_path) except EncodingError as exc: emit_unreadable("references/sources.md", exc) print_findings() sys.exit(1) sources_slugs = set(parse_h2_slugs(sources_content)) else: sources_slugs = set() # --- Check 0: source_keys without sources.md --- if not sources_md_exists: for rel, abs_path in files_with_source_keys: emit_fail( f"source_keys declared but references/sources.md is absent", rel, "source_keys references research provenance that has no sources index to validate against.", "Create references/sources.md with an H2 entry for each slug referenced by source_keys." ) print_findings() sys.exit(1) # --- Check 1: FILL IN: placeholders in sources.md --- for line in sources_content.splitlines(): if PLACEHOLDER_RE.search(line): emit_fail( "Unfilled FILL IN: placeholder", "references/sources.md", "sources.md contains an unfilled placeholder, meaning provenance is incomplete.", "Replace all 'FILL IN:' values in references/sources.md with real content." ) break # --- Check 2: source_keys in SKILL.md → slug exists in sources.md --- skill_md_path = os.path.join(skill_dir, "SKILL.md") if os.path.isfile(skill_md_path): try: skill_content = read_text(skill_md_path) except EncodingError as exc: emit_unreadable("SKILL.md", exc) skill_content = "" skill_fm, _ = parse_frontmatter(skill_content) skill_source_keys = parse_source_keys(skill_fm) for slug in skill_source_keys: if slug not in sources_slugs: emit_fail( f"source_keys slug '{slug}' not found in sources.md", "SKILL.md", f"SKILL.md declares '{slug}' as a source but there is no '## {slug}' heading in references/sources.md.", f"Add '## {slug}' entry to references/sources.md or remove '{slug}' from SKILL.md source_keys." ) # --- Check 3: source_keys in references/*.md → slug exists in sources.md (INFO if no source_keys) --- # os.walk, not os.listdir: find_files_with_source_keys() above already walks # references/ recursively, so a source_keys-bearing file in # references// was collected there — and then never validated here, # because the flat listdir could not see it. The two halves of the same check # disagreed about which files exist. if os.path.isdir(refs_dir): ref_paths = [] for root, dirs, files in os.walk(refs_dir): dirs[:] = sorted(d for d in dirs if not d.startswith('.')) for fname in sorted(files): if not fname.endswith('.md'): continue fpath = os.path.join(root, fname) if os.path.relpath(fpath, refs_dir) == "sources.md": continue ref_paths.append(fpath) for fpath in ref_paths: rel = os.path.relpath(fpath, skill_dir) try: ref_content = read_text(fpath) except EncodingError as exc: emit_unreadable(rel, exc) continue ref_fm, _ = parse_frontmatter(ref_content) ref_keys = parse_source_keys(ref_fm) if not ref_keys: # An explicit `source_keys: []` is a deliberate declaration that # the file is house-authored, and passes silently. The INFO is for # files that never said either way. if declares_empty_source_keys(ref_fm): continue emit_info( f"No source_keys frontmatter", rel, "This references file has no source_keys — provenance cannot be verified. " "Add source_keys frontmatter listing the slugs from references/sources.md that informed this file, " "or declare an explicit 'source_keys: []' if the file is house-authored and has no external source." ) else: for slug in ref_keys: if slug not in sources_slugs: emit_fail( f"source_keys slug '{slug}' not found in sources.md", rel, f"'{rel}' declares '{slug}' as a source but there is no '## {slug}' heading in references/sources.md.", f"Add '## {slug}' entry to references/sources.md or remove '{slug}' from {rel} source_keys." ) # --- Checks 4, 5, 6, 7, 8: Per-slug checks in sources.md --- repo_root = find_repo_root(skill_dir) # Collect all research doc paths we'll check (for Check 8) research_docs_seen = {} # abs_path → (rel_path, slugs referencing it, content) # Every per-slug parser below — parse_contributing_files, parse_research_docs, # parse_status — locates its block with pattern.search(), so a slug written # twice resolves to the FIRST block every time. Iterating the raw heading list # therefore checked the first block's fields twice and the second block's # never: a duplicated slug is half-validated, and looked fully validated. The # duplicate is announced and the repeat visit dropped. all_slugs = parse_h2_slugs(sources_content) unique_slugs = [] for _slug in all_slugs: if _slug in unique_slugs: continue unique_slugs.append(_slug) _count = all_slugs.count(_slug) if _count > 1: emit_info( f"Duplicate '## {_slug}' entry in sources.md — only the first block is checked", f"references/sources.md (## {_slug})", f"'## {_slug}' appears {_count} times. Every field parser here takes the first match, so the " f"second and later blocks' Contributing files, Research doc and Status are never validated — " f"checks 4, 5, 6, 7 and 8 did not run for them. " f"Merge the blocks into one entry, or give each a distinct slug and reference it from source_keys." ) for slug in unique_slugs: # Checks 4 and 5: Contributing files exist, and back-reference the slug. # `[]` and None are NOT the same answer here. `[]` is the author writing # "(none)" — there is nothing to check and the skip is correct. None is a # Contributing-files block this parser cannot read, and skipping THAT # silently disables both checks on the one entry least likely to be right, # which is the failure mode parse_contributing_files' own docstring warns # about. Say so out loud instead, the same way an unresolvable Research doc # value does. cf_files = parse_contributing_files(sources_content, slug) if cf_files is None: emit_info( f"Contributing-file checks skipped for '{slug}' — the Contributing files block could not be parsed", f"references/sources.md (## {slug})", f"The '## {slug}' entry has no Contributing files list this parser can read — a missing field, a bare heading, '*' bullets, a numbered list, or prose all read as unparsable rather than as an empty declaration. " f"Checks 4 and 5 did not run for this slug, so nothing verified that its contributing files exist or name it back. " f"Write the value as '- **Contributing files:** ', or as a '**Contributing files:**' heading followed by '- ' bullets — " f"or record '(none)' if this source contributed no files." ) elif cf_files: for cf_rel in cf_files: cf_abs = os.path.join(skill_dir, cf_rel) if not os.path.isfile(cf_abs): emit_fail( f"Contributing file '{cf_rel}' does not exist", f"references/sources.md (## {slug})", f"sources.md claims '{cf_rel}' was contributed to by slug '{slug}' but the file does not exist.", f"Create '{cf_rel}' relative to the skill directory, or correct the path in sources.md." ) else: # Check 5: Bidirectional — file should list slug in its source_keys # Skip sources.md itself if cf_rel == "references/sources.md": continue try: cf_content = read_text(cf_abs) except EncodingError as exc: emit_unreadable(cf_rel, exc) continue cf_fm, _ = parse_frontmatter(cf_content) cf_keys = parse_source_keys(cf_fm) if slug not in cf_keys: emit_fail( f"Contributing file '{cf_rel}' does not list '{slug}' in its source_keys", f"references/sources.md (## {slug})", f"sources.md says '{cf_rel}' was informed by '{slug}', but '{cf_rel}' does not declare '{slug}' in its source_keys frontmatter.", f"Add '{slug}' to the source_keys frontmatter of '{cf_rel}'." ) # Check 6: Research doc field required rd_values = parse_research_docs(sources_content, slug) if len(rd_values) > 1: emit_info( f"Multiple '- **Research doc:**' lines for '{slug}' — only the first is used", f"references/sources.md (## {slug})", f"The '## {slug}' entry has {len(rd_values)} Research doc lines; checks 7 and 8 ran against the first " f"('{rd_values[0]}') and never looked at the rest. " f"Keep one Research doc line per entry — if a slug genuinely came from two documents, split it into two slugs, " f"or name the extra document inside the first value's annotation where it is at least visible." ) rd_value = rd_values[0] if rd_values else None if rd_value is None: emit_fail( f"Research doc field missing", f"references/sources.md (## {slug})", f"The '## {slug}' entry in sources.md has no '- **Research doc:**' line.", f"Add '- **Research doc:** ' to the '## {slug}' entry in references/sources.md." ) elif rd_value == "" or PLACEHOLDER_RE.search(rd_value): emit_fail( f"Research doc field is empty or placeholder", f"references/sources.md (## {slug})", f"The '## {slug}' entry has an unfilled Research doc value.", f"Set '- **Research doc:**' to a real path relative to repo root, or '(none)' if not applicable." ) elif not research_doc_is_none(rd_value): # Check 7: Upstream forward — slug should appear in research doc. # Every path out of here that does NOT run the check says so out loud. rd_path = strip_research_doc_annotation(rd_value) if not repo_root: emit_info( f"Upstream checks skipped for '{slug}' — no repo root above the skill directory", f"references/sources.md (## {slug})", f"'{rd_value}' is a path relative to the repo root, but no ancestor of the skill directory contains a .git entry, " f"so it cannot be resolved. Checks 7 and 8 did not run for this slug. " f"Run this script against a skill inside a checkout." ) elif not rd_path: emit_info( f"Upstream checks skipped for '{slug}' — Research doc value names no path", f"references/sources.md (## {slug})", f"The Research doc value '{rd_value}' is entirely annotation — stripping the section marker leaves no path. " f"Checks 7 and 8 did not run for this slug. " f"Give the value a file path relative to the repo root, or record '(none)' if no research doc backs this entry." ) else: rd_abs = os.path.join(repo_root, rd_path) if not os.path.isfile(rd_abs): emit_info( f"Upstream checks skipped for '{slug}' — research doc '{rd_path}' does not exist", f"references/sources.md (## {slug})", f"'{rd_value}' resolves to '{rd_path}' relative to the repo root and no file is there. " f"Checks 7 and 8 did not run for this slug, so nothing verified that the research doc still backs it. " f"Point the value at one existing file — a brace expansion, a comma-separated list of paths, or a bare section title does not resolve — " f"or record '(none)' if no research doc backs this entry." ) elif os.path.basename(rd_path) != "sources.md": # Checks 7 and 8 both assume the Research doc is a research # SOURCE INDEX — a sources.md whose H2 headings ARE source # slugs. 30 of the 121 corpus entries point instead at a TOPIC # DOCUMENT (remotes.md, gitflow.md, api-reference.md), whose # H2s are headings like '## Core Philosophy'. A slug can never # match one, so check 7 reported all 30 as "slug not found" — # every one a false positive — and check 8, aimed at documents # that carry no '- **Status:**' line at all, was saved from a # matching flood of false FAILs only by an UNANNOUNCED skip on # that missing status. The premise, not the corpus, was wrong. # # A topic-document reference is a legitimate, useful value; it # just is not something these two checks can verify. Say that # once, out loud, instead of failing 30 entries for it. emit_info( f"Upstream checks not applicable for '{slug}' — research doc '{rd_path}' is a topic document, not a source index", f"references/sources.md (## {slug})", f"Checks 7 and 8 match slugs against the H2 headings of a research source index — a file named 'sources.md', " f"where each H2 IS a source slug. '{os.path.basename(rd_path)}' is a topic document, so its H2s are section " f"headings and no slug will ever match one. Checks 7 and 8 did not run for this slug. " f"This needs no fix: point the value at the research corpus's own sources.md only if you want the " f"provenance link machine-verified." ) else: try: rd_content = read_text(rd_abs) except EncodingError as exc: emit_info( f"Upstream checks skipped for '{slug}' — research doc '{rd_path}' is {exc}", f"references/sources.md (## {slug})", f"'{rd_path}' could not be decoded, so checks 7 and 8 did not run for this slug. " f"Re-save the research doc as UTF-8." ) continue rd_slugs = set(parse_h2_slugs(rd_content)) if slug not in rd_slugs: emit_info( f"Slug '{slug}' not found as H2 in research doc '{rd_path}'", f"references/sources.md (## {slug})", f"The research doc '{rd_path}' does not have a '## {slug}' heading. " f"The provenance link may be imprecise — the slug name in sources.md may differ from the research doc's heading." ) # Track for Check 8. The content is carried with the entry so # check 8 reuses this read rather than decoding the file a # second time, with a second chance to fail differently. if rd_abs not in research_docs_seen: research_docs_seen[rd_abs] = (rd_path, set(), rd_content) research_docs_seen[rd_abs][1].add(slug) # --- Check 8: Upstream reverse --- for rd_abs, (rd_rel, known_slugs, rd_content) in research_docs_seen.items(): for rd_slug in parse_h2_slugs(rd_content): # Parse this slug's Contributing files and Status in the research doc rd_cf = parse_contributing_files(rd_content, rd_slug) rd_status = parse_status(rd_content, rd_slug) # Skip if the research doc explicitly records no contributing files if rd_cf == []: continue # Skip if status is not `extracted` — and say so when the skip is what # kept the slug out of the FAIL below. A status of `referenced` or # `no content extracted` is a real reason not to demand the slug, but # it was applied in silence, so an entry that should have been in # sources.md and a status line nobody had updated produced the same # output: nothing. Only a MATERIAL skip is announced; when the slug is # already in sources.md the check passes either way and there is no # fail-open to disclose. if rd_status != "extracted": if rd_slug not in sources_slugs: shown = f"`{rd_status}`" if rd_status else "absent" emit_info( f"Check 8 skipped for research-doc slug '{rd_slug}' — its Status is {shown}, not `extracted`", f"{rd_rel} (## {rd_slug})", f"'{rd_rel}' has '## {rd_slug}' with contributing files but Status {shown}, and this skill's " f"sources.md has no '## {rd_slug}' entry. Check 8 only demands an entry for an `extracted` slug, " f"so it did not run here. If that status is stale — the content was extracted and the line was never " f"updated — this skill is missing a source entry; if it is accurate, nothing needs doing." ) continue # This slug should be in sources.md if rd_slug not in sources_slugs: emit_fail( f"Research doc slug '{rd_slug}' missing from skill sources.md", f"references/sources.md", f"The research doc '{rd_rel}' has '## {rd_slug}' with status `extracted` and contributing files, " f"but this skill's sources.md has no '## {rd_slug}' entry.", f"Add '## {rd_slug}' to references/sources.md or mark it as '(none)' in the research doc's Contributing files." ) # --- Check 9: Description / Contributing files changed since --base-ref --- # A structural fact — the field's TEXT differs from an earlier revision — is # all git can tell us. Whether the (possibly stronger) new wording is still # TRUE is a semantic question no parser here can answer; that is what sent # the earlier literal-text approaches (flagging a named-but-missing filename) # to 3/3 false positives against the real corpus without even catching the # bug that motivated this check. So check 9 does the one thing git reliably # can: detect the change, and hand the auditor the slug and field to go read, # never a verdict on the claim itself. Always INFO, never FAIL. if repo_root is None: emit_info( "Check 9 skipped — no repo root above the skill directory", "references/sources.md", "No ancestor of the skill directory contains a .git entry, so there is no git history to diff " "references/sources.md against. Check 9 did not run for any slug in this skill. " "Run this script against a skill inside a checkout to get this check." ) else: resolved_base_ref = base_ref_override.strip() resolve_error = None if not resolved_base_ref: rc, mb_out, mb_err = run_git(["merge-base", "HEAD", "origin/main"], repo_root) if rc == 0 and mb_out.strip(): resolved_base_ref = mb_out.strip() else: resolve_error = ( "`git merge-base HEAD origin/main` could not resolve a base ref" + (f" ({mb_err.strip()})" if mb_err.strip() else "") + " — there may be no origin/main remote, HEAD may be detached, or the clone may be shallow." ) elif not ref_exists(resolved_base_ref, repo_root): resolve_error = f"--base-ref value '{resolved_base_ref}' does not resolve to a commit in this repository." if resolve_error: emit_info( "Check 9 skipped — no base ref could be resolved", "references/sources.md", resolve_error + " Check 9 did not run for any slug in this skill. " "Pass --base-ref=, or set the VALIDATE_PROVENANCE_BASE_REF environment variable, " "to compare against something other than origin/main." ) else: sources_md_relpath = os.path.relpath(sources_md_path, repo_root) rc, old_sources_content, show_err = run_git( ["show", f"{resolved_base_ref}:{sources_md_relpath}"], repo_root ) if rc != 0: # The base ref resolved fine but `git show :` did not. # That single return code covers two situations this check cannot # tell apart, and only one of them is harmless: # # the file genuinely did not exist at the base ref — the whole # sources.md is new, every entry in it is a creation, and there # is nothing check 9 could have flagged; # # the path is not TRACKED under that name at the base ref — a # renamed skill directory, or a copy of the skill living # somewhere untracked or gitignored (an installed .claude/skills # tree is the everyday case). # # Treating both as "creation, nothing to flag" made the second one # a silent, whole-skill skip: the same directory audited at its # authoring path reported changed claims and at its deployed path # reported nothing, with no way to tell that from a clean run. # That is the exact fail-open this script's own header forbids — # "never a silent skip" — so announce it once for the whole check # and hand over git's own stderr, which is the only diagnostic # that separates the two cases. detail = show_err.strip().splitlines() detail = detail[0] if detail else "git gave no reason" emit_info( f"Check 9 skipped — '{sources_md_relpath}' is not tracked at {resolved_base_ref}", "references/sources.md", f"`git show {resolved_base_ref}:{sources_md_relpath}` failed ({detail}). " f"Either the file did not exist at that ref — in which case every entry is a " f"creation and there was nothing to flag — or this path is not tracked under " f"that name there: a renamed skill directory, or an untracked or gitignored copy " f"of the skill such as a deployed .claude/skills/ tree. " f"Check 9 did not run for any slug in this skill. " f"Re-run against the tracked authoring path, or pass --base-ref= naming a " f"commit where this path exists." ) old_sources_content = None if old_sources_content is not None: for slug in unique_slugs: changed_fields = [] removed_fields = [] for field_name in ("Description", "Contributing files"): old_value = parse_field_raw(old_sources_content, slug, field_name) new_value = parse_field_raw(sources_content, slug, field_name) if old_value is None and new_value is None: continue if old_value is None: # No earlier claim to compare against — a brand-new # entry, or a field that did not exist yet at the # base ref. That is a creation, not a change, and is # never flagged. continue if new_value is None: # The field existed at the base ref and is gone now. # This was folded into the creation skip above, which # justified only the other half: deleting a whole # '- **Description:**' line left NO finding anywhere — # no other check in this script requires the field, so # a claim could be removed as invisibly as it could be # strengthened. Announce it; the auditor decides # whether the removal was intended. removed_fields.append(field_name) continue if normalize_field_text(old_value) != normalize_field_text(new_value): changed_fields.append(field_name) if removed_fields: removed_list = " and ".join(removed_fields) emit_info( f"'{removed_list}' removed for '{slug}' since {resolved_base_ref}", f"references/sources.md (## {slug})", f"The '## {slug}' entry had {removed_list} at {resolved_base_ref} and has " f"none now. Nothing else in this script requires the field, so the removal " f"is otherwise invisible. Confirm it was deliberate — a provenance claim " f"withdrawn is as much a change to the chain as one rewritten — and " f"restore the field if it was lost to an edit." ) if changed_fields: field_list = " and ".join(changed_fields) emit_info( f"'{field_list}' changed for '{slug}' since {resolved_base_ref}", f"references/sources.md (## {slug})", f"The '## {slug}' entry's {field_list} text differs from the version at " f"{resolved_base_ref}. This script can confirm the entry is internally " f"consistent, but it cannot verify whether the claim itself is still true — a " f"retrofit once turned an honest hedge into an unsupported confident claim and " f"every structural check here passed it silently. Re-read the upstream research " f"doc named in this entry's Research doc field and the Contributing files it " f"lists, and confirm by hand that the wording still holds." ) print_findings() sys.exit(1 if has_fail else 0) KYBERFORGE_PROV_SKILL_BODY KYBERFORGE_PROV_SKILL_BODY_PY="${KYBERFORGE_PROV_SKILL_BODY_PY%$'\n'}"