@@ -11,7 +11,7 @@ Audit a skill directory against the agentskills.io specification and the house c
|
|||||||
|
|
||||||
`validate.sh` enforces two independent length families that must not be conflated: the agentskills.io spec conformance ceilings (500 lines, 2,770 words, both counting the whole file) and the ADR-0020 context budget (250/400 description characters, 600/900 body-only words).
|
`validate.sh` enforces two independent length families that must not be conflated: the agentskills.io spec conformance ceilings (500 lines, 2,770 words, both counting the whole file) and the ADR-0020 context budget (250/400 description characters, 600/900 body-only words).
|
||||||
|
|
||||||
Alongside those it runs four shape checks that are not length measurements at all. Two are FAILs: every routing target named in the description — in the compressed `Not <thing> -> <name>` arrow **and** in the prose form — must resolve to a real skill or agent, and every `references/<file>.md` the body names must exist on disk. Three are SUGGESTIONs: a missing boundary clause, a Gotchas section over five entries, and a Gotchas section over 25% of the body. The resolution universe for boundary targets is derived by walking up from the audited `SKILL.md` — the authoring root above it, its own apm package, and that package's declared `apm.yml` dependencies — so a fresh clone and a machine that has run `apm install` return the same verdict. When no universe can be determined the check prints `INFO ... DID NOT RUN` and does not silently pass.
|
Alongside those it runs shape checks that are not length measurements at all. Three are FAILs: every routing target named in the description — in the compressed `Not <thing> -> <name>` arrow **and** in the prose form — must resolve to a real skill or agent; every `references/<file>.md` the body names must exist on disk; and `metadata.version` must be present and three-part semver (ADR-0022). That last one is FAIL rather than SUGGESTION because the `skill-frontmatter` pre-commit hook rejects the file without it — an audit grading it lower would report ready-to-ship on a file the commit gate refuses. Three are SUGGESTIONs: a missing boundary clause, a Gotchas section over five entries, and a Gotchas section over 25% of the body. The resolution universe for boundary targets is derived by walking up from the audited `SKILL.md` — the authoring root above it, its own apm package, and that package's declared `apm.yml` dependencies — so a fresh clone and a machine that has run `apm install` return the same verdict. When no universe can be determined the check prints `INFO ... DID NOT RUN` and does not silently pass.
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
@@ -26,8 +26,8 @@ Provide the path to the skill directory to audit when invoking.
|
|||||||
| File | Purpose |
|
| File | Purpose |
|
||||||
|------|---------|
|
|------|---------|
|
||||||
| `SKILL.md` | Skill instructions for agents |
|
| `SKILL.md` | Skill instructions for agents |
|
||||||
| `scripts/validate.sh` | Structural validator — checks name format, name matches directory, description presence and length, body-only word count, line and whole-file word ceilings, boundary-clause presence, boundary-target resolution, `references/` pointer existence, Gotchas entry count and body share, placeholder detection, script executable bit, and interactive-prompt detection |
|
| `scripts/validate.sh` | Structural validator — checks name format, name matches directory, description presence and length, `metadata.version` presence and semver shape (ADR-0022), body-only word count, line and whole-file word ceilings, boundary-clause presence, boundary-target resolution, `references/` pointer existence, Gotchas entry count and body share, placeholder detection, script executable bit, and interactive-prompt detection |
|
||||||
| `scripts/validate-provenance.sh` | Provenance validator — checks sources.md completeness, source_keys/slug consistency, Contributing files existence, bidirectional linkage, Research doc: fields, and upstream research doc alignment |
|
| `scripts/validate-provenance.sh` | Provenance validator — checks sources.md completeness, source_keys/slug consistency, Contributing files existence, bidirectional linkage, Research doc: fields, upstream research doc alignment, and (check 9, INFO only) whether a slug's `Description` or `Contributing files` text has changed since a base ref — `--base-ref=<ref>` or `VALIDATE_PROVENANCE_BASE_REF`, defaulting to the merge base with `origin/main` |
|
||||||
| `scripts/vale-wrap.sh` | Vale prefilter wrapper — runs the bundled `Kyberforge` Vale styles against SKILL.md and reports alerts as deterministic FAILs ahead of Step 3's qualitative review |
|
| `scripts/vale-wrap.sh` | Vale prefilter wrapper — runs the bundled `Kyberforge` Vale styles against SKILL.md and reports alerts as deterministic FAILs ahead of Step 3's qualitative review |
|
||||||
| `assets/vale/.vale.ini` | Vale configuration — points Vale at the bundled `Kyberforge` style path, self-located relative to `vale-wrap.sh` |
|
| `assets/vale/.vale.ini` | Vale configuration — points Vale at the bundled `Kyberforge` style path, self-located relative to `vale-wrap.sh` |
|
||||||
| `assets/vale/styles/Kyberforge/CompositionNote.yml` | Vale rule — flags composition and architecture notes in a description (e.g. "cross-cutting", "entry point", "rather than duplicating") |
|
| `assets/vale/styles/Kyberforge/CompositionNote.yml` | Vale rule — flags composition and architecture notes in a description (e.g. "cross-cutting", "entry point", "rather than duplicating") |
|
||||||
@@ -41,7 +41,7 @@ Provide the path to the skill directory to audit when invoking.
|
|||||||
| `references/patterns.md` | Rubric for the patterns dimension — which instruction construct fits which job, and how each is correctly formed |
|
| `references/patterns.md` | Rubric for the patterns dimension — which instruction construct fits which job, and how each is correctly formed |
|
||||||
| `references/file-structure.md` | Rubric for the file-structure and internal-consistency dimensions — permitted directories, cross-plugin path rules and their two structural exemptions, README drift |
|
| `references/file-structure.md` | Rubric for the file-structure and internal-consistency dimensions — permitted directories, cross-plugin path rules and their two structural exemptions, README drift |
|
||||||
| `references/formatting-and-scripts.md` | Rubric for the formatting and scripts dimensions — heading and fencing conventions, and the agentic-use criteria for bundled scripts |
|
| `references/formatting-and-scripts.md` | Rubric for the formatting and scripts dimensions — heading and fencing conventions, and the agentic-use criteria for bundled scripts |
|
||||||
| `references/validation-scripts.md` | Step 1 troubleshooting — the manual structural fallback when `validate.sh` cannot run, and the script exit codes that are easy to misread (loaded only on a script failure) |
|
| `references/validation-scripts.md` | Step 1 troubleshooting — the manual structural fallback when `validate.sh` cannot run, and the script exit codes that are easy to misread (loaded on a script failure, and on any exit-0 run that printed something — `validate-provenance.sh`'s check 9 is INFO-only, so its findings arrive that way) |
|
||||||
| `references/sources.md` | Provenance record — agentskills.io sources that informed this skill and which files each contributed to |
|
| `references/sources.md` | Provenance record — agentskills.io sources that informed this skill and which files each contributed to |
|
||||||
| `tests/validate.bats` | (source-only) Bats test suite for validate.sh |
|
| `tests/validate.bats` | (source-only) Bats test suite for validate.sh |
|
||||||
| `tests/validate-provenance.bats` | (source-only) Bats test suite for validate-provenance.sh |
|
| `tests/validate-provenance.bats` | (source-only) Bats test suite for validate-provenance.sh |
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ description: >
|
|||||||
skill-author.
|
skill-author.
|
||||||
allowed-tools: Bash Read
|
allowed-tools: Bash Read
|
||||||
metadata:
|
metadata:
|
||||||
version: "1.0.0"
|
version: "1.0.1"
|
||||||
category: factory
|
category: factory
|
||||||
source_keys:
|
source_keys:
|
||||||
- agentskills-home
|
- agentskills-home
|
||||||
@@ -36,9 +36,9 @@ bash scripts/vale-wrap.sh <skill-dir>/SKILL.md
|
|||||||
|
|
||||||
`validate.sh` findings become the `### Structure` dimension — its FAILs and its SUGGESTIONs both, at the tier the script assigned. Report each once; never re-grade one under another dimension. Unresolved boundary targets are where this bites, because their tier turns on notation.
|
`validate.sh` findings become the `### Structure` dimension — its FAILs and its SUGGESTIONs both, at the tier the script assigned. Report each once; never re-grade one under another dimension. Unresolved boundary targets are where this bites, because their tier turns on notation.
|
||||||
|
|
||||||
If any of the three cannot run, or exits non-zero for a reason other than findings, read `references/validation-scripts.md` — it carries the manual fallback and the misleading exit codes. Ordinary content FAILs are the expected outcome here and need no fallback.
|
Read `references/validation-scripts.md` when any of the three cannot run or exits non-zero for a reason other than findings, **and whenever `validate-provenance.sh` exits 0 having printed anything**. Ordinary content FAILs are the expected outcome here and need no fallback.
|
||||||
|
|
||||||
`validate-provenance.sh` prints nothing on success, so read its exit code before you read its silence. **0** is a genuine pass. **1** means real findings: its FAILs and INFOs become a separate `### Provenance` dimension, and it emits Why and Fix itself — surface those verbatim. **2** means the check never ran — a usage or environment error, reason on stderr, no findings and often no stdout at all. On a 2, report `### Provenance` as unverified and quote the stderr reason. Never grade an exit 2 as a clean pass: empty stdout there means nothing was checked, not that nothing was wrong.
|
`validate-provenance.sh` reports through exit code **and** output; neither alone is the verdict. **0, silent** is a genuine pass. **0 with output** is INFO-only findings — still a `### Provenance` dimension; `references/validation-scripts.md` says what each obliges — for a check-9 INFO, reading rather than relaying. **1** is FAILs plus any INFOs; it emits Why and Fix itself — surface those verbatim. **2** means it never ran — a usage or environment error, reason on stderr, often no stdout — so report `### Provenance` unverified and quote that reason. Never grade an exit 2, or an exit 0 that printed, as a clean pass.
|
||||||
|
|
||||||
`vale-wrap.sh` applies the bundled `Kyberforge` style as a prefilter. Pass no `--config`; the wrapper locates its own. Every rule is graded `error`, so every alert is a FAIL. Report each one citing its rule ID, filed under the dimension it belongs to, and do not re-derive it by judgment:
|
`vale-wrap.sh` applies the bundled `Kyberforge` style as a prefilter. Pass no `--config`; the wrapper locates its own. Every rule is graded `error`, so every alert is a FAIL. Report each one citing its rule ID, filed under the dimension it belongs to, and do not re-derive it by judgment:
|
||||||
|
|
||||||
|
|||||||
@@ -6,8 +6,9 @@ source_keys:
|
|||||||
|
|
||||||
# Validation Scripts Reference
|
# Validation Scripts Reference
|
||||||
|
|
||||||
Read this when a Step 1 script fails, cannot run, or reports something that needs interpreting.
|
Read this when a Step 1 script fails, cannot run, or reports something that needs interpreting —
|
||||||
Nothing here is needed on a clean run.
|
including `validate-provenance.sh` exiting **0 having printed something**, which is INFO findings,
|
||||||
|
not a clean run. Its silent exit 0 is the only outcome that needs nothing here.
|
||||||
|
|
||||||
## Report the gap, do not guess
|
## Report the gap, do not guess
|
||||||
|
|
||||||
@@ -106,18 +107,33 @@ Three ways to read the result wrong:
|
|||||||
`python3` all exit **2** with a message on stderr. Exit 2 means the script never ran — report it
|
`python3` all exit **2** with a message on stderr. Exit 2 means the script never ran — report it
|
||||||
as an unaudited dimension, never as a pass and never as a finding. Exit 1 is findings.
|
as an unaudited dimension, never as a pass and never as a finding. Exit 1 is findings.
|
||||||
- **A check-9 INFO — `'<field>' changed for '<slug>' since <ref>` — means go read, not just relay.**
|
- **A check-9 INFO — `'<field>' changed for '<slug>' since <ref>` — means go read, not just relay.**
|
||||||
Check 9 diffs the current `references/sources.md` against a base ref (default: the merge base with
|
Check 9 diffs the current `references/sources.md` against a base ref and flags a slug whose
|
||||||
`origin/main`) and flags a slug whose `Description` or `Contributing files` text differs. It is
|
`Description` or `Contributing files` text differs. It is structurally incapable of telling you
|
||||||
structurally incapable of telling you whether the new wording is still *true* — it only detects
|
whether the new wording is still *true* — it only detects that the text changed — so when this
|
||||||
that the text changed — so when this INFO fires, open the Contributing files it names and the
|
INFO fires, open that slug's own entry: the document named in its `Research doc:` field, and the
|
||||||
document named in that slug's `Research doc:` field, and confirm by reading whether the (possibly
|
files its `Contributing files` list names. Read whichever the changed field is a claim *about* —
|
||||||
|
a Description-only change often leaves the file list untouched, so "open the Contributing files"
|
||||||
|
is where to look, not proof that they are what moved. Confirm by reading whether the (possibly
|
||||||
strengthened) claim genuinely holds. This is the one provenance finding this script cannot verify
|
strengthened) claim genuinely holds. This is the one provenance finding this script cannot verify
|
||||||
for you: every other check here is a structural fact you can relay as-is, but check 9's job is
|
for you: every other check here is a structural fact you can relay as-is, but check 9's job is
|
||||||
only to tell you *where* to spend that reading effort, not to replace it. Acknowledging the INFO
|
only to tell you *where* to spend that reading effort, not to replace it. Acknowledging the INFO
|
||||||
without opening those files is not auditing it. A single INFO naming "no base ref could be
|
without opening those files is not auditing it. Its companion — `'<field>' removed for '<slug>'
|
||||||
resolved" or "no repo root above the skill directory" is the same graceful-skip pattern as every
|
since <ref>` — is the same obligation in the other direction: a claim withdrawn rather than
|
||||||
other check here that cannot run — treat it as an unaudited dimension for that reason, not as a
|
rewritten. No other check here requires the field, so confirm the removal was deliberate.
|
||||||
finding about the skill.
|
- **The check-9 base ref defaults to `git merge-base HEAD origin/main`, and there are two ways to
|
||||||
|
override it.** `--base-ref=<ref>` on the command line, or the `VALIDATE_PROVENANCE_BASE_REF`
|
||||||
|
environment variable; the flag wins when both are given, including when it is given empty
|
||||||
|
(`--base-ref=`), which selects the default resolution and ignores the environment. Reach for one
|
||||||
|
on a fork, a long-lived branch, or a mirror whose remote is not called `origin` — and when a
|
||||||
|
review asks what changed since a specific commit rather than since the branch point.
|
||||||
|
- **A single check-9 INFO naming a whole-check skip is an unaudited dimension, not a finding about
|
||||||
|
the skill.** There are three: "no repo root above the skill directory", "no base ref could be
|
||||||
|
resolved", and "`<path>` is not tracked at `<ref>`". The third is the one to read carefully — it
|
||||||
|
fires when the base ref resolved but `git show <ref>:<path>` did not, which covers both a
|
||||||
|
genuinely new `sources.md` (nothing to flag) and a path git does not know under that name: a
|
||||||
|
renamed skill directory, or an installed, gitignored copy such as a deployed `.claude/skills/`
|
||||||
|
tree. Auditing the deployed copy silently checks nothing; re-run against the authoring path under
|
||||||
|
`plugins/*/.apm/skills/`.
|
||||||
- **`vale` reports `0 files`.** Treat the pass as NOT RUN, not as clean, and fall back to full
|
- **`vale` reports `0 files`.** Treat the pass as NOT RUN, not as clean, and fall back to full
|
||||||
Step 3 judgment for the dimensions it would have covered. The bundled `Kyberforge` style is
|
Step 3 judgment for the dimensions it would have covered. The bundled `Kyberforge` style is
|
||||||
scoped by glob in `assets/vale/.vale.ini`; a file outside those globs is silently not linted.
|
scoped by glob in `assets/vale/.vale.ini`; a file outside those globs is silently not linted.
|
||||||
|
|||||||
@@ -15,7 +15,8 @@ Arguments:
|
|||||||
a long-lived branch, a mirror with a different remote name).
|
a long-lived branch, a mirror with a different remote name).
|
||||||
The VALIDATE_PROVENANCE_BASE_REF environment variable is an
|
The VALIDATE_PROVENANCE_BASE_REF environment variable is an
|
||||||
equivalent, lower-precedence way to set it — the flag wins
|
equivalent, lower-precedence way to set it — the flag wins
|
||||||
if both are given.
|
if both are given, including when the flag is given empty
|
||||||
|
(\`--base-ref=\`), which selects the default resolution.
|
||||||
|
|
||||||
Exit codes:
|
Exit codes:
|
||||||
0 All checks passed (or nothing to validate)
|
0 All checks passed (or nothing to validate)
|
||||||
@@ -47,10 +48,13 @@ Checks performed:
|
|||||||
8 Extracted non-(none) slug in research doc present in sources.md
|
8 Extracted non-(none) slug in research doc present in sources.md
|
||||||
9 Description or Contributing files text changed since --base-ref (INFO
|
9 Description or Contributing files text changed since --base-ref (INFO
|
||||||
only — a bash script cannot verify the claim is still TRUE, only that it
|
only — a bash script cannot verify the claim is still TRUE, only that it
|
||||||
changed; the auditor reads the named files to check that). A slug absent
|
changed; the auditor reads the named files to check that). Wrapped values
|
||||||
at the base ref is a creation, not a change, and is not flagged. When the
|
are joined before comparison, so a re-wrap alone is not a change and a
|
||||||
base ref cannot be resolved at all, this is announced as ONE INFO for the
|
rewrite of any line of one is. A slug absent at the base ref is a
|
||||||
whole check, never a silent skip.
|
creation, not a change, and is not flagged; a field that WAS there and is
|
||||||
|
now gone is announced as a removal. When the base ref cannot be resolved,
|
||||||
|
or references/sources.md is not tracked under this path at that ref, this
|
||||||
|
is announced as ONE INFO for the whole check, never a silent skip.
|
||||||
|
|
||||||
Checks 7 and 8 apply ONLY when the Research doc value names a research SOURCE
|
Checks 7 and 8 apply ONLY when the Research doc value names a research SOURCE
|
||||||
INDEX — a file whose basename is sources.md, whose H2 headings ARE source
|
INDEX — a file whose basename is sources.md, whose H2 headings ARE source
|
||||||
@@ -70,8 +74,15 @@ fi
|
|||||||
# never counts against them — a caller passing it alongside skill-dir sees
|
# never counts against them — a caller passing it alongside skill-dir sees
|
||||||
# the same argument-count behaviour as one who does not pass it at all, and a
|
# the same argument-count behaviour as one who does not pass it at all, and a
|
||||||
# genuinely extra positional argument is still rejected.
|
# genuinely extra positional argument is still rejected.
|
||||||
|
#
|
||||||
|
# BASE_REF_OVERRIDE is deliberately left UNSET here rather than initialised to
|
||||||
|
# the empty string. `--base-ref=` (given, but empty) and "no flag at all" are
|
||||||
|
# different instructions — the first says "use the default resolution, ignoring
|
||||||
|
# the environment", the second says "fall back to the environment" — and an
|
||||||
|
# empty-string initialiser collapsed them: `${BASE_REF_OVERRIDE:-$ENV}` treats
|
||||||
|
# an empty flag value as absent, so the environment variable won and the usage
|
||||||
|
# text's "the flag wins if both are given" was false for exactly that spelling.
|
||||||
declare -a POSITIONAL_ARGS=()
|
declare -a POSITIONAL_ARGS=()
|
||||||
BASE_REF_OVERRIDE=""
|
|
||||||
for arg in "$@"; do
|
for arg in "$@"; do
|
||||||
case "$arg" in
|
case "$arg" in
|
||||||
--base-ref=*)
|
--base-ref=*)
|
||||||
@@ -107,10 +118,15 @@ fi
|
|||||||
|
|
||||||
SKILL_DIR_ARG="${POSITIONAL_ARGS[0]}"
|
SKILL_DIR_ARG="${POSITIONAL_ARGS[0]}"
|
||||||
|
|
||||||
# The flag wins over the environment variable when both are given; either is
|
# The flag wins over the environment variable whenever the flag was GIVEN —
|
||||||
# empty-string when unset, and an empty string tells the Python body to fall
|
# `+x` tests for presence, not for a non-empty value, which is the distinction
|
||||||
|
# `:-` could not make. An empty result either way tells the Python body to fall
|
||||||
# back to `git merge-base HEAD origin/main`.
|
# back to `git merge-base HEAD origin/main`.
|
||||||
BASE_REF="${BASE_REF_OVERRIDE:-${VALIDATE_PROVENANCE_BASE_REF:-}}"
|
if [[ -n "${BASE_REF_OVERRIDE+x}" ]]; then
|
||||||
|
BASE_REF="$BASE_REF_OVERRIDE"
|
||||||
|
else
|
||||||
|
BASE_REF="${VALIDATE_PROVENANCE_BASE_REF:-}"
|
||||||
|
fi
|
||||||
|
|
||||||
# python3 is a HARD dependency. Without this preflight a missing interpreter
|
# python3 is a HARD dependency. Without this preflight a missing interpreter
|
||||||
# produced 'line NN: python3: command not found' and exit 127 — an exit code no
|
# produced 'line NN: python3: command not found' and exit 127 — an exit code no
|
||||||
@@ -491,10 +507,21 @@ def find_repo_root(start_dir):
|
|||||||
# --- Check 9 helpers ---------------------------------------------------
|
# --- Check 9 helpers ---------------------------------------------------
|
||||||
# Check 9 needs a raw field VALUE (as text, to diff against an earlier
|
# Check 9 needs a raw field VALUE (as text, to diff against an earlier
|
||||||
# version), not the parsed structure parse_contributing_files() and
|
# version), not the parsed structure parse_contributing_files() and
|
||||||
# parse_status() return — a Contributing files list that reordered its
|
# parse_status() return. The ONE normalization applied is whitespace
|
||||||
# entries without changing them is not what this check is looking for, but
|
# collapsing, which is what makes a re-wrap or a re-indent invisible; nothing
|
||||||
# neither is normalizing so hard that a genuine rewrite disappears. Raw text,
|
# else is normalized away.
|
||||||
# whitespace-normalized, is the middle ground.
|
#
|
||||||
|
# In particular a REORDERED Contributing files list DOES fire this check, and
|
||||||
|
# that is deliberate — the header here used to claim the opposite, which the
|
||||||
|
# code never did. Order-insensitivity cannot be had for one field without
|
||||||
|
# distorting the other: the two fields share this parser, and the only way to
|
||||||
|
# ignore order is to split the value into items and sort them, which for a
|
||||||
|
# prose Description means splitting on commas and would then hide a genuine
|
||||||
|
# rewrite that merely permuted its clauses. Check 9 is always an INFO whose
|
||||||
|
# whole job is to point a human at a place to read; a reordered list costs
|
||||||
|
# that human one glance to dismiss, whereas a hidden rewrite is the exact
|
||||||
|
# failure #118 exists to catch. False positive over false negative, on this
|
||||||
|
# check, on purpose.
|
||||||
|
|
||||||
def run_git(args, cwd):
|
def run_git(args, cwd):
|
||||||
"""Run `git <args>` in cwd. Returns (returncode, stdout, stderr) — never
|
"""Run `git <args>` in cwd. Returns (returncode, stdout, stderr) — never
|
||||||
@@ -523,14 +550,44 @@ def find_slug_block(content, slug):
|
|||||||
m = pattern.search(content)
|
m = pattern.search(content)
|
||||||
return m.group(1) if m else None
|
return m.group(1) if m else None
|
||||||
|
|
||||||
|
# A field value ENDS at the next field, the next heading, or a blank line.
|
||||||
|
# Every other non-blank line is a continuation of the value the author wrapped
|
||||||
|
# across physical lines.
|
||||||
|
#
|
||||||
|
# This boundary is what the old `(.+)$` regex did not have. `.` does not cross
|
||||||
|
# a newline, so only the FIRST physical line of a wrapped value was ever
|
||||||
|
# compared — and a rewrite confined to a continuation line produced no finding
|
||||||
|
# at all. That is verbatim the hedge-to-confident-claim regression #118 exists
|
||||||
|
# to catch, invisible to the check written to catch it. The bullet branch had
|
||||||
|
# the same defect one level down: a wrapped bullet's continuation does not
|
||||||
|
# start with '- ', so the loop broke there and silently dropped every
|
||||||
|
# remaining bullet.
|
||||||
|
#
|
||||||
|
# A continuation line that itself opens with bold text ('**note** — ...') is
|
||||||
|
# read as a boundary and truncates the value. That is a known, narrow
|
||||||
|
# false-negative, accepted because the alternative — no boundary at all —
|
||||||
|
# is what produced the wide one above.
|
||||||
|
FIELD_BOUNDARY_RE = re.compile(r'^(?:- )?\*\*|^#{1,6} ')
|
||||||
|
|
||||||
|
|
||||||
|
def _is_field_boundary(stripped_line):
|
||||||
|
"""True when a stripped line starts a new field, bullet-less heading or H2."""
|
||||||
|
return bool(FIELD_BOUNDARY_RE.match(stripped_line))
|
||||||
|
|
||||||
|
|
||||||
def parse_field_raw(content, slug, field_name):
|
def parse_field_raw(content, slug, field_name):
|
||||||
"""Raw text of a '**<field_name>:**' field under a slug H2.
|
"""Raw text of a '**<field_name>:**' field under a slug H2, wrapping joined.
|
||||||
|
|
||||||
Mirrors the two authored shapes parse_contributing_files() and
|
Mirrors the two authored shapes parse_contributing_files() and
|
||||||
parse_status() already handle (inline value on the same line, or a
|
parse_status() already handle (inline value on the same line, or a
|
||||||
bare heading followed by '- ' bullets), but returns text rather than a
|
bare heading followed by '- ' bullets), but returns text rather than a
|
||||||
parsed structure, because check 9 diffs wording, not semantics.
|
parsed structure, because check 9 diffs wording, not semantics.
|
||||||
|
|
||||||
|
Continuation lines are joined into the value they belong to before the
|
||||||
|
caller normalizes and compares, so a value wrapped across two lines and
|
||||||
|
the same value on one line are the same text — and a change made on any
|
||||||
|
line of a wrapped value is visible, not just one made on the first.
|
||||||
|
|
||||||
Returns None when the H2 itself is absent (the slug did not exist at
|
Returns None when the H2 itself is absent (the slug did not exist at
|
||||||
this content's revision) or the field is absent — both read as "no
|
this content's revision) or the field is absent — both read as "no
|
||||||
earlier claim to compare against" to the caller, which is deliberate:
|
earlier claim to compare against" to the caller, which is deliberate:
|
||||||
@@ -539,28 +596,49 @@ def parse_field_raw(content, slug, field_name):
|
|||||||
block = find_slug_block(content, slug)
|
block = find_slug_block(content, slug)
|
||||||
if block is None:
|
if block is None:
|
||||||
return None
|
return None
|
||||||
inline_re = re.compile(r'^\- \*\*' + re.escape(field_name) + r':\*\* (.+)$', re.MULTILINE)
|
lines = block.splitlines()
|
||||||
im = inline_re.search(block)
|
inline_re = re.compile(r'^\- \*\*' + re.escape(field_name) + r':\*\*[ \t]*(.*)$')
|
||||||
|
heading_re = re.compile(r'^\*\*' + re.escape(field_name) + r':\*\*[ \t]*$')
|
||||||
|
|
||||||
|
for idx, line in enumerate(lines):
|
||||||
|
im = inline_re.match(line)
|
||||||
if im:
|
if im:
|
||||||
return im.group(1).strip()
|
parts = [im.group(1).strip()]
|
||||||
heading_re = re.compile(r'^\*\*' + re.escape(field_name) + r':\*\*\s*$', re.MULTILINE)
|
for cont in lines[idx + 1:]:
|
||||||
hm = heading_re.search(block)
|
stripped = cont.strip()
|
||||||
if not hm:
|
if not stripped or stripped.startswith("- ") or _is_field_boundary(stripped):
|
||||||
return None
|
break
|
||||||
lines = []
|
parts.append(stripped)
|
||||||
for line in block[hm.end():].splitlines():
|
joined = " ".join(p for p in parts if p).strip()
|
||||||
line = line.strip()
|
return joined or None
|
||||||
if not line:
|
if heading_re.match(line):
|
||||||
if lines:
|
entries = []
|
||||||
|
for cont in lines[idx + 1:]:
|
||||||
|
stripped = cont.strip()
|
||||||
|
if not stripped:
|
||||||
|
if entries:
|
||||||
break
|
break
|
||||||
continue
|
continue
|
||||||
if not line.startswith("- "):
|
if _is_field_boundary(stripped):
|
||||||
break
|
break
|
||||||
lines.append(line[2:].strip())
|
if stripped.startswith("- "):
|
||||||
return ", ".join(lines) if lines else None
|
entries.append(stripped[2:].strip())
|
||||||
|
elif entries:
|
||||||
|
# A wrapped bullet: fold it back into the bullet it
|
||||||
|
# continues rather than ending the list here.
|
||||||
|
entries[-1] = (entries[-1] + " " + stripped).strip()
|
||||||
|
else:
|
||||||
|
break
|
||||||
|
return ", ".join(e for e in entries if e) or None
|
||||||
|
return None
|
||||||
|
|
||||||
def normalize_field_text(value):
|
def normalize_field_text(value):
|
||||||
"""Collapse whitespace so reformatting alone never registers as a change."""
|
"""Collapse whitespace so reformatting alone never registers as a change.
|
||||||
|
|
||||||
|
True only because parse_field_raw() joins wrapped continuation lines
|
||||||
|
first: collapsing whitespace inside a value that had already been
|
||||||
|
truncated at its first newline normalized nothing a re-wrap could change.
|
||||||
|
"""
|
||||||
return re.sub(r'\s+', ' ', value).strip()
|
return re.sub(r'\s+', ' ', value).strip()
|
||||||
|
|
||||||
findings = []
|
findings = []
|
||||||
@@ -1025,28 +1103,82 @@ else:
|
|||||||
["show", f"{resolved_base_ref}:{sources_md_relpath}"], repo_root
|
["show", f"{resolved_base_ref}:{sources_md_relpath}"], repo_root
|
||||||
)
|
)
|
||||||
if rc != 0:
|
if rc != 0:
|
||||||
# The base ref resolved fine, but references/sources.md did not
|
# The base ref resolved fine but `git show <ref>:<path>` did not.
|
||||||
# exist there at all — the whole file is new. Every entry in it
|
# That single return code covers two situations this check cannot
|
||||||
# is therefore a creation, not a change: nothing to flag, and
|
# tell apart, and only one of them is harmless:
|
||||||
# this is not a structural failure of the check, so no INFO
|
#
|
||||||
# either. Same reasoning applies per-slug below when the ref
|
# the file genuinely did not exist at the base ref — the whole
|
||||||
# resolved but a given '## <slug>' heading did not exist yet.
|
# sources.md is new, every entry in it is a creation, and there
|
||||||
|
# is nothing check 9 could have flagged;
|
||||||
|
#
|
||||||
|
# the path is not TRACKED under that name at the base ref — a
|
||||||
|
# renamed skill directory, or a copy of the skill living
|
||||||
|
# somewhere untracked or gitignored (an installed .claude/skills
|
||||||
|
# tree is the everyday case).
|
||||||
|
#
|
||||||
|
# Treating both as "creation, nothing to flag" made the second one
|
||||||
|
# a silent, whole-skill skip: the same directory audited at its
|
||||||
|
# authoring path reported changed claims and at its deployed path
|
||||||
|
# reported nothing, with no way to tell that from a clean run.
|
||||||
|
# That is the exact fail-open this script's own header forbids —
|
||||||
|
# "never a silent skip" — so announce it once for the whole check
|
||||||
|
# and hand over git's own stderr, which is the only diagnostic
|
||||||
|
# that separates the two cases.
|
||||||
|
detail = show_err.strip().splitlines()
|
||||||
|
detail = detail[0] if detail else "git gave no reason"
|
||||||
|
emit_info(
|
||||||
|
f"Check 9 skipped — '{sources_md_relpath}' is not tracked at {resolved_base_ref}",
|
||||||
|
"references/sources.md",
|
||||||
|
f"`git show {resolved_base_ref}:{sources_md_relpath}` failed ({detail}). "
|
||||||
|
f"Either the file did not exist at that ref — in which case every entry is a "
|
||||||
|
f"creation and there was nothing to flag — or this path is not tracked under "
|
||||||
|
f"that name there: a renamed skill directory, or an untracked or gitignored copy "
|
||||||
|
f"of the skill such as a deployed .claude/skills/ tree. "
|
||||||
|
f"Check 9 did not run for any slug in this skill. "
|
||||||
|
f"Re-run against the tracked authoring path, or pass --base-ref=<ref> naming a "
|
||||||
|
f"commit where this path exists."
|
||||||
|
)
|
||||||
old_sources_content = None
|
old_sources_content = None
|
||||||
|
|
||||||
if old_sources_content is not None:
|
if old_sources_content is not None:
|
||||||
for slug in unique_slugs:
|
for slug in unique_slugs:
|
||||||
changed_fields = []
|
changed_fields = []
|
||||||
|
removed_fields = []
|
||||||
for field_name in ("Description", "Contributing files"):
|
for field_name in ("Description", "Contributing files"):
|
||||||
old_value = parse_field_raw(old_sources_content, slug, field_name)
|
old_value = parse_field_raw(old_sources_content, slug, field_name)
|
||||||
new_value = parse_field_raw(sources_content, slug, field_name)
|
new_value = parse_field_raw(sources_content, slug, field_name)
|
||||||
if old_value is None or new_value is None:
|
if old_value is None and new_value is None:
|
||||||
|
continue
|
||||||
|
if old_value is None:
|
||||||
# No earlier claim to compare against — a brand-new
|
# No earlier claim to compare against — a brand-new
|
||||||
# entry, or a field that did not exist yet at the
|
# entry, or a field that did not exist yet at the
|
||||||
# base ref. That is a creation, not a change, and is
|
# base ref. That is a creation, not a change, and is
|
||||||
# never flagged.
|
# never flagged.
|
||||||
continue
|
continue
|
||||||
|
if new_value is None:
|
||||||
|
# The field existed at the base ref and is gone now.
|
||||||
|
# This was folded into the creation skip above, which
|
||||||
|
# justified only the other half: deleting a whole
|
||||||
|
# '- **Description:**' line left NO finding anywhere —
|
||||||
|
# no other check in this script requires the field, so
|
||||||
|
# a claim could be removed as invisibly as it could be
|
||||||
|
# strengthened. Announce it; the auditor decides
|
||||||
|
# whether the removal was intended.
|
||||||
|
removed_fields.append(field_name)
|
||||||
|
continue
|
||||||
if normalize_field_text(old_value) != normalize_field_text(new_value):
|
if normalize_field_text(old_value) != normalize_field_text(new_value):
|
||||||
changed_fields.append(field_name)
|
changed_fields.append(field_name)
|
||||||
|
if removed_fields:
|
||||||
|
removed_list = " and ".join(removed_fields)
|
||||||
|
emit_info(
|
||||||
|
f"'{removed_list}' removed for '{slug}' since {resolved_base_ref}",
|
||||||
|
f"references/sources.md (## {slug})",
|
||||||
|
f"The '## {slug}' entry had {removed_list} at {resolved_base_ref} and has "
|
||||||
|
f"none now. Nothing else in this script requires the field, so the removal "
|
||||||
|
f"is otherwise invisible. Confirm it was deliberate — a provenance claim "
|
||||||
|
f"withdrawn is as much a change to the chain as one rewritten — and "
|
||||||
|
f"restore the field if it was lost to an edit."
|
||||||
|
)
|
||||||
if changed_fields:
|
if changed_fields:
|
||||||
field_list = " and ".join(changed_fields)
|
field_list = " and ".join(changed_fields)
|
||||||
emit_info(
|
emit_info(
|
||||||
|
|||||||
@@ -1289,6 +1289,50 @@ else:
|
|||||||
if desc:
|
if desc:
|
||||||
ok("description has no unfilled placeholders")
|
ok("description has no unfilled placeholders")
|
||||||
|
|
||||||
|
# --- ADR-0022: metadata.version is mandatory -------------------------------
|
||||||
|
# FAIL, not SUGGESTION, and the tier is set by the gate rather than by taste.
|
||||||
|
# `.pre-commit-config.yaml`'s `skill-frontmatter` hook REJECTS a SKILL.md with
|
||||||
|
# no `metadata.version`, and rejects a value that is not three-part semver.
|
||||||
|
# skill-author's Step 4 says to run this audit and "resolve every FAIL", so any
|
||||||
|
# tier below FAIL lets that step report done on a skill the commit gate then
|
||||||
|
# refuses — the same audit-disagrees-with-the-gate failure the MAX_LINES note
|
||||||
|
# below warns about, arrived at from the other direction. Verified before this
|
||||||
|
# check existed: a SKILL.md with no `metadata:` block at all reported "All
|
||||||
|
# checks passed".
|
||||||
|
#
|
||||||
|
# The rule is DUPLICATED from that hook for the same cache-isolation reason as
|
||||||
|
# every other constant here — an installed plugin's scripts cannot read the
|
||||||
|
# repo-root config. Keep the two in step: this check must accept exactly what
|
||||||
|
# the hook accepts.
|
||||||
|
SEMVER_RE = re.compile(r'^\d+\.\d+\.\d+$')
|
||||||
|
|
||||||
|
try:
|
||||||
|
fm_data = yaml.safe_load(fm)
|
||||||
|
except Exception:
|
||||||
|
# Unreachable in practice: description_value() above parses the same text
|
||||||
|
# and hard-exits on a YAML error, so anything arriving here already parsed.
|
||||||
|
fm_data = None
|
||||||
|
metadata_block = fm_data.get('metadata') if isinstance(fm_data, dict) else None
|
||||||
|
|
||||||
|
if not isinstance(metadata_block, dict) or metadata_block.get('version') is None:
|
||||||
|
fail("frontmatter has no metadata.version — ADR-0022 makes it mandatory for "
|
||||||
|
"every skill, and the skill-frontmatter pre-commit hook rejects the file "
|
||||||
|
"without it. Add `metadata:` / ` version: \"1.0.0\"` (new skills start "
|
||||||
|
"at \"0.1.0\")")
|
||||||
|
else:
|
||||||
|
version_value = metadata_block['version']
|
||||||
|
# NOT str()-coerced blind: `version: 1.0` is a YAML float, and its "1.0"
|
||||||
|
# spelling is exactly the two-part value the hook rejects — coercing and
|
||||||
|
# then matching keeps this check and the hook agreeing on that case.
|
||||||
|
version_text = version_value if isinstance(version_value, str) else str(version_value)
|
||||||
|
version_text = version_text.strip()
|
||||||
|
if SEMVER_RE.match(version_text):
|
||||||
|
ok(f"metadata.version present: '{version_text}' (ADR-0022)")
|
||||||
|
else:
|
||||||
|
fail(f"metadata.version '{version_text}' is not three-part semver — the "
|
||||||
|
f"skill-frontmatter pre-commit hook rejects it. Use MAJOR.MINOR.PATCH, "
|
||||||
|
f"e.g. \"1.0.0\"")
|
||||||
|
|
||||||
# SKILL.md size ceilings (agentskills.io skill-authoring.md: 500 lines,
|
# SKILL.md size ceilings (agentskills.io skill-authoring.md: 500 lines,
|
||||||
# ~5,000 tokens). Both constants are DUPLICATED from the repo-root pre-commit
|
# ~5,000 tokens). Both constants are DUPLICATED from the repo-root pre-commit
|
||||||
# hook scripts/skill-size-check.sh — a plugin skill's scripts cannot read files
|
# hook scripts/skill-size-check.sh — a plugin skill's scripts cannot read files
|
||||||
@@ -1515,11 +1559,66 @@ def stdin_redirected(line, prev_line):
|
|||||||
unquoted = re.sub(r'"[^"]*"|\'[^\']*\'', '', line)
|
unquoted = re.sub(r'"[^"]*"|\'[^\']*\'', '', line)
|
||||||
return '<' in unquoted or prev_line.rstrip().endswith('|')
|
return '<' in unquoted or prev_line.rstrip().endswith('|')
|
||||||
|
|
||||||
|
# A here-doc body is DATA, not command position. Every script in this corpus
|
||||||
|
# carries a `usage() { cat <<EOF ... EOF; }`, and prose wrapped inside one puts
|
||||||
|
# ordinary English at the start of a line — "read is reported as an INFO ..."
|
||||||
|
# in this skill's own validate-provenance.sh, which made skill-audit hard-FAIL
|
||||||
|
# on its own script. Reflowing that one sentence would have cleared the finding
|
||||||
|
# and left the cause: every future usage text is one wrap away from the same
|
||||||
|
# false positive, and the remedy an author reaches for is contorting working
|
||||||
|
# source, which the note above records has already happened twice.
|
||||||
|
#
|
||||||
|
# Detection is deliberately conservative in the direction that matters. A
|
||||||
|
# here-doc body is skipped only when its terminator is actually found further
|
||||||
|
# down the file; an opener with no terminator — the shape a stray `<<` inside a
|
||||||
|
# string would produce — is ignored rather than allowed to swallow the tail,
|
||||||
|
# because swallowing the tail is a false NEGATIVE and this check exists to fail
|
||||||
|
# closed. `<<<` here-strings open nothing and are excluded by the lookbehind.
|
||||||
|
HEREDOC_START_RE = re.compile(r'(?<!<)<<-?\s*(["\']?)([A-Za-z_][A-Za-z0-9_]*)\1')
|
||||||
|
|
||||||
|
|
||||||
|
def heredoc_delimiter(line):
|
||||||
|
"""The here-doc terminator this line opens, or None."""
|
||||||
|
m = HEREDOC_START_RE.search(line)
|
||||||
|
return m.group(2) if m else None
|
||||||
|
|
||||||
|
|
||||||
|
def heredoc_body_indices(lines):
|
||||||
|
"""Line indices that are here-doc BODY (plus its terminator), not code."""
|
||||||
|
skip = set()
|
||||||
|
i, n = 0, len(lines)
|
||||||
|
while i < n:
|
||||||
|
stripped = lines[i].strip()
|
||||||
|
delim = None if stripped.startswith('#') else heredoc_delimiter(lines[i])
|
||||||
|
if delim:
|
||||||
|
# `<<-` allows an indented terminator, so compare stripped.
|
||||||
|
for j in range(i + 1, n):
|
||||||
|
if lines[j].strip() == delim:
|
||||||
|
skip.update(range(i + 1, j + 1))
|
||||||
|
i = j
|
||||||
|
break
|
||||||
|
i += 1
|
||||||
|
return skip
|
||||||
|
|
||||||
|
|
||||||
|
# The here-doc exemption applies to the `read` heuristic ONLY, and the
|
||||||
|
# asymmetry is the point. `read` is an ordinary English verb, so any prose a
|
||||||
|
# script prints is one line-wrap away from opening with it. `input(` is not a
|
||||||
|
# word — a line beginning `input(` inside a here-doc is an embedded Python
|
||||||
|
# program pausing for a keypress, which is exactly what this check is for, and
|
||||||
|
# these scripts embed Python in a here-doc as a matter of course. Exempting the
|
||||||
|
# whole body would have disarmed the check across every script in the corpus.
|
||||||
def interactive_reads(source):
|
def interactive_reads(source):
|
||||||
hits = []
|
hits = []
|
||||||
prev_line = ''
|
prev_line = ''
|
||||||
for line in source.splitlines():
|
lines = source.splitlines()
|
||||||
|
in_heredoc = heredoc_body_indices(lines)
|
||||||
|
for idx, line in enumerate(lines):
|
||||||
stripped = line.strip()
|
stripped = line.strip()
|
||||||
|
if idx in in_heredoc:
|
||||||
|
if re.match(r'input\(', stripped):
|
||||||
|
hits.append(stripped)
|
||||||
|
continue
|
||||||
if re.match(r'read(\s|$)', stripped):
|
if re.match(r'read(\s|$)', stripped):
|
||||||
if not stdin_redirected(line, prev_line):
|
if not stdin_redirected(line, prev_line):
|
||||||
hits.append(stripped)
|
hits.append(stripped)
|
||||||
|
|||||||
@@ -67,6 +67,25 @@ EOF
|
|||||||
EOF
|
EOF
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Helper: a sources.md whose Description is wrapped across two physical
|
||||||
|
# lines — the shape check 9's parser used to truncate at the first newline.
|
||||||
|
make_wrapped_sources_md() {
|
||||||
|
local dir="$1"
|
||||||
|
mkdir -p "$dir/references"
|
||||||
|
cat > "$dir/references/sources.md" <<'EOF'
|
||||||
|
# Sources
|
||||||
|
|
||||||
|
## my-source
|
||||||
|
|
||||||
|
- **URL:** https://example.com/my-source
|
||||||
|
- **Description:** A test source, informing the dispatch table's shape with
|
||||||
|
no forge-specific content drawn directly from it beyond that.
|
||||||
|
- **Contributing files:** SKILL.md
|
||||||
|
- **Research doc:** (none)
|
||||||
|
- **Status:** `extracted`
|
||||||
|
EOF
|
||||||
|
}
|
||||||
|
|
||||||
# Helper: turn dir into a real git repo with one commit of its current
|
# Helper: turn dir into a real git repo with one commit of its current
|
||||||
# contents, and a refs/remotes/origin/main pointing at that same commit.
|
# contents, and a refs/remotes/origin/main pointing at that same commit.
|
||||||
# Check 9 diffs the skill's references/sources.md against `git merge-base
|
# Check 9 diffs the skill's references/sources.md against `git merge-base
|
||||||
@@ -1629,23 +1648,247 @@ EOF
|
|||||||
assert_output --partial "Check 9 skipped — no base ref could be resolved"
|
assert_output --partial "Check 9 skipped — no base ref could be resolved"
|
||||||
}
|
}
|
||||||
|
|
||||||
@test "check 9: --base-ref overrides the default origin/main resolution" {
|
@test "check 9: --base-ref overrides a default origin/main that resolves to something else" {
|
||||||
local skill="$TMPDIR/my-skill"
|
local skill="$TMPDIR/my-skill"
|
||||||
make_skill_with_source_keys "$skill"
|
make_skill_with_source_keys "$skill"
|
||||||
make_sources_md "$skill"
|
make_sources_md "$skill"
|
||||||
commit_as_base "$skill"
|
commit_as_base "$skill"
|
||||||
local base_sha
|
local old_sha
|
||||||
base_sha="$(git -C "$skill" rev-parse HEAD)"
|
old_sha="$(git -C "$skill" rev-parse HEAD)"
|
||||||
git -C "$skill" update-ref -d refs/remotes/origin/main >/dev/null 2>&1
|
|
||||||
|
|
||||||
|
# A SECOND commit carrying the rewritten claim, with origin/main moved onto
|
||||||
|
# it. The default base ref therefore resolves — to a commit that matches the
|
||||||
|
# working tree — so a silent default run proves there was a default here to
|
||||||
|
# override. Deleting origin/main instead, as this test used to, proved only
|
||||||
|
# that the flag works when nothing else does.
|
||||||
sed -i 's/^- \*\*Description:\*\* A test source\.$/- **Description:** A rewritten claim./' \
|
sed -i 's/^- \*\*Description:\*\* A test source\.$/- **Description:** A rewritten claim./' \
|
||||||
"$skill/references/sources.md"
|
"$skill/references/sources.md"
|
||||||
|
git -C "$skill" -c user.email=test@example.com -c user.name=test commit -aqm rewrite >/dev/null 2>&1
|
||||||
|
git -C "$skill" update-ref refs/remotes/origin/main HEAD >/dev/null 2>&1
|
||||||
|
|
||||||
run bash "$SCRIPT" "$skill" "--base-ref=$base_sha"
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_success
|
||||||
|
assert_output ""
|
||||||
|
|
||||||
|
run bash "$SCRIPT" "$skill" "--base-ref=$old_sha"
|
||||||
assert_success
|
assert_success
|
||||||
assert_output --partial "'Description' changed for 'my-source'"
|
assert_output --partial "'Description' changed for 'my-source'"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@test "check 9: VALIDATE_PROVENANCE_BASE_REF sets the base ref when no flag is given" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
make_sources_md "$skill"
|
||||||
|
commit_as_base "$skill"
|
||||||
|
local old_sha
|
||||||
|
old_sha="$(git -C "$skill" rev-parse HEAD)"
|
||||||
|
sed -i 's/^- \*\*Description:\*\* A test source\.$/- **Description:** A rewritten claim./' \
|
||||||
|
"$skill/references/sources.md"
|
||||||
|
git -C "$skill" -c user.email=test@example.com -c user.name=test commit -aqm rewrite >/dev/null 2>&1
|
||||||
|
git -C "$skill" update-ref refs/remotes/origin/main HEAD >/dev/null 2>&1
|
||||||
|
|
||||||
|
run env VALIDATE_PROVENANCE_BASE_REF="$old_sha" bash "$SCRIPT" "$skill"
|
||||||
|
assert_success
|
||||||
|
assert_output --partial "'Description' changed for 'my-source'"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "check 9: the --base-ref flag wins over the environment variable" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
make_sources_md "$skill"
|
||||||
|
commit_as_base "$skill"
|
||||||
|
local old_sha
|
||||||
|
old_sha="$(git -C "$skill" rev-parse HEAD)"
|
||||||
|
sed -i 's/^- \*\*Description:\*\* A test source\.$/- **Description:** A rewritten claim./' \
|
||||||
|
"$skill/references/sources.md"
|
||||||
|
git -C "$skill" -c user.email=test@example.com -c user.name=test commit -aqm rewrite >/dev/null 2>&1
|
||||||
|
local new_sha
|
||||||
|
new_sha="$(git -C "$skill" rev-parse HEAD)"
|
||||||
|
|
||||||
|
# The environment names the old commit (which would fire), the flag names
|
||||||
|
# the new one (which would not). The usage text promises the flag wins.
|
||||||
|
run env VALIDATE_PROVENANCE_BASE_REF="$old_sha" bash "$SCRIPT" "$skill" "--base-ref=$new_sha"
|
||||||
|
assert_success
|
||||||
|
assert_output ""
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "check 9: an EMPTY --base-ref is still 'given' and wins over the environment variable" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
make_sources_md "$skill"
|
||||||
|
commit_as_base "$skill"
|
||||||
|
local old_sha
|
||||||
|
old_sha="$(git -C "$skill" rev-parse HEAD)"
|
||||||
|
sed -i 's/^- \*\*Description:\*\* A test source\.$/- **Description:** A rewritten claim./' \
|
||||||
|
"$skill/references/sources.md"
|
||||||
|
git -C "$skill" -c user.email=test@example.com -c user.name=test commit -aqm rewrite >/dev/null 2>&1
|
||||||
|
git -C "$skill" update-ref refs/remotes/origin/main HEAD >/dev/null 2>&1
|
||||||
|
|
||||||
|
# `--base-ref=` selects the DEFAULT resolution (origin/main, which now
|
||||||
|
# matches the working tree), so nothing fires. Under the old `:-` spelling
|
||||||
|
# the empty value read as absent and the environment variable won, firing
|
||||||
|
# the INFO and contradicting the usage text.
|
||||||
|
run env VALIDATE_PROVENANCE_BASE_REF="$old_sha" bash "$SCRIPT" "$skill" "--base-ref="
|
||||||
|
assert_success
|
||||||
|
assert_output ""
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Cycle 29 — Check 9: a WRAPPED field value. The parser compared only the first
|
||||||
|
# physical line, so a rewrite confined to a continuation line — the exact
|
||||||
|
# hedge-to-confident-claim shape #118 exists to catch — produced no finding at
|
||||||
|
# all, while a pure re-wrap produced a false one.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@test "check 9: a rewrite confined to a wrapped Description's CONTINUATION line fires an INFO" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
make_wrapped_sources_md "$skill"
|
||||||
|
commit_as_base "$skill"
|
||||||
|
|
||||||
|
# Hedge to confident claim, on the second physical line only. This is the
|
||||||
|
# regression check 9 was written for, and the one it could not see.
|
||||||
|
sed -i "s|^ no forge-specific content drawn directly from it beyond that\.\$| it grounds Step 2's dispatch table in full.|" \
|
||||||
|
"$skill/references/sources.md"
|
||||||
|
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_success
|
||||||
|
assert_output --partial "'Description' changed for 'my-source'"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "check 9: re-wrapping a Description with no wording change produces no finding" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
make_wrapped_sources_md "$skill"
|
||||||
|
commit_as_base "$skill"
|
||||||
|
|
||||||
|
# Same words, different line breaks. normalize_field_text()'s docstring
|
||||||
|
# promises this is invisible; it was not, because the value was truncated
|
||||||
|
# at its first newline before the whitespace collapse ever ran.
|
||||||
|
python3 - "$skill/references/sources.md" <<'PY'
|
||||||
|
import sys
|
||||||
|
path = sys.argv[1]
|
||||||
|
text = open(path).read()
|
||||||
|
old = ("- **Description:** A test source, informing the dispatch table's shape with\n"
|
||||||
|
" no forge-specific content drawn directly from it beyond that.")
|
||||||
|
new = ("- **Description:** A test source, informing the dispatch\n"
|
||||||
|
" table's shape with no forge-specific content drawn\n"
|
||||||
|
" directly from it beyond that.")
|
||||||
|
assert old in text
|
||||||
|
open(path, 'w').write(text.replace(old, new))
|
||||||
|
PY
|
||||||
|
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_success
|
||||||
|
assert_output ""
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "check 9: the bullet form of Contributing files is compared, not skipped" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
mkdir -p "$skill/references"
|
||||||
|
cat > "$skill/references/sources.md" <<'EOF'
|
||||||
|
# Sources
|
||||||
|
|
||||||
|
## my-source
|
||||||
|
|
||||||
|
- **URL:** https://example.com/my-source
|
||||||
|
- **Description:** A test source.
|
||||||
|
|
||||||
|
**Contributing files:**
|
||||||
|
- SKILL.md (the dispatch table)
|
||||||
|
- references/other.md (the rubric)
|
||||||
|
|
||||||
|
- **Research doc:** (none)
|
||||||
|
- **Status:** `extracted`
|
||||||
|
EOF
|
||||||
|
printf -- '---\nsource_keys:\n - my-source\n---\n\nnotes\n' > "$skill/references/other.md"
|
||||||
|
commit_as_base "$skill"
|
||||||
|
|
||||||
|
# The change is in the SECOND bullet. The old loop joined bullets in
|
||||||
|
# document order too, but broke on any wrapped one — and no test covered
|
||||||
|
# this branch at all, both existing check-9 tests using the inline form.
|
||||||
|
sed -i 's|^- references/other.md (the rubric)$|- references/other.md (the whole rubric, verbatim)|' \
|
||||||
|
"$skill/references/sources.md"
|
||||||
|
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_success
|
||||||
|
assert_output --partial "'Contributing files' changed for 'my-source'"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "check 9: a wrapped bullet does not silently drop the bullets after it" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
mkdir -p "$skill/references"
|
||||||
|
cat > "$skill/references/sources.md" <<'EOF'
|
||||||
|
# Sources
|
||||||
|
|
||||||
|
## my-source
|
||||||
|
|
||||||
|
- **URL:** https://example.com/my-source
|
||||||
|
- **Description:** A test source.
|
||||||
|
|
||||||
|
**Contributing files:**
|
||||||
|
- SKILL.md (the dispatch table, and the gates
|
||||||
|
common to every branch of it)
|
||||||
|
- references/other.md (the rubric)
|
||||||
|
|
||||||
|
- **Research doc:** (none)
|
||||||
|
- **Status:** `extracted`
|
||||||
|
EOF
|
||||||
|
printf -- '---\nsource_keys:\n - my-source\n---\n\nnotes\n' > "$skill/references/other.md"
|
||||||
|
commit_as_base "$skill"
|
||||||
|
|
||||||
|
# The old loop broke at the wrapped continuation line, so everything from
|
||||||
|
# here down was never part of the compared value — a change to the last
|
||||||
|
# bullet was invisible.
|
||||||
|
sed -i 's|^- references/other.md (the rubric)$|- references/other.md (rewritten claim)|' \
|
||||||
|
"$skill/references/sources.md"
|
||||||
|
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_success
|
||||||
|
assert_output --partial "'Contributing files' changed for 'my-source'"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "check 9: a field present at the base ref and deleted since is announced" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
make_sources_md "$skill"
|
||||||
|
commit_as_base "$skill"
|
||||||
|
|
||||||
|
# No other check in this script requires a Description, so a deleted one
|
||||||
|
# used to leave no finding anywhere: a claim could be withdrawn as
|
||||||
|
# invisibly as it could be strengthened.
|
||||||
|
sed -i '/^- \*\*Description:\*\*/d' "$skill/references/sources.md"
|
||||||
|
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_success
|
||||||
|
assert_output --partial "'Description' removed for 'my-source'"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "check 9: a sources.md untracked at the base ref is announced, not silently skipped" {
|
||||||
|
local repo="$TMPDIR/repo"
|
||||||
|
local skill="$repo/tracked-skill"
|
||||||
|
mkdir -p "$skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
make_sources_md "$skill"
|
||||||
|
commit_as_base "$repo"
|
||||||
|
|
||||||
|
# A copy of the same skill at a path git does not know — the everyday case
|
||||||
|
# being an installed, gitignored .claude/skills/ tree. The base ref
|
||||||
|
# resolves fine; `git show <ref>:<path>` does not. Treating that as
|
||||||
|
# "creation, nothing to flag" made the whole check vanish without a word,
|
||||||
|
# so the same directory reported findings at one path and silence at the
|
||||||
|
# other.
|
||||||
|
cp -r "$skill" "$repo/untracked-copy"
|
||||||
|
|
||||||
|
run bash "$SCRIPT" "$repo/untracked-copy"
|
||||||
|
assert_success
|
||||||
|
assert_output --partial "INFO"
|
||||||
|
assert_output --partial "is not tracked at"
|
||||||
|
assert_output --partial "Check 9 did not run for any slug in this skill."
|
||||||
|
}
|
||||||
|
|
||||||
@test "check 9: an invalid --base-ref value is reported as unresolvable, not a crash" {
|
@test "check 9: an invalid --base-ref value is reported as unresolvable, not a crash" {
|
||||||
local skill="$TMPDIR/my-skill"
|
local skill="$TMPDIR/my-skill"
|
||||||
make_skill_with_source_keys "$skill"
|
make_skill_with_source_keys "$skill"
|
||||||
|
|||||||
@@ -16,6 +16,10 @@ setup() {
|
|||||||
# SUGGESTION-freedom would be asserting the boundary check's absence instead
|
# SUGGESTION-freedom would be asserting the boundary check's absence instead
|
||||||
# of the thing it names. "anything else" is not hyphenated, so the clause adds
|
# of the thing it names. "anything else" is not hyphenated, so the clause adds
|
||||||
# a boundary marker without adding a routing target to resolve.
|
# a boundary marker without adding a routing target to resolve.
|
||||||
|
#
|
||||||
|
# metadata.version is equally load-bearing: ADR-0022 makes it mandatory and
|
||||||
|
# validate.sh FAILs without it, so a fixture omitting it would not be
|
||||||
|
# "otherwise clean" either.
|
||||||
make_valid_skill() {
|
make_valid_skill() {
|
||||||
local dir="$1"
|
local dir="$1"
|
||||||
local name
|
local name
|
||||||
@@ -25,6 +29,8 @@ setup() {
|
|||||||
---
|
---
|
||||||
name: $name
|
name: $name
|
||||||
description: A valid skill description that is well within the limit. Do not use for anything else.
|
description: A valid skill description that is well within the limit. Do not use for anything else.
|
||||||
|
metadata:
|
||||||
|
version: "1.0.0"
|
||||||
---
|
---
|
||||||
|
|
||||||
## Step 1
|
## Step 1
|
||||||
@@ -59,6 +65,8 @@ PY
|
|||||||
echo "---"
|
echo "---"
|
||||||
echo "name: $name"
|
echo "name: $name"
|
||||||
echo "description: $desc"
|
echo "description: $desc"
|
||||||
|
echo "metadata:"
|
||||||
|
echo ' version: "1.0.0"'
|
||||||
echo "---"
|
echo "---"
|
||||||
echo ""
|
echo ""
|
||||||
python3 -c "print(' '.join(['word'] * $body_words))"
|
python3 -c "print(' '.join(['word'] * $body_words))"
|
||||||
@@ -294,6 +302,141 @@ EOF
|
|||||||
assert_success
|
assert_success
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@test "prose inside a usage() here-doc that wraps onto a line starting with 'read' does not fail" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_valid_skill "$skill"
|
||||||
|
# The exact shape that made skill-audit hard-FAIL on its own
|
||||||
|
# validate-provenance.sh: a usage() heredoc whose wrapped sentence puts the
|
||||||
|
# English verb "read" in column 0.
|
||||||
|
cat > "$skill/scripts/helper.sh" <<'SH'
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
usage() {
|
||||||
|
cat <<EOF
|
||||||
|
Checks performed:
|
||||||
|
4 A Contributing files block this parser cannot
|
||||||
|
read is reported as an INFO, never skipped silently.
|
||||||
|
EOF
|
||||||
|
}
|
||||||
|
usage
|
||||||
|
SH
|
||||||
|
chmod +x "$skill/scripts/helper.sh"
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_success
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a real interactive read AFTER a here-doc is still caught" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_valid_skill "$skill"
|
||||||
|
# Pins that the here-doc exemption ends at its terminator. A body skip that
|
||||||
|
# ran to end-of-file would swallow this read and report the script clean.
|
||||||
|
cat > "$skill/scripts/helper.sh" <<'SH'
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
cat <<EOF
|
||||||
|
read this text
|
||||||
|
EOF
|
||||||
|
read -r ANSWER
|
||||||
|
SH
|
||||||
|
chmod +x "$skill/scripts/helper.sh"
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_failure
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an unterminated here-doc opener does not disarm the check for the rest of the file" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_valid_skill "$skill"
|
||||||
|
# `<<` here is inside a string, not an opener. Treating it as one would skip
|
||||||
|
# every following line — a false negative, the direction this check must
|
||||||
|
# never fail in.
|
||||||
|
cat > "$skill/scripts/helper.sh" <<'SH'
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
echo "shift left with a << b"
|
||||||
|
read -r ANSWER
|
||||||
|
SH
|
||||||
|
chmod +x "$skill/scripts/helper.sh"
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_failure
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a bare input() inside an embedded-python here-doc is still caught" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_valid_skill "$skill"
|
||||||
|
# The here-doc exemption is for the `read` heuristic only: these scripts
|
||||||
|
# embed Python in a here-doc as a matter of course, so exempting the body
|
||||||
|
# wholesale would disarm the check across the corpus.
|
||||||
|
cat > "$skill/scripts/helper.sh" <<'SH'
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
python3 - <<'PY'
|
||||||
|
print("press enter")
|
||||||
|
input()
|
||||||
|
PY
|
||||||
|
SH
|
||||||
|
chmod +x "$skill/scripts/helper.sh"
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_failure
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# ADR-0022 — metadata.version is mandatory. FAIL tier, matching the
|
||||||
|
# skill-frontmatter pre-commit hook: an audit that graded this lower would
|
||||||
|
# report ready-to-ship on a file the commit gate rejects.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@test "ADR-0022: a SKILL.md with no metadata block at all FAILs" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_valid_skill "$skill"
|
||||||
|
python3 - "$skill/SKILL.md" <<'PY'
|
||||||
|
import sys
|
||||||
|
p = sys.argv[1]
|
||||||
|
s = open(p).read().replace('metadata:\n version: "1.0.0"\n', '')
|
||||||
|
open(p, 'w').write(s)
|
||||||
|
PY
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_failure
|
||||||
|
assert_output --partial "metadata.version"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "ADR-0022: a metadata block with no version key FAILs" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_valid_skill "$skill"
|
||||||
|
python3 - "$skill/SKILL.md" <<'PY'
|
||||||
|
import sys
|
||||||
|
p = sys.argv[1]
|
||||||
|
s = open(p).read().replace(' version: "1.0.0"\n', ' category: factory\n')
|
||||||
|
open(p, 'w').write(s)
|
||||||
|
PY
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_failure
|
||||||
|
assert_output --partial "metadata.version"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "ADR-0022: a two-part metadata.version FAILs as malformed, not passes as present" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_valid_skill "$skill"
|
||||||
|
python3 - "$skill/SKILL.md" <<'PY'
|
||||||
|
import sys
|
||||||
|
p = sys.argv[1]
|
||||||
|
s = open(p).read().replace(' version: "1.0.0"\n', ' version: 1.0\n')
|
||||||
|
open(p, 'w').write(s)
|
||||||
|
PY
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_failure
|
||||||
|
assert_output --partial "three-part semver"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "ADR-0022: an unquoted three-part metadata.version passes" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_valid_skill "$skill"
|
||||||
|
python3 - "$skill/SKILL.md" <<'PY'
|
||||||
|
import sys
|
||||||
|
p = sys.argv[1]
|
||||||
|
s = open(p).read().replace(' version: "1.0.0"\n', ' version: 0.1.3\n')
|
||||||
|
open(p, 'w').write(s)
|
||||||
|
PY
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_success
|
||||||
|
assert_output --partial "metadata.version present: '0.1.3'"
|
||||||
|
}
|
||||||
|
|
||||||
@test "fails when name contains consecutive hyphens" {
|
@test "fails when name contains consecutive hyphens" {
|
||||||
local skill="$TMPDIR/my--skill"
|
local skill="$TMPDIR/my--skill"
|
||||||
make_valid_skill "$skill"
|
make_valid_skill "$skill"
|
||||||
@@ -608,6 +751,8 @@ make_hand_invoked_skill() {
|
|||||||
echo "name: $name"
|
echo "name: $name"
|
||||||
echo "description: $desc"
|
echo "description: $desc"
|
||||||
echo "disable-model-invocation: true"
|
echo "disable-model-invocation: true"
|
||||||
|
echo "metadata:"
|
||||||
|
echo ' version: "1.0.0"'
|
||||||
echo "---"
|
echo "---"
|
||||||
echo ""
|
echo ""
|
||||||
python3 -c "print(' '.join(['word'] * $body_words))"
|
python3 -c "print(' '.join(['word'] * $body_words))"
|
||||||
@@ -761,6 +906,8 @@ make_hand_invoked_skill() {
|
|||||||
---
|
---
|
||||||
name: locale-skill
|
name: locale-skill
|
||||||
description: A valid skill description that is well within the limit.
|
description: A valid skill description that is well within the limit.
|
||||||
|
metadata:
|
||||||
|
version: "1.0.0"
|
||||||
---
|
---
|
||||||
|
|
||||||
## Step 1
|
## Step 1
|
||||||
|
|||||||
@@ -224,6 +224,8 @@ cat > "$SUBJECT_SKILL_DIR/SKILL.md" <<'EOF'
|
|||||||
---
|
---
|
||||||
name: my-skill
|
name: my-skill
|
||||||
description: A short valid description. Do not use for anything else.
|
description: A short valid description. Do not use for anything else.
|
||||||
|
metadata:
|
||||||
|
version: "1.0.0"
|
||||||
---
|
---
|
||||||
|
|
||||||
Do the thing.
|
Do the thing.
|
||||||
|
|||||||
Reference in New Issue
Block a user