Compare commits
5 Commits
c048d2320e
...
8dc5241c1c
| Author | SHA1 | Date | |
|---|---|---|---|
| 8dc5241c1c | |||
| 562527dfc4 | |||
| 229c7a4ab9 | |||
| 02b816cbb0 | |||
| 79da149935 |
@@ -95,9 +95,15 @@ Field (`source:`) in a skill's `META.md` tracking upstream provenance. An array
|
|||||||
<!-- ### META.md
|
<!-- ### META.md
|
||||||
A per-skill markdown file containing a single YAML code block with provenance and audit fields: `version`, `updated`, `when`, `source`, and `references`. Lives alongside the SKILL.md in the skill directory (either `.agents/skills/<name>/META.md` or `plugins/<plugin>/skills/<name>/META.md`). Not loaded at agent startup — progressive disclosure principle: name and description route the skill; provenance is only needed for upgrade reviews and audits. Prevents these fields from being scanned on every session start alongside every skill's name and description. The authoritative schema is `META-TEMPLATE.md` in `plugins/kyberforge/skills/write-skill/`. See also: [[Source field]]. -->
|
A per-skill markdown file containing a single YAML code block with provenance and audit fields: `version`, `updated`, `when`, `source`, and `references`. Lives alongside the SKILL.md in the skill directory (either `.agents/skills/<name>/META.md` or `plugins/<plugin>/skills/<name>/META.md`). Not loaded at agent startup — progressive disclosure principle: name and description route the skill; provenance is only needed for upgrade reviews and audits. Prevents these fields from being scanned on every session start alongside every skill's name and description. The authoritative schema is `META-TEMPLATE.md` in `plugins/kyberforge/skills/write-skill/`. See also: [[Source field]]. -->
|
||||||
|
|
||||||
|
### INFO (finding level)
|
||||||
|
A third finding level in `skill-audit` reports, below SUGGESTION. Observational — flags something worth noting that is not actionable and does not imply a defect. A skill with only INFO findings is a clean PASS. Counted separately in the result block as `· P info` and never mixed into the FAIL or SUGGESTION counts. Current use: a `references/*.md` file with no `source_keys` when `sources.md` is present; a skill-level source slug not found in the upstream research sources. See ADR-0014.
|
||||||
|
|
||||||
### Provider-agnostic issue tracker
|
### Provider-agnostic issue tracker
|
||||||
Skills and workflows reference "linked issue" generically rather than a specific provider. In the file-based phase, an issue is a `docs/issues/NNNN-<slug>.md` file. When Gitea MCP is configured, the same skills use it instead. The active backend is determined at runtime by MCP availability. "Issue" is the canonical cross-provider term (GitHub, GitLab, Gitea all use it).
|
Skills and workflows reference "linked issue" generically rather than a specific provider. In the file-based phase, an issue is a `docs/issues/NNNN-<slug>.md` file. When Gitea MCP is configured, the same skills use it instead. The active backend is determined at runtime by MCP availability. "Issue" is the canonical cross-provider term (GitHub, GitLab, Gitea all use it).
|
||||||
|
|
||||||
|
### Provenance chain
|
||||||
|
The three-stage traceability record linking a skill back to its research inputs: (1) `/research` produces topic docs and a `sources.md` in `plugins/<plugin>/docs/research/docs/<topic>/`; (2) `/skill-author` reads those docs and records which sources informed which skill files in `references/sources.md` (including a `Research doc:` back-pointer to the upstream research file) and `source_keys` frontmatter on `SKILL.md` and `references/*.md`; (3) `skill-audit` validates the chain is complete and internally consistent via `validate-provenance.sh`. A skill with research input but no `references/sources.md`, or with `source_keys` that don't match `references/sources.md` slugs, has a broken provenance chain.
|
||||||
|
|
||||||
### PRD scope
|
### PRD scope
|
||||||
A PRD contains: problem statement, goals, explicit non-goals, functional requirements at feature level, success criteria. Never contains: technical approach, implementation steps, or EARS-level detail. HOW is handled downstream: workstream-level technical approach belongs in `architecture-review` (≥2 options, tradeoffs, optional step after `write-prd`); issue-level HOW belongs in issue design notes. Prerequisite: a completed grill session. Validated by inline self-checks in the `write-prd` skill.
|
A PRD contains: problem statement, goals, explicit non-goals, functional requirements at feature level, success criteria. Never contains: technical approach, implementation steps, or EARS-level detail. HOW is handled downstream: workstream-level technical approach belongs in `architecture-review` (≥2 options, tradeoffs, optional step after `write-prd`); issue-level HOW belongs in issue design notes. Prerequisite: a completed grill session. Validated by inline self-checks in the `write-prd` skill.
|
||||||
|
|
||||||
|
|||||||
32
docs/adr/0014-skill-audit-info-finding-level.md
Normal file
32
docs/adr/0014-skill-audit-info-finding-level.md
Normal file
@@ -0,0 +1,32 @@
|
|||||||
|
# Add INFO as a third finding level in skill-audit reports
|
||||||
|
|
||||||
|
`skill-audit` shipped with two finding levels: FAIL (blocks shipping) and
|
||||||
|
SUGGESTION (optional improvement). Provenance validation introduced observations
|
||||||
|
that are worth surfacing but not actionable: a `references/*.md` file with no
|
||||||
|
`source_keys` when `sources.md` is present, and a skill-level source slug absent
|
||||||
|
from upstream research docs. Folding these into SUGGESTION would imply they
|
||||||
|
should be fixed — but retroactive source backfill after a reference file is
|
||||||
|
written is unreliable and not expected practice. A third level, INFO, is therefore
|
||||||
|
introduced: observational, no action implied, never changes the pass/fail verdict.
|
||||||
|
Counted separately in the result block as `· P info`.
|
||||||
|
|
||||||
|
## Considered options
|
||||||
|
|
||||||
|
**SUGGESTION with softer language (rejected)** — describe the finding as "worth
|
||||||
|
noting" rather than "should be fixed." Rejected because SUGGESTION already carries
|
||||||
|
an established meaning in the report; softening the language creates ambiguity
|
||||||
|
without changing the semantic level. Downstream consumers (humans, skill-improve)
|
||||||
|
would need to infer intent from prose rather than a stable token.
|
||||||
|
|
||||||
|
**Suppress entirely (rejected)** — omit findings that have no fix. Rejected
|
||||||
|
because the observations are useful for a human reviewing provenance completeness.
|
||||||
|
Silent omission loses information without reducing noise.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- Report format gains a third token: FAIL, SUGGESTION, INFO. INFO findings do not
|
||||||
|
affect pass/fail; counted as `· P info` in the result block.
|
||||||
|
- `skill-improve` currently ignores anything below FAIL — that behavior remains
|
||||||
|
correct; INFO findings are not forwarded to it.
|
||||||
|
- Future soft observations should use INFO rather than SUGGESTION when no fix is
|
||||||
|
actionable.
|
||||||
@@ -4,7 +4,7 @@ Audit a skill directory against the agentskills.io specification. Runs structura
|
|||||||
|
|
||||||
## What it does
|
## What it does
|
||||||
|
|
||||||
1. Runs `scripts/validate.sh` for structural checks (name format, description length, line count, placeholder detection, script rules)
|
1. Runs `scripts/validate.sh` and `scripts/validate-provenance.sh` for structural and provenance checks
|
||||||
2. Reads all files in the skill directory
|
2. Reads all files in the skill directory
|
||||||
3. Applies qualitative checks across seven dimensions
|
3. Applies qualitative checks across seven dimensions
|
||||||
4. Outputs a compact findings report — findings only, grouped by dimension, each with Why and Fix — and a result block with handoff to /skill-improve
|
4. Outputs a compact findings report — findings only, grouped by dimension, each with Why and Fix — and a result block with handoff to /skill-improve
|
||||||
@@ -23,8 +23,10 @@ Provide the path to the skill directory to audit when invoking.
|
|||||||
|------|---------|
|
|------|---------|
|
||||||
| `SKILL.md` | Skill instructions for agents |
|
| `SKILL.md` | Skill instructions for agents |
|
||||||
| `scripts/validate.sh` | Structural validator — checks name format, name matches directory, description length, line count, placeholder detection, script executable bit, and interactive-prompt detection |
|
| `scripts/validate.sh` | Structural validator — checks name format, name matches directory, description length, line count, placeholder detection, script executable bit, and interactive-prompt detection |
|
||||||
|
| `scripts/validate-provenance.sh` | Provenance validator — checks sources.md completeness, source_keys/slug consistency, Contributing files existence, bidirectional linkage, Research doc: fields, and upstream research doc alignment |
|
||||||
| `references/description-quality.md` | Spec-grounded rubric for description auditing — loaded when a finding is borderline |
|
| `references/description-quality.md` | Spec-grounded rubric for description auditing — loaded when a finding is borderline |
|
||||||
| `references/body-discipline.md` | Spec-grounded rubric for body discipline auditing — loaded when padding vs necessity is unclear |
|
| `references/body-discipline.md` | Spec-grounded rubric for body discipline auditing — loaded when padding vs necessity is unclear |
|
||||||
| `references/sources.md` | Provenance record — agentskills.io sources that informed this skill and which files each contributed to |
|
| `references/sources.md` | Provenance record — agentskills.io sources that informed this skill and which files each contributed to |
|
||||||
| `tests/validate.bats` | Bats test suite for validate.sh |
|
| `tests/validate.bats` | Bats test suite for validate.sh |
|
||||||
|
| `tests/validate-provenance.bats` | Bats test suite for validate-provenance.sh |
|
||||||
| `tests/README.md` | Setup instructions for bats-support and bats-assert test dependencies |
|
| `tests/README.md` | Setup instructions for bats-support and bats-assert test dependencies |
|
||||||
|
|||||||
@@ -30,10 +30,13 @@ metadata:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
bash scripts/validate.sh <skill-dir>
|
bash scripts/validate.sh <skill-dir>
|
||||||
|
bash scripts/validate-provenance.sh <skill-dir>
|
||||||
```
|
```
|
||||||
|
|
||||||
Note any structural FAILs — they will appear in the report as a `### Structure` dimension. If the script cannot execute (python3 unavailable, Bash denied, or permission error), perform structural checks manually: name format, name matches directory, description length ≤1024 chars, SKILL.md ≤500 lines, no unfilled `FILL IN:` placeholders, scripts executable and free of interactive prompts.
|
Note any structural FAILs — they will appear in the report as a `### Structure` dimension. If the script cannot execute (python3 unavailable, Bash denied, or permission error), perform structural checks manually: name format, name matches directory, description length ≤1024 chars, SKILL.md ≤500 lines, no unfilled `FILL IN:` placeholders, scripts executable and free of interactive prompts.
|
||||||
|
|
||||||
|
Note any Provenance FAILs and INFO findings from `validate-provenance.sh` — they surface in the report as a `### Provenance` dimension (separate from `### Structure`). The script embeds full FAIL/INFO format with Why and Fix per finding; surface them verbatim.
|
||||||
|
|
||||||
## Step 2 — Read all skill files
|
## Step 2 — Read all skill files
|
||||||
|
|
||||||
Read every file in the skill directory: `SKILL.md`, `README.md` (if present), all files in `scripts/`, `references/`, `assets/`, and `tests/`. Skip binary files only. Do not skip text files — internal consistency checks require the full picture.
|
Read every file in the skill directory: `SKILL.md`, `README.md` (if present), all files in `scripts/`, `references/`, `assets/`, and `tests/`. Skip binary files only. Do not skip text files — internal consistency checks require the full picture.
|
||||||
@@ -109,7 +112,7 @@ Check each pattern is appropriate and correctly formed:
|
|||||||
Open with a coverage line listing every dimension checked:
|
Open with a coverage line listing every dimension checked:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
Checked: structure · description · body-discipline · patterns · file-structure · formatting · scripts · internal-consistency
|
Checked: structure · description · body-discipline · patterns · file-structure · formatting · scripts · internal-consistency · provenance
|
||||||
```
|
```
|
||||||
|
|
||||||
Then output only dimensions that have findings, grouped under H3 headings, FAILs before SUGGESTIONs within each dimension. Omit clean dimensions entirely — their absence confirms they passed.
|
Then output only dimensions that have findings, grouped under H3 headings, FAILs before SUGGESTIONs within each dimension. Omit clean dimensions entirely — their absence confirms they passed.
|
||||||
@@ -127,8 +130,13 @@ Close with a result block:
|
|||||||
```text
|
```text
|
||||||
## Result
|
## Result
|
||||||
|
|
||||||
PASS / PASS (N suggestions) / FAIL (N fails · M suggestions)
|
PASS
|
||||||
|
PASS (N suggestions)
|
||||||
|
PASS · P info
|
||||||
|
PASS (N suggestions) · P info
|
||||||
|
FAIL (N fails · M suggestions)
|
||||||
|
FAIL (N fails · M suggestions) · P info
|
||||||
Run /skill-improve to address findings.
|
Run /skill-improve to address findings.
|
||||||
```
|
```
|
||||||
|
|
||||||
Omit the `/skill-improve` line when there are no findings. Do not apply fixes — report and propose only.
|
INFO findings are observational — do not affect PASS/FAIL. Omit `· P info` when there are no INFO findings. Omit the `/skill-improve` line when there are no findings at all. Do not apply fixes — report and propose only.
|
||||||
|
|||||||
@@ -5,6 +5,7 @@
|
|||||||
## agentskills-home
|
## agentskills-home
|
||||||
|
|
||||||
- **URL:** https://agentskills.io/home.md
|
- **URL:** https://agentskills.io/home.md
|
||||||
|
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||||
- **Description:** Agent Skills overview — what it is, why it exists, progressive disclosure model, ecosystem of 35+ implementing tools
|
- **Description:** Agent Skills overview — what it is, why it exists, progressive disclosure model, ecosystem of 35+ implementing tools
|
||||||
- **Contributing files:** SKILL.md
|
- **Contributing files:** SKILL.md
|
||||||
- **Status:** `extracted`
|
- **Status:** `extracted`
|
||||||
@@ -12,6 +13,7 @@
|
|||||||
## agentskills-spec
|
## agentskills-spec
|
||||||
|
|
||||||
- **URL:** https://agentskills.io/specification.md
|
- **URL:** https://agentskills.io/specification.md
|
||||||
|
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||||
- **Description:** Complete SKILL.md format specification — frontmatter fields, constraints, body content, optional directories, progressive disclosure levels, file references, validation
|
- **Description:** Complete SKILL.md format specification — frontmatter fields, constraints, body content, optional directories, progressive disclosure levels, file references, validation
|
||||||
- **Contributing files:** SKILL.md, references/body-discipline.md, references/description-quality.md
|
- **Contributing files:** SKILL.md, references/body-discipline.md, references/description-quality.md
|
||||||
- **Status:** `extracted`
|
- **Status:** `extracted`
|
||||||
@@ -19,6 +21,7 @@
|
|||||||
## agentskills-best-practices
|
## agentskills-best-practices
|
||||||
|
|
||||||
- **URL:** https://agentskills.io/skill-creation/best-practices.md
|
- **URL:** https://agentskills.io/skill-creation/best-practices.md
|
||||||
|
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||||
- **Description:** Best practices for skill creators — starting from real expertise, spending context wisely, calibrating control, instruction patterns (gotchas, templates, checklists, validation loops)
|
- **Description:** Best practices for skill creators — starting from real expertise, spending context wisely, calibrating control, instruction patterns (gotchas, templates, checklists, validation loops)
|
||||||
- **Contributing files:** SKILL.md, references/body-discipline.md
|
- **Contributing files:** SKILL.md, references/body-discipline.md
|
||||||
- **Status:** `extracted`
|
- **Status:** `extracted`
|
||||||
@@ -26,6 +29,7 @@
|
|||||||
## agentskills-optimizing-descriptions
|
## agentskills-optimizing-descriptions
|
||||||
|
|
||||||
- **URL:** https://agentskills.io/skill-creation/optimizing-descriptions.md
|
- **URL:** https://agentskills.io/skill-creation/optimizing-descriptions.md
|
||||||
|
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||||
- **Description:** How to systematically test and improve skill descriptions for triggering accuracy — eval queries, trigger rate testing, train/validation splits, optimization loop
|
- **Description:** How to systematically test and improve skill descriptions for triggering accuracy — eval queries, trigger rate testing, train/validation splits, optimization loop
|
||||||
- **Contributing files:** SKILL.md, references/description-quality.md
|
- **Contributing files:** SKILL.md, references/description-quality.md
|
||||||
- **Status:** `extracted`
|
- **Status:** `extracted`
|
||||||
@@ -33,6 +37,7 @@
|
|||||||
## agentskills-evaluating-skills
|
## agentskills-evaluating-skills
|
||||||
|
|
||||||
- **URL:** https://agentskills.io/skill-creation/evaluating-skills.md
|
- **URL:** https://agentskills.io/skill-creation/evaluating-skills.md
|
||||||
|
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||||
- **Description:** Eval-driven skill quality improvement — test case design, workspace structure, assertion writing, grading, benchmarking, human review, iteration loop
|
- **Description:** Eval-driven skill quality improvement — test case design, workspace structure, assertion writing, grading, benchmarking, human review, iteration loop
|
||||||
- **Contributing files:** (none — eval workflow not directly informing audit dimensions)
|
- **Contributing files:** (none — eval workflow not directly informing audit dimensions)
|
||||||
- **Status:** `extracted`
|
- **Status:** `extracted`
|
||||||
@@ -40,6 +45,7 @@
|
|||||||
## agentskills-using-scripts
|
## agentskills-using-scripts
|
||||||
|
|
||||||
- **URL:** https://agentskills.io/skill-creation/using-scripts.md
|
- **URL:** https://agentskills.io/skill-creation/using-scripts.md
|
||||||
|
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||||
- **Description:** Using scripts in skills — one-off commands, self-contained scripts with inline dependencies, designing scripts for agentic use (no interactive prompts, --help, structured output, idempotency)
|
- **Description:** Using scripts in skills — one-off commands, self-contained scripts with inline dependencies, designing scripts for agentic use (no interactive prompts, --help, structured output, idempotency)
|
||||||
- **Contributing files:** SKILL.md
|
- **Contributing files:** SKILL.md
|
||||||
- **Status:** `extracted`
|
- **Status:** `extracted`
|
||||||
@@ -47,6 +53,7 @@
|
|||||||
## agentskills-quickstart
|
## agentskills-quickstart
|
||||||
|
|
||||||
- **URL:** https://agentskills.io/skill-creation/quickstart.md
|
- **URL:** https://agentskills.io/skill-creation/quickstart.md
|
||||||
|
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||||
- **Description:** Step-by-step guide to creating a first skill (roll-dice example), how discovery/activation/execution work in practice
|
- **Description:** Step-by-step guide to creating a first skill (roll-dice example), how discovery/activation/execution work in practice
|
||||||
- **Contributing files:** (none — creation guide not directly informing audit criteria)
|
- **Contributing files:** (none — creation guide not directly informing audit criteria)
|
||||||
- **Status:** `extracted`
|
- **Status:** `extracted`
|
||||||
|
|||||||
395
plugins/kyberforge/skills/skill-audit/scripts/validate-provenance.sh
Executable file
395
plugins/kyberforge/skills/skill-audit/scripts/validate-provenance.sh
Executable file
@@ -0,0 +1,395 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
cat <<EOF
|
||||||
|
Usage: validate-provenance.sh <skill-dir>
|
||||||
|
|
||||||
|
Validate that a skill's sources provenance chain is complete and internally consistent.
|
||||||
|
|
||||||
|
Arguments:
|
||||||
|
skill-dir Path to the skill directory to validate.
|
||||||
|
|
||||||
|
Exit codes:
|
||||||
|
0 All checks passed (or nothing to validate)
|
||||||
|
1 One or more checks failed
|
||||||
|
|
||||||
|
Checks performed:
|
||||||
|
0 source_keys present but references/sources.md absent
|
||||||
|
1 FILL IN: placeholders in sources.md
|
||||||
|
2 source_keys in SKILL.md → slug exists in sources.md
|
||||||
|
3 source_keys in references/*.md → slug exists in sources.md (INFO if no source_keys)
|
||||||
|
4 Contributing files listed in sources.md exist on disk
|
||||||
|
5 Contributing files back-reference the parent slug in their source_keys
|
||||||
|
6 Research doc field present and not placeholder
|
||||||
|
7 Slug in sources.md present in upstream research doc (INFO only)
|
||||||
|
8 Extracted non-(none) slug in research doc present in sources.md
|
||||||
|
EOF
|
||||||
|
}
|
||||||
|
|
||||||
|
if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then
|
||||||
|
usage
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ $# -lt 1 ]]; then
|
||||||
|
echo "Error: skill-dir is required." >&2
|
||||||
|
echo "" >&2
|
||||||
|
usage >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
python3 -u - "$1" <<'PYTHON'
|
||||||
|
import sys
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
|
||||||
|
skill_dir = os.path.abspath(sys.argv[1])
|
||||||
|
sources_md_path = os.path.join(skill_dir, "references", "sources.md")
|
||||||
|
refs_dir = os.path.join(skill_dir, "references")
|
||||||
|
|
||||||
|
# --- Helpers ---
|
||||||
|
|
||||||
|
PLACEHOLDER_RE = re.compile(r'(?<!`)FILL IN:[^`\n]')
|
||||||
|
|
||||||
|
def parse_frontmatter(content):
|
||||||
|
"""Return (frontmatter_str, body_str) or (None, content) if no frontmatter."""
|
||||||
|
m = re.match(r'^---\n(.*?)\n---', content, re.DOTALL)
|
||||||
|
if not m:
|
||||||
|
return None, content
|
||||||
|
return m.group(1), content[m.end():]
|
||||||
|
|
||||||
|
def parse_source_keys(fm):
|
||||||
|
"""Extract list of source_keys from frontmatter string (handles metadata.source_keys and top-level)."""
|
||||||
|
if fm is None:
|
||||||
|
return []
|
||||||
|
keys = []
|
||||||
|
# Match either:
|
||||||
|
# metadata:\n source_keys:\n - slug
|
||||||
|
# or:
|
||||||
|
# source_keys:\n - slug
|
||||||
|
in_source_keys = False
|
||||||
|
in_metadata = False
|
||||||
|
for line in fm.splitlines():
|
||||||
|
if re.match(r'^metadata:', line):
|
||||||
|
in_metadata = True
|
||||||
|
continue
|
||||||
|
if in_metadata and re.match(r'^ source_keys:', line):
|
||||||
|
in_source_keys = True
|
||||||
|
continue
|
||||||
|
if not in_metadata and re.match(r'^source_keys:', line):
|
||||||
|
in_source_keys = True
|
||||||
|
continue
|
||||||
|
if in_source_keys:
|
||||||
|
m = re.match(r'^[ \t]+-\s+(\S+)', line)
|
||||||
|
if m:
|
||||||
|
keys.append(m.group(1).strip())
|
||||||
|
elif line and not line[0].isspace():
|
||||||
|
in_source_keys = False
|
||||||
|
in_metadata = False
|
||||||
|
return keys
|
||||||
|
|
||||||
|
def parse_h2_slugs(content):
|
||||||
|
"""Return list of H2 heading values from a markdown file."""
|
||||||
|
return re.findall(r'^## (.+)$', content, re.MULTILINE)
|
||||||
|
|
||||||
|
def parse_contributing_files(content, slug):
|
||||||
|
"""Find the Contributing files value for a given slug H2 in content."""
|
||||||
|
# Find the H2 block for slug, then look for Contributing files line
|
||||||
|
pattern = re.compile(
|
||||||
|
r'^## ' + re.escape(slug) + r'\s*\n(.*?)(?=^## |\Z)',
|
||||||
|
re.MULTILINE | re.DOTALL
|
||||||
|
)
|
||||||
|
m = pattern.search(content)
|
||||||
|
if not m:
|
||||||
|
return None
|
||||||
|
block = m.group(1)
|
||||||
|
cf_m = re.search(r'^\- \*\*Contributing files:\*\* (.+)$', block, re.MULTILINE)
|
||||||
|
if not cf_m:
|
||||||
|
return None
|
||||||
|
return cf_m.group(1).strip()
|
||||||
|
|
||||||
|
def parse_research_doc(content, slug):
|
||||||
|
"""Find the Research doc value for a given slug H2 in content."""
|
||||||
|
pattern = re.compile(
|
||||||
|
r'^## ' + re.escape(slug) + r'\s*\n(.*?)(?=^## |\Z)',
|
||||||
|
re.MULTILINE | re.DOTALL
|
||||||
|
)
|
||||||
|
m = pattern.search(content)
|
||||||
|
if not m:
|
||||||
|
return None
|
||||||
|
block = m.group(1)
|
||||||
|
rd_m = re.search(r'^\- \*\*Research doc:\*\* (.+)$', block, re.MULTILINE)
|
||||||
|
if not rd_m:
|
||||||
|
return None
|
||||||
|
return rd_m.group(1).strip()
|
||||||
|
|
||||||
|
def parse_status(content, slug):
|
||||||
|
"""Find the Status value for a given slug H2 in content."""
|
||||||
|
pattern = re.compile(
|
||||||
|
r'^## ' + re.escape(slug) + r'\s*\n(.*?)(?=^## |\Z)',
|
||||||
|
re.MULTILINE | re.DOTALL
|
||||||
|
)
|
||||||
|
m = pattern.search(content)
|
||||||
|
if not m:
|
||||||
|
return None
|
||||||
|
block = m.group(1)
|
||||||
|
st_m = re.search(r'^\- \*\*Status:\*\* (.+)$', block, re.MULTILINE)
|
||||||
|
if not st_m:
|
||||||
|
return None
|
||||||
|
return st_m.group(1).strip()
|
||||||
|
|
||||||
|
def find_repo_root(start_dir):
|
||||||
|
"""Walk up from start_dir until we find a directory containing .git."""
|
||||||
|
current = start_dir
|
||||||
|
while True:
|
||||||
|
if os.path.exists(os.path.join(current, ".git")):
|
||||||
|
return current
|
||||||
|
parent = os.path.dirname(current)
|
||||||
|
if parent == current:
|
||||||
|
return None
|
||||||
|
current = parent
|
||||||
|
|
||||||
|
findings = []
|
||||||
|
has_fail = False
|
||||||
|
|
||||||
|
def emit_fail(desc, fpath, why, fix):
|
||||||
|
global has_fail
|
||||||
|
has_fail = True
|
||||||
|
findings.append(("FAIL", desc, fpath, why, fix, None))
|
||||||
|
|
||||||
|
def emit_info(desc, fpath, note):
|
||||||
|
findings.append(("INFO", desc, fpath, None, None, note))
|
||||||
|
|
||||||
|
def print_findings():
|
||||||
|
for entry in findings:
|
||||||
|
kind = entry[0]
|
||||||
|
desc = entry[1]
|
||||||
|
fpath = entry[2]
|
||||||
|
why = entry[3]
|
||||||
|
fix = entry[4]
|
||||||
|
note = entry[5]
|
||||||
|
if kind == "FAIL":
|
||||||
|
print(f"FAIL {desc} — {fpath}")
|
||||||
|
print(f" Why: {why}")
|
||||||
|
print(f" Fix: {fix}")
|
||||||
|
print()
|
||||||
|
else:
|
||||||
|
print(f"INFO {desc} — {fpath}")
|
||||||
|
print(f" Note: {note}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
# --- Scan for any file with source_keys ---
|
||||||
|
|
||||||
|
def file_has_source_keys(fpath):
|
||||||
|
try:
|
||||||
|
with open(fpath) as f:
|
||||||
|
content = f.read()
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
fm, _ = parse_frontmatter(content)
|
||||||
|
if fm is None:
|
||||||
|
return False
|
||||||
|
return bool(parse_source_keys(fm))
|
||||||
|
|
||||||
|
def find_files_with_source_keys():
|
||||||
|
"""Return list of (relative_path, abs_path) for all skill files with source_keys."""
|
||||||
|
results = []
|
||||||
|
for root, dirs, files in os.walk(skill_dir):
|
||||||
|
# Skip hidden dirs
|
||||||
|
dirs[:] = [d for d in dirs if not d.startswith('.')]
|
||||||
|
for fname in files:
|
||||||
|
if fname.endswith('.md'):
|
||||||
|
abs_path = os.path.join(root, fname)
|
||||||
|
if file_has_source_keys(abs_path):
|
||||||
|
rel = os.path.relpath(abs_path, skill_dir)
|
||||||
|
results.append((rel, abs_path))
|
||||||
|
return results
|
||||||
|
|
||||||
|
sources_md_exists = os.path.isfile(sources_md_path)
|
||||||
|
files_with_source_keys = find_files_with_source_keys()
|
||||||
|
|
||||||
|
# Early exit: nothing to validate
|
||||||
|
if not sources_md_exists and not files_with_source_keys:
|
||||||
|
sys.exit(0)
|
||||||
|
|
||||||
|
# Load sources.md if it exists
|
||||||
|
sources_content = None
|
||||||
|
if sources_md_exists:
|
||||||
|
with open(sources_md_path) as f:
|
||||||
|
sources_content = f.read()
|
||||||
|
sources_slugs = set(parse_h2_slugs(sources_content))
|
||||||
|
else:
|
||||||
|
sources_slugs = set()
|
||||||
|
|
||||||
|
# --- Check 0: source_keys without sources.md ---
|
||||||
|
if not sources_md_exists:
|
||||||
|
for rel, abs_path in files_with_source_keys:
|
||||||
|
emit_fail(
|
||||||
|
f"source_keys declared but references/sources.md is absent",
|
||||||
|
rel,
|
||||||
|
"source_keys references research provenance that has no sources index to validate against.",
|
||||||
|
"Create references/sources.md with an H2 entry for each slug referenced by source_keys."
|
||||||
|
)
|
||||||
|
print_findings()
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
# --- Check 1: FILL IN: placeholders in sources.md ---
|
||||||
|
for line in sources_content.splitlines():
|
||||||
|
if PLACEHOLDER_RE.search(line):
|
||||||
|
emit_fail(
|
||||||
|
"Unfilled FILL IN: placeholder",
|
||||||
|
"references/sources.md",
|
||||||
|
"sources.md contains an unfilled placeholder, meaning provenance is incomplete.",
|
||||||
|
"Replace all 'FILL IN:' values in references/sources.md with real content."
|
||||||
|
)
|
||||||
|
break
|
||||||
|
|
||||||
|
# --- Check 2: source_keys in SKILL.md → slug exists in sources.md ---
|
||||||
|
skill_md_path = os.path.join(skill_dir, "SKILL.md")
|
||||||
|
if os.path.isfile(skill_md_path):
|
||||||
|
with open(skill_md_path) as f:
|
||||||
|
skill_content = f.read()
|
||||||
|
skill_fm, _ = parse_frontmatter(skill_content)
|
||||||
|
skill_source_keys = parse_source_keys(skill_fm)
|
||||||
|
for slug in skill_source_keys:
|
||||||
|
if slug not in sources_slugs:
|
||||||
|
emit_fail(
|
||||||
|
f"source_keys slug '{slug}' not found in sources.md",
|
||||||
|
"SKILL.md",
|
||||||
|
f"SKILL.md declares '{slug}' as a source but there is no '## {slug}' heading in references/sources.md.",
|
||||||
|
f"Add '## {slug}' entry to references/sources.md or remove '{slug}' from SKILL.md source_keys."
|
||||||
|
)
|
||||||
|
|
||||||
|
# --- Check 3: source_keys in references/*.md → slug exists in sources.md (INFO if no source_keys) ---
|
||||||
|
if os.path.isdir(refs_dir):
|
||||||
|
for fname in sorted(os.listdir(refs_dir)):
|
||||||
|
if not fname.endswith('.md'):
|
||||||
|
continue
|
||||||
|
if fname == "sources.md":
|
||||||
|
continue
|
||||||
|
fpath = os.path.join(refs_dir, fname)
|
||||||
|
rel = os.path.relpath(fpath, skill_dir)
|
||||||
|
with open(fpath) as f:
|
||||||
|
ref_content = f.read()
|
||||||
|
ref_fm, _ = parse_frontmatter(ref_content)
|
||||||
|
ref_keys = parse_source_keys(ref_fm)
|
||||||
|
if not ref_keys:
|
||||||
|
emit_info(
|
||||||
|
f"No source_keys frontmatter",
|
||||||
|
rel,
|
||||||
|
"This references file has no source_keys — provenance cannot be verified. "
|
||||||
|
"Add source_keys frontmatter listing the slugs from references/sources.md that informed this file."
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
for slug in ref_keys:
|
||||||
|
if slug not in sources_slugs:
|
||||||
|
emit_fail(
|
||||||
|
f"source_keys slug '{slug}' not found in sources.md",
|
||||||
|
rel,
|
||||||
|
f"'{rel}' declares '{slug}' as a source but there is no '## {slug}' heading in references/sources.md.",
|
||||||
|
f"Add '## {slug}' entry to references/sources.md or remove '{slug}' from {rel} source_keys."
|
||||||
|
)
|
||||||
|
|
||||||
|
# --- Checks 4, 5, 6, 7, 8: Per-slug checks in sources.md ---
|
||||||
|
|
||||||
|
repo_root = find_repo_root(skill_dir)
|
||||||
|
|
||||||
|
# Collect all research doc paths we'll check (for Check 8)
|
||||||
|
research_docs_seen = {} # abs_path → set of slugs in sources.md that reference it
|
||||||
|
|
||||||
|
for slug in parse_h2_slugs(sources_content):
|
||||||
|
# Check 4: Contributing files exist
|
||||||
|
cf_value = parse_contributing_files(sources_content, slug)
|
||||||
|
if cf_value and not cf_value.startswith("(none"):
|
||||||
|
# Split by comma
|
||||||
|
cf_files = [p.strip() for p in cf_value.split(",") if p.strip()]
|
||||||
|
for cf_rel in cf_files:
|
||||||
|
cf_abs = os.path.join(skill_dir, cf_rel)
|
||||||
|
if not os.path.isfile(cf_abs):
|
||||||
|
emit_fail(
|
||||||
|
f"Contributing file '{cf_rel}' does not exist",
|
||||||
|
f"references/sources.md (## {slug})",
|
||||||
|
f"sources.md claims '{cf_rel}' was contributed to by slug '{slug}' but the file does not exist.",
|
||||||
|
f"Create '{cf_rel}' relative to the skill directory, or correct the path in sources.md."
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# Check 5: Bidirectional — file should list slug in its source_keys
|
||||||
|
# Skip sources.md itself
|
||||||
|
if cf_rel == "references/sources.md":
|
||||||
|
continue
|
||||||
|
with open(cf_abs) as f:
|
||||||
|
cf_content = f.read()
|
||||||
|
cf_fm, _ = parse_frontmatter(cf_content)
|
||||||
|
cf_keys = parse_source_keys(cf_fm)
|
||||||
|
if slug not in cf_keys:
|
||||||
|
emit_fail(
|
||||||
|
f"Contributing file '{cf_rel}' does not list '{slug}' in its source_keys",
|
||||||
|
f"references/sources.md (## {slug})",
|
||||||
|
f"sources.md says '{cf_rel}' was informed by '{slug}', but '{cf_rel}' does not declare '{slug}' in its source_keys frontmatter.",
|
||||||
|
f"Add '{slug}' to the source_keys frontmatter of '{cf_rel}'."
|
||||||
|
)
|
||||||
|
|
||||||
|
# Check 6: Research doc field required
|
||||||
|
rd_value = parse_research_doc(sources_content, slug)
|
||||||
|
if rd_value is None:
|
||||||
|
emit_fail(
|
||||||
|
f"Research doc field missing",
|
||||||
|
f"references/sources.md (## {slug})",
|
||||||
|
f"The '## {slug}' entry in sources.md has no '- **Research doc:**' line.",
|
||||||
|
f"Add '- **Research doc:** <path-or-(none)>' to the '## {slug}' entry in references/sources.md."
|
||||||
|
)
|
||||||
|
elif rd_value == "" or PLACEHOLDER_RE.search(rd_value):
|
||||||
|
emit_fail(
|
||||||
|
f"Research doc field is empty or placeholder",
|
||||||
|
f"references/sources.md (## {slug})",
|
||||||
|
f"The '## {slug}' entry has an unfilled Research doc value.",
|
||||||
|
f"Set '- **Research doc:**' to a real path relative to repo root, or '(none)' if not applicable."
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# Check 7: Upstream forward — slug should appear in research doc
|
||||||
|
if repo_root and not rd_value.startswith("(none"):
|
||||||
|
rd_abs = os.path.join(repo_root, rd_value)
|
||||||
|
if os.path.isfile(rd_abs):
|
||||||
|
with open(rd_abs) as f:
|
||||||
|
rd_content = f.read()
|
||||||
|
rd_slugs = set(parse_h2_slugs(rd_content))
|
||||||
|
if slug not in rd_slugs:
|
||||||
|
emit_info(
|
||||||
|
f"Slug '{slug}' not found as H2 in research doc '{rd_value}'",
|
||||||
|
f"references/sources.md (## {slug})",
|
||||||
|
f"The research doc '{rd_value}' does not have a '## {slug}' heading. "
|
||||||
|
f"The provenance link may be imprecise — the slug name in sources.md may differ from the research doc's heading."
|
||||||
|
)
|
||||||
|
# Track for Check 8
|
||||||
|
if rd_abs not in research_docs_seen:
|
||||||
|
research_docs_seen[rd_abs] = (rd_value, set())
|
||||||
|
research_docs_seen[rd_abs][1].add(slug)
|
||||||
|
|
||||||
|
# --- Check 8: Upstream reverse ---
|
||||||
|
for rd_abs, (rd_rel, known_slugs) in research_docs_seen.items():
|
||||||
|
with open(rd_abs) as f:
|
||||||
|
rd_content = f.read()
|
||||||
|
for rd_slug in parse_h2_slugs(rd_content):
|
||||||
|
# Parse this slug's Contributing files and Status in the research doc
|
||||||
|
rd_cf = parse_contributing_files(rd_content, rd_slug)
|
||||||
|
rd_status = parse_status(rd_content, rd_slug)
|
||||||
|
# Skip if contributing files start with (none
|
||||||
|
if rd_cf and rd_cf.startswith("(none"):
|
||||||
|
continue
|
||||||
|
# Skip if status is not `extracted`
|
||||||
|
if rd_status != "`extracted`":
|
||||||
|
continue
|
||||||
|
# This slug should be in sources.md
|
||||||
|
if rd_slug not in sources_slugs:
|
||||||
|
emit_fail(
|
||||||
|
f"Research doc slug '{rd_slug}' missing from skill sources.md",
|
||||||
|
f"references/sources.md",
|
||||||
|
f"The research doc '{rd_rel}' has '## {rd_slug}' with status `extracted` and contributing files, "
|
||||||
|
f"but this skill's sources.md has no '## {rd_slug}' entry.",
|
||||||
|
f"Add '## {rd_slug}' to references/sources.md or mark it as '(none)' in the research doc's Contributing files."
|
||||||
|
)
|
||||||
|
|
||||||
|
print_findings()
|
||||||
|
sys.exit(1 if has_fail else 0)
|
||||||
|
PYTHON
|
||||||
@@ -26,3 +26,4 @@ bats plugins/kyberforge/skills/skill-audit/tests/
|
|||||||
| File | Purpose |
|
| File | Purpose |
|
||||||
|------|---------|
|
|------|---------|
|
||||||
| `validate.bats` | Bats test suite for `scripts/validate.sh` |
|
| `validate.bats` | Bats test suite for `scripts/validate.sh` |
|
||||||
|
| `validate-provenance.bats` | Bats test suite for `scripts/validate-provenance.sh` |
|
||||||
|
|||||||
@@ -0,0 +1,514 @@
|
|||||||
|
#!/usr/bin/env bats
|
||||||
|
|
||||||
|
setup() {
|
||||||
|
REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../../../../../" && pwd)"
|
||||||
|
load "$REPO_ROOT/tests/test_helper/bats-support/load"
|
||||||
|
load "$REPO_ROOT/tests/test_helper/bats-assert/load"
|
||||||
|
|
||||||
|
SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate-provenance.sh"
|
||||||
|
TMPDIR="$(mktemp -d)"
|
||||||
|
|
||||||
|
# Helper: create a minimal skill directory with no sources.md and no source_keys
|
||||||
|
make_clean_skill() {
|
||||||
|
local dir="$1"
|
||||||
|
local name
|
||||||
|
name="$(basename "$dir")"
|
||||||
|
mkdir -p "$dir"
|
||||||
|
cat > "$dir/SKILL.md" <<EOF
|
||||||
|
---
|
||||||
|
name: $name
|
||||||
|
description: A valid skill description.
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 1
|
||||||
|
|
||||||
|
Do the thing.
|
||||||
|
EOF
|
||||||
|
}
|
||||||
|
|
||||||
|
# Helper: create a skill with source_keys in SKILL.md
|
||||||
|
make_skill_with_source_keys() {
|
||||||
|
local dir="$1"
|
||||||
|
local name
|
||||||
|
name="$(basename "$dir")"
|
||||||
|
mkdir -p "$dir"
|
||||||
|
cat > "$dir/SKILL.md" <<EOF
|
||||||
|
---
|
||||||
|
name: $name
|
||||||
|
description: A valid skill description.
|
||||||
|
metadata:
|
||||||
|
source_keys:
|
||||||
|
- my-source
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 1
|
||||||
|
|
||||||
|
Do the thing.
|
||||||
|
EOF
|
||||||
|
}
|
||||||
|
|
||||||
|
# Helper: create a valid sources.md with one entry
|
||||||
|
make_sources_md() {
|
||||||
|
local dir="$1"
|
||||||
|
local slug="${2:-my-source}"
|
||||||
|
local contrib="${3:-SKILL.md}"
|
||||||
|
local research="${4:-(none)}"
|
||||||
|
mkdir -p "$dir/references"
|
||||||
|
cat > "$dir/references/sources.md" <<EOF
|
||||||
|
# Sources
|
||||||
|
|
||||||
|
## ${slug}
|
||||||
|
|
||||||
|
- **URL:** https://example.com/${slug}
|
||||||
|
- **Description:** A test source.
|
||||||
|
- **Contributing files:** ${contrib}
|
||||||
|
- **Research doc:** ${research}
|
||||||
|
- **Status:** \`extracted\`
|
||||||
|
EOF
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
teardown() {
|
||||||
|
rm -rf "$TMPDIR"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Cycle 1 — --help
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@test "--help exits 0" {
|
||||||
|
run bash "$SCRIPT" --help
|
||||||
|
assert_success
|
||||||
|
assert_output --partial "Usage:"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Cycle 2 — Early exit: no sources.md, no source_keys → exit 0, no output
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@test "clean pass: no sources.md and no source_keys anywhere → exit 0, no output" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_clean_skill "$skill"
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_success
|
||||||
|
assert_output ""
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Cycle 3 — Check 0: source_keys present but no sources.md → FAIL
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@test "FAIL: source_keys in SKILL.md but sources.md absent" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_failure
|
||||||
|
assert_output --partial "FAIL"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Cycle 4 — Check 1: FILL IN: placeholder in sources.md → FAIL
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@test "FAIL: FILL IN: placeholder in sources.md" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
mkdir -p "$skill/references"
|
||||||
|
cat > "$skill/references/sources.md" <<EOF
|
||||||
|
# Sources
|
||||||
|
|
||||||
|
## my-source
|
||||||
|
|
||||||
|
- **URL:** FILL IN: add url
|
||||||
|
- **Description:** A test source.
|
||||||
|
- **Contributing files:** SKILL.md
|
||||||
|
- **Research doc:** (none)
|
||||||
|
- **Status:** \`extracted\`
|
||||||
|
EOF
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_failure
|
||||||
|
assert_output --partial "FAIL"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "FILL IN: inside backticks in sources.md does not fail" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
make_sources_md "$skill"
|
||||||
|
echo "Use \`FILL IN: value\` as example." >> "$skill/references/sources.md"
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_success
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Cycle 5 — Check 2: source_keys slug missing from sources.md → FAIL
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@test "FAIL: source_keys slug in SKILL.md not present as H2 in sources.md" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
mkdir -p "$skill/references"
|
||||||
|
cat > "$skill/references/sources.md" <<EOF
|
||||||
|
# Sources
|
||||||
|
|
||||||
|
## different-source
|
||||||
|
|
||||||
|
- **URL:** https://example.com/different-source
|
||||||
|
- **Description:** A different source.
|
||||||
|
- **Contributing files:** SKILL.md
|
||||||
|
- **Research doc:** (none)
|
||||||
|
- **Status:** \`extracted\`
|
||||||
|
EOF
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_failure
|
||||||
|
assert_output --partial "FAIL"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Cycle 6 — Check 4: Contributing file path doesn't exist → FAIL
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@test "FAIL: Contributing file listed in sources.md does not exist" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
make_sources_md "$skill" "my-source" "references/nonexistent.md"
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_failure
|
||||||
|
assert_output --partial "FAIL"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "pass: (none) in Contributing files is skipped" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
make_sources_md "$skill" "my-source" "(none — not used directly)"
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_success
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Cycle 7 — Check 6: Research doc field missing → FAIL
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@test "FAIL: Research doc field missing from sources.md entry" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
mkdir -p "$skill/references"
|
||||||
|
cat > "$skill/references/sources.md" <<EOF
|
||||||
|
# Sources
|
||||||
|
|
||||||
|
## my-source
|
||||||
|
|
||||||
|
- **URL:** https://example.com/my-source
|
||||||
|
- **Description:** A test source.
|
||||||
|
- **Contributing files:** SKILL.md
|
||||||
|
- **Status:** \`extracted\`
|
||||||
|
EOF
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_failure
|
||||||
|
assert_output --partial "FAIL"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "FAIL: Research doc field is FILL IN: placeholder" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
mkdir -p "$skill/references"
|
||||||
|
cat > "$skill/references/sources.md" <<EOF
|
||||||
|
# Sources
|
||||||
|
|
||||||
|
## my-source
|
||||||
|
|
||||||
|
- **URL:** https://example.com/my-source
|
||||||
|
- **Description:** A test source.
|
||||||
|
- **Contributing files:** SKILL.md
|
||||||
|
- **Research doc:** FILL IN: path to research doc
|
||||||
|
- **Status:** \`extracted\`
|
||||||
|
EOF
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_failure
|
||||||
|
assert_output --partial "FAIL"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Cycle 8 — Check 5: Bidirectional mismatch → FAIL
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@test "FAIL: Contributing file exists but does not list parent slug in source_keys" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
make_sources_md "$skill" "my-source" "SKILL.md"
|
||||||
|
# SKILL.md has source_keys: my-source, but let's change it to NOT have my-source
|
||||||
|
cat > "$skill/SKILL.md" <<EOF
|
||||||
|
---
|
||||||
|
name: my-skill
|
||||||
|
description: A valid skill description.
|
||||||
|
metadata:
|
||||||
|
source_keys:
|
||||||
|
- other-source
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 1
|
||||||
|
|
||||||
|
Do the thing.
|
||||||
|
EOF
|
||||||
|
mkdir -p "$skill/references"
|
||||||
|
cat > "$skill/references/sources.md" <<EOF
|
||||||
|
# Sources
|
||||||
|
|
||||||
|
## other-source
|
||||||
|
|
||||||
|
- **URL:** https://example.com/other-source
|
||||||
|
- **Description:** A test source.
|
||||||
|
- **Contributing files:** SKILL.md
|
||||||
|
- **Research doc:** (none)
|
||||||
|
- **Status:** \`extracted\`
|
||||||
|
EOF
|
||||||
|
# Now add my-source that references SKILL.md but SKILL.md doesn't back-reference it
|
||||||
|
cat >> "$skill/references/sources.md" <<EOF
|
||||||
|
|
||||||
|
## my-source
|
||||||
|
|
||||||
|
- **URL:** https://example.com/my-source
|
||||||
|
- **Description:** Another source.
|
||||||
|
- **Contributing files:** SKILL.md
|
||||||
|
- **Research doc:** (none)
|
||||||
|
- **Status:** \`extracted\`
|
||||||
|
EOF
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_failure
|
||||||
|
assert_output --partial "FAIL"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Cycle 9 — Check 3: references/*.md with no source_keys → INFO (exit 0)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@test "INFO: references doc with no source_keys frontmatter emits INFO but exits 0" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
make_sources_md "$skill"
|
||||||
|
mkdir -p "$skill/references"
|
||||||
|
cat > "$skill/references/extra.md" <<EOF
|
||||||
|
# Extra Reference
|
||||||
|
|
||||||
|
No frontmatter here.
|
||||||
|
EOF
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_success
|
||||||
|
assert_output --partial "INFO"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "pass: references doc with source_keys all matching sources.md exits 0" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
make_sources_md "$skill"
|
||||||
|
mkdir -p "$skill/references"
|
||||||
|
cat > "$skill/references/extra.md" <<EOF
|
||||||
|
---
|
||||||
|
source_keys:
|
||||||
|
- my-source
|
||||||
|
---
|
||||||
|
|
||||||
|
# Extra Reference
|
||||||
|
|
||||||
|
Content here.
|
||||||
|
EOF
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_success
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Cycle 10 — Clean full pass: valid sources.md, all source_keys match, files exist
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@test "clean full pass: all checks satisfied" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
make_sources_md "$skill"
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_success
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Cycle 11 — Check 7: Upstream forward: slug in sources.md not in research doc → INFO
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@test "INFO: slug in sources.md not found in research doc → INFO, exits 0" {
|
||||||
|
local skill="$TMPDIR/my-skill"
|
||||||
|
make_skill_with_source_keys "$skill"
|
||||||
|
|
||||||
|
# Create a research doc that does NOT have the slug
|
||||||
|
local research_dir="$TMPDIR/research"
|
||||||
|
mkdir -p "$research_dir"
|
||||||
|
cat > "$research_dir/my-research.md" <<EOF
|
||||||
|
# Research
|
||||||
|
|
||||||
|
## different-slug
|
||||||
|
|
||||||
|
- **Contributing files:** (none)
|
||||||
|
- **Status:** \`extracted\`
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# Use a path relative to repo root — we'll place research doc inside TMPDIR
|
||||||
|
# and reference it as absolute for test purposes.
|
||||||
|
# The script finds repo root by walking up from skill-dir until .git is found.
|
||||||
|
# Since TMPDIR won't have .git, we simulate a repo structure.
|
||||||
|
local fake_repo="$TMPDIR/fakerepo"
|
||||||
|
mkdir -p "$fake_repo"
|
||||||
|
touch "$fake_repo/.git" # fake .git marker
|
||||||
|
|
||||||
|
local skill2="$fake_repo/my-skill"
|
||||||
|
mkdir -p "$skill2"
|
||||||
|
cat > "$skill2/SKILL.md" <<EOF
|
||||||
|
---
|
||||||
|
name: my-skill
|
||||||
|
description: A valid skill description.
|
||||||
|
metadata:
|
||||||
|
source_keys:
|
||||||
|
- my-source
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 1
|
||||||
|
|
||||||
|
Do the thing.
|
||||||
|
EOF
|
||||||
|
|
||||||
|
mkdir -p "$skill2/references"
|
||||||
|
mkdir -p "$fake_repo/docs/research"
|
||||||
|
cat > "$fake_repo/docs/research/my-research.md" <<EOF
|
||||||
|
# Research
|
||||||
|
|
||||||
|
## different-slug
|
||||||
|
|
||||||
|
- **Contributing files:** (none)
|
||||||
|
- **Status:** \`extracted\`
|
||||||
|
EOF
|
||||||
|
|
||||||
|
cat > "$skill2/references/sources.md" <<EOF
|
||||||
|
# Sources
|
||||||
|
|
||||||
|
## my-source
|
||||||
|
|
||||||
|
- **URL:** https://example.com/my-source
|
||||||
|
- **Description:** A test source.
|
||||||
|
- **Contributing files:** SKILL.md
|
||||||
|
- **Research doc:** docs/research/my-research.md
|
||||||
|
- **Status:** \`extracted\`
|
||||||
|
EOF
|
||||||
|
|
||||||
|
run bash "$SCRIPT" "$skill2"
|
||||||
|
assert_success
|
||||||
|
assert_output --partial "INFO"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Cycle 12 — Check 8: Upstream reverse: extracted slug in research doc not in sources.md → FAIL
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@test "FAIL: extracted non-(none) slug in research doc missing from skill sources.md" {
|
||||||
|
local fake_repo="$TMPDIR/fakerepo"
|
||||||
|
mkdir -p "$fake_repo"
|
||||||
|
touch "$fake_repo/.git"
|
||||||
|
|
||||||
|
local skill="$fake_repo/my-skill"
|
||||||
|
mkdir -p "$skill"
|
||||||
|
cat > "$skill/SKILL.md" <<EOF
|
||||||
|
---
|
||||||
|
name: my-skill
|
||||||
|
description: A valid skill description.
|
||||||
|
metadata:
|
||||||
|
source_keys:
|
||||||
|
- my-source
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 1
|
||||||
|
|
||||||
|
Do the thing.
|
||||||
|
EOF
|
||||||
|
|
||||||
|
mkdir -p "$skill/references"
|
||||||
|
mkdir -p "$fake_repo/docs/research"
|
||||||
|
|
||||||
|
# Research doc has my-source (extracted, with a contributing file) AND extra-source (also extracted)
|
||||||
|
cat > "$fake_repo/docs/research/my-research.md" <<EOF
|
||||||
|
# Research
|
||||||
|
|
||||||
|
## my-source
|
||||||
|
|
||||||
|
- **Contributing files:** some-skill/SKILL.md
|
||||||
|
- **Status:** \`extracted\`
|
||||||
|
|
||||||
|
## extra-source
|
||||||
|
|
||||||
|
- **Contributing files:** some-skill/references/extra.md
|
||||||
|
- **Status:** \`extracted\`
|
||||||
|
EOF
|
||||||
|
|
||||||
|
cat > "$skill/references/sources.md" <<EOF
|
||||||
|
# Sources
|
||||||
|
|
||||||
|
## my-source
|
||||||
|
|
||||||
|
- **URL:** https://example.com/my-source
|
||||||
|
- **Description:** A test source.
|
||||||
|
- **Contributing files:** SKILL.md
|
||||||
|
- **Research doc:** docs/research/my-research.md
|
||||||
|
- **Status:** \`extracted\`
|
||||||
|
EOF
|
||||||
|
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_failure
|
||||||
|
assert_output --partial "FAIL"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "pass: extracted slug in research doc with (none) contributing files is not required in sources.md" {
|
||||||
|
local fake_repo="$TMPDIR/fakerepo"
|
||||||
|
mkdir -p "$fake_repo"
|
||||||
|
touch "$fake_repo/.git"
|
||||||
|
|
||||||
|
local skill="$fake_repo/my-skill"
|
||||||
|
mkdir -p "$skill"
|
||||||
|
cat > "$skill/SKILL.md" <<EOF
|
||||||
|
---
|
||||||
|
name: my-skill
|
||||||
|
description: A valid skill description.
|
||||||
|
metadata:
|
||||||
|
source_keys:
|
||||||
|
- my-source
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 1
|
||||||
|
|
||||||
|
Do the thing.
|
||||||
|
EOF
|
||||||
|
|
||||||
|
mkdir -p "$skill/references"
|
||||||
|
mkdir -p "$fake_repo/docs/research"
|
||||||
|
|
||||||
|
cat > "$fake_repo/docs/research/my-research.md" <<EOF
|
||||||
|
# Research
|
||||||
|
|
||||||
|
## my-source
|
||||||
|
|
||||||
|
- **Contributing files:** some-skill/SKILL.md
|
||||||
|
- **Status:** \`extracted\`
|
||||||
|
|
||||||
|
## extra-source
|
||||||
|
|
||||||
|
- **Contributing files:** (none — not relevant)
|
||||||
|
- **Status:** \`extracted\`
|
||||||
|
EOF
|
||||||
|
|
||||||
|
cat > "$skill/references/sources.md" <<EOF
|
||||||
|
# Sources
|
||||||
|
|
||||||
|
## my-source
|
||||||
|
|
||||||
|
- **URL:** https://example.com/my-source
|
||||||
|
- **Description:** A test source.
|
||||||
|
- **Contributing files:** SKILL.md
|
||||||
|
- **Research doc:** docs/research/my-research.md
|
||||||
|
- **Status:** \`extracted\`
|
||||||
|
EOF
|
||||||
|
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_success
|
||||||
|
}
|
||||||
@@ -193,7 +193,7 @@ If a research `sources.md` is present in the conversation context:
|
|||||||
|
|
||||||
1. Read it and filter to entries with `` `extracted` `` status only.
|
1. Read it and filter to entries with `` `extracted` `` status only.
|
||||||
2. For each entry, determine which skill files it contributed to (SKILL.md and any files in references/ that drew from it). Update `Contributing files` accordingly — list skill files, not research topic files.
|
2. For each entry, determine which skill files it contributed to (SKILL.md and any files in references/ that drew from it). Update `Contributing files` accordingly — list skill files, not research topic files.
|
||||||
3. Write the updated content to `references/sources.md`.
|
3. Write the updated content to `references/sources.md`. For each entry, include `- **Research doc:** <path>` where `<path>` is the relative path from the repo root to the plugin-level research sources file this entry was drawn from (e.g. `plugins/myplugin/docs/research/docs/<topic>/sources.md`). This field is required on every entry — it makes the provenance chain explicit and is validated by `/skill-audit`.
|
||||||
4. Add `source_keys` to the frontmatter of `SKILL.md` (under `metadata`) listing the slugs of sources that informed it.
|
4. Add `source_keys` to the frontmatter of `SKILL.md` (under `metadata`) listing the slugs of sources that informed it.
|
||||||
5. For each file in `references/` that was informed by research sources, add `source_keys` frontmatter (same format as research topic files) listing the relevant slugs.
|
5. For each file in `references/` that was informed by research sources, add `source_keys` frontmatter (same format as research topic files) listing the relevant slugs.
|
||||||
|
|
||||||
|
|||||||
@@ -9,5 +9,6 @@
|
|||||||
|
|
||||||
- **URL:** FILL IN
|
- **URL:** FILL IN
|
||||||
- **Description:** FILL IN
|
- **Description:** FILL IN
|
||||||
- **Contributing files:** FILL IN: list skill files this source informed (e.g. SKILL.md, references/foo.md)
|
- **Research doc:** FILL IN: path to the plugin-level research sources file this entry came from (e.g. plugins/myplugin/docs/research/docs/<topic>/sources.md), relative to repo root
|
||||||
|
- **Contributing files:** FILL IN: comma-separated list of skill files this source informed (e.g. SKILL.md, references/foo.md). Use `(none)` if the source was consulted but contributed no file content directly.
|
||||||
- **Status:** `extracted`
|
- **Status:** `extracted`
|
||||||
|
|||||||
@@ -5,6 +5,7 @@
|
|||||||
## agentskills-home
|
## agentskills-home
|
||||||
|
|
||||||
- **URL:** https://agentskills.io/home.md
|
- **URL:** https://agentskills.io/home.md
|
||||||
|
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||||
- **Description:** Agent Skills overview — what it is, why it exists, progressive disclosure model, ecosystem of 35+ implementing tools
|
- **Description:** Agent Skills overview — what it is, why it exists, progressive disclosure model, ecosystem of 35+ implementing tools
|
||||||
- **Contributing files:** SKILL.md
|
- **Contributing files:** SKILL.md
|
||||||
- **Status:** `extracted`
|
- **Status:** `extracted`
|
||||||
@@ -12,6 +13,7 @@
|
|||||||
## agentskills-spec
|
## agentskills-spec
|
||||||
|
|
||||||
- **URL:** https://agentskills.io/specification.md
|
- **URL:** https://agentskills.io/specification.md
|
||||||
|
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||||
- **Description:** Complete SKILL.md format specification — frontmatter fields, constraints, body content, optional directories, progressive disclosure levels, file references, validation
|
- **Description:** Complete SKILL.md format specification — frontmatter fields, constraints, body content, optional directories, progressive disclosure levels, file references, validation
|
||||||
- **Contributing files:** SKILL.md, references/deployment-modes.md
|
- **Contributing files:** SKILL.md, references/deployment-modes.md
|
||||||
- **Status:** `extracted`
|
- **Status:** `extracted`
|
||||||
@@ -19,6 +21,7 @@
|
|||||||
## agentskills-best-practices
|
## agentskills-best-practices
|
||||||
|
|
||||||
- **URL:** https://agentskills.io/skill-creation/best-practices.md
|
- **URL:** https://agentskills.io/skill-creation/best-practices.md
|
||||||
|
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||||
- **Description:** Best practices for skill creators — starting from real expertise, spending context wisely, calibrating control, instruction patterns (gotchas, templates, checklists, validation loops)
|
- **Description:** Best practices for skill creators — starting from real expertise, spending context wisely, calibrating control, instruction patterns (gotchas, templates, checklists, validation loops)
|
||||||
- **Contributing files:** SKILL.md
|
- **Contributing files:** SKILL.md
|
||||||
- **Status:** `extracted`
|
- **Status:** `extracted`
|
||||||
@@ -26,6 +29,7 @@
|
|||||||
## agentskills-optimizing-descriptions
|
## agentskills-optimizing-descriptions
|
||||||
|
|
||||||
- **URL:** https://agentskills.io/skill-creation/optimizing-descriptions.md
|
- **URL:** https://agentskills.io/skill-creation/optimizing-descriptions.md
|
||||||
|
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||||
- **Description:** How to systematically test and improve skill descriptions for triggering accuracy — eval queries, trigger rate testing, train/validation splits, optimization loop
|
- **Description:** How to systematically test and improve skill descriptions for triggering accuracy — eval queries, trigger rate testing, train/validation splits, optimization loop
|
||||||
- **Contributing files:** SKILL.md
|
- **Contributing files:** SKILL.md
|
||||||
- **Status:** `extracted`
|
- **Status:** `extracted`
|
||||||
@@ -33,6 +37,7 @@
|
|||||||
## agentskills-evaluating-skills
|
## agentskills-evaluating-skills
|
||||||
|
|
||||||
- **URL:** https://agentskills.io/skill-creation/evaluating-skills.md
|
- **URL:** https://agentskills.io/skill-creation/evaluating-skills.md
|
||||||
|
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||||
- **Description:** Eval-driven skill quality improvement — test case design, workspace structure, assertion writing, grading, benchmarking, human review, iteration loop
|
- **Description:** Eval-driven skill quality improvement — test case design, workspace structure, assertion writing, grading, benchmarking, human review, iteration loop
|
||||||
- **Contributing files:** SKILL.md
|
- **Contributing files:** SKILL.md
|
||||||
- **Status:** `extracted`
|
- **Status:** `extracted`
|
||||||
@@ -40,6 +45,7 @@
|
|||||||
## agentskills-using-scripts
|
## agentskills-using-scripts
|
||||||
|
|
||||||
- **URL:** https://agentskills.io/skill-creation/using-scripts.md
|
- **URL:** https://agentskills.io/skill-creation/using-scripts.md
|
||||||
|
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||||
- **Description:** Using scripts in skills — one-off commands, self-contained scripts with inline dependencies, designing scripts for agentic use (no interactive prompts, --help, structured output, idempotency)
|
- **Description:** Using scripts in skills — one-off commands, self-contained scripts with inline dependencies, designing scripts for agentic use (no interactive prompts, --help, structured output, idempotency)
|
||||||
- **Contributing files:** SKILL.md, references/scripts.md
|
- **Contributing files:** SKILL.md, references/scripts.md
|
||||||
- **Status:** `extracted`
|
- **Status:** `extracted`
|
||||||
@@ -47,7 +53,7 @@
|
|||||||
## agentskills-quickstart
|
## agentskills-quickstart
|
||||||
|
|
||||||
- **URL:** https://agentskills.io/skill-creation/quickstart.md
|
- **URL:** https://agentskills.io/skill-creation/quickstart.md
|
||||||
|
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||||
- **Description:** Step-by-step guide to creating a first skill (roll-dice example), how discovery/activation/execution work in practice
|
- **Description:** Step-by-step guide to creating a first skill (roll-dice example), how discovery/activation/execution work in practice
|
||||||
- **Contributing files:** SKILL.md
|
- **Contributing files:** SKILL.md
|
||||||
- **Status:** `extracted`
|
- **Status:** `extracted`
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user