Compare commits
3 Commits
9a3f72b696
...
6910f1b5a5
| Author | SHA1 | Date | |
|---|---|---|---|
| 6910f1b5a5 | |||
| 050aec4c80 | |||
| 0a41b2c7d3 |
@@ -4,7 +4,7 @@ Audits a Claude Code and Copilot agent definition file pair for correctness and
|
|||||||
|
|
||||||
## What it does
|
## What it does
|
||||||
|
|
||||||
Accepts either file in a CC `.md` / Copilot `.agent.md` pair, derives the counterpart automatically, and validates both. Runs structural checks via `validate.sh` (required fields, kebab-case name, no placeholders, no CC-only fields in the Copilot file, silently-ignored fields at plugin scope), provenance chain validation via `validate-provenance.sh` (checks `source_keys` against `sources.md` at the plugin root), then qualitative checks on description phrasing and system prompt quality. Produces a compact findings report in the same format as `skill-audit`.
|
Accepts either file in a CC `.md` / Copilot `.agent.md` pair, derives the counterpart automatically, and validates both. Runs structural checks via `validate.sh` (required fields, kebab-case name, no placeholders, no CC-only fields in the Copilot file, silently-ignored fields at plugin scope), provenance chain validation via `validate-provenance.sh` (checks `source_keys` against `sources.md` at the plugin root), then qualitative checks on description phrasing and system prompt quality. Step 1 also runs a Vale-based prose sub-check via `vale-wrap.sh` against both files of the pair, using the `Kyberforge` style (both files) and `KyberforgeCopilot` style (Copilot file only) — every alert is a `FAIL`, cited by rule ID — falling back to Step 2 judgment when the `vale` binary is unavailable or reports `0 files` scanned. Produces a compact findings report in the same format as `skill-audit`.
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
@@ -19,6 +19,12 @@ Pass the path to either agent file as the argument.
|
|||||||
| File | Purpose |
|
| File | Purpose |
|
||||||
|------|---------|
|
|------|---------|
|
||||||
| `SKILL.md` | Skill instructions for agents |
|
| `SKILL.md` | Skill instructions for agents |
|
||||||
|
| `assets/vale/.vale.ini` | Vale config: scopes `Kyberforge` to `**/agents/*.md`, `Kyberforge`+`KyberforgeCopilot` to `**/*.agent.md` |
|
||||||
|
| `assets/vale/styles/Kyberforge/DescriptionOpener.yml` | Flags descriptions opening with "This skill/agent" instead of an imperative "Use when..." |
|
||||||
|
| `assets/vale/styles/Kyberforge/PaddingPhrase.yml` | Flags generic "see references/ for info" pointers instead of specific file references |
|
||||||
|
| `assets/vale/styles/Kyberforge/SentenceOpenerThereIs.yml` | Flags sentences opening with "There is/are" instead of naming the subject directly |
|
||||||
|
| `assets/vale/styles/Kyberforge/VagueWording.yml` | Flags vague capability wording ("helps with", "utilize", "assists with", "used for") in descriptions |
|
||||||
|
| `assets/vale/styles/KyberforgeCopilot/ProactivePhrase.yml` | Flags CC-specific "Use proactively" phrasing with no effect in Copilot descriptions |
|
||||||
| `references/README.md` | Directory documentation for references/ |
|
| `references/README.md` | Directory documentation for references/ |
|
||||||
| `references/description-quality.md` | Qualitative guide for borderline description findings |
|
| `references/description-quality.md` | Qualitative guide for borderline description findings |
|
||||||
| `references/field-inventory.md` | Authoritative list of valid CC and Copilot agent fields |
|
| `references/field-inventory.md` | Authoritative list of valid CC and Copilot agent fields |
|
||||||
@@ -26,6 +32,7 @@ Pass the path to either agent file as the argument.
|
|||||||
| `scripts/README.md` | Directory documentation for scripts/ |
|
| `scripts/README.md` | Directory documentation for scripts/ |
|
||||||
| `scripts/validate.sh` | Structural validation script for agent file pairs |
|
| `scripts/validate.sh` | Structural validation script for agent file pairs |
|
||||||
| `scripts/validate-provenance.sh` | Provenance chain validation script for agent pairs against `sources.md` (plugin root) |
|
| `scripts/validate-provenance.sh` | Provenance chain validation script for agent pairs against `sources.md` (plugin root) |
|
||||||
|
| `scripts/vale-wrap.sh` | Drop-in `vale` wrapper that works around a frontmatter-description NLP scope limitation |
|
||||||
| `tests/README.md` | Bats test dependency and run instructions |
|
| `tests/README.md` | Bats test dependency and run instructions |
|
||||||
| `tests/validate.bats` | Bats tests for validate.sh |
|
| `tests/validate.bats` | Bats tests for validate.sh |
|
||||||
| `tests/validate-provenance.bats` | Bats tests for validate-provenance.sh |
|
| `tests/validate-provenance.bats` | Bats tests for validate-provenance.sh |
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ Audit a skill directory against the agentskills.io specification. Runs structura
|
|||||||
|
|
||||||
## What it does
|
## What it does
|
||||||
|
|
||||||
1. Runs `scripts/validate.sh` and `scripts/validate-provenance.sh` for structural and provenance checks
|
1. Runs `scripts/validate.sh` and `scripts/validate-provenance.sh` for structural and provenance checks, plus `scripts/vale-wrap.sh` — a Vale prefilter that deterministically flags known-bad description openers, vague wording, padding phrases, and "There is/are" sentence openers
|
||||||
2. Reads all files in the skill directory
|
2. Reads all files in the skill directory
|
||||||
3. Applies qualitative checks across seven dimensions
|
3. Applies qualitative checks across seven dimensions
|
||||||
4. Outputs a compact findings report — findings only, grouped by dimension, each with Why and Fix — and a result block with handoff to /skill-improve
|
4. Outputs a compact findings report — findings only, grouped by dimension, each with Why and Fix — and a result block with handoff to /skill-improve
|
||||||
@@ -24,6 +24,12 @@ Provide the path to the skill directory to audit when invoking.
|
|||||||
| `SKILL.md` | Skill instructions for agents |
|
| `SKILL.md` | Skill instructions for agents |
|
||||||
| `scripts/validate.sh` | Structural validator — checks name format, name matches directory, description length, line count, placeholder detection, script executable bit, and interactive-prompt detection |
|
| `scripts/validate.sh` | Structural validator — checks name format, name matches directory, description length, line count, placeholder detection, script executable bit, and interactive-prompt detection |
|
||||||
| `scripts/validate-provenance.sh` | Provenance validator — checks sources.md completeness, source_keys/slug consistency, Contributing files existence, bidirectional linkage, Research doc: fields, and upstream research doc alignment |
|
| `scripts/validate-provenance.sh` | Provenance validator — checks sources.md completeness, source_keys/slug consistency, Contributing files existence, bidirectional linkage, Research doc: fields, and upstream research doc alignment |
|
||||||
|
| `scripts/vale-wrap.sh` | Vale prefilter wrapper — runs the bundled `Kyberforge` Vale styles against SKILL.md and reports alerts as deterministic FAILs ahead of Step 3's qualitative review |
|
||||||
|
| `assets/vale/.vale.ini` | Vale configuration — points Vale at the bundled `Kyberforge` style path, self-located relative to `vale-wrap.sh` |
|
||||||
|
| `assets/vale/styles/Kyberforge/DescriptionOpener.yml` | Vale rule — flags literal "This skill..."/"This agent..." description openers |
|
||||||
|
| `assets/vale/styles/Kyberforge/PaddingPhrase.yml` | Vale rule — flags generic "see references/" padding phrasing in conditional references |
|
||||||
|
| `assets/vale/styles/Kyberforge/SentenceOpenerThereIs.yml` | Vale rule — flags body sentences starting with "There is"/"There are" |
|
||||||
|
| `assets/vale/styles/Kyberforge/VagueWording.yml` | Vale rule — flags known filler wording (e.g. "helps with", "utilize") |
|
||||||
| `references/description-quality.md` | Spec-grounded rubric for description auditing — loaded when a finding is borderline |
|
| `references/description-quality.md` | Spec-grounded rubric for description auditing — loaded when a finding is borderline |
|
||||||
| `references/body-discipline.md` | Spec-grounded rubric for body discipline auditing — loaded when padding vs necessity is unclear |
|
| `references/body-discipline.md` | Spec-grounded rubric for body discipline auditing — loaded when padding vs necessity is unclear |
|
||||||
| `references/sources.md` | Provenance record — agentskills.io sources that informed this skill and which files each contributed to |
|
| `references/sources.md` | Provenance record — agentskills.io sources that informed this skill and which files each contributed to |
|
||||||
|
|||||||
@@ -55,6 +55,7 @@ Work through each dimension internally. Collect findings only; report them in St
|
|||||||
|
|
||||||
Vale's `Kyberforge.DescriptionOpener` ("This skill..." openers) and `Kyberforge.VagueWording` (filler like "helps with", "utilize") alerts from Step 1 — both FAILs — cover imperative phrasing and known vague-wording filler directly; report them as findings without re-deriving by judgment. The rest is still a judgment call:
|
Vale's `Kyberforge.DescriptionOpener` ("This skill..." openers) and `Kyberforge.VagueWording` (filler like "helps with", "utilize") alerts from Step 1 — both FAILs — cover imperative phrasing and known vague-wording filler directly; report them as findings without re-deriving by judgment. The rest is still a judgment call:
|
||||||
|
|
||||||
|
- **Action-verb opening**: does the description start with a verb ("Audits...", "Reviews...", "Validates...")? Vale's `Kyberforge.DescriptionOpener` alert only catches the literal "This skill..." pattern — confirming an arbitrary opening word is genuinely a strong verb still requires judgment.
|
||||||
- **Specificity beyond the filler blocklist**: are capabilities stated precisely ("parses OpenAPI specs") or genuinely vaguely ("handles files")?
|
- **Specificity beyond the filler blocklist**: are capabilities stated precisely ("parses OpenAPI specs") or genuinely vaguely ("handles files")?
|
||||||
- **Indirect triggers**: does it cover cases where the user doesn't name the domain directly?
|
- **Indirect triggers**: does it cover cases where the user doesn't name the domain directly?
|
||||||
- **Near-miss exclusions**: are "Do not use when..." clauses present if a near-miss skill could steal activations?
|
- **Near-miss exclusions**: are "Do not use when..." clauses present if a near-miss skill could steal activations?
|
||||||
|
|||||||
@@ -55,7 +55,12 @@ fi
|
|||||||
# experiment tag reachable from the pushed ref must not shift the diff baseline.
|
# experiment tag reachable from the pushed ref must not shift the diff baseline.
|
||||||
# The tag is resolved from $PUSHED_REF, not HEAD, for the same reason the diff
|
# The tag is resolved from $PUSHED_REF, not HEAD, for the same reason the diff
|
||||||
# is: a tag reachable only from HEAD is not part of the history being pushed.
|
# is: a tag reachable only from HEAD is not part of the history being pushed.
|
||||||
LAST_TAG="$(git describe --tags --abbrev=0 --match 'v[0-9]*.[0-9]*.[0-9]*' "$PUSHED_REF" 2>/dev/null || true)"
|
# --match is a shell glob, not a regex: its trailing `*`s match any suffix, so
|
||||||
|
# without --exclude a pre-release/checkpoint tag like v1.2.3-checkpoint or
|
||||||
|
# v1.2.3-rc1 also satisfies 'v[0-9]*.[0-9]*.[0-9]*' and could be picked over the
|
||||||
|
# true last release tag. --exclude is glob syntax too, so '*-*' is what actually
|
||||||
|
# rules out any tag carrying a hyphenated suffix, leaving only bare vMAJOR.MINOR.PATCH.
|
||||||
|
LAST_TAG="$(git describe --tags --abbrev=0 --match 'v[0-9]*.[0-9]*.[0-9]*' --exclude '*-*' "$PUSHED_REF" 2>/dev/null || true)"
|
||||||
|
|
||||||
if [[ -z "$LAST_TAG" ]]; then
|
if [[ -z "$LAST_TAG" ]]; then
|
||||||
echo "FAIL: no release tag exists yet, but .pre-commit-hooks.yaml already exposes hooks to external consumers." >&2
|
echo "FAIL: no release tag exists yet, but .pre-commit-hooks.yaml already exposes hooks to external consumers." >&2
|
||||||
|
|||||||
@@ -419,6 +419,24 @@ else
|
|||||||
fail "the repo's own .pre-commit-hooks.yaml no longer satisfies the entry constraints: $OUT20"
|
fail "the repo's own .pre-commit-hooks.yaml no longer satisfies the entry constraints: $OUT20"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# --- 21. A vX.Y.Z-suffixed checkpoint tag must not satisfy the release gate ---
|
||||||
|
# git describe --match uses shell-glob semantics, not regex: the trailing `*` in
|
||||||
|
# 'v[0-9]*.[0-9]*.[0-9]*' matches any suffix, so a pre-release/checkpoint tag like
|
||||||
|
# v1.0.1-checkpoint also satisfies the glob and can be picked as LAST_TAG instead
|
||||||
|
# of the true last release tag — hiding a real release-relevant change that landed
|
||||||
|
# before the checkpoint tag from the diff.
|
||||||
|
echo ""
|
||||||
|
echo "--- ignores a vX.Y.Z-checkpoint tag and still flags the change since the real release tag ---"
|
||||||
|
FIXTURE21="$(make_tagged_fixture)"; track "$FIXTURE21"
|
||||||
|
echo "v2" > "$FIXTURE21/scripts/skill-size-check.sh"
|
||||||
|
(cd "$FIXTURE21" && git add -A && git commit -q -m "real release-relevant change" && git tag v1.0.1-checkpoint)
|
||||||
|
OUT21=$(run_check "$FIXTURE21" "refs/heads/main" || true)
|
||||||
|
if echo "$OUT21" | grep -q "skill-size-check.sh"; then
|
||||||
|
pass "still flags the release-relevant change since v1.0.0, ignoring the vX.Y.Z-checkpoint tag"
|
||||||
|
else
|
||||||
|
fail "a vX.Y.Z-checkpoint tag satisfied the glob and hid a real release-relevant change"
|
||||||
|
fi
|
||||||
|
|
||||||
echo ""
|
echo ""
|
||||||
echo "Results: $PASS passed, $FAIL failed"
|
echo "Results: $PASS passed, $FAIL failed"
|
||||||
[[ $FAIL -eq 0 ]]
|
[[ $FAIL -eq 0 ]]
|
||||||
|
|||||||
Reference in New Issue
Block a user