diff --git a/plugins/kyberforge/skills/agent-audit/README.md b/plugins/kyberforge/skills/agent-audit/README.md index 1fb71a3..f701bc5 100644 --- a/plugins/kyberforge/skills/agent-audit/README.md +++ b/plugins/kyberforge/skills/agent-audit/README.md @@ -4,7 +4,7 @@ Audits a Claude Code and Copilot agent definition file pair for correctness and ## What it does -Accepts either file in a CC `.md` / Copilot `.agent.md` pair, derives the counterpart automatically, and validates both. Runs structural checks via `validate.sh` (required fields, kebab-case name, no placeholders, no CC-only fields in the Copilot file, silently-ignored fields at plugin scope), then qualitative checks on description phrasing and system prompt quality. Produces a compact findings report in the same format as `skill-audit`. +Accepts either file in a CC `.md` / Copilot `.agent.md` pair, derives the counterpart automatically, and validates both. Runs structural checks via `validate.sh` (required fields, kebab-case name, no placeholders, no CC-only fields in the Copilot file, silently-ignored fields at plugin scope), provenance chain validation via `validate-provenance.sh` (checks `source_keys` against `agents/sources.md` at plugin scope), then qualitative checks on description phrasing and system prompt quality. Produces a compact findings report in the same format as `skill-audit`. ## Usage @@ -24,5 +24,7 @@ Pass the path to either agent file as the argument. | `references/sources.md` | Research provenance for skill content | | `scripts/README.md` | Directory documentation for scripts/ | | `scripts/validate.sh` | Structural validation script for agent file pairs | +| `scripts/validate-provenance.sh` | Provenance chain validation script for agent pairs against `agents/sources.md` | | `tests/README.md` | Bats test dependency and run instructions | | `tests/validate.bats` | Bats tests for validate.sh | +| `tests/validate-provenance.bats` | Bats tests for validate-provenance.sh | diff --git a/plugins/kyberforge/skills/agent-audit/SKILL.md b/plugins/kyberforge/skills/agent-audit/SKILL.md index b2cdc58..a26ea17 100644 --- a/plugins/kyberforge/skills/agent-audit/SKILL.md +++ b/plugins/kyberforge/skills/agent-audit/SKILL.md @@ -31,11 +31,14 @@ metadata: ```bash bash scripts/validate.sh +bash scripts/validate-provenance.sh ``` The script accepts either the CC file or the Copilot file. It detects provider from extension, derives the counterpart, and runs all structural checks. Note FAILs for the `### Structure` and `### Provider safety` report dimensions. Findings about missing fields, bad name format, empty body, or missing frontmatter → `### Structure`. Findings about CC-only fields in a Copilot file or plugin-silently-ignored fields in a CC file → `### Provider safety`. -If the script cannot run (Bash denied, python3 unavailable), perform checks manually: required fields present (`name`, `description`, non-empty body), `name` is kebab-case, `name` matches filename stem, no `FILL IN:` placeholders, no CC-only fields in Copilot file. +`validate-provenance.sh` validates the provenance chain between the agent pair's `source_keys` and the plugin-scoped `agents/sources.md`. It exits 0 silently for non-plugin-scope agents and when no provenance data exists. Note FAILs from this script for the `### Provenance` dimension — surface them verbatim with Why and Fix. + +If the scripts cannot run (Bash denied, python3 unavailable), perform checks manually: required fields present (`name`, `description`, non-empty body), `name` is kebab-case, `name` matches filename stem, no `FILL IN:` placeholders, no CC-only fields in Copilot file. ## Step 2 — Qualitative checks @@ -59,10 +62,10 @@ Read both agent files. Work through each dimension internally. Collect findings Open with a coverage line: ```text -Checked: structure · provider-safety · description · body · pair-consistency +Checked: structure · provider-safety · description · body · pair-consistency · provenance ``` -Then output only dimensions that have findings, grouped under H3 headings, FAILs before SUGGESTIONs within each dimension. Omit clean dimensions entirely. +Then output only dimensions that have findings, grouped under H3 headings, FAILs before SUGGESTIONs within each dimension. Omit clean dimensions entirely. `### Provenance` findings are sourced verbatim from `validate-provenance.sh` output — copy them without rephrasing. For each finding: diff --git a/plugins/kyberforge/skills/agent-audit/scripts/validate-provenance.sh b/plugins/kyberforge/skills/agent-audit/scripts/validate-provenance.sh new file mode 100755 index 0000000..1f69dfa --- /dev/null +++ b/plugins/kyberforge/skills/agent-audit/scripts/validate-provenance.sh @@ -0,0 +1,276 @@ +#!/usr/bin/env bash +set -euo pipefail + +usage() { + cat < + +Validate that an agent pair's sources provenance chain is complete and internally consistent. +Operates at plugin scope only — exits 0 silently for project and user scope agents. + +Arguments: + agent-file Path to either the Claude Code .md or Copilot .agent.md agent file. + +Exit codes: + 0 All checks passed (or nothing to validate, or not plugin scope) + 1 One or more checks failed + +Checks performed: + 0 source_keys present in agent pair but agents/sources.md absent + 1 FILL IN: placeholders in agents/sources.md + 2 source_keys in agent files → slug exists in agents/sources.md + 4 Contributing files listed in agents/sources.md exist on disk (plugin-root relative) + 5 Contributing files back-reference the parent slug in their source_keys + 6 Research doc field present and not placeholder +EOF +} + +if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then + usage + exit 0 +fi + +if [[ $# -lt 1 ]]; then + echo "Error: agent-file is required." >&2 + echo "" >&2 + usage >&2 + exit 1 +fi + +python3 -u - "$1" <<'PYTHON' +import sys +import os +import re + +agent_file = os.path.abspath(sys.argv[1]) +fname = os.path.basename(agent_file) +agent_dir = os.path.dirname(agent_file) + +# --- Detect provider --- +if fname.endswith('.agent.md'): + provider = 'copilot' + name_stem = fname[:-len('.agent.md')] +elif fname.endswith('.md'): + provider = 'claude-code' + name_stem = fname[:-len('.md')] +else: + print(f"Error: unrecognized extension '{fname}' — expected .md or .agent.md", file=sys.stderr) + sys.exit(2) + +# --- Find plugin root --- +def find_plugin_root(start_dir): + current = os.path.abspath(start_dir) + while True: + if os.path.isfile(os.path.join(current, 'plugin.json')): + return current + parent = os.path.dirname(current) + if parent == current: + return None + current = parent + +plugin_root = find_plugin_root(agent_dir) +if plugin_root is None: + sys.exit(0) + +# --- Derive counterpart --- +if provider == 'copilot': + counterpart = os.path.join(agent_dir, name_stem + '.md') +else: + counterpart = os.path.join(agent_dir, name_stem + '.agent.md') + +sources_md_path = os.path.join(plugin_root, 'agents', 'sources.md') + +# --- Helpers --- +PLACEHOLDER_RE = re.compile(r'(?' to the '## {slug}' entry in agents/sources.md." + ) + elif rd_value == "" or PLACEHOLDER_RE.search(rd_value): + emit_fail( + "Research doc field is empty or placeholder", + f"agents/sources.md (## {slug})", + f"The '## {slug}' entry has an unfilled Research doc value.", + "Set '- **Research doc:**' to a real path relative to repo root, or '(none)' if not applicable." + ) + +print_findings() +sys.exit(1 if has_fail else 0) +PYTHON diff --git a/plugins/kyberforge/skills/agent-audit/tests/validate-provenance.bats b/plugins/kyberforge/skills/agent-audit/tests/validate-provenance.bats new file mode 100644 index 0000000..f79ce3c --- /dev/null +++ b/plugins/kyberforge/skills/agent-audit/tests/validate-provenance.bats @@ -0,0 +1,383 @@ +#!/usr/bin/env bats + +setup() { + REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../../../../../" && pwd)" + load "$REPO_ROOT/tests/test_helper/bats-support/load" + load "$REPO_ROOT/tests/test_helper/bats-assert/load" + + SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate-provenance.sh" + TMPDIR="$(mktemp -d)" + + # Helper: create a plugin root with plugin.json and an agents/ directory + make_plugin() { + local root="$1" + mkdir -p "$root/agents" + echo '{"name":"test-plugin","version":"0.1.0"}' > "$root/plugin.json" + } + + # Helper: create a clean agent pair (no source_keys) + make_clean_pair() { + local root="$1" + local name="${2:-my-agent}" + cat > "$root/agents/${name}.md" < "$root/agents/${name}.agent.md" < "$root/agents/${name}.md" < "$root/agents/${name}.agent.md" < "$root/agents/${name}.agent.md" < "$root/agents/sources.md" < "$dir/agents/my-agent.md" < "$root/agents/sources.md" <> "$root/agents/sources.md" + run bash "$SCRIPT" "$root/agents/my-agent.md" + assert_success +} + +# --------------------------------------------------------------------------- +# Check 2: source_keys slug missing from agents/sources.md → FAIL +# --------------------------------------------------------------------------- + +@test "FAIL: source_keys slug in CC file not present as H2 in agents/sources.md" { + local root="$TMPDIR/plugin" + make_plugin "$root" + make_cc_with_source_keys "$root" "my-agent" "my-source" + make_copilot_clean "$root" + make_sources_md "$root" "different-source" "(none)" "(none)" + run bash "$SCRIPT" "$root/agents/my-agent.md" + assert_failure + assert_output --partial "FAIL" +} + +@test "FAIL: source_keys slug in Copilot file not present as H2 in agents/sources.md" { + local root="$TMPDIR/plugin" + make_plugin "$root" + make_clean_pair "$root" + make_copilot_with_source_keys "$root" "my-agent" "my-source" + make_sources_md "$root" "different-source" "(none)" "(none)" + run bash "$SCRIPT" "$root/agents/my-agent.agent.md" + assert_failure + assert_output --partial "FAIL" +} + +# --------------------------------------------------------------------------- +# Check 4: Contributing file path doesn't exist → FAIL +# --------------------------------------------------------------------------- + +@test "FAIL: Contributing file listed in agents/sources.md does not exist" { + local root="$TMPDIR/plugin" + make_plugin "$root" + make_cc_with_source_keys "$root" + make_copilot_with_source_keys "$root" + make_sources_md "$root" "my-source" "agents/nonexistent.md" + run bash "$SCRIPT" "$root/agents/my-agent.md" + assert_failure + assert_output --partial "FAIL" +} + +@test "pass: (none) in Contributing files is skipped" { + local root="$TMPDIR/plugin" + make_plugin "$root" + make_cc_with_source_keys "$root" + make_copilot_with_source_keys "$root" + make_sources_md "$root" "my-source" "(none — not used directly)" + run bash "$SCRIPT" "$root/agents/my-agent.md" + assert_success +} + +# --------------------------------------------------------------------------- +# Check 6: Research doc field missing or placeholder → FAIL +# --------------------------------------------------------------------------- + +@test "FAIL: Research doc field missing from agents/sources.md entry" { + local root="$TMPDIR/plugin" + make_plugin "$root" + make_cc_with_source_keys "$root" + make_copilot_with_source_keys "$root" + cat > "$root/agents/sources.md" < "$root/agents/sources.md" < "$root/agents/my-agent.md" < "$root/agents/sources.md" <` per entry pointing to the upstream research sources file. +2. For each entry, identify which agent files in the pair it contributed to. +3. Write `agents/sources.md` using the format below. Paths in `Contributing files:` are relative to the plugin root. + +```markdown +# Sources + +## slug-name + +- **URL:** +- **Research doc:** +- **Description:** +- **Contributing files:** agents/.md, agents/.agent.md +- **Status:** `extracted` +``` + +Each slug must match an H2 heading, and each slug must also appear in the `source_keys` list of every file listed under `Contributing files:`. If no research sources are in context, delete `agents/sources.md`. diff --git a/plugins/kyberforge/skills/agent-author/assets/templates/claude-code.md b/plugins/kyberforge/skills/agent-author/assets/templates/claude-code.md index ffce8e2..26dbdb4 100644 --- a/plugins/kyberforge/skills/agent-author/assets/templates/claude-code.md +++ b/plugins/kyberforge/skills/agent-author/assets/templates/claude-code.md @@ -45,6 +45,11 @@ description: FILL IN: Action-first description of what this agent does and when # background: false # Optional. Set true to force background execution. +# source_keys: +# - slug-name +# Development-only. Add when research sources informed this agent (slugs must match agents/sources.md). +# Omit when no research was used. Not a runtime field — silently ignored by Claude Code. + # NOTE: hooks, mcpServers, and permissionMode are silently ignored for plugin agents. # Those fields only work in .claude/agents/ or ~/.claude/agents/. --- diff --git a/plugins/kyberforge/skills/agent-author/assets/templates/copilot.agent.md b/plugins/kyberforge/skills/agent-author/assets/templates/copilot.agent.md index a7fe894..44f028f 100644 --- a/plugins/kyberforge/skills/agent-author/assets/templates/copilot.agent.md +++ b/plugins/kyberforge/skills/agent-author/assets/templates/copilot.agent.md @@ -37,6 +37,11 @@ description: FILL IN: Action-first description of what this agent does and when # model: claude-sonnet-4-5 # Optional. Model to run this agent on. +# source_keys: +# - slug-name +# Development-only. Add when research sources informed this agent (slugs must match agents/sources.md). +# Omit when no research was used. Not a Copilot runtime field — silently ignored. + # DO NOT include these Claude Code-only fields: # maxTurns, isolation, memory, permissionMode, effort, hooks, mcpServers ---