feat(kyberforge): add agent-audit provenance chain validation (closes #60)
## Why agent-author produces agents/sources.md at plugin scope to record which research sources informed which agent files. agent-audit had no way to validate this chain, leaving stale or missing provenance undetected. ## Implementation Notes Validation is per-pair (the given agent file + its counterpart) rather than plugin-wide, keeping the scope consistent with validate.sh. The script exits 0 silently for non-plugin-scope agents. source_keys is top-level in both CC .md and Copilot .agent.md files (not under metadata:) to avoid conflict with Copilot's own metadata field semantics. Checks 0, 1, 2, 4, 5, 6 mirror the skill provenance set; upstream research-doc cross-reference checks (7, 8) are deferred. agent-author Steps 2, 3, and 4 updated to formally specify the agents/sources.md format and instruct authors to add source_keys to both files when research sources are in context. Refs: #60 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0147vXtL5sP6vorDdqXGJJU9
This commit is contained in:
@@ -4,7 +4,7 @@ Audits a Claude Code and Copilot agent definition file pair for correctness and
|
||||
|
||||
## What it does
|
||||
|
||||
Accepts either file in a CC `.md` / Copilot `.agent.md` pair, derives the counterpart automatically, and validates both. Runs structural checks via `validate.sh` (required fields, kebab-case name, no placeholders, no CC-only fields in the Copilot file, silently-ignored fields at plugin scope), then qualitative checks on description phrasing and system prompt quality. Produces a compact findings report in the same format as `skill-audit`.
|
||||
Accepts either file in a CC `.md` / Copilot `.agent.md` pair, derives the counterpart automatically, and validates both. Runs structural checks via `validate.sh` (required fields, kebab-case name, no placeholders, no CC-only fields in the Copilot file, silently-ignored fields at plugin scope), provenance chain validation via `validate-provenance.sh` (checks `source_keys` against `agents/sources.md` at plugin scope), then qualitative checks on description phrasing and system prompt quality. Produces a compact findings report in the same format as `skill-audit`.
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -24,5 +24,7 @@ Pass the path to either agent file as the argument.
|
||||
| `references/sources.md` | Research provenance for skill content |
|
||||
| `scripts/README.md` | Directory documentation for scripts/ |
|
||||
| `scripts/validate.sh` | Structural validation script for agent file pairs |
|
||||
| `scripts/validate-provenance.sh` | Provenance chain validation script for agent pairs against `agents/sources.md` |
|
||||
| `tests/README.md` | Bats test dependency and run instructions |
|
||||
| `tests/validate.bats` | Bats tests for validate.sh |
|
||||
| `tests/validate-provenance.bats` | Bats tests for validate-provenance.sh |
|
||||
|
||||
@@ -31,11 +31,14 @@ metadata:
|
||||
|
||||
```bash
|
||||
bash scripts/validate.sh <path-to-agent-file>
|
||||
bash scripts/validate-provenance.sh <path-to-agent-file>
|
||||
```
|
||||
|
||||
The script accepts either the CC file or the Copilot file. It detects provider from extension, derives the counterpart, and runs all structural checks. Note FAILs for the `### Structure` and `### Provider safety` report dimensions. Findings about missing fields, bad name format, empty body, or missing frontmatter → `### Structure`. Findings about CC-only fields in a Copilot file or plugin-silently-ignored fields in a CC file → `### Provider safety`.
|
||||
|
||||
If the script cannot run (Bash denied, python3 unavailable), perform checks manually: required fields present (`name`, `description`, non-empty body), `name` is kebab-case, `name` matches filename stem, no `FILL IN:` placeholders, no CC-only fields in Copilot file.
|
||||
`validate-provenance.sh` validates the provenance chain between the agent pair's `source_keys` and the plugin-scoped `agents/sources.md`. It exits 0 silently for non-plugin-scope agents and when no provenance data exists. Note FAILs from this script for the `### Provenance` dimension — surface them verbatim with Why and Fix.
|
||||
|
||||
If the scripts cannot run (Bash denied, python3 unavailable), perform checks manually: required fields present (`name`, `description`, non-empty body), `name` is kebab-case, `name` matches filename stem, no `FILL IN:` placeholders, no CC-only fields in Copilot file.
|
||||
|
||||
## Step 2 — Qualitative checks
|
||||
|
||||
@@ -59,10 +62,10 @@ Read both agent files. Work through each dimension internally. Collect findings
|
||||
Open with a coverage line:
|
||||
|
||||
```text
|
||||
Checked: structure · provider-safety · description · body · pair-consistency
|
||||
Checked: structure · provider-safety · description · body · pair-consistency · provenance
|
||||
```
|
||||
|
||||
Then output only dimensions that have findings, grouped under H3 headings, FAILs before SUGGESTIONs within each dimension. Omit clean dimensions entirely.
|
||||
Then output only dimensions that have findings, grouped under H3 headings, FAILs before SUGGESTIONs within each dimension. Omit clean dimensions entirely. `### Provenance` findings are sourced verbatim from `validate-provenance.sh` output — copy them without rephrasing.
|
||||
|
||||
For each finding:
|
||||
|
||||
|
||||
276
plugins/kyberforge/skills/agent-audit/scripts/validate-provenance.sh
Executable file
276
plugins/kyberforge/skills/agent-audit/scripts/validate-provenance.sh
Executable file
@@ -0,0 +1,276 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: validate-provenance.sh <agent-file>
|
||||
|
||||
Validate that an agent pair's sources provenance chain is complete and internally consistent.
|
||||
Operates at plugin scope only — exits 0 silently for project and user scope agents.
|
||||
|
||||
Arguments:
|
||||
agent-file Path to either the Claude Code .md or Copilot .agent.md agent file.
|
||||
|
||||
Exit codes:
|
||||
0 All checks passed (or nothing to validate, or not plugin scope)
|
||||
1 One or more checks failed
|
||||
|
||||
Checks performed:
|
||||
0 source_keys present in agent pair but agents/sources.md absent
|
||||
1 FILL IN: placeholders in agents/sources.md
|
||||
2 source_keys in agent files → slug exists in agents/sources.md
|
||||
4 Contributing files listed in agents/sources.md exist on disk (plugin-root relative)
|
||||
5 Contributing files back-reference the parent slug in their source_keys
|
||||
6 Research doc field present and not placeholder
|
||||
EOF
|
||||
}
|
||||
|
||||
if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then
|
||||
usage
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ $# -lt 1 ]]; then
|
||||
echo "Error: agent-file is required." >&2
|
||||
echo "" >&2
|
||||
usage >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
python3 -u - "$1" <<'PYTHON'
|
||||
import sys
|
||||
import os
|
||||
import re
|
||||
|
||||
agent_file = os.path.abspath(sys.argv[1])
|
||||
fname = os.path.basename(agent_file)
|
||||
agent_dir = os.path.dirname(agent_file)
|
||||
|
||||
# --- Detect provider ---
|
||||
if fname.endswith('.agent.md'):
|
||||
provider = 'copilot'
|
||||
name_stem = fname[:-len('.agent.md')]
|
||||
elif fname.endswith('.md'):
|
||||
provider = 'claude-code'
|
||||
name_stem = fname[:-len('.md')]
|
||||
else:
|
||||
print(f"Error: unrecognized extension '{fname}' — expected .md or .agent.md", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
# --- Find plugin root ---
|
||||
def find_plugin_root(start_dir):
|
||||
current = os.path.abspath(start_dir)
|
||||
while True:
|
||||
if os.path.isfile(os.path.join(current, 'plugin.json')):
|
||||
return current
|
||||
parent = os.path.dirname(current)
|
||||
if parent == current:
|
||||
return None
|
||||
current = parent
|
||||
|
||||
plugin_root = find_plugin_root(agent_dir)
|
||||
if plugin_root is None:
|
||||
sys.exit(0)
|
||||
|
||||
# --- Derive counterpart ---
|
||||
if provider == 'copilot':
|
||||
counterpart = os.path.join(agent_dir, name_stem + '.md')
|
||||
else:
|
||||
counterpart = os.path.join(agent_dir, name_stem + '.agent.md')
|
||||
|
||||
sources_md_path = os.path.join(plugin_root, 'agents', 'sources.md')
|
||||
|
||||
# --- Helpers ---
|
||||
PLACEHOLDER_RE = re.compile(r'(?<!`)FILL IN:[^`\n]')
|
||||
|
||||
def parse_frontmatter(content):
|
||||
m = re.match(r'^---\n(.*?)\n---', content, re.DOTALL)
|
||||
if not m:
|
||||
return None, content
|
||||
return m.group(1), content[m.end():]
|
||||
|
||||
def parse_source_keys(fm):
|
||||
"""Extract top-level source_keys list from frontmatter string."""
|
||||
if fm is None:
|
||||
return []
|
||||
keys = []
|
||||
in_source_keys = False
|
||||
for line in fm.splitlines():
|
||||
if re.match(r'^source_keys:', line):
|
||||
in_source_keys = True
|
||||
continue
|
||||
if in_source_keys:
|
||||
m = re.match(r'^[ \t]+-\s+(\S+)', line)
|
||||
if m:
|
||||
keys.append(m.group(1).strip())
|
||||
elif line and not line[0].isspace():
|
||||
in_source_keys = False
|
||||
return keys
|
||||
|
||||
def parse_h2_slugs(content):
|
||||
return re.findall(r'^## (.+)$', content, re.MULTILINE)
|
||||
|
||||
def parse_contributing_files(content, slug):
|
||||
pattern = re.compile(
|
||||
r'^## ' + re.escape(slug) + r'\s*\n(.*?)(?=^## |\Z)',
|
||||
re.MULTILINE | re.DOTALL
|
||||
)
|
||||
m = pattern.search(content)
|
||||
if not m:
|
||||
return None
|
||||
block = m.group(1)
|
||||
cf_m = re.search(r'^\- \*\*Contributing files:\*\* (.+)$', block, re.MULTILINE)
|
||||
if not cf_m:
|
||||
return None
|
||||
return cf_m.group(1).strip()
|
||||
|
||||
def parse_research_doc(content, slug):
|
||||
pattern = re.compile(
|
||||
r'^## ' + re.escape(slug) + r'\s*\n(.*?)(?=^## |\Z)',
|
||||
re.MULTILINE | re.DOTALL
|
||||
)
|
||||
m = pattern.search(content)
|
||||
if not m:
|
||||
return None
|
||||
block = m.group(1)
|
||||
rd_m = re.search(r'^\- \*\*Research doc:\*\* (.+)$', block, re.MULTILINE)
|
||||
if not rd_m:
|
||||
return None
|
||||
return rd_m.group(1).strip()
|
||||
|
||||
findings = []
|
||||
has_fail = False
|
||||
|
||||
def emit_fail(desc, fpath, why, fix):
|
||||
global has_fail
|
||||
has_fail = True
|
||||
findings.append(("FAIL", desc, fpath, why, fix))
|
||||
|
||||
def print_findings():
|
||||
for kind, desc, fpath, why, fix in findings:
|
||||
print(f"FAIL {desc} — {fpath}")
|
||||
print(f" Why: {why}")
|
||||
print(f" Fix: {fix}")
|
||||
print()
|
||||
|
||||
# --- Collect source_keys from agent pair ---
|
||||
def get_source_keys_from_file(fpath):
|
||||
if not os.path.isfile(fpath):
|
||||
return []
|
||||
try:
|
||||
with open(fpath) as f:
|
||||
content = f.read()
|
||||
except Exception:
|
||||
return []
|
||||
fm, _ = parse_frontmatter(content)
|
||||
return parse_source_keys(fm)
|
||||
|
||||
given_keys = get_source_keys_from_file(agent_file)
|
||||
counterpart_keys = get_source_keys_from_file(counterpart)
|
||||
# Deduplicated union, preserving order
|
||||
seen = set()
|
||||
all_source_keys = []
|
||||
for k in given_keys + counterpart_keys:
|
||||
if k not in seen:
|
||||
seen.add(k)
|
||||
all_source_keys.append(k)
|
||||
|
||||
sources_md_exists = os.path.isfile(sources_md_path)
|
||||
|
||||
# Early exit: nothing to validate
|
||||
if not all_source_keys and not sources_md_exists:
|
||||
sys.exit(0)
|
||||
|
||||
sources_content = None
|
||||
sources_slugs = set()
|
||||
if sources_md_exists:
|
||||
with open(sources_md_path) as f:
|
||||
sources_content = f.read()
|
||||
sources_slugs = set(parse_h2_slugs(sources_content))
|
||||
|
||||
# --- Check 0: source_keys present but agents/sources.md absent ---
|
||||
if not sources_md_exists and all_source_keys:
|
||||
rel_given = os.path.relpath(agent_file, plugin_root)
|
||||
emit_fail(
|
||||
"source_keys declared but agents/sources.md is absent",
|
||||
rel_given,
|
||||
"source_keys references research provenance that has no sources index to validate against.",
|
||||
"Create agents/sources.md with an H2 entry for each slug referenced by source_keys."
|
||||
)
|
||||
print_findings()
|
||||
sys.exit(1)
|
||||
|
||||
# --- Check 1: FILL IN: placeholders in agents/sources.md ---
|
||||
for line in sources_content.splitlines():
|
||||
if PLACEHOLDER_RE.search(line):
|
||||
emit_fail(
|
||||
"Unfilled FILL IN: placeholder",
|
||||
"agents/sources.md",
|
||||
"agents/sources.md contains an unfilled placeholder, meaning provenance is incomplete.",
|
||||
"Replace all 'FILL IN:' values in agents/sources.md with real content."
|
||||
)
|
||||
break
|
||||
|
||||
# --- Check 2: source_keys in agent files → slug exists in agents/sources.md ---
|
||||
for fpath, keys in [(agent_file, given_keys), (counterpart, counterpart_keys)]:
|
||||
if not keys:
|
||||
continue
|
||||
rel = os.path.relpath(fpath, plugin_root)
|
||||
for slug in keys:
|
||||
if slug not in sources_slugs:
|
||||
emit_fail(
|
||||
f"source_keys slug '{slug}' not found in agents/sources.md",
|
||||
rel,
|
||||
f"'{rel}' declares '{slug}' as a source but there is no '## {slug}' heading in agents/sources.md.",
|
||||
f"Add '## {slug}' entry to agents/sources.md or remove '{slug}' from {rel} source_keys."
|
||||
)
|
||||
|
||||
# --- Checks 4, 5, 6: Per-slug checks in agents/sources.md ---
|
||||
for slug in parse_h2_slugs(sources_content):
|
||||
# Check 4: Contributing files exist (paths relative to plugin root)
|
||||
cf_value = parse_contributing_files(sources_content, slug)
|
||||
if cf_value and not cf_value.startswith("(none"):
|
||||
cf_files = [p.strip() for p in cf_value.split(",") if p.strip()]
|
||||
for cf_rel in cf_files:
|
||||
cf_abs = os.path.join(plugin_root, cf_rel)
|
||||
if not os.path.isfile(cf_abs):
|
||||
emit_fail(
|
||||
f"Contributing file '{cf_rel}' does not exist",
|
||||
f"agents/sources.md (## {slug})",
|
||||
f"agents/sources.md claims '{cf_rel}' was contributed to by slug '{slug}' but the file does not exist.",
|
||||
f"Create '{cf_rel}' relative to the plugin root, or correct the path in agents/sources.md."
|
||||
)
|
||||
else:
|
||||
# Check 5: Bidirectional — file should list slug in its source_keys
|
||||
with open(cf_abs) as f:
|
||||
cf_content = f.read()
|
||||
cf_fm, _ = parse_frontmatter(cf_content)
|
||||
cf_keys = parse_source_keys(cf_fm)
|
||||
if slug not in cf_keys:
|
||||
emit_fail(
|
||||
f"Contributing file '{cf_rel}' does not list '{slug}' in its source_keys",
|
||||
f"agents/sources.md (## {slug})",
|
||||
f"agents/sources.md says '{cf_rel}' was informed by '{slug}', but '{cf_rel}' does not declare '{slug}' in its top-level source_keys.",
|
||||
f"Add '{slug}' to the top-level source_keys frontmatter in '{cf_rel}'."
|
||||
)
|
||||
|
||||
# Check 6: Research doc field required
|
||||
rd_value = parse_research_doc(sources_content, slug)
|
||||
if rd_value is None:
|
||||
emit_fail(
|
||||
"Research doc field missing",
|
||||
f"agents/sources.md (## {slug})",
|
||||
f"The '## {slug}' entry in agents/sources.md has no '- **Research doc:**' line.",
|
||||
f"Add '- **Research doc:** <path-or-(none)>' to the '## {slug}' entry in agents/sources.md."
|
||||
)
|
||||
elif rd_value == "" or PLACEHOLDER_RE.search(rd_value):
|
||||
emit_fail(
|
||||
"Research doc field is empty or placeholder",
|
||||
f"agents/sources.md (## {slug})",
|
||||
f"The '## {slug}' entry has an unfilled Research doc value.",
|
||||
"Set '- **Research doc:**' to a real path relative to repo root, or '(none)' if not applicable."
|
||||
)
|
||||
|
||||
print_findings()
|
||||
sys.exit(1 if has_fail else 0)
|
||||
PYTHON
|
||||
@@ -0,0 +1,383 @@
|
||||
#!/usr/bin/env bats
|
||||
|
||||
setup() {
|
||||
REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../../../../../" && pwd)"
|
||||
load "$REPO_ROOT/tests/test_helper/bats-support/load"
|
||||
load "$REPO_ROOT/tests/test_helper/bats-assert/load"
|
||||
|
||||
SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate-provenance.sh"
|
||||
TMPDIR="$(mktemp -d)"
|
||||
|
||||
# Helper: create a plugin root with plugin.json and an agents/ directory
|
||||
make_plugin() {
|
||||
local root="$1"
|
||||
mkdir -p "$root/agents"
|
||||
echo '{"name":"test-plugin","version":"0.1.0"}' > "$root/plugin.json"
|
||||
}
|
||||
|
||||
# Helper: create a clean agent pair (no source_keys)
|
||||
make_clean_pair() {
|
||||
local root="$1"
|
||||
local name="${2:-my-agent}"
|
||||
cat > "$root/agents/${name}.md" <<EOF
|
||||
---
|
||||
name: ${name}
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent.
|
||||
EOF
|
||||
cat > "$root/agents/${name}.agent.md" <<EOF
|
||||
---
|
||||
name: ${name}
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent.
|
||||
EOF
|
||||
}
|
||||
|
||||
# Helper: create a CC agent file with source_keys
|
||||
make_cc_with_source_keys() {
|
||||
local root="$1"
|
||||
local name="${2:-my-agent}"
|
||||
local slug="${3:-my-source}"
|
||||
cat > "$root/agents/${name}.md" <<EOF
|
||||
---
|
||||
name: ${name}
|
||||
description: A valid agent description.
|
||||
source_keys:
|
||||
- ${slug}
|
||||
---
|
||||
|
||||
You are a test agent.
|
||||
EOF
|
||||
}
|
||||
|
||||
# Helper: create a Copilot agent file with source_keys
|
||||
make_copilot_with_source_keys() {
|
||||
local root="$1"
|
||||
local name="${2:-my-agent}"
|
||||
local slug="${3:-my-source}"
|
||||
cat > "$root/agents/${name}.agent.md" <<EOF
|
||||
---
|
||||
name: ${name}
|
||||
description: A valid agent description.
|
||||
source_keys:
|
||||
- ${slug}
|
||||
---
|
||||
|
||||
You are a test agent.
|
||||
EOF
|
||||
}
|
||||
|
||||
# Helper: create a minimal Copilot file without source_keys
|
||||
make_copilot_clean() {
|
||||
local root="$1"
|
||||
local name="${2:-my-agent}"
|
||||
cat > "$root/agents/${name}.agent.md" <<EOF
|
||||
---
|
||||
name: ${name}
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent.
|
||||
EOF
|
||||
}
|
||||
|
||||
# Helper: create a valid agents/sources.md with one entry
|
||||
make_sources_md() {
|
||||
local root="$1"
|
||||
local slug="${2:-my-source}"
|
||||
local contrib="${3:-agents/my-agent.md, agents/my-agent.agent.md}"
|
||||
local research="${4:-(none)}"
|
||||
cat > "$root/agents/sources.md" <<EOF
|
||||
# Sources
|
||||
|
||||
## ${slug}
|
||||
|
||||
- **URL:** https://example.com/${slug}
|
||||
- **Description:** A test source.
|
||||
- **Contributing files:** ${contrib}
|
||||
- **Research doc:** ${research}
|
||||
- **Status:** \`extracted\`
|
||||
EOF
|
||||
}
|
||||
}
|
||||
|
||||
teardown() {
|
||||
rm -rf "$TMPDIR"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# --help
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "--help exits 0" {
|
||||
run bash "$SCRIPT" --help
|
||||
assert_success
|
||||
assert_output --partial "Usage:"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Non-plugin scope → exit 0 silently
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "non-plugin scope: no plugin.json in tree → exit 0, no output" {
|
||||
local dir="$TMPDIR/no-plugin"
|
||||
mkdir -p "$dir/agents"
|
||||
cat > "$dir/agents/my-agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
source_keys:
|
||||
- my-source
|
||||
---
|
||||
|
||||
You are a test agent.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$dir/agents/my-agent.md"
|
||||
assert_success
|
||||
assert_output ""
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Early exit: no sources.md, no source_keys → exit 0, no output
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "clean pass: no sources.md and no source_keys → exit 0, no output" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_clean_pair "$root"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
assert_success
|
||||
assert_output ""
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Check 0: source_keys present but agents/sources.md absent → FAIL
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "FAIL: source_keys in CC file but agents/sources.md absent" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root"
|
||||
make_copilot_clean "$root"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL"
|
||||
}
|
||||
|
||||
@test "FAIL: source_keys in Copilot file but agents/sources.md absent" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_clean_pair "$root"
|
||||
make_copilot_with_source_keys "$root"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Check 1: FILL IN: placeholder in agents/sources.md → FAIL
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "FAIL: FILL IN: placeholder in agents/sources.md" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root"
|
||||
make_copilot_with_source_keys "$root"
|
||||
cat > "$root/agents/sources.md" <<EOF
|
||||
# Sources
|
||||
|
||||
## my-source
|
||||
|
||||
- **URL:** FILL IN: add url
|
||||
- **Description:** A test source.
|
||||
- **Contributing files:** agents/my-agent.md, agents/my-agent.agent.md
|
||||
- **Research doc:** (none)
|
||||
- **Status:** \`extracted\`
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL"
|
||||
}
|
||||
|
||||
@test "FILL IN: inside backticks in agents/sources.md does not fail" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root"
|
||||
make_copilot_with_source_keys "$root"
|
||||
make_sources_md "$root"
|
||||
echo "Use \`FILL IN: value\` as example." >> "$root/agents/sources.md"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
assert_success
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Check 2: source_keys slug missing from agents/sources.md → FAIL
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "FAIL: source_keys slug in CC file not present as H2 in agents/sources.md" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root" "my-agent" "my-source"
|
||||
make_copilot_clean "$root"
|
||||
make_sources_md "$root" "different-source" "(none)" "(none)"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL"
|
||||
}
|
||||
|
||||
@test "FAIL: source_keys slug in Copilot file not present as H2 in agents/sources.md" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_clean_pair "$root"
|
||||
make_copilot_with_source_keys "$root" "my-agent" "my-source"
|
||||
make_sources_md "$root" "different-source" "(none)" "(none)"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Check 4: Contributing file path doesn't exist → FAIL
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "FAIL: Contributing file listed in agents/sources.md does not exist" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root"
|
||||
make_copilot_with_source_keys "$root"
|
||||
make_sources_md "$root" "my-source" "agents/nonexistent.md"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL"
|
||||
}
|
||||
|
||||
@test "pass: (none) in Contributing files is skipped" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root"
|
||||
make_copilot_with_source_keys "$root"
|
||||
make_sources_md "$root" "my-source" "(none — not used directly)"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
assert_success
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Check 6: Research doc field missing or placeholder → FAIL
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "FAIL: Research doc field missing from agents/sources.md entry" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root"
|
||||
make_copilot_with_source_keys "$root"
|
||||
cat > "$root/agents/sources.md" <<EOF
|
||||
# Sources
|
||||
|
||||
## my-source
|
||||
|
||||
- **URL:** https://example.com/my-source
|
||||
- **Description:** A test source.
|
||||
- **Contributing files:** agents/my-agent.md, agents/my-agent.agent.md
|
||||
- **Status:** \`extracted\`
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL"
|
||||
}
|
||||
|
||||
@test "FAIL: Research doc field is FILL IN: placeholder" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root"
|
||||
make_copilot_with_source_keys "$root"
|
||||
cat > "$root/agents/sources.md" <<EOF
|
||||
# Sources
|
||||
|
||||
## my-source
|
||||
|
||||
- **URL:** https://example.com/my-source
|
||||
- **Description:** A test source.
|
||||
- **Contributing files:** agents/my-agent.md, agents/my-agent.agent.md
|
||||
- **Research doc:** FILL IN: path to research doc
|
||||
- **Status:** \`extracted\`
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Check 5: Bidirectional — contributing file missing slug in source_keys → FAIL
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "FAIL: Contributing file exists but does not list parent slug in source_keys" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
# CC file has source_keys: other-source (not my-source)
|
||||
cat > "$root/agents/my-agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
source_keys:
|
||||
- other-source
|
||||
---
|
||||
|
||||
You are a test agent.
|
||||
EOF
|
||||
make_copilot_clean "$root"
|
||||
# sources.md says my-agent.md contributed to my-source, but my-agent.md doesn't list my-source
|
||||
cat > "$root/agents/sources.md" <<EOF
|
||||
# Sources
|
||||
|
||||
## other-source
|
||||
|
||||
- **URL:** https://example.com/other-source
|
||||
- **Description:** A test source.
|
||||
- **Contributing files:** agents/my-agent.md
|
||||
- **Research doc:** (none)
|
||||
- **Status:** \`extracted\`
|
||||
|
||||
## my-source
|
||||
|
||||
- **URL:** https://example.com/my-source
|
||||
- **Description:** Another source.
|
||||
- **Contributing files:** agents/my-agent.md
|
||||
- **Research doc:** (none)
|
||||
- **Status:** \`extracted\`
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Entry via Copilot file path
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "accepts Copilot file path as entry point" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root"
|
||||
make_copilot_with_source_keys "$root"
|
||||
make_sources_md "$root"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.agent.md"
|
||||
assert_success
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Clean full pass
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "clean full pass: all checks satisfied via CC file" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root"
|
||||
make_copilot_with_source_keys "$root"
|
||||
make_sources_md "$root"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
assert_success
|
||||
}
|
||||
@@ -102,6 +102,13 @@ Open the scaffolded Claude Code file. Replace every `FILL IN:` placeholder.
|
||||
- `memory`: `user`, `project`, or `local` — only when cross-session state is genuinely needed
|
||||
- `isolation: worktree` — only when the agent modifies files and needs an isolated copy
|
||||
|
||||
**`source_keys`** — top-level list of research source slugs that informed this agent. Add only when research sources were used (i.e. entries with `` `extracted` `` status are in context from a prior `/research` session). Each slug must match an H2 heading in `agents/sources.md`. Omit entirely when no research was used.
|
||||
|
||||
```yaml
|
||||
source_keys:
|
||||
- my-source-slug
|
||||
```
|
||||
|
||||
**System prompt body** — write as a direct role instruction:
|
||||
- Open with: "You are a [role]. When invoked, [primary action]."
|
||||
- Cover: inputs expected, process steps, output format, error handling
|
||||
@@ -119,6 +126,8 @@ Open the scaffolded Copilot file. Replace every `FILL IN:` placeholder.
|
||||
|
||||
**Do not include Claude Code-only fields**: `maxTurns`, `isolation`, `memory`, `permissionMode`, `effort`, `hooks`, `mcpServers`.
|
||||
|
||||
**`source_keys`** — add the same top-level list as the CC file when research sources were used. Omit when no research was used.
|
||||
|
||||
The system prompt body should match the Claude Code version — the agent's task definition is the same across providers.
|
||||
|
||||
### Step 4 — Populate or delete `agents/sources.md` (plugin scope only)
|
||||
@@ -127,8 +136,22 @@ Skip this step at project and user scope.
|
||||
|
||||
If a research `sources.md` is present in the conversation context:
|
||||
1. Filter to entries with `` `extracted` `` status only.
|
||||
2. For each entry, note which agent files it contributed to.
|
||||
3. Write `agents/sources.md` with those entries. Include `- **Research doc:** <path>` per entry pointing to the upstream research sources file.
|
||||
2. For each entry, identify which agent files in the pair it contributed to.
|
||||
3. Write `agents/sources.md` using the format below. Paths in `Contributing files:` are relative to the plugin root.
|
||||
|
||||
```markdown
|
||||
# Sources
|
||||
|
||||
## slug-name
|
||||
|
||||
- **URL:** <source URL>
|
||||
- **Research doc:** <path/to/research/sources.md relative to repo root>
|
||||
- **Description:** <what this source covers>
|
||||
- **Contributing files:** agents/<name>.md, agents/<name>.agent.md
|
||||
- **Status:** `extracted`
|
||||
```
|
||||
|
||||
Each slug must match an H2 heading, and each slug must also appear in the `source_keys` list of every file listed under `Contributing files:`.
|
||||
|
||||
If no research sources are in context, delete `agents/sources.md`.
|
||||
|
||||
|
||||
@@ -45,6 +45,11 @@ description: FILL IN: Action-first description of what this agent does and when
|
||||
# background: false
|
||||
# Optional. Set true to force background execution.
|
||||
|
||||
# source_keys:
|
||||
# - slug-name
|
||||
# Development-only. Add when research sources informed this agent (slugs must match agents/sources.md).
|
||||
# Omit when no research was used. Not a runtime field — silently ignored by Claude Code.
|
||||
|
||||
# NOTE: hooks, mcpServers, and permissionMode are silently ignored for plugin agents.
|
||||
# Those fields only work in .claude/agents/ or ~/.claude/agents/.
|
||||
---
|
||||
|
||||
@@ -37,6 +37,11 @@ description: FILL IN: Action-first description of what this agent does and when
|
||||
# model: claude-sonnet-4-5
|
||||
# Optional. Model to run this agent on.
|
||||
|
||||
# source_keys:
|
||||
# - slug-name
|
||||
# Development-only. Add when research sources informed this agent (slugs must match agents/sources.md).
|
||||
# Omit when no research was used. Not a Copilot runtime field — silently ignored.
|
||||
|
||||
# DO NOT include these Claude Code-only fields:
|
||||
# maxTurns, isolation, memory, permissionMode, effort, hooks, mcpServers
|
||||
---
|
||||
|
||||
Reference in New Issue
Block a user