fix(agents): relocate provenance sources.md outside agents/ dir #64

Merged
Defame1297 merged 2 commits from fix/63-relocate-agent-sources-provenance into main 2026-07-05 09:15:21 +00:00
16 changed files with 125 additions and 68 deletions
Showing only changes of commit 996d9be428 - Show all commits

View File

@@ -39,3 +39,8 @@ separate single-provider skill, adding complexity with no benefit.
- The file-by-file no-op in the script (skip existing files rather than
overwriting) means partial state — one provider file exists, the other does not —
is handled by routing in the skill body, not in the script.
**Update (ADR-0010):** the `agents/sources.md` path above is superseded. The provenance
file now lives at `<plugin-root>/sources.md`, outside the `agents/` directory, because
`claude plugin validate --strict` auto-discovers every `.md` under `agents/` as an agent
requiring frontmatter. See ADR-0010 for the empirical finding and rationale.

View File

@@ -0,0 +1,58 @@
# Plugin-scope agent provenance file moves to `<plugin-root>/sources.md`
**Partially supersedes:** ADR-0005 (agent-author dual-provider scaffold) — specifically the
claim that "both files share a single `agents/sources.md` for provenance." The rest of
ADR-0005 (dual-provider generation, scope detection, single-root script interface) is
unaffected and remains in force.
`claude plugin validate --strict` auto-discovers every `.md` file directly under a plugin's
`agents/` directory and treats it as an agent definition requiring YAML frontmatter (`name`,
`description`, etc.). A flat provenance file at `agents/sources.md` — no frontmatter, by
design, since it is not an agent — fails validation with a missing-frontmatter warning that
`--strict` promotes to an error.
This was first hit in `plugins/git/agents/sources.md` (added by the git-plugin skill suite).
It failed the `validate-plugins` pre-push hook. The stopgap in commit `0239b00` added
throwaway agent frontmatter to unblock the push:
```yaml
---
name: git-agents-sources
description: Provenance record for the git plugin's agents, not an invokable agent. Do not invoke.
tools: none
---
```
That workaround is now reverted — the file no longer lives where it needs to impersonate an
agent to pass validation.
## Considered options
**Exclude via an explicit `agents` manifest array (rejected)** — `plugin.json` supports
`"agents": ["./agents/reviewer.md"]` as an alternative to `"agents": "agents/"`. The
hypothesis was that listing only real agent files would stop the validator from also
discovering `sources.md` in the same directory. Tested empirically on a scratch copy of the
git plugin: `claude plugin validate --strict` still auto-discovered and failed on the
unlisted `sources.md`, regardless of the explicit array. The manifest field controls what
Claude Code loads as agents at runtime; it does not control what the validator scans on
disk. There is no manifest-level or CLI-flag mechanism to exclude a file from `agents/`
auto-discovery.
**Keep the frontmatter workaround permanently (rejected)** — cheapest fix, already applied,
but semantically wrong: it makes a plain provenance record indistinguishable from a real
invokable agent to any tooling or UI that lists available agents (e.g. it could appear as a
callable agent in the `/agents` picker), which is confusing and incorrect.
## Consequences
- The provenance file moves to `<plugin-root>/sources.md` — a flat file, plugin-root
relative, sitting outside any directory that Claude Code or its validator auto-scans. No
frontmatter is needed or added.
- `agent-author`'s `new-agent.sh` now writes `<root>/sources.md` instead of
`<root>/agents/sources.md` at plugin scope.
- `agent-audit`'s `validate-provenance.sh` now looks for `<plugin-root>/sources.md` when
checking `source_keys` provenance chains.
- All doc and template references to `agents/sources.md` (agent-author `SKILL.md`,
agent-audit `SKILL.md`/`README.md`, both provider templates) are updated to `sources.md`.
- `plugins/git/agents/sources.md` is relocated to `plugins/git/sources.md` and the
`0239b00` frontmatter workaround is removed.

View File

@@ -13,5 +13,5 @@
],
"license": "MIT",
"name": "git",
"version": "1.3.1"
"version": "1.3.2"
}

View File

@@ -18,5 +18,5 @@
"skills": [
"skills/"
],
"version": "1.3.1"
"version": "1.3.2"
}

View File

@@ -1,9 +1,3 @@
---
name: git-agents-sources
description: Provenance record for the git plugin's agents, not an invokable agent. Do not invoke.
tools: none
---
# Sources
## context7-git-htmldocs

View File

@@ -8,5 +8,5 @@
"keywords": [],
"license": "MIT",
"name": "kyberforge",
"version": "1.2.0"
"version": "1.2.1"
}

View File

@@ -13,5 +13,5 @@
"skills": [
"skills/"
],
"version": "1.2.0"
"version": "1.2.1"
}

View File

@@ -4,7 +4,7 @@ Audits a Claude Code and Copilot agent definition file pair for correctness and
## What it does
Accepts either file in a CC `.md` / Copilot `.agent.md` pair, derives the counterpart automatically, and validates both. Runs structural checks via `validate.sh` (required fields, kebab-case name, no placeholders, no CC-only fields in the Copilot file, silently-ignored fields at plugin scope), provenance chain validation via `validate-provenance.sh` (checks `source_keys` against `agents/sources.md` at plugin scope), then qualitative checks on description phrasing and system prompt quality. Produces a compact findings report in the same format as `skill-audit`.
Accepts either file in a CC `.md` / Copilot `.agent.md` pair, derives the counterpart automatically, and validates both. Runs structural checks via `validate.sh` (required fields, kebab-case name, no placeholders, no CC-only fields in the Copilot file, silently-ignored fields at plugin scope), provenance chain validation via `validate-provenance.sh` (checks `source_keys` against `sources.md` at the plugin root), then qualitative checks on description phrasing and system prompt quality. Produces a compact findings report in the same format as `skill-audit`.
## Usage
@@ -25,7 +25,7 @@ Pass the path to either agent file as the argument.
| `references/sources.md` | Research provenance for skill content |
| `scripts/README.md` | Directory documentation for scripts/ |
| `scripts/validate.sh` | Structural validation script for agent file pairs |
| `scripts/validate-provenance.sh` | Provenance chain validation script for agent pairs against `agents/sources.md` |
| `scripts/validate-provenance.sh` | Provenance chain validation script for agent pairs against `sources.md` (plugin root) |
| `tests/README.md` | Bats test dependency and run instructions |
| `tests/validate.bats` | Bats tests for validate.sh |
| `tests/validate-provenance.bats` | Bats tests for validate-provenance.sh |

View File

@@ -37,7 +37,7 @@ bash scripts/validate-provenance.sh <path-to-agent-file>
The script accepts either the CC file or the Copilot file. It detects provider from extension, derives the counterpart, and runs all structural checks. Note FAILs and SUGGESTIONs for the `### Structure` and `### Provider safety` report dimensions. Findings about missing fields, bad name format, empty body, or missing frontmatter → `### Structure`. Findings about CC-only fields in a Copilot file, Copilot-only fields in a CC file, plugin-silently-ignored fields, body length, or subagent-unavailable tools → `### Provider safety`.
`validate-provenance.sh` validates the provenance chain between the agent pair's `source_keys` and the plugin-scoped `agents/sources.md`. It exits 0 silently for non-plugin-scope agents and when no provenance data exists. Note FAILs from this script for the `### Provenance` dimension — surface them verbatim with Why and Fix.
`validate-provenance.sh` validates the provenance chain between the agent pair's `source_keys` and the plugin-scoped `sources.md` (plugin root — see ADR-0010). It exits 0 silently for non-plugin-scope agents and when no provenance data exists. Note FAILs from this script for the `### Provenance` dimension — surface them verbatim with Why and Fix.
If the scripts cannot run (Bash denied, python3 unavailable), perform checks manually: required fields present (`name`, `description`, non-empty body), `name` is kebab-case, Copilot CLI `.agent.md` `name` must match filename stem (CC files are exempt — the CC platform does not require name to match filename), no `FILL IN:` placeholders, no CC-only fields in Copilot file, no Copilot-only fields in CC file (read `references/field-inventory.md` for the authoritative field lists).

View File

@@ -16,10 +16,10 @@ Exit codes:
1 One or more checks failed
Checks performed:
0 source_keys present in agent pair but agents/sources.md absent
1 FILL IN: placeholders in agents/sources.md
2 source_keys in agent files → slug exists in agents/sources.md
3 Contributing files listed in agents/sources.md exist on disk (plugin-root relative)
0 source_keys present in agent pair but sources.md absent
1 FILL IN: placeholders in sources.md
2 source_keys in agent files → slug exists in sources.md
3 Contributing files listed in sources.md exist on disk (plugin-root relative)
4 Contributing files back-reference the parent slug in their source_keys
5 Research doc field present and not placeholder
EOF
@@ -78,7 +78,7 @@ if provider == 'copilot':
else:
counterpart = os.path.join(agent_dir, name_stem + '.agent.md')
sources_md_path = os.path.join(plugin_root, 'agents', 'sources.md')
sources_md_path = os.path.join(plugin_root, 'sources.md')
# --- Helpers ---
PLACEHOLDER_RE = re.compile(r'(?<!`)FILL IN:[^`\n]')
@@ -188,30 +188,30 @@ if sources_md_exists:
sources_content = f.read()
sources_slugs = set(parse_h2_slugs(sources_content))
# --- Check 0: source_keys present but agents/sources.md absent ---
# --- Check 0: source_keys present but sources.md absent ---
if not sources_md_exists and all_source_keys:
rel_given = os.path.relpath(agent_file, plugin_root)
emit_fail(
"source_keys declared but agents/sources.md is absent",
"source_keys declared but sources.md is absent",
rel_given,
"source_keys references research provenance that has no sources index to validate against.",
"Create agents/sources.md with an H2 entry for each slug referenced by source_keys."
"Create sources.md with an H2 entry for each slug referenced by source_keys."
)
print_findings()
sys.exit(1)
# --- Check 1: FILL IN: placeholders in agents/sources.md ---
# --- Check 1: FILL IN: placeholders in sources.md ---
for line in sources_content.splitlines():
if PLACEHOLDER_RE.search(line):
emit_fail(
"Unfilled FILL IN: placeholder",
"agents/sources.md",
"agents/sources.md contains an unfilled placeholder, meaning provenance is incomplete.",
"Replace all 'FILL IN:' values in agents/sources.md with real content."
"sources.md",
"sources.md contains an unfilled placeholder, meaning provenance is incomplete.",
"Replace all 'FILL IN:' values in sources.md with real content."
)
break
# --- Check 2: source_keys in agent files → slug exists in agents/sources.md ---
# --- Check 2: source_keys in agent files → slug exists in sources.md ---
for fpath, keys in [(agent_file, given_keys), (counterpart, counterpart_keys)]:
if not keys:
continue
@@ -219,13 +219,13 @@ for fpath, keys in [(agent_file, given_keys), (counterpart, counterpart_keys)]:
for slug in keys:
if slug not in sources_slugs:
emit_fail(
f"source_keys slug '{slug}' not found in agents/sources.md",
f"source_keys slug '{slug}' not found in sources.md",
rel,
f"'{rel}' declares '{slug}' as a source but there is no '## {slug}' heading in agents/sources.md.",
f"Add '## {slug}' entry to agents/sources.md or remove '{slug}' from {rel} source_keys."
f"'{rel}' declares '{slug}' as a source but there is no '## {slug}' heading in sources.md.",
f"Add '## {slug}' entry to sources.md or remove '{slug}' from {rel} source_keys."
)
# --- Checks 3, 4, 5: Per-slug checks in agents/sources.md ---
# --- Checks 3, 4, 5: Per-slug checks in sources.md ---
for slug in parse_h2_slugs(sources_content):
# Check 3: Contributing files exist (paths relative to plugin root)
cf_value = parse_contributing_files(sources_content, slug)
@@ -236,9 +236,9 @@ for slug in parse_h2_slugs(sources_content):
if not os.path.isfile(cf_abs):
emit_fail(
f"Contributing file '{cf_rel}' does not exist",
f"agents/sources.md (## {slug})",
f"agents/sources.md claims '{cf_rel}' was contributed to by slug '{slug}' but the file does not exist.",
f"Create '{cf_rel}' relative to the plugin root, or correct the path in agents/sources.md."
f"sources.md (## {slug})",
f"sources.md claims '{cf_rel}' was contributed to by slug '{slug}' but the file does not exist.",
f"Create '{cf_rel}' relative to the plugin root, or correct the path in sources.md."
)
else:
# Check 4: Bidirectional — file should list slug in its source_keys
@@ -249,8 +249,8 @@ for slug in parse_h2_slugs(sources_content):
if slug not in cf_keys:
emit_fail(
f"Contributing file '{cf_rel}' does not list '{slug}' in its source_keys",
f"agents/sources.md (## {slug})",
f"agents/sources.md says '{cf_rel}' was informed by '{slug}', but '{cf_rel}' does not declare '{slug}' in its top-level source_keys.",
f"sources.md (## {slug})",
f"sources.md says '{cf_rel}' was informed by '{slug}', but '{cf_rel}' does not declare '{slug}' in its top-level source_keys.",
f"Add '{slug}' to the top-level source_keys frontmatter in '{cf_rel}'."
)
@@ -259,14 +259,14 @@ for slug in parse_h2_slugs(sources_content):
if rd_value is None:
emit_fail(
"Research doc field missing",
f"agents/sources.md (## {slug})",
f"The '## {slug}' entry in agents/sources.md has no '- **Research doc:**' line.",
f"Add '- **Research doc:** <path-or-(none)>' to the '## {slug}' entry in agents/sources.md."
f"sources.md (## {slug})",
f"The '## {slug}' entry in sources.md has no '- **Research doc:**' line.",
f"Add '- **Research doc:** <path-or-(none)>' to the '## {slug}' entry in sources.md."
)
elif rd_value == "" or PLACEHOLDER_RE.search(rd_value):
emit_fail(
"Research doc field is empty or placeholder",
f"agents/sources.md (## {slug})",
f"sources.md (## {slug})",
f"The '## {slug}' entry has an unfilled Research doc value.",
"Set '- **Research doc:**' to a real path relative to repo root, or '(none)' if not applicable."
)

View File

@@ -85,13 +85,13 @@ You are a test agent.
EOF
}
# Helper: create a valid agents/sources.md with one entry
# Helper: create a valid sources.md with one entry
make_sources_md() {
local root="$1"
local slug="${2:-my-source}"
local contrib="${3:-agents/my-agent.md, agents/my-agent.agent.md}"
local research="${4:-(none)}"
cat > "$root/agents/sources.md" <<EOF
cat > "$root/sources.md" <<EOF
# Sources
## ${slug}
@@ -155,10 +155,10 @@ EOF
}
# ---------------------------------------------------------------------------
# Check 0: source_keys present but agents/sources.md absent → FAIL
# Check 0: source_keys present but sources.md absent → FAIL
# ---------------------------------------------------------------------------
@test "FAIL: source_keys in CC file but agents/sources.md absent" {
@test "FAIL: source_keys in CC file but sources.md absent" {
local root="$TMPDIR/plugin"
make_plugin "$root"
make_cc_with_source_keys "$root"
@@ -168,7 +168,7 @@ EOF
assert_output --partial "FAIL"
}
@test "FAIL: source_keys in Copilot file but agents/sources.md absent" {
@test "FAIL: source_keys in Copilot file but sources.md absent" {
local root="$TMPDIR/plugin"
make_plugin "$root"
make_clean_pair "$root"
@@ -179,15 +179,15 @@ EOF
}
# ---------------------------------------------------------------------------
# Check 1: FILL IN: placeholder in agents/sources.md → FAIL
# Check 1: FILL IN: placeholder in sources.md → FAIL
# ---------------------------------------------------------------------------
@test "FAIL: FILL IN: placeholder in agents/sources.md" {
@test "FAIL: FILL IN: placeholder in sources.md" {
local root="$TMPDIR/plugin"
make_plugin "$root"
make_cc_with_source_keys "$root"
make_copilot_with_source_keys "$root"
cat > "$root/agents/sources.md" <<EOF
cat > "$root/sources.md" <<EOF
# Sources
## my-source
@@ -203,22 +203,22 @@ EOF
assert_output --partial "FAIL"
}
@test "FILL IN: inside backticks in agents/sources.md does not fail" {
@test "FILL IN: inside backticks in sources.md does not fail" {
local root="$TMPDIR/plugin"
make_plugin "$root"
make_cc_with_source_keys "$root"
make_copilot_with_source_keys "$root"
make_sources_md "$root"
echo "Use \`FILL IN: value\` as example." >> "$root/agents/sources.md"
echo "Use \`FILL IN: value\` as example." >> "$root/sources.md"
run bash "$SCRIPT" "$root/agents/my-agent.md"
assert_success
}
# ---------------------------------------------------------------------------
# Check 2: source_keys slug missing from agents/sources.md → FAIL
# Check 2: source_keys slug missing from sources.md → FAIL
# ---------------------------------------------------------------------------
@test "FAIL: source_keys slug in CC file not present as H2 in agents/sources.md" {
@test "FAIL: source_keys slug in CC file not present as H2 in sources.md" {
local root="$TMPDIR/plugin"
make_plugin "$root"
make_cc_with_source_keys "$root" "my-agent" "my-source"
@@ -229,7 +229,7 @@ EOF
assert_output --partial "FAIL"
}
@test "FAIL: source_keys slug in Copilot file not present as H2 in agents/sources.md" {
@test "FAIL: source_keys slug in Copilot file not present as H2 in sources.md" {
local root="$TMPDIR/plugin"
make_plugin "$root"
make_clean_pair "$root"
@@ -244,7 +244,7 @@ EOF
# Check 4: Contributing file path doesn't exist → FAIL
# ---------------------------------------------------------------------------
@test "FAIL: Contributing file listed in agents/sources.md does not exist" {
@test "FAIL: Contributing file listed in sources.md does not exist" {
local root="$TMPDIR/plugin"
make_plugin "$root"
make_cc_with_source_keys "$root"
@@ -269,12 +269,12 @@ EOF
# Check 6: Research doc field missing or placeholder → FAIL
# ---------------------------------------------------------------------------
@test "FAIL: Research doc field missing from agents/sources.md entry" {
@test "FAIL: Research doc field missing from sources.md entry" {
local root="$TMPDIR/plugin"
make_plugin "$root"
make_cc_with_source_keys "$root"
make_copilot_with_source_keys "$root"
cat > "$root/agents/sources.md" <<EOF
cat > "$root/sources.md" <<EOF
# Sources
## my-source
@@ -294,7 +294,7 @@ EOF
make_plugin "$root"
make_cc_with_source_keys "$root"
make_copilot_with_source_keys "$root"
cat > "$root/agents/sources.md" <<EOF
cat > "$root/sources.md" <<EOF
# Sources
## my-source
@@ -330,7 +330,7 @@ You are a test agent.
EOF
make_copilot_clean "$root"
# sources.md says my-agent.md contributed to my-source, but my-agent.md doesn't list my-source
cat > "$root/agents/sources.md" <<EOF
cat > "$root/sources.md" <<EOF
# Sources
## other-source

View File

@@ -109,7 +109,7 @@ Open the scaffolded Claude Code file. Replace every `FILL IN:` placeholder. **Re
- `initialPrompt`: auto-submitted as the first turn when this agent activates as the main session thread; only set when this agent is intended for main-thread activation
- `background`: set `true` to force background execution
**`source_keys`** — top-level list of research source slugs that informed this agent. Add only when research sources were used (i.e. entries with `` `extracted` `` status are in context from a prior `/research` session). Each slug must match an H2 heading in `agents/sources.md`. Omit entirely when no research was used.
**`source_keys`** — top-level list of research source slugs that informed this agent. Add only when research sources were used (i.e. entries with `` `extracted` `` status are in context from a prior `/research` session). Each slug must match an H2 heading in `sources.md` (plugin root). Omit entirely when no research was used.
```yaml
source_keys:
@@ -145,14 +145,14 @@ There are **two distinct Copilot agent formats** with different paths and field
The system prompt body should match the Claude Code version — the agent's task definition is the same across providers.
### Step 4 — Populate or delete `agents/sources.md` (plugin scope only)
### Step 4 — Populate or delete `sources.md` (plugin scope only)
Skip this step at project and user scope.
Skip this step at project and user scope. The file lives at the plugin root, not inside `agents/` — `claude plugin validate --strict` auto-discovers every `.md` under `agents/` as an agent requiring frontmatter, and a flat provenance file would fail that check (see ADR-0010).
If a research `sources.md` is present in the conversation context:
1. Filter to entries with `` `extracted` `` status only.
2. For each entry, identify which agent files in the pair it contributed to.
3. Write `agents/sources.md` using the format below. Paths in `Contributing files:` are relative to the plugin root.
3. Write `sources.md` at the plugin root using the format below. Paths in `Contributing files:` are relative to the plugin root.
```markdown
# Sources
@@ -168,7 +168,7 @@ If a research `sources.md` is present in the conversation context:
Each slug must match an H2 heading, and each slug must also appear in the `source_keys` list of every file listed under `Contributing files:`.
If no research sources are in context, delete `agents/sources.md`.
If no research sources are in context, delete `sources.md`.
### Step 5 — Validate and close

View File

@@ -62,7 +62,7 @@ description: FILL IN: Action-first description of what this agent does and when
<!-- source_keys:
- slug-name
Development-only. Add when research sources informed this agent (slugs must match agents/sources.md).
Development-only. Add when research sources informed this agent (slugs must match sources.md at the plugin root).
Omit when no research was used. Not a runtime field — silently ignored by Claude Code. -->
<!-- NOTE: hooks, mcpServers, and permissionMode are silently ignored for plugin agents.

View File

@@ -35,7 +35,7 @@ description: FILL IN: Action-first description of what this agent does and when
<!-- source_keys:
- slug-name
Development-only. Add when research sources informed this agent (slugs must match agents/sources.md).
Development-only. Add when research sources informed this agent (slugs must match sources.md at the plugin root).
Omit when no research was used. Not a Copilot runtime field — silently ignored. -->
<!-- DO NOT include these Claude Code-only fields:

View File

@@ -17,7 +17,7 @@ Arguments:
plugin scope : root contains plugin.json
→ creates <root>/agents/<name>.md
→ creates <root>/agents/<name>.agent.md
→ creates <root>/agents/sources.md (if absent)
→ creates <root>/sources.md (if absent)
project scope : root is a project directory (no plugin.json)
→ creates <root>/.claude/agents/<name>.md
→ creates <root>/.github/agents/<name>.agent.md
@@ -86,7 +86,7 @@ case "$SCOPE" in
plugin)
CC_DIR="$ROOT/agents"
CP_DIR="$ROOT/agents"
SOURCES_DIR="$ROOT/agents"
SOURCES_DIR="$ROOT"
;;
project)
CC_DIR="$ROOT/.claude/agents"

View File

@@ -42,11 +42,11 @@ teardown() {
assert [ -d "$ROOT/agents" ]
}
@test "plugin scope: creates sources.md in agents/" {
@test "plugin scope: creates sources.md at plugin root" {
touch "$ROOT/plugin.json"
run bash "$SCRIPT" my-agent "$ROOT"
assert_success
assert [ -f "$ROOT/agents/sources.md" ]
assert [ -f "$ROOT/sources.md" ]
}
@test "plugin scope: no-op if claude code file already exists" {