diff --git a/docs/adr/0018-agent-audit-single-file-invocation.md b/docs/adr/0018-agent-audit-single-file-invocation.md new file mode 100644 index 0000000..642e675 --- /dev/null +++ b/docs/adr/0018-agent-audit-single-file-invocation.md @@ -0,0 +1,16 @@ +# agent-audit takes a single file path and derives the counterpart by scope detection + +`agent-audit` validates agent definition file pairs (Claude Code `.md` + Copilot `.agent.md`). The skill accepts a path to either file and derives the counterpart using scope detection rather than requiring the caller to name both files or supply a root directory. + +## Considered options + +**Directory input (rejected)** — analogous to `skill-audit `. Rejected because agents have no per-agent directory. At plugin scope both files are flat in `agents/`; at project scope they are in completely different directories (`.claude/agents/` and `.github/agents/`). No single directory contains both files across all scopes. + +**` ` signature (rejected)** — mirrors `new-agent.sh `. Rejected because it requires the caller to supply two pieces of information when one (the file path) is sufficient. The file path already implies the agent name (filename stem) and the root (found by walking up). Forcing the caller to re-supply what the script can infer is the kind of convention knowledge the script exists to encapsulate. + +## Consequences + +- The unit of validation is the pair. A missing counterpart is always a FAIL — an orphan file is incomplete by definition. +- Scope detection walks up from the input file: first directory containing `plugin.json` → plugin scope; first directory containing `.git` without `plugin.json` → project scope; path under `~` with neither → user scope. +- At user scope the derivation crosses filesystem locations (`~/.claude/agents/` ↔ `~/.copilot/agents/`); the script must handle the home directory case explicitly. +- The invocation signature is the public contract. Changing it is a breaking change to any caller — treat it as such. diff --git a/docs/adr/0019-agent-audit-field-inventory-reference.md b/docs/adr/0019-agent-audit-field-inventory-reference.md new file mode 100644 index 0000000..bc7c3e0 --- /dev/null +++ b/docs/adr/0019-agent-audit-field-inventory-reference.md @@ -0,0 +1,30 @@ +# agent-audit reads field lists from a reference file, not hardcoded script arrays + +`agent-audit`'s `validate.sh` checks for Claude Code-only fields in Copilot files and +silently-ignored fields in plugin agents. Rather than hardcoding those field lists in the +script, the script reads `references/field-inventory.md` at runtime. This keeps field list +maintenance decoupled from script logic and preserves a provenance chain back to the +research corpus that sourced the lists. + +## Considered options + +**Hardcode in validate.sh (rejected)** — field lists live as literal arrays in the +bash/python script. Rejected because: (1) the lists came from research docs +(`claude-code-plugins/agent-definition.md` and `github-copilot-plugins/agent-definition.md`) +and should maintain a provenance chain back to those sources via `source_keys` frontmatter; +(2) both provider APIs evolve — updating a structured markdown file is lower friction than +editing a script and less likely to introduce bugs; (3) it breaks the bidirectional reference +principle already established for this repo, where research-derived content carries explicit +source attribution. + +## Consequences + +- `validate.sh` must parse `references/field-inventory.md` to extract field lists — the + file format must be machine-parseable (section headings the script can grep, or a simple + list structure). +- `field-inventory.md` carries `source_keys` frontmatter referencing + `claude-code-plugins-docs` and `github-custom-agents-configuration` slugs. +- The script exits with a clear error if `references/field-inventory.md` is not found — + fail-fast, not silent. +- Field list updates (new provider field, deprecated field) require only editing + `field-inventory.md`; no script change needed. diff --git a/plugins/kyberforge/.claude-plugin/plugin.json b/plugins/kyberforge/.claude-plugin/plugin.json index 706e2aa..70fd94c 100644 --- a/plugins/kyberforge/.claude-plugin/plugin.json +++ b/plugins/kyberforge/.claude-plugin/plugin.json @@ -8,5 +8,5 @@ "keywords": [], "license": "MIT", "name": "kyberforge", - "version": "1.0.7" + "version": "1.1.2" } diff --git a/plugins/kyberforge/plugin.json b/plugins/kyberforge/plugin.json index aeaac46..eb08f9e 100644 --- a/plugins/kyberforge/plugin.json +++ b/plugins/kyberforge/plugin.json @@ -13,5 +13,5 @@ "skills": [ "skills/" ], - "version": "1.0.7" + "version": "1.1.2" } diff --git a/plugins/kyberforge/skills/agent-audit/README.md b/plugins/kyberforge/skills/agent-audit/README.md new file mode 100644 index 0000000..1fb71a3 --- /dev/null +++ b/plugins/kyberforge/skills/agent-audit/README.md @@ -0,0 +1,28 @@ +# agent-audit + +Audits a Claude Code and Copilot agent definition file pair for correctness and quality. + +## What it does + +Accepts either file in a CC `.md` / Copilot `.agent.md` pair, derives the counterpart automatically, and validates both. Runs structural checks via `validate.sh` (required fields, kebab-case name, no placeholders, no CC-only fields in the Copilot file, silently-ignored fields at plugin scope), then qualitative checks on description phrasing and system prompt quality. Produces a compact findings report in the same format as `skill-audit`. + +## Usage + +``` +/agent-audit +``` + +Pass the path to either agent file as the argument. + +## Files + +| File | Purpose | +|------|---------| +| `SKILL.md` | Skill instructions for agents | +| `references/README.md` | Directory documentation for references/ | +| `references/field-inventory.md` | Authoritative list of valid CC and Copilot agent fields | +| `references/sources.md` | Research provenance for skill content | +| `scripts/README.md` | Directory documentation for scripts/ | +| `scripts/validate.sh` | Structural validation script for agent file pairs | +| `tests/README.md` | Bats test dependency and run instructions | +| `tests/validate.bats` | Bats tests for validate.sh | diff --git a/plugins/kyberforge/skills/agent-audit/SKILL.md b/plugins/kyberforge/skills/agent-audit/SKILL.md new file mode 100644 index 0000000..b2cdc58 --- /dev/null +++ b/plugins/kyberforge/skills/agent-audit/SKILL.md @@ -0,0 +1,86 @@ +--- +name: agent-audit +description: > + Use when the user wants to review an agent definition they wrote, says "audit this + agent", "check if my agent follows best practices", "review my agent file", or wants + to know if an agent pair is ready to ship — even if they don't use the word "audit". + Audits a Claude Code .md and Copilot .agent.md agent file pair — structural + validation via validate.sh plus qualitative checks on description and system prompt. + Produces a compact findings report (findings only, no PASS noise) with Why and Fix + per finding, in the same format as skill-audit. Do not use to fix agent files — use + /agent-author instead. Do not use to audit SKILL.md files — use /skill-audit instead. +allowed-tools: Bash Read +metadata: + category: factory + source_keys: + - context7-websites-code-claude + - claude-code-plugins-docs + - claude-code-subagents-docs + - context7-github-en-copilot + - github-custom-agents-configuration +--- + +## Gotchas + +- The unit of authoring is always a pair (CC `.md` + Copilot `.agent.md`). A missing counterpart is always a FAIL, not a warning. +- Plugin scope is detected by the presence of `plugin.json` in the directory tree — not by the file path pattern. Walk up, don't guess. +- `references/field-inventory.md` must exist for `validate.sh` to run. The script exits with an error if it is missing. +- Do not output findings while auditing — gather internally, surface in Step 3 report. + +## Step 1 — Run structural validation + +```bash +bash scripts/validate.sh +``` + +The script accepts either the CC file or the Copilot file. It detects provider from extension, derives the counterpart, and runs all structural checks. Note FAILs for the `### Structure` and `### Provider safety` report dimensions. Findings about missing fields, bad name format, empty body, or missing frontmatter → `### Structure`. Findings about CC-only fields in a Copilot file or plugin-silently-ignored fields in a CC file → `### Provider safety`. + +If the script cannot run (Bash denied, python3 unavailable), perform checks manually: required fields present (`name`, `description`, non-empty body), `name` is kebab-case, `name` matches filename stem, no `FILL IN:` placeholders, no CC-only fields in Copilot file. + +## Step 2 — Qualitative checks + +Read both agent files. Work through each dimension internally. Collect findings only; report in Step 3. + +**Description (both files):** +- Action-verb opening: description starts with a verb ("Reviews...", "Analyzes...", "Generates...") — FAIL if absent +- Specificity: is the trigger condition stated precisely? — SUGGESTION if vague +- `Use proactively` in a Copilot description: CC-specific phrasing, has no effect in Copilot — SUGGESTION to remove + +**Body:** +- Direct role instruction: system prompt opens with `You are a [role]. When invoked, [action].` — SUGGESTION if absent +- One job per agent: system prompt describes a single bounded task — SUGGESTION if scope appears unbounded + +**Pair consistency (cross-file):** +- `name` field matches between CC and Copilot files — FAIL if mismatch +- Both system prompt bodies non-empty — FAIL if either is empty + +## Step 3 — Report + +Open with a coverage line: + +```text +Checked: structure · provider-safety · description · body · pair-consistency +``` + +Then output only dimensions that have findings, grouped under H3 headings, FAILs before SUGGESTIONs within each dimension. Omit clean dimensions entirely. + +For each finding: + +```text +FAIL/SUGGESTION — file:line + Why: + Fix: +``` + +Close with: + +```text +## Result + +PASS +PASS (N suggestions) +FAIL (N fails · M suggestions) +Run /agent-author to address findings. +``` + +Omit `Run /agent-author to address findings.` when there are no findings at all. Do not apply fixes — report and propose only. diff --git a/plugins/kyberforge/skills/agent-audit/references/README.md b/plugins/kyberforge/skills/agent-audit/references/README.md new file mode 100644 index 0000000..d32bf39 --- /dev/null +++ b/plugins/kyberforge/skills/agent-audit/references/README.md @@ -0,0 +1,10 @@ +# references/ + +Additional documentation agents load on demand. + +## Files + +| File | Purpose | +|------|---------| +| `field-inventory.md` | Canonical list of valid CC and Copilot agent definition fields. Load when the script needs authoritative field lists for structural validation. | +| `sources.md` | Research provenance records for skill content. Load only when tracing the origin of a specific rule or field constraint. | diff --git a/plugins/kyberforge/skills/agent-audit/references/field-inventory.md b/plugins/kyberforge/skills/agent-audit/references/field-inventory.md new file mode 100644 index 0000000..c0e215a --- /dev/null +++ b/plugins/kyberforge/skills/agent-audit/references/field-inventory.md @@ -0,0 +1,24 @@ +--- +source_keys: + - context7-websites-code-claude + - claude-code-plugins-docs + - claude-code-subagents-docs + - context7-github-en-copilot + - github-custom-agents-configuration +--- + +## claude-code-fields + +name description tools disallowedTools model effort maxTurns permissionMode skills mcpServers hooks memory background isolation color initialPrompt + +## claude-code-only-fields + +maxTurns isolation memory permissionMode effort hooks mcpServers disallowedTools skills initialPrompt color background + +## plugin-silently-ignored-fields + +hooks mcpServers permissionMode + +## copilot-fields + +name description tools target model disable-model-invocation user-invocable mcp-servers metadata diff --git a/plugins/kyberforge/skills/agent-audit/references/sources.md b/plugins/kyberforge/skills/agent-audit/references/sources.md new file mode 100644 index 0000000..95add20 --- /dev/null +++ b/plugins/kyberforge/skills/agent-audit/references/sources.md @@ -0,0 +1,90 @@ +--- +source_keys: + - context7-websites-code-claude + - claude-code-plugins-docs + - claude-code-subagents-docs + - context7-github-en-copilot + - github-custom-agents-configuration +--- + +# Sources + +## context7-websites-code-claude + +- **URL:** context7:/websites/code_claude +- **Research doc:** plugins/kyberforge/docs/research/docs/claude-code-plugins/sources.md +- **Description:** Official Claude Code documentation site indexed by Context7 — plugin manifest schema, subagent definition types, marketplace JSON format, agent markdown file format +- **Contributing files:** SKILL.md, references/field-inventory.md +- **Status:** `extracted` + +## claude-code-plugins-docs + +- **URL:** https://code.claude.com/docs/en/plugins +- **Research doc:** plugins/kyberforge/docs/research/docs/claude-code-plugins/sources.md +- **Description:** Official Claude Code plugin authoring guide — plugin structure, manifest fields, loading methods, skill namespacing, agent activation, marketplace submission +- **Contributing files:** SKILL.md, references/field-inventory.md +- **Status:** `extracted` + +## claude-code-subagents-docs + +- **URL:** https://code.claude.com/docs/en/sub-agents +- **Research doc:** plugins/kyberforge/docs/research/docs/claude-code-plugins/sources.md +- **Description:** Official Claude Code subagent reference — definition format, all frontmatter fields, scope priority, built-in agents, CLI flags, environment variables, known limitations +- **Contributing files:** SKILL.md, references/field-inventory.md +- **Status:** `extracted` + +## context7-github-en-copilot + +- **URL:** context7:/websites/github_en_copilot +- **Research doc:** plugins/kyberforge/docs/research/docs/github-copilot-plugins/sources.md +- **Description:** Official GitHub Copilot documentation indexed by Context7; covers CLI plugins, custom agents, SDK, and marketplace +- **Contributing files:** SKILL.md, references/field-inventory.md +- **Status:** `extracted` + +## github-custom-agents-configuration + +- **URL:** https://docs.github.com/en/copilot/reference/custom-agents-configuration +- **Research doc:** plugins/kyberforge/docs/research/docs/github-copilot-plugins/sources.md +- **Description:** Reference for cloud and IDE custom agent definition format — frontmatter fields, tool aliases, MCP server config, secrets interpolation, scoping hierarchy +- **Contributing files:** SKILL.md, references/field-inventory.md +- **Status:** `extracted` + +## github-cli-plugin-reference + +- **URL:** https://docs.github.com/en/copilot/reference/copilot-cli-reference/cli-plugin-reference +- **Research doc:** plugins/kyberforge/docs/research/docs/github-copilot-plugins/sources.md +- **Description:** Full CLI plugin reference — plugin.json schema, marketplace.json schema, all CLI commands and flags, install specification formats, loading precedence, env vars, LSP config +- **Contributing files:** (none) +- **Status:** `extracted` + +## github-plugins-creating + +- **URL:** https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/plugins-creating +- **Research doc:** plugins/kyberforge/docs/research/docs/github-copilot-plugins/sources.md +- **Description:** How-to for creating Copilot CLI plugins — plugin structure, agent and skill authoring, hooks format, MCP config, development lifecycle +- **Contributing files:** (none) +- **Status:** `extracted` + +## github-plugins-finding-installing + +- **URL:** https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/plugins-finding-installing +- **Research doc:** plugins/kyberforge/docs/research/docs/github-copilot-plugins/sources.md +- **Description:** User-facing guide to discovering and installing CLI plugins — marketplace browsing commands, install/update/uninstall workflow +- **Contributing files:** (none) +- **Status:** `extracted` + +## github-plugins-marketplace + +- **URL:** https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/plugins-marketplace +- **Research doc:** plugins/kyberforge/docs/research/docs/github-copilot-plugins/sources.md +- **Description:** How-to for creating and publishing a plugin marketplace — marketplace.json structure, hosting options, registration commands +- **Contributing files:** (none) +- **Status:** `extracted` + +## github-sdk-custom-agents + +- **URL:** https://docs.github.com/en/copilot/how-tos/copilot-sdk/features/custom-agents +- **Research doc:** plugins/kyberforge/docs/research/docs/github-copilot-plugins/sources.md +- **Description:** SDK custom agent API — CustomAgentConfig fields in all five languages, session config, sub-agent lifecycle events, tool scoping, permission handling +- **Contributing files:** (none) +- **Status:** `extracted` diff --git a/plugins/kyberforge/skills/agent-audit/scripts/README.md b/plugins/kyberforge/skills/agent-audit/scripts/README.md new file mode 100644 index 0000000..f71640e --- /dev/null +++ b/plugins/kyberforge/skills/agent-audit/scripts/README.md @@ -0,0 +1,47 @@ +# scripts/ + +Executable code bundled with this skill. Agents run scripts in this directory +to perform repeatable operations rather than reinventing the logic each run. + +## When to add a script + +Add a script when agents independently reinvent the same logic across runs — +building the same parser, chart, or validation routine from scratch each time. +Bundle it here once, tested and reliable. + +## Script requirements (agentskills.io) + +Scripts must be designed for non-interactive, agentic execution: + +- **No interactive prompts** — agents run in non-interactive shells. + Accept all input via flags, env vars, or stdin. A script that blocks on + TTY input hangs indefinitely. +- **Expose `--help`** — this is how agents learn your script's interface. + Keep the output concise; it enters the agent's context window. +- **Structured output** — write data (JSON, CSV, TSV) to stdout. + Write progress, warnings, and diagnostics to stderr. +- **Idempotent** — prefer "create if not exists" over "create and fail on + duplicate". Agents may retry on failure. +- **Meaningful exit codes** — `0` for success, non-zero for failure. + Use distinct codes for different failure types; document them in `--help`. +- **Dry-run support** — add `--dry-run` for destructive operations. + +## Self-contained scripts + +Bundle dependencies inline so the agent can run the script with a single command. + +Python (PEP 723 + uv): +```python +# /// script +# dependencies = ["requests>=2.31,<3"] +# requires-python = ">=3.11" +# /// +import requests +``` +```bash +uv run scripts/my-script.py +``` + +## If no scripts are needed + +Delete this README and the `scripts/` directory entirely. diff --git a/plugins/kyberforge/skills/agent-audit/scripts/validate.sh b/plugins/kyberforge/skills/agent-audit/scripts/validate.sh new file mode 100755 index 0000000..706ab48 --- /dev/null +++ b/plugins/kyberforge/skills/agent-audit/scripts/validate.sh @@ -0,0 +1,210 @@ +#!/usr/bin/env bash +set -euo pipefail + +usage() { + cat < + +Validate a Claude Code or Copilot agent file pair against the agent definition spec. + +Arguments: + agent-file Path to either the Claude Code .md or Copilot .agent.md agent file. + +Exit codes: + 0 All checks passed + 1 One or more checks failed + 2 Script error (unrecognized file extension or missing field-inventory.md) +EOF +} + +if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then + usage + exit 0 +fi + +if [[ $# -lt 1 ]]; then + echo "Error: agent-file is required." >&2 + echo "" >&2 + usage >&2 + exit 1 +fi + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +python3 -u - "$1" "$SCRIPT_DIR" <<'PYTHON' +import sys +import os +import re + +agent_file = os.path.abspath(sys.argv[1]) +script_dir = sys.argv[2] + +fname = os.path.basename(agent_file) + +# --- Detect provider (check .agent.md before .md) --- +if fname.endswith('.agent.md'): + provider = 'copilot' + name_stem = fname[:-len('.agent.md')] +elif fname.endswith('.md'): + provider = 'claude-code' + name_stem = fname[:-len('.md')] +else: + print(f"Error: unrecognized extension '{fname}' — expected .md or .agent.md", file=sys.stderr) + sys.exit(2) + +# --- Load field-inventory.md --- +inv_path = os.path.normpath(os.path.join(script_dir, '..', 'references', 'field-inventory.md')) +if not os.path.isfile(inv_path): + print(f"Error: field-inventory.md not found at {inv_path}", file=sys.stderr) + sys.exit(2) + +with open(inv_path) as f: + inv_content = f.read() + +def parse_section_tokens(content, section_name): + lines = content.splitlines() + for i, line in enumerate(lines): + if line.strip() == f'## {section_name}': + for j in range(i + 1, len(lines)): + stripped = lines[j].strip() + if stripped and not stripped.startswith('#') and not stripped.startswith('---'): + return set(stripped.split()) + return set() + +cc_only_fields = parse_section_tokens(inv_content, 'claude-code-only-fields') +plugin_ignored_fields = parse_section_tokens(inv_content, 'plugin-silently-ignored-fields') + +# --- Detect scope --- +def detect_scope(start_dir): + current = os.path.abspath(start_dir) + while True: + if os.path.isfile(os.path.join(current, 'plugin.json')): + return 'plugin', current + if os.path.isdir(os.path.join(current, '.git')): + return 'project', current + parent = os.path.dirname(current) + if parent == current: + return 'user', os.path.expanduser('~') + current = parent + +agent_dir = os.path.dirname(agent_file) +scope, scope_root = detect_scope(agent_dir) + +# --- Derive counterpart path --- +if scope == 'plugin': + if provider == 'copilot': + counterpart = os.path.join(agent_dir, name_stem + '.md') + counterpart_provider = 'claude-code' + else: + counterpart = os.path.join(agent_dir, name_stem + '.agent.md') + counterpart_provider = 'copilot' +elif scope == 'project': + if provider == 'claude-code': + counterpart = os.path.join(scope_root, '.github', 'agents', name_stem + '.agent.md') + counterpart_provider = 'copilot' + else: + counterpart = os.path.join(scope_root, '.claude', 'agents', name_stem + '.md') + counterpart_provider = 'claude-code' +else: # user + home = os.path.expanduser('~') + if provider == 'claude-code': + counterpart = os.path.join(home, '.copilot', 'agents', name_stem + '.agent.md') + counterpart_provider = 'copilot' + else: + counterpart = os.path.join(home, '.claude', 'agents', name_stem + '.md') + counterpart_provider = 'claude-code' + +# --- Helpers --- +failed = False + +def fail(msg): + global failed + failed = True + print(f"FAIL {msg}") + +PLACEHOLDER_RE = re.compile(r'(?/agent-audit/tests/ +``` + +## If no tests are needed + +Delete this README and the `tests/` directory entirely. diff --git a/plugins/kyberforge/skills/agent-audit/tests/validate.bats b/plugins/kyberforge/skills/agent-audit/tests/validate.bats new file mode 100644 index 0000000..0aa4e53 --- /dev/null +++ b/plugins/kyberforge/skills/agent-audit/tests/validate.bats @@ -0,0 +1,218 @@ +#!/usr/bin/env bats + +setup() { + REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../../../../../" && pwd)" + load "$REPO_ROOT/tests/test_helper/bats-support/load" + load "$REPO_ROOT/tests/test_helper/bats-assert/load" + + SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate.sh" + TMPDIR="$(mktemp -d)" + + # Helper: create a plugin-scope pair in with given + make_plugin_pair() { + local dir="$1" + local name="$2" + mkdir -p "$dir" + echo '{}' > "$dir/plugin.json" + cat > "$dir/${name}.md" < "$dir/${name}.agent.md" < "$root/.claude/agents/my-agent.md" < "$root/.github/agents/my-agent.agent.md" < "$dir/my-agent.agent.md" < "$dir/my-agent.md" < "$dir/plugin.json" + cat > "$dir/my-agent.md" < "$dir/my-agent.agent.md" < "$dir/my-agent.md" < "$dir/my-agent.md" < "$dir/my-agent.md" < "$dir/my-agent.md" <