feat(kyberforge): add agent-audit skill (closes #11)
## Why agent-author produces paired agent definition files (Claude Code .md + Copilot .agent.md) but had no companion audit skill to validate them. agent-audit fills that gap, giving the same structured PASS/FAIL report that skill-audit provides for SKILL.md files. ## Implementation Notes - validate.sh uses scope detection (walk up for plugin.json / .git) to locate the counterpart file and determine whether plugin-silently-ignored fields (hooks, mcpServers, permissionMode) should be flagged - CC-only and silently-ignored field lists are read from references/field-inventory.md at runtime rather than hardcoded — provenance back to the research corpus; see ADR-0019 - Single-file invocation (pass either file, counterpart derived) chosen over directory or name+root — see ADR-0018 - 12 bats tests cover provider detection, scope detection, all FAIL paths, and clean-pair pass ## Impact - kyberforge bumped to v1.1.2 - agent-author close step should be updated to reference agent-audit (#11) - Provenance/sources chain check deferred to #60 ADR: docs/adr/0018-agent-audit-single-file-invocation.md ADR: docs/adr/0019-agent-audit-field-inventory-reference.md Refs: #11 Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0147vXtL5sP6vorDdqXGJJU9
This commit is contained in:
@@ -8,5 +8,5 @@
|
||||
"keywords": [],
|
||||
"license": "MIT",
|
||||
"name": "kyberforge",
|
||||
"version": "1.0.7"
|
||||
"version": "1.1.2"
|
||||
}
|
||||
|
||||
@@ -13,5 +13,5 @@
|
||||
"skills": [
|
||||
"skills/"
|
||||
],
|
||||
"version": "1.0.7"
|
||||
"version": "1.1.2"
|
||||
}
|
||||
|
||||
28
plugins/kyberforge/skills/agent-audit/README.md
Normal file
28
plugins/kyberforge/skills/agent-audit/README.md
Normal file
@@ -0,0 +1,28 @@
|
||||
# agent-audit
|
||||
|
||||
Audits a Claude Code and Copilot agent definition file pair for correctness and quality.
|
||||
|
||||
## What it does
|
||||
|
||||
Accepts either file in a CC `.md` / Copilot `.agent.md` pair, derives the counterpart automatically, and validates both. Runs structural checks via `validate.sh` (required fields, kebab-case name, no placeholders, no CC-only fields in the Copilot file, silently-ignored fields at plugin scope), then qualitative checks on description phrasing and system prompt quality. Produces a compact findings report in the same format as `skill-audit`.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
/agent-audit
|
||||
```
|
||||
|
||||
Pass the path to either agent file as the argument.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `SKILL.md` | Skill instructions for agents |
|
||||
| `references/README.md` | Directory documentation for references/ |
|
||||
| `references/field-inventory.md` | Authoritative list of valid CC and Copilot agent fields |
|
||||
| `references/sources.md` | Research provenance for skill content |
|
||||
| `scripts/README.md` | Directory documentation for scripts/ |
|
||||
| `scripts/validate.sh` | Structural validation script for agent file pairs |
|
||||
| `tests/README.md` | Bats test dependency and run instructions |
|
||||
| `tests/validate.bats` | Bats tests for validate.sh |
|
||||
86
plugins/kyberforge/skills/agent-audit/SKILL.md
Normal file
86
plugins/kyberforge/skills/agent-audit/SKILL.md
Normal file
@@ -0,0 +1,86 @@
|
||||
---
|
||||
name: agent-audit
|
||||
description: >
|
||||
Use when the user wants to review an agent definition they wrote, says "audit this
|
||||
agent", "check if my agent follows best practices", "review my agent file", or wants
|
||||
to know if an agent pair is ready to ship — even if they don't use the word "audit".
|
||||
Audits a Claude Code .md and Copilot .agent.md agent file pair — structural
|
||||
validation via validate.sh plus qualitative checks on description and system prompt.
|
||||
Produces a compact findings report (findings only, no PASS noise) with Why and Fix
|
||||
per finding, in the same format as skill-audit. Do not use to fix agent files — use
|
||||
/agent-author instead. Do not use to audit SKILL.md files — use /skill-audit instead.
|
||||
allowed-tools: Bash Read
|
||||
metadata:
|
||||
category: factory
|
||||
source_keys:
|
||||
- context7-websites-code-claude
|
||||
- claude-code-plugins-docs
|
||||
- claude-code-subagents-docs
|
||||
- context7-github-en-copilot
|
||||
- github-custom-agents-configuration
|
||||
---
|
||||
|
||||
## Gotchas
|
||||
|
||||
- The unit of authoring is always a pair (CC `.md` + Copilot `.agent.md`). A missing counterpart is always a FAIL, not a warning.
|
||||
- Plugin scope is detected by the presence of `plugin.json` in the directory tree — not by the file path pattern. Walk up, don't guess.
|
||||
- `references/field-inventory.md` must exist for `validate.sh` to run. The script exits with an error if it is missing.
|
||||
- Do not output findings while auditing — gather internally, surface in Step 3 report.
|
||||
|
||||
## Step 1 — Run structural validation
|
||||
|
||||
```bash
|
||||
bash scripts/validate.sh <path-to-agent-file>
|
||||
```
|
||||
|
||||
The script accepts either the CC file or the Copilot file. It detects provider from extension, derives the counterpart, and runs all structural checks. Note FAILs for the `### Structure` and `### Provider safety` report dimensions. Findings about missing fields, bad name format, empty body, or missing frontmatter → `### Structure`. Findings about CC-only fields in a Copilot file or plugin-silently-ignored fields in a CC file → `### Provider safety`.
|
||||
|
||||
If the script cannot run (Bash denied, python3 unavailable), perform checks manually: required fields present (`name`, `description`, non-empty body), `name` is kebab-case, `name` matches filename stem, no `FILL IN:` placeholders, no CC-only fields in Copilot file.
|
||||
|
||||
## Step 2 — Qualitative checks
|
||||
|
||||
Read both agent files. Work through each dimension internally. Collect findings only; report in Step 3.
|
||||
|
||||
**Description (both files):**
|
||||
- Action-verb opening: description starts with a verb ("Reviews...", "Analyzes...", "Generates...") — FAIL if absent
|
||||
- Specificity: is the trigger condition stated precisely? — SUGGESTION if vague
|
||||
- `Use proactively` in a Copilot description: CC-specific phrasing, has no effect in Copilot — SUGGESTION to remove
|
||||
|
||||
**Body:**
|
||||
- Direct role instruction: system prompt opens with `You are a [role]. When invoked, [action].` — SUGGESTION if absent
|
||||
- One job per agent: system prompt describes a single bounded task — SUGGESTION if scope appears unbounded
|
||||
|
||||
**Pair consistency (cross-file):**
|
||||
- `name` field matches between CC and Copilot files — FAIL if mismatch
|
||||
- Both system prompt bodies non-empty — FAIL if either is empty
|
||||
|
||||
## Step 3 — Report
|
||||
|
||||
Open with a coverage line:
|
||||
|
||||
```text
|
||||
Checked: structure · provider-safety · description · body · pair-consistency
|
||||
```
|
||||
|
||||
Then output only dimensions that have findings, grouped under H3 headings, FAILs before SUGGESTIONs within each dimension. Omit clean dimensions entirely.
|
||||
|
||||
For each finding:
|
||||
|
||||
```text
|
||||
FAIL/SUGGESTION <finding> — file:line
|
||||
Why: <why this is a problem>
|
||||
Fix: <exact change — quote before/after where applicable>
|
||||
```
|
||||
|
||||
Close with:
|
||||
|
||||
```text
|
||||
## Result
|
||||
|
||||
PASS
|
||||
PASS (N suggestions)
|
||||
FAIL (N fails · M suggestions)
|
||||
Run /agent-author to address findings.
|
||||
```
|
||||
|
||||
Omit `Run /agent-author to address findings.` when there are no findings at all. Do not apply fixes — report and propose only.
|
||||
10
plugins/kyberforge/skills/agent-audit/references/README.md
Normal file
10
plugins/kyberforge/skills/agent-audit/references/README.md
Normal file
@@ -0,0 +1,10 @@
|
||||
# references/
|
||||
|
||||
Additional documentation agents load on demand.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `field-inventory.md` | Canonical list of valid CC and Copilot agent definition fields. Load when the script needs authoritative field lists for structural validation. |
|
||||
| `sources.md` | Research provenance records for skill content. Load only when tracing the origin of a specific rule or field constraint. |
|
||||
@@ -0,0 +1,24 @@
|
||||
---
|
||||
source_keys:
|
||||
- context7-websites-code-claude
|
||||
- claude-code-plugins-docs
|
||||
- claude-code-subagents-docs
|
||||
- context7-github-en-copilot
|
||||
- github-custom-agents-configuration
|
||||
---
|
||||
|
||||
## claude-code-fields
|
||||
|
||||
name description tools disallowedTools model effort maxTurns permissionMode skills mcpServers hooks memory background isolation color initialPrompt
|
||||
|
||||
## claude-code-only-fields
|
||||
|
||||
maxTurns isolation memory permissionMode effort hooks mcpServers disallowedTools skills initialPrompt color background
|
||||
|
||||
## plugin-silently-ignored-fields
|
||||
|
||||
hooks mcpServers permissionMode
|
||||
|
||||
## copilot-fields
|
||||
|
||||
name description tools target model disable-model-invocation user-invocable mcp-servers metadata
|
||||
90
plugins/kyberforge/skills/agent-audit/references/sources.md
Normal file
90
plugins/kyberforge/skills/agent-audit/references/sources.md
Normal file
@@ -0,0 +1,90 @@
|
||||
---
|
||||
source_keys:
|
||||
- context7-websites-code-claude
|
||||
- claude-code-plugins-docs
|
||||
- claude-code-subagents-docs
|
||||
- context7-github-en-copilot
|
||||
- github-custom-agents-configuration
|
||||
---
|
||||
|
||||
# Sources
|
||||
|
||||
## context7-websites-code-claude
|
||||
|
||||
- **URL:** context7:/websites/code_claude
|
||||
- **Research doc:** plugins/kyberforge/docs/research/docs/claude-code-plugins/sources.md
|
||||
- **Description:** Official Claude Code documentation site indexed by Context7 — plugin manifest schema, subagent definition types, marketplace JSON format, agent markdown file format
|
||||
- **Contributing files:** SKILL.md, references/field-inventory.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## claude-code-plugins-docs
|
||||
|
||||
- **URL:** https://code.claude.com/docs/en/plugins
|
||||
- **Research doc:** plugins/kyberforge/docs/research/docs/claude-code-plugins/sources.md
|
||||
- **Description:** Official Claude Code plugin authoring guide — plugin structure, manifest fields, loading methods, skill namespacing, agent activation, marketplace submission
|
||||
- **Contributing files:** SKILL.md, references/field-inventory.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## claude-code-subagents-docs
|
||||
|
||||
- **URL:** https://code.claude.com/docs/en/sub-agents
|
||||
- **Research doc:** plugins/kyberforge/docs/research/docs/claude-code-plugins/sources.md
|
||||
- **Description:** Official Claude Code subagent reference — definition format, all frontmatter fields, scope priority, built-in agents, CLI flags, environment variables, known limitations
|
||||
- **Contributing files:** SKILL.md, references/field-inventory.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## context7-github-en-copilot
|
||||
|
||||
- **URL:** context7:/websites/github_en_copilot
|
||||
- **Research doc:** plugins/kyberforge/docs/research/docs/github-copilot-plugins/sources.md
|
||||
- **Description:** Official GitHub Copilot documentation indexed by Context7; covers CLI plugins, custom agents, SDK, and marketplace
|
||||
- **Contributing files:** SKILL.md, references/field-inventory.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## github-custom-agents-configuration
|
||||
|
||||
- **URL:** https://docs.github.com/en/copilot/reference/custom-agents-configuration
|
||||
- **Research doc:** plugins/kyberforge/docs/research/docs/github-copilot-plugins/sources.md
|
||||
- **Description:** Reference for cloud and IDE custom agent definition format — frontmatter fields, tool aliases, MCP server config, secrets interpolation, scoping hierarchy
|
||||
- **Contributing files:** SKILL.md, references/field-inventory.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## github-cli-plugin-reference
|
||||
|
||||
- **URL:** https://docs.github.com/en/copilot/reference/copilot-cli-reference/cli-plugin-reference
|
||||
- **Research doc:** plugins/kyberforge/docs/research/docs/github-copilot-plugins/sources.md
|
||||
- **Description:** Full CLI plugin reference — plugin.json schema, marketplace.json schema, all CLI commands and flags, install specification formats, loading precedence, env vars, LSP config
|
||||
- **Contributing files:** (none)
|
||||
- **Status:** `extracted`
|
||||
|
||||
## github-plugins-creating
|
||||
|
||||
- **URL:** https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/plugins-creating
|
||||
- **Research doc:** plugins/kyberforge/docs/research/docs/github-copilot-plugins/sources.md
|
||||
- **Description:** How-to for creating Copilot CLI plugins — plugin structure, agent and skill authoring, hooks format, MCP config, development lifecycle
|
||||
- **Contributing files:** (none)
|
||||
- **Status:** `extracted`
|
||||
|
||||
## github-plugins-finding-installing
|
||||
|
||||
- **URL:** https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/plugins-finding-installing
|
||||
- **Research doc:** plugins/kyberforge/docs/research/docs/github-copilot-plugins/sources.md
|
||||
- **Description:** User-facing guide to discovering and installing CLI plugins — marketplace browsing commands, install/update/uninstall workflow
|
||||
- **Contributing files:** (none)
|
||||
- **Status:** `extracted`
|
||||
|
||||
## github-plugins-marketplace
|
||||
|
||||
- **URL:** https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/plugins-marketplace
|
||||
- **Research doc:** plugins/kyberforge/docs/research/docs/github-copilot-plugins/sources.md
|
||||
- **Description:** How-to for creating and publishing a plugin marketplace — marketplace.json structure, hosting options, registration commands
|
||||
- **Contributing files:** (none)
|
||||
- **Status:** `extracted`
|
||||
|
||||
## github-sdk-custom-agents
|
||||
|
||||
- **URL:** https://docs.github.com/en/copilot/how-tos/copilot-sdk/features/custom-agents
|
||||
- **Research doc:** plugins/kyberforge/docs/research/docs/github-copilot-plugins/sources.md
|
||||
- **Description:** SDK custom agent API — CustomAgentConfig fields in all five languages, session config, sub-agent lifecycle events, tool scoping, permission handling
|
||||
- **Contributing files:** (none)
|
||||
- **Status:** `extracted`
|
||||
47
plugins/kyberforge/skills/agent-audit/scripts/README.md
Normal file
47
plugins/kyberforge/skills/agent-audit/scripts/README.md
Normal file
@@ -0,0 +1,47 @@
|
||||
# scripts/
|
||||
|
||||
Executable code bundled with this skill. Agents run scripts in this directory
|
||||
to perform repeatable operations rather than reinventing the logic each run.
|
||||
|
||||
## When to add a script
|
||||
|
||||
Add a script when agents independently reinvent the same logic across runs —
|
||||
building the same parser, chart, or validation routine from scratch each time.
|
||||
Bundle it here once, tested and reliable.
|
||||
|
||||
## Script requirements (agentskills.io)
|
||||
|
||||
Scripts must be designed for non-interactive, agentic execution:
|
||||
|
||||
- **No interactive prompts** — agents run in non-interactive shells.
|
||||
Accept all input via flags, env vars, or stdin. A script that blocks on
|
||||
TTY input hangs indefinitely.
|
||||
- **Expose `--help`** — this is how agents learn your script's interface.
|
||||
Keep the output concise; it enters the agent's context window.
|
||||
- **Structured output** — write data (JSON, CSV, TSV) to stdout.
|
||||
Write progress, warnings, and diagnostics to stderr.
|
||||
- **Idempotent** — prefer "create if not exists" over "create and fail on
|
||||
duplicate". Agents may retry on failure.
|
||||
- **Meaningful exit codes** — `0` for success, non-zero for failure.
|
||||
Use distinct codes for different failure types; document them in `--help`.
|
||||
- **Dry-run support** — add `--dry-run` for destructive operations.
|
||||
|
||||
## Self-contained scripts
|
||||
|
||||
Bundle dependencies inline so the agent can run the script with a single command.
|
||||
|
||||
Python (PEP 723 + uv):
|
||||
```python
|
||||
# /// script
|
||||
# dependencies = ["requests>=2.31,<3"]
|
||||
# requires-python = ">=3.11"
|
||||
# ///
|
||||
import requests
|
||||
```
|
||||
```bash
|
||||
uv run scripts/my-script.py
|
||||
```
|
||||
|
||||
## If no scripts are needed
|
||||
|
||||
Delete this README and the `scripts/` directory entirely.
|
||||
210
plugins/kyberforge/skills/agent-audit/scripts/validate.sh
Executable file
210
plugins/kyberforge/skills/agent-audit/scripts/validate.sh
Executable file
@@ -0,0 +1,210 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: validate.sh <agent-file>
|
||||
|
||||
Validate a Claude Code or Copilot agent file pair against the agent definition spec.
|
||||
|
||||
Arguments:
|
||||
agent-file Path to either the Claude Code .md or Copilot .agent.md agent file.
|
||||
|
||||
Exit codes:
|
||||
0 All checks passed
|
||||
1 One or more checks failed
|
||||
2 Script error (unrecognized file extension or missing field-inventory.md)
|
||||
EOF
|
||||
}
|
||||
|
||||
if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then
|
||||
usage
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ $# -lt 1 ]]; then
|
||||
echo "Error: agent-file is required." >&2
|
||||
echo "" >&2
|
||||
usage >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
python3 -u - "$1" "$SCRIPT_DIR" <<'PYTHON'
|
||||
import sys
|
||||
import os
|
||||
import re
|
||||
|
||||
agent_file = os.path.abspath(sys.argv[1])
|
||||
script_dir = sys.argv[2]
|
||||
|
||||
fname = os.path.basename(agent_file)
|
||||
|
||||
# --- Detect provider (check .agent.md before .md) ---
|
||||
if fname.endswith('.agent.md'):
|
||||
provider = 'copilot'
|
||||
name_stem = fname[:-len('.agent.md')]
|
||||
elif fname.endswith('.md'):
|
||||
provider = 'claude-code'
|
||||
name_stem = fname[:-len('.md')]
|
||||
else:
|
||||
print(f"Error: unrecognized extension '{fname}' — expected .md or .agent.md", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
# --- Load field-inventory.md ---
|
||||
inv_path = os.path.normpath(os.path.join(script_dir, '..', 'references', 'field-inventory.md'))
|
||||
if not os.path.isfile(inv_path):
|
||||
print(f"Error: field-inventory.md not found at {inv_path}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
with open(inv_path) as f:
|
||||
inv_content = f.read()
|
||||
|
||||
def parse_section_tokens(content, section_name):
|
||||
lines = content.splitlines()
|
||||
for i, line in enumerate(lines):
|
||||
if line.strip() == f'## {section_name}':
|
||||
for j in range(i + 1, len(lines)):
|
||||
stripped = lines[j].strip()
|
||||
if stripped and not stripped.startswith('#') and not stripped.startswith('---'):
|
||||
return set(stripped.split())
|
||||
return set()
|
||||
|
||||
cc_only_fields = parse_section_tokens(inv_content, 'claude-code-only-fields')
|
||||
plugin_ignored_fields = parse_section_tokens(inv_content, 'plugin-silently-ignored-fields')
|
||||
|
||||
# --- Detect scope ---
|
||||
def detect_scope(start_dir):
|
||||
current = os.path.abspath(start_dir)
|
||||
while True:
|
||||
if os.path.isfile(os.path.join(current, 'plugin.json')):
|
||||
return 'plugin', current
|
||||
if os.path.isdir(os.path.join(current, '.git')):
|
||||
return 'project', current
|
||||
parent = os.path.dirname(current)
|
||||
if parent == current:
|
||||
return 'user', os.path.expanduser('~')
|
||||
current = parent
|
||||
|
||||
agent_dir = os.path.dirname(agent_file)
|
||||
scope, scope_root = detect_scope(agent_dir)
|
||||
|
||||
# --- Derive counterpart path ---
|
||||
if scope == 'plugin':
|
||||
if provider == 'copilot':
|
||||
counterpart = os.path.join(agent_dir, name_stem + '.md')
|
||||
counterpart_provider = 'claude-code'
|
||||
else:
|
||||
counterpart = os.path.join(agent_dir, name_stem + '.agent.md')
|
||||
counterpart_provider = 'copilot'
|
||||
elif scope == 'project':
|
||||
if provider == 'claude-code':
|
||||
counterpart = os.path.join(scope_root, '.github', 'agents', name_stem + '.agent.md')
|
||||
counterpart_provider = 'copilot'
|
||||
else:
|
||||
counterpart = os.path.join(scope_root, '.claude', 'agents', name_stem + '.md')
|
||||
counterpart_provider = 'claude-code'
|
||||
else: # user
|
||||
home = os.path.expanduser('~')
|
||||
if provider == 'claude-code':
|
||||
counterpart = os.path.join(home, '.copilot', 'agents', name_stem + '.agent.md')
|
||||
counterpart_provider = 'copilot'
|
||||
else:
|
||||
counterpart = os.path.join(home, '.claude', 'agents', name_stem + '.md')
|
||||
counterpart_provider = 'claude-code'
|
||||
|
||||
# --- Helpers ---
|
||||
failed = False
|
||||
|
||||
def fail(msg):
|
||||
global failed
|
||||
failed = True
|
||||
print(f"FAIL {msg}")
|
||||
|
||||
PLACEHOLDER_RE = re.compile(r'(?<!`)FILL IN:[^`\n]')
|
||||
|
||||
def parse_frontmatter(content):
|
||||
m = re.match(r'^---\n(.*?)\n---', content, re.DOTALL)
|
||||
if not m:
|
||||
return None, content
|
||||
return m.group(1), content[m.end():]
|
||||
|
||||
def extract_field(fm, field):
|
||||
m = re.search(rf'^{re.escape(field)}:\s*(.+)', fm, re.MULTILINE)
|
||||
return m.group(1).strip() if m else None
|
||||
|
||||
def get_frontmatter_keys(fm):
|
||||
keys = set()
|
||||
for line in fm.splitlines():
|
||||
m = re.match(r'^([a-zA-Z][a-zA-Z0-9_-]*):', line)
|
||||
if m:
|
||||
keys.add(m.group(1))
|
||||
return keys
|
||||
|
||||
def check_file(fpath, file_provider, is_plugin_scope):
|
||||
local_fname = os.path.basename(fpath)
|
||||
with open(fpath) as f:
|
||||
content = f.read()
|
||||
|
||||
fm, body = parse_frontmatter(content)
|
||||
if fm is None:
|
||||
fail(f"no valid YAML frontmatter (---...---) — {local_fname}")
|
||||
return
|
||||
|
||||
# name
|
||||
name_val = extract_field(fm, 'name')
|
||||
if not name_val:
|
||||
fail(f"name field is missing or empty — {local_fname}")
|
||||
else:
|
||||
if not re.match(r'^[a-z0-9]+(-[a-z0-9]+)*$', name_val):
|
||||
fail(f"name '{name_val}' is not kebab-case — {local_fname}")
|
||||
# stem check
|
||||
if file_provider == 'copilot':
|
||||
stem = local_fname[:-len('.agent.md')]
|
||||
else:
|
||||
stem = local_fname[:-len('.md')]
|
||||
if name_val != stem:
|
||||
fail(f"name '{name_val}' does not match filename stem '{stem}' — {local_fname}")
|
||||
|
||||
# description
|
||||
desc_val = extract_field(fm, 'description')
|
||||
if not desc_val:
|
||||
fail(f"description field is missing or empty — {local_fname}")
|
||||
else:
|
||||
if PLACEHOLDER_RE.search(desc_val):
|
||||
fail(f"description contains unfilled FILL IN: placeholder — {local_fname}")
|
||||
|
||||
# body
|
||||
if not body.strip():
|
||||
fail(f"system prompt body is empty — {local_fname}")
|
||||
else:
|
||||
if PLACEHOLDER_RE.search(body):
|
||||
fail(f"body contains unfilled FILL IN: placeholder — {local_fname}")
|
||||
|
||||
# CC-only fields in Copilot file
|
||||
if file_provider == 'copilot':
|
||||
fm_keys = get_frontmatter_keys(fm)
|
||||
for key in sorted(fm_keys):
|
||||
if key in cc_only_fields:
|
||||
fail(f"CC-only field '{key}' present in Copilot file — {local_fname}")
|
||||
|
||||
# Silently-ignored fields in plugin-scope CC file
|
||||
if file_provider == 'claude-code' and is_plugin_scope:
|
||||
fm_keys = get_frontmatter_keys(fm)
|
||||
for key in sorted(fm_keys):
|
||||
if key in plugin_ignored_fields:
|
||||
fail(f"plugin-silently-ignored field '{key}' present in plugin-scope CC file — {local_fname}")
|
||||
|
||||
# --- Check counterpart exists ---
|
||||
if not os.path.isfile(counterpart):
|
||||
fail(f"counterpart file not found: {counterpart}")
|
||||
sys.exit(1)
|
||||
|
||||
# --- Check both files ---
|
||||
is_plugin = (scope == 'plugin')
|
||||
check_file(agent_file, provider, is_plugin)
|
||||
check_file(counterpart, counterpart_provider, is_plugin)
|
||||
|
||||
sys.exit(1 if failed else 0)
|
||||
PYTHON
|
||||
33
plugins/kyberforge/skills/agent-audit/tests/README.md
Normal file
33
plugins/kyberforge/skills/agent-audit/tests/README.md
Normal file
@@ -0,0 +1,33 @@
|
||||
# tests/
|
||||
|
||||
Test files for scripts bundled with this skill.
|
||||
|
||||
## When to add tests
|
||||
|
||||
Add tests here when the skill has scripts in `scripts/` that are complex enough
|
||||
to break silently — validators, parsers, generators, anything with branching
|
||||
logic or edge cases. Test infrastructure (`.bats`, `*_test.*`, `test_*.sh`)
|
||||
belongs here, not in `scripts/`.
|
||||
|
||||
## Dependencies
|
||||
|
||||
Tests require [bats-support](https://github.com/bats-core/bats-support) and
|
||||
[bats-assert](https://github.com/bats-core/bats-assert). The test files load
|
||||
helpers from the repo root's `tests/test_helper/`.
|
||||
|
||||
From the repo root:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/bats-core/bats-support tests/test_helper/bats-support
|
||||
git clone https://github.com/bats-core/bats-assert tests/test_helper/bats-assert
|
||||
```
|
||||
|
||||
Run all tests for this skill (from the repo root):
|
||||
|
||||
```bash
|
||||
bats <destination-dir>/agent-audit/tests/
|
||||
```
|
||||
|
||||
## If no tests are needed
|
||||
|
||||
Delete this README and the `tests/` directory entirely.
|
||||
218
plugins/kyberforge/skills/agent-audit/tests/validate.bats
Normal file
218
plugins/kyberforge/skills/agent-audit/tests/validate.bats
Normal file
@@ -0,0 +1,218 @@
|
||||
#!/usr/bin/env bats
|
||||
|
||||
setup() {
|
||||
REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../../../../../" && pwd)"
|
||||
load "$REPO_ROOT/tests/test_helper/bats-support/load"
|
||||
load "$REPO_ROOT/tests/test_helper/bats-assert/load"
|
||||
|
||||
SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate.sh"
|
||||
TMPDIR="$(mktemp -d)"
|
||||
|
||||
# Helper: create a plugin-scope pair in <dir> with given <name>
|
||||
make_plugin_pair() {
|
||||
local dir="$1"
|
||||
local name="$2"
|
||||
mkdir -p "$dir"
|
||||
echo '{}' > "$dir/plugin.json"
|
||||
cat > "$dir/${name}.md" <<EOF
|
||||
---
|
||||
name: ${name}
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
cat > "$dir/${name}.agent.md" <<EOF
|
||||
---
|
||||
name: ${name}
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
}
|
||||
}
|
||||
|
||||
teardown() {
|
||||
rm -rf "$TMPDIR"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Passing cases
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "passes on a clean plugin-scope pair (CC file as input)" {
|
||||
local dir="$TMPDIR/agents"
|
||||
make_plugin_pair "$dir" "my-agent"
|
||||
run bash "$SCRIPT" "$dir/my-agent.md"
|
||||
assert_success
|
||||
refute_output --partial "FAIL"
|
||||
}
|
||||
|
||||
@test "passes on a clean project-scope pair (CC file as input)" {
|
||||
local root="$TMPDIR/project"
|
||||
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
||||
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
||||
assert_success
|
||||
refute_output --partial "FAIL"
|
||||
}
|
||||
|
||||
@test "--help exits 0 and shows Usage:" {
|
||||
run bash "$SCRIPT" --help
|
||||
assert_success
|
||||
assert_output --partial "Usage:"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Failing cases
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "fails when Copilot counterpart is missing" {
|
||||
local dir="$TMPDIR/agents"
|
||||
make_plugin_pair "$dir" "my-agent"
|
||||
rm "$dir/my-agent.agent.md"
|
||||
run bash "$SCRIPT" "$dir/my-agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "counterpart"
|
||||
}
|
||||
|
||||
@test "fails when CC-only field 'maxTurns' is in Copilot file" {
|
||||
local dir="$TMPDIR/agents"
|
||||
make_plugin_pair "$dir" "my-agent"
|
||||
cat > "$dir/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
maxTurns: 10
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$dir/my-agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "maxTurns"
|
||||
}
|
||||
|
||||
@test "fails when plugin-silently-ignored field 'hooks' is in plugin-scope CC file" {
|
||||
local dir="$TMPDIR/agents"
|
||||
make_plugin_pair "$dir" "my-agent"
|
||||
cat > "$dir/my-agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
hooks:
|
||||
PostToolUse:
|
||||
- match: ".*"
|
||||
command: "echo done"
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$dir/my-agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "hooks"
|
||||
}
|
||||
|
||||
@test "fails when CC file name is not kebab-case" {
|
||||
local dir="$TMPDIR/agents"
|
||||
mkdir -p "$dir"
|
||||
echo '{}' > "$dir/plugin.json"
|
||||
cat > "$dir/my-agent.md" <<EOF
|
||||
---
|
||||
name: MyAgent
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
cat > "$dir/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: MyAgent
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$dir/my-agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "kebab"
|
||||
}
|
||||
|
||||
@test "fails when 'name' field is missing from CC file" {
|
||||
local dir="$TMPDIR/agents"
|
||||
make_plugin_pair "$dir" "my-agent"
|
||||
cat > "$dir/my-agent.md" <<EOF
|
||||
---
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$dir/my-agent.md"
|
||||
assert_failure
|
||||
}
|
||||
|
||||
@test "fails when 'description' field is missing from CC file" {
|
||||
local dir="$TMPDIR/agents"
|
||||
make_plugin_pair "$dir" "my-agent"
|
||||
cat > "$dir/my-agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$dir/my-agent.md"
|
||||
assert_failure
|
||||
}
|
||||
|
||||
@test "fails when body contains unfilled FILL IN: placeholder" {
|
||||
local dir="$TMPDIR/agents"
|
||||
make_plugin_pair "$dir" "my-agent"
|
||||
cat > "$dir/my-agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
FILL IN: replace this with your system prompt.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$dir/my-agent.md"
|
||||
assert_failure
|
||||
}
|
||||
|
||||
@test "fails when frontmatter name does not match filename stem" {
|
||||
local dir="$TMPDIR/agents"
|
||||
make_plugin_pair "$dir" "my-agent"
|
||||
cat > "$dir/my-agent.md" <<EOF
|
||||
---
|
||||
name: wrong-name
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$dir/my-agent.md"
|
||||
assert_failure
|
||||
}
|
||||
|
||||
@test "fails when no arguments are given" {
|
||||
run bash "$SCRIPT"
|
||||
assert_failure
|
||||
}
|
||||
Reference in New Issue
Block a user