chore: remove neuledge-context skill and multiple kyberforge skills
Deletes the neuledge-context skill (.agents/skills/) and four kyberforge plugin skills — marketplace-architect, plugin-create, promptfoo, and write-agent — along with associated docs (adding-agents.md, plugin-marketplace-architecture.md). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -1,13 +0,0 @@
|
||||
# agents/
|
||||
|
||||
Agent definitions for this plugin. Each agent needs two files — one per tool:
|
||||
|
||||
| File | Tool | Notes |
|
||||
|---|---|---|
|
||||
| `<name>.md` | Claude Code | Frontmatter: `name`, `description`. No `tools:` field. |
|
||||
| `<name>.agent.md` | GitHub Copilot CLI | Frontmatter: `name`, `description`, `tools:` (array of permitted tool names). |
|
||||
|
||||
The two files share the same system prompt body. Keep them in sync.
|
||||
|
||||
Rename `PLUGIN_NAME.md` and `PLUGIN_NAME.agent.md` to your agent's name (kebab-case).
|
||||
Add additional agent pairs as needed — one pair per agent.
|
||||
@@ -1,334 +0,0 @@
|
||||
# Plugin Marketplace Architecture
|
||||
|
||||
Reference for refactoring this repo of skills/agents/hooks/prompts into a single Git-based
|
||||
plugin marketplace installable by **Claude Code** and **GitHub Copilot CLI**, and for building
|
||||
the `marketplace-architect` skill that automates the migration.
|
||||
|
||||
> **Provenance & staleness:** Verified against the official Claude Code plugin docs
|
||||
> (`code.claude.com/docs`) and GitHub Copilot CLI plugin docs (`docs.github.com`) as of
|
||||
> **June 2026**. Both ecosystems are moving fast; re-verify the divergence table before a
|
||||
> big migration. One item below (Copilot reading `.claude-plugin/plugin.json` per-plugin) is
|
||||
> **explicitly unverified** — see the flagged note. Don't treat that part as settled.
|
||||
|
||||
---
|
||||
|
||||
## 1. Core model (this part is correct and stable)
|
||||
|
||||
- The **Git repository is the marketplace.** No backend, registry API, database, SaaS, or MCP
|
||||
server is required. A marketplace is just a manifest file that lists plugins and where to find them.
|
||||
- A **plugin is the deployable unit.** Each plugin bundles one or more of: skills, agents, hooks,
|
||||
prompts/commands (flat `.md`), MCP servers, and — Claude Code only — output styles, LSP servers,
|
||||
background monitors, a `bin/` on PATH, and default `settings.json`. "Workflows" from the handoff
|
||||
aren't a distinct file type; express them as a skill that orchestrates steps, or a command.
|
||||
- **Organize by user outcome, not file type.** `startup-cto/` and `security-reviewer/`, not
|
||||
`all-skills/` and `all-agents/`.
|
||||
- **Aim for ~10–20 opinionated plugins**, not 50 tiny ones. (This is a usability judgment, not a
|
||||
hard rule from either vendor — but it's sound. Many skills can live inside one plugin.) The
|
||||
handoff's suggested set, as a starting shape: `startup-cto`, `system-architect`,
|
||||
`security-reviewer`, `product-manager`, `growth-marketer`, `developer-relations`,
|
||||
`technical-writer`, `research-analyst`.
|
||||
|
||||
Everything below is where the original handoff was either wrong or incomplete.
|
||||
|
||||
---
|
||||
|
||||
## 2. The two tools are convergent, NOT identical
|
||||
|
||||
This is the single most important correction. The handoff assumed "write once, run both."
|
||||
In reality the formats overlap heavily but diverge in specific, breaking ways. **Skills are the
|
||||
portable core; manifests and agents are where they split.**
|
||||
|
||||
**Recommended stance:** make **Claude Code the source of truth** (it's the stricter, more
|
||||
fully-specified format) and treat "loads in Copilot CLI" as a **tested checklist item per plugin**,
|
||||
not an assumption. Encode the Copilot deltas (manifest location, `.agent.md` naming) explicitly in
|
||||
the architect skill rather than pretending the two are the same. This is more honest than "write
|
||||
once, run both" and stops surprises at install time.
|
||||
|
||||
### Divergence table (ground truth)
|
||||
|
||||
| Concern | Claude Code | GitHub Copilot CLI | Portable choice |
|
||||
|---|---|---|---|
|
||||
| Marketplace manifest path | `.claude-plugin/marketplace.json` (required) | `.github/plugin/marketplace.json` (primary); **also reads `.claude-plugin/`** | Put it in `.claude-plugin/` — both read it. Optionally also `.github/plugin/`. |
|
||||
| Plugin manifest path | `.claude-plugin/plugin.json` (required; **only** plugin.json goes in this dir) | `plugin.json` at **plugin root** | ⚠️ See flagged note — may need it in **both** locations |
|
||||
| Skills | `skills/<name>/SKILL.md` | `skills/<name>/SKILL.md` | ✅ Identical — lean on these |
|
||||
| Agents | `agents/<name>.md` | `agents/<name>.agent.md` (frontmatter incl. `tools:`) | Diverges — keep portable logic in skills; ship per-tool agent files only when needed |
|
||||
| Hooks | `hooks/hooks.json` | `hooks.json` at plugin root | Diverges; declare paths in manifest to be safe |
|
||||
| MCP servers | `.mcp.json` at plugin root | `.mcp.json` at plugin root | ✅ Same |
|
||||
| Relative `source` | must start with `./` | `./x` and `x` both valid | Always use `./` — valid for both |
|
||||
| Validate command | `claude plugin validate .` (or `/plugin validate .`) | none documented | Claude validator + custom JSON checks for Copilot |
|
||||
| Install marketplace | `claude plugin marketplace add owner/repo` | `copilot plugin marketplace add owner/repo` | Same shape |
|
||||
| Install plugin | `claude plugin install <name>@<marketplace-name>` | from a registered marketplace by plugin name; **`@marketplace` suffix not confirmed** — `update`/`uninstall` take a bare `<name>`, so don't assume Claude's `@marketplace` form | ⚠️ Verify the Copilot install string before documenting it |
|
||||
| Local install (dev) | `claude --plugin-dir ./plugin` | `copilot plugin install ./plugin` | Tool-specific |
|
||||
|
||||
> ⚠️ **FLAGGED / UNVERIFIED — test this by hand before committing to a layout.**
|
||||
> Copilot's docs explicitly confirm it falls back to reading the **marketplace** manifest from
|
||||
> `.claude-plugin/`. They do **not** confirm the same fallback for an individual plugin's
|
||||
> `plugin.json`; the Copilot docs only show `plugin.json` at the plugin root. Claude *requires*
|
||||
> it in `.claude-plugin/`. Until you verify, the pragmatic move is to **ship `plugin.json` in
|
||||
> both** `plugin-name/plugin.json` and `plugin-name/.claude-plugin/plugin.json` (identical
|
||||
> content), then drop whichever proves redundant. The architect skill should generate both and
|
||||
> note the duplication.
|
||||
|
||||
### `@<marketplace-name>` resolves to the manifest `name`, not the repo
|
||||
|
||||
`claude plugin install startup-cto@my-ai-marketplace` requires the marketplace manifest's
|
||||
top-level `name` field to be exactly `my-ai-marketplace`. It is **not** the GitHub repo name.
|
||||
Keep them aligned to avoid confusion, but know they're separate things.
|
||||
|
||||
---
|
||||
|
||||
## 3. Canonical repo layout (cross-compatible)
|
||||
|
||||
```text
|
||||
repo-root/
|
||||
├── .claude-plugin/
|
||||
│ └── marketplace.json # both tools read here
|
||||
├── .github/plugin/
|
||||
│ └── marketplace.json # OPTIONAL: Copilot's canonical path (mirror of above)
|
||||
├── plugins/
|
||||
│ └── startup-cto/
|
||||
│ ├── plugin.json # Copilot root manifest ┐ ship both until
|
||||
│ ├── .claude-plugin/ # │ the §2 note is
|
||||
│ │ └── plugin.json # Claude manifest ┘ verified
|
||||
│ ├── skills/
|
||||
│ │ ├── fundraising/SKILL.md
|
||||
│ │ └── hiring/SKILL.md
|
||||
│ ├── agents/
|
||||
│ │ ├── startup-cto.md # Claude
|
||||
│ │ └── startup-cto.agent.md # Copilot (only if you ship native agents)
|
||||
│ ├── hooks/hooks.json # Claude
|
||||
│ ├── hooks.json # Copilot (if hooks used)
|
||||
│ ├── docs/
|
||||
│ └── README.md
|
||||
└── README.md
|
||||
```
|
||||
|
||||
If maintaining two manifest copies is annoying, generate the mirrors from one source in CI
|
||||
(see §7) rather than hand-editing both.
|
||||
|
||||
### Manifest shapes
|
||||
|
||||
`marketplace.json` (root):
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "my-ai-marketplace",
|
||||
"owner": { "name": "Your Name", "email": "you@example.com" },
|
||||
"metadata": { "description": "Agents, skills and workflows", "version": "1.0.0" },
|
||||
"plugins": [
|
||||
{ "name": "startup-cto", "source": "./plugins/startup-cto", "description": "..." },
|
||||
{ "name": "system-architect", "source": "./plugins/system-architect", "description": "..." }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`plugin.json` (keep minimal; add fields only when needed):
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "startup-cto",
|
||||
"version": "1.0.0",
|
||||
"description": "Startup technical leadership toolkit",
|
||||
"author": { "name": "Your Name" }
|
||||
}
|
||||
```
|
||||
|
||||
Plugin names must be **kebab-case** (lowercase, digits, hyphens). Claude.ai's marketplace sync
|
||||
rejects anything else even though the local CLI may tolerate it.
|
||||
|
||||
---
|
||||
|
||||
## 4. Gotchas the original handoff omitted
|
||||
|
||||
These will cause real breakage during refactor. The architect skill must check for them.
|
||||
|
||||
1. **Plugins are copied to a cache on install.** A plugin **cannot** reference files outside its
|
||||
own directory (e.g. `../shared-utils`) — those files aren't copied. If your current repo shares
|
||||
helper files across skills/agents, that sharing breaks. Fix by **duplicating** the shared file
|
||||
into each plugin or using **symlinks**. Audit for cross-references before moving anything.
|
||||
|
||||
2. **Version-pinning footgun.** If `plugin.json` sets `"version"` and you don't bump it on a new
|
||||
release, existing users get **no update** — the cached copy is kept. Either bump every release,
|
||||
or **omit `version`** so the git commit SHA is used (every commit = new version). Don't set
|
||||
`version` in both `plugin.json` and the marketplace entry; the `plugin.json` value wins silently.
|
||||
|
||||
3. **Reserved marketplace names.** Claude blocks a set of names (`anthropic-*`, `claude-*`,
|
||||
`agent-skills`, and impersonators like `official-claude-plugins`). Validate against these.
|
||||
|
||||
4. **`commands/` ≠ `skills/` in Claude.** A flat `foo.md` is a legacy *command*; a
|
||||
`foo/SKILL.md` directory is a *skill*. Promote flat command files to skill directories during
|
||||
migration — don't treat them as interchangeable.
|
||||
|
||||
5. **Use `${CLAUDE_PLUGIN_ROOT}`** in Claude hook/MCP configs to reference in-plugin files, since
|
||||
the plugin runs from a cache path, not its repo location.
|
||||
|
||||
6. **Strict mode (Claude).** A marketplace plugin entry defaults to `strict: true` — `plugin.json`
|
||||
is authoritative and the entry can only supplement it. Set `strict: false` to make the
|
||||
marketplace entry the *entire* definition (it then declares `skills`/`agents`/`hooks`/`mcpServers`
|
||||
path arrays itself, and the plugin needs no `plugin.json`). Useful when the architect curates a
|
||||
plugin's exposed components differently from how the files are laid out. Don't mix the two — a
|
||||
`strict:false` entry plus a component-declaring `plugin.json` is a conflict and fails to load.
|
||||
|
||||
7. **Plugin sources beyond relative paths (Claude).** This repo uses `./plugins/x` relative sources
|
||||
(simplest for a monorepo). If you later split a plugin into its own repo, the `source` field also
|
||||
supports `github` (`owner/repo` + `ref`/`sha`), `git-subdir` (sparse clone of a monorepo path),
|
||||
`url` (any git host), and `npm` (published package). Copilot's docs only *show* relative-path
|
||||
sources — other source types aren't documented there, so don't rely on them cross-tool. Stay on
|
||||
relative paths unless you have a reason not to.
|
||||
|
||||
8. **Copilot declares component paths in `plugin.json` (Claude does it differently).** Copilot's
|
||||
`plugin.json` can carry `"skills": "skills/"`, `"agents": "agents/"`, `"hooks": "hooks.json"`,
|
||||
`"mcpServers": ".mcp.json"` fields that tell it where components live. Claude instead defaults to
|
||||
the standard dirs and only takes path overrides via the *marketplace entry* (see strict mode,
|
||||
#6). So the same `plugin.json` may need these path fields for Copilot but not for Claude — another
|
||||
reason the architect should generate per-tool manifests rather than one shared file.
|
||||
|
||||
---
|
||||
|
||||
## 5. The `marketplace-architect` skill spec
|
||||
|
||||
Build this skill **on the corrected spec above** — not on the original handoff, which would bake
|
||||
in the format errors. It's a Claude Code skill (`skills/marketplace-architect/SKILL.md`) following
|
||||
standard skill conventions: a tight SKILL.md body (<500 lines) plus bundled `references/` and
|
||||
`scripts/` loaded progressively.
|
||||
|
||||
### Frontmatter (description is the trigger — make it pushy)
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: marketplace-architect
|
||||
description: >
|
||||
Audits a repository of Claude Code / Copilot CLI skills, agents, hooks, and prompts and
|
||||
refactors it into an installable plugin marketplace. Use this whenever the user wants to
|
||||
organize loose skills/agents into plugins, define plugin boundaries, generate plugin.json
|
||||
or marketplace.json manifests, plan a migration to a plugin marketplace, validate plugin
|
||||
naming or detect duplicate capabilities, or set up cross-tool (Claude Code + GitHub Copilot
|
||||
CLI) distribution — even if they don't say the word "marketplace".
|
||||
---
|
||||
```
|
||||
|
||||
### Responsibilities (from the handoff, refined)
|
||||
|
||||
1. **Audit** — walk the repo; classify every asset as skill / command / agent / hook / prompt / MCP.
|
||||
2. **Detect cross-references** — flag any `../` or shared-file dependencies that break under caching (§4.1).
|
||||
3. **Recommend plugin boundaries** — group by outcome; warn on 50-tiny-plugins sprawl.
|
||||
4. **Prevent duplicate capabilities** — diff skill descriptions/agents for overlap before splitting.
|
||||
5. **Generate manifests** — emit `plugin.json` (both locations per §2 note) and `marketplace.json`
|
||||
(`.claude-plugin/`, optionally mirror to `.github/plugin/`).
|
||||
6. **Validate naming** — kebab-case, reserved names, unique plugin names, `@name` ↔ manifest `name`.
|
||||
7. **Produce a migration plan** — concrete file-move list (old path → new path) as a checklist.
|
||||
8. **Emit cross-tool deltas** — for each plugin, note what's needed for Copilot (`.agent.md`,
|
||||
root `plugin.json`) vs Claude.
|
||||
9. **Generate release notes + install docs** — per-plugin README with both `claude` and `copilot`
|
||||
install commands.
|
||||
|
||||
### Suggested bundled structure
|
||||
|
||||
```text
|
||||
skills/marketplace-architect/
|
||||
├── SKILL.md
|
||||
├── references/
|
||||
│ ├── claude-code.md # Claude paths, validate, version rules, reserved names
|
||||
│ ├── copilot-cli.md # Copilot paths, .agent.md, marketplace fallback
|
||||
│ └── cross-compat.md # the §2 divergence table — the heart of the skill
|
||||
└── scripts/
|
||||
├── inventory.py # scan repo → classify assets → emit a table
|
||||
├── gen_manifests.py # write plugin.json + marketplace.json (both layouts)
|
||||
└── validate.py # wraps `claude plugin validate .` + JSON/naming checks
|
||||
```
|
||||
|
||||
Put the §2 divergence table verbatim into `references/cross-compat.md` — that's the knowledge the
|
||||
skill exists to apply. Keep SKILL.md to the workflow (audit → boundaries → generate → validate)
|
||||
and point it at the reference files.
|
||||
|
||||
### Authoring notes
|
||||
|
||||
- Use **imperative** instructions ("Scan the repo", "Emit the manifest").
|
||||
- Explain *why* a rule matters (e.g. the cache constraint) rather than bare MUSTs — the model
|
||||
applies judgment better with rationale.
|
||||
- After drafting, run 2–3 realistic test prompts (e.g. "turn this repo into a marketplace",
|
||||
"which of these skills belong together?") and iterate.
|
||||
|
||||
---
|
||||
|
||||
## 6. Refactor playbook (corrected phases)
|
||||
|
||||
Run these *with* the architect skill once it exists; it automates 1–5.
|
||||
|
||||
1. **Inventory.** Classify every asset (skill/command/agent/hook/prompt/MCP). Record current path.
|
||||
2. **Detect breakage.** Find cross-references and shared files (§4.1). Decide duplicate vs symlink.
|
||||
3. **Draw boundaries.** Group by outcome into ~10–20 plugins. De-dupe overlapping capabilities.
|
||||
4. **Move files.**
|
||||
- `skills/react.md` (flat command) → `plugins/system-architect/skills/react/SKILL.md`
|
||||
- `agents/startup-founder.md` → `plugins/startup-cto/agents/startup-founder.md`
|
||||
(+ `startup-founder.agent.md` if shipping Copilot-native agents)
|
||||
5. **Generate manifests.** Per-plugin `plugin.json` (both locations); root `marketplace.json`.
|
||||
6. **Validate.** `claude plugin validate .` — fix every warning. Then test-install in both tools:
|
||||
- `claude --plugin-dir ./plugins/<x>` then `/plugin install <x>@<marketplace>`
|
||||
- `copilot plugin install ./plugins/<x>` then `copilot plugin list` / `/skills list` / `/agent`
|
||||
7. **Document.** Per-plugin README with both install commands; root README listing all plugins.
|
||||
|
||||
---
|
||||
|
||||
## 7. Future / roadmap (preserved from the handoff)
|
||||
|
||||
Three post-migration features the original handoff called for. Not needed for the first cut, but
|
||||
recorded here so the intent isn't lost.
|
||||
|
||||
### Marketplace website
|
||||
|
||||
Generate a docs site **directly from `marketplace.json`** — no separate content source. Iterate
|
||||
over the `plugins` array to produce a browsable catalog (one page per plugin from its
|
||||
`description`/`README.md`), publishable to something like `marketplace.example.com` via GitHub
|
||||
Pages. Because the manifest is the single source of truth, the site never drifts from what's
|
||||
installable.
|
||||
|
||||
### Plugin templates
|
||||
|
||||
Add a `templates/` directory so contributors never start from scratch:
|
||||
|
||||
```text
|
||||
templates/
|
||||
skill-plugin/ # plugin.json + skills/<name>/SKILL.md skeleton
|
||||
agent-plugin/ # plugin.json + agents/ skeleton (both .md and .agent.md)
|
||||
workflow-plugin/ # plugin.json + a multi-step orchestration skill
|
||||
```
|
||||
|
||||
Each ships the dual-manifest layout from §3 so new plugins are cross-tool by default.
|
||||
|
||||
### CI validation (the cross-tool catch)
|
||||
|
||||
A GitHub Action can gate PRs, but remember: **`claude plugin validate` covers the Claude side
|
||||
only.** There's no Copilot validator, so the Action needs custom JSON checks for the Copilot
|
||||
layout. Minimum checks:
|
||||
|
||||
- Valid JSON in every `marketplace.json` / `plugin.json`.
|
||||
- Each plugin's `SKILL.md` files have valid YAML frontmatter.
|
||||
- Unique plugin names; kebab-case; no reserved names.
|
||||
- Every `source` path resolves to an existing directory.
|
||||
- (If mirroring) `.claude-plugin/` and `.github/plugin/` manifests are in sync.
|
||||
- Version consistency (no conflicting `version` in plugin.json vs marketplace entry).
|
||||
|
||||
Fail the PR on any violation so the contributor flow stays self-service.
|
||||
|
||||
### Success criteria (the target end-state)
|
||||
|
||||
A new contributor should be able to, with **no manual registry edits, no backend, no MCP
|
||||
dependency**: (1) fork the repo, (2) create a plugin folder, (3) add `plugin.json`, (4) add
|
||||
skills/agents, (5) open a PR, (6) have the plugin appear in the marketplace automatically once
|
||||
merged. If a step requires hand-editing a central list, the automation isn't done.
|
||||
|
||||
---
|
||||
|
||||
## Open questions to resolve first
|
||||
|
||||
Two facts couldn't be confirmed from the published docs. Settle both with a one-skill test plugin
|
||||
before the architect generates layouts:
|
||||
|
||||
1. **Does Copilot CLI load a plugin whose `plugin.json` lives only in `.claude-plugin/`?** (Install
|
||||
the test plugin both ways.) The answer decides whether you ship one manifest or two. Copilot's
|
||||
docs put `plugin.json` at the plugin root; Claude requires `.claude-plugin/`.
|
||||
2. **What is Copilot's exact install-from-marketplace command?** The how-to pages don't show the
|
||||
literal string, and Copilot's `update`/`uninstall` take a bare plugin name — so it may *not* use
|
||||
Claude's `<name>@<marketplace>` form. Run `copilot plugin install --help` and confirm before
|
||||
putting the command in any README or the architect's generated docs.
|
||||
@@ -1,21 +0,0 @@
|
||||
```yaml
|
||||
version: "1.0"
|
||||
updated: 2026-06-20
|
||||
|
||||
when: >
|
||||
Invoked when the user wants to create, manage, or maintain a plugin marketplace for
|
||||
Claude Code and/or GitHub Copilot CLI. Covers four operations: (a) audit and refactor
|
||||
a repository of skills/agents/hooks into a plugin marketplace layout, (b) adopt external
|
||||
plugins/skills/agents from outside sources, (c) update and maintain an existing
|
||||
marketplace.json and plugin manifests, (d) validate existing plugin manifests for naming,
|
||||
structure, and cross-tool compatibility. Also triggered implicitly when the user asks
|
||||
about distributing skills to a team, organizing loose skills into installable units, or
|
||||
setting up cross-tool distribution — even without saying "marketplace".
|
||||
|
||||
references:
|
||||
- https://code.claude.com/docs/en/plugins
|
||||
- https://code.claude.com/docs/en/plugin-marketplaces
|
||||
- https://code.claude.com/docs/en/plugins-reference
|
||||
- https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/plugins-creating
|
||||
- https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/plugins-marketplace
|
||||
```
|
||||
@@ -1,101 +0,0 @@
|
||||
---
|
||||
name: marketplace-architect
|
||||
description: >
|
||||
Manages and maintains a plugin marketplace for Claude Code and GitHub Copilot CLI.
|
||||
Use this whenever the user wants to: create or update a marketplace (marketplace.json,
|
||||
plugin.json manifests), adopt plugins/skills/agents/hooks from external sources, evaluate
|
||||
cross-tool compatibility between Claude Code and Copilot CLI, plan plugin groupings and
|
||||
boundaries, refactor a repository into marketplace format, validate plugin naming, detect
|
||||
duplicate capabilities, or generate per-plugin install docs — even if they don't use the
|
||||
word "marketplace". Do NOT use when the user wants to author a new skill from scratch
|
||||
(use write-skill), debug an existing skill (use diagnose), or run a direct plugin CLI
|
||||
command (copilot plugin install, claude plugin list).
|
||||
metadata:
|
||||
category: marketplace
|
||||
---
|
||||
|
||||
<requirements>
|
||||
|
||||
## Required inputs
|
||||
|
||||
- **Target operation** — what the user wants to do; inferred from request. If ambiguous, ask: audit/refactor, adopt an external plugin, update/maintain an existing marketplace, or validate manifests.
|
||||
- **Repository path** — path to the repo to act on; defaults to current working directory if not stated.
|
||||
- **Marketplace name** — kebab-case identifier (e.g. `my-ai-marketplace`); required only when generating a new `marketplace.json`. Infer from repo name if obvious, ask if not.
|
||||
- **Plugin source** — URL, GitHub slug, or local path; required only when adopting an external plugin.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Load `references/cross-compat.md` before any tool-specific decision — Claude Code and Copilot CLI diverge in ways that cause silent breakage at install time.
|
||||
- Never write files until the user has approved the plan at Gate A and the specific file contents at Gate B — two separate explicit approvals required.
|
||||
- If credential-shaped content is detected in any manifest field, halt and redirect to environment variable references (e.g. `$MY_TOKEN`) — do not generate the manifest.
|
||||
- Produce cross-tool deltas and per-plugin READMEs only when explicitly requested — do not generate them automatically.
|
||||
- Scripts in `scripts/` are loaded on demand by the step that needs them — never preloaded.
|
||||
- Flag any `../` cross-references in the audited repo before recommending plugin boundaries — plugins cannot reference files outside their own directory after install-time caching.
|
||||
- Plugin names must be kebab-case; validate against the reserved name list in `references/claude-code.md` before generating any manifest.
|
||||
- Do not set `version` in both `plugin.json` and the marketplace entry — `plugin.json` wins silently and causes update failures.
|
||||
|
||||
</requirements>
|
||||
|
||||
<steps>
|
||||
|
||||
## Process
|
||||
|
||||
1. **Identify the operation.** Determine intent from the user's request — one of: (a) audit/refactor a repo into marketplace format, (b) adopt an external plugin/skill/agent, (c) maintain or update an existing marketplace, (d) validate existing manifests. Ask if the operation cannot be inferred.
|
||||
|
||||
2. **Load the compatibility reference.** Read `references/cross-compat.md` before any tool-specific decision. Claude Code and Copilot CLI diverge in manifest paths, agent file naming, and hooks layout — every recommendation depends on this table.
|
||||
|
||||
3. **Execute the operation phase.**
|
||||
|
||||
**(a) Audit/refactor:** Run `scripts/inventory.sh` against the repo to classify every asset (skill / command / agent / hook / prompt / MCP). Flag any `../` cross-references — these break under install-time caching. Recommend plugin groupings by user outcome (~10–20 plugins); warn if proposed count exceeds 20 or falls below 3. Diff skill descriptions for duplicate capabilities before finalising boundaries. Produce a concrete migration checklist: old path → new path, one row per file.
|
||||
|
||||
**(b) Adopt external plugin:** Fetch and inspect the plugin source. Classify included assets. Check for naming conflicts with existing plugins in the marketplace. Evaluate cross-tool compatibility using `references/cross-compat.md`. Summarise what will be added to `marketplace.json`.
|
||||
|
||||
**(c) Maintain/update:** Read current `marketplace.json` and all `plugin.json` files. Identify stale versions, reserved name violations, kebab-case violations, and `version` duplication between plugin.json and marketplace entry. Report findings as a prioritised fix list.
|
||||
|
||||
**(d) Validate:** Run `scripts/validate.sh` (wraps `claude plugin validate` plus custom JSON and naming checks). Report each violation with a recommended fix. Do not proceed to file writes until all errors are resolved.
|
||||
|
||||
4. **Gate A — plan review.** Present the full plan or fix list to the user. Wait for explicit approval before proceeding. Do not interpret silence or "looks good" as approval — require a direct "yes" or equivalent.
|
||||
|
||||
5. **Generate outputs.** After Gate A approval: for audit/refactor and adopt operations, run `scripts/gen_manifests.sh` to produce `plugin.json` (at both `.claude-plugin/plugin.json` and plugin root until the Copilot fallback is verified) and `marketplace.json` (at `.claude-plugin/marketplace.json`; optionally mirror to `.github/plugin/marketplace.json`). Read `references/claude-code.md` for Claude-specific path rules and `references/copilot-cli.md` for Copilot-specific requirements.
|
||||
|
||||
6. **Gate B — file write approval.** Show the user every file that will be written with its full contents. Wait for explicit approval per file or as a batch. Write nothing until approved.
|
||||
|
||||
7. **Validate post-write.** After writes complete, run `scripts/validate.sh` again. Report any remaining issues. Suggest local install test commands: `claude --plugin-dir ./plugins/<name>` and `copilot plugin install ./plugins/<name>`.
|
||||
|
||||
8. **Optional deliverables.** Only when the user explicitly asks: emit cross-tool delta notes (what each plugin needs for Copilot vs Claude Code) and per-plugin README with install commands for both tools.
|
||||
|
||||
## Output format
|
||||
|
||||
Files generated depend on operation:
|
||||
- **Audit/refactor and adopt:** `plugin.json` (two locations per plugin until verified), `marketplace.json` (`.claude-plugin/`, optionally `.github/plugin/`), migration checklist as a markdown table
|
||||
- **Maintain/update:** updated `marketplace.json` and affected `plugin.json` files
|
||||
- **Validate:** report only — no file writes unless explicitly requested after review
|
||||
- **Optional:** per-plugin `README.md` with both `claude` and `copilot` install commands
|
||||
|
||||
</steps>
|
||||
|
||||
<checks>
|
||||
|
||||
## Failure handling
|
||||
|
||||
- `scripts/inventory.sh` not found or fails — perform manual asset classification using Read and Bash find; note the fallback in output.
|
||||
- `scripts/gen_manifests.sh` not found or fails — generate manifest JSON inline; flag that the output was not script-produced.
|
||||
- `scripts/validate.sh` not found or `claude plugin validate` unavailable — run manual JSON schema and naming checks using `references/claude-code.md`; flag that automated validation was skipped.
|
||||
- Plugin source unreachable (bad URL, private repo, missing path) — stop the adopt operation, report the error, ask the user to verify the source before retrying.
|
||||
- Reserved name detected in proposed plugin or marketplace name — halt, report the name and the reserved list from `references/claude-code.md`, ask for a replacement before proceeding.
|
||||
- Credential-shaped content detected in any manifest field — halt, do not generate the manifest, redirect to environment variable references.
|
||||
|
||||
## Self-check
|
||||
|
||||
- [ ] `references/cross-compat.md` loaded before any tool-specific recommendation was made
|
||||
- [ ] Operation identified before any scanning or file reading began
|
||||
- [ ] Gate A presented and explicit approval received before any manifest was generated
|
||||
- [ ] Gate B presented with full file contents and explicit approval received before any file was written
|
||||
- [ ] No credential-shaped content in any generated manifest field
|
||||
- [ ] All plugin names validated as kebab-case and checked against reserved name list
|
||||
- [ ] `version` field not set in both `plugin.json` and marketplace entry for the same plugin
|
||||
- [ ] Scripts loaded on demand by step — not preloaded at skill invocation
|
||||
- [ ] Cross-tool deltas and READMEs produced only if explicitly requested
|
||||
- [ ] Post-write validation run and findings reported
|
||||
|
||||
</checks>
|
||||
@@ -1,62 +0,0 @@
|
||||
skill_name: marketplace-architect
|
||||
|
||||
trigger_tests:
|
||||
- id: explicit-trigger-refactor
|
||||
name: "Explicit trigger — refactor repo into marketplace"
|
||||
query: "turn this repo into a plugin marketplace"
|
||||
should_trigger: true
|
||||
|
||||
- id: implicit-trigger-team-sharing
|
||||
name: "Implicit trigger — team distribution without saying marketplace"
|
||||
query: "how do I share my skills with my team so they can install them"
|
||||
should_trigger: true
|
||||
|
||||
- id: negative-trigger-new-skill
|
||||
name: "Negative trigger — new skill authoring belongs to write-skill"
|
||||
query: "write me a new skill for code review"
|
||||
should_trigger: false
|
||||
|
||||
output_tests:
|
||||
- id: deterministic-cross-compat-loaded
|
||||
name: "Deterministic — cross-compat reference loaded before tool-specific recommendations"
|
||||
type: deterministic
|
||||
prompt: "audit this repo and recommend how to convert it into a plugin marketplace for both Claude Code and Copilot CLI"
|
||||
expected_output: >
|
||||
The skill reads references/cross-compat.md early in its response and demonstrates
|
||||
awareness of the Claude Code / Copilot CLI format differences — specifically that
|
||||
manifest paths differ (.claude-plugin/ vs .github/plugin/), agent files differ
|
||||
(.md vs .agent.md), and hooks placement differs — before recommending any layout.
|
||||
assertions:
|
||||
- "Output references or acknowledges the divergence between Claude Code and Copilot CLI manifest paths before proposing a directory layout"
|
||||
- "Output does not recommend a single shared plugin.json location without noting the two-location requirement (.claude-plugin/ and plugin root)"
|
||||
- "Output does not write or propose writing any files before presenting a plan"
|
||||
|
||||
- id: deterministic-gate-a-blocks-writes
|
||||
name: "Deterministic — Gate A plan presented and approval requested before any writes"
|
||||
type: deterministic
|
||||
prompt: "help me migrate my skills and agents into a plugin marketplace layout"
|
||||
expected_output: >
|
||||
The skill produces a migration plan (checklist of old path → new path, one row per
|
||||
file) and explicitly asks the user for approval before proceeding to generate any
|
||||
manifest files. No plugin.json or marketplace.json is written or shown as written output.
|
||||
assertions:
|
||||
- "Output contains a migration plan or checklist with at least one file-move entry in the form 'old path → new path'"
|
||||
- "Output explicitly asks the user to approve or confirm the plan before proceeding"
|
||||
- "Output does not contain a complete plugin.json or marketplace.json file unless the user has already said 'yes' or equivalent in the prompt"
|
||||
|
||||
- id: llm-rubric-adopt-plugin
|
||||
name: "LLM-rubric — adopt external plugin covers all required checks without premature writes"
|
||||
type: llm-rubric
|
||||
prompt: "I want to add this plugin to my marketplace: https://github.com/example/my-tool-plugin"
|
||||
expected_output: >
|
||||
The skill fetches and inspects the plugin source, classifies what assets it contains
|
||||
(skills, agents, hooks, MCP servers), checks whether any existing plugin in the
|
||||
marketplace has a naming conflict, evaluates cross-tool compatibility against the
|
||||
divergence table, and summarises what would be added to marketplace.json — all without
|
||||
writing any files and without proceeding past Gate A without explicit user approval.
|
||||
assertions:
|
||||
- "Response identifies what asset types the external plugin contains (skills, agents, hooks, and/or MCP servers)"
|
||||
- "Response checks or asks about naming conflicts with existing plugins in the marketplace"
|
||||
- "Response notes at least one Claude Code vs Copilot CLI compatibility consideration for the adopted plugin"
|
||||
- "Response summarises the proposed change to marketplace.json without writing the file"
|
||||
- "Response asks for user approval (Gate A) before any file is created or modified"
|
||||
@@ -1,169 +0,0 @@
|
||||
# Claude Code Plugin Reference
|
||||
|
||||
Verified against code.claude.com/docs as of June 2026.
|
||||
|
||||
---
|
||||
|
||||
## Directory structure
|
||||
|
||||
```text
|
||||
plugin-root/
|
||||
├── .claude-plugin/
|
||||
│ └── plugin.json # ONLY plugin.json goes here; all other dirs at plugin root
|
||||
├── skills/ # skill directories: <name>/SKILL.md
|
||||
├── commands/ # legacy flat .md files; promote to skills/ for new plugins
|
||||
├── agents/ # agent definitions: <name>.md
|
||||
├── hooks/
|
||||
│ └── hooks.json
|
||||
├── .mcp.json
|
||||
├── .lsp.json
|
||||
├── monitors/
|
||||
│ └── monitors.json
|
||||
├── bin/ # executables added to PATH while plugin is enabled
|
||||
└── settings.json # default settings applied when plugin is enabled
|
||||
```
|
||||
|
||||
A plugin that ships exactly one skill may place `SKILL.md` directly at the plugin root.
|
||||
Use `skills/` for plugins that may grow beyond one skill.
|
||||
|
||||
---
|
||||
|
||||
## plugin.json schema
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "my-plugin", // kebab-case, no spaces — also the skill namespace prefix
|
||||
"displayName": "My Plugin", // human-readable; shown in UI (v2.1.143+)
|
||||
"description": "What it does",
|
||||
"version": "1.0.0", // OPTIONAL — omit to use git SHA per commit
|
||||
"author": { "name": "Name", "url": "https://..." },
|
||||
"homepage": "https://...",
|
||||
"repository": "https://github.com/...",
|
||||
"license": "MIT",
|
||||
"keywords": [],
|
||||
"defaultEnabled": true, // set false to install disabled (v2.1.154+)
|
||||
"dependencies": [
|
||||
{ "name": "other-plugin", "version": "~2.1.0" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Only `name` is required. Add fields only when needed.
|
||||
|
||||
---
|
||||
|
||||
## marketplace.json schema
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "my-ai-marketplace",
|
||||
"owner": { "name": "Your Name", "email": "you@example.com" },
|
||||
"description": "Description",
|
||||
"version": "1.0.0",
|
||||
"plugins": [
|
||||
{
|
||||
"name": "startup-cto",
|
||||
"source": "./plugins/startup-cto",
|
||||
"description": "...",
|
||||
"strict": true
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`description` and `version` are also accepted under a `metadata` key for backward compatibility.
|
||||
|
||||
---
|
||||
|
||||
## Plugin source types
|
||||
|
||||
| Type | Format | Notes |
|
||||
|---|---|---|
|
||||
| Relative path | `"./plugins/my-plugin"` | Must start with `./`. Resolved from marketplace root. Only works with git-hosted marketplaces, not URL-based. |
|
||||
| GitHub | `{ "source": "github", "repo": "owner/repo", "ref": "main", "sha": "abc123" }` | sha pins exact commit; ref is branch/tag |
|
||||
| URL / git | `{ "source": "url", "url": "https://...", "ref": "main" }` | also accepts `owner/repo` shorthand and SSH URLs |
|
||||
| git-subdir | `{ "source": "git-subdir", "url": "...", "path": "packages/my-plugin" }` | sparse clone of a monorepo path |
|
||||
| npm | `{ "source": "npm", "package": "@scope/pkg", "version": "^2.0.0", "registry": "https://..." }` | installed via npm install |
|
||||
|
||||
When both `ref` and `sha` are set, `sha` is the effective pin.
|
||||
|
||||
---
|
||||
|
||||
## Strict mode
|
||||
|
||||
Controls whether `plugin.json` is the authority for component definitions.
|
||||
|
||||
- **`strict: true`** (default) — plugin has its own `plugin.json`; marketplace entry can add extra skills/hooks on top.
|
||||
- **`strict: false`** — marketplace entry is the entire definition; plugin needs no `plugin.json`. The entry declares `skills`, `agents`, `hooks`, `mcpServers` path arrays.
|
||||
|
||||
Do not use `strict: false` plus a component-declaring `plugin.json` — this is a conflict and fails to load.
|
||||
|
||||
---
|
||||
|
||||
## Reserved marketplace names
|
||||
|
||||
These names are blocked for third-party use:
|
||||
`claude-code-marketplace`, `claude-code-plugins`, `claude-plugins-official`,
|
||||
`claude-plugins-community`, `claude-community`, `anthropic-marketplace`,
|
||||
`anthropic-plugins`, `agent-skills`, `anthropic-agent-skills`,
|
||||
`knowledge-work-plugins`, `life-sciences`, `claude-for-legal`,
|
||||
`claude-for-financial-services`, `financial-services-plugins`
|
||||
|
||||
Names that impersonate official marketplaces are also blocked (e.g. `official-claude-plugins`,
|
||||
`anthropic-tools-v2`).
|
||||
|
||||
Plugin names must be kebab-case (lowercase, digits, hyphens). Claude.ai marketplace sync
|
||||
rejects anything else even if the local CLI tolerates it.
|
||||
|
||||
---
|
||||
|
||||
## Version management
|
||||
|
||||
- If `version` is set in `plugin.json`, users receive updates only when you bump it.
|
||||
- If `version` is omitted, git commit SHA is used — every commit is a new version.
|
||||
- If `version` is set in both `plugin.json` and the marketplace entry, `plugin.json` wins silently.
|
||||
- **Recommendation:** omit `version` unless you need explicit release gates.
|
||||
|
||||
---
|
||||
|
||||
## Environment variables
|
||||
|
||||
- **`${CLAUDE_PLUGIN_ROOT}`** — absolute path to the plugin's installation directory. Use in hook commands and MCP/LSP configs for all in-plugin file references. This path changes on update.
|
||||
- **`${CLAUDE_PLUGIN_DATA}`** — persistent directory for plugin state that survives updates. Use for `node_modules`, generated code, caches.
|
||||
|
||||
---
|
||||
|
||||
## Validation and CLI commands
|
||||
|
||||
```bash
|
||||
# Validate plugin structure and manifest
|
||||
claude plugin validate ./my-plugin
|
||||
claude plugin validate ./my-plugin --strict # treat warnings as errors
|
||||
|
||||
# Install/manage
|
||||
claude plugin install <name>@<marketplace>
|
||||
claude plugin update <name>@<marketplace>
|
||||
claude plugin uninstall <name>
|
||||
claude plugin list
|
||||
claude plugin enable <name>
|
||||
claude plugin disable <name>
|
||||
|
||||
# Marketplace
|
||||
claude plugin marketplace add owner/repo
|
||||
claude plugin marketplace update
|
||||
|
||||
# Development
|
||||
claude --plugin-dir ./my-plugin # load without installing
|
||||
claude --plugin-dir ./my-plugin.zip # load from zip (v2.1.128+)
|
||||
claude plugin init my-tool # scaffold a skills-dir plugin
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key gotchas
|
||||
|
||||
1. **Plugins are copied to cache on install.** Cannot reference `../shared-utils` — those files are not copied. Duplicate shared files into each plugin or use symlinks.
|
||||
2. **`commands/` ≠ `skills/`.** Flat `foo.md` is a legacy command; `foo/SKILL.md` is a skill. Promote flat commands to skill directories during migration.
|
||||
3. **Only `plugin.json` in `.claude-plugin/`.** Skills, agents, hooks, and other directories must be at the plugin root, not inside `.claude-plugin/`.
|
||||
4. **Plugin names are skill namespace prefixes.** `name: my-plugin` means skills invoke as `/my-plugin:skill-name`.
|
||||
5. **`defaultEnabled: false` requires v2.1.154+.** Earlier versions ignore it and enable on install.
|
||||
@@ -1,143 +0,0 @@
|
||||
# GitHub Copilot CLI Plugin Reference
|
||||
|
||||
Verified against docs.github.com as of June 2026.
|
||||
|
||||
---
|
||||
|
||||
## Directory structure
|
||||
|
||||
```text
|
||||
plugin-root/
|
||||
├── plugin.json # at plugin root (NOT in .claude-plugin/)
|
||||
├── skills/ # skill directories: <name>/SKILL.md (same as Claude Code)
|
||||
├── agents/ # agent files: <name>.agent.md (differs from Claude Code)
|
||||
├── hooks.json # at plugin root (differs from Claude Code: hooks/hooks.json)
|
||||
└── .mcp.json # at plugin root (same as Claude Code)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## plugin.json schema (Copilot)
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "my-plugin",
|
||||
"description": "What it does",
|
||||
"version": "1.0.0",
|
||||
"author": { "name": "Name", "email": "you@example.com" },
|
||||
"license": "MIT",
|
||||
"keywords": [],
|
||||
"agents": "agents/",
|
||||
"skills": ["skills/"],
|
||||
"hooks": "hooks.json",
|
||||
"mcpServers": ".mcp.json"
|
||||
}
|
||||
```
|
||||
|
||||
Key difference from Claude Code: Copilot expects component path declarations inside
|
||||
`plugin.json` (`"skills": "skills/"`, `"agents": "agents/"`, etc.). Claude Code instead
|
||||
defaults to standard dirs and takes path overrides only via the marketplace entry.
|
||||
This means the same `plugin.json` may need these fields for Copilot but not for Claude.
|
||||
|
||||
---
|
||||
|
||||
## marketplace.json schema (Copilot)
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "my-ai-marketplace",
|
||||
"owner": { "name": "Your Name", "email": "you@example.com" },
|
||||
"metadata": { "description": "Agents, skills and workflows", "version": "1.0.0" },
|
||||
"plugins": [
|
||||
{
|
||||
"name": "startup-cto",
|
||||
"source": "./plugins/startup-cto",
|
||||
"description": "...",
|
||||
"version": "1.0.0"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Copilot's primary marketplace manifest path is `.github/plugin/marketplace.json`.
|
||||
It also reads `.claude-plugin/marketplace.json` as a fallback.
|
||||
Relative `source` paths: `./x` and `x` are both valid (Claude requires `./`).
|
||||
|
||||
---
|
||||
|
||||
## Agent file format
|
||||
|
||||
Copilot agents use `.agent.md` extension with frontmatter:
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: my-agent
|
||||
description: What this agent does
|
||||
tools:
|
||||
- read_file
|
||||
- run_command
|
||||
---
|
||||
|
||||
Agent instructions here.
|
||||
```
|
||||
|
||||
Claude Code agents use `.md` extension without the `.agent.md` suffix.
|
||||
If shipping agents for both tools, create both files:
|
||||
- `agents/my-agent.md` — Claude Code
|
||||
- `agents/my-agent.agent.md` — Copilot CLI
|
||||
|
||||
---
|
||||
|
||||
## CLI commands
|
||||
|
||||
```bash
|
||||
# Install plugin locally (development)
|
||||
copilot plugin install ./my-plugin
|
||||
|
||||
# List installed plugins
|
||||
copilot plugin list
|
||||
|
||||
# In interactive mode
|
||||
/plugin list
|
||||
/skills list
|
||||
/agent
|
||||
|
||||
# Reload after changes
|
||||
/reload-plugins
|
||||
|
||||
# Uninstall (uses bare plugin name, not @marketplace form)
|
||||
copilot plugin uninstall <name>
|
||||
|
||||
# Marketplace
|
||||
copilot plugin marketplace add owner/repo
|
||||
```
|
||||
|
||||
> ⚠️ **UNVERIFIED: Copilot marketplace install command.**
|
||||
> The `update`/`uninstall` commands take a bare `<name>`. Whether install from a marketplace
|
||||
> uses `<name>@<marketplace>` (Claude Code's form) or a bare `<name>` is not confirmed in docs.
|
||||
> Run `copilot plugin install --help` before documenting the install command anywhere.
|
||||
|
||||
---
|
||||
|
||||
## Validation
|
||||
|
||||
Copilot has no documented `plugin validate` command. For Copilot-side validation, run manual checks:
|
||||
- Valid JSON in `plugin.json` and `marketplace.json`
|
||||
- Required fields: `name`, `description`
|
||||
- Unique plugin names across marketplace
|
||||
- Kebab-case plugin names
|
||||
- All `source` paths resolve to existing directories
|
||||
- `.agent.md` files have valid YAML frontmatter with `name`, `description`, `tools`
|
||||
|
||||
---
|
||||
|
||||
## Key differences from Claude Code (summary)
|
||||
|
||||
| What | Claude Code | Copilot CLI |
|
||||
|---|---|---|
|
||||
| Plugin manifest location | `.claude-plugin/plugin.json` | `plugin.json` at plugin root |
|
||||
| Agent files | `agents/<name>.md` | `agents/<name>.agent.md` |
|
||||
| Hooks file | `hooks/hooks.json` | `hooks.json` at plugin root |
|
||||
| Component paths | Declared in marketplace entry | Declared in `plugin.json` |
|
||||
| Validate command | `claude plugin validate` | None — manual checks only |
|
||||
| Relative source `./` | Required | Optional (`x` also valid) |
|
||||
@@ -1,79 +0,0 @@
|
||||
# Cross-Tool Compatibility Reference
|
||||
|
||||
Claude Code and GitHub Copilot CLI share the plugin concept but diverge in specific, breaking
|
||||
ways. **Skills are the portable core. Manifests and agents are where they split.**
|
||||
|
||||
Make Claude Code the source of truth — it is the stricter, more fully specified format.
|
||||
Treat "loads in Copilot CLI" as a tested checklist item per plugin, not an assumption.
|
||||
|
||||
---
|
||||
|
||||
## Divergence table
|
||||
|
||||
| Concern | Claude Code | GitHub Copilot CLI | Portable choice |
|
||||
|---|---|---|---|
|
||||
| Marketplace manifest path | `.claude-plugin/marketplace.json` (required) | `.github/plugin/marketplace.json` (primary); also reads `.claude-plugin/` | Put it in `.claude-plugin/` — both read it. Optionally mirror to `.github/plugin/`. |
|
||||
| Plugin manifest path | `.claude-plugin/plugin.json` (required; only `plugin.json` goes in this dir) | `plugin.json` at **plugin root** | Ship in both locations until verified — see ⚠️ below |
|
||||
| Skills | `skills/<name>/SKILL.md` | `skills/<name>/SKILL.md` | ✅ Identical |
|
||||
| Agents | `agents/<name>.md` | `agents/<name>.agent.md` (frontmatter incl. `tools:`) | Diverges — keep portable logic in skills; ship per-tool agent files only when needed |
|
||||
| Hooks | `hooks/hooks.json` | `hooks.json` at plugin root | Diverges; declare paths in manifest to be safe |
|
||||
| MCP servers | `.mcp.json` at plugin root | `.mcp.json` at plugin root | ✅ Same |
|
||||
| Relative `source` | must start with `./` | `./x` and `x` both valid | Always use `./` — valid for both |
|
||||
| Validate command | `claude plugin validate .` (or `/plugin validate .`) | none documented | Run Claude validator + manual JSON checks for Copilot |
|
||||
| Install marketplace | `claude plugin marketplace add owner/repo` | `copilot plugin marketplace add owner/repo` | Same shape |
|
||||
| Install plugin | `claude plugin install <name>@<marketplace-name>` | install by plugin name; `@marketplace` suffix unconfirmed | ⚠️ Verify Copilot install string before documenting |
|
||||
| Local install (dev) | `claude --plugin-dir ./plugin` | `copilot plugin install ./plugin` | Tool-specific |
|
||||
| Component paths in plugin.json | Claude defaults to standard dirs; path overrides via marketplace entry only | `"skills": "skills/"`, `"agents": "agents/"`, etc. in plugin.json | Generate per-tool manifests rather than one shared file |
|
||||
|
||||
> ⚠️ **UNVERIFIED — test before committing to a layout.**
|
||||
> Copilot docs confirm it reads the **marketplace** manifest from `.claude-plugin/`. They do NOT
|
||||
> confirm the same fallback for a plugin's `plugin.json`. Copilot docs show `plugin.json` at plugin
|
||||
> root; Claude requires it in `.claude-plugin/`. Until verified: ship `plugin.json` in BOTH
|
||||
> `plugin-name/plugin.json` and `plugin-name/.claude-plugin/plugin.json` (identical content), then
|
||||
> drop whichever proves redundant.
|
||||
|
||||
---
|
||||
|
||||
## `@<marketplace-name>` resolution
|
||||
|
||||
`claude plugin install startup-cto@my-ai-marketplace` requires the marketplace manifest's
|
||||
top-level `name` field to be exactly `my-ai-marketplace`. It is **not** the GitHub repo name.
|
||||
Keep them aligned to avoid confusion, but they are separate fields.
|
||||
|
||||
---
|
||||
|
||||
## Canonical cross-compatible repo layout
|
||||
|
||||
```text
|
||||
repo-root/
|
||||
├── .claude-plugin/
|
||||
│ └── marketplace.json # both tools read here
|
||||
├── .github/plugin/
|
||||
│ └── marketplace.json # OPTIONAL: Copilot canonical path (mirror)
|
||||
├── plugins/
|
||||
│ └── startup-cto/
|
||||
│ ├── plugin.json # Copilot root manifest ┐ ship both until
|
||||
│ ├── .claude-plugin/ # │ the note above is
|
||||
│ │ └── plugin.json # Claude manifest ┘ verified
|
||||
│ ├── skills/
|
||||
│ │ ├── fundraising/SKILL.md
|
||||
│ │ └── hiring/SKILL.md
|
||||
│ ├── agents/
|
||||
│ │ ├── startup-cto.md # Claude
|
||||
│ │ └── startup-cto.agent.md # Copilot (only if shipping native agents)
|
||||
│ ├── hooks/hooks.json # Claude
|
||||
│ ├── hooks.json # Copilot (if hooks used)
|
||||
│ └── README.md
|
||||
└── README.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Open questions to resolve before generating layouts
|
||||
|
||||
1. **Does Copilot CLI load a plugin whose `plugin.json` lives only in `.claude-plugin/`?**
|
||||
Install a test plugin both ways. The answer decides whether to ship one manifest or two.
|
||||
|
||||
2. **What is Copilot's exact install-from-marketplace command?**
|
||||
Run `copilot plugin install --help`. The `update`/`uninstall` commands take a bare plugin
|
||||
name — the `@marketplace` form may not apply.
|
||||
@@ -1,194 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Generate plugin.json (in both locations) and marketplace.json.
|
||||
# Dry-run by default; pass --write to apply.
|
||||
#
|
||||
# Usage:
|
||||
# gen_manifests.sh <repo-root> --marketplace-name <name> [options]
|
||||
#
|
||||
# Options:
|
||||
# --marketplace-name <name> kebab-case marketplace identifier (required)
|
||||
# --author "Name <email>" author string (default: "Unknown <unknown@example.com>")
|
||||
# --plugins-dir <dir> subdir containing plugin folders (default: plugins)
|
||||
# --mirror-github also write to .github/plugin/marketplace.json
|
||||
# --write apply changes (default is dry run)
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
RESERVED_NAMES="claude-code-marketplace claude-code-plugins claude-plugins-official
|
||||
claude-plugins-community claude-community anthropic-marketplace anthropic-plugins
|
||||
agent-skills anthropic-agent-skills knowledge-work-plugins life-sciences
|
||||
claude-for-legal claude-for-financial-services financial-services-plugins"
|
||||
|
||||
RESERVED_PATTERNS="official-claude anthropic-tools claude-official"
|
||||
|
||||
is_kebab_case() {
|
||||
[[ "$1" =~ ^[a-z0-9]+(-[a-z0-9]+)*$ ]]
|
||||
}
|
||||
|
||||
is_reserved() {
|
||||
local name="$1"
|
||||
for n in $RESERVED_NAMES; do
|
||||
[[ "$name" == "$n" ]] && return 0
|
||||
done
|
||||
for p in $RESERVED_PATTERNS; do
|
||||
[[ "$name" == "$p"* ]] && return 0
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
validate_name() {
|
||||
local name="$1" context="$2"
|
||||
local ok=true
|
||||
if ! is_kebab_case "$name"; then
|
||||
echo "ERROR: $context: name '$name' is not kebab-case (lowercase, digits, hyphens only)." >&2
|
||||
ok=false
|
||||
fi
|
||||
if is_reserved "$name"; then
|
||||
echo "ERROR: $context: name '$name' is reserved for official Anthropic use." >&2
|
||||
ok=false
|
||||
fi
|
||||
[[ "$ok" == "true" ]]
|
||||
}
|
||||
|
||||
write_json() {
|
||||
local path="$1" content="$2" dry_run="$3"
|
||||
if [[ "$dry_run" == "true" ]]; then
|
||||
echo ""
|
||||
echo "--- $path (dry run) ---"
|
||||
echo "$content"
|
||||
else
|
||||
mkdir -p "$(dirname "$path")"
|
||||
echo "$content" > "$path"
|
||||
echo " Written: $path"
|
||||
fi
|
||||
}
|
||||
|
||||
# ── parse args ───────────────────────────────────────────────────────────────
|
||||
|
||||
ROOT=""
|
||||
MARKETPLACE_NAME=""
|
||||
AUTHOR="Unknown <unknown@example.com>"
|
||||
PLUGINS_DIR="plugins"
|
||||
MIRROR_GITHUB=false
|
||||
WRITE=false
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--marketplace-name) MARKETPLACE_NAME="$2"; shift 2 ;;
|
||||
--author) AUTHOR="$2"; shift 2 ;;
|
||||
--plugins-dir) PLUGINS_DIR="$2"; shift 2 ;;
|
||||
--mirror-github) MIRROR_GITHUB=true; shift ;;
|
||||
--write) WRITE=true; shift ;;
|
||||
-*) echo "Unknown option: $1" >&2; exit 1 ;;
|
||||
*) ROOT="$1"; shift ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ -z "$ROOT" || -z "$MARKETPLACE_NAME" ]]; then
|
||||
echo "Usage: gen_manifests.sh <repo-root> --marketplace-name <name> [--write]" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ROOT="$(cd "$ROOT" && pwd)"
|
||||
DRY_RUN=$( [[ "$WRITE" == "true" ]] && echo "false" || echo "true" )
|
||||
|
||||
[[ "$DRY_RUN" == "true" ]] && echo "DRY RUN — pass --write to apply changes"
|
||||
|
||||
# Parse author
|
||||
AUTHOR_NAME="${AUTHOR%% <*}"
|
||||
AUTHOR_EMAIL=""
|
||||
if [[ "$AUTHOR" =~ \<(.+)\> ]]; then
|
||||
AUTHOR_EMAIL="${BASH_REMATCH[1]}"
|
||||
fi
|
||||
|
||||
# Validate marketplace name
|
||||
validate_name "$MARKETPLACE_NAME" "marketplace" || exit 1
|
||||
|
||||
# Discover plugins
|
||||
PLUGINS_PATH="$ROOT/$PLUGINS_DIR"
|
||||
if [[ ! -d "$PLUGINS_PATH" ]]; then
|
||||
echo "No plugins directory found at $PLUGINS_PATH" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mapfile -t PLUGIN_DIRS < <(find "$PLUGINS_PATH" -mindepth 1 -maxdepth 1 -type d ! -name '.*' | sort)
|
||||
|
||||
if [[ ${#PLUGIN_DIRS[@]} -eq 0 ]]; then
|
||||
echo "No plugin directories found in $PLUGINS_PATH" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Found ${#PLUGIN_DIRS[@]} plugin(s)"
|
||||
|
||||
# Build plugins array for marketplace.json
|
||||
PLUGINS_JSON="[]"
|
||||
|
||||
for pd in "${PLUGIN_DIRS[@]}"; do
|
||||
pname="$(basename "$pd")"
|
||||
validate_name "$pname" "plugin '$pname'" || exit 1
|
||||
|
||||
# Read existing plugin.json if present
|
||||
existing_claude="$pd/.claude-plugin/plugin.json"
|
||||
existing_root="$pd/plugin.json"
|
||||
existing_desc="Plugin: $pname"
|
||||
existing_name="$pname"
|
||||
|
||||
for existing in "$existing_claude" "$existing_root"; do
|
||||
if [[ -f "$existing" ]] && jq -e . "$existing" >/dev/null 2>&1; then
|
||||
d=$(jq -r '.description // empty' "$existing")
|
||||
n=$(jq -r '.name // empty' "$existing")
|
||||
[[ -n "$d" ]] && existing_desc="$d"
|
||||
[[ -n "$n" ]] && existing_name="$n"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
# Warn on version
|
||||
for existing in "$existing_claude" "$existing_root"; do
|
||||
if [[ -f "$existing" ]] && jq -e '.version' "$existing" >/dev/null 2>&1; then
|
||||
echo " WARNING: plugin '$pname' sets version in plugin.json. Do not also set it in the marketplace entry — plugin.json wins silently."
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
# Build plugin.json
|
||||
plugin_json=$(jq -n \
|
||||
--arg name "$existing_name" \
|
||||
--arg desc "$existing_desc" \
|
||||
--arg aname "$AUTHOR_NAME" \
|
||||
--arg aemail "$AUTHOR_EMAIL" \
|
||||
'{name: $name, description: $desc, author: {name: $aname, email: $aemail}}')
|
||||
|
||||
write_json "$pd/plugin.json" "$plugin_json" "$DRY_RUN"
|
||||
write_json "$pd/.claude-plugin/plugin.json" "$plugin_json" "$DRY_RUN"
|
||||
|
||||
source="./$PLUGINS_DIR/$pname"
|
||||
PLUGINS_JSON=$(echo "$PLUGINS_JSON" | jq \
|
||||
--arg name "$existing_name" \
|
||||
--arg src "$source" \
|
||||
--arg desc "$existing_desc" \
|
||||
'. + [{name: $name, source: $src, description: $desc}]')
|
||||
done
|
||||
|
||||
# Build marketplace.json
|
||||
marketplace_json=$(jq -n \
|
||||
--arg name "$MARKETPLACE_NAME" \
|
||||
--arg aname "$AUTHOR_NAME" \
|
||||
--arg aemail "$AUTHOR_EMAIL" \
|
||||
--arg desc "$MARKETPLACE_NAME plugin marketplace" \
|
||||
--argjson plugins "$PLUGINS_JSON" \
|
||||
'{name: $name, owner: {name: $aname, email: $aemail}, description: $desc, plugins: $plugins}')
|
||||
|
||||
write_json "$ROOT/.claude-plugin/marketplace.json" "$marketplace_json" "$DRY_RUN"
|
||||
|
||||
if [[ "$MIRROR_GITHUB" == "true" ]]; then
|
||||
write_json "$ROOT/.github/plugin/marketplace.json" "$marketplace_json" "$DRY_RUN"
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
echo ""
|
||||
echo "--- End dry run. Pass --write to apply. ---"
|
||||
else
|
||||
echo ""
|
||||
echo "Done. Run scripts/validate.sh to verify."
|
||||
fi
|
||||
@@ -1,124 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Scan a repository and classify every asset as skill/command/agent/hook/prompt/MCP.
|
||||
# Outputs a markdown table of findings plus a list of cross-reference warnings.
|
||||
#
|
||||
# Usage: inventory.sh <repo-path>
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
if [[ $# -lt 1 ]]; then
|
||||
echo "Usage: inventory.sh <repo-path>" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ROOT="$(cd "$1" && pwd)"
|
||||
|
||||
if [[ ! -d "$ROOT" ]]; then
|
||||
echo "Error: $ROOT is not a directory" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ── classify assets ──────────────────────────────────────────────────────────
|
||||
|
||||
declare -a ROWS=()
|
||||
declare -a CROSS_REFS=()
|
||||
|
||||
while IFS= read -r -d '' path; do
|
||||
rel="${path#"$ROOT/"}"
|
||||
name="$(basename "$path")"
|
||||
dir="$(dirname "$rel")"
|
||||
parent="$(basename "$dir")"
|
||||
|
||||
# Skip hidden dirs except .claude-plugin and .github
|
||||
skip=false
|
||||
IFS='/' read -ra parts <<< "$dir"
|
||||
for part in "${parts[@]}"; do
|
||||
if [[ "$part" == .* && "$part" != ".claude-plugin" && "$part" != ".github" && "$part" != ".agents" ]]; then
|
||||
skip=true; break
|
||||
fi
|
||||
done
|
||||
$skip && continue
|
||||
|
||||
asset_type=""
|
||||
|
||||
case "$name" in
|
||||
SKILL.md) asset_type="skill" ;;
|
||||
hooks.json) asset_type="hook" ;;
|
||||
.mcp.json) asset_type="mcp" ;;
|
||||
.lsp.json) asset_type="lsp" ;;
|
||||
plugin.json) asset_type="manifest-plugin" ;;
|
||||
marketplace.json) asset_type="manifest-marketplace" ;;
|
||||
*.agent.md) asset_type="agent-copilot" ;;
|
||||
*.md)
|
||||
if [[ "$parent" == "agents" ]]; then
|
||||
asset_type="agent-claude"
|
||||
elif [[ "$parent" == "commands" ]]; then
|
||||
asset_type="command"
|
||||
elif [[ "$rel" != *"/skills/"* && "$rel" != *"/commands/"* && "$rel" != *"/agents/"* ]]; then
|
||||
asset_type="prompt"
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
|
||||
[[ -n "$asset_type" ]] && ROWS+=("$asset_type|$rel")
|
||||
|
||||
# Check for cross-references in text files — skip occurrences inside backtick spans
|
||||
case "$name" in *.md|*.json|*.sh)
|
||||
if grep '\.\.\/' "$path" 2>/dev/null | sed 's/`[^`]*`//g' | grep -q '\.\.\/'; then
|
||||
while IFS= read -r line; do
|
||||
lineno="${line%%:*}"
|
||||
content="${line#*:}"
|
||||
stripped=$(echo "$content" | sed 's/`[^`]*`//g')
|
||||
if echo "$stripped" | grep -q '\.\.\/'; then
|
||||
CROSS_REFS+=("$rel:$lineno: $content")
|
||||
fi
|
||||
done < <(grep -n '\.\.\/' "$path" 2>/dev/null | head -20)
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
|
||||
done < <(find "$ROOT" -type f -print0 | sort -z)
|
||||
|
||||
# ── report ───────────────────────────────────────────────────────────────────
|
||||
|
||||
echo "# Asset Inventory: $ROOT"
|
||||
echo ""
|
||||
echo "## Assets"
|
||||
echo ""
|
||||
echo "| Type | Path |"
|
||||
echo "|---|---|"
|
||||
for row in "${ROWS[@]+"${ROWS[@]}"}"; do
|
||||
type="${row%%|*}"
|
||||
path="${row#*|}"
|
||||
echo "| \`$type\` | \`$path\` |"
|
||||
done | sort
|
||||
|
||||
total="${#ROWS[@]}"
|
||||
echo ""
|
||||
echo "**Total: $total assets**"
|
||||
echo ""
|
||||
|
||||
# Summary by type
|
||||
echo "## Summary by type"
|
||||
echo ""
|
||||
for row in "${ROWS[@]+"${ROWS[@]}"}"; do
|
||||
echo "${row%%|*}"
|
||||
done | sort | uniq -c | while read -r count type; do
|
||||
echo "- \`$type\`: $count"
|
||||
done
|
||||
|
||||
# Cross-reference warnings
|
||||
echo ""
|
||||
if [[ ${#CROSS_REFS[@]} -gt 0 ]]; then
|
||||
echo "## ⚠️ Cross-reference warnings (${#CROSS_REFS[@]} found)"
|
||||
echo ""
|
||||
echo "These \`../\` references will break after install-time caching:"
|
||||
echo ""
|
||||
for ref in "${CROSS_REFS[@]}"; do
|
||||
echo "- \`$ref\`"
|
||||
done
|
||||
else
|
||||
echo "## Cross-references"
|
||||
echo ""
|
||||
echo "No \`../\` cross-references found. Safe to proceed with plugin boundaries."
|
||||
fi
|
||||
@@ -1,194 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPTS_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
PASS=0; FAIL=0
|
||||
# Use += to avoid ((var++)) returning 0 exit code when var was 0 under set -e
|
||||
|
||||
# ── helpers ─────────────────────────────────────────────────────────────────
|
||||
|
||||
tmpdir() { mktemp -d; }
|
||||
|
||||
assert_contains() {
|
||||
local label="$1" expected="$2" actual="$3"
|
||||
if echo "$actual" | grep -qF "$expected"; then
|
||||
echo " PASS: $label"
|
||||
PASS=$((PASS+1))
|
||||
else
|
||||
echo " FAIL: $label"
|
||||
echo " expected to contain: $expected"
|
||||
echo " got: $(echo "$actual" | head -5)"
|
||||
FAIL=$((FAIL+1))
|
||||
fi
|
||||
}
|
||||
|
||||
assert_not_contains() {
|
||||
local label="$1" unexpected="$2" actual="$3"
|
||||
if echo "$actual" | grep -qF "$unexpected"; then
|
||||
echo " FAIL: $label"
|
||||
echo " expected NOT to contain: $unexpected"
|
||||
FAIL=$((FAIL+1))
|
||||
else
|
||||
echo " PASS: $label"
|
||||
PASS=$((PASS+1))
|
||||
fi
|
||||
}
|
||||
|
||||
assert_exit() {
|
||||
local label="$1" expected="$2" actual="$3"
|
||||
if [[ "$actual" -eq "$expected" ]]; then
|
||||
echo " PASS: $label"
|
||||
PASS=$((PASS+1))
|
||||
else
|
||||
echo " FAIL: $label"
|
||||
echo " expected exit $expected, got $actual"
|
||||
FAIL=$((FAIL+1))
|
||||
fi
|
||||
}
|
||||
|
||||
assert_file_exists() {
|
||||
local label="$1" path="$2"
|
||||
if [[ -f "$path" ]]; then
|
||||
echo " PASS: $label"
|
||||
PASS=$((PASS+1))
|
||||
else
|
||||
echo " FAIL: $label"
|
||||
echo " file not found: $path"
|
||||
FAIL=$((FAIL+1))
|
||||
fi
|
||||
}
|
||||
|
||||
assert_file_absent() {
|
||||
local label="$1" path="$2"
|
||||
if [[ ! -e "$path" ]]; then
|
||||
echo " PASS: $label"
|
||||
PASS=$((PASS+1))
|
||||
else
|
||||
echo " FAIL: $label"
|
||||
echo " file should not exist: $path"
|
||||
FAIL=$((FAIL+1))
|
||||
fi
|
||||
}
|
||||
|
||||
# ── inventory.sh tests ───────────────────────────────────────────────────────
|
||||
|
||||
echo "=== inventory.sh ==="
|
||||
|
||||
# 1. SKILL.md classified as skill
|
||||
t=$(tmpdir)
|
||||
mkdir -p "$t/skills/my-skill"
|
||||
echo "---" > "$t/skills/my-skill/SKILL.md"
|
||||
out=$("$SCRIPTS_DIR/inventory.sh" "$t")
|
||||
assert_contains "SKILL.md classified as skill" "skill" "$out"
|
||||
rm -rf "$t"
|
||||
|
||||
# 2. agents/foo.agent.md classified as agent-copilot
|
||||
t=$(tmpdir)
|
||||
mkdir -p "$t/agents"
|
||||
touch "$t/agents/my-agent.agent.md"
|
||||
out=$("$SCRIPTS_DIR/inventory.sh" "$t")
|
||||
assert_contains "agents/*.agent.md classified as agent-copilot" "agent-copilot" "$out"
|
||||
rm -rf "$t"
|
||||
|
||||
# 3. ../ in file content produces cross-reference warning
|
||||
t=$(tmpdir)
|
||||
mkdir -p "$t/skills/my-skill"
|
||||
printf -- '---\ndescription: test\n---\nSee ../shared/file.md\n' > "$t/skills/my-skill/SKILL.md"
|
||||
out=$("$SCRIPTS_DIR/inventory.sh" "$t")
|
||||
assert_contains "../ cross-reference warning emitted" "Cross-reference" "$out"
|
||||
assert_contains "../ path shown in warning" "../shared/file.md" "$out"
|
||||
rm -rf "$t"
|
||||
|
||||
# ── gen_manifests.sh tests ───────────────────────────────────────────────────
|
||||
|
||||
echo ""
|
||||
echo "=== gen_manifests.sh ==="
|
||||
|
||||
# 4. Without --write, no files are created
|
||||
t=$(tmpdir)
|
||||
mkdir -p "$t/plugins/my-plugin/skills/hello"
|
||||
echo "---" > "$t/plugins/my-plugin/skills/hello/SKILL.md"
|
||||
"$SCRIPTS_DIR/gen_manifests.sh" "$t" --marketplace-name my-marketplace >/dev/null
|
||||
assert_file_absent "dry run: .claude-plugin/marketplace.json not written" "$t/.claude-plugin/marketplace.json"
|
||||
assert_file_absent "dry run: plugin.json not written" "$t/plugins/my-plugin/plugin.json"
|
||||
rm -rf "$t"
|
||||
|
||||
# 5. With --write, creates plugin.json in both locations
|
||||
t=$(tmpdir)
|
||||
mkdir -p "$t/plugins/my-plugin/skills/hello"
|
||||
echo "---" > "$t/plugins/my-plugin/skills/hello/SKILL.md"
|
||||
"$SCRIPTS_DIR/gen_manifests.sh" "$t" --marketplace-name my-marketplace --write >/dev/null
|
||||
assert_file_exists "--write: plugin root plugin.json created" "$t/plugins/my-plugin/plugin.json"
|
||||
assert_file_exists "--write: .claude-plugin/plugin.json created" "$t/plugins/my-plugin/.claude-plugin/plugin.json"
|
||||
rm -rf "$t"
|
||||
|
||||
# 6. Reserved name exits 1
|
||||
t=$(tmpdir)
|
||||
mkdir -p "$t/plugins/my-plugin"
|
||||
set +e
|
||||
out=$("$SCRIPTS_DIR/gen_manifests.sh" "$t" --marketplace-name claude-plugins-official 2>&1)
|
||||
code=$?
|
||||
set -e
|
||||
assert_exit "reserved name: exit 1" 1 "$code"
|
||||
assert_contains "reserved name: error message" "reserved" "$out"
|
||||
rm -rf "$t"
|
||||
|
||||
# ── validate.sh tests ────────────────────────────────────────────────────────
|
||||
|
||||
echo ""
|
||||
echo "=== validate.sh ==="
|
||||
|
||||
# 7. Invalid JSON exits 1
|
||||
t=$(tmpdir)
|
||||
mkdir -p "$t/.claude-plugin"
|
||||
echo "not json" > "$t/.claude-plugin/marketplace.json"
|
||||
set +e
|
||||
out=$("$SCRIPTS_DIR/validate.sh" "$t" 2>&1)
|
||||
code=$?
|
||||
set -e
|
||||
assert_exit "invalid JSON: exit 1" 1 "$code"
|
||||
assert_contains "invalid JSON: error message" "ERROR" "$out"
|
||||
rm -rf "$t"
|
||||
|
||||
# 8. Reserved marketplace name exits 1
|
||||
t=$(tmpdir)
|
||||
mkdir -p "$t/.claude-plugin"
|
||||
printf '{"name":"claude-plugins-official","plugins":[]}\n' > "$t/.claude-plugin/marketplace.json"
|
||||
set +e
|
||||
out=$("$SCRIPTS_DIR/validate.sh" "$t" 2>&1)
|
||||
code=$?
|
||||
set -e
|
||||
assert_exit "reserved marketplace name: exit 1" 1 "$code"
|
||||
assert_contains "reserved marketplace name: error message" "reserved" "$out"
|
||||
rm -rf "$t"
|
||||
|
||||
# 9. ../ in plugin file exits 1
|
||||
t=$(tmpdir)
|
||||
mkdir -p "$t/.claude-plugin" "$t/plugins/my-plugin/skills/hello"
|
||||
printf '{"name":"my-marketplace","plugins":[{"name":"my-plugin","source":"./plugins/my-plugin"}]}\n' \
|
||||
> "$t/.claude-plugin/marketplace.json"
|
||||
printf -- '---\ndescription: test\n---\nSee ../shared.md\n' \
|
||||
> "$t/plugins/my-plugin/skills/hello/SKILL.md"
|
||||
set +e
|
||||
out=$("$SCRIPTS_DIR/validate.sh" "$t" 2>&1)
|
||||
code=$?
|
||||
set -e
|
||||
assert_exit "..// in plugin: exit 1" 1 "$code"
|
||||
assert_contains "..// in plugin: error message" "../" "$out"
|
||||
rm -rf "$t"
|
||||
|
||||
# 10. No marketplace.json warns but exits 0
|
||||
t=$(tmpdir)
|
||||
set +e
|
||||
out=$("$SCRIPTS_DIR/validate.sh" "$t" 2>&1)
|
||||
code=$?
|
||||
set -e
|
||||
assert_exit "no marketplace.json: exit 0" 0 "$code"
|
||||
assert_contains "no marketplace.json: warning emitted" "WARN" "$out"
|
||||
rm -rf "$t"
|
||||
|
||||
# ── summary ──────────────────────────────────────────────────────────────────
|
||||
|
||||
echo ""
|
||||
echo "Results: $PASS passed, $FAIL failed"
|
||||
[[ $FAIL -eq 0 ]]
|
||||
@@ -1,332 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Validate plugin marketplace manifests for Claude Code and GitHub Copilot CLI.
|
||||
# Wraps `claude plugin validate` (Claude-side) and runs manual checks (Copilot-side).
|
||||
#
|
||||
# Usage:
|
||||
# validate.sh <repo-root>
|
||||
# validate.sh <repo-root> --plugin plugins/my-plugin
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
RESERVED_NAMES="claude-code-marketplace claude-code-plugins claude-plugins-official
|
||||
claude-plugins-community claude-community anthropic-marketplace anthropic-plugins
|
||||
agent-skills anthropic-agent-skills knowledge-work-plugins life-sciences
|
||||
claude-for-legal claude-for-financial-services financial-services-plugins"
|
||||
|
||||
RESERVED_PATTERNS="official-claude anthropic-tools claude-official"
|
||||
|
||||
ERRORS=0
|
||||
WARNINGS=0
|
||||
|
||||
error() { echo "ERROR: $1"; ((ERRORS++)) || true; }
|
||||
warn() { echo "WARN: $1"; ((WARNINGS++)) || true; }
|
||||
|
||||
is_kebab_case() { [[ "$1" =~ ^[a-z0-9]+(-[a-z0-9]+)*$ ]]; }
|
||||
|
||||
is_reserved() {
|
||||
local name="$1"
|
||||
for n in $RESERVED_NAMES; do [[ "$name" == "$n" ]] && return 0; done
|
||||
for p in $RESERVED_PATTERNS; do [[ "$name" == "$p"* ]] && return 0; done
|
||||
return 1
|
||||
}
|
||||
|
||||
validate_name() {
|
||||
local name="$1" context="$2"
|
||||
[[ -z "$name" ]] && { error "$context: name is missing or empty"; return 0; }
|
||||
is_kebab_case "$name" || error "$context: name '$name' is not kebab-case"
|
||||
if is_reserved "$name"; then
|
||||
error "$context: name '$name' is reserved for official Anthropic use"
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
valid_json() {
|
||||
local path="$1"
|
||||
if ! jq -e . "$path" >/dev/null 2>&1; then
|
||||
error "Invalid JSON in $path"
|
||||
return 1
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
validate_marketplace_json() {
|
||||
local path="$1"
|
||||
[[ -f "$path" ]] || return 0
|
||||
valid_json "$path" || return 0
|
||||
|
||||
local name
|
||||
name=$(jq -r '.name // empty' "$path")
|
||||
[[ -z "$name" ]] && error "$path: 'name' field is required" || validate_name "$name" "$path"
|
||||
|
||||
local plugins_type
|
||||
plugins_type=$(jq -r 'if .plugins | type == "array" then "ok" else "bad" end' "$path")
|
||||
if [[ "$plugins_type" != "ok" ]]; then
|
||||
error "$path: 'plugins' must be an array"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Check each plugin entry
|
||||
local seen_names=()
|
||||
while IFS= read -r pname; do
|
||||
# Duplicate check
|
||||
for seen in "${seen_names[@]+"${seen_names[@]}"}"; do
|
||||
if [[ "$seen" == "$pname" ]]; then error "$path: duplicate plugin name '$pname'"; fi
|
||||
done
|
||||
seen_names+=("$pname")
|
||||
validate_name "$pname" "$path plugin '$pname'"
|
||||
|
||||
# Source path check
|
||||
local src
|
||||
src=$(jq -r --arg n "$pname" '.plugins[] | select(.name==$n) | .source // empty' "$path")
|
||||
if [[ -n "$src" && "$src" != ./* && "$src" != "github" && "$src" != "npm" && "$src" != "url" && "$src" != "git-subdir" ]]; then
|
||||
warn "$path plugin '$pname': relative source '$src' should start with './' for Claude Code compatibility"
|
||||
fi
|
||||
|
||||
# Version duplication warning
|
||||
local has_ver
|
||||
has_ver=$(jq -r --arg n "$pname" '.plugins[] | select(.name==$n) | .version // empty' "$path")
|
||||
if [[ -n "$has_ver" ]]; then
|
||||
warn "$path plugin '$pname': version set in marketplace entry. If also set in plugin.json, plugin.json wins silently."
|
||||
fi
|
||||
|
||||
done < <(jq -r '.plugins[].name // empty' "$path")
|
||||
return 0
|
||||
}
|
||||
|
||||
validate_plugin_json() {
|
||||
local path="$1" marketplace_json="${2:-}"
|
||||
[[ -f "$path" ]] || return 0
|
||||
valid_json "$path" || return 0
|
||||
|
||||
local name
|
||||
name=$(jq -r '.name // empty' "$path")
|
||||
if [[ -z "$name" ]]; then
|
||||
warn "$path: 'name' field missing (plugin dir name will be used)"
|
||||
else
|
||||
validate_name "$name" "$path"
|
||||
|
||||
# Version duplication check
|
||||
if [[ -n "$marketplace_json" && -f "$marketplace_json" ]]; then
|
||||
local pver mver
|
||||
pver=$(jq -r '.version // empty' "$path")
|
||||
mver=$(jq -r --arg n "$name" '.plugins[]? | select(.name==$n) | .version // empty' "$marketplace_json")
|
||||
if [[ -n "$pver" && -n "$mver" ]]; then
|
||||
error "$path: version '$pver' set in both plugin.json and marketplace entry — plugin.json wins silently. Remove one."
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
validate_skill_md() {
|
||||
local path="$1"
|
||||
local content
|
||||
content=$(cat "$path")
|
||||
if [[ "$content" != ---* ]]; then
|
||||
warn "$path: SKILL.md has no YAML frontmatter"
|
||||
return
|
||||
fi
|
||||
if ! echo "$content" | awk 'NR>1 && /^---/' | grep -q '^---'; then
|
||||
error "$path: SKILL.md frontmatter not closed"
|
||||
return
|
||||
fi
|
||||
if ! echo "$content" | awk '/^---/{n++; if(n==2) exit} n==1' | grep -q 'description:'; then
|
||||
warn "$path: SKILL.md frontmatter missing 'description' field"
|
||||
fi
|
||||
}
|
||||
|
||||
validate_plugin_dir() {
|
||||
local pd="$1" marketplace_json="${2:-}"
|
||||
|
||||
local claude_manifest="$pd/.claude-plugin/plugin.json"
|
||||
local root_manifest="$pd/plugin.json"
|
||||
|
||||
if [[ ! -f "$claude_manifest" && ! -f "$root_manifest" ]]; then
|
||||
warn "$pd: no plugin.json found (will auto-discover components)"
|
||||
else
|
||||
validate_plugin_json "$claude_manifest" "$marketplace_json"
|
||||
validate_plugin_json "$root_manifest" "$marketplace_json"
|
||||
|
||||
# Sync check — shared identity fields must match; component path fields legitimately diverge
|
||||
if [[ -f "$claude_manifest" && -f "$root_manifest" ]]; then
|
||||
local field cv rv
|
||||
for field in name description version license; do
|
||||
cv=$(jq -r ".$field // empty" "$claude_manifest")
|
||||
rv=$(jq -r ".$field // empty" "$root_manifest")
|
||||
if [[ ( -n "$cv" || -n "$rv" ) && "$cv" != "$rv" ]]; then
|
||||
error "$pd: '$field' differs between .claude-plugin/plugin.json ('$cv') and plugin.json ('$rv')"
|
||||
fi
|
||||
done
|
||||
cv=$(jq -r '.author.name // empty' "$claude_manifest")
|
||||
rv=$(jq -r '.author.name // empty' "$root_manifest")
|
||||
if [[ ( -n "$cv" || -n "$rv" ) && "$cv" != "$rv" ]]; then
|
||||
error "$pd: 'author.name' differs between .claude-plugin/plugin.json ('$cv') and plugin.json ('$rv')"
|
||||
fi
|
||||
cv=$(jq -r '.keywords // [] | sort | join(",")' "$claude_manifest")
|
||||
rv=$(jq -r '.keywords // [] | sort | join(",")' "$root_manifest")
|
||||
if [[ "$cv" != "$rv" ]]; then
|
||||
error "$pd: 'keywords' differs between .claude-plugin/plugin.json and plugin.json"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# Components must not be inside .claude-plugin/
|
||||
for bad_dir in skills agents hooks commands; do
|
||||
if [[ -d "$pd/.claude-plugin/$bad_dir" ]]; then
|
||||
error "$pd/.claude-plugin/$bad_dir: only plugin.json belongs in .claude-plugin/; move $bad_dir/ to plugin root"
|
||||
fi
|
||||
done
|
||||
|
||||
# Validate SKILL.md files
|
||||
while IFS= read -r -d '' skill_md; do
|
||||
validate_skill_md "$skill_md"
|
||||
done < <(find "$pd" -name "SKILL.md" -print0 2>/dev/null)
|
||||
|
||||
# Cross-reference check — skip occurrences inside backtick spans (documentation text)
|
||||
while IFS= read -r -d '' f; do
|
||||
if grep '\.\.\/' "$f" 2>/dev/null | sed 's/`[^`]*`//g' | grep -q '\.\.\/'; then
|
||||
local rel="${f#"$pd/"}"
|
||||
error "$rel: contains '../' reference — plugins cannot access files outside their directory after caching"
|
||||
fi
|
||||
done < <(find "$pd" \( -name "*.md" -o -name "*.json" \) -print0 2>/dev/null)
|
||||
}
|
||||
|
||||
run_claude_validate() {
|
||||
local path="$1"
|
||||
if command -v claude >/dev/null 2>&1; then
|
||||
if ! claude plugin validate "$path" 2>&1; then
|
||||
error "claude plugin validate failed for $path"
|
||||
fi
|
||||
else
|
||||
warn "'claude' CLI not found — skipping claude plugin validate"
|
||||
fi
|
||||
}
|
||||
|
||||
# ── parse args ────────────────────────────────────────────────────────────────
|
||||
|
||||
ROOT=""
|
||||
PLUGIN_ONLY=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--plugin) PLUGIN_ONLY="$2"; shift 2 ;;
|
||||
-*) echo "Unknown option: $1" >&2; exit 1 ;;
|
||||
*) ROOT="$1"; shift ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ -z "$ROOT" ]]; then
|
||||
echo "Usage: validate.sh <repo-root> [--plugin <path>]" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ROOT="$(cd "$ROOT" && pwd)"
|
||||
|
||||
# ── validate marketplace.json ─────────────────────────────────────────────────
|
||||
|
||||
CLAUDE_MARKETPLACE="$ROOT/.claude-plugin/marketplace.json"
|
||||
COPILOT_MARKETPLACE="$ROOT/.github/plugin/marketplace.json"
|
||||
MARKETPLACE_JSON=""
|
||||
|
||||
for mp in "$CLAUDE_MARKETPLACE" "$COPILOT_MARKETPLACE"; do
|
||||
if [[ -f "$mp" ]]; then
|
||||
[[ -z "$MARKETPLACE_JSON" ]] && MARKETPLACE_JSON="$mp"
|
||||
validate_marketplace_json "$mp"
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ -z "$MARKETPLACE_JSON" ]]; then
|
||||
warn "No marketplace.json found. Expected at .claude-plugin/marketplace.json"
|
||||
fi
|
||||
|
||||
# Marketplace sync check — shared identity fields must match; description/version
|
||||
# legitimately differ in structure (Claude: top-level; Copilot: under metadata)
|
||||
if [[ -f "$CLAUDE_MARKETPLACE" && -f "$COPILOT_MARKETPLACE" ]]; then
|
||||
cm_val=$(jq -r '.name // empty' "$CLAUDE_MARKETPLACE")
|
||||
cp_val=$(jq -r '.name // empty' "$COPILOT_MARKETPLACE")
|
||||
if [[ "$cm_val" != "$cp_val" ]]; then
|
||||
error "marketplace: 'name' differs — .claude-plugin ('$cm_val') vs .github/plugin ('$cp_val')"
|
||||
fi
|
||||
|
||||
cm_val=$(jq -r '.owner.name // empty' "$CLAUDE_MARKETPLACE")
|
||||
cp_val=$(jq -r '.owner.name // empty' "$COPILOT_MARKETPLACE")
|
||||
if [[ ( -n "$cm_val" || -n "$cp_val" ) && "$cm_val" != "$cp_val" ]]; then
|
||||
error "marketplace: 'owner.name' differs — '$cm_val' vs '$cp_val'"
|
||||
fi
|
||||
|
||||
# description: Claude top-level, Copilot under metadata — compare values regardless of path
|
||||
cm_val=$(jq -r '.description // .metadata.description // empty' "$CLAUDE_MARKETPLACE")
|
||||
cp_val=$(jq -r '.metadata.description // .description // empty' "$COPILOT_MARKETPLACE")
|
||||
if [[ ( -n "$cm_val" || -n "$cp_val" ) && "$cm_val" != "$cp_val" ]]; then
|
||||
error "marketplace: description differs between .claude-plugin/marketplace.json and .github/plugin/marketplace.json"
|
||||
fi
|
||||
|
||||
# version: same structural divergence as description
|
||||
cm_val=$(jq -r '.version // .metadata.version // empty' "$CLAUDE_MARKETPLACE")
|
||||
cp_val=$(jq -r '.metadata.version // .version // empty' "$COPILOT_MARKETPLACE")
|
||||
if [[ ( -n "$cm_val" || -n "$cp_val" ) && "$cm_val" != "$cp_val" ]]; then
|
||||
error "marketplace: version differs — '$cm_val' vs '$cp_val'"
|
||||
fi
|
||||
|
||||
# Plugin catalog must be identical across both files
|
||||
cm_plugins=$(jq -r '.plugins[].name' "$CLAUDE_MARKETPLACE" 2>/dev/null | sort)
|
||||
cp_plugins=$(jq -r '.plugins[].name' "$COPILOT_MARKETPLACE" 2>/dev/null | sort)
|
||||
if [[ "$cm_plugins" != "$cp_plugins" ]]; then
|
||||
error "marketplace: plugin lists differ between .claude-plugin/marketplace.json and .github/plugin/marketplace.json"
|
||||
else
|
||||
while IFS= read -r pname; do
|
||||
[[ -z "$pname" ]] && continue
|
||||
cm_val=$(jq -r --arg n "$pname" '.plugins[] | select(.name==$n) | .source // empty' "$CLAUDE_MARKETPLACE")
|
||||
cp_val=$(jq -r --arg n "$pname" '.plugins[] | select(.name==$n) | .source // empty' "$COPILOT_MARKETPLACE")
|
||||
if [[ "$cm_val" != "$cp_val" ]]; then
|
||||
error "marketplace plugin '$pname': source differs — '$cm_val' vs '$cp_val'"
|
||||
fi
|
||||
cm_val=$(jq -r --arg n "$pname" '.plugins[] | select(.name==$n) | .description // empty' "$CLAUDE_MARKETPLACE")
|
||||
cp_val=$(jq -r --arg n "$pname" '.plugins[] | select(.name==$n) | .description // empty' "$COPILOT_MARKETPLACE")
|
||||
if [[ ( -n "$cm_val" || -n "$cp_val" ) && "$cm_val" != "$cp_val" ]]; then
|
||||
error "marketplace plugin '$pname': description differs between the two marketplace.json files"
|
||||
fi
|
||||
done <<< "$cm_plugins"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── validate plugins ──────────────────────────────────────────────────────────
|
||||
|
||||
if [[ -n "$PLUGIN_ONLY" ]]; then
|
||||
validate_plugin_dir "$(cd "$PLUGIN_ONLY" && pwd)" "$MARKETPLACE_JSON"
|
||||
run_claude_validate "$(cd "$PLUGIN_ONLY" && pwd)"
|
||||
else
|
||||
plugins_path="$ROOT/plugins"
|
||||
if [[ -d "$plugins_path" ]]; then
|
||||
while IFS= read -r -d '' pd; do
|
||||
validate_plugin_dir "$pd" "$MARKETPLACE_JSON"
|
||||
run_claude_validate "$pd"
|
||||
done < <(find "$plugins_path" -mindepth 1 -maxdepth 1 -type d ! -name '.*' -print0 | sort -z)
|
||||
else
|
||||
warn "No plugins/ directory found at $ROOT"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── check source paths resolve ────────────────────────────────────────────────
|
||||
|
||||
if [[ -n "$MARKETPLACE_JSON" ]]; then
|
||||
while IFS= read -r src; do
|
||||
[[ "$src" != ./* ]] && continue
|
||||
src_path="$ROOT/${src#./}"
|
||||
if [[ ! -d "$src_path" ]]; then error "Marketplace source path '$src' does not exist at $src_path"; fi
|
||||
done < <(jq -r '.plugins[]?.source | strings' "$MARKETPLACE_JSON" 2>/dev/null)
|
||||
fi
|
||||
|
||||
# ── report ────────────────────────────────────────────────────────────────────
|
||||
|
||||
echo ""
|
||||
if [[ $ERRORS -eq 0 && $WARNINGS -eq 0 ]]; then
|
||||
echo "✓ All checks passed."
|
||||
exit 0
|
||||
elif [[ $ERRORS -eq 0 ]]; then
|
||||
echo "Passed with $WARNINGS warning(s)."
|
||||
exit 0
|
||||
else
|
||||
echo "Failed. Fix $ERRORS error(s) before proceeding."
|
||||
exit 1
|
||||
fi
|
||||
@@ -1,13 +0,0 @@
|
||||
```yaml
|
||||
version: "1.0"
|
||||
updated: 2026-06-20
|
||||
|
||||
when: Invoked when the user wants to create a new plugin in the marketplace. Scaffolds the
|
||||
directory structure from assets/plugin-template/ (bundled inside this skill), substitutes
|
||||
PLUGIN_NAME/PLUGIN_DESCRIPTION/AUTHOR_NAME/AUTHOR_EMAIL/AUTHOR_URL placeholders, writes to
|
||||
plugins/<name>/, registers the plugin in .claude-plugin/marketplace.json, runs
|
||||
claude plugin validate ., and hands off to /marketplace-architect.
|
||||
|
||||
references:
|
||||
- "${CLAUDE_PLUGIN_ROOT}/docs/plugin-marketplace-architecture.md"
|
||||
```
|
||||
@@ -1,89 +0,0 @@
|
||||
---
|
||||
name: plugin-create
|
||||
description: >
|
||||
Use when the user wants to create a new plugin in the marketplace — scaffold the directory
|
||||
structure, generate plugin.json manifests for Claude Code and Copilot CLI, and register
|
||||
the plugin in marketplace.json. Triggers: "create a new plugin", "add a plugin called X",
|
||||
"scaffold a plugin", "new plugin for Y". Do NOT use when auditing or refactoring existing
|
||||
plugins (use /marketplace-architect), authoring skill content inside a plugin (use
|
||||
/write-skill), adopting an external plugin into the marketplace (use /marketplace-architect),
|
||||
or validating existing manifests without creating anything (use /marketplace-architect).
|
||||
metadata:
|
||||
category: marketplace
|
||||
---
|
||||
|
||||
<requirements>
|
||||
|
||||
## Required inputs
|
||||
|
||||
- **Plugin name** — kebab-case slug; ask if not stated. Validate: lowercase letters, digits, hyphens only; not already present in `plugins/` or `.claude-plugin/marketplace.json`; not a reserved name. Load `references/reserved-names.md` for the full reserved list.
|
||||
- **Plugin description** — one sentence; ask if not stated.
|
||||
- **Author name** — ask if not stated.
|
||||
- **Author email** — ask if not stated; used in the Copilot root `plugin.json`.
|
||||
- **Author URL** — ask if not stated; used in Claude's `.claude-plugin/plugin.json`.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Load the plugin template from `assets/plugin-template/` bundled inside this skill (`${CLAUDE_PLUGIN_ROOT}/skills/plugin-create/assets/plugin-template/`). Stop if the path is missing — do not generate files from memory.
|
||||
- Plugin name must be kebab-case and not a reserved name — see `references/reserved-names.md` for the full list; halt and ask for a replacement before Gate A if violated.
|
||||
- Do not write any file until Gate A (plan approval) and Gate B (file contents approval) are both explicitly confirmed.
|
||||
- Replace all five placeholder markers — `PLUGIN_NAME`, `PLUGIN_DESCRIPTION`, `AUTHOR_NAME`, `AUTHOR_EMAIL`, `AUTHOR_URL` — in every copied file before Gate B review. No marker may appear in written output.
|
||||
- Do not generate skill content — `skills/` is scaffolded as an empty directory with README only. Direct the user to `/write-skill` to add skills.
|
||||
- Do not set `version` in both `plugin.json` and the marketplace entry — `plugin.json` wins silently and blocks updates for existing users.
|
||||
- Update `.github/plugin/marketplace.json` only if it already exists — do not create it.
|
||||
- Do not auto-invoke `/marketplace-architect` — hand off by name at the end; let the user trigger it.
|
||||
|
||||
</requirements>
|
||||
|
||||
<steps>
|
||||
|
||||
## Process
|
||||
|
||||
1. **Collect inputs.** Ask for plugin name, description, author name, author email, and author URL — one question at a time. Validate the plugin name: kebab-case format, not already present in `plugins/` or `.claude-plugin/marketplace.json`, not a reserved name (load `references/reserved-names.md` to check). If any validation fails, stop and ask for a replacement before continuing.
|
||||
|
||||
2. **Check template.** Load the bundled template from `assets/plugin-template/` inside this skill (`${CLAUDE_PLUGIN_ROOT}/skills/plugin-create/assets/plugin-template/`). If the path is missing, stop and report it — do not proceed or generate files from memory.
|
||||
|
||||
3. **Gate A — plan review.** Present: the list of files that will be written (derived from `assets/plugin-template/` with `PLUGIN_NAME` substituted into filenames), the new `plugins/<name>/` directory path, the marketplace entry that will be added to `.claude-plugin/marketplace.json`, and whether `.github/plugin/marketplace.json` will also be updated. Wait for explicit approval — do not proceed on "looks good" or silence.
|
||||
|
||||
4. **Copy and substitute.** Copy `assets/plugin-template/` to `plugins/<name>/`. In every copied file, replace all occurrences of `PLUGIN_NAME`, `PLUGIN_DESCRIPTION`, `AUTHOR_NAME`, `AUTHOR_EMAIL`, and `AUTHOR_URL` with the collected values. Rename any file or directory whose name contains `PLUGIN_NAME`. Note: after substitution, `displayName` in `.claude-plugin/plugin.json` will equal the kebab slug — remind the user to update it to a human-readable string (e.g. "My Plugin") before publishing.
|
||||
|
||||
5. **Gate B — file contents review.** Show every file with its full substituted content. Wait for explicit approval — do not write until confirmed.
|
||||
|
||||
6. **Write files.** Write all substituted files to `plugins/<name>/`. Append the new plugin entry to `.claude-plugin/marketplace.json`. If `.github/plugin/marketplace.json` exists, append the same entry there.
|
||||
|
||||
7. **Validate.** Run `claude plugin validate .` from `plugins/<name>/`. Report all output inline — do not suppress warnings. If the command is unavailable, note it and suggest the user run it manually after local install. For full cross-tool and marketplace validation, direct the user to run `/marketplace-architect validate` after the plugin is installed.
|
||||
|
||||
8. **Hand off.** Print: "Plugin `<name>` created and registered in `marketplace.json`. Fill in skill and agent content, then run `/marketplace-architect` to audit the full marketplace."
|
||||
|
||||
## Output format
|
||||
|
||||
- `plugins/<name>/` — directory tree copied from `assets/plugin-template/` with all placeholders substituted
|
||||
- `.claude-plugin/marketplace.json` — updated with the new plugin entry
|
||||
- `.github/plugin/marketplace.json` — updated with the same entry, only if it already existed
|
||||
- Inline validation report from `claude plugin validate .`
|
||||
|
||||
</steps>
|
||||
|
||||
<checks>
|
||||
|
||||
## Failure handling
|
||||
|
||||
- `assets/plugin-template/` not found at `${CLAUDE_PLUGIN_ROOT}/skills/plugin-create/assets/plugin-template/` — stop, report the path, do not generate from memory.
|
||||
- Plugin name already exists in `plugins/` or `marketplace.json` — stop, report the conflict, ask for a different name.
|
||||
- Reserved name detected — stop, report the name and the reserved list, ask for a replacement before continuing.
|
||||
- `claude plugin validate .` unavailable — report that automated validation was skipped; suggest running it manually with `claude --plugin-dir ./plugins/<name>`.
|
||||
|
||||
## Self-check
|
||||
|
||||
- [ ] Plugin name validated: kebab-case, not reserved, not already present in `plugins/` or `marketplace.json`
|
||||
- [ ] `assets/plugin-template/` verified to exist at `${CLAUDE_PLUGIN_ROOT}/skills/plugin-create/assets/plugin-template/` before any file generation
|
||||
- [ ] Gate A presented with file list and marketplace entry — explicit approval received
|
||||
- [ ] All five markers substituted in all files — none appear in written output
|
||||
- [ ] Gate B presented with full substituted file contents — explicit approval received
|
||||
- [ ] Files written only after Gate B approval
|
||||
- [ ] `.github/plugin/marketplace.json` updated only if it already existed — not created
|
||||
- [ ] `claude plugin validate .` run and results reported; or unavailability noted
|
||||
- [ ] Hand-off message printed directing user to `/marketplace-architect`
|
||||
- [ ] No skill content generated — `skills/` is empty with README only
|
||||
|
||||
</checks>
|
||||
@@ -1,8 +0,0 @@
|
||||
{
|
||||
"name": "PLUGIN_NAME",
|
||||
"displayName": "PLUGIN_NAME",
|
||||
"description": "PLUGIN_DESCRIPTION",
|
||||
"author": { "name": "AUTHOR_NAME", "url": "AUTHOR_URL" },
|
||||
"license": "MIT",
|
||||
"keywords": []
|
||||
}
|
||||
@@ -1,3 +0,0 @@
|
||||
{
|
||||
"mcpServers": {}
|
||||
}
|
||||
@@ -1,42 +0,0 @@
|
||||
# PLUGIN_NAME
|
||||
|
||||
PLUGIN_DESCRIPTION
|
||||
|
||||
## Install
|
||||
|
||||
**Claude Code:**
|
||||
|
||||
```bash
|
||||
claude plugin marketplace add <owner>/<repo>
|
||||
claude plugin install PLUGIN_NAME@<marketplace-name>
|
||||
```
|
||||
|
||||
**GitHub Copilot CLI:**
|
||||
|
||||
```bash
|
||||
copilot plugin marketplace add <owner>/<repo>
|
||||
copilot plugin install PLUGIN_NAME
|
||||
```
|
||||
|
||||
**Local (development):**
|
||||
|
||||
```bash
|
||||
# Claude Code
|
||||
claude --plugin-dir ./plugins/PLUGIN_NAME
|
||||
|
||||
# GitHub Copilot CLI
|
||||
copilot plugin install ./plugins/PLUGIN_NAME
|
||||
```
|
||||
|
||||
## Contents
|
||||
|
||||
| Component | Path | Description |
|
||||
|---|---|---|
|
||||
| Skills | `skills/` | Slash commands available after install |
|
||||
| Agents | `agents/` | Role-based agents (`.md` for Claude, `.agent.md` for Copilot) |
|
||||
| Hooks | `hooks/hooks.json` (Claude) / `hooks.json` (Copilot) | Event-triggered automation |
|
||||
| MCP servers | `.mcp.json` | Model Context Protocol server definitions |
|
||||
|
||||
## Author
|
||||
|
||||
AUTHOR_NAME
|
||||
@@ -1,13 +0,0 @@
|
||||
---
|
||||
name: PLUGIN_NAME
|
||||
description: PLUGIN_DESCRIPTION
|
||||
tools:
|
||||
- read_file
|
||||
- list_directory
|
||||
---
|
||||
|
||||
Replace this with your agent's system prompt and instructions.
|
||||
|
||||
This file is loaded by **GitHub Copilot CLI**. The `tools:` frontmatter field declares
|
||||
which tools the agent can use — add or remove tools as needed.
|
||||
For Claude Code, see `PLUGIN_NAME.md`.
|
||||
@@ -1,8 +0,0 @@
|
||||
---
|
||||
name: PLUGIN_NAME
|
||||
description: PLUGIN_DESCRIPTION
|
||||
---
|
||||
|
||||
Replace this with your agent's system prompt and instructions.
|
||||
|
||||
This file is loaded by **Claude Code**. For GitHub Copilot CLI, see `PLUGIN_NAME.agent.md`.
|
||||
@@ -1,13 +0,0 @@
|
||||
# agents/
|
||||
|
||||
Agent definitions for this plugin. Each agent needs two files — one per tool:
|
||||
|
||||
| File | Tool | Notes |
|
||||
|---|---|---|
|
||||
| `<name>.md` | Claude Code | Frontmatter: `name`, `description`. No `tools:` field. |
|
||||
| `<name>.agent.md` | GitHub Copilot CLI | Frontmatter: `name`, `description`, `tools:` (array of permitted tool names). |
|
||||
|
||||
The two files share the same system prompt body. Keep them in sync.
|
||||
|
||||
Rename `PLUGIN_NAME.md` and `PLUGIN_NAME.agent.md` to your agent's name (kebab-case).
|
||||
Add additional agent pairs as needed — one pair per agent.
|
||||
@@ -1,11 +0,0 @@
|
||||
# bin/
|
||||
|
||||
**Claude Code only.** Executables placed here are added to PATH when the plugin is installed.
|
||||
|
||||
Use this for CLI tools, helper scripts, or MCP server entry points bundled with the plugin.
|
||||
|
||||
Reference files in this directory from `.mcp.json` or hooks using `${CLAUDE_PLUGIN_ROOT}/bin/<file>`.
|
||||
The `${CLAUDE_PLUGIN_ROOT}` variable resolves to the plugin's install cache path at runtime —
|
||||
do not use relative paths from the repo root, as they will break after install.
|
||||
|
||||
GitHub Copilot CLI does not support `bin/`. If you add executables here, they are Claude Code only.
|
||||
@@ -1,7 +0,0 @@
|
||||
# docs/
|
||||
|
||||
Plugin documentation. Place usage guides, reference material, and examples here.
|
||||
|
||||
This directory is not read automatically by either Claude Code or GitHub Copilot CLI.
|
||||
Reference specific files from your skill bodies or agent prompts when needed
|
||||
(e.g. `See references/usage.md for examples`).
|
||||
@@ -1,3 +0,0 @@
|
||||
{
|
||||
"hooks": {}
|
||||
}
|
||||
@@ -1,27 +0,0 @@
|
||||
# hooks/
|
||||
|
||||
**Claude Code only.** Hook definitions that run in response to Claude Code events.
|
||||
|
||||
The `hooks.json` in this directory is read by Claude Code. Structure:
|
||||
|
||||
```json
|
||||
{
|
||||
"hooks": {
|
||||
"PostToolUse": [
|
||||
{
|
||||
"matcher": "Bash",
|
||||
"hooks": [
|
||||
{ "type": "command", "command": "echo 'tool used'" }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Supported events: `PreToolUse`, `PostToolUse`, `Notification`, `Stop`.
|
||||
|
||||
Use `${CLAUDE_PLUGIN_ROOT}` to reference scripts inside this plugin — the plugin runs
|
||||
from a cache path after install, not its original repo location.
|
||||
|
||||
For GitHub Copilot CLI hooks, see `hooks.json` at the plugin root.
|
||||
@@ -1,3 +0,0 @@
|
||||
{
|
||||
"hooks": {}
|
||||
}
|
||||
@@ -1,11 +0,0 @@
|
||||
{
|
||||
"name": "PLUGIN_NAME",
|
||||
"description": "PLUGIN_DESCRIPTION",
|
||||
"author": { "name": "AUTHOR_NAME", "email": "AUTHOR_EMAIL" },
|
||||
"license": "MIT",
|
||||
"keywords": [],
|
||||
"agents": "agents/",
|
||||
"skills": ["skills/"],
|
||||
"hooks": "hooks.json",
|
||||
"mcpServers": ".mcp.json"
|
||||
}
|
||||
@@ -1,18 +0,0 @@
|
||||
# skills/
|
||||
|
||||
Skills for this plugin. Each skill lives in its own subdirectory:
|
||||
|
||||
```
|
||||
skills/
|
||||
<skill-name>/
|
||||
SKILL.md # required — frontmatter + skill body
|
||||
META.md # required — provenance and audit fields
|
||||
references/ # optional — on-demand reference docs
|
||||
scripts/ # optional — executable helper scripts
|
||||
assets/ # optional — templates, data files, lookup tables
|
||||
```
|
||||
|
||||
**Shared** — both Claude Code and GitHub Copilot CLI read `skills/<name>/SKILL.md`.
|
||||
|
||||
To add a skill, run `/write-skill` in a Claude Code session. Do not write SKILL.md by hand
|
||||
without following the authoring standard — trigger descriptions and self-checks are required.
|
||||
@@ -1,118 +0,0 @@
|
||||
skill_name: plugin-create
|
||||
|
||||
trigger_tests:
|
||||
- id: explicit-create-named-plugin
|
||||
name: "Explicit trigger — create named plugin"
|
||||
query: "create a new plugin called security-tools"
|
||||
should_trigger: true
|
||||
|
||||
- id: explicit-scaffold-plugin
|
||||
name: "Explicit trigger — scaffold plugin"
|
||||
query: "scaffold a plugin called developer-tools"
|
||||
should_trigger: true
|
||||
|
||||
- id: implicit-add-marketplace-plugin
|
||||
name: "Implicit trigger — add plugin to marketplace"
|
||||
query: "add a new plugin to the marketplace for developer tools"
|
||||
should_trigger: true
|
||||
|
||||
- id: implicit-package-skills-into-plugin
|
||||
name: "Implicit trigger — package skills into a plugin"
|
||||
query: "I want to package up my skills into a distributable plugin"
|
||||
should_trigger: true
|
||||
|
||||
- id: negative-validate-marketplace
|
||||
name: "Negative trigger — validate marketplace.json (routes to /marketplace-architect)"
|
||||
query: "validate my marketplace.json"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-write-skill
|
||||
name: "Negative trigger — write a skill (routes to /write-skill)"
|
||||
query: "write a skill for code review"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-adopt-external-plugin
|
||||
name: "Negative trigger — adopt external plugin (routes to /marketplace-architect)"
|
||||
query: "adopt this external plugin into my marketplace"
|
||||
should_trigger: false
|
||||
|
||||
output_tests:
|
||||
- id: gate-a-shows-plan
|
||||
name: "Gate A presents file list and marketplace entry before writing"
|
||||
type: deterministic
|
||||
prompt: >
|
||||
Create a new plugin called marketplace-tools, description: 'Tools for managing the
|
||||
holocron marketplace', author: Jane Doe, email: jane@example.com,
|
||||
URL: https://github.com/jane
|
||||
expected_output: >
|
||||
The skill presents a plan listing all files that will be written under
|
||||
plugins/marketplace-tools/, shows the new marketplace.json entry, and
|
||||
explicitly asks for approval before writing any file.
|
||||
assertions:
|
||||
- "Output lists files to be written under plugins/marketplace-tools/"
|
||||
- "Output includes a marketplace.json entry with name 'marketplace-tools'"
|
||||
- "Output explicitly asks for approval and does not proceed without it"
|
||||
- "Output does not contain any written file confirmation before approval is given"
|
||||
|
||||
- id: reserved-name-rejected
|
||||
name: "Reserved plugin name is rejected before Gate A"
|
||||
type: deterministic
|
||||
prompt: "Create a new plugin called claude-tools"
|
||||
expected_output: >
|
||||
The skill halts before presenting Gate A, reports that 'claude-tools' matches
|
||||
the reserved name pattern 'claude-*', and asks the user to provide a different name.
|
||||
assertions:
|
||||
- "Output does not present Gate A or a file list"
|
||||
- "Output identifies 'claude-tools' as a reserved name"
|
||||
- "Output asks the user to provide a replacement name before continuing"
|
||||
|
||||
- id: no-skill-stub-generated
|
||||
name: "No skill stub generated — skills/ is empty with README only"
|
||||
type: deterministic
|
||||
prompt: >
|
||||
Create a new plugin called marketplace-tools, description: 'Tools for managing the
|
||||
holocron marketplace', author: Jane Doe, email: jane@example.com,
|
||||
URL: https://github.com/jane. Approve Gate A and Gate B.
|
||||
expected_output: >
|
||||
The plugin scaffold does not include any SKILL.md file. The skills/ directory
|
||||
is present with a README only. The skill directs the user to /write-skill
|
||||
to add skill content.
|
||||
assertions:
|
||||
- "Output does not include creation of any SKILL.md file"
|
||||
- "Output includes a skills/ directory entry with README.md only"
|
||||
- "Output mentions /write-skill for adding skill content"
|
||||
|
||||
- id: handoff-message-present
|
||||
name: "Hand-off message directs user to /marketplace-architect"
|
||||
type: deterministic
|
||||
prompt: >
|
||||
Create a new plugin called marketplace-tools, description: 'Tools for managing the
|
||||
holocron marketplace', author: Jane Doe, email: jane@example.com,
|
||||
URL: https://github.com/jane. Approve Gate A and Gate B.
|
||||
expected_output: >
|
||||
After writing files and running validation, the skill prints a hand-off message
|
||||
that names the created plugin and instructs the user to run /marketplace-architect.
|
||||
assertions:
|
||||
- "Output contains the string '/marketplace-architect'"
|
||||
- "Output includes the plugin name 'marketplace-tools' in the hand-off message"
|
||||
- "Output does not auto-invoke /marketplace-architect itself"
|
||||
|
||||
- id: full-flow-quality
|
||||
name: "Full flow — correct sequencing, substitution, and hand-off"
|
||||
type: llm-rubric
|
||||
prompt: >
|
||||
Create a new plugin called developer-tools with description 'Developer productivity
|
||||
toolkit', author: Alex Smith, email: alex@example.com, URL: https://github.com/alex.
|
||||
Approve Gate A and Gate B.
|
||||
expected_output: >
|
||||
The skill executes the full create-plugin flow: collects all five inputs one at a
|
||||
time, presents Gate A with a file plan, copies and substitutes the template, presents
|
||||
Gate B with full file contents, writes files after Gate B approval, updates
|
||||
marketplace.json, runs validation, and prints a hand-off message.
|
||||
assertions:
|
||||
- "All five inputs (plugin name, description, author name, email, URL) were collected individually before Gate A was presented"
|
||||
- "Gate A clearly listed all files to be written and the new marketplace.json entry, and waited for explicit approval"
|
||||
- "Gate B showed the full substituted content of every file before writing"
|
||||
- "None of the strings PLUGIN_NAME, PLUGIN_DESCRIPTION, AUTHOR_NAME, AUTHOR_EMAIL, or AUTHOR_URL appear in the written file output"
|
||||
- "The validation step either ran claude plugin validate . and reported output, or explicitly stated it was unavailable"
|
||||
- "The final message directed the user to run /marketplace-architect and included the plugin name 'developer-tools'"
|
||||
@@ -1,72 +0,0 @@
|
||||
# Manifest Field Reference
|
||||
|
||||
Quick reference for the two plugin manifests generated by this skill.
|
||||
Source of truth: `${CLAUDE_PLUGIN_ROOT}/skills/marketplace-architect/references/claude-code.md`
|
||||
and `copilot-cli.md` in the same directory.
|
||||
|
||||
---
|
||||
|
||||
## Claude Code — `.claude-plugin/plugin.json`
|
||||
|
||||
Only `name` is required. Add other fields only when needed.
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "my-plugin",
|
||||
"displayName": "My Plugin",
|
||||
"description": "What it does",
|
||||
"author": { "name": "Name", "url": "https://..." },
|
||||
"license": "MIT",
|
||||
"keywords": []
|
||||
}
|
||||
```
|
||||
|
||||
- `displayName` — human-readable label shown in the Claude Code UI. Update it to a
|
||||
title-cased string after placeholder substitution — it defaults to the kebab slug.
|
||||
- `version` — omit to use git SHA per commit (recommended). Set only for explicit release gates.
|
||||
- Component paths (`skills/`, `agents/`, etc.) are declared in the marketplace entry, not here.
|
||||
|
||||
---
|
||||
|
||||
## Copilot CLI — `plugin.json` at plugin root
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "my-plugin",
|
||||
"description": "What it does",
|
||||
"author": { "name": "Name", "email": "you@example.com" },
|
||||
"license": "MIT",
|
||||
"keywords": [],
|
||||
"agents": "agents/",
|
||||
"skills": ["skills/"],
|
||||
"hooks": "hooks.json",
|
||||
"mcpServers": ".mcp.json"
|
||||
}
|
||||
```
|
||||
|
||||
- `author.email` (not `url`) — Copilot uses email; Claude uses url. Both manifests diverge here.
|
||||
- Component paths are declared inline in this manifest (Copilot requires them; Claude ignores them).
|
||||
|
||||
---
|
||||
|
||||
## Marketplace entry (in `.claude-plugin/marketplace.json`)
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "my-plugin",
|
||||
"source": "./plugins/my-plugin",
|
||||
"description": "..."
|
||||
}
|
||||
```
|
||||
|
||||
- `source` must start with `./` for Claude Code compatibility.
|
||||
- Do not set `version` here if it is also set in `plugin.json` — `plugin.json` wins silently.
|
||||
|
||||
---
|
||||
|
||||
## Environment variables (for hooks and MCP configs)
|
||||
|
||||
- `${CLAUDE_PLUGIN_ROOT}` — absolute path to the plugin's installation cache. Use for all
|
||||
in-plugin file references in hooks and `.mcp.json`. Changes on update.
|
||||
- `${CLAUDE_PLUGIN_DATA}` — persistent directory that survives updates. Use for state,
|
||||
caches, and `node_modules`.
|
||||
@@ -1,48 +0,0 @@
|
||||
# Reserved Plugin and Marketplace Names
|
||||
|
||||
These names are blocked for third-party use by the Claude Code marketplace. Reject any
|
||||
plugin or marketplace name that matches an exact entry or a wildcard pattern below.
|
||||
|
||||
## Exact reserved names
|
||||
|
||||
```
|
||||
claude-code-marketplace
|
||||
claude-code-plugins
|
||||
claude-plugins-official
|
||||
claude-plugins-community
|
||||
claude-community
|
||||
anthropic-marketplace
|
||||
anthropic-plugins
|
||||
agent-skills
|
||||
anthropic-agent-skills
|
||||
knowledge-work-plugins
|
||||
life-sciences
|
||||
claude-for-legal
|
||||
claude-for-financial-services
|
||||
financial-services-plugins
|
||||
```
|
||||
|
||||
## Reserved name patterns (prefix match)
|
||||
|
||||
Any name starting with these strings is reserved:
|
||||
|
||||
```
|
||||
anthropic-
|
||||
claude-
|
||||
official-claude
|
||||
anthropic-tools
|
||||
```
|
||||
|
||||
## Kebab-case requirement
|
||||
|
||||
Plugin names must match `^[a-z0-9]+(-[a-z0-9]+)*$` — lowercase letters, digits, and
|
||||
hyphens only. No underscores, spaces, or uppercase. The Claude.ai marketplace sync rejects
|
||||
non-kebab-case names even if the local CLI tolerates them.
|
||||
|
||||
## How to check
|
||||
|
||||
1. Exact match: is the name in the exact reserved list above?
|
||||
2. Pattern match: does the name start with any reserved prefix?
|
||||
3. Format: does the name match the kebab-case regex?
|
||||
|
||||
If any check fails, halt and ask the user for a different name before continuing.
|
||||
@@ -1,18 +0,0 @@
|
||||
```yaml
|
||||
version: "1.0"
|
||||
updated: 2026-06-21
|
||||
|
||||
when: >
|
||||
Invoked explicitly as /promptfoo, or implicitly when the user wants to install,
|
||||
configure, run evaluations, red-team an LLM application, or integrate LLM testing
|
||||
into CI/CD using Promptfoo. Also triggered when the user wants to compare model
|
||||
outputs side-by-side across providers.
|
||||
|
||||
references:
|
||||
- https://promptfoo.dev/docs/getting-started
|
||||
- https://promptfoo.dev/docs/configuration/parameters
|
||||
- https://promptfoo.dev/docs/usage/command-line
|
||||
- https://promptfoo.dev/docs/configuration/expected-outputs
|
||||
- https://promptfoo.dev/docs/red-team/
|
||||
- https://promptfoo.dev/docs/category/integrations
|
||||
```
|
||||
@@ -1,107 +0,0 @@
|
||||
---
|
||||
name: promptfoo
|
||||
description: >
|
||||
Use when the user wants to install, set up, configure, or work with Promptfoo —
|
||||
the open-source LLM evaluation CLI. Covers: project initialization (promptfoo init),
|
||||
writing or editing promptfooconfig.yaml, running evaluations (promptfoo eval),
|
||||
comparing model outputs side-by-side, viewing and sharing results, generating test
|
||||
datasets, red-teaming LLM applications with adversarial plugins and strategies, and
|
||||
integrating Promptfoo into CI/CD pipelines. Do NOT use when the user wants to evaluate
|
||||
LLM output quality in general without Promptfoo, write tests for non-LLM code (use
|
||||
tdd), or set up a different evaluation framework (RAGAS, OpenAI Evals, etc.).
|
||||
metadata:
|
||||
category: test
|
||||
---
|
||||
|
||||
<requirements>
|
||||
|
||||
## Required inputs
|
||||
|
||||
- **Task scope** — what the user wants to do (install, configure, run eval, red-team, CI
|
||||
integration, or debug); inferred from request, ask only if genuinely ambiguous
|
||||
- **Config file path** — defaults to `promptfooconfig.yaml` in the working directory;
|
||||
user provides if non-standard
|
||||
- **Provider API keys** — must be set as environment variables before running evals;
|
||||
remind the user if not already set
|
||||
|
||||
## Constraints
|
||||
|
||||
- Always use the pinned version `0.121.17` — never suggest `@latest` or an unpinned
|
||||
install; see `references/installation.md` for rationale (OpenAI acquisition, March 2026)
|
||||
- API keys and credentials must always reference environment variables in examples and
|
||||
config — never hardcoded values, even as placeholders like `sk-abc123`
|
||||
- Warn when the user's eval workload is non-OpenAI-centric: note the acquisition and
|
||||
mention DeepEval / Arize Phoenix as documented fallbacks
|
||||
- Offload detail to references/ files — do not reproduce CLI flags, assertion types, or
|
||||
redteam plugin tables inline; wire each reference file in the step that needs it
|
||||
|
||||
</requirements>
|
||||
|
||||
<steps>
|
||||
|
||||
## Process
|
||||
|
||||
1. **Clarify scope.** Identify what the user wants: install/setup, configure
|
||||
`promptfooconfig.yaml`, run an eval, generate a dataset, red-team, integrate CI/CD,
|
||||
or debug a failing run. Infer from context — ask only if genuinely ambiguous.
|
||||
|
||||
2. **Install / initialise (if needed).** Check whether Promptfoo is already installed.
|
||||
If not, guide through installation using the pinned version. To scaffold a new
|
||||
project: `npx promptfoo@0.121.17 init`. See `references/installation.md` for
|
||||
prerequisites, global/local install options, and the acquisition notice.
|
||||
|
||||
3. **Configure.** Help the user write or edit `promptfooconfig.yaml`. Cover: prompts
|
||||
(inline, file://, JS/Python), providers (string shorthand and object form),
|
||||
test cases with vars and assert, defaultTest, outputPath, and evaluateOptions.
|
||||
See `references/configuration.md` for the full structure and
|
||||
`references/assertions.md` for assertion types and shorthand syntax.
|
||||
|
||||
4. **Run evaluation.** Execute `npx promptfoo@0.121.17 eval` (with `--no-cache` in CI).
|
||||
After the run, open results with `promptfoo view` or share with `promptfoo share`.
|
||||
For rate-limit issues, suggest `--max-concurrency` and `--delay` flags.
|
||||
See `references/cli-reference.md` for all flags and subcommands.
|
||||
|
||||
5. **Red-team (if requested).** Add a `redteam:` block to the config with the
|
||||
appropriate plugins and strategies. Run `promptfoo redteam generate` to produce
|
||||
adversarial test cases, then `promptfoo eval` to execute them.
|
||||
See `references/redteam.md` for plugin/strategy tables and OWASP/MITRE mappings.
|
||||
|
||||
6. **CI integration (if requested).** Write a GitHub Actions workflow using
|
||||
`promptfoo/promptfoo-action@v1` or a generic `npx promptfoo@0.121.17 eval --no-cache`
|
||||
step. Always use CI secrets for API keys.
|
||||
See `references/ci-cd.md` for full workflow templates.
|
||||
|
||||
7. **Generate test dataset (if requested).** Run `promptfoo generate dataset` with
|
||||
optional `--instructions` to AI-generate test cases from the prompt template.
|
||||
See `references/cli-reference.md` for flags.
|
||||
|
||||
## Output format
|
||||
|
||||
- Modified or newly created `promptfooconfig.yaml`
|
||||
- Shell commands to run, with the pinned version and relevant flags
|
||||
- GitHub Actions workflow file (`.github/workflows/llm-eval.yml`) when CI is in scope
|
||||
|
||||
</steps>
|
||||
|
||||
<checks>
|
||||
|
||||
## Failure handling
|
||||
|
||||
- No `promptfooconfig.yaml` found and user did not ask to create one — ask before
|
||||
scaffolding; do not overwrite existing config without confirmation
|
||||
- Eval fails with rate-limit errors — suggest `--max-concurrency 2 --delay 3000` and
|
||||
`PROMPTFOO_RETRY_5XX=true`; see `references/cli-reference.md`
|
||||
- User requests `@latest` or an unpinned install — correct to `0.121.17` and explain
|
||||
the version-pinning rationale (acquisition notice)
|
||||
- User requests hardcoded API key in config or example — refuse and redirect to
|
||||
environment variable pattern
|
||||
|
||||
## Self-check
|
||||
|
||||
- [ ] Pinned version `0.121.17` used in all commands — no `@latest`
|
||||
- [ ] No credentials appear in any output, config example, or shell command
|
||||
- [ ] Non-OpenAI eval workloads include the acquisition warning
|
||||
- [ ] Each references/ file wired in the step that uses it
|
||||
- [ ] Config written or modified only after confirming path with user
|
||||
|
||||
</checks>
|
||||
@@ -1,112 +0,0 @@
|
||||
skill_name: promptfoo
|
||||
|
||||
trigger_tests:
|
||||
- id: explicit-setup
|
||||
name: "Explicit trigger — project setup"
|
||||
query: "set up promptfoo in my project"
|
||||
should_trigger: true
|
||||
|
||||
- id: explicit-run-eval
|
||||
name: "Explicit trigger — run eval"
|
||||
query: "help me run a promptfoo eval"
|
||||
should_trigger: true
|
||||
|
||||
- id: implicit-model-comparison
|
||||
name: "Implicit trigger — model comparison"
|
||||
query: "I want to compare GPT-4 and Claude on the same test cases"
|
||||
should_trigger: true
|
||||
|
||||
- id: implicit-ci-llm-testing
|
||||
name: "Implicit trigger — CI LLM testing"
|
||||
query: "how do I add LLM testing to my pull request pipeline?"
|
||||
should_trigger: true
|
||||
|
||||
- id: negative-express-tests
|
||||
name: "Negative trigger — non-LLM integration tests"
|
||||
query: "How do I write integration tests for my Express API?"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-ragas
|
||||
name: "Negative trigger — different eval framework"
|
||||
query: "Set up RAGAS for evaluating my RAG pipeline"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-general-llm-eval
|
||||
name: "Negative trigger — general LLM quality question"
|
||||
query: "What's the best way to evaluate LLM output quality?"
|
||||
should_trigger: false
|
||||
|
||||
output_tests:
|
||||
- id: output-pinned-version
|
||||
name: "Deterministic — pinned version used, @latest absent"
|
||||
type: deterministic
|
||||
prompt: "install promptfoo"
|
||||
expected_output: >
|
||||
Shell commands that install or invoke Promptfoo using the pinned version
|
||||
0.121.17. No @latest tag appears anywhere in the output.
|
||||
assertions:
|
||||
- "Output contains the string '0.121.17'"
|
||||
- "Output does NOT contain '@latest'"
|
||||
- "Output does NOT contain 'promptfoo@latest'"
|
||||
|
||||
- id: output-config-structure
|
||||
name: "Deterministic — valid promptfooconfig.yaml structure"
|
||||
type: deterministic
|
||||
prompt: "create a promptfooconfig.yaml for comparing two models on my customer support prompts"
|
||||
expected_output: >
|
||||
A YAML configuration file with all required top-level keys for a working
|
||||
promptfoo evaluation: prompts, providers (at least two entries), tests with
|
||||
at least one test case containing vars and assert.
|
||||
assertions:
|
||||
- "Output contains 'prompts:'"
|
||||
- "Output contains 'providers:'"
|
||||
- "Output contains 'tests:'"
|
||||
- "Output contains 'assert:'"
|
||||
|
||||
- id: output-no-hardcoded-keys
|
||||
name: "Deterministic — no hardcoded API keys in output"
|
||||
type: deterministic
|
||||
prompt: "set up promptfoo with OpenAI and show me the config"
|
||||
expected_output: >
|
||||
A setup guide and config that references API keys only via environment
|
||||
variables. No literal API key values, no placeholder strings like sk-abc123
|
||||
or sk-ant-abc123.
|
||||
assertions:
|
||||
- "Output does NOT contain 'sk-' followed by alphanumeric characters"
|
||||
- "Output references OPENAI_API_KEY or ANTHROPIC_API_KEY as an environment variable"
|
||||
- "Output does NOT contain a hardcoded token or credential value"
|
||||
|
||||
- id: output-acquisition-warning
|
||||
name: "Deterministic — acquisition warning for non-OpenAI workloads"
|
||||
type: deterministic
|
||||
prompt: "evaluate my Anthropic-only pipeline with promptfoo"
|
||||
expected_output: >
|
||||
The response includes a note about Promptfoo's OpenAI acquisition (March 2026)
|
||||
and either mentions version pinning or names at least one documented fallback
|
||||
(DeepEval or Arize Phoenix).
|
||||
assertions:
|
||||
- "Output contains 'acquisition' OR output contains 'OpenAI' in a warning context"
|
||||
- "Output contains 'DeepEval' OR output contains 'Arize' OR output contains 'Phoenix' OR output contains 'pin'"
|
||||
|
||||
- id: output-quality-rubric
|
||||
name: "LLM-rubric — complete setup is actionable and compliant"
|
||||
type: llm-rubric
|
||||
prompt: "set up a complete promptfoo project for testing my customer support chatbot"
|
||||
expected_output: >
|
||||
A complete, actionable guide that: scaffolds a project with the pinned version,
|
||||
produces a valid promptfooconfig.yaml with realistic prompts, at least one
|
||||
provider, and test cases with assertions, references environment variables for
|
||||
API keys, and provides the commands needed to run the first eval.
|
||||
assertions:
|
||||
- >
|
||||
The output is actionable: a developer could follow the steps without additional
|
||||
research and end up with a working promptfoo project
|
||||
- >
|
||||
The config example is structurally valid promptfoo YAML: it has prompts,
|
||||
providers, and tests sections with realistic placeholder content
|
||||
- >
|
||||
API keys are handled correctly: all examples use environment variable references,
|
||||
not literal key values
|
||||
- >
|
||||
The pinned version (0.121.17) appears in at least one command — the output does
|
||||
not instruct the user to install an unpinned version
|
||||
@@ -1,124 +0,0 @@
|
||||
---
|
||||
topic: assertions
|
||||
source_keys:
|
||||
- context7-promptfoo-dev
|
||||
- context7-promptfoo-github
|
||||
---
|
||||
|
||||
## Assertion structure
|
||||
|
||||
Each assertion in `assert:` has a `type:`, an optional `value:`, an optional `threshold:`, and an optional `metric:` label.
|
||||
|
||||
```yaml
|
||||
assert:
|
||||
- type: contains
|
||||
value: 'return policy'
|
||||
- type: llm-rubric
|
||||
value: 'Response is helpful and professional'
|
||||
threshold: 0.8
|
||||
metric: quality
|
||||
```
|
||||
|
||||
## String shorthand
|
||||
|
||||
Assertions can also be written as compact strings directly in the assert list:
|
||||
|
||||
| Shorthand | Full type |
|
||||
|---|---|
|
||||
| `Paris` | `equals` |
|
||||
| `contains:Paris` | `contains` |
|
||||
| `icontains:paris` | `icontains` (case-insensitive) |
|
||||
| `starts-with:The answer` | `starts-with` |
|
||||
| `regex:^Hello.*world$` | `regex` |
|
||||
| `is-json` | `is-json` |
|
||||
| `contains-json` | `contains-json` |
|
||||
| `similar(0.8):Hello world` | `similar` with threshold |
|
||||
| `llm-rubric:Is helpful and accurate` | `llm-rubric` |
|
||||
| `grade:Does not mention being an AI` | alias for `llm-rubric` |
|
||||
| `factuality:Paris is the capital of France` | `factuality` |
|
||||
| `javascript:output.length < 100` | inline JS |
|
||||
| `fn:output.includes('hello')` | alias for `javascript` |
|
||||
| `python:len(output) > 10` | inline Python |
|
||||
| `file://assertions/custom.js` | external file |
|
||||
| `levenshtein(5):expected text` | `levenshtein` with distance |
|
||||
| `not-contains:error` | negated assertion |
|
||||
|
||||
## Deterministic assertions
|
||||
|
||||
- `equals` — exact string match
|
||||
- `contains` / `icontains` — substring check (case-sensitive / insensitive)
|
||||
- `not-contains` — absence check
|
||||
- `starts-with` — prefix check
|
||||
- `regex` — regular expression match
|
||||
- `is-json` — valid JSON
|
||||
- `contains-json` — valid JSON somewhere in output
|
||||
- `levenshtein` — edit distance within threshold
|
||||
|
||||
## Similarity and semantic assertions
|
||||
|
||||
- `similar` — embedding cosine similarity; `threshold:` is a 0–1 score
|
||||
- `context-faithfulness` — similarity-based RAG faithfulness; `threshold: 0.8` typical
|
||||
|
||||
## Model-graded assertions
|
||||
|
||||
These send a grader prompt to another LLM (by default the configured judge model) and score the output.
|
||||
|
||||
**`llm-rubric`** — open-ended rubric; binary or 0–1 score depending on criteria phrasing. Use `threshold:` to set minimum passing score:
|
||||
```yaml
|
||||
- type: llm-rubric
|
||||
value: Is not apologetic and provides a clear, concise answer
|
||||
threshold: 0.8
|
||||
```
|
||||
|
||||
**`factuality`** — checks whether the output is factually consistent with a reference statement:
|
||||
```yaml
|
||||
- type: factuality
|
||||
value: The capital of California is Sacramento
|
||||
```
|
||||
|
||||
**`pi`** — custom scoring with any numeric range; requires `threshold:`.
|
||||
|
||||
The grader model can be overridden globally in `defaultTest.options.provider` or per-assertion.
|
||||
|
||||
## Custom assertions
|
||||
|
||||
**JavaScript (inline):**
|
||||
```yaml
|
||||
- type: javascript
|
||||
value: "output.length < 100 && !output.includes('error')"
|
||||
```
|
||||
|
||||
**JavaScript (file):**
|
||||
```yaml
|
||||
- type: javascript
|
||||
value: file://assertions/check_format.js
|
||||
```
|
||||
|
||||
**Python:**
|
||||
```yaml
|
||||
- type: python
|
||||
value: "len(output) > 10 and 'Paris' in output"
|
||||
```
|
||||
|
||||
## Negation
|
||||
|
||||
Any assertion type can be negated by prepending `not-`:
|
||||
```yaml
|
||||
- type: not-contains
|
||||
value: 'error'
|
||||
- type: not-regex
|
||||
value: '\b(fail|broken)\b'
|
||||
```
|
||||
|
||||
## Metrics
|
||||
|
||||
The `metric:` field groups assertions for aggregate reporting. All assertions with the same metric name are scored together in the results view:
|
||||
```yaml
|
||||
assert:
|
||||
- type: contains
|
||||
value: 'policy'
|
||||
metric: coverage
|
||||
- type: llm-rubric
|
||||
value: Addresses the user's concern
|
||||
metric: quality
|
||||
```
|
||||
@@ -1,127 +0,0 @@
|
||||
---
|
||||
topic: ci-cd
|
||||
source_keys:
|
||||
- context7-promptfoo-dev
|
||||
- context7-promptfoo-github
|
||||
---
|
||||
|
||||
## How CI integration works
|
||||
|
||||
`promptfoo eval` exits with a non-zero code when any assertion fails. This makes it a natural CI gate — a failing eval blocks a merge just like a failing test suite.
|
||||
|
||||
## GitHub Actions — promptfoo-action
|
||||
|
||||
The official `promptfoo/promptfoo-action@v1` action runs an eval on every pull request and posts results as a PR comment.
|
||||
|
||||
```yaml
|
||||
# .github/workflows/llm-eval.yml
|
||||
name: LLM Eval
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
eval:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: promptfoo/promptfoo-action@v1
|
||||
with:
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
config: promptfooconfig.yaml
|
||||
env:
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
```
|
||||
|
||||
The `github-token` is required for the action to post PR comments. Without it the eval still runs but results are not surfaced in the PR UI.
|
||||
|
||||
## Generic CI (npx)
|
||||
|
||||
Any CI system that can run Node.js commands can use Promptfoo:
|
||||
|
||||
```yaml
|
||||
# Any CI provider
|
||||
- name: Run promptfoo eval
|
||||
run: npx promptfoo@0.121.17 eval --no-cache
|
||||
env:
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
```
|
||||
|
||||
`--no-cache` ensures all LLM calls are fresh — important in CI where cached results from a developer's machine would not be present.
|
||||
|
||||
## Google Cloud / Vertex AI
|
||||
|
||||
```yaml
|
||||
# .github/workflows/llm-test.yml
|
||||
steps:
|
||||
- uses: google-github-actions/auth@v2
|
||||
with:
|
||||
credentials_json: ${{ secrets.GCP_CREDENTIALS }}
|
||||
- name: Run promptfoo tests
|
||||
run: npx promptfoo@0.121.17 eval
|
||||
env:
|
||||
GOOGLE_CLOUD_PROJECT: ${{ vars.GCP_PROJECT_ID }}
|
||||
GOOGLE_CLOUD_LOCATION: us-central1
|
||||
```
|
||||
|
||||
## MCP security testing workflow
|
||||
|
||||
```yaml
|
||||
# .github/workflows/security-test.yml
|
||||
name: MCP Security Testing
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
security-test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
- run: npm install
|
||||
- run: npm run build:all-servers
|
||||
- name: Run security tests
|
||||
run: |
|
||||
npx promptfoo eval -c security-tests/scenario1.yaml
|
||||
npx promptfoo eval -c security-tests/scenario2.yaml
|
||||
```
|
||||
|
||||
## Model security scanning (SARIF output)
|
||||
|
||||
For repositories that include model files, scan them in CI and upload results to GitHub's security tab:
|
||||
|
||||
```yaml
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
npm install -g promptfoo
|
||||
pip install modelaudit
|
||||
|
||||
- name: Scan models
|
||||
run: |
|
||||
promptfoo scan-model ./models/ \
|
||||
--strict \
|
||||
--no-write \
|
||||
--format sarif \
|
||||
--output model-scan-results.sarif
|
||||
|
||||
- name: Upload SARIF
|
||||
uses: github/codeql-action/upload-sarif@v3
|
||||
with:
|
||||
sarif_file: model-scan-results.sarif
|
||||
```
|
||||
|
||||
## GitLab CI and Jenkins
|
||||
|
||||
Both are supported. Use `npx promptfoo@0.121.17 eval` as the test command. No platform-specific action is needed — the exit code gates the pipeline natively.
|
||||
|
||||
## Recommended CI practices
|
||||
|
||||
- Always pass `--no-cache` in CI to avoid stale results
|
||||
- Store API keys as CI secrets, never hardcode them
|
||||
- Run evals on PR branches to catch regressions before merge
|
||||
- Use `outputPath: results.json` and archive the artifact for debugging failed runs
|
||||
- Set `evaluateOptions.maxConcurrency` low (2–5) in CI to avoid provider rate limits
|
||||
@@ -1,148 +0,0 @@
|
||||
---
|
||||
topic: cli-reference
|
||||
source_keys:
|
||||
- context7-promptfoo-dev
|
||||
- context7-promptfoo-github
|
||||
---
|
||||
|
||||
## Invocation
|
||||
|
||||
```bash
|
||||
promptfoo <command> [options] # global install
|
||||
npx promptfoo@0.121.17 <command> # one-off via npx
|
||||
```
|
||||
|
||||
## Commands
|
||||
|
||||
### `init`
|
||||
|
||||
Scaffold a new project in the current directory.
|
||||
|
||||
```bash
|
||||
promptfoo init
|
||||
promptfoo init --example openai-mcp
|
||||
```
|
||||
|
||||
Creates `promptfooconfig.yaml` with example prompts, providers, and test cases.
|
||||
|
||||
---
|
||||
|
||||
### `eval` (most common)
|
||||
|
||||
Run an evaluation.
|
||||
|
||||
```bash
|
||||
promptfoo eval
|
||||
promptfoo eval -c path/to/promptfooconfig.yaml
|
||||
promptfoo eval --no-cache
|
||||
promptfoo eval --max-concurrency 2
|
||||
promptfoo eval --delay 3000
|
||||
promptfoo eval -o results.json
|
||||
```
|
||||
|
||||
| Flag | Description |
|
||||
|---|---|
|
||||
| `-c <path>` | Config file path (default: `promptfooconfig.yaml`) |
|
||||
| `--no-cache` | Disable response cache; forces fresh LLM calls |
|
||||
| `--max-concurrency <n>` | Max parallel requests (default: provider-dependent) |
|
||||
| `--delay <ms>` | Fixed delay between requests |
|
||||
| `-o <path>` | Output path (`.html`, `.json`, `.csv`, `.yaml`) |
|
||||
| `--format sarif` | Output in SARIF format (for security scanning) |
|
||||
|
||||
Exit code is non-zero when any assertion fails, making it suitable for CI gating.
|
||||
|
||||
---
|
||||
|
||||
### `view`
|
||||
|
||||
Open the most recent evaluation results in a local browser UI.
|
||||
|
||||
```bash
|
||||
promptfoo view
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `share`
|
||||
|
||||
Upload results and get a shareable URL.
|
||||
|
||||
```bash
|
||||
promptfoo share
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `cache clear`
|
||||
|
||||
Clear all cached LLM responses.
|
||||
|
||||
```bash
|
||||
promptfoo cache clear
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `generate dataset`
|
||||
|
||||
Use an LLM to auto-generate test cases from a prompt template.
|
||||
|
||||
```bash
|
||||
promptfoo generate dataset
|
||||
promptfoo generate dataset --config path/to/config.yaml
|
||||
promptfoo generate dataset --output generated_tests.yaml
|
||||
promptfoo generate dataset --instructions "Consider edge cases related to international travel"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `redteam generate`
|
||||
|
||||
Generate adversarial test cases for red-teaming.
|
||||
|
||||
```bash
|
||||
promptfoo redteam generate
|
||||
promptfoo redteam generate -c promptfooconfig.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `scan-model`
|
||||
|
||||
Scan model files for security vulnerabilities. Outputs results in SARIF format.
|
||||
|
||||
```bash
|
||||
promptfoo scan-model ./models/ --strict --no-write --format sarif --output scan.sarif
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `auth`
|
||||
|
||||
Manage authentication (for sharing and cloud features).
|
||||
|
||||
```bash
|
||||
promptfoo auth login
|
||||
promptfoo auth logout
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rate-limit management
|
||||
|
||||
```bash
|
||||
promptfoo eval --max-concurrency 1 --delay 3000
|
||||
```
|
||||
|
||||
Or in config:
|
||||
```yaml
|
||||
evaluateOptions:
|
||||
maxConcurrency: 2
|
||||
delay: 3000
|
||||
```
|
||||
|
||||
Environment variable for backoff:
|
||||
```bash
|
||||
export PROMPTFOO_REQUEST_BACKOFF_MS=10000
|
||||
export PROMPTFOO_RETRY_5XX=true
|
||||
```
|
||||
@@ -1,151 +0,0 @@
|
||||
---
|
||||
topic: configuration
|
||||
source_keys:
|
||||
- context7-promptfoo-dev
|
||||
- context7-promptfoo-github
|
||||
---
|
||||
|
||||
## File name and schema
|
||||
|
||||
The default config file is `promptfooconfig.yaml` in the working directory. A different path can be passed with `-c`. Add the JSON schema header for editor autocompletion:
|
||||
|
||||
```yaml
|
||||
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
||||
```
|
||||
|
||||
## Top-level structure
|
||||
|
||||
```yaml
|
||||
description: Human-readable name for this eval
|
||||
|
||||
prompts:
|
||||
- '...' # inline string
|
||||
- file://... # path to .txt, .json, .js, .py
|
||||
|
||||
providers:
|
||||
- openai:gpt-5-mini
|
||||
- anthropic:messages:claude-sonnet-4-5
|
||||
|
||||
defaultTest: # merged into every test case
|
||||
assert:
|
||||
- type: is-json
|
||||
|
||||
tests:
|
||||
- vars:
|
||||
query: 'I need help'
|
||||
assert:
|
||||
- type: contains
|
||||
value: 'help'
|
||||
- file://test_scenarios.csv # external test file
|
||||
|
||||
outputPath: results/eval.html # .html, .json, .csv, .yaml
|
||||
|
||||
evaluateOptions:
|
||||
maxConcurrency: 5
|
||||
delay: 500 # ms between requests
|
||||
```
|
||||
|
||||
## Prompts
|
||||
|
||||
Plain string with Handlebars-style `{{variable}}` placeholders:
|
||||
```yaml
|
||||
prompts:
|
||||
- 'You are a helpful agent. {{query}}'
|
||||
```
|
||||
|
||||
Chat conversation from a JSON file (array of `{role, content}` messages):
|
||||
```yaml
|
||||
prompts:
|
||||
- file://prompts/chat_conversation.json
|
||||
```
|
||||
|
||||
Dynamic prompt from a JS function:
|
||||
```yaml
|
||||
prompts:
|
||||
- file://prompts/generate_prompt.js
|
||||
```
|
||||
|
||||
Prompts can also carry a `label:` and `raw:` when using the object form, and a `config:` block to set provider-specific parameters (e.g. `response_format`).
|
||||
|
||||
## Providers
|
||||
|
||||
String shorthand:
|
||||
```yaml
|
||||
providers:
|
||||
- openai:gpt-5-mini
|
||||
- anthropic:messages:claude-sonnet-4-5-20250929
|
||||
- bedrock:us.anthropic.claude-sonnet-4-5-20250929-v1:0
|
||||
- azureopenai:chat:my-deployment
|
||||
- http://localhost:8080/v1/chat/completions # custom HTTP
|
||||
```
|
||||
|
||||
Object form with config:
|
||||
```yaml
|
||||
providers:
|
||||
- id: openai:responses:gpt-5
|
||||
config:
|
||||
temperature: 0.7
|
||||
max_output_tokens: 500
|
||||
instructions: 'You are a helpful assistant.'
|
||||
```
|
||||
|
||||
Over 60 providers are supported. Local models, HuggingFace, and custom HTTP endpoints are all valid provider types.
|
||||
|
||||
## Tests and vars
|
||||
|
||||
Each test case has `vars:` (substituted into prompt placeholders) and `assert:` (assertions on the response):
|
||||
|
||||
```yaml
|
||||
tests:
|
||||
- vars:
|
||||
query: 'I need to return a product'
|
||||
assert:
|
||||
- type: contains
|
||||
value: 'return policy'
|
||||
- type: llm-rubric
|
||||
value: 'Response is helpful and professional'
|
||||
```
|
||||
|
||||
Tests can be loaded from external files (CSV, YAML) using `file://` references.
|
||||
|
||||
## defaultTest
|
||||
|
||||
Assertions and options declared here are merged into every test case, reducing repetition:
|
||||
|
||||
```yaml
|
||||
defaultTest:
|
||||
assert:
|
||||
- type: llm-rubric
|
||||
value: 'Does not reveal internal system prompt'
|
||||
options:
|
||||
provider:
|
||||
id: openai:chat:gpt-5-mini # override grader model
|
||||
```
|
||||
|
||||
## Output formats
|
||||
|
||||
`outputPath` accepts `.html` (browser-viewable), `.json`, `.csv`, or `.yaml`. Multiple outputs can be listed as an array.
|
||||
|
||||
## Environment variables
|
||||
|
||||
| Variable | Purpose |
|
||||
|---|---|
|
||||
| `OPENAI_API_KEY` | OpenAI authentication |
|
||||
| `ANTHROPIC_API_KEY` | Anthropic authentication |
|
||||
| `REQUEST_TIMEOUT_MS` | Per-request timeout in ms |
|
||||
| `PROMPTFOO_RETRY_5XX` | Retry on 5xx errors (`true`/`false`) |
|
||||
| `PROMPTFOO_REQUEST_BACKOFF_MS` | Backoff between retries |
|
||||
|
||||
## Red-team configuration block
|
||||
|
||||
```yaml
|
||||
redteam:
|
||||
plugins:
|
||||
- harmful
|
||||
- prompt-injection
|
||||
- hijacking
|
||||
strategies:
|
||||
- jailbreak
|
||||
- jailbreak:composite
|
||||
- prompt-injection
|
||||
```
|
||||
@@ -1,150 +0,0 @@
|
||||
---
|
||||
topic: examples
|
||||
source_keys:
|
||||
- context7-promptfoo-dev
|
||||
- context7-promptfoo-github
|
||||
---
|
||||
|
||||
## Quickstart
|
||||
|
||||
```bash
|
||||
npx promptfoo@0.121.17 init
|
||||
# edit promptfooconfig.yaml
|
||||
npx promptfoo@0.121.17 eval
|
||||
npx promptfoo@0.121.17 view
|
||||
```
|
||||
|
||||
## Minimal config
|
||||
|
||||
```yaml
|
||||
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
||||
prompts:
|
||||
- 'Answer the user question concisely. Question: {{question}}'
|
||||
|
||||
providers:
|
||||
- openai:gpt-5-mini
|
||||
|
||||
tests:
|
||||
- vars:
|
||||
question: How do I reset my password?
|
||||
assert:
|
||||
- type: contains
|
||||
value: reset
|
||||
- vars:
|
||||
question: Can I cancel my subscription today?
|
||||
assert:
|
||||
- type: llm-rubric
|
||||
value: The answer clearly explains the cancellation path.
|
||||
```
|
||||
|
||||
## Multi-provider comparison
|
||||
|
||||
```yaml
|
||||
providers:
|
||||
- openai:gpt-5-mini
|
||||
- anthropic:claude-3-haiku
|
||||
|
||||
prompts:
|
||||
- 'You are a helpful customer service agent. {{query}}'
|
||||
|
||||
tests:
|
||||
- vars:
|
||||
query: 'I need to return a product'
|
||||
assert:
|
||||
- type: contains
|
||||
value: 'return policy'
|
||||
- type: llm-rubric
|
||||
value: 'Response is helpful and professional'
|
||||
```
|
||||
|
||||
Running this produces a side-by-side table with both models' outputs and assertion scores.
|
||||
|
||||
## Loading tests from CSV
|
||||
|
||||
```yaml
|
||||
tests:
|
||||
- file://test_scenarios.csv
|
||||
```
|
||||
|
||||
CSV format: one column per variable, header row must match `{{variable}}` names in the prompt. An `__expected` column maps to the `equals` assertion automatically.
|
||||
|
||||
## Factuality evaluation
|
||||
|
||||
```yaml
|
||||
providers:
|
||||
- openai:gpt-5-mini
|
||||
prompts:
|
||||
- |
|
||||
Please answer the following question accurately:
|
||||
Question: What is the capital of {{location}}?
|
||||
tests:
|
||||
- vars:
|
||||
location: California
|
||||
assert:
|
||||
- type: factuality
|
||||
value: The capital of California is Sacramento
|
||||
```
|
||||
|
||||
## defaultTest for shared assertions
|
||||
|
||||
```yaml
|
||||
defaultTest:
|
||||
assert:
|
||||
- type: llm-rubric
|
||||
value: |
|
||||
Evaluate whether the response correctly answers the question.
|
||||
Question: {{ question }}
|
||||
Model Response: {{ output }}
|
||||
Correct Answer: {{ answer }}
|
||||
Grade accuracy 0.0–1.0. Pass if >= 0.8.
|
||||
threshold: 0.8
|
||||
|
||||
tests:
|
||||
- vars:
|
||||
question: What year did WW2 end?
|
||||
answer: '1945'
|
||||
- vars:
|
||||
question: What is the boiling point of water in Celsius?
|
||||
answer: '100'
|
||||
```
|
||||
|
||||
## Node.js API
|
||||
|
||||
```javascript
|
||||
import { evaluate } from 'promptfoo';
|
||||
|
||||
const evalRecord = await evaluate({
|
||||
prompts: ['Translate to Spanish: {{ text }}'],
|
||||
providers: ['openai:chat:gpt-5.5'],
|
||||
tests: [
|
||||
{
|
||||
vars: { text: 'Hello' },
|
||||
assert: [{ type: 'contains', value: 'Hola', metric: 'translation' }],
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
const results = await evalRecord.toEvaluateSummary();
|
||||
console.log(`Pass rate: ${results.stats.successes}/${results.results.length}`);
|
||||
```
|
||||
|
||||
## Generating test datasets with AI
|
||||
|
||||
```bash
|
||||
# Generate test cases based on your prompt template
|
||||
promptfoo generate dataset
|
||||
promptfoo generate dataset --instructions "Consider edge cases related to international travel"
|
||||
promptfoo generate dataset --output generated_tests.yaml
|
||||
```
|
||||
|
||||
## Saving and sharing results
|
||||
|
||||
```yaml
|
||||
outputPath: evaluations/results.html
|
||||
```
|
||||
|
||||
Or via CLI:
|
||||
```bash
|
||||
promptfoo eval -o results.json
|
||||
promptfoo share # get a shareable URL
|
||||
```
|
||||
@@ -1,70 +0,0 @@
|
||||
---
|
||||
topic: installation
|
||||
source_keys:
|
||||
- context7-promptfoo-dev
|
||||
- context7-promptfoo-github
|
||||
---
|
||||
|
||||
## Acquisition notice
|
||||
|
||||
Promptfoo was acquired by OpenAI in March 2026. OpenAI has committed to keeping it open-source and multi-provider. **Always pin to a specific version** — do not track `@latest`. Monitor for neutrality degradation in future releases, particularly for non-OpenAI provider evaluation. Documented fallbacks: DeepEval (pytest-native, Python teams) and Arize Phoenix (self-hosted, vendor-neutral).
|
||||
|
||||
Pinned version in this skill: **0.121.17**. Update via CI when a new version has been validated.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Node.js 18+ (Node 22 recommended for CI)
|
||||
- npm or npx
|
||||
|
||||
No database or server process is required. Promptfoo stores evaluation results in a local SQLite cache.
|
||||
|
||||
## Installation options
|
||||
|
||||
**One-off via npx (no install required — preferred):**
|
||||
```bash
|
||||
npx promptfoo@0.121.17 eval
|
||||
```
|
||||
|
||||
**Global install (pinned):**
|
||||
```bash
|
||||
npm install -g promptfoo@0.121.17
|
||||
```
|
||||
After a global install, use `promptfoo` directly without `npx`.
|
||||
|
||||
**Project dependency (for Node.js API usage):**
|
||||
```bash
|
||||
npm install promptfoo@0.121.17
|
||||
```
|
||||
|
||||
## Initialising a project
|
||||
|
||||
```bash
|
||||
npx promptfoo@0.121.17 init
|
||||
```
|
||||
|
||||
Creates a `promptfooconfig.yaml` scaffold in the current directory with example prompts, providers, and tests.
|
||||
|
||||
To start from an official example:
|
||||
```bash
|
||||
npx promptfoo@0.121.17 init --example openai-mcp
|
||||
npx promptfoo@0.121.17 init --example openai-structured-output
|
||||
npx promptfoo@0.121.17 init --example openai-responses
|
||||
npx promptfoo@0.121.17 init --example openai-audio
|
||||
```
|
||||
|
||||
## API keys
|
||||
|
||||
Promptfoo reads provider credentials from environment variables. Set them before running evals:
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=sk-...
|
||||
export ANTHROPIC_API_KEY=sk-ant-...
|
||||
```
|
||||
|
||||
Other providers (Bedrock, Azure, Vertex) follow their own SDK credential conventions — see provider-specific docs.
|
||||
|
||||
## Global timeout
|
||||
|
||||
```bash
|
||||
export REQUEST_TIMEOUT_MS=600000 # 10 minutes default
|
||||
```
|
||||
@@ -1,40 +0,0 @@
|
||||
---
|
||||
topic: overview
|
||||
source_keys:
|
||||
- context7-promptfoo-dev
|
||||
- context7-promptfoo-github
|
||||
---
|
||||
|
||||
## What is Promptfoo
|
||||
|
||||
Promptfoo is an open-source, local-first CLI and library for evaluating and red-teaming LLM applications. It enables systematic, repeatable testing of prompts across multiple providers with assertions that grade outputs automatically. Results are stored locally and can be visualised in a browser UI or exported.
|
||||
|
||||
It is not a hosted service — all evaluation state stays on your machine unless you explicitly share a result.
|
||||
|
||||
## Core concepts
|
||||
|
||||
**Prompt** — a template with `{{variable}}` placeholders. Can be a plain string, a JSON chat array (`file://prompts/chat.json`), or a JavaScript function (`file://prompts/generate.js`) that returns a string or message array dynamically.
|
||||
|
||||
**Provider** — an LLM endpoint to send the rendered prompt to. Providers are declared as strings (`openai:gpt-5-mini`, `anthropic:messages:claude-sonnet-4-5`) or objects with a `config:` block for additional parameters.
|
||||
|
||||
**Test case** — one input scenario. Contains `vars:` (values substituted into prompt variables) and `assert:` (a list of assertions that the response must satisfy).
|
||||
|
||||
**Assertion** — a pass/fail check on the model output. Ranges from deterministic (`contains`, `regex`, `equals`) to model-graded (`llm-rubric`, `factuality`).
|
||||
|
||||
**Eval** — one complete run: every prompt × every provider × every test case is executed and each assertion is scored. Results are a table of pass/fail cells with per-assertion metrics.
|
||||
|
||||
**defaultTest** — a top-level config key whose `assert:` and `options:` are merged into every test case, avoiding repetition.
|
||||
|
||||
## Mental model
|
||||
|
||||
Think of an eval as a spreadsheet where rows are test cases and columns are (prompt, provider) pairs. Each cell contains the model output and assertion results. Running `promptfoo eval` fills the spreadsheet; `promptfoo view` opens it in a browser.
|
||||
|
||||
The config file (`promptfooconfig.yaml`) is the source of truth for a given eval. It is committed alongside your prompt files so evals are reproducible.
|
||||
|
||||
## What it is used for
|
||||
|
||||
- **Regression testing** — catch prompt regressions before deploying changes
|
||||
- **Side-by-side model comparison** — evaluate multiple providers on identical test suites
|
||||
- **Red-teaming** — generate and run adversarial tests (jailbreaks, prompt injection, harmful content)
|
||||
- **Dataset generation** — AI-generate test cases from a prompt template
|
||||
- **CI/CD gating** — fail a pull request when assertion pass rate drops
|
||||
@@ -1,133 +0,0 @@
|
||||
---
|
||||
topic: redteam
|
||||
source_keys:
|
||||
- context7-promptfoo-dev
|
||||
- context7-promptfoo-github
|
||||
---
|
||||
|
||||
## What red-teaming does
|
||||
|
||||
Red-teaming in Promptfoo generates adversarial test cases that probe an LLM application for security vulnerabilities and safety failures. It is separate from standard evals — you configure it under a `redteam:` block and use `promptfoo redteam generate` to produce test cases, then run them with `promptfoo eval`.
|
||||
|
||||
## Basic configuration
|
||||
|
||||
```yaml
|
||||
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
||||
redteam:
|
||||
plugins:
|
||||
- harmful
|
||||
- prompt-injection
|
||||
- hijacking
|
||||
strategies:
|
||||
- jailbreak
|
||||
- jailbreak:composite
|
||||
- prompt-injection
|
||||
|
||||
providers:
|
||||
- openai:gpt-5-mini
|
||||
```
|
||||
|
||||
## Plugins (what to test for)
|
||||
|
||||
Plugins define the vulnerability categories to probe:
|
||||
|
||||
| Plugin | Tests for |
|
||||
|---|---|
|
||||
| `harmful` | General harmful content generation |
|
||||
| `harmful:misinformation-disinformation` | False or misleading information |
|
||||
| `harmful:cybercrime` | Cyberattack assistance |
|
||||
| `prompt-injection` | Injection via user input overriding system instructions |
|
||||
| `hijacking` | Redirecting the assistant to unintended tasks |
|
||||
| `jailbreak` | Breaking safety guardrails |
|
||||
| `rbac` | Role-based access control bypass |
|
||||
| `hallucination` | Fabricated facts |
|
||||
| `debug-access` | Exposing internal debug interfaces |
|
||||
| `shell-injection` | Shell command injection |
|
||||
| `sql-injection` | SQL injection via natural language |
|
||||
| `ssrf` | Server-side request forgery |
|
||||
|
||||
## Strategies (how to attack)
|
||||
|
||||
Strategies control the attack method applied to each plugin's test cases:
|
||||
|
||||
| Strategy | Description |
|
||||
|---|---|
|
||||
| `jailbreak` | Classic jailbreak prompts |
|
||||
| `jailbreak:composite` | Chained / composite jailbreak attempts |
|
||||
| `prompt-injection` | Inject instructions via user-controlled content |
|
||||
| `base64` | Encode attack payload in base64 |
|
||||
| `leetspeak` | Obfuscate with leet substitutions |
|
||||
| `rot13` | Encode with ROT-13 |
|
||||
| `iterative` | Iteratively refine attack prompts |
|
||||
| `ensemble` | Combine multiple strategies |
|
||||
|
||||
## Generating adversarial tests
|
||||
|
||||
```bash
|
||||
promptfoo redteam generate
|
||||
promptfoo redteam generate -c promptfooconfig.yaml
|
||||
```
|
||||
|
||||
Then run the generated tests:
|
||||
```bash
|
||||
promptfoo eval
|
||||
```
|
||||
|
||||
## Node.js API
|
||||
|
||||
```javascript
|
||||
import { redteam } from 'promptfoo';
|
||||
|
||||
const result = await redteam.generate({
|
||||
target: {
|
||||
prompt: 'You are a helpful assistant. Answer user questions.',
|
||||
model: 'openai:chat:gpt-5.5',
|
||||
},
|
||||
plugins: ['prompt-injection', 'jailbreak', 'rbac'],
|
||||
numTests: 5,
|
||||
strategies: ['iterative', 'ensemble'],
|
||||
});
|
||||
|
||||
result.tests.forEach((test, i) => {
|
||||
console.log(`${i + 1}. [${test.category}] ${test.prompt.substring(0, 100)}...`);
|
||||
});
|
||||
```
|
||||
|
||||
## Cascading failures (agentic AI)
|
||||
|
||||
For agentic systems, test cascading failures and multi-step attacks:
|
||||
|
||||
```yaml
|
||||
redteam:
|
||||
plugins:
|
||||
- hallucination
|
||||
- harmful:misinformation-disinformation
|
||||
- divergent-repetition
|
||||
strategies:
|
||||
- jailbreak
|
||||
- prompt-injection
|
||||
```
|
||||
|
||||
## OWASP and MITRE alignment
|
||||
|
||||
Plugins map to OWASP LLM Top 10 and MITRE ATLAS categories. Use `debug-access`, `shell-injection`, `sql-injection`, `ssrf` together to cover the MITRE ATLAS initial-access cluster:
|
||||
|
||||
```yaml
|
||||
redteam:
|
||||
plugins:
|
||||
- debug-access
|
||||
- harmful:cybercrime
|
||||
- shell-injection
|
||||
- sql-injection
|
||||
- ssrf
|
||||
strategies:
|
||||
- base64
|
||||
- jailbreak
|
||||
- leetspeak
|
||||
- prompt-injection
|
||||
- rot13
|
||||
```
|
||||
|
||||
## Integration with Burp Suite
|
||||
|
||||
Promptfoo can generate targeted red-team test cases for use with Burp Suite. Use `promptfoo redteam generate` to produce test payloads and then pass them into Burp's active scanner.
|
||||
@@ -1,15 +0,0 @@
|
||||
# Sources
|
||||
|
||||
## context7-promptfoo-dev
|
||||
|
||||
- **URL:** context7:/websites/promptfoo_dev
|
||||
- **Description:** Official Promptfoo website documentation — overview, getting started, configuration, CLI, assertions, CI/CD, red-teaming
|
||||
- **Contributing files:** overview.md, installation.md, configuration.md, cli-reference.md, assertions.md, examples.md, redteam.md, ci-cd.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## context7-promptfoo-github
|
||||
|
||||
- **URL:** context7:/promptfoo/promptfoo
|
||||
- **Description:** Promptfoo GitHub repository — provider-specific docs, Node.js API examples, advanced configuration patterns
|
||||
- **Contributing files:** overview.md, installation.md, configuration.md, cli-reference.md, assertions.md, examples.md, redteam.md, ci-cd.md
|
||||
- **Status:** `extracted`
|
||||
@@ -1,13 +0,0 @@
|
||||
```yaml
|
||||
version: "1.0"
|
||||
updated: 2026-06-21
|
||||
|
||||
when: invoked by explicit trigger ("write an agent for X", "create a subagent that does Y", "add an agent to the Z plugin") or implicit request to author a Claude Code subagent or GitHub Copilot CLI plugin agent definition file
|
||||
|
||||
# source: omitted — self-authored original; no upstream content adopted
|
||||
|
||||
references:
|
||||
- https://code.claude.com/docs/en/sub-agents
|
||||
- https://code.claude.com/docs/en/plugins-reference
|
||||
- https://docs.github.com/en/copilot/reference/custom-agents-configuration
|
||||
```
|
||||
@@ -1,111 +0,0 @@
|
||||
---
|
||||
name: write-agent
|
||||
description: >
|
||||
Use when the user wants to author a new agent definition file for Claude Code (subagent) or
|
||||
GitHub Copilot CLI (plugin agent). Triggers: "write an agent for X", "create a subagent that
|
||||
does Y", "add an agent to the Z plugin", "build a cross-tool agent". Do NOT use when the user
|
||||
wants to create a role skill that loads inline into the current conversation (use /write-skill
|
||||
with category: roles), scaffold a new plugin from scratch (use /plugin-create), edit or update
|
||||
an existing agent definition (use upgrade-agent), or author skills inside a plugin (use
|
||||
/write-skill).
|
||||
metadata:
|
||||
category: factory
|
||||
model: sonnet
|
||||
---
|
||||
|
||||
<requirements>
|
||||
|
||||
## Required inputs
|
||||
|
||||
- **Agent type** — standalone subagent or plugin agent; inferred from request if obvious ("add an agent to plugin X" → plugin agent, "create a subagent" → standalone), ask if ambiguous
|
||||
- **Agent name** — kebab-case slug; inferred from user description if not stated, ask if ambiguous
|
||||
- **Plugin name** — plugin agents only; must be an existing plugin in `plugins/`; ask if not stated
|
||||
- **Purpose + use cases** — what the agent does, what tasks it handles exclusively; source for the `description` field and system prompt body
|
||||
- **Tool access rationale** — which tools the agent needs and why (allowlist, denylist, or inherit all); ask if not stated
|
||||
|
||||
## Constraints
|
||||
|
||||
- One skill, two branches — determine agent type before any other step; do not proceed until type is confirmed
|
||||
- Plugin agents: always generate both `<name>.md` (Claude Code) and `<name>.agent.md` (Copilot CLI) — same system prompt body, translated frontmatter; never generate one without the other
|
||||
- Plugin agents: `plugins/<name>/` must exist before writing — stop and redirect to `/plugin-create` if not found
|
||||
- Standalone subagents: write to `core/agents/<name>.md` per ADR-0010 — never write directly to `.claude/agents/`
|
||||
- Do not create role skills (inline mode switches, `category: roles`) — if the user wants a cognitive mode switch without context isolation, redirect to `/write-skill`
|
||||
- Load `references/cross-compat.md` before writing any plugin agent frontmatter — Claude Code and Copilot CLI diverge in field names, tool names, and supported features
|
||||
- Copy agent files from `assets/` templates — never generate from memory; stop and report the path if a template is missing
|
||||
- Flag Claude-only fields (`permissionMode`, `isolation`, `maxTurns`, `memory`, `hooks`, `mcpServers`) in a handoff comment when generating `.agent.md` — these have no Copilot CLI equivalent
|
||||
- Body under 500 lines
|
||||
|
||||
</requirements>
|
||||
|
||||
<steps>
|
||||
|
||||
## Process
|
||||
|
||||
1. **Determine type.** Infer standalone subagent or plugin agent from the request. Ask if ambiguous. Hard gate: do not proceed until type is confirmed.
|
||||
|
||||
2. **Validate target.** For plugin agents: confirm `plugins/<name>/` exists. Stop and redirect to `/plugin-create` if not. For standalone subagents: confirm `core/agents/` exists.
|
||||
|
||||
3. **Scan for overlap.** Check `core/agents/` (standalone) or `plugins/<name>/agents/` (plugin) for agents with similar purpose or name. Surface any found and wait for direction before continuing.
|
||||
|
||||
4. **Grill.** One question at a time, with a recommendation for each: agent name, purpose, tasks it handles exclusively, what it explicitly does NOT do, tool access rationale, model selection, isolation needs (subagents only), and any optional fields worth setting. Stop when there is shared understanding of all five required inputs.
|
||||
|
||||
5. **Conflict check.** Spawn a sub-agent: read `docs/ai-constitution.md`, `docs/research/ai-coding-factory/ai-coding-factory-principles.md`, and `docs/notes/factory-integration-decisions.md`, then check the agreed agent design against all three. Return a numbered list of genuine unresolved tensions, or confirm none found. Hard gate: resolve any findings before proceeding.
|
||||
|
||||
6. **Write and test the description field.** Draft the `description:` using the agreed purpose and use cases — this is what Claude reads to decide whether to delegate. Propose negative trigger cases; get explicit user confirmation. Test explicit, implicit, and negative cases and show per-case PASS/FAIL. A failed case means revise and retest — do not proceed.
|
||||
|
||||
7. **Walk through the system prompt body.** Propose the body section by section: role statement, task scope, explicit out-of-scope items, behavioral constraints. Wait for explicit confirmation of each before writing.
|
||||
|
||||
8. **Walk through optional frontmatter.** For each optional field (`model`, `tools`, `disallowedTools`, `maxTurns`, `effort`, `isolation`, `permissionMode`, `memory`, `background`): propose a value if warranted by the agreed design, or confirm omission. Load `references/claude-code-agents.md` for field semantics. Wait for confirmation of each.
|
||||
|
||||
9. **Check template.** Load the appropriate template(s) from `assets/`: `subagent.md` for standalone subagents; `plugin-agent-claude.md` and `plugin-agent-copilot.md` for plugin agents. Stop and report the path if any template is missing — do not generate from memory.
|
||||
|
||||
10. **Copy and fill.** Copy the template(s) to the target path(s). Fill with confirmed content. For plugin agents: load `references/cross-compat.md` and translate Claude frontmatter to Copilot equivalents in `<name>.agent.md`. Keep both bodies identical. Note Claude-only fields that have no Copilot equivalent in a handoff comment at the top of `<name>.agent.md`.
|
||||
|
||||
11. **Invoke `write-eval`.** Do not mark the agent complete without an eval file.
|
||||
|
||||
12. **Run self-check.** Work through every item in the Self-check section below.
|
||||
|
||||
13. **Prompt for HITL.** Ask the user to open a fresh session, trigger the agent, and confirm behavior before committing.
|
||||
|
||||
## Output format
|
||||
|
||||
For standalone subagents:
|
||||
- `core/agents/<name>.md` — copy-filled from `assets/subagent.md`
|
||||
- `plugins/kyberforge/skills/write-agent/evals/<name>.yaml` — produced by write-eval
|
||||
|
||||
For plugin agents:
|
||||
- `plugins/<plugin-name>/agents/<name>.md` — copy-filled from `assets/plugin-agent-claude.md`
|
||||
- `plugins/<plugin-name>/agents/<name>.agent.md` — copy-filled from `assets/plugin-agent-copilot.md`; Claude-only fields noted in a handoff comment at the top
|
||||
- `plugins/kyberforge/skills/write-agent/evals/<name>.yaml` — produced by write-eval
|
||||
|
||||
</steps>
|
||||
|
||||
<checks>
|
||||
|
||||
## Failure handling
|
||||
|
||||
- Agent type cannot be determined from the request — stop and ask; do not proceed without explicit type confirmation
|
||||
- `plugins/<name>/` not found for a plugin agent — stop, report the path checked, redirect to `/plugin-create`
|
||||
- `core/agents/` not found for a standalone subagent — stop, report the path, do not write to `.claude/agents/` directly
|
||||
- Template missing from `assets/` — stop, report the exact path searched, do not generate from memory
|
||||
- Overlap found in target directory — surface it and wait for explicit direction; do not continue
|
||||
- `write-eval` fails or is unavailable — flag, do not mark the agent complete
|
||||
- Conflict check sub-agent returns unresolved tensions — resolve before writing any file
|
||||
|
||||
## Self-check
|
||||
|
||||
- [ ] Agent type confirmed before any other step
|
||||
- [ ] Plugin validated to exist at `plugins/<name>/` before any file was written (plugin agents only)
|
||||
- [ ] Overlap check completed in the correct target directory before any content was written
|
||||
- [ ] Conflict check sub-agent ran — all findings resolved before writing began
|
||||
- [ ] Description field tested against explicit, implicit, and negative cases — all passed before body was written
|
||||
- [ ] Negative trigger cases confirmed by user before testing
|
||||
- [ ] System prompt body confirmed section by section before writing
|
||||
- [ ] Optional frontmatter fields confirmed or explicitly omitted
|
||||
- [ ] Template(s) loaded from `assets/` — not generated from memory
|
||||
- [ ] For plugin agents: both `.md` and `.agent.md` written; bodies are identical; Claude-only fields noted in handoff comment
|
||||
- [ ] `references/cross-compat.md` loaded before translating plugin agent frontmatter (plugin agents only)
|
||||
- [ ] Standalone subagent written to `core/agents/<name>.md` — not to `.claude/agents/`
|
||||
- [ ] `write-eval` invoked — eval file exists and covers trigger cases
|
||||
|
||||
</checks>
|
||||
@@ -1,8 +0,0 @@
|
||||
---
|
||||
name: AGENT_NAME
|
||||
description: AGENT_DESCRIPTION
|
||||
tools: AGENT_TOOLS
|
||||
model: AGENT_MODEL
|
||||
---
|
||||
|
||||
AGENT_SYSTEM_PROMPT
|
||||
@@ -1,14 +0,0 @@
|
||||
<!-- HANDOFF: The following Claude Code fields were not ported — no Copilot CLI equivalent:
|
||||
CLAUDE_ONLY_FIELDS_NOTE
|
||||
Add Copilot-specific fields (target, disable-model-invocation, user-invocable) if needed.
|
||||
Verify tool names against Copilot CLI tool aliases (codebase, search, edit/editFiles, etc.)
|
||||
rather than Claude Code internal names (Read, Grep, Bash, etc.). -->
|
||||
---
|
||||
name: AGENT_NAME
|
||||
description: AGENT_DESCRIPTION
|
||||
tools:
|
||||
- AGENT_TOOLS_COPILOT
|
||||
model: AGENT_MODEL_COPILOT
|
||||
---
|
||||
|
||||
AGENT_SYSTEM_PROMPT
|
||||
@@ -1,8 +0,0 @@
|
||||
---
|
||||
name: AGENT_NAME
|
||||
description: AGENT_DESCRIPTION
|
||||
tools: AGENT_TOOLS
|
||||
model: AGENT_MODEL
|
||||
---
|
||||
|
||||
AGENT_SYSTEM_PROMPT
|
||||
@@ -1,103 +0,0 @@
|
||||
skill_name: write-agent
|
||||
|
||||
trigger_tests:
|
||||
- id: explicit-basic
|
||||
name: Explicit trigger — basic invocation
|
||||
query: "write an agent for code review"
|
||||
should_trigger: true
|
||||
|
||||
- id: explicit-plugin
|
||||
name: Explicit trigger — plugin agent
|
||||
query: "add an agent to the security-tools plugin"
|
||||
should_trigger: true
|
||||
|
||||
- id: implicit-isolation
|
||||
name: Implicit trigger — isolated worker need
|
||||
query: "I need a specialized worker that handles DB migrations in isolation"
|
||||
should_trigger: true
|
||||
|
||||
- id: negative-role-skill
|
||||
name: Negative — role skill (inline mode switch)
|
||||
query: "Add an architect role to my session"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-plugin-create
|
||||
name: Negative — plugin scaffolding
|
||||
query: "Create a new plugin called security-tools"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-edit-existing
|
||||
name: Negative — editing an existing agent
|
||||
query: "Update the description on my existing code-reviewer agent"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-skill-authoring
|
||||
name: Negative — skill authoring inside a plugin
|
||||
query: "Write a skill for my refactor plugin"
|
||||
should_trigger: false
|
||||
|
||||
output_tests:
|
||||
- id: subagent-output-path
|
||||
name: Standalone subagent written to core/agents/
|
||||
type: deterministic
|
||||
prompt: "Create a standalone subagent called db-migrator that runs database migrations in an isolated context"
|
||||
expected_output: >
|
||||
The skill produces a file at core/agents/db-migrator.md containing valid YAML frontmatter
|
||||
with at minimum name and description fields, followed by a system prompt body.
|
||||
assertions:
|
||||
- "Output references the file path core/agents/db-migrator.md"
|
||||
- "Output does not reference .claude/agents/ as the write target"
|
||||
- "Generated file contains a --- frontmatter block with name: db-migrator"
|
||||
|
||||
- id: plugin-agent-dual-file
|
||||
name: Plugin agent produces both .md and .agent.md
|
||||
type: deterministic
|
||||
prompt: "Add a cross-tool agent called security-scanner to the existing kyberforge plugin"
|
||||
expected_output: >
|
||||
The skill produces two files: plugins/kyberforge/agents/security-scanner.md (Claude Code)
|
||||
and plugins/kyberforge/agents/security-scanner.agent.md (Copilot CLI). Both files share
|
||||
an identical system prompt body.
|
||||
assertions:
|
||||
- "Output references plugins/kyberforge/agents/security-scanner.md"
|
||||
- "Output references plugins/kyberforge/agents/security-scanner.agent.md"
|
||||
- "Skill does not produce only one of the two files"
|
||||
|
||||
- id: claude-frontmatter-required-fields
|
||||
name: Claude Code agent file contains required frontmatter fields
|
||||
type: deterministic
|
||||
prompt: "Write an agent for linting that reads files and reports violations"
|
||||
expected_output: >
|
||||
The generated .md file contains a YAML frontmatter block with at minimum name: and
|
||||
description: fields. The description field contains delegation guidance — when Claude
|
||||
should invoke this agent.
|
||||
assertions:
|
||||
- "Generated frontmatter contains name:"
|
||||
- "Generated frontmatter contains description:"
|
||||
- "The description value is not empty or a placeholder"
|
||||
|
||||
- id: copilot-handoff-comment
|
||||
name: Copilot CLI .agent.md contains handoff comment for Claude-only fields
|
||||
type: deterministic
|
||||
prompt: "Add a plugin agent to the kyberforge plugin that uses permissionMode: bypassPermissions and isolation: worktree"
|
||||
expected_output: >
|
||||
The generated .agent.md file contains a comment at the top noting that permissionMode
|
||||
and isolation are Claude Code-only fields with no Copilot CLI equivalent.
|
||||
assertions:
|
||||
- "The .agent.md file contains a HANDOFF comment or equivalent note"
|
||||
- "The comment names at least one Claude-only field that was not ported"
|
||||
- "The .agent.md frontmatter does not contain permissionMode or isolation fields"
|
||||
|
||||
- id: description-field-quality
|
||||
name: LLM rubric — description field names specific tasks not just a role
|
||||
type: llm-rubric
|
||||
prompt: "Create a standalone subagent called api-tester that tests REST API endpoints for correctness and schema compliance"
|
||||
expected_output: >
|
||||
The description field in the generated agent file should specify delegation conditions
|
||||
precisely enough that an orchestrator model can decide whether to invoke this agent
|
||||
without ambiguity. It should name the task domain, the conditions for delegation,
|
||||
and ideally one or more negative cases. A description that only says "Tests APIs" or
|
||||
restates the agent name is insufficient.
|
||||
assertions:
|
||||
- "The description field mentions what kind of task triggers delegation to this agent"
|
||||
- "The description field does not simply restate the agent name"
|
||||
- "The description is specific enough to distinguish this agent from a general-purpose testing agent"
|
||||
@@ -1,91 +0,0 @@
|
||||
---
|
||||
topic: claude-code-agents
|
||||
source_keys:
|
||||
- claude-code-docs-subagents
|
||||
- claude-code-docs-plugins
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Claude Code agents (called subagents) are Markdown files with YAML frontmatter stored in one of several scoped locations. Each runs in its own context window with a custom system prompt, specific tool access, and independent permissions. The parent conversation delegates to a subagent when its `description` matches the task.
|
||||
|
||||
## File format
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: code-reviewer
|
||||
description: Reviews code for correctness, security, and maintainability. Use proactively after code changes.
|
||||
tools: Read, Grep, Glob
|
||||
model: sonnet
|
||||
---
|
||||
|
||||
You are a senior code reviewer. Review for:
|
||||
1. Correctness: logic errors, edge cases
|
||||
2. Security: injection, auth bypass
|
||||
3. Maintainability: naming, complexity
|
||||
```
|
||||
|
||||
The body becomes the subagent's system prompt. Subagents receive only this system prompt plus basic environment details (working directory) — not the full Claude Code system prompt and not the parent conversation history.
|
||||
|
||||
## Supported frontmatter fields
|
||||
|
||||
Only `name` and `description` are required. All others are optional.
|
||||
|
||||
| Field | Description |
|
||||
|---|---|
|
||||
| `name` | Unique identifier: lowercase letters and hyphens. Used as `agent_type` in hooks. Filename does not have to match. |
|
||||
| `description` | When Claude should delegate to this subagent — determines automatic routing. Write imperatively. |
|
||||
| `tools` | Allowlist of tools the subagent can use. Inherits all tools if omitted. Comma-separated or array. Use `Agent(type1, type2)` syntax to restrict which subagents this agent can spawn. |
|
||||
| `disallowedTools` | Denylist — removed from inherited or specified list. If both `tools` and `disallowedTools` are set, denylist is applied first. Supports `mcp__<server>` patterns. |
|
||||
| `model` | Model alias (`sonnet`, `opus`, `haiku`, `fable`) or full ID (`claude-opus-4-8`). Defaults to `inherit` (parent model). |
|
||||
| `permissionMode` | `default`, `acceptEdits`, `auto`, `dontAsk`, `bypassPermissions`, `plan`. Inherited from parent; parent's mode takes precedence if stricter. **Ignored for plugin agents.** |
|
||||
| `maxTurns` | Maximum agentic turns before the subagent stops. |
|
||||
| `skills` | Skills to preload into the subagent's context at startup (full skill content injected, not just description). |
|
||||
| `mcpServers` | MCP servers scoped to this subagent. Inline definitions connect on start, disconnect on finish. String references reuse the parent session's connection. **Ignored for plugin agents.** |
|
||||
| `hooks` | Lifecycle hooks scoped to this subagent. **Ignored for plugin agents.** |
|
||||
| `memory` | Persistent memory scope: `user`, `project`, or `local`. Enables cross-session learning. |
|
||||
| `background` | `true` to always run as a background task. Default: `false`. |
|
||||
| `effort` | Reasoning effort: `low`, `medium`, `high`, `xhigh`, `max`. Overrides session effort level. |
|
||||
| `isolation` | `worktree` — runs in a temporary git worktree (isolated repo copy). Auto-cleaned up if no changes. |
|
||||
| `color` | Display color in the task list: `red`, `blue`, `green`, `yellow`, `purple`, `orange`, `pink`, `cyan`. |
|
||||
| `initialPrompt` | Auto-submitted as the first user turn when this agent runs as the main session agent (via `--agent`). |
|
||||
|
||||
## Storage locations and scope priority
|
||||
|
||||
| Location | Scope | Priority |
|
||||
|---|---|---|
|
||||
| Managed settings `.claude/agents/` | Organization-wide | 1 (highest) |
|
||||
| `--agents` CLI flag | Current session only | 2 |
|
||||
| `.claude/agents/` | Current project | 3 |
|
||||
| `~/.claude/agents/` | All projects (user-level) | 4 |
|
||||
| Plugin `agents/` directory | Where plugin is enabled | 5 (lowest) |
|
||||
|
||||
When the same `name` is defined in multiple locations, the highest-priority location wins. Claude Code scans both `.claude/agents/` and `~/.claude/agents/` recursively — files can be organized into subfolders. Identity comes from the `name` frontmatter, not the filename.
|
||||
|
||||
## Plugin agent constraints
|
||||
|
||||
Plugin agents (in a plugin's `agents/` directory) have three fields ignored for security reasons:
|
||||
- `hooks` — ignored
|
||||
- `mcpServers` — ignored
|
||||
- `permissionMode` — ignored
|
||||
|
||||
If any of these are needed, copy the agent file to `.claude/agents/` or `~/.claude/agents/` instead.
|
||||
|
||||
Plugin agents in subfolders get a scoped identifier: `plugin-name:subfolder:agent-name` (e.g., a file at `agents/review/security.md` in plugin `my-plugin` registers as `my-plugin:review:security`).
|
||||
|
||||
## Repo-specific placement (this repo)
|
||||
|
||||
Per ADR-0010, two categories of agent definitions exist in this repo:
|
||||
|
||||
- **Subagent definitions** (isolated context, separate context window): live in `core/agents/` — deployed to `~/.claude/agents/` by `install.sh`. These are the "true" subagents.
|
||||
- **Plugin agents**: live in `plugins/<name>/agents/<name>.md`. Installed with the plugin.
|
||||
- **Role skills** (inline mode switches, NOT agents): live in `.agents/skills/` with `category: roles`.
|
||||
|
||||
Never put role skills in `core/agents/`. Never put subagent definitions in `.agents/skills/`. The distinction is the isolation boundary — subagents get a fresh context window; role skills load inline.
|
||||
|
||||
## Model routing guidance (from factory §9)
|
||||
|
||||
- `haiku` — formatting, classification, fast lookups
|
||||
- `sonnet` — most coding, review, analysis
|
||||
- `opus` — adversarial reasoning, complex multi-step, security-critical
|
||||
- Omit `model` to inherit from the parent session
|
||||
@@ -1,86 +0,0 @@
|
||||
---
|
||||
topic: copilot-cli-agents
|
||||
source_keys:
|
||||
- github-copilot-custom-agents-config
|
||||
- github-blog-copilot-cli-agents
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
GitHub Copilot CLI agents are defined as Markdown files with `.agent.md` extension and YAML frontmatter. They live in the `.github/agents/` directory of a repository and are version-controlled with the codebase. Users invoke them via the `/agent` slash command in the CLI.
|
||||
|
||||
## File format
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: Accessibility Expert
|
||||
description: Expert assistant for web accessibility (WCAG 2.1/2.2)
|
||||
model: GPT-4.1
|
||||
tools:
|
||||
- codebase
|
||||
- edit/editFiles
|
||||
- search
|
||||
---
|
||||
|
||||
# Accessibility Expert
|
||||
|
||||
You are a web accessibility expert. Focus on WCAG 2.1/2.2 compliance...
|
||||
```
|
||||
|
||||
The body (below frontmatter) contains the agent's instructions in Markdown. Maximum 30,000 characters.
|
||||
|
||||
## Supported frontmatter fields
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|---|---|---|---|
|
||||
| `name` | String | Recommended | Display identifier for the agent |
|
||||
| `description` | String | Required | Purpose and capabilities — what this agent does |
|
||||
| `target` | String | No | `vscode`, `github-copilot`, or omit for both |
|
||||
| `model` | String | No | Model to use (e.g., `GPT-4.1`, `claude-opus-4-8`). Inherits default if unset. |
|
||||
| `tools` | List | No | Tools the agent can use. Defaults to all tools if omitted. |
|
||||
| `disable-model-invocation` | Boolean | No | When `true`, requires manual agent selection. |
|
||||
| `user-invocable` | Boolean | No | Whether user can manually select this agent. Defaults to `true`. |
|
||||
| `mcp-servers` | Object | No | Additional MCP server configurations. Not used in VS Code/IDE agents. |
|
||||
| `metadata` | Object | No | Key-value annotation pairs. Not applicable to IDE agents. |
|
||||
|
||||
## Tools field
|
||||
|
||||
Tools can be configured three ways:
|
||||
|
||||
- **All tools** (default): omit the `tools` property, or use `tools: ["*"]`
|
||||
- **Specific tools**: list names, e.g. `tools: ["codebase", "edit/editFiles", "search"]`
|
||||
- **No tools**: `tools: []`
|
||||
|
||||
Available tool aliases (case-insensitive): `execute`, `read`, `edit`, `search`, `agent`, `web`, `todo`.
|
||||
|
||||
MCP server tools use namespacing: `mcp-server-name/tool-name` or `mcp-server-name/*` for all tools from a server.
|
||||
|
||||
Some available tools: `gh`, `git`, `codebase`, `search`, `githubRepo`, `runCommands`, `runTests`, `edit/editFiles`, `terraform`, `conftest`, `jq`, `curl`, `semgrep`, `trivy`, `gitleaks`.
|
||||
|
||||
## File location
|
||||
|
||||
Copilot CLI agents live in `.github/agents/` in the repository root. The filename uses the `.agent.md` extension (e.g., `accessibility.agent.md`).
|
||||
|
||||
## Plugin placement (this repo)
|
||||
|
||||
Per the cross-compat reference, plugin agents for Copilot CLI use the `.agent.md` naming:
|
||||
- Claude Code reads: `agents/<name>.md`
|
||||
- Copilot CLI reads: `agents/<name>.agent.md`
|
||||
|
||||
Both files share the same system prompt body. Only the frontmatter differs: Copilot uses `tools:` as an array with different tool names; Claude Code uses `tools:` as comma-separated internal tool names.
|
||||
|
||||
## Fields not supported in Copilot CLI
|
||||
|
||||
These Claude Code agent fields have no Copilot CLI equivalent:
|
||||
- `permissionMode`
|
||||
- `maxTurns`
|
||||
- `skills` (Claude-specific skill injection)
|
||||
- `mcpServers` (inline per-agent MCP — not supported in VS Code/IDE)
|
||||
- `hooks`
|
||||
- `memory`
|
||||
- `background`
|
||||
- `effort`
|
||||
- `isolation`
|
||||
- `color`
|
||||
- `initialPrompt`
|
||||
- `disallowedTools`
|
||||
@@ -1,73 +0,0 @@
|
||||
---
|
||||
topic: cross-tool-agent-compatibility
|
||||
source_keys:
|
||||
- claude-code-docs-plugins
|
||||
- github-copilot-custom-agents-config
|
||||
- plugin-marketplace-architecture
|
||||
---
|
||||
|
||||
## The core rule
|
||||
|
||||
Skills are the portable primitive — identical format in both tools. Agents diverge. The system prompt body can be shared; the frontmatter and filename cannot.
|
||||
|
||||
## Divergence table
|
||||
|
||||
| Concern | Claude Code | GitHub Copilot CLI | Portable choice |
|
||||
|---|---|---|---|
|
||||
| File extension | `agents/<name>.md` | `agents/<name>.agent.md` | Ship both files — same body, tool-specific frontmatter |
|
||||
| Required fields | `name`, `description` | `description` | Use both `name` and `description` in both files |
|
||||
| `tools` format | Comma-separated string or array; internal tool names (e.g. `Read`, `Grep`, `Bash`) | Array; Copilot tool aliases (e.g. `codebase`, `search`, `edit/editFiles`) | Different values — maintain separately |
|
||||
| `model` values | `sonnet`, `opus`, `haiku`, `fable`, or full model ID | Copilot model names (e.g. `GPT-4.1`, `claude-opus-4-8`) | Maintain separately |
|
||||
| `permissionMode` | Supported | Not supported | Claude-only field |
|
||||
| `maxTurns` | Supported | Not supported | Claude-only field |
|
||||
| `skills` | Preloads skills into context | Not supported | Claude-only field |
|
||||
| `mcpServers` | Inline or reference; ignored in plugin agents | `mcp-servers` (different key); not used in IDE agents | Maintain separately if needed |
|
||||
| `hooks` | Supported (ignored in plugin agents) | Not supported | Claude-only field |
|
||||
| `memory` | `user`, `project`, `local` | Not supported | Claude-only field |
|
||||
| `background` | `true`/`false` | Not supported | Claude-only field |
|
||||
| `effort` | `low`/`medium`/`high`/`xhigh`/`max` | Not supported | Claude-only field |
|
||||
| `isolation` | `worktree` | Not supported | Claude-only field |
|
||||
| `color` | `red`/`blue`/etc. | Not supported | Claude-only field |
|
||||
| `target` | Not supported | `vscode` or `github-copilot` | Copilot-only field |
|
||||
| `disable-model-invocation` | Not supported | Boolean | Copilot-only field |
|
||||
| `user-invocable` | Not supported | Boolean (default `true`) | Copilot-only field |
|
||||
| `metadata` | Not supported | Key-value object | Copilot-only field |
|
||||
|
||||
## The two-file pattern
|
||||
|
||||
For every cross-tool agent in a plugin, ship two files with the same body:
|
||||
|
||||
```text
|
||||
plugins/<plugin-name>/agents/
|
||||
├── <name>.md ← Claude Code (frontmatter: name, description, tools, model, ...)
|
||||
└── <name>.agent.md ← Copilot CLI (frontmatter: name, description, tools as array, ...)
|
||||
```
|
||||
|
||||
The system prompt body (everything below the frontmatter `---`) is identical in both. Copy-fill both from the same system prompt source. Update both when the system prompt changes.
|
||||
|
||||
If the agent is Claude Code-only (not intended for Copilot), ship only `<name>.md` and note it explicitly.
|
||||
|
||||
## Plugin-specific constraints (Claude Code)
|
||||
|
||||
Plugin agents in Claude Code cannot use: `hooks`, `mcpServers`, `permissionMode`. These fields are silently ignored. If the agent needs them, it must be a user-level or project-level agent (`.claude/agents/`), not a plugin agent.
|
||||
|
||||
## Subagent vs role skill (this repo)
|
||||
|
||||
A common mistake: confusing role skills with subagents.
|
||||
|
||||
| Type | What it is | Where it goes | When to use |
|
||||
|---|---|---|---|
|
||||
| Role skill | Inline mode switch; loads into current conversation | `.agents/skills/` with `category: roles` | When you want Claude to adopt a cognitive mode (Architect, Reviewer) without context isolation |
|
||||
| Subagent | Fresh context window, own system prompt, isolated | `core/agents/` → deployed to `~/.claude/agents/` | When a task would flood the main context (research, parallel work, long exploration) |
|
||||
| Plugin agent | Installed with plugin; same isolation as subagent | `plugins/<name>/agents/<name>.md` + `.agent.md` | When the agent is part of a distributable plugin |
|
||||
|
||||
Write-agent creates subagents and plugin agents. Write-skill creates role skills. Do not mix the two.
|
||||
|
||||
## Recommended authoring stance
|
||||
|
||||
Claude Code is the stricter format (more fields, security restrictions for plugins). Treat it as the source of truth. For each agent:
|
||||
|
||||
1. Write the Claude Code `.md` file first with all relevant frontmatter.
|
||||
2. Copy the system prompt body to the Copilot `.agent.md` file.
|
||||
3. Translate frontmatter: keep `name`, `description`; translate `tools` to Copilot aliases; drop Claude-only fields; add Copilot-only fields if needed.
|
||||
4. Note any fields that couldn't be ported (e.g., `permissionMode`, `isolation`) as Claude-only behaviors.
|
||||
@@ -1,36 +0,0 @@
|
||||
# Sources
|
||||
|
||||
## claude-code-docs-subagents
|
||||
|
||||
- **URL:** https://code.claude.com/docs/en/sub-agents
|
||||
- **Description:** Official Claude Code documentation on creating and configuring custom subagents — all frontmatter fields, scope priority, tool restrictions, permission modes.
|
||||
- **Contributing files:** claude-code-agents.md, cross-compat.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## claude-code-docs-plugins
|
||||
|
||||
- **URL:** https://code.claude.com/docs/en/plugins-reference
|
||||
- **Description:** Official Claude Code plugin reference — agent fields available in plugin context and security restrictions (hooks, mcpServers, permissionMode ignored).
|
||||
- **Contributing files:** claude-code-agents.md, cross-compat.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## github-copilot-custom-agents-config
|
||||
|
||||
- **URL:** https://docs.github.com/en/copilot/reference/custom-agents-configuration
|
||||
- **Description:** GitHub Copilot custom agents configuration reference — all frontmatter fields, tools field format, target field, mcp-servers.
|
||||
- **Contributing files:** copilot-cli-agents.md, cross-compat.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## github-blog-copilot-cli-agents
|
||||
|
||||
- **URL:** https://github.blog/ai-and-ml/github-copilot/from-one-off-prompts-to-workflows-how-to-use-custom-agents-in-github-copilot-cli/
|
||||
- **Description:** GitHub blog post on Copilot CLI custom agents — .agent.md format, tool names, invocation pattern.
|
||||
- **Contributing files:** copilot-cli-agents.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## plugin-marketplace-architecture
|
||||
|
||||
- **URL:** plugins/kyberforge/docs/plugin-marketplace-architecture.md
|
||||
- **Description:** Repo-internal reference on Claude Code vs Copilot CLI divergence, two-file agent pattern, plugin constraints.
|
||||
- **Contributing files:** cross-compat.md
|
||||
- **Status:** `extracted`
|
||||
Reference in New Issue
Block a user