Defame1297
5dac52680f
Move all evals into each skill's own evals/ directory and test_scripts.sh into marketplace-architect/scripts/ so test artefacts live alongside the code they test. Also fix: hooks.json array→object, displayName title-case, marketplace.json owner placeholders, and .github/plugin/marketplace.json metadata-wrapper schema divergence. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
63 lines
3.8 KiB
YAML
63 lines
3.8 KiB
YAML
skill_name: marketplace-architect
|
|
|
|
trigger_tests:
|
|
- id: explicit-trigger-refactor
|
|
name: "Explicit trigger — refactor repo into marketplace"
|
|
query: "turn this repo into a plugin marketplace"
|
|
should_trigger: true
|
|
|
|
- id: implicit-trigger-team-sharing
|
|
name: "Implicit trigger — team distribution without saying marketplace"
|
|
query: "how do I share my skills with my team so they can install them"
|
|
should_trigger: true
|
|
|
|
- id: negative-trigger-new-skill
|
|
name: "Negative trigger — new skill authoring belongs to write-skill"
|
|
query: "write me a new skill for code review"
|
|
should_trigger: false
|
|
|
|
output_tests:
|
|
- id: deterministic-cross-compat-loaded
|
|
name: "Deterministic — cross-compat reference loaded before tool-specific recommendations"
|
|
type: deterministic
|
|
prompt: "audit this repo and recommend how to convert it into a plugin marketplace for both Claude Code and Copilot CLI"
|
|
expected_output: >
|
|
The skill reads references/cross-compat.md early in its response and demonstrates
|
|
awareness of the Claude Code / Copilot CLI format differences — specifically that
|
|
manifest paths differ (.claude-plugin/ vs .github/plugin/), agent files differ
|
|
(.md vs .agent.md), and hooks placement differs — before recommending any layout.
|
|
assertions:
|
|
- "Output references or acknowledges the divergence between Claude Code and Copilot CLI manifest paths before proposing a directory layout"
|
|
- "Output does not recommend a single shared plugin.json location without noting the two-location requirement (.claude-plugin/ and plugin root)"
|
|
- "Output does not write or propose writing any files before presenting a plan"
|
|
|
|
- id: deterministic-gate-a-blocks-writes
|
|
name: "Deterministic — Gate A plan presented and approval requested before any writes"
|
|
type: deterministic
|
|
prompt: "help me migrate my skills and agents into a plugin marketplace layout"
|
|
expected_output: >
|
|
The skill produces a migration plan (checklist of old path → new path, one row per
|
|
file) and explicitly asks the user for approval before proceeding to generate any
|
|
manifest files. No plugin.json or marketplace.json is written or shown as written output.
|
|
assertions:
|
|
- "Output contains a migration plan or checklist with at least one file-move entry in the form 'old path → new path'"
|
|
- "Output explicitly asks the user to approve or confirm the plan before proceeding"
|
|
- "Output does not contain a complete plugin.json or marketplace.json file unless the user has already said 'yes' or equivalent in the prompt"
|
|
|
|
- id: llm-rubric-adopt-plugin
|
|
name: "LLM-rubric — adopt external plugin covers all required checks without premature writes"
|
|
type: llm-rubric
|
|
prompt: "I want to add this plugin to my marketplace: https://github.com/example/my-tool-plugin"
|
|
expected_output: >
|
|
The skill fetches and inspects the plugin source, classifies what assets it contains
|
|
(skills, agents, hooks, MCP servers), checks whether any existing plugin in the
|
|
marketplace has a naming conflict, evaluates cross-tool compatibility against the
|
|
divergence table, and summarises what would be added to marketplace.json — all without
|
|
writing any files and without proceeding past Gate A without explicit user approval.
|
|
assertions:
|
|
- "Response identifies what asset types the external plugin contains (skills, agents, hooks, and/or MCP servers)"
|
|
- "Response checks or asks about naming conflicts with existing plugins in the marketplace"
|
|
- "Response notes at least one Claude Code vs Copilot CLI compatibility consideration for the adopted plugin"
|
|
- "Response summarises the proposed change to marketplace.json without writing the file"
|
|
- "Response asks for user approval (Gate A) before any file is created or modified"
|