Defame1297/holocron
0
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases 3 Wiki Activity

fix(kyberforge): co-locate evals/tests with skills and fix manifest inconsistencies

Browse Source 
Move all evals into each skill's own evals/ directory and test_scripts.sh into
marketplace-architect/scripts/ so test artefacts live alongside the code they test.
Also fix: hooks.json array→object, displayName title-case, marketplace.json owner
placeholders, and .github/plugin/marketplace.json metadata-wrapper schema divergence.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Defame1297
2026-06-20 18:56:52 +00:00
parent 2b649782a2
commit 5dac52680f
9 changed files with 7 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

62
plugins/kyberforge/skills/marketplace-architect/evals/eval.yaml Normal file
View File

@@ -0,0 +1,62 @@
skill_name: marketplace-architect
trigger_tests:
- id: explicit-trigger-refactor
name: "Explicit trigger — refactor repo into marketplace"
query: "turn this repo into a plugin marketplace"
should_trigger: true
- id: implicit-trigger-team-sharing
name: "Implicit trigger — team distribution without saying marketplace"
query: "how do I share my skills with my team so they can install them"
should_trigger: true
- id: negative-trigger-new-skill
name: "Negative trigger — new skill authoring belongs to write-skill"
query: "write me a new skill for code review"
should_trigger: false
output_tests:
- id: deterministic-cross-compat-loaded
name: "Deterministic — cross-compat reference loaded before tool-specific recommendations"
type: deterministic
prompt: "audit this repo and recommend how to convert it into a plugin marketplace for both Claude Code and Copilot CLI"
expected_output: >
The skill reads references/cross-compat.md early in its response and demonstrates
awareness of the Claude Code / Copilot CLI format differences — specifically that
manifest paths differ (.claude-plugin/ vs .github/plugin/), agent files differ
(.md vs .agent.md), and hooks placement differs — before recommending any layout.
assertions:
- "Output references or acknowledges the divergence between Claude Code and Copilot CLI manifest paths before proposing a directory layout"
- "Output does not recommend a single shared plugin.json location without noting the two-location requirement (.claude-plugin/ and plugin root)"
- "Output does not write or propose writing any files before presenting a plan"
- id: deterministic-gate-a-blocks-writes
name: "Deterministic — Gate A plan presented and approval requested before any writes"
type: deterministic
prompt: "help me migrate my skills and agents into a plugin marketplace layout"
expected_output: >
The skill produces a migration plan (checklist of old path → new path, one row per
file) and explicitly asks the user for approval before proceeding to generate any
manifest files. No plugin.json or marketplace.json is written or shown as written output.
assertions:
- "Output contains a migration plan or checklist with at least one file-move entry in the form 'old path → new path'"
- "Output explicitly asks the user to approve or confirm the plan before proceeding"
- "Output does not contain a complete plugin.json or marketplace.json file unless the user has already said 'yes' or equivalent in the prompt"
- id: llm-rubric-adopt-plugin
name: "LLM-rubric — adopt external plugin covers all required checks without premature writes"
type: llm-rubric
prompt: "I want to add this plugin to my marketplace: https://github.com/example/my-tool-plugin"
expected_output: >
The skill fetches and inspects the plugin source, classifies what assets it contains
(skills, agents, hooks, MCP servers), checks whether any existing plugin in the
marketplace has a naming conflict, evaluates cross-tool compatibility against the
divergence table, and summarises what would be added to marketplace.json — all without
writing any files and without proceeding past Gate A without explicit user approval.
assertions:
- "Response identifies what asset types the external plugin contains (skills, agents, hooks, and/or MCP servers)"
- "Response checks or asks about naming conflicts with existing plugins in the marketplace"
- "Response notes at least one Claude Code vs Copilot CLI compatibility consideration for the adopted plugin"
- "Response summarises the proposed change to marketplace.json without writing the file"
- "Response asks for user approval (Gate A) before any file is created or modified"

194
plugins/kyberforge/skills/marketplace-architect/scripts/test_scripts.sh Executable file
View File

@@ -0,0 +1,194 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPTS_DIR="$(cd "$(dirname "$0")" && pwd)"
PASS=0; FAIL=0
# Use += to avoid ((var++)) returning 0 exit code when var was 0 under set -e
# ── helpers ─────────────────────────────────────────────────────────────────
tmpdir() { mktemp -d; }
assert_contains() {
local label="$1" expected="$2" actual="$3"
if echo "$actual" | grep -qF "$expected"; then
echo " PASS: $label"
PASS=$((PASS+1))
else
echo " FAIL: $label"
echo " expected to contain: $expected"
echo " got: $(echo "$actual" | head -5)"
FAIL=$((FAIL+1))
fi
}
assert_not_contains() {
local label="$1" unexpected="$2" actual="$3"
if echo "$actual" | grep -qF "$unexpected"; then
echo " FAIL: $label"
echo " expected NOT to contain: $unexpected"
FAIL=$((FAIL+1))
else
echo " PASS: $label"
PASS=$((PASS+1))
fi
}
assert_exit() {
local label="$1" expected="$2" actual="$3"
if [[ "$actual" -eq "$expected" ]]; then
echo " PASS: $label"
PASS=$((PASS+1))
else
echo " FAIL: $label"
echo " expected exit $expected, got $actual"
FAIL=$((FAIL+1))
fi
}
assert_file_exists() {
local label="$1" path="$2"
if [[ -f "$path" ]]; then
echo " PASS: $label"
PASS=$((PASS+1))
else
echo " FAIL: $label"
echo " file not found: $path"
FAIL=$((FAIL+1))
fi
}
assert_file_absent() {
local label="$1" path="$2"
if [[ ! -e "$path" ]]; then
echo " PASS: $label"
PASS=$((PASS+1))
else
echo " FAIL: $label"
echo " file should not exist: $path"
FAIL=$((FAIL+1))
fi
}
# ── inventory.sh tests ───────────────────────────────────────────────────────
echo "=== inventory.sh ==="
# 1. SKILL.md classified as skill
t=$(tmpdir)
mkdir -p "$t/skills/my-skill"
echo "---" > "$t/skills/my-skill/SKILL.md"
out=$("$SCRIPTS_DIR/inventory.sh" "$t")
assert_contains "SKILL.md classified as skill" "skill" "$out"
rm -rf "$t"
# 2. agents/foo.agent.md classified as agent-copilot
t=$(tmpdir)
mkdir -p "$t/agents"
touch "$t/agents/my-agent.agent.md"
out=$("$SCRIPTS_DIR/inventory.sh" "$t")
assert_contains "agents/*.agent.md classified as agent-copilot" "agent-copilot" "$out"
rm -rf "$t"
# 3. ../ in file content produces cross-reference warning
t=$(tmpdir)
mkdir -p "$t/skills/my-skill"
printf -- '---\ndescription: test\n---\nSee ../shared/file.md\n' > "$t/skills/my-skill/SKILL.md"
out=$("$SCRIPTS_DIR/inventory.sh" "$t")
assert_contains "../ cross-reference warning emitted" "Cross-reference" "$out"
assert_contains "../ path shown in warning" "../shared/file.md" "$out"
rm -rf "$t"
# ── gen_manifests.sh tests ───────────────────────────────────────────────────
echo ""
echo "=== gen_manifests.sh ==="
# 4. Without --write, no files are created
t=$(tmpdir)
mkdir -p "$t/plugins/my-plugin/skills/hello"
echo "---" > "$t/plugins/my-plugin/skills/hello/SKILL.md"
"$SCRIPTS_DIR/gen_manifests.sh" "$t" --marketplace-name my-marketplace >/dev/null
assert_file_absent "dry run: .claude-plugin/marketplace.json not written" "$t/.claude-plugin/marketplace.json"
assert_file_absent "dry run: plugin.json not written" "$t/plugins/my-plugin/plugin.json"
rm -rf "$t"
# 5. With --write, creates plugin.json in both locations
t=$(tmpdir)
mkdir -p "$t/plugins/my-plugin/skills/hello"
echo "---" > "$t/plugins/my-plugin/skills/hello/SKILL.md"
"$SCRIPTS_DIR/gen_manifests.sh" "$t" --marketplace-name my-marketplace --write >/dev/null
assert_file_exists "--write: plugin root plugin.json created" "$t/plugins/my-plugin/plugin.json"
assert_file_exists "--write: .claude-plugin/plugin.json created" "$t/plugins/my-plugin/.claude-plugin/plugin.json"
rm -rf "$t"
# 6. Reserved name exits 1
t=$(tmpdir)
mkdir -p "$t/plugins/my-plugin"
set +e
out=$("$SCRIPTS_DIR/gen_manifests.sh" "$t" --marketplace-name claude-plugins-official 2>&1)
code=$?
set -e
assert_exit "reserved name: exit 1" 1 "$code"
assert_contains "reserved name: error message" "reserved" "$out"
rm -rf "$t"
# ── validate.sh tests ────────────────────────────────────────────────────────
echo ""
echo "=== validate.sh ==="
# 7. Invalid JSON exits 1
t=$(tmpdir)
mkdir -p "$t/.claude-plugin"
echo "not json" > "$t/.claude-plugin/marketplace.json"
set +e
out=$("$SCRIPTS_DIR/validate.sh" "$t" 2>&1)
code=$?
set -e
assert_exit "invalid JSON: exit 1" 1 "$code"
assert_contains "invalid JSON: error message" "ERROR" "$out"
rm -rf "$t"
# 8. Reserved marketplace name exits 1
t=$(tmpdir)
mkdir -p "$t/.claude-plugin"
printf '{"name":"claude-plugins-official","plugins":[]}\n' > "$t/.claude-plugin/marketplace.json"
set +e
out=$("$SCRIPTS_DIR/validate.sh" "$t" 2>&1)
code=$?
set -e
assert_exit "reserved marketplace name: exit 1" 1 "$code"
assert_contains "reserved marketplace name: error message" "reserved" "$out"
rm -rf "$t"
# 9. ../ in plugin file exits 1
t=$(tmpdir)
mkdir -p "$t/.claude-plugin" "$t/plugins/my-plugin/skills/hello"
printf '{"name":"my-marketplace","plugins":[{"name":"my-plugin","source":"./plugins/my-plugin"}]}\n' \
> "$t/.claude-plugin/marketplace.json"
printf -- '---\ndescription: test\n---\nSee ../shared.md\n' \
> "$t/plugins/my-plugin/skills/hello/SKILL.md"
set +e
out=$("$SCRIPTS_DIR/validate.sh" "$t" 2>&1)
code=$?
set -e
assert_exit "..// in plugin: exit 1" 1 "$code"
assert_contains "..// in plugin: error message" "../" "$out"
rm -rf "$t"
# 10. No marketplace.json warns but exits 0
t=$(tmpdir)
set +e
out=$("$SCRIPTS_DIR/validate.sh" "$t" 2>&1)
code=$?
set -e
assert_exit "no marketplace.json: exit 0" 0 "$code"
assert_contains "no marketplace.json: warning emitted" "WARN" "$out"
rm -rf "$t"
# ── summary ──────────────────────────────────────────────────────────────────
echo ""
echo "Results: $PASS passed, $FAIL failed"
[[ $FAIL -eq 0 ]]

118
plugins/kyberforge/skills/plugin-create/evals/eval.yaml Normal file
View File

@@ -0,0 +1,118 @@
skill_name: plugin-create
trigger_tests:
- id: explicit-create-named-plugin
name: "Explicit trigger — create named plugin"
query: "create a new plugin called security-tools"
should_trigger: true
- id: explicit-scaffold-plugin
name: "Explicit trigger — scaffold plugin"
query: "scaffold a plugin called developer-tools"
should_trigger: true
- id: implicit-add-marketplace-plugin
name: "Implicit trigger — add plugin to marketplace"
query: "add a new plugin to the marketplace for developer tools"
should_trigger: true
- id: implicit-package-skills-into-plugin
name: "Implicit trigger — package skills into a plugin"
query: "I want to package up my skills into a distributable plugin"
should_trigger: true
- id: negative-validate-marketplace
name: "Negative trigger — validate marketplace.json (routes to /marketplace-architect)"
query: "validate my marketplace.json"
should_trigger: false
- id: negative-write-skill
name: "Negative trigger — write a skill (routes to /write-skill)"
query: "write a skill for code review"
should_trigger: false
- id: negative-adopt-external-plugin
name: "Negative trigger — adopt external plugin (routes to /marketplace-architect)"
query: "adopt this external plugin into my marketplace"
should_trigger: false
output_tests:
- id: gate-a-shows-plan
name: "Gate A presents file list and marketplace entry before writing"
type: deterministic
prompt: >
Create a new plugin called marketplace-tools, description: 'Tools for managing the
holocron marketplace', author: Jane Doe, email: jane@example.com,
URL: https://github.com/jane
expected_output: >
The skill presents a plan listing all files that will be written under
plugins/marketplace-tools/, shows the new marketplace.json entry, and
explicitly asks for approval before writing any file.
assertions:
- "Output lists files to be written under plugins/marketplace-tools/"
- "Output includes a marketplace.json entry with name 'marketplace-tools'"
- "Output explicitly asks for approval and does not proceed without it"
- "Output does not contain any written file confirmation before approval is given"
- id: reserved-name-rejected
name: "Reserved plugin name is rejected before Gate A"
type: deterministic
prompt: "Create a new plugin called claude-tools"
expected_output: >
The skill halts before presenting Gate A, reports that 'claude-tools' matches
the reserved name pattern 'claude-*', and asks the user to provide a different name.
assertions:
- "Output does not present Gate A or a file list"
- "Output identifies 'claude-tools' as a reserved name"
- "Output asks the user to provide a replacement name before continuing"
- id: no-skill-stub-generated
name: "No skill stub generated — skills/ is empty with README only"
type: deterministic
prompt: >
Create a new plugin called marketplace-tools, description: 'Tools for managing the
holocron marketplace', author: Jane Doe, email: jane@example.com,
URL: https://github.com/jane. Approve Gate A and Gate B.
expected_output: >
The plugin scaffold does not include any SKILL.md file. The skills/ directory
is present with a README only. The skill directs the user to /write-skill
to add skill content.
assertions:
- "Output does not include creation of any SKILL.md file"
- "Output includes a skills/ directory entry with README.md only"
- "Output mentions /write-skill for adding skill content"
- id: handoff-message-present
name: "Hand-off message directs user to /marketplace-architect"
type: deterministic
prompt: >
Create a new plugin called marketplace-tools, description: 'Tools for managing the
holocron marketplace', author: Jane Doe, email: jane@example.com,
URL: https://github.com/jane. Approve Gate A and Gate B.
expected_output: >
After writing files and running validation, the skill prints a hand-off message
that names the created plugin and instructs the user to run /marketplace-architect.
assertions:
- "Output contains the string '/marketplace-architect'"
- "Output includes the plugin name 'marketplace-tools' in the hand-off message"
- "Output does not auto-invoke /marketplace-architect itself"
- id: full-flow-quality
name: "Full flow — correct sequencing, substitution, and hand-off"
type: llm-rubric
prompt: >
Create a new plugin called developer-tools with description 'Developer productivity
toolkit', author: Alex Smith, email: alex@example.com, URL: https://github.com/alex.
Approve Gate A and Gate B.
expected_output: >
The skill executes the full create-plugin flow: collects all five inputs one at a
time, presents Gate A with a file plan, copies and substitutes the template, presents
Gate B with full file contents, writes files after Gate B approval, updates
marketplace.json, runs validation, and prints a hand-off message.
assertions:
- "All five inputs (plugin name, description, author name, email, URL) were collected individually before Gate A was presented"
- "Gate A clearly listed all files to be written and the new marketplace.json entry, and waited for explicit approval"
- "Gate B showed the full substituted content of every file before writing"
- "None of the strings PLUGIN_NAME, PLUGIN_DESCRIPTION, AUTHOR_NAME, AUTHOR_EMAIL, or AUTHOR_URL appear in the written file output"
- "The validation step either ran claude plugin validate . and reported output, or explicitly stated it was unavailable"
- "The final message directed the user to run /marketplace-architect and included the plugin name 'developer-tools'"

85
plugins/kyberforge/skills/write-eval/evals/eval.yaml Normal file
View File

@@ -0,0 +1,85 @@
skill_name: write-eval
trigger_tests:
- id: explicit-trigger-write-evals
name: "Explicit trigger — write evals"
query: "write evals for this skill"
should_trigger: true
- id: explicit-trigger-create-eval-yaml
name: "Explicit trigger — create eval.yaml"
query: "create eval.yaml for the tdd skill"
should_trigger: true
- id: implicit-trigger-test-coverage
name: "Implicit trigger — test coverage request"
query: "I need test coverage for the grill-me skill"
should_trigger: true
- id: negative-trigger-run-evals
name: "Negative — run evals (runner concern, not writer)"
query: "run my evals"
should_trigger: false
- id: negative-trigger-code-unit-tests
name: "Negative — unit tests for application code"
query: "write unit tests for my Python file"
should_trigger: false
- id: negative-trigger-debug-failing-eval
name: "Negative — debug failing eval"
query: "my eval is failing, help me debug it"
should_trigger: false
output_tests:
- id: deterministic-correct-output-path
name: "Deterministic — eval.yaml written to correct path"
type: deterministic
prompt: "write evals for the tdd skill"
expected_output: "eval.yaml written to .agents/evals/implement/tdd/eval.yaml containing skill_name: tdd"
assertions:
- "Output references the path .agents/evals/implement/tdd/eval.yaml"
- "Output file contains 'skill_name: tdd'"
- id: deterministic-all-five-types-present
name: "Deterministic — eval.yaml contains all five required test types"
type: deterministic
prompt: "create eval.yaml for the grill-me skill"
expected_output: "eval.yaml contains trigger_tests and output_tests sections with all five required test types represented"
assertions:
- "Output contains 'trigger_tests:'"
- "Output contains 'output_tests:'"
- "Output contains at least one entry with 'should_trigger: true'"
- "Output contains at least one entry with 'should_trigger: false'"
- "Output contains at least one entry with 'type: deterministic'"
- "Output contains at least one entry with 'type: llm-rubric'"
- id: deterministic-plan-shown-before-write
name: "Deterministic — test plan presented before file is written"
type: deterministic
prompt: "write evals for the diagnose skill"
expected_output: "Skill presents each proposed test case with its id, type, and query before writing any file, then requests confirmation"
assertions:
- "Response presents each proposed test case individually — showing at minimum the query and test type — before any file is written"
- "Response requests confirmation before proceeding to write"
- id: deterministic-merge-conflict-flagged
name: "Deterministic — conflict flagged in plan on re-run with existing eval"
type: deterministic
prompt: "write evals for the tdd skill — eval.yaml already exists at .agents/evals/implement/tdd/eval.yaml with a test case id 'explicit-trigger-basic'"
expected_output: "Skill identifies the existing eval.yaml, classifies the conflicting case as CONFLICT, and does not write until the user resolves it"
assertions:
- "Response indicates eval.yaml already exists at the target path"
- "Response labels the conflicting test case as CONFLICT or equivalent"
- "Response does not write the file before the user resolves the conflict"
- id: llm-rubric-assertion-quality
name: "LLM rubric — assertions are specific and verifiable"
type: llm-rubric
prompt: "write evals for the write-skill skill"
expected_output: "eval.yaml contains high-quality assertions that are specific, observable, and not vague"
assertions:
- "All assertions describe observable, verifiable conditions — not vague quality claims like 'output is good' or 'the response is helpful'"
- "Trigger test queries reflect realistic user phrasings, not just the exact skill description verbatim"
- "Negative trigger tests target adjacent tasks that share surface-level similarity with the skill's trigger"
- "Deterministic assertions are machine-checkable without LLM inference — presence of strings, path patterns, required sections"

69
plugins/kyberforge/skills/write-skill/evals/eval.yaml Normal file
View File

@@ -0,0 +1,69 @@
skill_name: write-skill
trigger_tests:
- id: explicit-trigger-new-skill
name: Explicit — new skill phrase
query: "Write a new skill for handling database migrations"
should_trigger: true
- id: implicit-trigger-no-phrase
name: Implicit — no trigger phrase
query: "I want to add a skill that automates our deploy process"
should_trigger: true
- id: implicit-trigger-conversion
name: Implicit — placeholder conversion
query: "The grill-me skill is a Pocock placeholder, can you convert it to our standard?"
should_trigger: true
- id: negative-trigger-upgrade
name: Negative — existing skill fix
query: "The tdd skill is producing wrong output, fix it"
should_trigger: false
- id: negative-trigger-code-refactor
name: Negative — code refactor
query: "Refactor this module to use the new API client"
should_trigger: false
- id: negative-trigger-write-eval
name: Negative — eval request
query: "Write evals for the diagnose skill"
should_trigger: false
output_tests:
- id: output-has-all-sections
name: All 8 body sections present in order
type: deterministic
prompt: "Write a new skill for linting markdown files, category: implement"
expected_output: A complete SKILL.md containing all 8 required body sections in the prescribed order.
assertions:
- "Output contains '## Role'"
- "Output contains '## When to use / When not to use'"
- "Output contains '## Required inputs'"
- "Output contains '## Constraints'"
- "Output contains '## Process'"
- "Output contains '## Output format'"
- "Output contains '## Failure handling'"
- "Output contains '## Self-check'"
- "Sections appear in this order: ## Role, ## When to use / When not to use, ## Required inputs, ## Constraints, ## Process, ## Output format, ## Failure handling, ## Self-check"
- id: output-path-correct
name: Output path and frontmatter fields correct
type: deterministic
prompt: "Write a new skill for sending Slack notifications on deploy events, category: deploy"
expected_output: A SKILL.md with correct output path stated and all required frontmatter fields present.
assertions:
- "Output contains '.agents/skills/' in the stated output path"
- "Output contains 'metadata:' and 'category:' in frontmatter"
- "Output contains 'version:'"
- "Output contains 'when:'"
- id: output-trigger-tested-before-body
name: Trigger description tested before body content written
type: llm-rubric
prompt: "Write a new skill for summarising pull request diffs"
expected_output: The skill presents a trigger description and tests it against at least 3 cases (explicit, implicit, negative) before proposing or writing any body section content.
assertions:
- "The skill proposes a trigger description and explicitly tests it against an explicit query, an implicit query, and a negative query before writing any body section"
- "The skill walks through each body section individually and seeks confirmation before writing the file"