Defame1297/holocron
0
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases 3 Wiki Activity

chore: move skills and evals to plugins/bin, remove legacy root configs

Browse Source 
Skills and evals migrated from .agents/ to plugins/bin/ plugin directory.
Remove .mcp.json, provider-manifest.sh, and skills-lock.json legacy artifacts.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Defame1297
2026-06-25 19:18:41 +00:00
parent 2bf0365aa5
commit e50c98f722
48 changed files with 77 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files

84
plugins/bin/evals/cross-cutting/gitleaks/eval.yaml Normal file
View File

@@ -0,0 +1,84 @@
skill_name: gitleaks
trigger_tests:
- id: explicit-trigger-install
name: Explicit trigger — install and configure
query: "set up gitleaks in this repo"
should_trigger: true
- id: explicit-trigger-update-hook
name: Explicit trigger — update hook
query: "update the gitleaks pre-commit hook"
should_trigger: true
- id: implicit-trigger-false-positive
name: Implicit trigger — suppress false positive in pre-commit hook
query: "my pre-commit hook keeps blocking commits because it thinks my test fixture has an API key, how do I suppress it?"
should_trigger: true
- id: implicit-trigger-scan-history
name: Implicit trigger — audit repo history for secrets
query: "I want to scan my entire git history to make sure no credentials were ever committed"
should_trigger: true
- id: negative-trigger-security-review
name: Negative trigger — general code security review
query: "do a security review of this pull request"
should_trigger: false
- id: negative-trigger-ci-scan
name: Negative trigger — CI pipeline secret scanning
query: "add secret scanning to my GitHub Actions CI pipeline"
should_trigger: false
- id: negative-trigger-other-tool
name: Negative trigger — different secret scanning tool
query: "trufflehog is flagging false positives in my repo, how do I configure it?"
should_trigger: false
output_tests:
- id: output-states-intent-before-running
name: Deterministic — states intent and waits for confirmation before running setup script
type: deterministic
prompt: "set up gitleaks in /home/user/myproject"
expected_output: >
The skill states what it is about to do (run setup-gitleaks.sh, which will modify
.git/hooks/pre-commit and seed .gitleaks.toml) and asks for confirmation before
executing the script. The confirmation request appears before any Bash command
that runs the script.
assertions:
- "Output contains 'setup-gitleaks.sh' before any Bash tool invocation that runs it"
- "Output contains a confirmation prompt (e.g. 'Proceed?') before running the setup script"
- "Output does not run the setup script without an explicit confirmation step"
- id: output-redacts-secret-values
name: Deterministic — redacts secret value and prescribes rotation before history rewrite
type: deterministic
prompt: "gitleaks found an AWS secret key on line 42 of config/settings.py — it says the value is AKIAIOSFODNN7EXAMPLE. What do I do?"
expected_output: >
The skill does not echo or repeat the secret value. It references the finding by
rule ID, file path, and line number only. It states that the secret must be rotated
immediately. It mentions git filter-repo (not git filter-branch) for history rewrite
and asks for user confirmation before running the rewrite.
assertions:
- "Output does not contain the literal string 'AKIAIOSFODNN7EXAMPLE'"
- "Output mentions rotating or revoking the credential as the first action"
- "Output references 'git filter-repo' for history rewrite, not 'git filter-branch'"
- "Output states that user confirmation is required before running the history rewrite"
- id: output-quality-allowlist-guidance
name: LLM-rubric — allowlist guidance is correct, version-aware, and minimal
type: llm-rubric
prompt: "gitleaks keeps flagging my docs/research/ directory as containing secrets, how do I suppress it?"
expected_output: >
High-quality output checks the installed gitleaks version before prescribing any
TOML syntax, recommends a path-based allowlist entry in .gitleaks.toml (not a
.gitleaksignore fingerprint), uses the correct TOML syntax for the detected version,
adds only the minimal allowlist entry needed for the identified false positive, and
includes a verification step (re-run gitleaks dir -v or gitleaks dir --log-level debug)
after making the change.
assertions:
- "Output checks or asks about the gitleaks version before writing TOML syntax"
- "Output recommends a path-based allowlist entry in .gitleaks.toml rather than .gitleaksignore"
- "Output includes a command to verify the suppression works after the change"
- "Output explains why .gitleaksignore fingerprints are fragile (line numbers shift)"

97
plugins/bin/evals/cross-cutting/neuledge-context/eval.yaml Normal file
View File

@@ -0,0 +1,97 @@
skill_name: neuledge-context
trigger_tests:
- id: explicit-install-register
name: "Explicit trigger — install and register"
query: "install @neuledge/context and register it as an MCP server in Claude Code"
should_trigger: true
- id: explicit-package-management
name: "Explicit trigger — package management"
query: "install the react documentation package using neuledge context"
should_trigger: true
- id: implicit-offline-docs
name: "Implicit trigger — offline docs for AI agent"
query: "I need React and Next.js docs available to my AI agent without web searches"
should_trigger: true
- id: negative-different-mcp
name: "Negative — different MCP server"
query: "Add the Gitea MCP server to Claude Code"
should_trigger: false
- id: negative-query-existing
name: "Negative — querying an already-running server"
query: "How do I query React docs using the context server that's already running?"
should_trigger: false
- id: negative-cursor-setup
name: "Negative — different provider"
query: "Set up context serve for Cursor"
should_trigger: false
output_tests:
- id: install-script-announced
name: "Install — script announced before running, version verified after"
type: deterministic
prompt: "install @neuledge/context"
expected_output: >
The skill announces that it will run scripts/setup-neuledge-context.sh before executing it,
then verifies the installation by running `context --version`.
assertions:
- "Output mentions 'scripts/setup-neuledge-context.sh' before any install command is run"
- "Output includes a `context --version` call after the install step"
- "Output does not contain `npm install -g @neuledge/context@latest` (no floating @latest)"
- id: mcp-list-before-add
name: "MCP registration — list checked before add, skipped if present"
type: deterministic
prompt: "register @neuledge/context as a Claude Code MCP server"
expected_output: >
The skill runs `claude mcp list` and checks for an existing 'context' entry before
running `claude mcp add`. If already registered, the add step is skipped.
assertions:
- "Output includes `claude mcp list` before `claude mcp add context`"
- "Output states that registration is skipped when the server is already present"
- "The `claude mcp add` command uses stdio form: `claude mcp add context -- context serve`"
- id: auth-chmod-paired
name: "Auth — secure-context-config.sh run immediately after auth add"
type: deterministic
prompt: "add auth credentials for docs.example.com to neuledge context"
expected_output: >
The skill runs `context auth add docs.example.com` with an environment variable reference
for the credential, then immediately runs scripts/secure-context-config.sh.
No credential value appears in the output.
assertions:
- "Output references an environment variable (e.g. $TOKEN) rather than a literal credential value"
- "Output runs `scripts/secure-context-config.sh` in the same step as or immediately after `context auth add`"
- "No bearer token, cookie value, or other credential string appears in the output"
- id: git-check-url-source
name: "context add — git prerequisite checked for URL sources only"
type: deterministic
prompt: "add documentation from https://github.com/prisma/prisma using context add"
expected_output: >
Before running `context add`, the skill checks `git --version` because the source is a
GitHub URL. The check is present for URL/repo sources and absent for local .db file paths.
assertions:
- "Output includes `git --version` before the `context add https://github.com/...` command"
- "If given a local .db file path instead, the git check is absent"
- id: install-flow-quality
name: "Full install + register flow quality"
type: llm-rubric
prompt: "install @neuledge/context and set it up as my Claude Code MCP server"
expected_output: >
A complete, ordered install-then-register flow: (1) announce the install script,
(2) run setup-neuledge-context.sh, (3) verify with context --version,
(4) check claude mcp list, (5) run claude mcp add if not already registered,
(6) confirm with claude mcp list. Steps are in the correct order with verification
between install and registration.
assertions:
- "Install step comes before MCP registration step"
- "A verification command (context --version) appears between install and registration"
- "The output would leave a user with a working @neuledge/context MCP server in Claude Code"
- "No step is skipped without an explanation of why it was skipped"

61
plugins/bin/evals/implement/write-docs/eval.yaml Normal file
View File

@@ -0,0 +1,61 @@
skill_name: write-docs
trigger_tests:
- id: explicit-trigger-document-module
name: "Explicit trigger — document a script"
query: "Write documentation for the install.sh script"
should_trigger: true
- id: explicit-trigger-create-docs
name: "Explicit trigger — create docs for a feature"
query: "Create docs for this feature"
should_trigger: true
- id: implicit-trigger-readme-update
name: "Implicit trigger — outdated README section, no trigger phrase"
query: "We need to update the README section for the auth module, the current one is outdated"
should_trigger: true
- id: negative-trigger-prd
name: "Negative — PRD request should route to to-prd"
query: "Write a PRD for the new logging feature"
should_trigger: false
- id: negative-trigger-write-skill
name: "Negative — skill authoring request should route to write-skill"
query: "Write a skill for generating documentation automatically"
should_trigger: false
- id: negative-trigger-skill-file
name: "Negative — SKILL.md update (skill files are self-describing)"
query: "Document how the write-docs skill works by updating its SKILL.md"
should_trigger: false
output_tests:
- id: output-proposes-files-before-reading
name: "Deterministic — candidates proposed or approval sought before reading files"
type: deterministic
prompt: "Write documentation for the config module"
expected_output: "Skill proposes candidate files or asks the user to name specific files before reading any file content"
assertions:
- "Response proposes candidate file paths or asks the user to confirm which files to read before showing any extracted content"
- "Response does not display extracted code content or API surface without first receiving file approval"
- id: output-gap-check-present
name: "Deterministic — gap check step present before drafting"
type: deterministic
prompt: "Write documentation for the install.sh script, audience: developer"
expected_output: "Skill presents extracted behaviour to the user and asks them to fill gaps before drafting any section"
assertions:
- "Response includes a gap check step that presents extracted behaviour and asks what the code does not explain"
- "Response does not skip directly to a drafted documentation section without presenting extracted content first"
- id: output-never-invents-behaviour
name: "LLM rubric — no invented behaviour, all claims sourced"
type: llm-rubric
prompt: "Document the src/config.py file for internal developers"
expected_output: "Documentation where every claim is attributed to code content or explicit user input, with no invented explanations, assumptions about intent, or unverifiable behaviour claims."
assertions:
- "The skill explicitly derives each documented claim from a named source — a code line, spec section, or user statement — and does not add claims without attribution"
- "The skill does not include descriptions of caller intent, design rationale, or future behaviour that are not present in the source material"
- "If a behaviour is undocumentable (internal detail with no public spec), the skill notes it as out-of-scope rather than inventing an explanation"

149
plugins/bin/evals/research/research/eval.yaml Normal file
View File

@@ -0,0 +1,149 @@
skill_name: research
trigger_tests:
- id: explicit-basic
name: "Explicit trigger — basic invocation with topic and path"
query: "/research stripe .agents/skills/stripe/references/"
should_trigger: true
- id: explicit-with-urls
name: "Explicit trigger — topic, path, and starting URLs provided"
query: "Research the FastMCP library and put refs in .agents/skills/mcp/references/ — start here: https://fastmcp.dev/llms.txt"
should_trigger: true
- id: implicit-pre-skill
name: "Implicit trigger — looking up docs before writing a skill"
query: "Before we write the GitHub Actions skill, can you look up the API and put the docs somewhere useful?"
should_trigger: true
- id: negative-write-docs
name: "Negative — documenting existing code goes to write-docs"
query: "Document the auth module for me"
should_trigger: false
- id: negative-diagnose
name: "Negative — debugging investigation goes to diagnose"
query: "Research why these integration tests are failing"
should_trigger: false
- id: negative-neuledge
name: "Negative — MCP server setup goes to neuledge-context"
query: "Install the neuledge context server and set it up"
should_trigger: false
- id: negative-context7-direct-question
name: "Negative — direct doc question goes to context7-mcp, not research"
query: "What are the Next.js middleware options?"
should_trigger: false
- id: negative-starting-urls-skips-context7
name: "Explicit starting URLs — Context7 step skipped, user source respected"
query: "Research fastapi at docs/refs/fastapi/ — start here: https://fastapi.tiangolo.com/llms.txt"
should_trigger: true
- id: implicit-library-context7-attempted
name: "Implicit library research — skill triggers and should attempt Context7 before websearch"
query: "Before we write the Redis skill, grab the docs and put them in .agents/skills/redis/references/"
should_trigger: true
- id: concept-topic-still-triggers-research
name: "Concept topic — research triggers but Context7 step is skipped (not a named library)"
query: "/research 'JWT authentication concepts' docs/refs/jwt/"
should_trigger: true
output_tests:
- id: sources-always-written
name: "sources.md is always produced at the output path"
type: deterministic
prompt: "/research redis .agents/skills/redis/references/"
expected_output: >-
A sources.md file written to .agents/skills/redis/references/sources.md
containing at least one source section with a URL and status field.
assertions:
- "Output path .agents/skills/redis/references/sources.md is created"
- "sources.md contains at least one ## <slug> section"
- "Each section in sources.md includes a URL: line and a Status: line"
- id: topic-frontmatter
name: "Every topic file has required frontmatter fields"
type: deterministic
prompt: "/research stripe .agents/skills/stripe/references/"
expected_output: >-
One or more topic .md files written to .agents/skills/stripe/references/,
each with YAML frontmatter containing topic: and source_keys: fields.
assertions:
- "Every .md file except sources.md contains a YAML frontmatter block delimited by ---"
- "Every topic file frontmatter contains a topic: field"
- "Every topic file frontmatter contains a source_keys: field with at least one entry"
- id: no-empty-files
name: "No topic files written when no content is found"
type: deterministic
prompt: "/research xyzzy-nonexistent-tool-404 /tmp/refs-test/"
expected_output: >-
No topic .md files written. Skill reports what was searched and asks user
to provide starting URLs. sources.md may still be written if sources were attempted.
assertions:
- "Skill does not write any topic .md files to the output path"
- "Skill reports the search terms or URLs it tried"
- "Skill asks the user to provide starting URLs or confirm the topic name"
- id: output-path-respected
name: "All files written inside the provided output path only"
type: deterministic
prompt: "/research openai .agents/skills/openai/references/"
expected_output: >-
All written files are under .agents/skills/openai/references/. No files
created outside that directory.
assertions:
- "Every file path in the write operations starts with .agents/skills/openai/references/"
- "No files are written to the current working directory root or any other path"
- id: context7-resolves-source-recorded
name: "Context7 source recorded in sources.md with context7: URL when library resolves"
type: deterministic
prompt: "/research next.js docs/refs/nextjs/"
expected_output: >-
sources.md contains at least one entry with a URL starting with context7:
(e.g. context7:/vercel/next.js) and a slug matching context7-*.
assertions:
- "sources.md contains a ## context7-* section"
- "The URL: line for that section starts with context7:"
- "At least one topic file lists the context7-* slug in its source_keys frontmatter"
- id: context7-fallback-to-web
name: "Fallback to websearch when Context7 does not resolve the library"
type: deterministic
prompt: "/research xyzzy-internal-tool-abc docs/refs/xyzzy/"
expected_output: >-
No context7-* source in sources.md. Skill falls back to websearch.
If no web results found, skill reports failure and does not write topic files.
assertions:
- "sources.md does not contain a ## context7-* section"
- "Skill either writes topic files sourced from web URLs or reports no sources found"
- id: context7-skipped-for-non-library
name: "Context7 step skipped when topic is not a library/framework/API"
type: deterministic
prompt: "/research 'JWT authentication concepts' docs/refs/jwt/"
expected_output: >-
No resolve-library-id call made. Skill proceeds directly to websearch.
sources.md contains only web-sourced slugs.
assertions:
- "sources.md does not contain a ## context7-* section"
- "Topic files are sourced from web URLs, not Context7"
- id: content-quality
name: "Extracted content is relevant, concise, and well-organised"
type: llm-rubric
prompt: "/research fastapi .agents/skills/fastapi/references/"
expected_output: >-
A set of topic files covering FastAPI concepts relevant to skill authoring:
installation, API reference, configuration, and examples at minimum. Content
is concise summaries, not verbatim page dumps. sources.md accurately maps
each source to the files it contributed to.
assertions:
- "Topic files cover concepts relevant to FastAPI (routing, request handling, dependency injection, or similar) — not generic web framework content"
- "Body prose in each topic file is a synthesis or extraction, not a verbatim copy of the source page"
- "source_keys in each topic file match slugs present in sources.md"
- "sources.md contributing files list accurately reflects which topic files each source contributed to"