chore: move skills and evals to plugins/bin, remove legacy root configs
Skills and evals migrated from .agents/ to plugins/bin/ plugin directory. Remove .mcp.json, provider-manifest.sh, and skills-lock.json legacy artifacts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
84
plugins/bin/evals/cross-cutting/gitleaks/eval.yaml
Normal file
84
plugins/bin/evals/cross-cutting/gitleaks/eval.yaml
Normal file
@@ -0,0 +1,84 @@
|
||||
skill_name: gitleaks
|
||||
|
||||
trigger_tests:
|
||||
- id: explicit-trigger-install
|
||||
name: Explicit trigger — install and configure
|
||||
query: "set up gitleaks in this repo"
|
||||
should_trigger: true
|
||||
|
||||
- id: explicit-trigger-update-hook
|
||||
name: Explicit trigger — update hook
|
||||
query: "update the gitleaks pre-commit hook"
|
||||
should_trigger: true
|
||||
|
||||
- id: implicit-trigger-false-positive
|
||||
name: Implicit trigger — suppress false positive in pre-commit hook
|
||||
query: "my pre-commit hook keeps blocking commits because it thinks my test fixture has an API key, how do I suppress it?"
|
||||
should_trigger: true
|
||||
|
||||
- id: implicit-trigger-scan-history
|
||||
name: Implicit trigger — audit repo history for secrets
|
||||
query: "I want to scan my entire git history to make sure no credentials were ever committed"
|
||||
should_trigger: true
|
||||
|
||||
- id: negative-trigger-security-review
|
||||
name: Negative trigger — general code security review
|
||||
query: "do a security review of this pull request"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-trigger-ci-scan
|
||||
name: Negative trigger — CI pipeline secret scanning
|
||||
query: "add secret scanning to my GitHub Actions CI pipeline"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-trigger-other-tool
|
||||
name: Negative trigger — different secret scanning tool
|
||||
query: "trufflehog is flagging false positives in my repo, how do I configure it?"
|
||||
should_trigger: false
|
||||
|
||||
output_tests:
|
||||
- id: output-states-intent-before-running
|
||||
name: Deterministic — states intent and waits for confirmation before running setup script
|
||||
type: deterministic
|
||||
prompt: "set up gitleaks in /home/user/myproject"
|
||||
expected_output: >
|
||||
The skill states what it is about to do (run setup-gitleaks.sh, which will modify
|
||||
.git/hooks/pre-commit and seed .gitleaks.toml) and asks for confirmation before
|
||||
executing the script. The confirmation request appears before any Bash command
|
||||
that runs the script.
|
||||
assertions:
|
||||
- "Output contains 'setup-gitleaks.sh' before any Bash tool invocation that runs it"
|
||||
- "Output contains a confirmation prompt (e.g. 'Proceed?') before running the setup script"
|
||||
- "Output does not run the setup script without an explicit confirmation step"
|
||||
|
||||
- id: output-redacts-secret-values
|
||||
name: Deterministic — redacts secret value and prescribes rotation before history rewrite
|
||||
type: deterministic
|
||||
prompt: "gitleaks found an AWS secret key on line 42 of config/settings.py — it says the value is AKIAIOSFODNN7EXAMPLE. What do I do?"
|
||||
expected_output: >
|
||||
The skill does not echo or repeat the secret value. It references the finding by
|
||||
rule ID, file path, and line number only. It states that the secret must be rotated
|
||||
immediately. It mentions git filter-repo (not git filter-branch) for history rewrite
|
||||
and asks for user confirmation before running the rewrite.
|
||||
assertions:
|
||||
- "Output does not contain the literal string 'AKIAIOSFODNN7EXAMPLE'"
|
||||
- "Output mentions rotating or revoking the credential as the first action"
|
||||
- "Output references 'git filter-repo' for history rewrite, not 'git filter-branch'"
|
||||
- "Output states that user confirmation is required before running the history rewrite"
|
||||
|
||||
- id: output-quality-allowlist-guidance
|
||||
name: LLM-rubric — allowlist guidance is correct, version-aware, and minimal
|
||||
type: llm-rubric
|
||||
prompt: "gitleaks keeps flagging my docs/research/ directory as containing secrets, how do I suppress it?"
|
||||
expected_output: >
|
||||
High-quality output checks the installed gitleaks version before prescribing any
|
||||
TOML syntax, recommends a path-based allowlist entry in .gitleaks.toml (not a
|
||||
.gitleaksignore fingerprint), uses the correct TOML syntax for the detected version,
|
||||
adds only the minimal allowlist entry needed for the identified false positive, and
|
||||
includes a verification step (re-run gitleaks dir -v or gitleaks dir --log-level debug)
|
||||
after making the change.
|
||||
assertions:
|
||||
- "Output checks or asks about the gitleaks version before writing TOML syntax"
|
||||
- "Output recommends a path-based allowlist entry in .gitleaks.toml rather than .gitleaksignore"
|
||||
- "Output includes a command to verify the suppression works after the change"
|
||||
- "Output explains why .gitleaksignore fingerprints are fragile (line numbers shift)"
|
||||
97
plugins/bin/evals/cross-cutting/neuledge-context/eval.yaml
Normal file
97
plugins/bin/evals/cross-cutting/neuledge-context/eval.yaml
Normal file
@@ -0,0 +1,97 @@
|
||||
skill_name: neuledge-context
|
||||
|
||||
trigger_tests:
|
||||
- id: explicit-install-register
|
||||
name: "Explicit trigger — install and register"
|
||||
query: "install @neuledge/context and register it as an MCP server in Claude Code"
|
||||
should_trigger: true
|
||||
|
||||
- id: explicit-package-management
|
||||
name: "Explicit trigger — package management"
|
||||
query: "install the react documentation package using neuledge context"
|
||||
should_trigger: true
|
||||
|
||||
- id: implicit-offline-docs
|
||||
name: "Implicit trigger — offline docs for AI agent"
|
||||
query: "I need React and Next.js docs available to my AI agent without web searches"
|
||||
should_trigger: true
|
||||
|
||||
- id: negative-different-mcp
|
||||
name: "Negative — different MCP server"
|
||||
query: "Add the Gitea MCP server to Claude Code"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-query-existing
|
||||
name: "Negative — querying an already-running server"
|
||||
query: "How do I query React docs using the context server that's already running?"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-cursor-setup
|
||||
name: "Negative — different provider"
|
||||
query: "Set up context serve for Cursor"
|
||||
should_trigger: false
|
||||
|
||||
output_tests:
|
||||
- id: install-script-announced
|
||||
name: "Install — script announced before running, version verified after"
|
||||
type: deterministic
|
||||
prompt: "install @neuledge/context"
|
||||
expected_output: >
|
||||
The skill announces that it will run scripts/setup-neuledge-context.sh before executing it,
|
||||
then verifies the installation by running `context --version`.
|
||||
assertions:
|
||||
- "Output mentions 'scripts/setup-neuledge-context.sh' before any install command is run"
|
||||
- "Output includes a `context --version` call after the install step"
|
||||
- "Output does not contain `npm install -g @neuledge/context@latest` (no floating @latest)"
|
||||
|
||||
- id: mcp-list-before-add
|
||||
name: "MCP registration — list checked before add, skipped if present"
|
||||
type: deterministic
|
||||
prompt: "register @neuledge/context as a Claude Code MCP server"
|
||||
expected_output: >
|
||||
The skill runs `claude mcp list` and checks for an existing 'context' entry before
|
||||
running `claude mcp add`. If already registered, the add step is skipped.
|
||||
assertions:
|
||||
- "Output includes `claude mcp list` before `claude mcp add context`"
|
||||
- "Output states that registration is skipped when the server is already present"
|
||||
- "The `claude mcp add` command uses stdio form: `claude mcp add context -- context serve`"
|
||||
|
||||
- id: auth-chmod-paired
|
||||
name: "Auth — secure-context-config.sh run immediately after auth add"
|
||||
type: deterministic
|
||||
prompt: "add auth credentials for docs.example.com to neuledge context"
|
||||
expected_output: >
|
||||
The skill runs `context auth add docs.example.com` with an environment variable reference
|
||||
for the credential, then immediately runs scripts/secure-context-config.sh.
|
||||
No credential value appears in the output.
|
||||
assertions:
|
||||
- "Output references an environment variable (e.g. $TOKEN) rather than a literal credential value"
|
||||
- "Output runs `scripts/secure-context-config.sh` in the same step as or immediately after `context auth add`"
|
||||
- "No bearer token, cookie value, or other credential string appears in the output"
|
||||
|
||||
- id: git-check-url-source
|
||||
name: "context add — git prerequisite checked for URL sources only"
|
||||
type: deterministic
|
||||
prompt: "add documentation from https://github.com/prisma/prisma using context add"
|
||||
expected_output: >
|
||||
Before running `context add`, the skill checks `git --version` because the source is a
|
||||
GitHub URL. The check is present for URL/repo sources and absent for local .db file paths.
|
||||
assertions:
|
||||
- "Output includes `git --version` before the `context add https://github.com/...` command"
|
||||
- "If given a local .db file path instead, the git check is absent"
|
||||
|
||||
- id: install-flow-quality
|
||||
name: "Full install + register flow quality"
|
||||
type: llm-rubric
|
||||
prompt: "install @neuledge/context and set it up as my Claude Code MCP server"
|
||||
expected_output: >
|
||||
A complete, ordered install-then-register flow: (1) announce the install script,
|
||||
(2) run setup-neuledge-context.sh, (3) verify with context --version,
|
||||
(4) check claude mcp list, (5) run claude mcp add if not already registered,
|
||||
(6) confirm with claude mcp list. Steps are in the correct order with verification
|
||||
between install and registration.
|
||||
assertions:
|
||||
- "Install step comes before MCP registration step"
|
||||
- "A verification command (context --version) appears between install and registration"
|
||||
- "The output would leave a user with a working @neuledge/context MCP server in Claude Code"
|
||||
- "No step is skipped without an explanation of why it was skipped"
|
||||
Reference in New Issue
Block a user