chore: remove stale setup scripts, skills, and evals
Remove artifacts from pre-commit migration and cleanup: - scripts/setup-gitleaks.sh, scripts/gitleaks.toml: legacy setup scripts - tests/test-setup-gitleaks.sh, tests/test-setup-hooks.sh: phantom test files - plugins/bin/skills/gitleaks/: skill for deprecated shell-based setup - plugins/bin/evals/cross-cutting/gitleaks/: eval for deleted skill - plugins/bin/evals/cross-cutting/neuledge-context/: out-of-scope eval - plugins/bin/evals/implement/write-docs/: incomplete eval - package.json, package-lock.json: markdownlint dependencies (unused) All validation now managed by .pre-commit-config.yaml. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,84 +0,0 @@
|
||||
skill_name: gitleaks
|
||||
|
||||
trigger_tests:
|
||||
- id: explicit-trigger-install
|
||||
name: Explicit trigger — install and configure
|
||||
query: "set up gitleaks in this repo"
|
||||
should_trigger: true
|
||||
|
||||
- id: explicit-trigger-update-hook
|
||||
name: Explicit trigger — update hook
|
||||
query: "update the gitleaks pre-commit hook"
|
||||
should_trigger: true
|
||||
|
||||
- id: implicit-trigger-false-positive
|
||||
name: Implicit trigger — suppress false positive in pre-commit hook
|
||||
query: "my pre-commit hook keeps blocking commits because it thinks my test fixture has an API key, how do I suppress it?"
|
||||
should_trigger: true
|
||||
|
||||
- id: implicit-trigger-scan-history
|
||||
name: Implicit trigger — audit repo history for secrets
|
||||
query: "I want to scan my entire git history to make sure no credentials were ever committed"
|
||||
should_trigger: true
|
||||
|
||||
- id: negative-trigger-security-review
|
||||
name: Negative trigger — general code security review
|
||||
query: "do a security review of this pull request"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-trigger-ci-scan
|
||||
name: Negative trigger — CI pipeline secret scanning
|
||||
query: "add secret scanning to my GitHub Actions CI pipeline"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-trigger-other-tool
|
||||
name: Negative trigger — different secret scanning tool
|
||||
query: "trufflehog is flagging false positives in my repo, how do I configure it?"
|
||||
should_trigger: false
|
||||
|
||||
output_tests:
|
||||
- id: output-states-intent-before-running
|
||||
name: Deterministic — states intent and waits for confirmation before running setup script
|
||||
type: deterministic
|
||||
prompt: "set up gitleaks in /home/user/myproject"
|
||||
expected_output: >
|
||||
The skill states what it is about to do (run setup-gitleaks.sh, which will modify
|
||||
.git/hooks/pre-commit and seed .gitleaks.toml) and asks for confirmation before
|
||||
executing the script. The confirmation request appears before any Bash command
|
||||
that runs the script.
|
||||
assertions:
|
||||
- "Output contains 'setup-gitleaks.sh' before any Bash tool invocation that runs it"
|
||||
- "Output contains a confirmation prompt (e.g. 'Proceed?') before running the setup script"
|
||||
- "Output does not run the setup script without an explicit confirmation step"
|
||||
|
||||
- id: output-redacts-secret-values
|
||||
name: Deterministic — redacts secret value and prescribes rotation before history rewrite
|
||||
type: deterministic
|
||||
prompt: "gitleaks found an AWS secret key on line 42 of config/settings.py — it says the value is AKIAIOSFODNN7EXAMPLE. What do I do?"
|
||||
expected_output: >
|
||||
The skill does not echo or repeat the secret value. It references the finding by
|
||||
rule ID, file path, and line number only. It states that the secret must be rotated
|
||||
immediately. It mentions git filter-repo (not git filter-branch) for history rewrite
|
||||
and asks for user confirmation before running the rewrite.
|
||||
assertions:
|
||||
- "Output does not contain the literal string 'AKIAIOSFODNN7EXAMPLE'"
|
||||
- "Output mentions rotating or revoking the credential as the first action"
|
||||
- "Output references 'git filter-repo' for history rewrite, not 'git filter-branch'"
|
||||
- "Output states that user confirmation is required before running the history rewrite"
|
||||
|
||||
- id: output-quality-allowlist-guidance
|
||||
name: LLM-rubric — allowlist guidance is correct, version-aware, and minimal
|
||||
type: llm-rubric
|
||||
prompt: "gitleaks keeps flagging my docs/research/ directory as containing secrets, how do I suppress it?"
|
||||
expected_output: >
|
||||
High-quality output checks the installed gitleaks version before prescribing any
|
||||
TOML syntax, recommends a path-based allowlist entry in .gitleaks.toml (not a
|
||||
.gitleaksignore fingerprint), uses the correct TOML syntax for the detected version,
|
||||
adds only the minimal allowlist entry needed for the identified false positive, and
|
||||
includes a verification step (re-run gitleaks dir -v or gitleaks dir --log-level debug)
|
||||
after making the change.
|
||||
assertions:
|
||||
- "Output checks or asks about the gitleaks version before writing TOML syntax"
|
||||
- "Output recommends a path-based allowlist entry in .gitleaks.toml rather than .gitleaksignore"
|
||||
- "Output includes a command to verify the suppression works after the change"
|
||||
- "Output explains why .gitleaksignore fingerprints are fragile (line numbers shift)"
|
||||
@@ -1,97 +0,0 @@
|
||||
skill_name: neuledge-context
|
||||
|
||||
trigger_tests:
|
||||
- id: explicit-install-register
|
||||
name: "Explicit trigger — install and register"
|
||||
query: "install @neuledge/context and register it as an MCP server in Claude Code"
|
||||
should_trigger: true
|
||||
|
||||
- id: explicit-package-management
|
||||
name: "Explicit trigger — package management"
|
||||
query: "install the react documentation package using neuledge context"
|
||||
should_trigger: true
|
||||
|
||||
- id: implicit-offline-docs
|
||||
name: "Implicit trigger — offline docs for AI agent"
|
||||
query: "I need React and Next.js docs available to my AI agent without web searches"
|
||||
should_trigger: true
|
||||
|
||||
- id: negative-different-mcp
|
||||
name: "Negative — different MCP server"
|
||||
query: "Add the Gitea MCP server to Claude Code"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-query-existing
|
||||
name: "Negative — querying an already-running server"
|
||||
query: "How do I query React docs using the context server that's already running?"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-cursor-setup
|
||||
name: "Negative — different provider"
|
||||
query: "Set up context serve for Cursor"
|
||||
should_trigger: false
|
||||
|
||||
output_tests:
|
||||
- id: install-script-announced
|
||||
name: "Install — script announced before running, version verified after"
|
||||
type: deterministic
|
||||
prompt: "install @neuledge/context"
|
||||
expected_output: >
|
||||
The skill announces that it will run scripts/setup-neuledge-context.sh before executing it,
|
||||
then verifies the installation by running `context --version`.
|
||||
assertions:
|
||||
- "Output mentions 'scripts/setup-neuledge-context.sh' before any install command is run"
|
||||
- "Output includes a `context --version` call after the install step"
|
||||
- "Output does not contain `npm install -g @neuledge/context@latest` (no floating @latest)"
|
||||
|
||||
- id: mcp-list-before-add
|
||||
name: "MCP registration — list checked before add, skipped if present"
|
||||
type: deterministic
|
||||
prompt: "register @neuledge/context as a Claude Code MCP server"
|
||||
expected_output: >
|
||||
The skill runs `claude mcp list` and checks for an existing 'context' entry before
|
||||
running `claude mcp add`. If already registered, the add step is skipped.
|
||||
assertions:
|
||||
- "Output includes `claude mcp list` before `claude mcp add context`"
|
||||
- "Output states that registration is skipped when the server is already present"
|
||||
- "The `claude mcp add` command uses stdio form: `claude mcp add context -- context serve`"
|
||||
|
||||
- id: auth-chmod-paired
|
||||
name: "Auth — secure-context-config.sh run immediately after auth add"
|
||||
type: deterministic
|
||||
prompt: "add auth credentials for docs.example.com to neuledge context"
|
||||
expected_output: >
|
||||
The skill runs `context auth add docs.example.com` with an environment variable reference
|
||||
for the credential, then immediately runs scripts/secure-context-config.sh.
|
||||
No credential value appears in the output.
|
||||
assertions:
|
||||
- "Output references an environment variable (e.g. $TOKEN) rather than a literal credential value"
|
||||
- "Output runs `scripts/secure-context-config.sh` in the same step as or immediately after `context auth add`"
|
||||
- "No bearer token, cookie value, or other credential string appears in the output"
|
||||
|
||||
- id: git-check-url-source
|
||||
name: "context add — git prerequisite checked for URL sources only"
|
||||
type: deterministic
|
||||
prompt: "add documentation from https://github.com/prisma/prisma using context add"
|
||||
expected_output: >
|
||||
Before running `context add`, the skill checks `git --version` because the source is a
|
||||
GitHub URL. The check is present for URL/repo sources and absent for local .db file paths.
|
||||
assertions:
|
||||
- "Output includes `git --version` before the `context add https://github.com/...` command"
|
||||
- "If given a local .db file path instead, the git check is absent"
|
||||
|
||||
- id: install-flow-quality
|
||||
name: "Full install + register flow quality"
|
||||
type: llm-rubric
|
||||
prompt: "install @neuledge/context and set it up as my Claude Code MCP server"
|
||||
expected_output: >
|
||||
A complete, ordered install-then-register flow: (1) announce the install script,
|
||||
(2) run setup-neuledge-context.sh, (3) verify with context --version,
|
||||
(4) check claude mcp list, (5) run claude mcp add if not already registered,
|
||||
(6) confirm with claude mcp list. Steps are in the correct order with verification
|
||||
between install and registration.
|
||||
assertions:
|
||||
- "Install step comes before MCP registration step"
|
||||
- "A verification command (context --version) appears between install and registration"
|
||||
- "The output would leave a user with a working @neuledge/context MCP server in Claude Code"
|
||||
- "No step is skipped without an explanation of why it was skipped"
|
||||
@@ -1,61 +0,0 @@
|
||||
skill_name: write-docs
|
||||
|
||||
trigger_tests:
|
||||
- id: explicit-trigger-document-module
|
||||
name: "Explicit trigger — document a script"
|
||||
query: "Write documentation for the install.sh script"
|
||||
should_trigger: true
|
||||
|
||||
- id: explicit-trigger-create-docs
|
||||
name: "Explicit trigger — create docs for a feature"
|
||||
query: "Create docs for this feature"
|
||||
should_trigger: true
|
||||
|
||||
- id: implicit-trigger-readme-update
|
||||
name: "Implicit trigger — outdated README section, no trigger phrase"
|
||||
query: "We need to update the README section for the auth module, the current one is outdated"
|
||||
should_trigger: true
|
||||
|
||||
- id: negative-trigger-prd
|
||||
name: "Negative — PRD request should route to to-prd"
|
||||
query: "Write a PRD for the new logging feature"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-trigger-write-skill
|
||||
name: "Negative — skill authoring request should route to write-skill"
|
||||
query: "Write a skill for generating documentation automatically"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-trigger-skill-file
|
||||
name: "Negative — SKILL.md update (skill files are self-describing)"
|
||||
query: "Document how the write-docs skill works by updating its SKILL.md"
|
||||
should_trigger: false
|
||||
|
||||
output_tests:
|
||||
- id: output-proposes-files-before-reading
|
||||
name: "Deterministic — candidates proposed or approval sought before reading files"
|
||||
type: deterministic
|
||||
prompt: "Write documentation for the config module"
|
||||
expected_output: "Skill proposes candidate files or asks the user to name specific files before reading any file content"
|
||||
assertions:
|
||||
- "Response proposes candidate file paths or asks the user to confirm which files to read before showing any extracted content"
|
||||
- "Response does not display extracted code content or API surface without first receiving file approval"
|
||||
|
||||
- id: output-gap-check-present
|
||||
name: "Deterministic — gap check step present before drafting"
|
||||
type: deterministic
|
||||
prompt: "Write documentation for the install.sh script, audience: developer"
|
||||
expected_output: "Skill presents extracted behaviour to the user and asks them to fill gaps before drafting any section"
|
||||
assertions:
|
||||
- "Response includes a gap check step that presents extracted behaviour and asks what the code does not explain"
|
||||
- "Response does not skip directly to a drafted documentation section without presenting extracted content first"
|
||||
|
||||
- id: output-never-invents-behaviour
|
||||
name: "LLM rubric — no invented behaviour, all claims sourced"
|
||||
type: llm-rubric
|
||||
prompt: "Document the src/config.py file for internal developers"
|
||||
expected_output: "Documentation where every claim is attributed to code content or explicit user input, with no invented explanations, assumptions about intent, or unverifiable behaviour claims."
|
||||
assertions:
|
||||
- "The skill explicitly derives each documented claim from a named source — a code line, spec section, or user statement — and does not add claims without attribution"
|
||||
- "The skill does not include descriptions of caller intent, design rationale, or future behaviour that are not present in the source material"
|
||||
- "If a behaviour is undocumentable (internal detail with no public spec), the skill notes it as out-of-scope rather than inventing an explanation"
|
||||
Reference in New Issue
Block a user