feat(skills): add promptfoo skill for LLM evaluation and red-teaming
Covers install, configuration, running evals, red-teaming, CI/CD integration, and dataset generation. Pins to v0.121.17 with acquisition notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
107
plugins/kyberforge/skills/promptfoo/SKILL.md
Normal file
107
plugins/kyberforge/skills/promptfoo/SKILL.md
Normal file
@@ -0,0 +1,107 @@
|
||||
---
|
||||
name: promptfoo
|
||||
description: >
|
||||
Use when the user wants to install, set up, configure, or work with Promptfoo —
|
||||
the open-source LLM evaluation CLI. Covers: project initialization (promptfoo init),
|
||||
writing or editing promptfooconfig.yaml, running evaluations (promptfoo eval),
|
||||
comparing model outputs side-by-side, viewing and sharing results, generating test
|
||||
datasets, red-teaming LLM applications with adversarial plugins and strategies, and
|
||||
integrating Promptfoo into CI/CD pipelines. Do NOT use when the user wants to evaluate
|
||||
LLM output quality in general without Promptfoo, write tests for non-LLM code (use
|
||||
tdd), or set up a different evaluation framework (RAGAS, OpenAI Evals, etc.).
|
||||
metadata:
|
||||
category: test
|
||||
---
|
||||
|
||||
<requirements>
|
||||
|
||||
## Required inputs
|
||||
|
||||
- **Task scope** — what the user wants to do (install, configure, run eval, red-team, CI
|
||||
integration, or debug); inferred from request, ask only if genuinely ambiguous
|
||||
- **Config file path** — defaults to `promptfooconfig.yaml` in the working directory;
|
||||
user provides if non-standard
|
||||
- **Provider API keys** — must be set as environment variables before running evals;
|
||||
remind the user if not already set
|
||||
|
||||
## Constraints
|
||||
|
||||
- Always use the pinned version `0.121.17` — never suggest `@latest` or an unpinned
|
||||
install; see `references/installation.md` for rationale (OpenAI acquisition, March 2026)
|
||||
- API keys and credentials must always reference environment variables in examples and
|
||||
config — never hardcoded values, even as placeholders like `sk-abc123`
|
||||
- Warn when the user's eval workload is non-OpenAI-centric: note the acquisition and
|
||||
mention DeepEval / Arize Phoenix as documented fallbacks
|
||||
- Offload detail to references/ files — do not reproduce CLI flags, assertion types, or
|
||||
redteam plugin tables inline; wire each reference file in the step that needs it
|
||||
|
||||
</requirements>
|
||||
|
||||
<steps>
|
||||
|
||||
## Process
|
||||
|
||||
1. **Clarify scope.** Identify what the user wants: install/setup, configure
|
||||
`promptfooconfig.yaml`, run an eval, generate a dataset, red-team, integrate CI/CD,
|
||||
or debug a failing run. Infer from context — ask only if genuinely ambiguous.
|
||||
|
||||
2. **Install / initialise (if needed).** Check whether Promptfoo is already installed.
|
||||
If not, guide through installation using the pinned version. To scaffold a new
|
||||
project: `npx promptfoo@0.121.17 init`. See `references/installation.md` for
|
||||
prerequisites, global/local install options, and the acquisition notice.
|
||||
|
||||
3. **Configure.** Help the user write or edit `promptfooconfig.yaml`. Cover: prompts
|
||||
(inline, file://, JS/Python), providers (string shorthand and object form),
|
||||
test cases with vars and assert, defaultTest, outputPath, and evaluateOptions.
|
||||
See `references/configuration.md` for the full structure and
|
||||
`references/assertions.md` for assertion types and shorthand syntax.
|
||||
|
||||
4. **Run evaluation.** Execute `npx promptfoo@0.121.17 eval` (with `--no-cache` in CI).
|
||||
After the run, open results with `promptfoo view` or share with `promptfoo share`.
|
||||
For rate-limit issues, suggest `--max-concurrency` and `--delay` flags.
|
||||
See `references/cli-reference.md` for all flags and subcommands.
|
||||
|
||||
5. **Red-team (if requested).** Add a `redteam:` block to the config with the
|
||||
appropriate plugins and strategies. Run `promptfoo redteam generate` to produce
|
||||
adversarial test cases, then `promptfoo eval` to execute them.
|
||||
See `references/redteam.md` for plugin/strategy tables and OWASP/MITRE mappings.
|
||||
|
||||
6. **CI integration (if requested).** Write a GitHub Actions workflow using
|
||||
`promptfoo/promptfoo-action@v1` or a generic `npx promptfoo@0.121.17 eval --no-cache`
|
||||
step. Always use CI secrets for API keys.
|
||||
See `references/ci-cd.md` for full workflow templates.
|
||||
|
||||
7. **Generate test dataset (if requested).** Run `promptfoo generate dataset` with
|
||||
optional `--instructions` to AI-generate test cases from the prompt template.
|
||||
See `references/cli-reference.md` for flags.
|
||||
|
||||
## Output format
|
||||
|
||||
- Modified or newly created `promptfooconfig.yaml`
|
||||
- Shell commands to run, with the pinned version and relevant flags
|
||||
- GitHub Actions workflow file (`.github/workflows/llm-eval.yml`) when CI is in scope
|
||||
|
||||
</steps>
|
||||
|
||||
<checks>
|
||||
|
||||
## Failure handling
|
||||
|
||||
- No `promptfooconfig.yaml` found and user did not ask to create one — ask before
|
||||
scaffolding; do not overwrite existing config without confirmation
|
||||
- Eval fails with rate-limit errors — suggest `--max-concurrency 2 --delay 3000` and
|
||||
`PROMPTFOO_RETRY_5XX=true`; see `references/cli-reference.md`
|
||||
- User requests `@latest` or an unpinned install — correct to `0.121.17` and explain
|
||||
the version-pinning rationale (acquisition notice)
|
||||
- User requests hardcoded API key in config or example — refuse and redirect to
|
||||
environment variable pattern
|
||||
|
||||
## Self-check
|
||||
|
||||
- [ ] Pinned version `0.121.17` used in all commands — no `@latest`
|
||||
- [ ] No credentials appear in any output, config example, or shell command
|
||||
- [ ] Non-OpenAI eval workloads include the acquisition warning
|
||||
- [ ] Each references/ file wired in the step that uses it
|
||||
- [ ] Config written or modified only after confirming path with user
|
||||
|
||||
</checks>
|
||||
Reference in New Issue
Block a user