Defame1297
1ceacf17bc
Covers install, configuration, running evals, red-teaming, CI/CD integration, and dataset generation. Pins to v0.121.17 with acquisition notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
108 lines
5.0 KiB
Markdown
108 lines
5.0 KiB
Markdown
---
|
|
name: promptfoo
|
|
description: >
|
|
Use when the user wants to install, set up, configure, or work with Promptfoo —
|
|
the open-source LLM evaluation CLI. Covers: project initialization (promptfoo init),
|
|
writing or editing promptfooconfig.yaml, running evaluations (promptfoo eval),
|
|
comparing model outputs side-by-side, viewing and sharing results, generating test
|
|
datasets, red-teaming LLM applications with adversarial plugins and strategies, and
|
|
integrating Promptfoo into CI/CD pipelines. Do NOT use when the user wants to evaluate
|
|
LLM output quality in general without Promptfoo, write tests for non-LLM code (use
|
|
tdd), or set up a different evaluation framework (RAGAS, OpenAI Evals, etc.).
|
|
metadata:
|
|
category: test
|
|
---
|
|
|
|
<requirements>
|
|
|
|
## Required inputs
|
|
|
|
- **Task scope** — what the user wants to do (install, configure, run eval, red-team, CI
|
|
integration, or debug); inferred from request, ask only if genuinely ambiguous
|
|
- **Config file path** — defaults to `promptfooconfig.yaml` in the working directory;
|
|
user provides if non-standard
|
|
- **Provider API keys** — must be set as environment variables before running evals;
|
|
remind the user if not already set
|
|
|
|
## Constraints
|
|
|
|
- Always use the pinned version `0.121.17` — never suggest `@latest` or an unpinned
|
|
install; see `references/installation.md` for rationale (OpenAI acquisition, March 2026)
|
|
- API keys and credentials must always reference environment variables in examples and
|
|
config — never hardcoded values, even as placeholders like `sk-abc123`
|
|
- Warn when the user's eval workload is non-OpenAI-centric: note the acquisition and
|
|
mention DeepEval / Arize Phoenix as documented fallbacks
|
|
- Offload detail to references/ files — do not reproduce CLI flags, assertion types, or
|
|
redteam plugin tables inline; wire each reference file in the step that needs it
|
|
|
|
</requirements>
|
|
|
|
<steps>
|
|
|
|
## Process
|
|
|
|
1. **Clarify scope.** Identify what the user wants: install/setup, configure
|
|
`promptfooconfig.yaml`, run an eval, generate a dataset, red-team, integrate CI/CD,
|
|
or debug a failing run. Infer from context — ask only if genuinely ambiguous.
|
|
|
|
2. **Install / initialise (if needed).** Check whether Promptfoo is already installed.
|
|
If not, guide through installation using the pinned version. To scaffold a new
|
|
project: `npx promptfoo@0.121.17 init`. See `references/installation.md` for
|
|
prerequisites, global/local install options, and the acquisition notice.
|
|
|
|
3. **Configure.** Help the user write or edit `promptfooconfig.yaml`. Cover: prompts
|
|
(inline, file://, JS/Python), providers (string shorthand and object form),
|
|
test cases with vars and assert, defaultTest, outputPath, and evaluateOptions.
|
|
See `references/configuration.md` for the full structure and
|
|
`references/assertions.md` for assertion types and shorthand syntax.
|
|
|
|
4. **Run evaluation.** Execute `npx promptfoo@0.121.17 eval` (with `--no-cache` in CI).
|
|
After the run, open results with `promptfoo view` or share with `promptfoo share`.
|
|
For rate-limit issues, suggest `--max-concurrency` and `--delay` flags.
|
|
See `references/cli-reference.md` for all flags and subcommands.
|
|
|
|
5. **Red-team (if requested).** Add a `redteam:` block to the config with the
|
|
appropriate plugins and strategies. Run `promptfoo redteam generate` to produce
|
|
adversarial test cases, then `promptfoo eval` to execute them.
|
|
See `references/redteam.md` for plugin/strategy tables and OWASP/MITRE mappings.
|
|
|
|
6. **CI integration (if requested).** Write a GitHub Actions workflow using
|
|
`promptfoo/promptfoo-action@v1` or a generic `npx promptfoo@0.121.17 eval --no-cache`
|
|
step. Always use CI secrets for API keys.
|
|
See `references/ci-cd.md` for full workflow templates.
|
|
|
|
7. **Generate test dataset (if requested).** Run `promptfoo generate dataset` with
|
|
optional `--instructions` to AI-generate test cases from the prompt template.
|
|
See `references/cli-reference.md` for flags.
|
|
|
|
## Output format
|
|
|
|
- Modified or newly created `promptfooconfig.yaml`
|
|
- Shell commands to run, with the pinned version and relevant flags
|
|
- GitHub Actions workflow file (`.github/workflows/llm-eval.yml`) when CI is in scope
|
|
|
|
</steps>
|
|
|
|
<checks>
|
|
|
|
## Failure handling
|
|
|
|
- No `promptfooconfig.yaml` found and user did not ask to create one — ask before
|
|
scaffolding; do not overwrite existing config without confirmation
|
|
- Eval fails with rate-limit errors — suggest `--max-concurrency 2 --delay 3000` and
|
|
`PROMPTFOO_RETRY_5XX=true`; see `references/cli-reference.md`
|
|
- User requests `@latest` or an unpinned install — correct to `0.121.17` and explain
|
|
the version-pinning rationale (acquisition notice)
|
|
- User requests hardcoded API key in config or example — refuse and redirect to
|
|
environment variable pattern
|
|
|
|
## Self-check
|
|
|
|
- [ ] Pinned version `0.121.17` used in all commands — no `@latest`
|
|
- [ ] No credentials appear in any output, config example, or shell command
|
|
- [ ] Non-OpenAI eval workloads include the acquisition warning
|
|
- [ ] Each references/ file wired in the step that uses it
|
|
- [ ] Config written or modified only after confirming path with user
|
|
|
|
</checks>
|