---
name: promptfoo
description: >
  Use when the user wants to install, set up, configure, or work with Promptfoo —
  the open-source LLM evaluation CLI. Covers: project initialization (promptfoo init),
  writing or editing promptfooconfig.yaml, running evaluations (promptfoo eval),
  comparing model outputs side-by-side, viewing and sharing results, generating test
  datasets, red-teaming LLM applications with adversarial plugins and strategies, and
  integrating Promptfoo into CI/CD pipelines. Do NOT use when the user wants to evaluate
  LLM output quality in general without Promptfoo, write tests for non-LLM code (use
  tdd), or set up a different evaluation framework (RAGAS, OpenAI Evals, etc.).
metadata:
  category: test
---

<requirements>

## Required inputs

- **Task scope** — what the user wants to do (install, configure, run eval, red-team, CI
  integration, or debug); inferred from request, ask only if genuinely ambiguous
- **Config file path** — defaults to `promptfooconfig.yaml` in the working directory;
  user provides if non-standard
- **Provider API keys** — must be set as environment variables before running evals;
  remind the user if not already set

## Constraints

- Always use the pinned version `0.121.17` — never suggest `@latest` or an unpinned
  install; see `references/installation.md` for rationale (OpenAI acquisition, March 2026)
- API keys and credentials must always reference environment variables in examples and
  config — never hardcoded values, even as placeholders like `sk-abc123`
- Warn when the user's eval workload is non-OpenAI-centric: note the acquisition and
  mention DeepEval / Arize Phoenix as documented fallbacks
- Offload detail to references/ files — do not reproduce CLI flags, assertion types, or
  redteam plugin tables inline; wire each reference file in the step that needs it

</requirements>

<steps>

## Process

1. **Clarify scope.** Identify what the user wants: install/setup, configure
   `promptfooconfig.yaml`, run an eval, generate a dataset, red-team, integrate CI/CD,
   or debug a failing run. Infer from context — ask only if genuinely ambiguous.

2. **Install / initialise (if needed).** Check whether Promptfoo is already installed.
   If not, guide through installation using the pinned version. To scaffold a new
   project: `npx promptfoo@0.121.17 init`. See `references/installation.md` for
   prerequisites, global/local install options, and the acquisition notice.

3. **Configure.** Help the user write or edit `promptfooconfig.yaml`. Cover: prompts
   (inline, file://, JS/Python), providers (string shorthand and object form),
   test cases with vars and assert, defaultTest, outputPath, and evaluateOptions.
   See `references/configuration.md` for the full structure and
   `references/assertions.md` for assertion types and shorthand syntax.

4. **Run evaluation.** Execute `npx promptfoo@0.121.17 eval` (with `--no-cache` in CI).
   After the run, open results with `promptfoo view` or share with `promptfoo share`.
   For rate-limit issues, suggest `--max-concurrency` and `--delay` flags.
   See `references/cli-reference.md` for all flags and subcommands.

5. **Red-team (if requested).** Add a `redteam:` block to the config with the
   appropriate plugins and strategies. Run `promptfoo redteam generate` to produce
   adversarial test cases, then `promptfoo eval` to execute them.
   See `references/redteam.md` for plugin/strategy tables and OWASP/MITRE mappings.

6. **CI integration (if requested).** Write a GitHub Actions workflow using
   `promptfoo/promptfoo-action@v1` or a generic `npx promptfoo@0.121.17 eval --no-cache`
   step. Always use CI secrets for API keys.
   See `references/ci-cd.md` for full workflow templates.

7. **Generate test dataset (if requested).** Run `promptfoo generate dataset` with
   optional `--instructions` to AI-generate test cases from the prompt template.
   See `references/cli-reference.md` for flags.

## Output format

- Modified or newly created `promptfooconfig.yaml`
- Shell commands to run, with the pinned version and relevant flags
- GitHub Actions workflow file (`.github/workflows/llm-eval.yml`) when CI is in scope

</steps>

<checks>

## Failure handling

- No `promptfooconfig.yaml` found and user did not ask to create one — ask before
  scaffolding; do not overwrite existing config without confirmation
- Eval fails with rate-limit errors — suggest `--max-concurrency 2 --delay 3000` and
  `PROMPTFOO_RETRY_5XX=true`; see `references/cli-reference.md`
- User requests `@latest` or an unpinned install — correct to `0.121.17` and explain
  the version-pinning rationale (acquisition notice)
- User requests hardcoded API key in config or example — refuse and redirect to
  environment variable pattern

## Self-check

- [ ] Pinned version `0.121.17` used in all commands — no `@latest`
- [ ] No credentials appear in any output, config example, or shell command
- [ ] Non-OpenAI eval workloads include the acquisition warning
- [ ] Each references/ file wired in the step that uses it
- [ ] Config written or modified only after confirming path with user

</checks>