Defame1297
1ceacf17bc
Covers install, configuration, running evals, red-teaming, CI/CD integration, and dataset generation. Pins to v0.121.17 with acquisition notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
41 lines
2.5 KiB
Markdown
41 lines
2.5 KiB
Markdown
---
|
||
topic: overview
|
||
source_keys:
|
||
- context7-promptfoo-dev
|
||
- context7-promptfoo-github
|
||
---
|
||
|
||
## What is Promptfoo
|
||
|
||
Promptfoo is an open-source, local-first CLI and library for evaluating and red-teaming LLM applications. It enables systematic, repeatable testing of prompts across multiple providers with assertions that grade outputs automatically. Results are stored locally and can be visualised in a browser UI or exported.
|
||
|
||
It is not a hosted service — all evaluation state stays on your machine unless you explicitly share a result.
|
||
|
||
## Core concepts
|
||
|
||
**Prompt** — a template with `{{variable}}` placeholders. Can be a plain string, a JSON chat array (`file://prompts/chat.json`), or a JavaScript function (`file://prompts/generate.js`) that returns a string or message array dynamically.
|
||
|
||
**Provider** — an LLM endpoint to send the rendered prompt to. Providers are declared as strings (`openai:gpt-5-mini`, `anthropic:messages:claude-sonnet-4-5`) or objects with a `config:` block for additional parameters.
|
||
|
||
**Test case** — one input scenario. Contains `vars:` (values substituted into prompt variables) and `assert:` (a list of assertions that the response must satisfy).
|
||
|
||
**Assertion** — a pass/fail check on the model output. Ranges from deterministic (`contains`, `regex`, `equals`) to model-graded (`llm-rubric`, `factuality`).
|
||
|
||
**Eval** — one complete run: every prompt × every provider × every test case is executed and each assertion is scored. Results are a table of pass/fail cells with per-assertion metrics.
|
||
|
||
**defaultTest** — a top-level config key whose `assert:` and `options:` are merged into every test case, avoiding repetition.
|
||
|
||
## Mental model
|
||
|
||
Think of an eval as a spreadsheet where rows are test cases and columns are (prompt, provider) pairs. Each cell contains the model output and assertion results. Running `promptfoo eval` fills the spreadsheet; `promptfoo view` opens it in a browser.
|
||
|
||
The config file (`promptfooconfig.yaml`) is the source of truth for a given eval. It is committed alongside your prompt files so evals are reproducible.
|
||
|
||
## What it is used for
|
||
|
||
- **Regression testing** — catch prompt regressions before deploying changes
|
||
- **Side-by-side model comparison** — evaluate multiple providers on identical test suites
|
||
- **Red-teaming** — generate and run adversarial tests (jailbreaks, prompt injection, harmful content)
|
||
- **Dataset generation** — AI-generate test cases from a prompt template
|
||
- **CI/CD gating** — fail a pull request when assertion pass rate drops
|