Defame1297
1ceacf17bc
Covers install, configuration, running evals, red-teaming, CI/CD integration, and dataset generation. Pins to v0.121.17 with acquisition notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2.5 KiB
topic, source_keys
| topic | source_keys | ||
|---|---|---|---|
| overview |
|
What is Promptfoo
Promptfoo is an open-source, local-first CLI and library for evaluating and red-teaming LLM applications. It enables systematic, repeatable testing of prompts across multiple providers with assertions that grade outputs automatically. Results are stored locally and can be visualised in a browser UI or exported.
It is not a hosted service — all evaluation state stays on your machine unless you explicitly share a result.
Core concepts
Prompt — a template with {{variable}} placeholders. Can be a plain string, a JSON chat array (file://prompts/chat.json), or a JavaScript function (file://prompts/generate.js) that returns a string or message array dynamically.
Provider — an LLM endpoint to send the rendered prompt to. Providers are declared as strings (openai:gpt-5-mini, anthropic:messages:claude-sonnet-4-5) or objects with a config: block for additional parameters.
Test case — one input scenario. Contains vars: (values substituted into prompt variables) and assert: (a list of assertions that the response must satisfy).
Assertion — a pass/fail check on the model output. Ranges from deterministic (contains, regex, equals) to model-graded (llm-rubric, factuality).
Eval — one complete run: every prompt × every provider × every test case is executed and each assertion is scored. Results are a table of pass/fail cells with per-assertion metrics.
defaultTest — a top-level config key whose assert: and options: are merged into every test case, avoiding repetition.
Mental model
Think of an eval as a spreadsheet where rows are test cases and columns are (prompt, provider) pairs. Each cell contains the model output and assertion results. Running promptfoo eval fills the spreadsheet; promptfoo view opens it in a browser.
The config file (promptfooconfig.yaml) is the source of truth for a given eval. It is committed alongside your prompt files so evals are reproducible.
What it is used for
- Regression testing — catch prompt regressions before deploying changes
- Side-by-side model comparison — evaluate multiple providers on identical test suites
- Red-teaming — generate and run adversarial tests (jailbreaks, prompt injection, harmful content)
- Dataset generation — AI-generate test cases from a prompt template
- CI/CD gating — fail a pull request when assertion pass rate drops