---
topic: overview
source_keys:
  - context7-promptfoo-dev
  - context7-promptfoo-github
---

## What is Promptfoo

Promptfoo is an open-source, local-first CLI and library for evaluating and red-teaming LLM applications. It enables systematic, repeatable testing of prompts across multiple providers with assertions that grade outputs automatically. Results are stored locally and can be visualised in a browser UI or exported.

It is not a hosted service — all evaluation state stays on your machine unless you explicitly share a result.

## Core concepts

**Prompt** — a template with `{{variable}}` placeholders. Can be a plain string, a JSON chat array (`file://prompts/chat.json`), or a JavaScript function (`file://prompts/generate.js`) that returns a string or message array dynamically.

**Provider** — an LLM endpoint to send the rendered prompt to. Providers are declared as strings (`openai:gpt-5-mini`, `anthropic:messages:claude-sonnet-4-5`) or objects with a `config:` block for additional parameters.

**Test case** — one input scenario. Contains `vars:` (values substituted into prompt variables) and `assert:` (a list of assertions that the response must satisfy).

**Assertion** — a pass/fail check on the model output. Ranges from deterministic (`contains`, `regex`, `equals`) to model-graded (`llm-rubric`, `factuality`).

**Eval** — one complete run: every prompt × every provider × every test case is executed and each assertion is scored. Results are a table of pass/fail cells with per-assertion metrics.

**defaultTest** — a top-level config key whose `assert:` and `options:` are merged into every test case, avoiding repetition.

## Mental model

Think of an eval as a spreadsheet where rows are test cases and columns are (prompt, provider) pairs. Each cell contains the model output and assertion results. Running `promptfoo eval` fills the spreadsheet; `promptfoo view` opens it in a browser.

The config file (`promptfooconfig.yaml`) is the source of truth for a given eval. It is committed alongside your prompt files so evals are reproducible.

## What it is used for

- **Regression testing** — catch prompt regressions before deploying changes
- **Side-by-side model comparison** — evaluate multiple providers on identical test suites
- **Red-teaming** — generate and run adversarial tests (jailbreaks, prompt injection, harmful content)
- **Dataset generation** — AI-generate test cases from a prompt template
- **CI/CD gating** — fail a pull request when assertion pass rate drops