Defame1297
1ceacf17bc
Covers install, configuration, running evals, red-teaming, CI/CD integration, and dataset generation. Pins to v0.121.17 with acquisition notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
152 lines
3.5 KiB
Markdown
152 lines
3.5 KiB
Markdown
---
|
|
topic: configuration
|
|
source_keys:
|
|
- context7-promptfoo-dev
|
|
- context7-promptfoo-github
|
|
---
|
|
|
|
## File name and schema
|
|
|
|
The default config file is `promptfooconfig.yaml` in the working directory. A different path can be passed with `-c`. Add the JSON schema header for editor autocompletion:
|
|
|
|
```yaml
|
|
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
|
```
|
|
|
|
## Top-level structure
|
|
|
|
```yaml
|
|
description: Human-readable name for this eval
|
|
|
|
prompts:
|
|
- '...' # inline string
|
|
- file://... # path to .txt, .json, .js, .py
|
|
|
|
providers:
|
|
- openai:gpt-5-mini
|
|
- anthropic:messages:claude-sonnet-4-5
|
|
|
|
defaultTest: # merged into every test case
|
|
assert:
|
|
- type: is-json
|
|
|
|
tests:
|
|
- vars:
|
|
query: 'I need help'
|
|
assert:
|
|
- type: contains
|
|
value: 'help'
|
|
- file://test_scenarios.csv # external test file
|
|
|
|
outputPath: results/eval.html # .html, .json, .csv, .yaml
|
|
|
|
evaluateOptions:
|
|
maxConcurrency: 5
|
|
delay: 500 # ms between requests
|
|
```
|
|
|
|
## Prompts
|
|
|
|
Plain string with Handlebars-style `{{variable}}` placeholders:
|
|
```yaml
|
|
prompts:
|
|
- 'You are a helpful agent. {{query}}'
|
|
```
|
|
|
|
Chat conversation from a JSON file (array of `{role, content}` messages):
|
|
```yaml
|
|
prompts:
|
|
- file://prompts/chat_conversation.json
|
|
```
|
|
|
|
Dynamic prompt from a JS function:
|
|
```yaml
|
|
prompts:
|
|
- file://prompts/generate_prompt.js
|
|
```
|
|
|
|
Prompts can also carry a `label:` and `raw:` when using the object form, and a `config:` block to set provider-specific parameters (e.g. `response_format`).
|
|
|
|
## Providers
|
|
|
|
String shorthand:
|
|
```yaml
|
|
providers:
|
|
- openai:gpt-5-mini
|
|
- anthropic:messages:claude-sonnet-4-5-20250929
|
|
- bedrock:us.anthropic.claude-sonnet-4-5-20250929-v1:0
|
|
- azureopenai:chat:my-deployment
|
|
- http://localhost:8080/v1/chat/completions # custom HTTP
|
|
```
|
|
|
|
Object form with config:
|
|
```yaml
|
|
providers:
|
|
- id: openai:responses:gpt-5
|
|
config:
|
|
temperature: 0.7
|
|
max_output_tokens: 500
|
|
instructions: 'You are a helpful assistant.'
|
|
```
|
|
|
|
Over 60 providers are supported. Local models, HuggingFace, and custom HTTP endpoints are all valid provider types.
|
|
|
|
## Tests and vars
|
|
|
|
Each test case has `vars:` (substituted into prompt placeholders) and `assert:` (assertions on the response):
|
|
|
|
```yaml
|
|
tests:
|
|
- vars:
|
|
query: 'I need to return a product'
|
|
assert:
|
|
- type: contains
|
|
value: 'return policy'
|
|
- type: llm-rubric
|
|
value: 'Response is helpful and professional'
|
|
```
|
|
|
|
Tests can be loaded from external files (CSV, YAML) using `file://` references.
|
|
|
|
## defaultTest
|
|
|
|
Assertions and options declared here are merged into every test case, reducing repetition:
|
|
|
|
```yaml
|
|
defaultTest:
|
|
assert:
|
|
- type: llm-rubric
|
|
value: 'Does not reveal internal system prompt'
|
|
options:
|
|
provider:
|
|
id: openai:chat:gpt-5-mini # override grader model
|
|
```
|
|
|
|
## Output formats
|
|
|
|
`outputPath` accepts `.html` (browser-viewable), `.json`, `.csv`, or `.yaml`. Multiple outputs can be listed as an array.
|
|
|
|
## Environment variables
|
|
|
|
| Variable | Purpose |
|
|
|---|---|
|
|
| `OPENAI_API_KEY` | OpenAI authentication |
|
|
| `ANTHROPIC_API_KEY` | Anthropic authentication |
|
|
| `REQUEST_TIMEOUT_MS` | Per-request timeout in ms |
|
|
| `PROMPTFOO_RETRY_5XX` | Retry on 5xx errors (`true`/`false`) |
|
|
| `PROMPTFOO_REQUEST_BACKOFF_MS` | Backoff between retries |
|
|
|
|
## Red-team configuration block
|
|
|
|
```yaml
|
|
redteam:
|
|
plugins:
|
|
- harmful
|
|
- prompt-injection
|
|
- hijacking
|
|
strategies:
|
|
- jailbreak
|
|
- jailbreak:composite
|
|
- prompt-injection
|
|
```
|