Defame1297
1ceacf17bc
Covers install, configuration, running evals, red-teaming, CI/CD integration, and dataset generation. Pins to v0.121.17 with acquisition notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2.6 KiB
2.6 KiB
topic, source_keys
| topic | source_keys | ||
|---|---|---|---|
| cli-reference |
|
Invocation
promptfoo <command> [options] # global install
npx promptfoo@0.121.17 <command> # one-off via npx
Commands
init
Scaffold a new project in the current directory.
promptfoo init
promptfoo init --example openai-mcp
Creates promptfooconfig.yaml with example prompts, providers, and test cases.
eval (most common)
Run an evaluation.
promptfoo eval
promptfoo eval -c path/to/promptfooconfig.yaml
promptfoo eval --no-cache
promptfoo eval --max-concurrency 2
promptfoo eval --delay 3000
promptfoo eval -o results.json
| Flag | Description |
|---|---|
-c <path> |
Config file path (default: promptfooconfig.yaml) |
--no-cache |
Disable response cache; forces fresh LLM calls |
--max-concurrency <n> |
Max parallel requests (default: provider-dependent) |
--delay <ms> |
Fixed delay between requests |
-o <path> |
Output path (.html, .json, .csv, .yaml) |
--format sarif |
Output in SARIF format (for security scanning) |
Exit code is non-zero when any assertion fails, making it suitable for CI gating.
view
Open the most recent evaluation results in a local browser UI.
promptfoo view
share
Upload results and get a shareable URL.
promptfoo share
cache clear
Clear all cached LLM responses.
promptfoo cache clear
generate dataset
Use an LLM to auto-generate test cases from a prompt template.
promptfoo generate dataset
promptfoo generate dataset --config path/to/config.yaml
promptfoo generate dataset --output generated_tests.yaml
promptfoo generate dataset --instructions "Consider edge cases related to international travel"
redteam generate
Generate adversarial test cases for red-teaming.
promptfoo redteam generate
promptfoo redteam generate -c promptfooconfig.yaml
scan-model
Scan model files for security vulnerabilities. Outputs results in SARIF format.
promptfoo scan-model ./models/ --strict --no-write --format sarif --output scan.sarif
auth
Manage authentication (for sharing and cloud features).
promptfoo auth login
promptfoo auth logout
Rate-limit management
promptfoo eval --max-concurrency 1 --delay 3000
Or in config:
evaluateOptions:
maxConcurrency: 2
delay: 3000
Environment variable for backoff:
export PROMPTFOO_REQUEST_BACKOFF_MS=10000
export PROMPTFOO_RETRY_5XX=true