Defame1297
1ceacf17bc
Covers install, configuration, running evals, red-teaming, CI/CD integration, and dataset generation. Pins to v0.121.17 with acquisition notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1.9 KiB
topic, source_keys
| topic | source_keys | ||
|---|---|---|---|
| installation |
|
Acquisition notice
Promptfoo was acquired by OpenAI in March 2026. OpenAI has committed to keeping it open-source and multi-provider. Always pin to a specific version — do not track @latest. Monitor for neutrality degradation in future releases, particularly for non-OpenAI provider evaluation. Documented fallbacks: DeepEval (pytest-native, Python teams) and Arize Phoenix (self-hosted, vendor-neutral).
Pinned version in this skill: 0.121.17. Update via CI when a new version has been validated.
Prerequisites
- Node.js 18+ (Node 22 recommended for CI)
- npm or npx
No database or server process is required. Promptfoo stores evaluation results in a local SQLite cache.
Installation options
One-off via npx (no install required — preferred):
npx promptfoo@0.121.17 eval
Global install (pinned):
npm install -g promptfoo@0.121.17
After a global install, use promptfoo directly without npx.
Project dependency (for Node.js API usage):
npm install promptfoo@0.121.17
Initialising a project
npx promptfoo@0.121.17 init
Creates a promptfooconfig.yaml scaffold in the current directory with example prompts, providers, and tests.
To start from an official example:
npx promptfoo@0.121.17 init --example openai-mcp
npx promptfoo@0.121.17 init --example openai-structured-output
npx promptfoo@0.121.17 init --example openai-responses
npx promptfoo@0.121.17 init --example openai-audio
API keys
Promptfoo reads provider credentials from environment variables. Set them before running evals:
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
Other providers (Bedrock, Azure, Vertex) follow their own SDK credential conventions — see provider-specific docs.
Global timeout
export REQUEST_TIMEOUT_MS=600000 # 10 minutes default