feat(skills): add promptfoo skill for LLM evaluation and red-teaming
Covers install, configuration, running evals, red-teaming, CI/CD integration, and dataset generation. Pins to v0.121.17 with acquisition notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,70 @@
|
||||
---
|
||||
topic: installation
|
||||
source_keys:
|
||||
- context7-promptfoo-dev
|
||||
- context7-promptfoo-github
|
||||
---
|
||||
|
||||
## Acquisition notice
|
||||
|
||||
Promptfoo was acquired by OpenAI in March 2026. OpenAI has committed to keeping it open-source and multi-provider. **Always pin to a specific version** — do not track `@latest`. Monitor for neutrality degradation in future releases, particularly for non-OpenAI provider evaluation. Documented fallbacks: DeepEval (pytest-native, Python teams) and Arize Phoenix (self-hosted, vendor-neutral).
|
||||
|
||||
Pinned version in this skill: **0.121.17**. Update via CI when a new version has been validated.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Node.js 18+ (Node 22 recommended for CI)
|
||||
- npm or npx
|
||||
|
||||
No database or server process is required. Promptfoo stores evaluation results in a local SQLite cache.
|
||||
|
||||
## Installation options
|
||||
|
||||
**One-off via npx (no install required — preferred):**
|
||||
```bash
|
||||
npx promptfoo@0.121.17 eval
|
||||
```
|
||||
|
||||
**Global install (pinned):**
|
||||
```bash
|
||||
npm install -g promptfoo@0.121.17
|
||||
```
|
||||
After a global install, use `promptfoo` directly without `npx`.
|
||||
|
||||
**Project dependency (for Node.js API usage):**
|
||||
```bash
|
||||
npm install promptfoo@0.121.17
|
||||
```
|
||||
|
||||
## Initialising a project
|
||||
|
||||
```bash
|
||||
npx promptfoo@0.121.17 init
|
||||
```
|
||||
|
||||
Creates a `promptfooconfig.yaml` scaffold in the current directory with example prompts, providers, and tests.
|
||||
|
||||
To start from an official example:
|
||||
```bash
|
||||
npx promptfoo@0.121.17 init --example openai-mcp
|
||||
npx promptfoo@0.121.17 init --example openai-structured-output
|
||||
npx promptfoo@0.121.17 init --example openai-responses
|
||||
npx promptfoo@0.121.17 init --example openai-audio
|
||||
```
|
||||
|
||||
## API keys
|
||||
|
||||
Promptfoo reads provider credentials from environment variables. Set them before running evals:
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=sk-...
|
||||
export ANTHROPIC_API_KEY=sk-ant-...
|
||||
```
|
||||
|
||||
Other providers (Bedrock, Azure, Vertex) follow their own SDK credential conventions — see provider-specific docs.
|
||||
|
||||
## Global timeout
|
||||
|
||||
```bash
|
||||
export REQUEST_TIMEOUT_MS=600000 # 10 minutes default
|
||||
```
|
||||
Reference in New Issue
Block a user