feat(skills): add promptfoo skill for LLM evaluation and red-teaming

Covers install, configuration, running evals, red-teaming, CI/CD
integration, and dataset generation. Pins to v0.121.17 with acquisition
notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-06-21 11:11:59 +00:00
parent 0155fcec26
commit 1ceacf17bc
12 changed files with 1195 additions and 0 deletions

View File

@@ -0,0 +1,70 @@
---
topic: installation
source_keys:
- context7-promptfoo-dev
- context7-promptfoo-github
---
## Acquisition notice
Promptfoo was acquired by OpenAI in March 2026. OpenAI has committed to keeping it open-source and multi-provider. **Always pin to a specific version** — do not track `@latest`. Monitor for neutrality degradation in future releases, particularly for non-OpenAI provider evaluation. Documented fallbacks: DeepEval (pytest-native, Python teams) and Arize Phoenix (self-hosted, vendor-neutral).
Pinned version in this skill: **0.121.17**. Update via CI when a new version has been validated.
## Prerequisites
- Node.js 18+ (Node 22 recommended for CI)
- npm or npx
No database or server process is required. Promptfoo stores evaluation results in a local SQLite cache.
## Installation options
**One-off via npx (no install required — preferred):**
```bash
npx promptfoo@0.121.17 eval
```
**Global install (pinned):**
```bash
npm install -g promptfoo@0.121.17
```
After a global install, use `promptfoo` directly without `npx`.
**Project dependency (for Node.js API usage):**
```bash
npm install promptfoo@0.121.17
```
## Initialising a project
```bash
npx promptfoo@0.121.17 init
```
Creates a `promptfooconfig.yaml` scaffold in the current directory with example prompts, providers, and tests.
To start from an official example:
```bash
npx promptfoo@0.121.17 init --example openai-mcp
npx promptfoo@0.121.17 init --example openai-structured-output
npx promptfoo@0.121.17 init --example openai-responses
npx promptfoo@0.121.17 init --example openai-audio
```
## API keys
Promptfoo reads provider credentials from environment variables. Set them before running evals:
```bash
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
```
Other providers (Bedrock, Azure, Vertex) follow their own SDK credential conventions — see provider-specific docs.
## Global timeout
```bash
export REQUEST_TIMEOUT_MS=600000 # 10 minutes default
```