feat(skills): add promptfoo skill for LLM evaluation and red-teaming

Covers install, configuration, running evals, red-teaming, CI/CD
integration, and dataset generation. Pins to v0.121.17 with acquisition
notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-06-21 11:11:59 +00:00
parent 0155fcec26
commit 1ceacf17bc
12 changed files with 1195 additions and 0 deletions

View File

@@ -0,0 +1,148 @@
---
topic: cli-reference
source_keys:
- context7-promptfoo-dev
- context7-promptfoo-github
---
## Invocation
```bash
promptfoo <command> [options] # global install
npx promptfoo@0.121.17 <command> # one-off via npx
```
## Commands
### `init`
Scaffold a new project in the current directory.
```bash
promptfoo init
promptfoo init --example openai-mcp
```
Creates `promptfooconfig.yaml` with example prompts, providers, and test cases.
---
### `eval` (most common)
Run an evaluation.
```bash
promptfoo eval
promptfoo eval -c path/to/promptfooconfig.yaml
promptfoo eval --no-cache
promptfoo eval --max-concurrency 2
promptfoo eval --delay 3000
promptfoo eval -o results.json
```
| Flag | Description |
|---|---|
| `-c <path>` | Config file path (default: `promptfooconfig.yaml`) |
| `--no-cache` | Disable response cache; forces fresh LLM calls |
| `--max-concurrency <n>` | Max parallel requests (default: provider-dependent) |
| `--delay <ms>` | Fixed delay between requests |
| `-o <path>` | Output path (`.html`, `.json`, `.csv`, `.yaml`) |
| `--format sarif` | Output in SARIF format (for security scanning) |
Exit code is non-zero when any assertion fails, making it suitable for CI gating.
---
### `view`
Open the most recent evaluation results in a local browser UI.
```bash
promptfoo view
```
---
### `share`
Upload results and get a shareable URL.
```bash
promptfoo share
```
---
### `cache clear`
Clear all cached LLM responses.
```bash
promptfoo cache clear
```
---
### `generate dataset`
Use an LLM to auto-generate test cases from a prompt template.
```bash
promptfoo generate dataset
promptfoo generate dataset --config path/to/config.yaml
promptfoo generate dataset --output generated_tests.yaml
promptfoo generate dataset --instructions "Consider edge cases related to international travel"
```
---
### `redteam generate`
Generate adversarial test cases for red-teaming.
```bash
promptfoo redteam generate
promptfoo redteam generate -c promptfooconfig.yaml
```
---
### `scan-model`
Scan model files for security vulnerabilities. Outputs results in SARIF format.
```bash
promptfoo scan-model ./models/ --strict --no-write --format sarif --output scan.sarif
```
---
### `auth`
Manage authentication (for sharing and cloud features).
```bash
promptfoo auth login
promptfoo auth logout
```
---
## Rate-limit management
```bash
promptfoo eval --max-concurrency 1 --delay 3000
```
Or in config:
```yaml
evaluateOptions:
maxConcurrency: 2
delay: 3000
```
Environment variable for backoff:
```bash
export PROMPTFOO_REQUEST_BACKOFF_MS=10000
export PROMPTFOO_RETRY_5XX=true
```