feat(skills): add promptfoo skill for LLM evaluation and red-teaming
Covers install, configuration, running evals, red-teaming, CI/CD integration, and dataset generation. Pins to v0.121.17 with acquisition notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
148
plugins/kyberforge/skills/promptfoo/references/cli-reference.md
Normal file
148
plugins/kyberforge/skills/promptfoo/references/cli-reference.md
Normal file
@@ -0,0 +1,148 @@
|
||||
---
|
||||
topic: cli-reference
|
||||
source_keys:
|
||||
- context7-promptfoo-dev
|
||||
- context7-promptfoo-github
|
||||
---
|
||||
|
||||
## Invocation
|
||||
|
||||
```bash
|
||||
promptfoo <command> [options] # global install
|
||||
npx promptfoo@0.121.17 <command> # one-off via npx
|
||||
```
|
||||
|
||||
## Commands
|
||||
|
||||
### `init`
|
||||
|
||||
Scaffold a new project in the current directory.
|
||||
|
||||
```bash
|
||||
promptfoo init
|
||||
promptfoo init --example openai-mcp
|
||||
```
|
||||
|
||||
Creates `promptfooconfig.yaml` with example prompts, providers, and test cases.
|
||||
|
||||
---
|
||||
|
||||
### `eval` (most common)
|
||||
|
||||
Run an evaluation.
|
||||
|
||||
```bash
|
||||
promptfoo eval
|
||||
promptfoo eval -c path/to/promptfooconfig.yaml
|
||||
promptfoo eval --no-cache
|
||||
promptfoo eval --max-concurrency 2
|
||||
promptfoo eval --delay 3000
|
||||
promptfoo eval -o results.json
|
||||
```
|
||||
|
||||
| Flag | Description |
|
||||
|---|---|
|
||||
| `-c <path>` | Config file path (default: `promptfooconfig.yaml`) |
|
||||
| `--no-cache` | Disable response cache; forces fresh LLM calls |
|
||||
| `--max-concurrency <n>` | Max parallel requests (default: provider-dependent) |
|
||||
| `--delay <ms>` | Fixed delay between requests |
|
||||
| `-o <path>` | Output path (`.html`, `.json`, `.csv`, `.yaml`) |
|
||||
| `--format sarif` | Output in SARIF format (for security scanning) |
|
||||
|
||||
Exit code is non-zero when any assertion fails, making it suitable for CI gating.
|
||||
|
||||
---
|
||||
|
||||
### `view`
|
||||
|
||||
Open the most recent evaluation results in a local browser UI.
|
||||
|
||||
```bash
|
||||
promptfoo view
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `share`
|
||||
|
||||
Upload results and get a shareable URL.
|
||||
|
||||
```bash
|
||||
promptfoo share
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `cache clear`
|
||||
|
||||
Clear all cached LLM responses.
|
||||
|
||||
```bash
|
||||
promptfoo cache clear
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `generate dataset`
|
||||
|
||||
Use an LLM to auto-generate test cases from a prompt template.
|
||||
|
||||
```bash
|
||||
promptfoo generate dataset
|
||||
promptfoo generate dataset --config path/to/config.yaml
|
||||
promptfoo generate dataset --output generated_tests.yaml
|
||||
promptfoo generate dataset --instructions "Consider edge cases related to international travel"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `redteam generate`
|
||||
|
||||
Generate adversarial test cases for red-teaming.
|
||||
|
||||
```bash
|
||||
promptfoo redteam generate
|
||||
promptfoo redteam generate -c promptfooconfig.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `scan-model`
|
||||
|
||||
Scan model files for security vulnerabilities. Outputs results in SARIF format.
|
||||
|
||||
```bash
|
||||
promptfoo scan-model ./models/ --strict --no-write --format sarif --output scan.sarif
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `auth`
|
||||
|
||||
Manage authentication (for sharing and cloud features).
|
||||
|
||||
```bash
|
||||
promptfoo auth login
|
||||
promptfoo auth logout
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rate-limit management
|
||||
|
||||
```bash
|
||||
promptfoo eval --max-concurrency 1 --delay 3000
|
||||
```
|
||||
|
||||
Or in config:
|
||||
```yaml
|
||||
evaluateOptions:
|
||||
maxConcurrency: 2
|
||||
delay: 3000
|
||||
```
|
||||
|
||||
Environment variable for backoff:
|
||||
```bash
|
||||
export PROMPTFOO_REQUEST_BACKOFF_MS=10000
|
||||
export PROMPTFOO_RETRY_5XX=true
|
||||
```
|
||||
Reference in New Issue
Block a user