feat(skills): add promptfoo skill for LLM evaluation and red-teaming

Covers install, configuration, running evals, red-teaming, CI/CD
integration, and dataset generation. Pins to v0.121.17 with acquisition
notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-06-21 11:11:59 +00:00
parent 0155fcec26
commit 1ceacf17bc
12 changed files with 1195 additions and 0 deletions

View File

@@ -0,0 +1,150 @@
---
topic: examples
source_keys:
- context7-promptfoo-dev
- context7-promptfoo-github
---
## Quickstart
```bash
npx promptfoo@0.121.17 init
# edit promptfooconfig.yaml
npx promptfoo@0.121.17 eval
npx promptfoo@0.121.17 view
```
## Minimal config
```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
prompts:
- 'Answer the user question concisely. Question: {{question}}'
providers:
- openai:gpt-5-mini
tests:
- vars:
question: How do I reset my password?
assert:
- type: contains
value: reset
- vars:
question: Can I cancel my subscription today?
assert:
- type: llm-rubric
value: The answer clearly explains the cancellation path.
```
## Multi-provider comparison
```yaml
providers:
- openai:gpt-5-mini
- anthropic:claude-3-haiku
prompts:
- 'You are a helpful customer service agent. {{query}}'
tests:
- vars:
query: 'I need to return a product'
assert:
- type: contains
value: 'return policy'
- type: llm-rubric
value: 'Response is helpful and professional'
```
Running this produces a side-by-side table with both models' outputs and assertion scores.
## Loading tests from CSV
```yaml
tests:
- file://test_scenarios.csv
```
CSV format: one column per variable, header row must match `{{variable}}` names in the prompt. An `__expected` column maps to the `equals` assertion automatically.
## Factuality evaluation
```yaml
providers:
- openai:gpt-5-mini
prompts:
- |
Please answer the following question accurately:
Question: What is the capital of {{location}}?
tests:
- vars:
location: California
assert:
- type: factuality
value: The capital of California is Sacramento
```
## defaultTest for shared assertions
```yaml
defaultTest:
assert:
- type: llm-rubric
value: |
Evaluate whether the response correctly answers the question.
Question: {{ question }}
Model Response: {{ output }}
Correct Answer: {{ answer }}
Grade accuracy 0.0–1.0. Pass if >= 0.8.
threshold: 0.8
tests:
- vars:
question: What year did WW2 end?
answer: '1945'
- vars:
question: What is the boiling point of water in Celsius?
answer: '100'
```
## Node.js API
```javascript
import { evaluate } from 'promptfoo';
const evalRecord = await evaluate({
prompts: ['Translate to Spanish: {{ text }}'],
providers: ['openai:chat:gpt-5.5'],
tests: [
{
vars: { text: 'Hello' },
assert: [{ type: 'contains', value: 'Hola', metric: 'translation' }],
},
],
});
const results = await evalRecord.toEvaluateSummary();
console.log(`Pass rate: ${results.stats.successes}/${results.results.length}`);
```
## Generating test datasets with AI
```bash
# Generate test cases based on your prompt template
promptfoo generate dataset
promptfoo generate dataset --instructions "Consider edge cases related to international travel"
promptfoo generate dataset --output generated_tests.yaml
```
## Saving and sharing results
```yaml
outputPath: evaluations/results.html
```
Or via CLI:
```bash
promptfoo eval -o results.json
promptfoo share # get a shareable URL
```