feat(skills): add promptfoo skill for LLM evaluation and red-teaming
Covers install, configuration, running evals, red-teaming, CI/CD integration, and dataset generation. Pins to v0.121.17 with acquisition notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
150
plugins/kyberforge/skills/promptfoo/references/examples.md
Normal file
150
plugins/kyberforge/skills/promptfoo/references/examples.md
Normal file
@@ -0,0 +1,150 @@
|
||||
---
|
||||
topic: examples
|
||||
source_keys:
|
||||
- context7-promptfoo-dev
|
||||
- context7-promptfoo-github
|
||||
---
|
||||
|
||||
## Quickstart
|
||||
|
||||
```bash
|
||||
npx promptfoo@0.121.17 init
|
||||
# edit promptfooconfig.yaml
|
||||
npx promptfoo@0.121.17 eval
|
||||
npx promptfoo@0.121.17 view
|
||||
```
|
||||
|
||||
## Minimal config
|
||||
|
||||
```yaml
|
||||
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
||||
prompts:
|
||||
- 'Answer the user question concisely. Question: {{question}}'
|
||||
|
||||
providers:
|
||||
- openai:gpt-5-mini
|
||||
|
||||
tests:
|
||||
- vars:
|
||||
question: How do I reset my password?
|
||||
assert:
|
||||
- type: contains
|
||||
value: reset
|
||||
- vars:
|
||||
question: Can I cancel my subscription today?
|
||||
assert:
|
||||
- type: llm-rubric
|
||||
value: The answer clearly explains the cancellation path.
|
||||
```
|
||||
|
||||
## Multi-provider comparison
|
||||
|
||||
```yaml
|
||||
providers:
|
||||
- openai:gpt-5-mini
|
||||
- anthropic:claude-3-haiku
|
||||
|
||||
prompts:
|
||||
- 'You are a helpful customer service agent. {{query}}'
|
||||
|
||||
tests:
|
||||
- vars:
|
||||
query: 'I need to return a product'
|
||||
assert:
|
||||
- type: contains
|
||||
value: 'return policy'
|
||||
- type: llm-rubric
|
||||
value: 'Response is helpful and professional'
|
||||
```
|
||||
|
||||
Running this produces a side-by-side table with both models' outputs and assertion scores.
|
||||
|
||||
## Loading tests from CSV
|
||||
|
||||
```yaml
|
||||
tests:
|
||||
- file://test_scenarios.csv
|
||||
```
|
||||
|
||||
CSV format: one column per variable, header row must match `{{variable}}` names in the prompt. An `__expected` column maps to the `equals` assertion automatically.
|
||||
|
||||
## Factuality evaluation
|
||||
|
||||
```yaml
|
||||
providers:
|
||||
- openai:gpt-5-mini
|
||||
prompts:
|
||||
- |
|
||||
Please answer the following question accurately:
|
||||
Question: What is the capital of {{location}}?
|
||||
tests:
|
||||
- vars:
|
||||
location: California
|
||||
assert:
|
||||
- type: factuality
|
||||
value: The capital of California is Sacramento
|
||||
```
|
||||
|
||||
## defaultTest for shared assertions
|
||||
|
||||
```yaml
|
||||
defaultTest:
|
||||
assert:
|
||||
- type: llm-rubric
|
||||
value: |
|
||||
Evaluate whether the response correctly answers the question.
|
||||
Question: {{ question }}
|
||||
Model Response: {{ output }}
|
||||
Correct Answer: {{ answer }}
|
||||
Grade accuracy 0.0–1.0. Pass if >= 0.8.
|
||||
threshold: 0.8
|
||||
|
||||
tests:
|
||||
- vars:
|
||||
question: What year did WW2 end?
|
||||
answer: '1945'
|
||||
- vars:
|
||||
question: What is the boiling point of water in Celsius?
|
||||
answer: '100'
|
||||
```
|
||||
|
||||
## Node.js API
|
||||
|
||||
```javascript
|
||||
import { evaluate } from 'promptfoo';
|
||||
|
||||
const evalRecord = await evaluate({
|
||||
prompts: ['Translate to Spanish: {{ text }}'],
|
||||
providers: ['openai:chat:gpt-5.5'],
|
||||
tests: [
|
||||
{
|
||||
vars: { text: 'Hello' },
|
||||
assert: [{ type: 'contains', value: 'Hola', metric: 'translation' }],
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
const results = await evalRecord.toEvaluateSummary();
|
||||
console.log(`Pass rate: ${results.stats.successes}/${results.results.length}`);
|
||||
```
|
||||
|
||||
## Generating test datasets with AI
|
||||
|
||||
```bash
|
||||
# Generate test cases based on your prompt template
|
||||
promptfoo generate dataset
|
||||
promptfoo generate dataset --instructions "Consider edge cases related to international travel"
|
||||
promptfoo generate dataset --output generated_tests.yaml
|
||||
```
|
||||
|
||||
## Saving and sharing results
|
||||
|
||||
```yaml
|
||||
outputPath: evaluations/results.html
|
||||
```
|
||||
|
||||
Or via CLI:
|
||||
```bash
|
||||
promptfoo eval -o results.json
|
||||
promptfoo share # get a shareable URL
|
||||
```
|
||||
Reference in New Issue
Block a user