feat(skills): add promptfoo skill for LLM evaluation and red-teaming
Covers install, configuration, running evals, red-teaming, CI/CD integration, and dataset generation. Pins to v0.121.17 with acquisition notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
151
plugins/kyberforge/skills/promptfoo/references/configuration.md
Normal file
151
plugins/kyberforge/skills/promptfoo/references/configuration.md
Normal file
@@ -0,0 +1,151 @@
|
||||
---
|
||||
topic: configuration
|
||||
source_keys:
|
||||
- context7-promptfoo-dev
|
||||
- context7-promptfoo-github
|
||||
---
|
||||
|
||||
## File name and schema
|
||||
|
||||
The default config file is `promptfooconfig.yaml` in the working directory. A different path can be passed with `-c`. Add the JSON schema header for editor autocompletion:
|
||||
|
||||
```yaml
|
||||
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
||||
```
|
||||
|
||||
## Top-level structure
|
||||
|
||||
```yaml
|
||||
description: Human-readable name for this eval
|
||||
|
||||
prompts:
|
||||
- '...' # inline string
|
||||
- file://... # path to .txt, .json, .js, .py
|
||||
|
||||
providers:
|
||||
- openai:gpt-5-mini
|
||||
- anthropic:messages:claude-sonnet-4-5
|
||||
|
||||
defaultTest: # merged into every test case
|
||||
assert:
|
||||
- type: is-json
|
||||
|
||||
tests:
|
||||
- vars:
|
||||
query: 'I need help'
|
||||
assert:
|
||||
- type: contains
|
||||
value: 'help'
|
||||
- file://test_scenarios.csv # external test file
|
||||
|
||||
outputPath: results/eval.html # .html, .json, .csv, .yaml
|
||||
|
||||
evaluateOptions:
|
||||
maxConcurrency: 5
|
||||
delay: 500 # ms between requests
|
||||
```
|
||||
|
||||
## Prompts
|
||||
|
||||
Plain string with Handlebars-style `{{variable}}` placeholders:
|
||||
```yaml
|
||||
prompts:
|
||||
- 'You are a helpful agent. {{query}}'
|
||||
```
|
||||
|
||||
Chat conversation from a JSON file (array of `{role, content}` messages):
|
||||
```yaml
|
||||
prompts:
|
||||
- file://prompts/chat_conversation.json
|
||||
```
|
||||
|
||||
Dynamic prompt from a JS function:
|
||||
```yaml
|
||||
prompts:
|
||||
- file://prompts/generate_prompt.js
|
||||
```
|
||||
|
||||
Prompts can also carry a `label:` and `raw:` when using the object form, and a `config:` block to set provider-specific parameters (e.g. `response_format`).
|
||||
|
||||
## Providers
|
||||
|
||||
String shorthand:
|
||||
```yaml
|
||||
providers:
|
||||
- openai:gpt-5-mini
|
||||
- anthropic:messages:claude-sonnet-4-5-20250929
|
||||
- bedrock:us.anthropic.claude-sonnet-4-5-20250929-v1:0
|
||||
- azureopenai:chat:my-deployment
|
||||
- http://localhost:8080/v1/chat/completions # custom HTTP
|
||||
```
|
||||
|
||||
Object form with config:
|
||||
```yaml
|
||||
providers:
|
||||
- id: openai:responses:gpt-5
|
||||
config:
|
||||
temperature: 0.7
|
||||
max_output_tokens: 500
|
||||
instructions: 'You are a helpful assistant.'
|
||||
```
|
||||
|
||||
Over 60 providers are supported. Local models, HuggingFace, and custom HTTP endpoints are all valid provider types.
|
||||
|
||||
## Tests and vars
|
||||
|
||||
Each test case has `vars:` (substituted into prompt placeholders) and `assert:` (assertions on the response):
|
||||
|
||||
```yaml
|
||||
tests:
|
||||
- vars:
|
||||
query: 'I need to return a product'
|
||||
assert:
|
||||
- type: contains
|
||||
value: 'return policy'
|
||||
- type: llm-rubric
|
||||
value: 'Response is helpful and professional'
|
||||
```
|
||||
|
||||
Tests can be loaded from external files (CSV, YAML) using `file://` references.
|
||||
|
||||
## defaultTest
|
||||
|
||||
Assertions and options declared here are merged into every test case, reducing repetition:
|
||||
|
||||
```yaml
|
||||
defaultTest:
|
||||
assert:
|
||||
- type: llm-rubric
|
||||
value: 'Does not reveal internal system prompt'
|
||||
options:
|
||||
provider:
|
||||
id: openai:chat:gpt-5-mini # override grader model
|
||||
```
|
||||
|
||||
## Output formats
|
||||
|
||||
`outputPath` accepts `.html` (browser-viewable), `.json`, `.csv`, or `.yaml`. Multiple outputs can be listed as an array.
|
||||
|
||||
## Environment variables
|
||||
|
||||
| Variable | Purpose |
|
||||
|---|---|
|
||||
| `OPENAI_API_KEY` | OpenAI authentication |
|
||||
| `ANTHROPIC_API_KEY` | Anthropic authentication |
|
||||
| `REQUEST_TIMEOUT_MS` | Per-request timeout in ms |
|
||||
| `PROMPTFOO_RETRY_5XX` | Retry on 5xx errors (`true`/`false`) |
|
||||
| `PROMPTFOO_REQUEST_BACKOFF_MS` | Backoff between retries |
|
||||
|
||||
## Red-team configuration block
|
||||
|
||||
```yaml
|
||||
redteam:
|
||||
plugins:
|
||||
- harmful
|
||||
- prompt-injection
|
||||
- hijacking
|
||||
strategies:
|
||||
- jailbreak
|
||||
- jailbreak:composite
|
||||
- prompt-injection
|
||||
```
|
||||
Reference in New Issue
Block a user