feat(skills): add promptfoo skill for LLM evaluation and red-teaming

Covers install, configuration, running evals, red-teaming, CI/CD
integration, and dataset generation. Pins to v0.121.17 with acquisition
notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-06-21 11:11:59 +00:00
parent 0155fcec26
commit 1ceacf17bc
12 changed files with 1195 additions and 0 deletions

View File

@@ -0,0 +1,124 @@
---
topic: assertions
source_keys:
- context7-promptfoo-dev
- context7-promptfoo-github
---
## Assertion structure
Each assertion in `assert:` has a `type:`, an optional `value:`, an optional `threshold:`, and an optional `metric:` label.
```yaml
assert:
- type: contains
value: 'return policy'
- type: llm-rubric
value: 'Response is helpful and professional'
threshold: 0.8
metric: quality
```
## String shorthand
Assertions can also be written as compact strings directly in the assert list:
| Shorthand | Full type |
|---|---|
| `Paris` | `equals` |
| `contains:Paris` | `contains` |
| `icontains:paris` | `icontains` (case-insensitive) |
| `starts-with:The answer` | `starts-with` |
| `regex:^Hello.*world$` | `regex` |
| `is-json` | `is-json` |
| `contains-json` | `contains-json` |
| `similar(0.8):Hello world` | `similar` with threshold |
| `llm-rubric:Is helpful and accurate` | `llm-rubric` |
| `grade:Does not mention being an AI` | alias for `llm-rubric` |
| `factuality:Paris is the capital of France` | `factuality` |
| `javascript:output.length < 100` | inline JS |
| `fn:output.includes('hello')` | alias for `javascript` |
| `python:len(output) > 10` | inline Python |
| `file://assertions/custom.js` | external file |
| `levenshtein(5):expected text` | `levenshtein` with distance |
| `not-contains:error` | negated assertion |
## Deterministic assertions
- `equals` — exact string match
- `contains` / `icontains` — substring check (case-sensitive / insensitive)
- `not-contains` — absence check
- `starts-with` — prefix check
- `regex` — regular expression match
- `is-json` — valid JSON
- `contains-json` — valid JSON somewhere in output
- `levenshtein` — edit distance within threshold
## Similarity and semantic assertions
- `similar` — embedding cosine similarity; `threshold:` is a 0–1 score
- `context-faithfulness` — similarity-based RAG faithfulness; `threshold: 0.8` typical
## Model-graded assertions
These send a grader prompt to another LLM (by default the configured judge model) and score the output.
**`llm-rubric`** — open-ended rubric; binary or 0–1 score depending on criteria phrasing. Use `threshold:` to set minimum passing score:
```yaml
- type: llm-rubric
value: Is not apologetic and provides a clear, concise answer
threshold: 0.8
```
**`factuality`** — checks whether the output is factually consistent with a reference statement:
```yaml
- type: factuality
value: The capital of California is Sacramento
```
**`pi`** — custom scoring with any numeric range; requires `threshold:`.
The grader model can be overridden globally in `defaultTest.options.provider` or per-assertion.
## Custom assertions
**JavaScript (inline):**
```yaml
- type: javascript
value: "output.length < 100 && !output.includes('error')"
```
**JavaScript (file):**
```yaml
- type: javascript
value: file://assertions/check_format.js
```
**Python:**
```yaml
- type: python
value: "len(output) > 10 and 'Paris' in output"
```
## Negation
Any assertion type can be negated by prepending `not-`:
```yaml
- type: not-contains
value: 'error'
- type: not-regex
value: '\b(fail|broken)\b'
```
## Metrics
The `metric:` field groups assertions for aggregate reporting. All assertions with the same metric name are scored together in the results view:
```yaml
assert:
- type: contains
value: 'policy'
metric: coverage
- type: llm-rubric
value: Addresses the user's concern
metric: quality
```