Defame1297
1ceacf17bc
Covers install, configuration, running evals, red-teaming, CI/CD integration, and dataset generation. Pins to v0.121.17 with acquisition notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
125 lines
3.4 KiB
Markdown
125 lines
3.4 KiB
Markdown
---
|
||
topic: assertions
|
||
source_keys:
|
||
- context7-promptfoo-dev
|
||
- context7-promptfoo-github
|
||
---
|
||
|
||
## Assertion structure
|
||
|
||
Each assertion in `assert:` has a `type:`, an optional `value:`, an optional `threshold:`, and an optional `metric:` label.
|
||
|
||
```yaml
|
||
assert:
|
||
- type: contains
|
||
value: 'return policy'
|
||
- type: llm-rubric
|
||
value: 'Response is helpful and professional'
|
||
threshold: 0.8
|
||
metric: quality
|
||
```
|
||
|
||
## String shorthand
|
||
|
||
Assertions can also be written as compact strings directly in the assert list:
|
||
|
||
| Shorthand | Full type |
|
||
|---|---|
|
||
| `Paris` | `equals` |
|
||
| `contains:Paris` | `contains` |
|
||
| `icontains:paris` | `icontains` (case-insensitive) |
|
||
| `starts-with:The answer` | `starts-with` |
|
||
| `regex:^Hello.*world$` | `regex` |
|
||
| `is-json` | `is-json` |
|
||
| `contains-json` | `contains-json` |
|
||
| `similar(0.8):Hello world` | `similar` with threshold |
|
||
| `llm-rubric:Is helpful and accurate` | `llm-rubric` |
|
||
| `grade:Does not mention being an AI` | alias for `llm-rubric` |
|
||
| `factuality:Paris is the capital of France` | `factuality` |
|
||
| `javascript:output.length < 100` | inline JS |
|
||
| `fn:output.includes('hello')` | alias for `javascript` |
|
||
| `python:len(output) > 10` | inline Python |
|
||
| `file://assertions/custom.js` | external file |
|
||
| `levenshtein(5):expected text` | `levenshtein` with distance |
|
||
| `not-contains:error` | negated assertion |
|
||
|
||
## Deterministic assertions
|
||
|
||
- `equals` — exact string match
|
||
- `contains` / `icontains` — substring check (case-sensitive / insensitive)
|
||
- `not-contains` — absence check
|
||
- `starts-with` — prefix check
|
||
- `regex` — regular expression match
|
||
- `is-json` — valid JSON
|
||
- `contains-json` — valid JSON somewhere in output
|
||
- `levenshtein` — edit distance within threshold
|
||
|
||
## Similarity and semantic assertions
|
||
|
||
- `similar` — embedding cosine similarity; `threshold:` is a 0–1 score
|
||
- `context-faithfulness` — similarity-based RAG faithfulness; `threshold: 0.8` typical
|
||
|
||
## Model-graded assertions
|
||
|
||
These send a grader prompt to another LLM (by default the configured judge model) and score the output.
|
||
|
||
**`llm-rubric`** — open-ended rubric; binary or 0–1 score depending on criteria phrasing. Use `threshold:` to set minimum passing score:
|
||
```yaml
|
||
- type: llm-rubric
|
||
value: Is not apologetic and provides a clear, concise answer
|
||
threshold: 0.8
|
||
```
|
||
|
||
**`factuality`** — checks whether the output is factually consistent with a reference statement:
|
||
```yaml
|
||
- type: factuality
|
||
value: The capital of California is Sacramento
|
||
```
|
||
|
||
**`pi`** — custom scoring with any numeric range; requires `threshold:`.
|
||
|
||
The grader model can be overridden globally in `defaultTest.options.provider` or per-assertion.
|
||
|
||
## Custom assertions
|
||
|
||
**JavaScript (inline):**
|
||
```yaml
|
||
- type: javascript
|
||
value: "output.length < 100 && !output.includes('error')"
|
||
```
|
||
|
||
**JavaScript (file):**
|
||
```yaml
|
||
- type: javascript
|
||
value: file://assertions/check_format.js
|
||
```
|
||
|
||
**Python:**
|
||
```yaml
|
||
- type: python
|
||
value: "len(output) > 10 and 'Paris' in output"
|
||
```
|
||
|
||
## Negation
|
||
|
||
Any assertion type can be negated by prepending `not-`:
|
||
```yaml
|
||
- type: not-contains
|
||
value: 'error'
|
||
- type: not-regex
|
||
value: '\b(fail|broken)\b'
|
||
```
|
||
|
||
## Metrics
|
||
|
||
The `metric:` field groups assertions for aggregate reporting. All assertions with the same metric name are scored together in the results view:
|
||
```yaml
|
||
assert:
|
||
- type: contains
|
||
value: 'policy'
|
||
metric: coverage
|
||
- type: llm-rubric
|
||
value: Addresses the user's concern
|
||
metric: quality
|
||
```
|