feat(skills): add promptfoo skill for LLM evaluation and red-teaming
Covers install, configuration, running evals, red-teaming, CI/CD integration, and dataset generation. Pins to v0.121.17 with acquisition notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
124
plugins/kyberforge/skills/promptfoo/references/assertions.md
Normal file
124
plugins/kyberforge/skills/promptfoo/references/assertions.md
Normal file
@@ -0,0 +1,124 @@
|
||||
---
|
||||
topic: assertions
|
||||
source_keys:
|
||||
- context7-promptfoo-dev
|
||||
- context7-promptfoo-github
|
||||
---
|
||||
|
||||
## Assertion structure
|
||||
|
||||
Each assertion in `assert:` has a `type:`, an optional `value:`, an optional `threshold:`, and an optional `metric:` label.
|
||||
|
||||
```yaml
|
||||
assert:
|
||||
- type: contains
|
||||
value: 'return policy'
|
||||
- type: llm-rubric
|
||||
value: 'Response is helpful and professional'
|
||||
threshold: 0.8
|
||||
metric: quality
|
||||
```
|
||||
|
||||
## String shorthand
|
||||
|
||||
Assertions can also be written as compact strings directly in the assert list:
|
||||
|
||||
| Shorthand | Full type |
|
||||
|---|---|
|
||||
| `Paris` | `equals` |
|
||||
| `contains:Paris` | `contains` |
|
||||
| `icontains:paris` | `icontains` (case-insensitive) |
|
||||
| `starts-with:The answer` | `starts-with` |
|
||||
| `regex:^Hello.*world$` | `regex` |
|
||||
| `is-json` | `is-json` |
|
||||
| `contains-json` | `contains-json` |
|
||||
| `similar(0.8):Hello world` | `similar` with threshold |
|
||||
| `llm-rubric:Is helpful and accurate` | `llm-rubric` |
|
||||
| `grade:Does not mention being an AI` | alias for `llm-rubric` |
|
||||
| `factuality:Paris is the capital of France` | `factuality` |
|
||||
| `javascript:output.length < 100` | inline JS |
|
||||
| `fn:output.includes('hello')` | alias for `javascript` |
|
||||
| `python:len(output) > 10` | inline Python |
|
||||
| `file://assertions/custom.js` | external file |
|
||||
| `levenshtein(5):expected text` | `levenshtein` with distance |
|
||||
| `not-contains:error` | negated assertion |
|
||||
|
||||
## Deterministic assertions
|
||||
|
||||
- `equals` — exact string match
|
||||
- `contains` / `icontains` — substring check (case-sensitive / insensitive)
|
||||
- `not-contains` — absence check
|
||||
- `starts-with` — prefix check
|
||||
- `regex` — regular expression match
|
||||
- `is-json` — valid JSON
|
||||
- `contains-json` — valid JSON somewhere in output
|
||||
- `levenshtein` — edit distance within threshold
|
||||
|
||||
## Similarity and semantic assertions
|
||||
|
||||
- `similar` — embedding cosine similarity; `threshold:` is a 0–1 score
|
||||
- `context-faithfulness` — similarity-based RAG faithfulness; `threshold: 0.8` typical
|
||||
|
||||
## Model-graded assertions
|
||||
|
||||
These send a grader prompt to another LLM (by default the configured judge model) and score the output.
|
||||
|
||||
**`llm-rubric`** — open-ended rubric; binary or 0–1 score depending on criteria phrasing. Use `threshold:` to set minimum passing score:
|
||||
```yaml
|
||||
- type: llm-rubric
|
||||
value: Is not apologetic and provides a clear, concise answer
|
||||
threshold: 0.8
|
||||
```
|
||||
|
||||
**`factuality`** — checks whether the output is factually consistent with a reference statement:
|
||||
```yaml
|
||||
- type: factuality
|
||||
value: The capital of California is Sacramento
|
||||
```
|
||||
|
||||
**`pi`** — custom scoring with any numeric range; requires `threshold:`.
|
||||
|
||||
The grader model can be overridden globally in `defaultTest.options.provider` or per-assertion.
|
||||
|
||||
## Custom assertions
|
||||
|
||||
**JavaScript (inline):**
|
||||
```yaml
|
||||
- type: javascript
|
||||
value: "output.length < 100 && !output.includes('error')"
|
||||
```
|
||||
|
||||
**JavaScript (file):**
|
||||
```yaml
|
||||
- type: javascript
|
||||
value: file://assertions/check_format.js
|
||||
```
|
||||
|
||||
**Python:**
|
||||
```yaml
|
||||
- type: python
|
||||
value: "len(output) > 10 and 'Paris' in output"
|
||||
```
|
||||
|
||||
## Negation
|
||||
|
||||
Any assertion type can be negated by prepending `not-`:
|
||||
```yaml
|
||||
- type: not-contains
|
||||
value: 'error'
|
||||
- type: not-regex
|
||||
value: '\b(fail|broken)\b'
|
||||
```
|
||||
|
||||
## Metrics
|
||||
|
||||
The `metric:` field groups assertions for aggregate reporting. All assertions with the same metric name are scored together in the results view:
|
||||
```yaml
|
||||
assert:
|
||||
- type: contains
|
||||
value: 'policy'
|
||||
metric: coverage
|
||||
- type: llm-rubric
|
||||
value: Addresses the user's concern
|
||||
metric: quality
|
||||
```
|
||||
Reference in New Issue
Block a user