holocron/plugins/kyberforge/skills/promptfoo/references/assertions.md

topic, source_keys

topic	source_keys
assertions	context7-promptfoo-dev context7-promptfoo-github

Assertion structure

Each assertion in assert: has a type:, an optional value:, an optional threshold:, and an optional metric: label.

assert:
  - type: contains
    value: 'return policy'
  - type: llm-rubric
    value: 'Response is helpful and professional'
    threshold: 0.8
    metric: quality

String shorthand

Assertions can also be written as compact strings directly in the assert list:

Shorthand	Full type
Paris	equals
contains:Paris	contains
icontains:paris	icontains (case-insensitive)
starts-with:The answer	starts-with
regex:^Hello.*world$	regex
is-json	is-json
contains-json	contains-json
similar(0.8):Hello world	similar with threshold
llm-rubric:Is helpful and accurate	llm-rubric
grade:Does not mention being an AI	alias for llm-rubric
factuality:Paris is the capital of France	factuality
javascript:output.length < 100	inline JS
fn:output.includes('hello')	alias for javascript
python:len(output) > 10	inline Python
file://assertions/custom.js	external file
levenshtein(5):expected text	levenshtein with distance
not-contains:error	negated assertion

Deterministic assertions

• equals — exact string match
• contains / icontains — substring check (case-sensitive / insensitive)
• not-contains — absence check
• starts-with — prefix check
• regex — regular expression match
• is-json — valid JSON
• contains-json — valid JSON somewhere in output
• levenshtein — edit distance within threshold

Similarity and semantic assertions

• similar — embedding cosine similarity; threshold: is a 0–1 score
• context-faithfulness — similarity-based RAG faithfulness; threshold: 0.8 typical

Model-graded assertions

These send a grader prompt to another LLM (by default the configured judge model) and score the output.

llm-rubric — open-ended rubric; binary or 0–1 score depending on criteria phrasing. Use threshold: to set minimum passing score:

- type: llm-rubric
  value: Is not apologetic and provides a clear, concise answer
  threshold: 0.8

factuality — checks whether the output is factually consistent with a reference statement:

- type: factuality
  value: The capital of California is Sacramento

pi — custom scoring with any numeric range; requires threshold:.

The grader model can be overridden globally in defaultTest.options.provider or per-assertion.

Custom assertions

JavaScript (inline):

- type: javascript
  value: "output.length < 100 && !output.includes('error')"

JavaScript (file):

- type: javascript
  value: file://assertions/check_format.js

Python:

- type: python
  value: "len(output) > 10 and 'Paris' in output"

Negation

Any assertion type can be negated by prepending not-:

- type: not-contains
  value: 'error'
- type: not-regex
  value: '\b(fail|broken)\b'

Metrics

The metric: field groups assertions for aggregate reporting. All assertions with the same metric name are scored together in the results view:

assert:
  - type: contains
    value: 'policy'
    metric: coverage
  - type: llm-rubric
    value: Addresses the user's concern
    metric: quality