holocron/plugins/kyberforge/skills/promptfoo/references/examples.md

topic, source_keys

topic	source_keys
examples	context7-promptfoo-dev context7-promptfoo-github

Quickstart

npx promptfoo@0.121.17 init
# edit promptfooconfig.yaml
npx promptfoo@0.121.17 eval
npx promptfoo@0.121.17 view

Minimal config

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
prompts:
  - 'Answer the user question concisely. Question: {{question}}'

providers:
  - openai:gpt-5-mini

tests:
  - vars:
      question: How do I reset my password?
    assert:
      - type: contains
        value: reset
  - vars:
      question: Can I cancel my subscription today?
    assert:
      - type: llm-rubric
        value: The answer clearly explains the cancellation path.

Multi-provider comparison

providers:
  - openai:gpt-5-mini
  - anthropic:claude-3-haiku

prompts:
  - 'You are a helpful customer service agent. {{query}}'

tests:
  - vars:
      query: 'I need to return a product'
    assert:
      - type: contains
        value: 'return policy'
      - type: llm-rubric
        value: 'Response is helpful and professional'

Running this produces a side-by-side table with both models' outputs and assertion scores.

Loading tests from CSV

tests:
  - file://test_scenarios.csv

CSV format: one column per variable, header row must match {{variable}} names in the prompt. An __expected column maps to the equals assertion automatically.

Factuality evaluation

providers:
  - openai:gpt-5-mini
prompts:
  - |
    Please answer the following question accurately:
    Question: What is the capital of {{location}}?
tests:
  - vars:
      location: California
    assert:
      - type: factuality
        value: The capital of California is Sacramento

defaultTest for shared assertions

defaultTest:
  assert:
    - type: llm-rubric
      value: |
        Evaluate whether the response correctly answers the question.
        Question: {{ question }}
        Model Response: {{ output }}
        Correct Answer: {{ answer }}
        Grade accuracy 0.0–1.0. Pass if >= 0.8.
      threshold: 0.8

tests:
  - vars:
      question: What year did WW2 end?
      answer: '1945'
  - vars:
      question: What is the boiling point of water in Celsius?
      answer: '100'

Node.js API

import { evaluate } from 'promptfoo';

const evalRecord = await evaluate({
  prompts: ['Translate to Spanish: {{ text }}'],
  providers: ['openai:chat:gpt-5.5'],
  tests: [
    {
      vars: { text: 'Hello' },
      assert: [{ type: 'contains', value: 'Hola', metric: 'translation' }],
    },
  ],
});

const results = await evalRecord.toEvaluateSummary();
console.log(`Pass rate: ${results.stats.successes}/${results.results.length}`);

Generating test datasets with AI

# Generate test cases based on your prompt template
promptfoo generate dataset
promptfoo generate dataset --instructions "Consider edge cases related to international travel"
promptfoo generate dataset --output generated_tests.yaml

Saving and sharing results

outputPath: evaluations/results.html

Or via CLI:

promptfoo eval -o results.json
promptfoo share  # get a shareable URL