holocron/plugins/kyberforge/skills/promptfoo/references/configuration.md

topic, source_keys

topic	source_keys
configuration	context7-promptfoo-dev context7-promptfoo-github

File name and schema

The default config file is promptfooconfig.yaml in the working directory. A different path can be passed with -c. Add the JSON schema header for editor autocompletion:

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json

Top-level structure

description: Human-readable name for this eval

prompts:
  - '...'          # inline string
  - file://...     # path to .txt, .json, .js, .py

providers:
  - openai:gpt-5-mini
  - anthropic:messages:claude-sonnet-4-5

defaultTest:       # merged into every test case
  assert:
    - type: is-json

tests:
  - vars:
      query: 'I need help'
    assert:
      - type: contains
        value: 'help'
  - file://test_scenarios.csv  # external test file

outputPath: results/eval.html  # .html, .json, .csv, .yaml

evaluateOptions:
  maxConcurrency: 5
  delay: 500       # ms between requests

Prompts

Plain string with Handlebars-style {{variable}} placeholders:

prompts:
  - 'You are a helpful agent. {{query}}'

Chat conversation from a JSON file (array of {role, content} messages):

prompts:
  - file://prompts/chat_conversation.json

Dynamic prompt from a JS function:

prompts:
  - file://prompts/generate_prompt.js

Prompts can also carry a label: and raw: when using the object form, and a config: block to set provider-specific parameters (e.g. response_format).

Providers

String shorthand:

providers:
  - openai:gpt-5-mini
  - anthropic:messages:claude-sonnet-4-5-20250929
  - bedrock:us.anthropic.claude-sonnet-4-5-20250929-v1:0
  - azureopenai:chat:my-deployment
  - http://localhost:8080/v1/chat/completions  # custom HTTP

Object form with config:

providers:
  - id: openai:responses:gpt-5
    config:
      temperature: 0.7
      max_output_tokens: 500
      instructions: 'You are a helpful assistant.'

Over 60 providers are supported. Local models, HuggingFace, and custom HTTP endpoints are all valid provider types.

Tests and vars

Each test case has vars: (substituted into prompt placeholders) and assert: (assertions on the response):

tests:
  - vars:
      query: 'I need to return a product'
    assert:
      - type: contains
        value: 'return policy'
      - type: llm-rubric
        value: 'Response is helpful and professional'

Tests can be loaded from external files (CSV, YAML) using file:// references.

defaultTest

Assertions and options declared here are merged into every test case, reducing repetition:

defaultTest:
  assert:
    - type: llm-rubric
      value: 'Does not reveal internal system prompt'
  options:
    provider:
      id: openai:chat:gpt-5-mini  # override grader model

Output formats

outputPath accepts .html (browser-viewable), .json, .csv, or .yaml. Multiple outputs can be listed as an array.

Environment variables

Variable	Purpose
OPENAI_API_KEY	OpenAI authentication
ANTHROPIC_API_KEY	Anthropic authentication
REQUEST_TIMEOUT_MS	Per-request timeout in ms
PROMPTFOO_RETRY_5XX	Retry on 5xx errors (true/false)
PROMPTFOO_REQUEST_BACKOFF_MS	Backoff between retries

Red-team configuration block

redteam:
  plugins:
    - harmful
    - prompt-injection
    - hijacking
  strategies:
    - jailbreak
    - jailbreak:composite
    - prompt-injection