holocron/plugins/kyberforge/skills/promptfoo/references/ci-cd.md

topic, source_keys

topic	source_keys
ci-cd	context7-promptfoo-dev context7-promptfoo-github

How CI integration works

promptfoo eval exits with a non-zero code when any assertion fails. This makes it a natural CI gate — a failing eval blocks a merge just like a failing test suite.

GitHub Actions — promptfoo-action

The official promptfoo/promptfoo-action@v1 action runs an eval on every pull request and posts results as a PR comment.

# .github/workflows/llm-eval.yml
name: LLM Eval

on:
  pull_request:
  workflow_dispatch:

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: promptfoo/promptfoo-action@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          config: promptfooconfig.yaml
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

The github-token is required for the action to post PR comments. Without it the eval still runs but results are not surfaced in the PR UI.

Generic CI (npx)

Any CI system that can run Node.js commands can use Promptfoo:

# Any CI provider
- name: Run promptfoo eval
  run: npx promptfoo@0.121.17 eval --no-cache
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

--no-cache ensures all LLM calls are fresh — important in CI where cached results from a developer's machine would not be present.

Google Cloud / Vertex AI

# .github/workflows/llm-test.yml
steps:
  - uses: google-github-actions/auth@v2
    with:
      credentials_json: ${{ secrets.GCP_CREDENTIALS }}
  - name: Run promptfoo tests
    run: npx promptfoo@0.121.17 eval
    env:
      GOOGLE_CLOUD_PROJECT: ${{ vars.GCP_PROJECT_ID }}
      GOOGLE_CLOUD_LOCATION: us-central1

MCP security testing workflow

# .github/workflows/security-test.yml
name: MCP Security Testing

on: [push, pull_request]

jobs:
  security-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '22'
      - run: npm install
      - run: npm run build:all-servers
      - name: Run security tests
        run: |
          npx promptfoo eval -c security-tests/scenario1.yaml
          npx promptfoo eval -c security-tests/scenario2.yaml

Model security scanning (SARIF output)

For repositories that include model files, scan them in CI and upload results to GitHub's security tab:

- name: Install dependencies
  run: |
    npm install -g promptfoo
    pip install modelaudit

- name: Scan models
  run: |
    promptfoo scan-model ./models/ \
      --strict \
      --no-write \
      --format sarif \
      --output model-scan-results.sarif

- name: Upload SARIF
  uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: model-scan-results.sarif

GitLab CI and Jenkins

Both are supported. Use npx promptfoo@0.121.17 eval as the test command. No platform-specific action is needed — the exit code gates the pipeline natively.

Recommended CI practices

• Always pass --no-cache in CI to avoid stale results
• Store API keys as CI secrets, never hardcode them
• Run evals on PR branches to catch regressions before merge
• Use outputPath: results.json and archive the artifact for debugging failed runs
• Set evaluateOptions.maxConcurrency low (2–5) in CI to avoid provider rate limits