Defame1297
1ceacf17bc
Covers install, configuration, running evals, red-teaming, CI/CD integration, and dataset generation. Pins to v0.121.17 with acquisition notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
128 lines
3.4 KiB
Markdown
128 lines
3.4 KiB
Markdown
---
|
||
topic: ci-cd
|
||
source_keys:
|
||
- context7-promptfoo-dev
|
||
- context7-promptfoo-github
|
||
---
|
||
|
||
## How CI integration works
|
||
|
||
`promptfoo eval` exits with a non-zero code when any assertion fails. This makes it a natural CI gate — a failing eval blocks a merge just like a failing test suite.
|
||
|
||
## GitHub Actions — promptfoo-action
|
||
|
||
The official `promptfoo/promptfoo-action@v1` action runs an eval on every pull request and posts results as a PR comment.
|
||
|
||
```yaml
|
||
# .github/workflows/llm-eval.yml
|
||
name: LLM Eval
|
||
|
||
on:
|
||
pull_request:
|
||
workflow_dispatch:
|
||
|
||
jobs:
|
||
eval:
|
||
runs-on: ubuntu-latest
|
||
steps:
|
||
- uses: actions/checkout@v4
|
||
- uses: promptfoo/promptfoo-action@v1
|
||
with:
|
||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||
config: promptfooconfig.yaml
|
||
env:
|
||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||
```
|
||
|
||
The `github-token` is required for the action to post PR comments. Without it the eval still runs but results are not surfaced in the PR UI.
|
||
|
||
## Generic CI (npx)
|
||
|
||
Any CI system that can run Node.js commands can use Promptfoo:
|
||
|
||
```yaml
|
||
# Any CI provider
|
||
- name: Run promptfoo eval
|
||
run: npx promptfoo@0.121.17 eval --no-cache
|
||
env:
|
||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||
```
|
||
|
||
`--no-cache` ensures all LLM calls are fresh — important in CI where cached results from a developer's machine would not be present.
|
||
|
||
## Google Cloud / Vertex AI
|
||
|
||
```yaml
|
||
# .github/workflows/llm-test.yml
|
||
steps:
|
||
- uses: google-github-actions/auth@v2
|
||
with:
|
||
credentials_json: ${{ secrets.GCP_CREDENTIALS }}
|
||
- name: Run promptfoo tests
|
||
run: npx promptfoo@0.121.17 eval
|
||
env:
|
||
GOOGLE_CLOUD_PROJECT: ${{ vars.GCP_PROJECT_ID }}
|
||
GOOGLE_CLOUD_LOCATION: us-central1
|
||
```
|
||
|
||
## MCP security testing workflow
|
||
|
||
```yaml
|
||
# .github/workflows/security-test.yml
|
||
name: MCP Security Testing
|
||
|
||
on: [push, pull_request]
|
||
|
||
jobs:
|
||
security-test:
|
||
runs-on: ubuntu-latest
|
||
steps:
|
||
- uses: actions/checkout@v4
|
||
- uses: actions/setup-node@v4
|
||
with:
|
||
node-version: '22'
|
||
- run: npm install
|
||
- run: npm run build:all-servers
|
||
- name: Run security tests
|
||
run: |
|
||
npx promptfoo eval -c security-tests/scenario1.yaml
|
||
npx promptfoo eval -c security-tests/scenario2.yaml
|
||
```
|
||
|
||
## Model security scanning (SARIF output)
|
||
|
||
For repositories that include model files, scan them in CI and upload results to GitHub's security tab:
|
||
|
||
```yaml
|
||
- name: Install dependencies
|
||
run: |
|
||
npm install -g promptfoo
|
||
pip install modelaudit
|
||
|
||
- name: Scan models
|
||
run: |
|
||
promptfoo scan-model ./models/ \
|
||
--strict \
|
||
--no-write \
|
||
--format sarif \
|
||
--output model-scan-results.sarif
|
||
|
||
- name: Upload SARIF
|
||
uses: github/codeql-action/upload-sarif@v3
|
||
with:
|
||
sarif_file: model-scan-results.sarif
|
||
```
|
||
|
||
## GitLab CI and Jenkins
|
||
|
||
Both are supported. Use `npx promptfoo@0.121.17 eval` as the test command. No platform-specific action is needed — the exit code gates the pipeline natively.
|
||
|
||
## Recommended CI practices
|
||
|
||
- Always pass `--no-cache` in CI to avoid stale results
|
||
- Store API keys as CI secrets, never hardcode them
|
||
- Run evals on PR branches to catch regressions before merge
|
||
- Use `outputPath: results.json` and archive the artifact for debugging failed runs
|
||
- Set `evaluateOptions.maxConcurrency` low (2–5) in CI to avoid provider rate limits
|