feat(skills): add promptfoo skill for LLM evaluation and red-teaming

Covers install, configuration, running evals, red-teaming, CI/CD
integration, and dataset generation. Pins to v0.121.17 with acquisition
notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-06-21 11:11:59 +00:00
parent 0155fcec26
commit 1ceacf17bc
12 changed files with 1195 additions and 0 deletions

View File

@@ -0,0 +1,127 @@
---
topic: ci-cd
source_keys:
- context7-promptfoo-dev
- context7-promptfoo-github
---
## How CI integration works
`promptfoo eval` exits with a non-zero code when any assertion fails. This makes it a natural CI gate — a failing eval blocks a merge just like a failing test suite.
## GitHub Actions — promptfoo-action
The official `promptfoo/promptfoo-action@v1` action runs an eval on every pull request and posts results as a PR comment.
```yaml
# .github/workflows/llm-eval.yml
name: LLM Eval
on:
pull_request:
workflow_dispatch:
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: promptfoo/promptfoo-action@v1
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
config: promptfooconfig.yaml
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
```
The `github-token` is required for the action to post PR comments. Without it the eval still runs but results are not surfaced in the PR UI.
## Generic CI (npx)
Any CI system that can run Node.js commands can use Promptfoo:
```yaml
# Any CI provider
- name: Run promptfoo eval
run: npx promptfoo@0.121.17 eval --no-cache
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
```
`--no-cache` ensures all LLM calls are fresh — important in CI where cached results from a developer's machine would not be present.
## Google Cloud / Vertex AI
```yaml
# .github/workflows/llm-test.yml
steps:
- uses: google-github-actions/auth@v2
with:
credentials_json: ${{ secrets.GCP_CREDENTIALS }}
- name: Run promptfoo tests
run: npx promptfoo@0.121.17 eval
env:
GOOGLE_CLOUD_PROJECT: ${{ vars.GCP_PROJECT_ID }}
GOOGLE_CLOUD_LOCATION: us-central1
```
## MCP security testing workflow
```yaml
# .github/workflows/security-test.yml
name: MCP Security Testing
on: [push, pull_request]
jobs:
security-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '22'
- run: npm install
- run: npm run build:all-servers
- name: Run security tests
run: |
npx promptfoo eval -c security-tests/scenario1.yaml
npx promptfoo eval -c security-tests/scenario2.yaml
```
## Model security scanning (SARIF output)
For repositories that include model files, scan them in CI and upload results to GitHub's security tab:
```yaml
- name: Install dependencies
run: |
npm install -g promptfoo
pip install modelaudit
- name: Scan models
run: |
promptfoo scan-model ./models/ \
--strict \
--no-write \
--format sarif \
--output model-scan-results.sarif
- name: Upload SARIF
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: model-scan-results.sarif
```
## GitLab CI and Jenkins
Both are supported. Use `npx promptfoo@0.121.17 eval` as the test command. No platform-specific action is needed — the exit code gates the pipeline natively.
## Recommended CI practices
- Always pass `--no-cache` in CI to avoid stale results
- Store API keys as CI secrets, never hardcode them
- Run evals on PR branches to catch regressions before merge
- Use `outputPath: results.json` and archive the artifact for debugging failed runs
- Set `evaluateOptions.maxConcurrency` low (2–5) in CI to avoid provider rate limits