feat(skills): add promptfoo skill for LLM evaluation and red-teaming
Covers install, configuration, running evals, red-teaming, CI/CD integration, and dataset generation. Pins to v0.121.17 with acquisition notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
127
plugins/kyberforge/skills/promptfoo/references/ci-cd.md
Normal file
127
plugins/kyberforge/skills/promptfoo/references/ci-cd.md
Normal file
@@ -0,0 +1,127 @@
|
||||
---
|
||||
topic: ci-cd
|
||||
source_keys:
|
||||
- context7-promptfoo-dev
|
||||
- context7-promptfoo-github
|
||||
---
|
||||
|
||||
## How CI integration works
|
||||
|
||||
`promptfoo eval` exits with a non-zero code when any assertion fails. This makes it a natural CI gate — a failing eval blocks a merge just like a failing test suite.
|
||||
|
||||
## GitHub Actions — promptfoo-action
|
||||
|
||||
The official `promptfoo/promptfoo-action@v1` action runs an eval on every pull request and posts results as a PR comment.
|
||||
|
||||
```yaml
|
||||
# .github/workflows/llm-eval.yml
|
||||
name: LLM Eval
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
eval:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: promptfoo/promptfoo-action@v1
|
||||
with:
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
config: promptfooconfig.yaml
|
||||
env:
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
```
|
||||
|
||||
The `github-token` is required for the action to post PR comments. Without it the eval still runs but results are not surfaced in the PR UI.
|
||||
|
||||
## Generic CI (npx)
|
||||
|
||||
Any CI system that can run Node.js commands can use Promptfoo:
|
||||
|
||||
```yaml
|
||||
# Any CI provider
|
||||
- name: Run promptfoo eval
|
||||
run: npx promptfoo@0.121.17 eval --no-cache
|
||||
env:
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
```
|
||||
|
||||
`--no-cache` ensures all LLM calls are fresh — important in CI where cached results from a developer's machine would not be present.
|
||||
|
||||
## Google Cloud / Vertex AI
|
||||
|
||||
```yaml
|
||||
# .github/workflows/llm-test.yml
|
||||
steps:
|
||||
- uses: google-github-actions/auth@v2
|
||||
with:
|
||||
credentials_json: ${{ secrets.GCP_CREDENTIALS }}
|
||||
- name: Run promptfoo tests
|
||||
run: npx promptfoo@0.121.17 eval
|
||||
env:
|
||||
GOOGLE_CLOUD_PROJECT: ${{ vars.GCP_PROJECT_ID }}
|
||||
GOOGLE_CLOUD_LOCATION: us-central1
|
||||
```
|
||||
|
||||
## MCP security testing workflow
|
||||
|
||||
```yaml
|
||||
# .github/workflows/security-test.yml
|
||||
name: MCP Security Testing
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
security-test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
- run: npm install
|
||||
- run: npm run build:all-servers
|
||||
- name: Run security tests
|
||||
run: |
|
||||
npx promptfoo eval -c security-tests/scenario1.yaml
|
||||
npx promptfoo eval -c security-tests/scenario2.yaml
|
||||
```
|
||||
|
||||
## Model security scanning (SARIF output)
|
||||
|
||||
For repositories that include model files, scan them in CI and upload results to GitHub's security tab:
|
||||
|
||||
```yaml
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
npm install -g promptfoo
|
||||
pip install modelaudit
|
||||
|
||||
- name: Scan models
|
||||
run: |
|
||||
promptfoo scan-model ./models/ \
|
||||
--strict \
|
||||
--no-write \
|
||||
--format sarif \
|
||||
--output model-scan-results.sarif
|
||||
|
||||
- name: Upload SARIF
|
||||
uses: github/codeql-action/upload-sarif@v3
|
||||
with:
|
||||
sarif_file: model-scan-results.sarif
|
||||
```
|
||||
|
||||
## GitLab CI and Jenkins
|
||||
|
||||
Both are supported. Use `npx promptfoo@0.121.17 eval` as the test command. No platform-specific action is needed — the exit code gates the pipeline natively.
|
||||
|
||||
## Recommended CI practices
|
||||
|
||||
- Always pass `--no-cache` in CI to avoid stale results
|
||||
- Store API keys as CI secrets, never hardcode them
|
||||
- Run evals on PR branches to catch regressions before merge
|
||||
- Use `outputPath: results.json` and archive the artifact for debugging failed runs
|
||||
- Set `evaluateOptions.maxConcurrency` low (2–5) in CI to avoid provider rate limits
|
||||
Reference in New Issue
Block a user