Defame1297/holocron
0
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases 3 Wiki Activity

feat(skills): add promptfoo skill for LLM evaluation and red-teaming

Browse Source 
Covers install, configuration, running evals, red-teaming, CI/CD
integration, and dataset generation. Pins to v0.121.17 with acquisition
notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Defame1297
2026-06-21 11:11:59 +00:00
parent 0155fcec26
commit 1ceacf17bc
12 changed files with 1195 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
plugins/kyberforge/skills/promptfoo/META.md Normal file
View File

@@ -0,0 +1,18 @@
```yaml
version: "1.0"
updated: 2026-06-21
when: >
Invoked explicitly as /promptfoo, or implicitly when the user wants to install,
configure, run evaluations, red-team an LLM application, or integrate LLM testing
into CI/CD using Promptfoo. Also triggered when the user wants to compare model
outputs side-by-side across providers.
references:
- https://promptfoo.dev/docs/getting-started
- https://promptfoo.dev/docs/configuration/parameters
- https://promptfoo.dev/docs/usage/command-line
- https://promptfoo.dev/docs/configuration/expected-outputs
- https://promptfoo.dev/docs/red-team/
- https://promptfoo.dev/docs/category/integrations
```

107
plugins/kyberforge/skills/promptfoo/SKILL.md Normal file
View File

@@ -0,0 +1,107 @@
---
name: promptfoo
description: >
Use when the user wants to install, set up, configure, or work with Promptfoo —
the open-source LLM evaluation CLI. Covers: project initialization (promptfoo init),
writing or editing promptfooconfig.yaml, running evaluations (promptfoo eval),
comparing model outputs side-by-side, viewing and sharing results, generating test
datasets, red-teaming LLM applications with adversarial plugins and strategies, and
integrating Promptfoo into CI/CD pipelines. Do NOT use when the user wants to evaluate
LLM output quality in general without Promptfoo, write tests for non-LLM code (use
tdd), or set up a different evaluation framework (RAGAS, OpenAI Evals, etc.).
metadata:
category: test
---
<requirements>
## Required inputs
- **Task scope** — what the user wants to do (install, configure, run eval, red-team, CI
integration, or debug); inferred from request, ask only if genuinely ambiguous
- **Config file path** — defaults to `promptfooconfig.yaml` in the working directory;
user provides if non-standard
- **Provider API keys** — must be set as environment variables before running evals;
remind the user if not already set
## Constraints
- Always use the pinned version `0.121.17` — never suggest `@latest` or an unpinned
install; see `references/installation.md` for rationale (OpenAI acquisition, March 2026)
- API keys and credentials must always reference environment variables in examples and
config — never hardcoded values, even as placeholders like `sk-abc123`
- Warn when the user's eval workload is non-OpenAI-centric: note the acquisition and
mention DeepEval / Arize Phoenix as documented fallbacks
- Offload detail to references/ files — do not reproduce CLI flags, assertion types, or
redteam plugin tables inline; wire each reference file in the step that needs it
</requirements>
<steps>
## Process
1. **Clarify scope.** Identify what the user wants: install/setup, configure
`promptfooconfig.yaml`, run an eval, generate a dataset, red-team, integrate CI/CD,
or debug a failing run. Infer from context — ask only if genuinely ambiguous.
2. **Install / initialise (if needed).** Check whether Promptfoo is already installed.
If not, guide through installation using the pinned version. To scaffold a new
project: `npx promptfoo@0.121.17 init`. See `references/installation.md` for
prerequisites, global/local install options, and the acquisition notice.
3. **Configure.** Help the user write or edit `promptfooconfig.yaml`. Cover: prompts
(inline, file://, JS/Python), providers (string shorthand and object form),
test cases with vars and assert, defaultTest, outputPath, and evaluateOptions.
See `references/configuration.md` for the full structure and
`references/assertions.md` for assertion types and shorthand syntax.
4. **Run evaluation.** Execute `npx promptfoo@0.121.17 eval` (with `--no-cache` in CI).
After the run, open results with `promptfoo view` or share with `promptfoo share`.
For rate-limit issues, suggest `--max-concurrency` and `--delay` flags.
See `references/cli-reference.md` for all flags and subcommands.
5. **Red-team (if requested).** Add a `redteam:` block to the config with the
appropriate plugins and strategies. Run `promptfoo redteam generate` to produce
adversarial test cases, then `promptfoo eval` to execute them.
See `references/redteam.md` for plugin/strategy tables and OWASP/MITRE mappings.
6. **CI integration (if requested).** Write a GitHub Actions workflow using
`promptfoo/promptfoo-action@v1` or a generic `npx promptfoo@0.121.17 eval --no-cache`
step. Always use CI secrets for API keys.
See `references/ci-cd.md` for full workflow templates.
7. **Generate test dataset (if requested).** Run `promptfoo generate dataset` with
optional `--instructions` to AI-generate test cases from the prompt template.
See `references/cli-reference.md` for flags.
## Output format
- Modified or newly created `promptfooconfig.yaml`
- Shell commands to run, with the pinned version and relevant flags
- GitHub Actions workflow file (`.github/workflows/llm-eval.yml`) when CI is in scope
</steps>
<checks>
## Failure handling
- No `promptfooconfig.yaml` found and user did not ask to create one — ask before
scaffolding; do not overwrite existing config without confirmation
- Eval fails with rate-limit errors — suggest `--max-concurrency 2 --delay 3000` and
`PROMPTFOO_RETRY_5XX=true`; see `references/cli-reference.md`
- User requests `@latest` or an unpinned install — correct to `0.121.17` and explain
the version-pinning rationale (acquisition notice)
- User requests hardcoded API key in config or example — refuse and redirect to
environment variable pattern
## Self-check
- [ ] Pinned version `0.121.17` used in all commands — no `@latest`
- [ ] No credentials appear in any output, config example, or shell command
- [ ] Non-OpenAI eval workloads include the acquisition warning
- [ ] Each references/ file wired in the step that uses it
- [ ] Config written or modified only after confirming path with user
</checks>

112
plugins/kyberforge/skills/promptfoo/evals/eval.yaml Normal file
View File

@@ -0,0 +1,112 @@
skill_name: promptfoo
trigger_tests:
- id: explicit-setup
name: "Explicit trigger — project setup"
query: "set up promptfoo in my project"
should_trigger: true
- id: explicit-run-eval
name: "Explicit trigger — run eval"
query: "help me run a promptfoo eval"
should_trigger: true
- id: implicit-model-comparison
name: "Implicit trigger — model comparison"
query: "I want to compare GPT-4 and Claude on the same test cases"
should_trigger: true
- id: implicit-ci-llm-testing
name: "Implicit trigger — CI LLM testing"
query: "how do I add LLM testing to my pull request pipeline?"
should_trigger: true
- id: negative-express-tests
name: "Negative trigger — non-LLM integration tests"
query: "How do I write integration tests for my Express API?"
should_trigger: false
- id: negative-ragas
name: "Negative trigger — different eval framework"
query: "Set up RAGAS for evaluating my RAG pipeline"
should_trigger: false
- id: negative-general-llm-eval
name: "Negative trigger — general LLM quality question"
query: "What's the best way to evaluate LLM output quality?"
should_trigger: false
output_tests:
- id: output-pinned-version
name: "Deterministic — pinned version used, @latest absent"
type: deterministic
prompt: "install promptfoo"
expected_output: >
Shell commands that install or invoke Promptfoo using the pinned version
0.121.17. No @latest tag appears anywhere in the output.
assertions:
- "Output contains the string '0.121.17'"
- "Output does NOT contain '@latest'"
- "Output does NOT contain 'promptfoo@latest'"
- id: output-config-structure
name: "Deterministic — valid promptfooconfig.yaml structure"
type: deterministic
prompt: "create a promptfooconfig.yaml for comparing two models on my customer support prompts"
expected_output: >
A YAML configuration file with all required top-level keys for a working
promptfoo evaluation: prompts, providers (at least two entries), tests with
at least one test case containing vars and assert.
assertions:
- "Output contains 'prompts:'"
- "Output contains 'providers:'"
- "Output contains 'tests:'"
- "Output contains 'assert:'"
- id: output-no-hardcoded-keys
name: "Deterministic — no hardcoded API keys in output"
type: deterministic
prompt: "set up promptfoo with OpenAI and show me the config"
expected_output: >
A setup guide and config that references API keys only via environment
variables. No literal API key values, no placeholder strings like sk-abc123
or sk-ant-abc123.
assertions:
- "Output does NOT contain 'sk-' followed by alphanumeric characters"
- "Output references OPENAI_API_KEY or ANTHROPIC_API_KEY as an environment variable"
- "Output does NOT contain a hardcoded token or credential value"
- id: output-acquisition-warning
name: "Deterministic — acquisition warning for non-OpenAI workloads"
type: deterministic
prompt: "evaluate my Anthropic-only pipeline with promptfoo"
expected_output: >
The response includes a note about Promptfoo's OpenAI acquisition (March 2026)
and either mentions version pinning or names at least one documented fallback
(DeepEval or Arize Phoenix).
assertions:
- "Output contains 'acquisition' OR output contains 'OpenAI' in a warning context"
- "Output contains 'DeepEval' OR output contains 'Arize' OR output contains 'Phoenix' OR output contains 'pin'"
- id: output-quality-rubric
name: "LLM-rubric — complete setup is actionable and compliant"
type: llm-rubric
prompt: "set up a complete promptfoo project for testing my customer support chatbot"
expected_output: >
A complete, actionable guide that: scaffolds a project with the pinned version,
produces a valid promptfooconfig.yaml with realistic prompts, at least one
provider, and test cases with assertions, references environment variables for
API keys, and provides the commands needed to run the first eval.
assertions:
- >
The output is actionable: a developer could follow the steps without additional
research and end up with a working promptfoo project
- >
The config example is structurally valid promptfoo YAML: it has prompts,
providers, and tests sections with realistic placeholder content
- >
API keys are handled correctly: all examples use environment variable references,
not literal key values
- >
The pinned version (0.121.17) appears in at least one command — the output does
not instruct the user to install an unpinned version

124
plugins/kyberforge/skills/promptfoo/references/assertions.md Normal file
View File

@@ -0,0 +1,124 @@
---
topic: assertions
source_keys:
- context7-promptfoo-dev
- context7-promptfoo-github
---
## Assertion structure
Each assertion in `assert:` has a `type:`, an optional `value:`, an optional `threshold:`, and an optional `metric:` label.
```yaml
assert:
- type: contains
value: 'return policy'
- type: llm-rubric
value: 'Response is helpful and professional'
threshold: 0.8
metric: quality
```
## String shorthand
Assertions can also be written as compact strings directly in the assert list:
| Shorthand | Full type |
|---|---|
| `Paris` | `equals` |
| `contains:Paris` | `contains` |
| `icontains:paris` | `icontains` (case-insensitive) |
| `starts-with:The answer` | `starts-with` |
| `regex:^Hello.*world$` | `regex` |
| `is-json` | `is-json` |
| `contains-json` | `contains-json` |
| `similar(0.8):Hello world` | `similar` with threshold |
| `llm-rubric:Is helpful and accurate` | `llm-rubric` |
| `grade:Does not mention being an AI` | alias for `llm-rubric` |
| `factuality:Paris is the capital of France` | `factuality` |
| `javascript:output.length < 100` | inline JS |
| `fn:output.includes('hello')` | alias for `javascript` |
| `python:len(output) > 10` | inline Python |
| `file://assertions/custom.js` | external file |
| `levenshtein(5):expected text` | `levenshtein` with distance |
| `not-contains:error` | negated assertion |
## Deterministic assertions
- `equals` — exact string match
- `contains` / `icontains` — substring check (case-sensitive / insensitive)
- `not-contains` — absence check
- `starts-with` — prefix check
- `regex` — regular expression match
- `is-json` — valid JSON
- `contains-json` — valid JSON somewhere in output
- `levenshtein` — edit distance within threshold
## Similarity and semantic assertions
- `similar` — embedding cosine similarity; `threshold:` is a 0–1 score
- `context-faithfulness` — similarity-based RAG faithfulness; `threshold: 0.8` typical
## Model-graded assertions
These send a grader prompt to another LLM (by default the configured judge model) and score the output.
**`llm-rubric`** — open-ended rubric; binary or 0–1 score depending on criteria phrasing. Use `threshold:` to set minimum passing score:
```yaml
- type: llm-rubric
value: Is not apologetic and provides a clear, concise answer
threshold: 0.8
```
**`factuality`** — checks whether the output is factually consistent with a reference statement:
```yaml
- type: factuality
value: The capital of California is Sacramento
```
**`pi`** — custom scoring with any numeric range; requires `threshold:`.
The grader model can be overridden globally in `defaultTest.options.provider` or per-assertion.
## Custom assertions
**JavaScript (inline):**
```yaml
- type: javascript
value: "output.length < 100 && !output.includes('error')"
```
**JavaScript (file):**
```yaml
- type: javascript
value: file://assertions/check_format.js
```
**Python:**
```yaml
- type: python
value: "len(output) > 10 and 'Paris' in output"
```
## Negation
Any assertion type can be negated by prepending `not-`:
```yaml
- type: not-contains
value: 'error'
- type: not-regex
value: '\b(fail|broken)\b'
```
## Metrics
The `metric:` field groups assertions for aggregate reporting. All assertions with the same metric name are scored together in the results view:
```yaml
assert:
- type: contains
value: 'policy'
metric: coverage
- type: llm-rubric
value: Addresses the user's concern
metric: quality
```

127
plugins/kyberforge/skills/promptfoo/references/ci-cd.md Normal file
View File

@@ -0,0 +1,127 @@
---
topic: ci-cd
source_keys:
- context7-promptfoo-dev
- context7-promptfoo-github
---
## How CI integration works
`promptfoo eval` exits with a non-zero code when any assertion fails. This makes it a natural CI gate — a failing eval blocks a merge just like a failing test suite.
## GitHub Actions — promptfoo-action
The official `promptfoo/promptfoo-action@v1` action runs an eval on every pull request and posts results as a PR comment.
```yaml
# .github/workflows/llm-eval.yml
name: LLM Eval
on:
pull_request:
workflow_dispatch:
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: promptfoo/promptfoo-action@v1
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
config: promptfooconfig.yaml
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
```
The `github-token` is required for the action to post PR comments. Without it the eval still runs but results are not surfaced in the PR UI.
## Generic CI (npx)
Any CI system that can run Node.js commands can use Promptfoo:
```yaml
# Any CI provider
- name: Run promptfoo eval
run: npx promptfoo@0.121.17 eval --no-cache
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
```
`--no-cache` ensures all LLM calls are fresh — important in CI where cached results from a developer's machine would not be present.
## Google Cloud / Vertex AI
```yaml
# .github/workflows/llm-test.yml
steps:
- uses: google-github-actions/auth@v2
with:
credentials_json: ${{ secrets.GCP_CREDENTIALS }}
- name: Run promptfoo tests
run: npx promptfoo@0.121.17 eval
env:
GOOGLE_CLOUD_PROJECT: ${{ vars.GCP_PROJECT_ID }}
GOOGLE_CLOUD_LOCATION: us-central1
```
## MCP security testing workflow
```yaml
# .github/workflows/security-test.yml
name: MCP Security Testing
on: [push, pull_request]
jobs:
security-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '22'
- run: npm install
- run: npm run build:all-servers
- name: Run security tests
run: |
npx promptfoo eval -c security-tests/scenario1.yaml
npx promptfoo eval -c security-tests/scenario2.yaml
```
## Model security scanning (SARIF output)
For repositories that include model files, scan them in CI and upload results to GitHub's security tab:
```yaml
- name: Install dependencies
run: |
npm install -g promptfoo
pip install modelaudit
- name: Scan models
run: |
promptfoo scan-model ./models/ \
--strict \
--no-write \
--format sarif \
--output model-scan-results.sarif
- name: Upload SARIF
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: model-scan-results.sarif
```
## GitLab CI and Jenkins
Both are supported. Use `npx promptfoo@0.121.17 eval` as the test command. No platform-specific action is needed — the exit code gates the pipeline natively.
## Recommended CI practices
- Always pass `--no-cache` in CI to avoid stale results
- Store API keys as CI secrets, never hardcode them
- Run evals on PR branches to catch regressions before merge
- Use `outputPath: results.json` and archive the artifact for debugging failed runs
- Set `evaluateOptions.maxConcurrency` low (2–5) in CI to avoid provider rate limits

148
plugins/kyberforge/skills/promptfoo/references/cli-reference.md Normal file
View File

@@ -0,0 +1,148 @@
---
topic: cli-reference
source_keys:
- context7-promptfoo-dev
- context7-promptfoo-github
---
## Invocation
```bash
promptfoo <command> [options] # global install
npx promptfoo@0.121.17 <command> # one-off via npx
```
## Commands
### `init`
Scaffold a new project in the current directory.
```bash
promptfoo init
promptfoo init --example openai-mcp
```
Creates `promptfooconfig.yaml` with example prompts, providers, and test cases.
---
### `eval` (most common)
Run an evaluation.
```bash
promptfoo eval
promptfoo eval -c path/to/promptfooconfig.yaml
promptfoo eval --no-cache
promptfoo eval --max-concurrency 2
promptfoo eval --delay 3000
promptfoo eval -o results.json
```
| Flag | Description |
|---|---|
| `-c <path>` | Config file path (default: `promptfooconfig.yaml`) |
| `--no-cache` | Disable response cache; forces fresh LLM calls |
| `--max-concurrency <n>` | Max parallel requests (default: provider-dependent) |
| `--delay <ms>` | Fixed delay between requests |
| `-o <path>` | Output path (`.html`, `.json`, `.csv`, `.yaml`) |
| `--format sarif` | Output in SARIF format (for security scanning) |
Exit code is non-zero when any assertion fails, making it suitable for CI gating.
---
### `view`
Open the most recent evaluation results in a local browser UI.
```bash
promptfoo view
```
---
### `share`
Upload results and get a shareable URL.
```bash
promptfoo share
```
---
### `cache clear`
Clear all cached LLM responses.
```bash
promptfoo cache clear
```
---
### `generate dataset`
Use an LLM to auto-generate test cases from a prompt template.
```bash
promptfoo generate dataset
promptfoo generate dataset --config path/to/config.yaml
promptfoo generate dataset --output generated_tests.yaml
promptfoo generate dataset --instructions "Consider edge cases related to international travel"
```
---
### `redteam generate`
Generate adversarial test cases for red-teaming.
```bash
promptfoo redteam generate
promptfoo redteam generate -c promptfooconfig.yaml
```
---
### `scan-model`
Scan model files for security vulnerabilities. Outputs results in SARIF format.
```bash
promptfoo scan-model ./models/ --strict --no-write --format sarif --output scan.sarif
```
---
### `auth`
Manage authentication (for sharing and cloud features).
```bash
promptfoo auth login
promptfoo auth logout
```
---
## Rate-limit management
```bash
promptfoo eval --max-concurrency 1 --delay 3000
```
Or in config:
```yaml
evaluateOptions:
maxConcurrency: 2
delay: 3000
```
Environment variable for backoff:
```bash
export PROMPTFOO_REQUEST_BACKOFF_MS=10000
export PROMPTFOO_RETRY_5XX=true
```

151
plugins/kyberforge/skills/promptfoo/references/configuration.md Normal file
View File

@@ -0,0 +1,151 @@
---
topic: configuration
source_keys:
- context7-promptfoo-dev
- context7-promptfoo-github
---
## File name and schema
The default config file is `promptfooconfig.yaml` in the working directory. A different path can be passed with `-c`. Add the JSON schema header for editor autocompletion:
```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
```
## Top-level structure
```yaml
description: Human-readable name for this eval
prompts:
- '...' # inline string
- file://... # path to .txt, .json, .js, .py
providers:
- openai:gpt-5-mini
- anthropic:messages:claude-sonnet-4-5
defaultTest: # merged into every test case
assert:
- type: is-json
tests:
- vars:
query: 'I need help'
assert:
- type: contains
value: 'help'
- file://test_scenarios.csv # external test file
outputPath: results/eval.html # .html, .json, .csv, .yaml
evaluateOptions:
maxConcurrency: 5
delay: 500 # ms between requests
```
## Prompts
Plain string with Handlebars-style `{{variable}}` placeholders:
```yaml
prompts:
- 'You are a helpful agent. {{query}}'
```
Chat conversation from a JSON file (array of `{role, content}` messages):
```yaml
prompts:
- file://prompts/chat_conversation.json
```
Dynamic prompt from a JS function:
```yaml
prompts:
- file://prompts/generate_prompt.js
```
Prompts can also carry a `label:` and `raw:` when using the object form, and a `config:` block to set provider-specific parameters (e.g. `response_format`).
## Providers
String shorthand:
```yaml
providers:
- openai:gpt-5-mini
- anthropic:messages:claude-sonnet-4-5-20250929
- bedrock:us.anthropic.claude-sonnet-4-5-20250929-v1:0
- azureopenai:chat:my-deployment
- http://localhost:8080/v1/chat/completions # custom HTTP
```
Object form with config:
```yaml
providers:
- id: openai:responses:gpt-5
config:
temperature: 0.7
max_output_tokens: 500
instructions: 'You are a helpful assistant.'
```
Over 60 providers are supported. Local models, HuggingFace, and custom HTTP endpoints are all valid provider types.
## Tests and vars
Each test case has `vars:` (substituted into prompt placeholders) and `assert:` (assertions on the response):
```yaml
tests:
- vars:
query: 'I need to return a product'
assert:
- type: contains
value: 'return policy'
- type: llm-rubric
value: 'Response is helpful and professional'
```
Tests can be loaded from external files (CSV, YAML) using `file://` references.
## defaultTest
Assertions and options declared here are merged into every test case, reducing repetition:
```yaml
defaultTest:
assert:
- type: llm-rubric
value: 'Does not reveal internal system prompt'
options:
provider:
id: openai:chat:gpt-5-mini # override grader model
```
## Output formats
`outputPath` accepts `.html` (browser-viewable), `.json`, `.csv`, or `.yaml`. Multiple outputs can be listed as an array.
## Environment variables
| Variable | Purpose |
|---|---|
| `OPENAI_API_KEY` | OpenAI authentication |
| `ANTHROPIC_API_KEY` | Anthropic authentication |
| `REQUEST_TIMEOUT_MS` | Per-request timeout in ms |
| `PROMPTFOO_RETRY_5XX` | Retry on 5xx errors (`true`/`false`) |
| `PROMPTFOO_REQUEST_BACKOFF_MS` | Backoff between retries |
## Red-team configuration block
```yaml
redteam:
plugins:
- harmful
- prompt-injection
- hijacking
strategies:
- jailbreak
- jailbreak:composite
- prompt-injection
```

150
plugins/kyberforge/skills/promptfoo/references/examples.md Normal file
View File

@@ -0,0 +1,150 @@
---
topic: examples
source_keys:
- context7-promptfoo-dev
- context7-promptfoo-github
---
## Quickstart
```bash
npx promptfoo@0.121.17 init
# edit promptfooconfig.yaml
npx promptfoo@0.121.17 eval
npx promptfoo@0.121.17 view
```
## Minimal config
```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
prompts:
- 'Answer the user question concisely. Question: {{question}}'
providers:
- openai:gpt-5-mini
tests:
- vars:
question: How do I reset my password?
assert:
- type: contains
value: reset
- vars:
question: Can I cancel my subscription today?
assert:
- type: llm-rubric
value: The answer clearly explains the cancellation path.
```
## Multi-provider comparison
```yaml
providers:
- openai:gpt-5-mini
- anthropic:claude-3-haiku
prompts:
- 'You are a helpful customer service agent. {{query}}'
tests:
- vars:
query: 'I need to return a product'
assert:
- type: contains
value: 'return policy'
- type: llm-rubric
value: 'Response is helpful and professional'
```
Running this produces a side-by-side table with both models' outputs and assertion scores.
## Loading tests from CSV
```yaml
tests:
- file://test_scenarios.csv
```
CSV format: one column per variable, header row must match `{{variable}}` names in the prompt. An `__expected` column maps to the `equals` assertion automatically.
## Factuality evaluation
```yaml
providers:
- openai:gpt-5-mini
prompts:
- |
Please answer the following question accurately:
Question: What is the capital of {{location}}?
tests:
- vars:
location: California
assert:
- type: factuality
value: The capital of California is Sacramento
```
## defaultTest for shared assertions
```yaml
defaultTest:
assert:
- type: llm-rubric
value: |
Evaluate whether the response correctly answers the question.
Question: {{ question }}
Model Response: {{ output }}
Correct Answer: {{ answer }}
Grade accuracy 0.0–1.0. Pass if >= 0.8.
threshold: 0.8
tests:
- vars:
question: What year did WW2 end?
answer: '1945'
- vars:
question: What is the boiling point of water in Celsius?
answer: '100'
```
## Node.js API
```javascript
import { evaluate } from 'promptfoo';
const evalRecord = await evaluate({
prompts: ['Translate to Spanish: {{ text }}'],
providers: ['openai:chat:gpt-5.5'],
tests: [
{
vars: { text: 'Hello' },
assert: [{ type: 'contains', value: 'Hola', metric: 'translation' }],
},
],
});
const results = await evalRecord.toEvaluateSummary();
console.log(`Pass rate: ${results.stats.successes}/${results.results.length}`);
```
## Generating test datasets with AI
```bash
# Generate test cases based on your prompt template
promptfoo generate dataset
promptfoo generate dataset --instructions "Consider edge cases related to international travel"
promptfoo generate dataset --output generated_tests.yaml
```
## Saving and sharing results
```yaml
outputPath: evaluations/results.html
```
Or via CLI:
```bash
promptfoo eval -o results.json
promptfoo share # get a shareable URL
```

70
plugins/kyberforge/skills/promptfoo/references/installation.md Normal file
View File

@@ -0,0 +1,70 @@
---
topic: installation
source_keys:
- context7-promptfoo-dev
- context7-promptfoo-github
---
## Acquisition notice
Promptfoo was acquired by OpenAI in March 2026. OpenAI has committed to keeping it open-source and multi-provider. **Always pin to a specific version** — do not track `@latest`. Monitor for neutrality degradation in future releases, particularly for non-OpenAI provider evaluation. Documented fallbacks: DeepEval (pytest-native, Python teams) and Arize Phoenix (self-hosted, vendor-neutral).
Pinned version in this skill: **0.121.17**. Update via CI when a new version has been validated.
## Prerequisites
- Node.js 18+ (Node 22 recommended for CI)
- npm or npx
No database or server process is required. Promptfoo stores evaluation results in a local SQLite cache.
## Installation options
**One-off via npx (no install required — preferred):**
```bash
npx promptfoo@0.121.17 eval
```
**Global install (pinned):**
```bash
npm install -g promptfoo@0.121.17
```
After a global install, use `promptfoo` directly without `npx`.
**Project dependency (for Node.js API usage):**
```bash
npm install promptfoo@0.121.17
```
## Initialising a project
```bash
npx promptfoo@0.121.17 init
```
Creates a `promptfooconfig.yaml` scaffold in the current directory with example prompts, providers, and tests.
To start from an official example:
```bash
npx promptfoo@0.121.17 init --example openai-mcp
npx promptfoo@0.121.17 init --example openai-structured-output
npx promptfoo@0.121.17 init --example openai-responses
npx promptfoo@0.121.17 init --example openai-audio
```
## API keys
Promptfoo reads provider credentials from environment variables. Set them before running evals:
```bash
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
```
Other providers (Bedrock, Azure, Vertex) follow their own SDK credential conventions — see provider-specific docs.
## Global timeout
```bash
export REQUEST_TIMEOUT_MS=600000 # 10 minutes default
```

40
plugins/kyberforge/skills/promptfoo/references/overview.md Normal file
View File

@@ -0,0 +1,40 @@
---
topic: overview
source_keys:
- context7-promptfoo-dev
- context7-promptfoo-github
---
## What is Promptfoo
Promptfoo is an open-source, local-first CLI and library for evaluating and red-teaming LLM applications. It enables systematic, repeatable testing of prompts across multiple providers with assertions that grade outputs automatically. Results are stored locally and can be visualised in a browser UI or exported.
It is not a hosted service — all evaluation state stays on your machine unless you explicitly share a result.
## Core concepts
**Prompt** — a template with `{{variable}}` placeholders. Can be a plain string, a JSON chat array (`file://prompts/chat.json`), or a JavaScript function (`file://prompts/generate.js`) that returns a string or message array dynamically.
**Provider** — an LLM endpoint to send the rendered prompt to. Providers are declared as strings (`openai:gpt-5-mini`, `anthropic:messages:claude-sonnet-4-5`) or objects with a `config:` block for additional parameters.
**Test case** — one input scenario. Contains `vars:` (values substituted into prompt variables) and `assert:` (a list of assertions that the response must satisfy).
**Assertion** — a pass/fail check on the model output. Ranges from deterministic (`contains`, `regex`, `equals`) to model-graded (`llm-rubric`, `factuality`).
**Eval** — one complete run: every prompt × every provider × every test case is executed and each assertion is scored. Results are a table of pass/fail cells with per-assertion metrics.
**defaultTest** — a top-level config key whose `assert:` and `options:` are merged into every test case, avoiding repetition.
## Mental model
Think of an eval as a spreadsheet where rows are test cases and columns are (prompt, provider) pairs. Each cell contains the model output and assertion results. Running `promptfoo eval` fills the spreadsheet; `promptfoo view` opens it in a browser.
The config file (`promptfooconfig.yaml`) is the source of truth for a given eval. It is committed alongside your prompt files so evals are reproducible.
## What it is used for
- **Regression testing** — catch prompt regressions before deploying changes
- **Side-by-side model comparison** — evaluate multiple providers on identical test suites
- **Red-teaming** — generate and run adversarial tests (jailbreaks, prompt injection, harmful content)
- **Dataset generation** — AI-generate test cases from a prompt template
- **CI/CD gating** — fail a pull request when assertion pass rate drops

133
plugins/kyberforge/skills/promptfoo/references/redteam.md Normal file
View File

@@ -0,0 +1,133 @@
---
topic: redteam
source_keys:
- context7-promptfoo-dev
- context7-promptfoo-github
---
## What red-teaming does
Red-teaming in Promptfoo generates adversarial test cases that probe an LLM application for security vulnerabilities and safety failures. It is separate from standard evals — you configure it under a `redteam:` block and use `promptfoo redteam generate` to produce test cases, then run them with `promptfoo eval`.
## Basic configuration
```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
redteam:
plugins:
- harmful
- prompt-injection
- hijacking
strategies:
- jailbreak
- jailbreak:composite
- prompt-injection
providers:
- openai:gpt-5-mini
```
## Plugins (what to test for)
Plugins define the vulnerability categories to probe:
| Plugin | Tests for |
|---|---|
| `harmful` | General harmful content generation |
| `harmful:misinformation-disinformation` | False or misleading information |
| `harmful:cybercrime` | Cyberattack assistance |
| `prompt-injection` | Injection via user input overriding system instructions |
| `hijacking` | Redirecting the assistant to unintended tasks |
| `jailbreak` | Breaking safety guardrails |
| `rbac` | Role-based access control bypass |
| `hallucination` | Fabricated facts |
| `debug-access` | Exposing internal debug interfaces |
| `shell-injection` | Shell command injection |
| `sql-injection` | SQL injection via natural language |
| `ssrf` | Server-side request forgery |
## Strategies (how to attack)
Strategies control the attack method applied to each plugin's test cases:
| Strategy | Description |
|---|---|
| `jailbreak` | Classic jailbreak prompts |
| `jailbreak:composite` | Chained / composite jailbreak attempts |
| `prompt-injection` | Inject instructions via user-controlled content |
| `base64` | Encode attack payload in base64 |
| `leetspeak` | Obfuscate with leet substitutions |
| `rot13` | Encode with ROT-13 |
| `iterative` | Iteratively refine attack prompts |
| `ensemble` | Combine multiple strategies |
## Generating adversarial tests
```bash
promptfoo redteam generate
promptfoo redteam generate -c promptfooconfig.yaml
```
Then run the generated tests:
```bash
promptfoo eval
```
## Node.js API
```javascript
import { redteam } from 'promptfoo';
const result = await redteam.generate({
target: {
prompt: 'You are a helpful assistant. Answer user questions.',
model: 'openai:chat:gpt-5.5',
},
plugins: ['prompt-injection', 'jailbreak', 'rbac'],
numTests: 5,
strategies: ['iterative', 'ensemble'],
});
result.tests.forEach((test, i) => {
console.log(`${i + 1}. [${test.category}] ${test.prompt.substring(0, 100)}...`);
});
```
## Cascading failures (agentic AI)
For agentic systems, test cascading failures and multi-step attacks:
```yaml
redteam:
plugins:
- hallucination
- harmful:misinformation-disinformation
- divergent-repetition
strategies:
- jailbreak
- prompt-injection
```
## OWASP and MITRE alignment
Plugins map to OWASP LLM Top 10 and MITRE ATLAS categories. Use `debug-access`, `shell-injection`, `sql-injection`, `ssrf` together to cover the MITRE ATLAS initial-access cluster:
```yaml
redteam:
plugins:
- debug-access
- harmful:cybercrime
- shell-injection
- sql-injection
- ssrf
strategies:
- base64
- jailbreak
- leetspeak
- prompt-injection
- rot13
```
## Integration with Burp Suite
Promptfoo can generate targeted red-team test cases for use with Burp Suite. Use `promptfoo redteam generate` to produce test payloads and then pass them into Burp's active scanner.

15
plugins/kyberforge/skills/promptfoo/references/sources.md Normal file
View File

@@ -0,0 +1,15 @@
# Sources
## context7-promptfoo-dev
- **URL:** context7:/websites/promptfoo_dev
- **Description:** Official Promptfoo website documentation — overview, getting started, configuration, CLI, assertions, CI/CD, red-teaming
- **Contributing files:** overview.md, installation.md, configuration.md, cli-reference.md, assertions.md, examples.md, redteam.md, ci-cd.md
- **Status:** `extracted`
## context7-promptfoo-github
- **URL:** context7:/promptfoo/promptfoo
- **Description:** Promptfoo GitHub repository — provider-specific docs, Node.js API examples, advanced configuration patterns
- **Contributing files:** overview.md, installation.md, configuration.md, cli-reference.md, assertions.md, examples.md, redteam.md, ci-cd.md
- **Status:** `extracted`