Defame1297
1ceacf17bc
Covers install, configuration, running evals, red-teaming, CI/CD integration, and dataset generation. Pins to v0.121.17 with acquisition notice (OpenAI, March 2026) and documented fallbacks (DeepEval, Arize Phoenix). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
134 lines
3.6 KiB
Markdown
134 lines
3.6 KiB
Markdown
---
|
|
topic: redteam
|
|
source_keys:
|
|
- context7-promptfoo-dev
|
|
- context7-promptfoo-github
|
|
---
|
|
|
|
## What red-teaming does
|
|
|
|
Red-teaming in Promptfoo generates adversarial test cases that probe an LLM application for security vulnerabilities and safety failures. It is separate from standard evals — you configure it under a `redteam:` block and use `promptfoo redteam generate` to produce test cases, then run them with `promptfoo eval`.
|
|
|
|
## Basic configuration
|
|
|
|
```yaml
|
|
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
|
redteam:
|
|
plugins:
|
|
- harmful
|
|
- prompt-injection
|
|
- hijacking
|
|
strategies:
|
|
- jailbreak
|
|
- jailbreak:composite
|
|
- prompt-injection
|
|
|
|
providers:
|
|
- openai:gpt-5-mini
|
|
```
|
|
|
|
## Plugins (what to test for)
|
|
|
|
Plugins define the vulnerability categories to probe:
|
|
|
|
| Plugin | Tests for |
|
|
|---|---|
|
|
| `harmful` | General harmful content generation |
|
|
| `harmful:misinformation-disinformation` | False or misleading information |
|
|
| `harmful:cybercrime` | Cyberattack assistance |
|
|
| `prompt-injection` | Injection via user input overriding system instructions |
|
|
| `hijacking` | Redirecting the assistant to unintended tasks |
|
|
| `jailbreak` | Breaking safety guardrails |
|
|
| `rbac` | Role-based access control bypass |
|
|
| `hallucination` | Fabricated facts |
|
|
| `debug-access` | Exposing internal debug interfaces |
|
|
| `shell-injection` | Shell command injection |
|
|
| `sql-injection` | SQL injection via natural language |
|
|
| `ssrf` | Server-side request forgery |
|
|
|
|
## Strategies (how to attack)
|
|
|
|
Strategies control the attack method applied to each plugin's test cases:
|
|
|
|
| Strategy | Description |
|
|
|---|---|
|
|
| `jailbreak` | Classic jailbreak prompts |
|
|
| `jailbreak:composite` | Chained / composite jailbreak attempts |
|
|
| `prompt-injection` | Inject instructions via user-controlled content |
|
|
| `base64` | Encode attack payload in base64 |
|
|
| `leetspeak` | Obfuscate with leet substitutions |
|
|
| `rot13` | Encode with ROT-13 |
|
|
| `iterative` | Iteratively refine attack prompts |
|
|
| `ensemble` | Combine multiple strategies |
|
|
|
|
## Generating adversarial tests
|
|
|
|
```bash
|
|
promptfoo redteam generate
|
|
promptfoo redteam generate -c promptfooconfig.yaml
|
|
```
|
|
|
|
Then run the generated tests:
|
|
```bash
|
|
promptfoo eval
|
|
```
|
|
|
|
## Node.js API
|
|
|
|
```javascript
|
|
import { redteam } from 'promptfoo';
|
|
|
|
const result = await redteam.generate({
|
|
target: {
|
|
prompt: 'You are a helpful assistant. Answer user questions.',
|
|
model: 'openai:chat:gpt-5.5',
|
|
},
|
|
plugins: ['prompt-injection', 'jailbreak', 'rbac'],
|
|
numTests: 5,
|
|
strategies: ['iterative', 'ensemble'],
|
|
});
|
|
|
|
result.tests.forEach((test, i) => {
|
|
console.log(`${i + 1}. [${test.category}] ${test.prompt.substring(0, 100)}...`);
|
|
});
|
|
```
|
|
|
|
## Cascading failures (agentic AI)
|
|
|
|
For agentic systems, test cascading failures and multi-step attacks:
|
|
|
|
```yaml
|
|
redteam:
|
|
plugins:
|
|
- hallucination
|
|
- harmful:misinformation-disinformation
|
|
- divergent-repetition
|
|
strategies:
|
|
- jailbreak
|
|
- prompt-injection
|
|
```
|
|
|
|
## OWASP and MITRE alignment
|
|
|
|
Plugins map to OWASP LLM Top 10 and MITRE ATLAS categories. Use `debug-access`, `shell-injection`, `sql-injection`, `ssrf` together to cover the MITRE ATLAS initial-access cluster:
|
|
|
|
```yaml
|
|
redteam:
|
|
plugins:
|
|
- debug-access
|
|
- harmful:cybercrime
|
|
- shell-injection
|
|
- sql-injection
|
|
- ssrf
|
|
strategies:
|
|
- base64
|
|
- jailbreak
|
|
- leetspeak
|
|
- prompt-injection
|
|
- rot13
|
|
```
|
|
|
|
## Integration with Burp Suite
|
|
|
|
Promptfoo can generate targeted red-team test cases for use with Burp Suite. Use `promptfoo redteam generate` to produce test payloads and then pass them into Burp's active scanner.
|