holocron/plugins/kyberforge/skills/promptfoo/SKILL.md

name, description, metadata

name	description	metadata
promptfoo	Use when the user wants to install, set up, configure, or work with Promptfoo — the open-source LLM evaluation CLI. Covers: project initialization (promptfoo init), writing or editing promptfooconfig.yaml, running evaluations (promptfoo eval), comparing model outputs side-by-side, viewing and sharing results, generating test datasets, red-teaming LLM applications with adversarial plugins and strategies, and integrating Promptfoo into CI/CD pipelines. Do NOT use when the user wants to evaluate LLM output quality in general without Promptfoo, write tests for non-LLM code (use tdd), or set up a different evaluation framework (RAGAS, OpenAI Evals, etc.).	category test

Required inputs

• Task scope — what the user wants to do (install, configure, run eval, red-team, CI integration, or debug); inferred from request, ask only if genuinely ambiguous
• Config file path — defaults to promptfooconfig.yaml in the working directory; user provides if non-standard
• Provider API keys — must be set as environment variables before running evals; remind the user if not already set

Constraints

• Always use the pinned version 0.121.17 — never suggest @latest or an unpinned install; see references/installation.md for rationale (OpenAI acquisition, March 2026)
• API keys and credentials must always reference environment variables in examples and config — never hardcoded values, even as placeholders like sk-abc123
• Warn when the user's eval workload is non-OpenAI-centric: note the acquisition and mention DeepEval / Arize Phoenix as documented fallbacks
• Offload detail to references/ files — do not reproduce CLI flags, assertion types, or redteam plugin tables inline; wire each reference file in the step that needs it

Process

1. Clarify scope. Identify what the user wants: install/setup, configure promptfooconfig.yaml, run an eval, generate a dataset, red-team, integrate CI/CD, or debug a failing run. Infer from context — ask only if genuinely ambiguous.

2. Install / initialise (if needed). Check whether Promptfoo is already installed. If not, guide through installation using the pinned version. To scaffold a new project: npx promptfoo@0.121.17 init. See references/installation.md for prerequisites, global/local install options, and the acquisition notice.

3. Configure. Help the user write or edit promptfooconfig.yaml. Cover: prompts (inline, file://, JS/Python), providers (string shorthand and object form), test cases with vars and assert, defaultTest, outputPath, and evaluateOptions. See references/configuration.md for the full structure and references/assertions.md for assertion types and shorthand syntax.

4. Run evaluation. Execute npx promptfoo@0.121.17 eval (with --no-cache in CI). After the run, open results with promptfoo view or share with promptfoo share. For rate-limit issues, suggest --max-concurrency and --delay flags. See references/cli-reference.md for all flags and subcommands.

5. Red-team (if requested). Add a redteam: block to the config with the appropriate plugins and strategies. Run promptfoo redteam generate to produce adversarial test cases, then promptfoo eval to execute them. See references/redteam.md for plugin/strategy tables and OWASP/MITRE mappings.

6. CI integration (if requested). Write a GitHub Actions workflow using promptfoo/promptfoo-action@v1 or a generic npx promptfoo@0.121.17 eval --no-cache step. Always use CI secrets for API keys. See references/ci-cd.md for full workflow templates.

7. Generate test dataset (if requested). Run promptfoo generate dataset with optional --instructions to AI-generate test cases from the prompt template. See references/cli-reference.md for flags.

Output format

• Modified or newly created promptfooconfig.yaml
• Shell commands to run, with the pinned version and relevant flags
• GitHub Actions workflow file (.github/workflows/llm-eval.yml) when CI is in scope

Failure handling

• No promptfooconfig.yaml found and user did not ask to create one — ask before scaffolding; do not overwrite existing config without confirmation
• Eval fails with rate-limit errors — suggest --max-concurrency 2 --delay 3000 and PROMPTFOO_RETRY_5XX=true; see references/cli-reference.md
• User requests @latest or an unpinned install — correct to 0.121.17 and explain the version-pinning rationale (acquisition notice)
• User requests hardcoded API key in config or example — refuse and redirect to environment variable pattern

Self-check

• Pinned version 0.121.17 used in all commands — no @latest
• No credentials appear in any output, config example, or shell command
• Non-OpenAI eval workloads include the acquisition warning
• Each references/ file wired in the step that uses it
• Config written or modified only after confirming path with user