holocron/.agents/skills/deepeval/references/metrics.md

# Metrics

Use 3-5 metrics for the first eval suite when the user is unsure. More metrics
make iteration slower and harder to interpret. Reuse existing project metrics
and thresholds before adding new ones.

Keep metric instances in a separate `metrics.py` module (or the project's
existing metrics module). Eval test files should import metric lists rather than
constructing several ad hoc metrics inline.

Name component/span metric lists after the exact component they evaluate. Avoid
generic names like `COMPONENT_METRICS` because one suite can evaluate several
components with different metric requirements.

## Required Rule

Single-turn `LLMTestCase` evals must use single-turn metrics.

Multi-turn `ConversationalTestCase` evals must use multi-turn conversational
metrics. Do not use `AnswerRelevancyMetric`, `FaithfulnessMetric`, or other
single-turn `LLMTestCase` metrics on multi-turn end-to-end evals.

## Metric Types

Choose metrics by what the user wants to measure, not only by app type.

| Type | Use when | Examples |
| --- | --- | --- |
| Custom criteria | The success criteria is product- or domain-specific | `GEval`, `DAGMetric`, `ConversationalGEval`, `ConversationalDAGMetric` |
| RAG retriever | You need to evaluate retrieved context quality | `ContextualRelevancyMetric`, `ContextualPrecisionMetric`, `ContextualRecallMetric` |
| RAG generator | You need to evaluate the final answer against context | `AnswerRelevancyMetric`, `FaithfulnessMetric` |
| Agentic flow | You need to evaluate task completion, plans, steps, tools, or arguments | `TaskCompletionMetric`, `ToolCorrectnessMetric`, `ArgumentCorrectnessMetric`, `PlanAdherenceMetric`, `PlanQualityMetric`, `StepEfficiencyMetric` |
| Multi-turn chatbot | You need to evaluate an entire conversation | `ConversationCompletenessMetric`, `RoleAdherenceMetric`, `TurnRelevancyMetric`, `ConversationalGEval` |
| Safety and compliance | You need to detect risky or policy-violating outputs | `BiasMetric`, `ToxicityMetric`, `PIILeakageMetric`, `MisuseMetric`, `RoleViolationMetric`, `NonAdviceMetric` |
| Format / structure | You need output to match a schema or instruction set | `JsonCorrectnessMetric`, `PromptAlignmentMetric` |
| Other task-specific quality | The app is summarization, hallucination-sensitive, image-based, or otherwise specialized | `SummarizationMetric`, `HallucinationMetric`, multimodal metrics |

Aim to include at least one custom metric when the user's definition of success
is not fully captured by a predefined metric. In practice, custom metrics should
usually be `GEval` for single-turn evals or `ConversationalGEval` for multi-turn
evals.

## Default If User Is Unsure

If the user says "I don't know" or gives no metric preference:

- Use 3-5 metrics.
- Put metrics on the end-to-end eval first.
- Do not add safety metrics by default unless the app is safety/compliance
  sensitive or the user asks for them.
- Use about half custom metrics and half system-specific metrics.
- Add component/span metrics only after E2E/traces show component failures, or
  if the user explicitly wants component-level scoring.

Good system-specific defaults:

- Single-turn tracing E2E: strongly prefer `TaskCompletionMetric` and
  `StepEfficiencyMetric` as the baseline pair, especially for agents and
  multi-step AI apps.
- Agent: `TaskCompletionMetric` plus tool/argument correctness only when
  `tools_called` data exists.
- RAG: `FaithfulnessMetric`, `AnswerRelevancyMetric`, and
  `ContextualRelevancyMetric` are strong candidates.
- Multi-turn chatbot: use conversational metrics only, plus a
  `ConversationalGEval` custom criterion when product-specific behavior matters.

For custom metrics, assume `GEval` for single-turn or `ConversationalGEval` for
multi-turn. There is a very high chance this is the right custom metric type.
Do not start with DAG unless the user already has a DAG metric or specifically
needs decision-tree scoring.

Use `GEval` when scoring is subjective or there is no predefined metric for the
thing the user cares about. Correctness is a common example: there is no generic
"correctness metric" because correctness depends on the task. Define a `GEval`
named `Correctness` and write criteria that explain what correct means for this
app.

Use `DAGMetric` only when the metric is decision-based: the score should follow
explicit branches, checks, or deterministic rubric paths. DAG is useful when the
metric is more like a decision tree than a subjective judge. Do not start with
DAG for ordinary subjective scoring.

When choosing `GEval.evaluation_params`, include only fields the test case will
actually have. Be especially careful with reference-space params like
`expected_output`, `context`, `retrieval_context`, or `expected_tools`; if the
dataset or app does not provide them, the metric will fail at runtime. Prefer
`input` and `actual_output` unless the eval plan explicitly creates the
reference fields.

If existing project metrics are present, use them first. If there are too many,
tell the user: "You already have a lot of metrics here, which may make evals
slow or hard to interpret. I recommend narrowing the first run to the highest
signal metrics."

## Reference-Based Metrics

Some metrics require reference fields. Use them sparingly unless the plan
includes those expected values, because missing fields will cause metric errors.

Reference-based fields include:

- `expected_output`
- `expected_outcome`
- `expected_tools`
- `context`
- `retrieval_context`

Examples:

- `ContextualPrecisionMetric` and `ContextualRecallMetric` need
  `expected_output`.
- `ToolCorrectnessMetric` needs `expected_tools`.
- Multi-turn outcome metrics may depend on `expected_outcome`.
- RAG grounding metrics need `retrieval_context`.

If the dataset does not include the required fields, choose metrics that match
available fields or update the dataset generation/loading plan first.

## Common Single-Turn Metrics

| Metric | What it checks | Required test case fields |
| --- | --- | --- |
| `AnswerRelevancyMetric` | Output answers the input | `input`, `actual_output` |
| `FaithfulnessMetric` | Output is grounded in retrieved context | `input`, `actual_output`, `retrieval_context` |
| `ContextualRelevancyMetric` | Retrieved context is relevant to input | `input`, `retrieval_context` |
| `ContextualPrecisionMetric` | Relevant context is ranked highly | `input`, `retrieval_context`, `expected_output` |
| `ContextualRecallMetric` | Retrieved context covers expected answer | `input`, `retrieval_context`, `expected_output` |
| `TaskCompletionMetric` | Agent/app completed the task | `input`, `actual_output` |
| `StepEfficiencyMetric` | Agent/app completed the task efficiently without unnecessary steps | trace steps/tool activity |
| `ToolCorrectnessMetric` | Called tools match expected tools | `input`, `tools_called`, `expected_tools` |
| `ArgumentCorrectnessMetric` | Tool arguments are correct | `input`, `tools_called` |
| `JsonCorrectnessMetric` | Output matches expected schema | `input`, `actual_output`; constructor needs `expected_schema` |
| `PromptAlignmentMetric` | Output follows prompt instructions | `input`, `actual_output`; constructor needs `prompt_instructions` |
| `GEval` | Custom single-turn criteria | constructor needs `name`, `criteria` or `evaluation_steps`, and `evaluation_params` |

## Common Multi-Turn Metrics

| Metric | What it checks | Required test case fields |
| --- | --- | --- |
| `ConversationCompletenessMetric` | Conversation achieved the expected outcome | `turns` with `role`, `content` |
| `RoleAdherenceMetric` | Assistant stayed in role across turns | `turns` with `role`, `content` |
| `TurnRelevancyMetric` | Assistant turns are relevant | `turns` with `role`, `content` |
| `TurnFaithfulnessMetric` | Turns are faithful to retrieval context | `turns` with `role`, `content`, `retrieval_context` |
| `TurnContextualRelevancyMetric` | Turn retrieval context is relevant | `turns` with `role`, `content`, retrieval context |
| `GoalAccuracyMetric` | Conversation achieved the user's goal | `turns` with `role`, `content` |
| `TopicAdherenceMetric` | Conversation stayed on allowed topics | `turns` with `role`, `content`; constructor needs `relevant_topics` |
| `ConversationalGEval` | Custom multi-turn criteria | constructor needs `name` and `criteria` or `evaluation_steps` |

## Choosing Metrics

Ask what the user cares about in product terms first. Then map that to metrics.

Ask:

- What failure would be unacceptable in production?
- Is success about final answer quality, retrieved context, tool use, safety,
  conversation completion, or output format?
- Do we need a custom criterion because the product definition of "good" is
  domain-specific?
- Which fields does the dataset/test case actually contain?

Mappings:

- "Does it answer correctly?" -> `AnswerRelevancyMetric` or task-specific `GEval`
- "Is it grounded in docs?" -> `FaithfulnessMetric` plus contextual metrics
- "Did the agent finish the task?" -> `TaskCompletionMetric`
- "Did the agent take efficient steps?" -> `StepEfficiencyMetric`
- "Did it use the right tool?" -> `ToolCorrectnessMetric`
- "Did the chatbot complete the conversation?" -> `ConversationCompletenessMetric`
- "Did it stay in character?" -> `RoleAdherenceMetric`

If unsure for single-turn tracing, start with `TaskCompletionMetric` and
`StepEfficiencyMetric`, then add 1-3 more E2E metrics only when the app's
success criteria need them. Add component/span metrics only after the first run
reveals where the app is failing.