holocron/.agents/skills/deepeval/references/metrics.md

Metrics

Use 3-5 metrics for the first eval suite when the user is unsure. More metrics make iteration slower and harder to interpret. Reuse existing project metrics and thresholds before adding new ones.

Keep metric instances in a separate metrics.py module (or the project's existing metrics module). Eval test files should import metric lists rather than constructing several ad hoc metrics inline.

Name component/span metric lists after the exact component they evaluate. Avoid generic names like COMPONENT_METRICS because one suite can evaluate several components with different metric requirements.

Required Rule

Single-turn LLMTestCase evals must use single-turn metrics.

Multi-turn ConversationalTestCase evals must use multi-turn conversational metrics. Do not use AnswerRelevancyMetric, FaithfulnessMetric, or other single-turn LLMTestCase metrics on multi-turn end-to-end evals.

Metric Types

Choose metrics by what the user wants to measure, not only by app type.

Type	Use when	Examples
Custom criteria	The success criteria is product- or domain-specific	GEval, DAGMetric, ConversationalGEval, ConversationalDAGMetric
RAG retriever	You need to evaluate retrieved context quality	ContextualRelevancyMetric, ContextualPrecisionMetric, ContextualRecallMetric
RAG generator	You need to evaluate the final answer against context	AnswerRelevancyMetric, FaithfulnessMetric
Agentic flow	You need to evaluate task completion, plans, steps, tools, or arguments	TaskCompletionMetric, ToolCorrectnessMetric, ArgumentCorrectnessMetric, PlanAdherenceMetric, PlanQualityMetric, StepEfficiencyMetric
Multi-turn chatbot	You need to evaluate an entire conversation	ConversationCompletenessMetric, RoleAdherenceMetric, TurnRelevancyMetric, ConversationalGEval
Safety and compliance	You need to detect risky or policy-violating outputs	BiasMetric, ToxicityMetric, PIILeakageMetric, MisuseMetric, RoleViolationMetric, NonAdviceMetric
Format / structure	You need output to match a schema or instruction set	JsonCorrectnessMetric, PromptAlignmentMetric
Other task-specific quality	The app is summarization, hallucination-sensitive, image-based, or otherwise specialized	SummarizationMetric, HallucinationMetric, multimodal metrics

Aim to include at least one custom metric when the user's definition of success is not fully captured by a predefined metric. In practice, custom metrics should usually be GEval for single-turn evals or ConversationalGEval for multi-turn evals.

Default If User Is Unsure

If the user says "I don't know" or gives no metric preference:

• Use 3-5 metrics.
• Put metrics on the end-to-end eval first.
• Do not add safety metrics by default unless the app is safety/compliance sensitive or the user asks for them.
• Use about half custom metrics and half system-specific metrics.
• Add component/span metrics only after E2E/traces show component failures, or if the user explicitly wants component-level scoring.

Good system-specific defaults:

• Single-turn tracing E2E: strongly prefer TaskCompletionMetric and StepEfficiencyMetric as the baseline pair, especially for agents and multi-step AI apps.
• Agent: TaskCompletionMetric plus tool/argument correctness only when tools_called data exists.
• RAG: FaithfulnessMetric, AnswerRelevancyMetric, and ContextualRelevancyMetric are strong candidates.
• Multi-turn chatbot: use conversational metrics only, plus a ConversationalGEval custom criterion when product-specific behavior matters.

For custom metrics, assume GEval for single-turn or ConversationalGEval for multi-turn. There is a very high chance this is the right custom metric type. Do not start with DAG unless the user already has a DAG metric or specifically needs decision-tree scoring.

Use GEval when scoring is subjective or there is no predefined metric for the thing the user cares about. Correctness is a common example: there is no generic "correctness metric" because correctness depends on the task. Define a GEval named Correctness and write criteria that explain what correct means for this app.

Use DAGMetric only when the metric is decision-based: the score should follow explicit branches, checks, or deterministic rubric paths. DAG is useful when the metric is more like a decision tree than a subjective judge. Do not start with DAG for ordinary subjective scoring.

When choosing GEval.evaluation_params, include only fields the test case will actually have. Be especially careful with reference-space params like expected_output, context, retrieval_context, or expected_tools; if the dataset or app does not provide them, the metric will fail at runtime. Prefer input and actual_output unless the eval plan explicitly creates the reference fields.

If existing project metrics are present, use them first. If there are too many, tell the user: "You already have a lot of metrics here, which may make evals slow or hard to interpret. I recommend narrowing the first run to the highest signal metrics."

Reference-Based Metrics

Some metrics require reference fields. Use them sparingly unless the plan includes those expected values, because missing fields will cause metric errors.

Reference-based fields include:

• expected_output
• expected_outcome
• expected_tools
• context
• retrieval_context

Examples:

• ContextualPrecisionMetric and ContextualRecallMetric need expected_output.
• ToolCorrectnessMetric needs expected_tools.
• Multi-turn outcome metrics may depend on expected_outcome.
• RAG grounding metrics need retrieval_context.

If the dataset does not include the required fields, choose metrics that match available fields or update the dataset generation/loading plan first.

Common Single-Turn Metrics

Metric	What it checks	Required test case fields
AnswerRelevancyMetric	Output answers the input	input, actual_output
FaithfulnessMetric	Output is grounded in retrieved context	input, actual_output, retrieval_context
ContextualRelevancyMetric	Retrieved context is relevant to input	input, retrieval_context
ContextualPrecisionMetric	Relevant context is ranked highly	input, retrieval_context, expected_output
ContextualRecallMetric	Retrieved context covers expected answer	input, retrieval_context, expected_output
TaskCompletionMetric	Agent/app completed the task	input, actual_output
StepEfficiencyMetric	Agent/app completed the task efficiently without unnecessary steps	trace steps/tool activity
ToolCorrectnessMetric	Called tools match expected tools	input, tools_called, expected_tools
ArgumentCorrectnessMetric	Tool arguments are correct	input, tools_called
JsonCorrectnessMetric	Output matches expected schema	input, actual_output; constructor needs expected_schema
PromptAlignmentMetric	Output follows prompt instructions	input, actual_output; constructor needs prompt_instructions
GEval	Custom single-turn criteria	constructor needs name, criteria or evaluation_steps, and evaluation_params

Common Multi-Turn Metrics

Metric	What it checks	Required test case fields
ConversationCompletenessMetric	Conversation achieved the expected outcome	turns with role, content
RoleAdherenceMetric	Assistant stayed in role across turns	turns with role, content
TurnRelevancyMetric	Assistant turns are relevant	turns with role, content
TurnFaithfulnessMetric	Turns are faithful to retrieval context	turns with role, content, retrieval_context
TurnContextualRelevancyMetric	Turn retrieval context is relevant	turns with role, content, retrieval context
GoalAccuracyMetric	Conversation achieved the user's goal	turns with role, content
TopicAdherenceMetric	Conversation stayed on allowed topics	turns with role, content; constructor needs relevant_topics
ConversationalGEval	Custom multi-turn criteria	constructor needs name and criteria or evaluation_steps

Choosing Metrics

Ask what the user cares about in product terms first. Then map that to metrics.

Ask:

• What failure would be unacceptable in production?
• Is success about final answer quality, retrieved context, tool use, safety, conversation completion, or output format?
• Do we need a custom criterion because the product definition of "good" is domain-specific?
• Which fields does the dataset/test case actually contain?

Mappings:

• "Does it answer correctly?" -> AnswerRelevancyMetric or task-specific GEval
• "Is it grounded in docs?" -> FaithfulnessMetric plus contextual metrics
• "Did the agent finish the task?" -> TaskCompletionMetric
• "Did the agent take efficient steps?" -> StepEfficiencyMetric
• "Did it use the right tool?" -> ToolCorrectnessMetric
• "Did the chatbot complete the conversation?" -> ConversationCompletenessMetric
• "Did it stay in character?" -> RoleAdherenceMetric

If unsure for single-turn tracing, start with TaskCompletionMetric and StepEfficiencyMetric, then add 1-3 more E2E metrics only when the app's success criteria need them. Add component/span metrics only after the first run reveals where the app is failing.