Add & manage Metrics

Learn how Metrics are divided into Default and Custom, which Metrics each type includes, and how to add them to a Task.

A Metric is how you calculate or judge the Output of an Application or Agent and assign a score. You can connect multiple Metrics to one Task. Metrics are divided into Default, which DATUMO prepares for you, and Custom, which you build yourself.

1. Default and Custom

①How the two types differ

	Default	Custom
What it is	Metrics built and provided by DATUMO	Metrics you build with your own service criteria
How you make it	Pick from the list and add it	Write it with a Generative Model or a Decision Model
Scoring method	LLM as a Judge, Rule-based, Embedding, Statistical	A Judge model judges using criteria you define
Eval result	LLM as a Judge returns a score and a reason; the others return a score	Generative Model returns a score and the result Keys you define; Decision Model returns a score and answers to Additional questions
Key Mapping	Required	Not needed (the Keys you choose are connected as-is)

②Which type to choose

Goal	Type
The evaluation method you want is already in the list	Default
You evaluate with your own criteria and want a reason along with the score	Custom › Generative Model
You evaluate with your own criteria and want to score a lot of data quickly, cheaply, and consistently	Custom › Decision Model

In a Task, click + Add Metric. Metric Type shows the DATUMO Default card under Default, and the Generative Model and Decision Model cards under Custom.
The card tags also show the difference. Generative Model: Reasons, Flexible, Slower, Higher cost. Decision Model: Fast, Low cost, Consistent, No reasons.
The same answer can get different scores depending on the Metric. An answer with the same meaning but different wording scores 0 on ExactMatch and high on Similarity. If your goals differ, connect several Metrics together.

③Score range

Default Metric scores range from 0 to 1, and a higher score means better quality. For Custom, we also recommend 0–1 so scores are easy to compare with other Metrics. A Decision Model converts the chosen answer to a 0–1 score automatically.

1-1Start adding a Metric
1. Click + Add Metric on the Task Detail screen.
2. Choose a creation method in Metric Type. See Which type to choose for the difference.
- DATUMO Default: Choose from the provided Metric templates.
- Generative Model: Enter your own evaluation prompt. You also get reasons.
- Decision Model: Set questions and options to score quickly.

2. Default Metrics

①Metrics in Default

DATUMO provides 131 Default Metrics you can use right away. They also come included in Task templates, so when you pick a template, the matching Metrics are selected with it. There are four scoring methods.

Scoring method	Count	How it scores	Example
LLM as a Judge	101	A Judge model judges with a set prompt and rubric	Faithfulness, Answer Accuracy
Rule-based	23	Calculates a score with a formula such as match, word overlap, or ranking	Exact Match, BLEU, ROUGE, Precision@k
Embedding	4	Turns the answer and the reference into vectors and calculates how close their meanings are	Semantic Similarity, Factual Consistency
Statistical	3	Collects per-item judgments across the whole Benchmark Set and calculates ratios	F1 Score, Recall, FPR

The full list by evaluation area (10 areas, including safety, retrieval quality, answer accuracy, and agents) is in Introduction to Datumo Metrics. If you're new, start with these.

What you want to check	Metric	Required Keys
Whether harmful questions are blocked	F1 Score	`query`, `query_safety_label`, `safety_taxonomy_lv2`
Whether it makes up anything not in the documents	Faithfulness	`response`, `retrieved_context`
Whether it says the same thing as the correct answer	Answer Accuracy	`query`, `response`, `expected_response`

②Add and map Keys

Choose DATUMO Default to see the list of Metric cards. Each card shows Method, Required (the Keys used for scoring), and Eval Result. You can pick several at once.
After adding, use Key Mapping to connect the Keys the Metric requires to the Task's Keys. For each Metric Required Key on the left, choose a Task Key on the right.
Keys that aren't connected are marked Mapping incomplete and appear in red on the Metric card.

For example, connecting Answer Correctness looks like this:

Metric Required Key	Task Key
`query`	`query` (Input)
`response`	`answer` (Output)
`expected_response`	`expected_response` (Reference)

2-1Add from a template
1. Select DATUMO Default.
2. Select the Metric to add.
3. Click Add 1 Metric.
You can select multiple Metrics at once. The number on the button is how many Metrics you've selected.
2-2Connect Keys
1. In the Metric list, click a Metric with red Keys.
2. Check the Keys the Metric requires in Required Key. See Add and map Keys for how connections work.
3. Click Mapping incomplete and choose the Task Key to connect.
4. Click Save.
You must connect every required Key before you can evaluate with this Metric.

3. Custom Metrics

①Metrics in Custom

For criteria that aren't in the list, build a Custom Metric. There are two ways to build one: Generative Model and Decision Model.

	Generative Model	Decision Model
How it judges	Reads the prompt and answers in text	Picks one of the options you set
How you make it	Write a System Prompt and a User Prompt yourself	Follow the on-screen guide to choose and enter evaluation Keys, questions, and options
Result	A score and the result Keys you define (e.g., reason)	A 0–1 score and answers to Additional questions. No reason
Speed and cost	Slower and more expensive	Faster and cheaper
Judge model	An LLM you registered	Davi, Jev

②Build with a Generative Model

In Metric Info, enter a name and an optional description, then choose the Judge model in Judgement Model under Select Model.
In System Prompt, write the role, evaluation criteria, scoring rules, and output format.
In User Prompt, mark where the values to evaluate go using Keys such as {{query}}. Click a Key shown above the prompts to insert it at the cursor. You must insert at least one Key in either field before you can save.
In Evaluation Result, set the fields you want back. Score (Number) is there by default; add fields such as reason (text) with + Add Key. A column with each name appears in the results table.

Prompt example: build Answer Correctness yourself

System Prompt

You are an evaluation judge for answer correctness.

Evaluate whether the response correctly answers the user's question based on the reference answer.

Scoring rubric:
- 1.0: Fully correct and consistent with the reference answer
- 0.5: Partially correct but missing or misstating important details
- 0.0: Incorrect, misleading, or contradicts the reference answer

Return:
- score: float between 0 and 1
- reasoning: brief explanation of the score

Output must be valid JSON.

User Prompt

Query: {{query}}
Response: {{answer}}
Reference: {{expected_response}}

The System Prompt asks for score and reasoning, so set Score and Reasoning (text) in Evaluation Result.

3-1Write the prompt
1. Select Generative Model.
2. Enter a Metric Name.
3. Enter the System Prompt and User Prompt. Read Build with a Generative Model first to see what to write.
4. Set the Keys to receive as results in Evaluation Result. See Build with a Generative Model for details.
5. Click Save.
- You must insert at least one of the Keys shown above into the prompt.
- We recommend a 0–1 score range so scores are easy to compare with other Metrics. See Score range.

③Build with a Decision Model

In Evaluation Key, choose the Keys the Judge model reads (Input, Output, and Reference if needed).
In Evaluation Instructions, write a question and choose a score type: Binary (Yes 1 / No 0), Continuous (levels from 1 to 0), or Choice (1 to 0 in option order). Only the first question becomes the score; Additional questions appear separately in the results.
In Judgement Model, choose the Judge model. We recommend Davi for Korean data and Jev for English data.
Enter a name in Metric Info and click Save. A Custom Decision Model card is added to the Task's Metric list.

Question examples and score types in detail

Score type	Use it when	Options	Score
`Binary`	You judge right or wrong	`Yes`, `No`	Yes 1, No 0
`Continuous`	You rate a degree in levels. Selected by default.	3 levels by default (1, 0.5, 0). 2–101 levels	A value between 1 and 0. Values between levels are kept.
`Choice`	You pick one of set options	3 by default	Values spread evenly from 1 to 0 in option order

Describe what kind of answer fits each level or option. The more specific the description, the more accurate the scoring.
Add more with + Add Level (Continuous) or + Add Option (Choice). Add Additional questions with + Add Instruction; each needs a name (e.g., clarity).
If you save without an Output Key, a confirmation appears. Click JSON to view and edit the same content as JSON.

For example, to check whether answers explain the refund policy correctly:

Question	Type	Options
Score: Does the answer clearly explain whether a refund is available?	`Binary`	Yes: States clearly whether a refund is available and on what terms / No: Is vague or invents terms
`clarity`: How easy is the answer to understand?	`Continuous`	1: Clear and concise / 0.5: Mostly clear / 0: Hard to follow
`response_tone`: What is the tone of the answer?	`Choice`	formal / casual / curt

Choose the Judge model from Davi (Datumo) and Typesafe (Datumo) (Jev) under DATUMO Model.

Model	Strengths	Use it when
Davi	Built by DATUMO. More accurate for Korean evaluations.	You evaluate Korean data
Jev	Built by Typesafe. Always answers in the required format, so scores stay consistent. Choose `Jev-latest` or `Jev-preview`.	You evaluate English data

3-1Choose Decision Model
1. On the Task Detail screen, click + Add Metric.
2. Under Custom, choose Decision Model. See Which type to choose for the difference.
Decision Model scores quickly and cheaply but gives no reasons. If you need reasons, choose Generative Model.
3-2Enter Keys and questions
1. In Evaluation Key, choose the Keys the model reads. The guide on the right explains each Key.
2. In Evaluation Instructions, choose a score type. See Build with a Decision Model.
3. Write the question in Instructions and describe each level.
4. To add a question, click + Add Instruction.
Only the first question becomes the score. Additional questions don't affect the score and appear separately in the results.
3-3Choose a model and save
1. Choose the Judge model in Judgement Model. Davi is selected by default.
2. To use Jev, choose a version under Typesafe (Datumo).
3. Enter the Metric Name in Metric Info.
4. Click Save.
5. A Custom Decision Model card is added to the Task's Metric list.
For Korean data, we recommend Davi.

④Evaluation Preview

Test-run a saved Custom Metric on example data to check that it scores as intended. Enter a Gold Score (the expected correct score) and click Run Test to see the difference between the Evaluation Score and the Gold Score. For a Decision Model, you can also expand a result row to see the probability for each option. If results aren't what you expect, revise the prompt or questions and run it again.

Preview results in detail

The table columns are the Keys the Metric uses. These cells can't be left empty; the Gold Score is optional.
Accuracy shows how close the Evaluation Score is to the Gold Score. For example, if a row with a Gold Score of 1.0 gets 0.95, the difference is -0.05; if the two scores are the same, it shows Match.
For a Decision Model, Run Test results stack as Test #1 and so on, with the time, Judge model, and score type. Binary and Choice show Match or Mismatch, and Continuous shows the difference (e.g., +0.08).
Avg.Difference is the average difference between the model score and the Gold Score on rows that have a Gold Score. The closer to 0, the closer to Gold.

What to check	Where to look
Whether scores match Gold	`Match`, `Mismatch`, difference, `Avg.Difference`
How confident the model was	Option probabilities in an expanded row
Answers to Additional questions	Question name columns under `Additional`

3-1Enter examples
1. Add rows with + Add Query/Response.
2. Enter an example value in each cell.
- You must save the Metric settings before you can use Preview.
- Gold Score is optional. See Evaluation Preview for what it means.
3-2Run Test
1. Click Run Test.
2. Check the scores and Accuracy in the results.
3-3Check Decision Model results
1. On a Decision Model Metric, enter example values and Gold values, then click Run Test.
2. Check Match, Mismatch, or the difference for each score.
3. Use Avg.Difference to see how close the scores are to the Gold Score.
4. Click a row to expand the probability for each option.
If results aren't what you expect, click Edit, revise the questions and option descriptions, and run it again. See Evaluation Preview for details.

Frequently asked questions

Why do I see red Keys on a Metric card?Because some Keys aren't connected yet.

Cause: Keys the Metric requires that aren't connected to a Task Key appear in red.
Solution: Open the Metric and choose a Task Key from Mapping incomplete in Required Key. Then click Save.

Why can't I click the Run Test button?Because the Metric settings haven't been saved yet.

Cause: You can't run Preview on an unsaved Metric.
Solution: Click Save in the bottom-left corner to save your settings.

Why are Preview cells highlighted in red?Because some required example values are empty.

Cause: Cells for Keys the Metric requires can't be left empty. Only Gold Score can be left empty.
Solution: Enter example values in the red cells, then click Run Test again.

Why is there no reason in my Decision Model results?Because Decision Models don't write reasons.

Cause: A Decision Model doesn't generate text. It only picks one of your options, which makes it fast and cheap but gives no reason.
Solution: If you need reasons, create the Metric with a Generative Model. With a Decision Model, click a result row to see the probability for each option.

Why can't I save my Generative Model Metric?Because the prompt doesn't contain any Keys.

Cause: System Prompt or User Prompt must contain at least one Key.
Solution: Insert a Key shown above, such as {{query}}, into the prompt. See Build with a Generative Model for an example.