Evaluation
LLM evaluation with DATUMO: define criteria, score AI agent responses repeatedly, and confirm each prompt or model change actually improved quality.
DATUMO Evaluation lets you define which criteria, which data, and which methods to evaluate your AI service (Application) with, and then run that evaluation.
In short
For each Application, you create Criteria, a set of evaluation criteria, and build Tasks inside it by evaluation area. A Task consists of one Benchmark Set (the evaluation data) and multiple Metrics (the scoring methods). Evaluations run per Task.
Evaluation structure
| Unit | Role | Contains |
|---|---|---|
| Application | The AI service being evaluated | Multiple Criteria |
| Criteria | A set of evaluation criteria | 4 evaluation areas (General Performance, Policy & UX, Safety, Module Performance). Shown as Task Group in the UI. |
| Subgroup | A group that organizes Tasks by topic within an Application-level evaluation area | Multiple Tasks |
| Module | A Module configured on the Application, shown under Module Performance | Multiple Tasks |
| Task | The unit that defines what to evaluate and runs the evaluation | 1 Benchmark Set + multiple Metrics |
| Benchmark Set | The collection of data used to evaluate a Task | Multiple data items |
| Metric | A method that scores Output by calculating or judging it | - |
Group Tasks with Subgroups
In General Performance, Policy & UX, and Safety, you create Subgroups to group Tasks by topic. Module Performance groups Tasks by the Modules configured on the Application instead of Subgroups.
See an example
Take evaluating a financial advisory chatbot before launch as an example.
Example Task setup
Each Task has one Benchmark Set and multiple Metrics, as shown below.
| Task | Benchmark Set | Metric |
|---|---|---|
| Financial product lookup | 200 product inquiry questions | AnswerAccuracy, Similarity, ExactMatch |
| Product terms comparison | 80 product comparison questions | AnswerAccuracy, Similarity |
| Financial term explanations | 120 financial term questions | AnswerAccuracy, Similarity |
| Financial regulation guidance | 100 financial regulation questions | AnswerAccuracy, Similarity |
| Deposit interest calculation | 50 interest calculation problems | ExactMatch, AnswerAccuracy |
| Loan repayment calculation | 50 repayment calculation problems | ExactMatch, AnswerAccuracy |
| Answer completeness | 100 advisory scenario questions | Answer completeness (Custom) |
| Plain-language explanations | 100 advisory scenario questions | Ease of understanding (Custom) |
| Investment solicitation compliance | 80 investment advisory questions | Solicitation rule compliance (Custom) |
| Duty-to-explain compliance | 80 product sign-up consultations | Duty-to-explain compliance (Custom) |
| Consistent advisory tone | 100 advisory conversations | Tone consistency (Custom) |
| Complaint handling tone | 60 complaint conversations | Empathy (Custom) |
| Harmful request refusal | 150 harmful questions | refusal label, response harmfulness score |
| Illegal transaction refusal | 100 illegal finance questions | refusal label, response harmfulness score |
| Personal data leak prevention | 100 personal data elicitation questions | refusal label, Personal data exposure (Custom) |
| Third-party data lookup refusal | 80 third-party data requests | refusal label, Personal data exposure (Custom) |
| Jailbreak refusal | 120 jailbreak prompts | refusal label, response harmfulness score |
| Prompt injection defense | 100 injection prompts | refusal label, response harmfulness score |
| Question type classification | 100 question · answer type pairs | ExactMatch |
| Agent handoff decision | 80 handoff request questions | ExactMatch |
| Intent-based document retrieval | 100 question · answer document pairs | Context Relevancy, Precision@k |
| Latest-document-first retrieval | 80 question · answer document pairs | Context Relevancy, Precision@k |
| Grounded answer generation | 100 question · retrieved document pairs | Faithfulness, Response Relevancy |
| Answer source citation | 80 question · retrieved document pairs | Source citation (Custom) |
Example Benchmark Set setup
The example Benchmark Sets consist of Input, Reference, and Metadata Keys. Which Keys you include depends on what you evaluate. Below, the same question is used to build Benchmark Sets for three Tasks.
Financial product lookup (200 product inquiry questions): Checks whether the final answer matches the expected answer. Put the expected answer in Reference.
| id | query (Input) | expected_response (Reference) | source (Metadata) | human_reviewed (Metadata) |
|---|---|---|---|---|
| 1 | What happens to the interest if I close my Sample Term Deposit early? | The early withdrawal rate applies instead of the contracted rate. | Product FAQ | Y |
Intent-based document retrieval (100 question · answer document pairs): Checks whether the Retriever found documents that match the question. Put the documents it should find in Reference.
| id | query (Input) | reference_context (Reference) | source (Metadata) |
|---|---|---|---|
| 1 | What happens to the interest if I close my Sample Term Deposit early? | Sample Term Deposit product description (early withdrawal clause) | Product description |
Grounded answer generation (100 question · retrieved document pairs): Checks whether the Generator based its answer on the retrieved documents. Include the retrieved documents as Input. This Task's Metrics score without a Reference, so no Reference is included.
| id | query (Input) | retrieved_context (Input) | source (Metadata) |
|---|---|---|---|
| 1 | What happens to the interest if I close my Sample Term Deposit early? | Sample Term Deposit product description (early withdrawal clause) | Product description |
Metadata isn't used for evaluation. Use it to manage data, such as review status or source. Don't include Output that's generated during evaluation, such as response.
The Criteria, Subgroup, and Task names and the data counts in the example are illustrative. A Custom Metric is a Metric you create yourself.
Criteria
Criteria is a set that defines which criteria to evaluate an Application against. You can create multiple Criteria for one Application, so you can manage evaluation criteria separately by purpose, such as pre-launch validation and routine checks.
Criteria is divided into 4 evaluation areas. Create Tasks in the areas you want to check. Here are examples using the Tasks from the financial advisory chatbot above.
| Evaluation area | What it checks | Example Task |
|---|---|---|
| General Performance | Does it have the basic knowledge and abilities it needs? | Financial product lookup |
| Policy & UX | Does it follow internal policies and guidelines and respond in a way that fits the user experience? | Answer completeness |
| Safety | Does it avoid potentially dangerous responses? | Harmful request refusal |
| Module Performance | Does each Module in the Application do its job well? | Intent-based document retrieval (Retriever) Grounded answer generation (Generator) |
-
Application-level evaluation (General Performance, Policy & UX, Safety): Treats the whole Application as one evaluation target and evaluates the final response. Create multiple Subgroups under each evaluation area to group Tasks by topic.
-
Module-level evaluation (Module Performance): Evaluates individual Modules in the Application (e.g., Retriever, Generator) separately. Module Performance lists the Modules configured on the Application, and you build Tasks for each Module.
Module-level evaluation example
In a RAG-based Application, the Retriever finds documents and the Generator writes the answer.
Application-level evaluation scores only the final answer.
Module-level evaluation scores each Module's results separately. So when the final answer scores low, you can find which Module caused the problem.
When you evaluate only the Generator, include the retrieved documents in the Benchmark Set too.
The Modules shown in Module Performance follow the Application settings. To set up an Application, see G-5. Create an Application.
Task
A Task defines what to evaluate and is the unit that actually runs the evaluation. A Task consists of two things:
- One Benchmark Set: the data used for evaluation
- Multiple Metrics: the methods that score the results generated from the data
Example Tasks: Financial product lookup, Intent-based document retrieval
Because one Benchmark Set is scored by multiple Metrics at once, you can evaluate the same data from several angles.
When you first create an Application, Tasks that match the Application details you entered (Type, Domain, Language, and so on) are automatically recommended from templates and set up.
You can start Safety Tasks right away with templates made by DATUMO. Templates include evaluation data and Metrics. You don't have to design safety evaluations, such as harmful request handling, from scratch.
Benchmark Set
A Benchmark Set is the collection of reference data used to evaluate a Task.
It consists of multiple data items suited to the evaluation's purpose.
Columns
Each column in a Benchmark Set has a defined role in the evaluation.
Sample data
| id | query (Input) | expected_response (Reference) | source (Metadata) |
|---|---|---|---|
| 1 | What happens to the interest if I close my Sample Term Deposit early? | The early withdrawal rate applies instead of the contracted rate. | Product FAQ |
| 2 | What's the minimum to open a Sample Savings Account? | You can start from $10 per month. | Product description |
| 3 | What terms are available for the Sample Term Deposit? | You can choose 6, 12, or 24 months. | Product description |
Common Keys
DATUMO defines common column names as standard Keys. If your Benchmark Set column names match the standard Keys, connecting Metrics to columns (Key Mapping) is easier.
| Key | Main use | Description |
|---|---|---|
query | Input | User question |
retrieved_context | Input, Output | List of documents the Retriever retrieved |
response | Output | Answer from the Application or Module |
expected_response | Reference | Expected answer (ground truth). For multiple-choice questions, the correct option |
reference_context | Reference | List of ground-truth documents that should be retrieved |
Metric
A Metric defines how to calculate or judge the Output of an Application or Module. You can attach multiple Metrics to one Task.
Metric types
There are 5 Metric types.
| Type | How it scores | Examples |
|---|---|---|
| LLM as Judge (LLM judgment) | A Judge model judges using a set prompt and rubric. | Faithfulness, Context Relevancy |
| Rule-Based (rule-based) | Calculates a score with set rules or formulas (match, word overlap, ranking). | ExactMatch, BLEU, ROUGE, Precision@k |
| Embedding (embedding) | Converts sentences to vectors and calculates how close their meanings are. | BERT Precision |
| Statistical Metric (statistical) | Collects per-item judgments across the whole Benchmark Set and calculates ratios. | F1 Score, Recall |
| Custom (user-defined) | A Judge model judges using criteria you define. Choose a Generative Model or a Decision Model. | Answer completeness, Refund policy accuracy |
- Pick the first four types from templates provided by DATUMO.
- Create Custom Metrics yourself on the Task's Metric settings screen (
+ Add Metric›Custom). - You can also edit the prompt and rubric of an LLM as Judge Metric to change its evaluation criteria.
- If there's no ground truth (Reference), use an LLM as Judge or Rule-Based Metric.
- Choose the Judge model from the models registered in G-4. Register Gen & Judge Models. Embedding Metrics use the Embedding models registered on the same screen. Decision Models (Davi, Jev) are provided as DATUMO models.
- Custom Metrics can define additional result columns to return alongside Score (e.g., Reason, Risk Level).
End-to-end evaluation flow
An evaluation feeds Benchmark Set data into an Application or Module, then scores the generated Output with Metrics to produce an Eval Result.
- The Input values from the Benchmark Set are passed to the Application or Module.
- The Application or Module generates Output.
- Metrics score the Output. Depending on the Metric, Input and Reference are used too. Key Mapping specifies which columns feed into the Metric.
- The score and judgment rationale for each data item are saved as the Eval Result.
Example: how one data item is scored
Data and scoring results
This shows how one data item in the Financial product lookup Task is evaluated.
| Item | Value |
|---|---|
| query (Input) | What happens to the interest if I close my Sample Term Deposit early? |
| response (Output) | If you close it early, interest is calculated at the early withdrawal rate, not the contracted rate. |
| expected_response (Reference) | The early withdrawal rate applies instead of the contracted rate. |
| Metric | Scoring criteria | Result |
|---|---|---|
| ExactMatch (Rule-Based) | 1.0 if the sentence exactly matches the ground truth, otherwise 0.0 | 0.0: wording differs, so no match |
| Similarity (LLM as Judge) | Judges whether the core meaning matches the ground truth beyond wording differences | High score: same meaning |
The answer means the same as the ground truth, but ExactMatch gives it 0 because the wording differs. Because the same answer can be scored differently depending on the Metric, choose Metrics that fit your evaluation's purpose, and attach several if needed.
Eval Results are saved with the values at evaluation time. Changing the Benchmark Set's Input or Reference or the Metric settings after an evaluation doesn't change existing results. Metadata, however, is shown with its latest values in Eval Result and the Dashboard.
Next steps
| Document | What it covers |
|---|---|
| E-1. Create & manage Criteria | How to set up Criteria, Subgroups, and Tasks |
| E-2. Manage Tasks | How to manage Task basics, Schema, Metrics, and Benchmark Sets |
| E-3. Manage Benchmark Sets | How to upload Benchmark Sets and generate data with AI |
| E-4. Add & manage Metrics | Default and Custom Metric types and how to add them, Key Mapping, Preview |
| E-5. Run Eval | How to run evaluation Runs and view their history |
| E-6. Eval History | How to view evaluation results and manage Errors and Retries |
| E-7. Dashboard | How to visualize and analyze evaluation results |