> For the complete documentation index, see [llms.txt](https://docs.datumo.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.datumo.com/documentation/documentation-en/evaluation/editor.md).

# Core concepts and structure

## Core Concept: Task·Benchmark set· Metric

These are the three basic units you should learn first in Datumo Evaluation.

***

## Evaluation structure

One **Task**can have multiple **Benchmarkset**can have, and each **Benchmark set**is paired with a **Metric** one.

<figure><img src="/files/8wjGgoHZ182CfcHB3pAm" alt=""><figcaption></figcaption></figure>

<table><thead><tr><th width="156.53125">Concept</th><th>Question answered</th><th>In one word</th></tr></thead><tbody><tr><td><strong>Task</strong></td><td>What does AI do?</td><td><strong>What</strong></td></tr><tr><td><strong>Benchmark set</strong></td><td>Where will the evaluation be conducted?</td><td><strong>Where</strong></td></tr><tr><td><strong>Metric</strong></td><td>How is that perspective scored?</td><td><strong>How</strong> </td></tr></tbody></table>

### Concept definition

**Task: Capability to evaluate**

Task is a unit that defines the ability or task to be evaluated.

* Examples: search for financial product information, document retrieval based on question intent, evaluation of answer completeness
* One Task can have multiple Metrics.
* One Task is connected to one Benchmark Set.

***

**Benchmark Set: Data to be evaluated**

The Benchmark Set is a collection of reference data used to evaluate a Task.

* It is composed of multiple data points suitable for the evaluation purpose.
* It can have column information such as Input, Reference, and Metadata.
* When the evaluation data is representative, the term Benchmark Set is used instead of Dataset.

***

**Metric: Method of evaluation**

Metric is a unit that defines how to calculate or judge evaluation results.

* Examples: LLM-as-a-Judge, Exact Match, Pass/Fail, Similarity Score
* Multiple Metrics can be connected to one Task.
* Even when using the same Benchmark Set, the meaning of the evaluation result can vary depending on the Metric.

***

## Evaluation Framework example

An Evaluation Framework is a structure that inputs Benchmark Set data into an Application or Module, evaluates the generated Output, and produces an Eval Result.

### 1. Common Evaluation Framework

The basic evaluation structure that passes the Input of the Benchmark Set to an Application or Module, evaluates the generated Output with a Metric, and produces an Eval Result.

<figure><img src="/files/kZgSqXobnyzYgmG76Y2f" alt=""><figcaption></figcaption></figure>

### **2. Retriever metric-based evaluation example**

Retriever evaluation checks whether an appropriate Context was retrieved for the Query.

<figure><img src="/files/iHO50vEOdYQLvkowloj4" alt=""><figcaption></figcaption></figure>

Based on the search results, the following Metrics can be applied.

* Context Relevancy
* Context Precision
* Precision\@K

### **3. Generator metric-based evaluation example**

Generator evaluation checks the quality of the Response generated based on the Query and Reference Context.

<figure><img src="/files/8dV3l9MN630aWIHL80mh" alt=""><figcaption></figcaption></figure>

Based on the generated response, the following Metrics can be applied.

* Response Relevancy
* Faithfulness
* Precision

***

## Evaluation tasks

* E-1. Creating and managing Criteria — How to configure Criteria, Subgroup, and Task
* E-2. Task Management — How to manage Task basic information, Schema, Metric, and Benchmark Set
* E-3. Managing Benchmark Sets — How to upload Benchmark Sets and generate AI-based data
* E-4. Adding and managing Metrics — How to create Metrics, map Keys, and set up Preview
* E-5. Run Eval — How to run an evaluation Run and check history
* E-6. Eval Result — How to check evaluation results and manage Errors and Retries
* E-7. Dashboard — How to visualize and analyze evaluation results