Manage Benchmark Sets

Learn how to build a Benchmark Set by uploading evaluation data, how to edit, delete, and add data, and how to generate data with AI Tools.

1. Open a Benchmark Set

A Benchmark Set is the collection of data used as the basis for evaluating a Task. Build it from data that represents the evaluation target. Each row in the table is one evaluation data item.

①Key types

Each column in a Benchmark Set is one of the Task's Keys. Keys fall into three types based on their role in evaluation.

Key type	Meaning	What it's used for	Example
Input	The value passed to the Application or Agent	Passed as-is to the Application during evaluation.	`query`
Reference	Ground-truth information used for scoring	Metrics compare it with the Output. Some Metrics score without a Reference.	`expected_response`, `gold_context_id`
Metadata	Extra information not used in evaluation	Used to manage data, such as reviewer, creation date, and source.	`query_author`, `query_created_at`, `human_reviewed`

You don't add Output to the Benchmark Set in advance. When you run an evaluation, the Application takes the Input and produces the Output (e.g., response, answer, retrieved_context).

Here's an example with values filled in for each Key type:

id	query (Input)	expected_response (Reference)	source (Metadata)
1	What happens to the interest if I close a sample term deposit early?	The early termination rate applies instead of the agreed rate.	Product FAQ
2	What's the minimum amount to open a sample installment savings account?	You can open one starting at 10,000 KRW per month.	Product description

②Common Keys

DATUMO defines commonly used column names as standard Keys. Matching your column names to the standard Keys makes it easier to connect Metrics and Keys.

Key	Type	Meaning
`query`	Input	The user's question
`retrieved_context`	Input, Output	The list of documents found by the Retriever
`response`	Output	The Application's or Agent's answer
`expected_response`	Reference	The expected answer (ground truth). For multiple-choice questions, the correct option
`reference_context`	Reference	The list of ground-truth documents that should be retrieved

AI Tools templates such as Generator show names the template uses, like context and source_document. These names aren't Keys. When you use the template, you pick an actual column for each name.

1-1Open the Benchmark Set screen
1. In the left menu, click a Task under Criteria.
2. Click Manage Benchmark on the Task screen to open this screen.

2. Upload data

①Connect columns when uploading

When you upload a file, the Key Mapping window opens. In this window, you connect the file's columns to Task Keys.

On the left, Task Key lists this Task's Keys, grouped into Input Key, Reference Key, and Metadata Key.
On the right, under CSV Column, pick one file column for each Key.
The info box at the top shows all the file's columns. Columns you've already connected are blue.
Keys without a selected column are marked Mapping incomplete.
For a Metadata Key, you can choose Manual Input instead of a column. The value you enter goes into every uploaded row.

File columns you don't connect are collected under Unselected Columns. Unselect columns will be added as Metadata. is checked by default. When it's checked, these columns are added as new Metadata Keys. When you uncheck it, these columns aren't uploaded.

For example, say the file columns are query, expected_response, created_by, and id.

File column	Result
`query`	Connected to Input Key `query`
`expected_response`	Connected to Reference Key `expected_response`
`created_by`	Connected to Metadata Key `query_author`
`id`	Not connected. Added as Metadata Key `id` if the box is checked

2-1Upload a file
1. Click Upload File in the top-right corner.
2. Drag and drop a file into the upload area. You can also click the area to choose one.
3. Click Next.
You can upload only csv and xlsx files.
2-2Connect columns
1. Check the file's column names in the info box at the top.
2. Pick the file column to connect to each Task Key. If you're new to Keys, read Key types first.
3. Fill in any cell marked Mapping incomplete by picking a column.
4. Click Complete.
- Columns you don't connect are added as Metadata. If you don't want that, uncheck the box below.
- Uploaded data is added after the existing data.
- Connect columns when uploading explains the rules in detail.

3. Edit data

3-1Modify data
1. Click Edit in the top-right corner.
2. Click the cell you want to change and edit its content.
3. Click Save.
4. Click Save in the confirmation dialog.
If you click Cancel, your changes aren't saved.
3-2Delete data
1. On the Edit screen, select the rows to delete with the checkboxes in the ID column.
2. Click - Delete Data.
3. Click Delete in the confirmation dialog.
4. Click Save to save.
You can't undo deleting data.
3-3Add data
1. On the Edit screen, click + Add Data.
2. Enter values in the new row at the bottom.
3. Click Save to save.

4. Generate data with AI Tools

①AI Tools

AI Tools use a generation model to create or modify values in a Benchmark Set. There are four tools.

Tool	What it does	When to use it	Result
Generator	Creates or replaces column values based on other columns.	When you have questions but no expected answers, or documents but no questions.	Fills the selected column. It can also create a new column.
Modifier	Edits the content of an existing column.	When you want to change wording, such as fixing typos, changing tone, or shortening sentences.	Replaces the selected column's values with new ones.
Augmenter	Adds new rows based on existing columns and rows.	When you don't have enough data, or want more data of an underrepresented type.	Adds new rows at the bottom of the table.
Validator	Validates data in an existing column.	When you want to check grammar, fluency, or how well answers match questions.	Adds a score column (`<name>_score`) and a reason column (`<name>_reason`).

When you choose a tool, a list of templates appears (e.g., Expected Response Generator for Generator, Proofreading for Modifier).
To write your own instructions without a template, choose Custom.
If the Benchmark Set has no data, you can use only Augmenter.
For Validator, choose a scoring method from 1-5, Pass/Fail, High/Medium/Low, and Custom.

②Source Column and Target Column

The Source Column is the column used as a reference when generating values. It's only referenced, and its values don't change. The Target Column is the column to fill or change. Each tool asks for different columns.

Tool	What you choose
Generator	A Source Column and a Target Column. Choose `Create New Column` as the Target to create a new column. Set the new column as Reference or Metadata.
Modifier	Only a Target Column. Its values are edited.
Augmenter	The columns to fill in new rows and the rows to reference. We recommend 3–50 reference rows, with a maximum of 50. You also enter the number of rows to create.
Validator	Only a Source Column. Results go into new columns.

Rows with an empty Source Column value are skipped. Check the number of usable rows in Available data to generate on the screen.

In Fill Options, set the range of cells to fill (e.g., for Generator, choose from Update or fill all cells, Fill empty cells only, and Update or fill selected cells only).

③Prompt Setup and Draft

Prompt Setup is where you write instructions for the generation model. There are two modes:

Guided Mode: Fill in just two fields (e.g., for Generator, the column relationship (Source–Target Column Relationship) and the instruction (Instruction)).
Edit Prompt: Edit the System Prompt directly. If you switch back to Guided Mode, your direct edits are lost.

A Draft is one set of settings that combines instructions and a generation model. Drafts are numbered at the top of the screen, like Draft #1.

+ Create New Draft: Creates a new empty Draft. Previous Drafts are kept.
Draft History: Select a previous Draft to view it again.

④Samples and full data

AI Tools generate a sample first. If the results look good, you then generate the full data.

Type	Button	What it generates	Applied to
Sample	`Generate 15 Sample Data`	Up to 15 rows. If there are fewer than 15 rows, all of them.	Preview only, in the table on the right.
Full data	`Generate Full Data`	Every row in the selected range	Applied to the Benchmark Set.

Full data is generated with the settings of the Draft currently on screen.
For Validator, the buttons are Validate 15 Sample Data and Validate Full Data.
For Augmenter, the sample button is Generate 15 Sample Row. If you're creating fewer than 15 rows, it generates that many.

4-1Choose a tool and template
1. Click AI Tools in the top-right corner.
2. Choose a tool. See AI Tools for how the tools differ.
3. Choose a template. To work without a template, choose Custom.
If the Benchmark Set has no data, you can use only Augmenter.
4-2Choose columns to generate
1. Choose the column to use as a reference (Source Column). If this is your first time, read Source Column and Target Column first.
2. Click Next.
3. Choose the column to fill (Target Column).
4. Choose the range to fill in Fill Options.
5. Click Start.
To generate values in a new column, choose Create New Column.
4-3Review the sample
1. Enter the column relationship and instructions in Prompt Setup. See Prompt Setup and Draft for the input modes.
2. Choose a generation model.
3. Click Generate 15 Sample Data.
4. Review the generated results in the table on the right.
- A sample has up to 15 rows. Samples aren't applied to the Benchmark Set.
- See Samples and full data for how they differ from full data.
4-4Add a Draft
1. If you're not happy with the results, click + Create New Draft.
2. Edit the instructions in the new Draft and generate a sample again.
You can view previous Drafts again in Draft History.
4-5Generate full data
1. Click Generate Full Data.
2. Click Generate in the confirmation dialog.
3. Check the generation progress in the notice above the table.
Full data is generated with the settings of the Draft currently on screen.

Frequently asked questions

Why can I choose only Augmenter in AI Tools?Because the Benchmark Set has no data. With no data, you can use only Augmenter.

Cause: The other tools create or edit values based on existing data. With zero data items, there's nothing to base them on. See AI Tools for how the tools differ.
Solution: Upload data with Upload File first, then choose another tool.

Why can't I upload my file?Because you can upload only csv and xlsx files.

Cause: The upload area doesn't accept other file formats.
Solution: Save the file as csv or xlsx and upload it again.