Manage Tasks

Learn how to create a Task from a template or as a Custom Task, how the Task screen is laid out, and how to manage Key information.

1. Create a Task

①Task

A Task is the unit that defines what to evaluate. Create one Task for each ability you want to evaluate. Evaluations also run per Task.

A Task is made up of two things:

One Benchmark Set: the data used for evaluation
Multiple Metrics: the methods for scoring the results

Add Tasks under a Subgroup or Module on the Criteria screen.

Task: Financial product information search
├─ Benchmark Set: 200 product inquiry questions
└─ Metric: AnswerAccuracy, Similarity, ExactMatch

Because the same data is scored with multiple Metrics, you can evaluate one Task from several angles.

②Templates and Custom Tasks

There are two ways to create a Task.

Method	What it starts with	Use it when
Template	A description, Metrics, and evaluation data prepared by DATUMO	A template matches what you want to evaluate
`Custom Task`	Only a name and description	No template fits

You can filter the template list by Task Group, Application Type, Module Type, Domain, and Language.
Each template card shows a description, data count, Metrics, and tags.
When you create a Task from a template, you need to connect the template Keys to Task Keys. See Connect Keys.
For a Custom Task, you add the Benchmark Set and Metrics yourself.

③Key types

A Key is the name of a field in the evaluation data. Think of it as a column name in a table. There are four types of Keys, and each plays a different role in evaluation.

Key type	Meaning	Where to edit	Example
Input	The value passed to the Application	Application Setting (Settings > Application > `Agent Connection`)	`query`
Output	The result the Application produces from the Input	Application Setting	`response`, `retrieved_context`
Reference	Ground-truth information a Metric compares against when scoring	Task screen	`expected_response`, `reference_context`
Metadata	Extra information not used in evaluation	Task screen	Reviewer, creation date, source

Input and Output Keys are defined by the Application. All Tasks in the same Application share them.
You don't add Output to the Benchmark Set in advance. The Application generates it when the evaluation runs.
Reference is optional. Some Metrics score without a Reference.
Reference and Metadata Keys are connected to file columns when you import Benchmark Set data.

For example, one Benchmark Set data item looks like this:

query (Input)	expected_response (Reference)	source (Metadata)
What's the minimum amount to open a sample installment savings account?	You can open one starting at 10,000 KRW per month.	Product description

See Evaluation Core Concept for commonly used standard Key names.

④Connect Keys

Templates use their own Key names, which may differ from your Application's Key names. So when you create a Task from a template, you connect each template Key to a Task Key.

Template Required Key: a Key used by the template you're adding
Task Key: the Key in this Task to connect to the template Key

For each template Key, choose one of the following:

Option	What it does
Input or Output Key	Connects it to the Application's Input or Output Key.
`Auto-create as Reference Key`	Creates a new Reference Key with the same name.
`Auto-create as Metadata Key`	Creates a new Metadata Key with the same name.
`Clear Column Values`	Doesn't use this Key and clears the values in that column.

For example, if the template Key is response and the Application's Output Key is answer, connect the two. You can create the template's expected_response as a new Key with Auto-create as Reference Key.

You must connect every template Key before you can add the Task.
You can't connect two template Keys to the same Task Key.

1-1Create from a template
1. Click + New Task on the Criteria screen.
2. Select a template.
3. Click Add Task.
- If you don't see the template you want, use the filters or search box at the top.
- See Templates and Custom Tasks for the difference between the two.
1-2Connect template Keys
1. Click a Key marked Mapping incomplete. If this is your first time, read Connect Keys first.
2. Choose the Task Key to connect.
3. Click Complete.
4. Find the added Task in the Task list.
- You must connect every template Key to a Task Key.
- If there's no Key to connect to, you can create a new Reference or Metadata Key.
1-3Create a Custom Task
1. Click + New Task.
2. Select Custom Task.
3. Enter a Task Name and Description.
4. Click Complete.
5. Find the added Task on the Task Detail screen.
Description is optional.

2. View the Task screen

①Task screen layout

Click a Task name on the Criteria screen to open the Task screen. The Task screen is divided into four areas.

Area	What it shows
`Task Info`	Task Group, Task name, and description
`Schema Setting`	This Task's Keys, divided into Input, Output, Reference, and Metadata.
`Metric`	The Metrics in this Task, with their type, required Keys, result fields, and Judge model.
`Benchmark Set`	The evaluation data, with Keys shown as table columns.

Add & manage Metrics explains how to add Metrics, and Manage Benchmark Sets explains how to add data.

2-1Screen layout
1. Click a Task name on the Criteria screen. Task screen layout describes each area.
2. Check the name, description, and Task Group in Task Info.
3. Check the Task's Keys in Schema Setting. Key types explains what each Key does.
4. Check the Task's Metrics in Metric.
5. Check the evaluation data in Benchmark Set.

3. Manage Key information

3-1Edit Input and Output Keys
1. Click Go to Application Setting.
Edit Input and Output Keys on the Application Setting screen.
3-2Rename a Key
1. Click Edit in Reference.
2. Click Edit next to the Key you want to change.
3. Change the name and click Save.
4. Click Complete.
Edit Metadata Keys the same way from Edit in Metadata.
3-3Add a Key
1. Click + Add Key.
2. Enter a new Key name and click Save.
3. Click Complete.
3-4Delete a Key
1. Click − next to the Key you want to delete.
2. Click Complete.
You can't undo deleting a saved Key. Existing evaluation results are kept.

Frequently asked questions

Why can't I edit Input and Output Keys on the Task screen?Because Input and Output Keys are managed in Application Setting.

Cause: Input and Output Keys are managed in Application Setting. The Task screen shows only a link there instead of Edit.
Solution: Click Go to Application Setting and edit them on the Application Setting screen. Key types lists where to edit each Key type.

Why can't I click Complete after editing a Key?Because a Key hasn't been saved yet.

Cause: Complete is disabled while you're renaming a Key or entering a new one.
Solution: Click Save next to the edited Key first. Delete any Keys you don't need with −.

Why can't I click Complete when adding a Task from a template?Because some template Keys aren't connected yet.

Cause: You can't add the Task while any Key is marked Mapping incomplete.
Solution: Click the red cell and choose a Task Key, or create a new Reference or Metadata Key. Connect Keys lists the available options.