Run Eval

Select a Task, Metric, and Version to run an evaluation Run, then review your Run history.

1. Start an evaluation run

①Run

A Run is a record of one evaluation run. It evaluates with the Criteria, Version, Task, and Metric you select in the Run Evaluation dialog.

Runs you start are listed on the Runs screen. The most recent Run is at the top.
Each Run contains the results of the selected Tasks for each Version.
Runs are numbered by default, such as Run#9.

Evaluations run per Task. Evaluation Core Concept explains how Tasks, Benchmark Sets, and Metrics relate.

②Applied Criteria

The Criteria selected by default depends on which screen you click Run Evaluation from.

Starting screen	Criteria selected by default
Runs screen	`Default Criteria`
Criteria Setting screen	The Criteria you were viewing

You can switch to another Criteria in the Criteria area at the top of the dialog.
If you switch to another Criteria and come back, the Tasks and Metrics you selected stay selected.
If you select a Criteria with no Tasks, you see This Criteria has no Tasks to evaluate. Add Tasks and Metrics in Criteria Setting, then run again.

1-1Start from the Runs screen
1. Click Runs in the left menu.
2. Click Run Evaluation in the top right.
3. Check that the Criteria at the top is Default Criteria.
When you start from the Runs screen, Default Criteria is applied by default. Read Applied Criteria first.
1-2Start from the Criteria Setting screen
1. Click Criteria in the left menu.
2. Open the tab of the Criteria to use for the evaluation.
3. Click Run Evaluation in the top right.
4. Check that the Criteria you were viewing is selected.

2. Select Run conditions

①Select evaluation targets

In the Run Evaluation dialog, select the Tasks and Metrics this Run evaluates. The dialog title is Select Tasks To Evaluate. Only the selected Tasks and Metrics are evaluated.

Area	What it evaluates	Display structure
`Application Eval (End to End)`	The final response of the whole Application	Task Group (evaluation area) → Subgroup → Task
`Module Eval`	Each Module that makes up the Application	Module (e.g., `Text to SQL`, `Router`) → Task

When you select a Task, all of its Metrics are selected too. Clear only the Metrics you don't need.
You can't select a Metric with a warning icon. It means a connection setting is required (Connection required to evaluate this metric.).
Turn on Get Output Only to skip Metric evaluation and fetch Outputs only. The Next button then becomes Start.

②Version

A Version is the unit for configuring the API to evaluate. Turn on the switch for each Version you want to evaluate among those registered to the Application.

Each Version you turn on generates its own Outputs from the same Task and Benchmark Set, so you can compare Versions against the same criteria.
For Application Eval (End to End), turn on Versions at the top of the dialog (e.g., Gemini-Pro, RAG-v1.2).
For Module Eval, turn on Versions in the Version row at the bottom of each Module (e.g., Router_v1, Router_v2).
Turning on 2 Versions doubles the number of Outputs and evaluations.

To run an evaluation, finish the API integration and Output Mapping for each Version. Application API integration explains how.

③Run confirmation dialog

Click Next to open the run confirmation dialog (Confirm Evaluation Run). Check the Run name and expected counts here.

Field	Meaning
`Run Name`	The name shown in the Run list. It can't be empty.
`Expected Output`	The number of new Outputs to generate: Task data count × number of Versions turned on.
`Expected Eval Result`	The number of new scoring results to generate: number of Outputs × number of selected Metrics.

Clicking Run Evaluation starts Output generation and evaluation immediately. You can't stop or cancel an evaluation once it starts. Check the expected counts before you run.

You can rename a Run after it runs, using the pencil icon next to its name in the Run list.

2-1Change the Criteria
1. Click the Criteria area at the top of the dialog.
2. Select the Criteria to use for the evaluation from the list.
If you switch Criteria and come back, the Tasks and Metrics you selected stay selected.
2-2Select evaluation items
1. Turn on the switch for each Version to evaluate.
2. Select the Tasks to evaluate.
3. Select the Metrics to evaluate for each Task.
4. Click Next.
You can select only the Tasks and Metrics you need. See Select evaluation targets.
2-3Confirm the run
1. Check the Run Name and expected counts. Run confirmation dialog explains each field.
2. Click Run Evaluation.
Once the run starts, you can't stop or cancel it.

3. View Run history

①Run status

Expand a Run on the Runs screen to see the progress of each Task and Version. The scoring stage (Evaluation) starts only after the Output fetching stage (Fetch) finishes.

Fetching Status shows the status of Output fetching.

Status	Meaning
`Waiting`	Outputs haven't been fetched yet.
`Fetching`	Outputs are being fetched.
`Fetched`	Outputs were fetched for all data.
`Partial Fetch`	Outputs couldn't be fetched for some data.
`Fetched Error`	Outputs couldn't be fetched for any data.

Metrics shows the scoring status of each Metric.

Status	Meaning
`Waiting`	Scoring hasn't started.
`Evaluating`	Scoring is in progress.
Score (e.g., `0.77`)	The average score of the scored data. A warning icon means the average excludes data that errored.
`Error`	Scoring failed for all data.
`Start Eval`	A Metric you didn't select at run time. Click it to evaluate only that Metric.

When all evaluations finish, Run Completed appears next to the Run name.
To evaluate all Metrics in the Start Eval state at once, click Run All Evals.
Rerun items that errored from the results screen. Eval History explains how.

3-1Check the Run list
1. Find the new Run at the top of the list.
2. Click the arrow to the left of the Run to expand results by Task. See Run status for what each status means.
3. Click a Task name to open its evaluation results.
The Eval Result document explains the results screen.

Frequently asked questions

Why is Default Criteria selected instead of the Criteria I want?Because Default Criteria is applied first when you start from the Runs screen.

Cause: Run Evaluation on the Runs screen always opens the dialog with Default Criteria.
Solution: Change the Criteria in the Criteria area at the top of the dialog. You can also click Run Evaluation on the Criteria Setting screen.

Why are the Tasks and Metrics I selected earlier still selected after I change the Criteria?Because the Tasks and Metrics you select are remembered for each Criteria. This is expected.

Cause: If you move to another Criteria and come back, the Tasks and Metrics you selected earlier are remembered.
Solution: Double-check your Task and Metric selections before you run. Also read Applied Criteria.

Why can't I stop a Run I started?Because there's no way to stop or cancel a Run once it starts.

Cause: Clicking Run Evaluation in the confirmation dialog starts response generation and evaluation immediately.
Solution: Check the expected counts and Run Name in the confirmation dialog before you run. Run confirmation dialog explains each field.