Platform Docs
Try DatumoEval

Eval Report

Review the release verdict, analysis by judgment group, and version comparison results in the Run Report.

1. Open the Run Report

①Run Report layout

The Run Report is a results report generated automatically when a Run finishes. It judges whether you can release based on the Release Gate. The Release Gate is the judgment criteria you set for each Criteria. You don't need to create any charts.

The Report screen is divided into the following parts, from top to bottom.

AreaWhat it shows
HeaderWhich Run was evaluated against which criteria.
Run Detail, Criteria Detail, Share
Run Detail also shows usage (cost and tokens).
Conclusion SummaryPass or Fail on each judgment group card for each version
Usage tabThe cost, tokens, and time this Run used. See View usage.
Judgment group tabsVersion comparison and failure cause analysis by group
  • When you add a Metric to a Run, the Report is regenerated. During this time, the status is Run Updating and Report Updating.
  • You can still view the existing Report while it's being regenerated.
  • Create & manage Criteria explains how to set the Release Gate.

  1. 1-1Open the Run Report

    1. Click Evaluation Dashboard in the left menu.
    2. The Run Report under Default Page opens. See Run Report layout for the screen layout.
    • The Report is generated automatically when a Run finishes.
    • A Run that's in progress has no Report.

2. Check the Run and criteria

①Judgment groups

A judgment group is a unit the Report judges as Pass or Fail separately. Each evaluation area of the Criteria and each Module becomes a group.

Judgment groupEvaluation targetResult structureDefault Fail Threshold
General PerformanceApplicationSub Group › Task › Metric10%
Policy & UXApplicationSub Group › Task › Metric5%
SafetyApplicationSub Group › Task › Metric2%
Module PerformanceModuleTask › Metric10%
  • Application group cards show one row per Application version.
  • Module cards show one card per Module version (e.g., RAG BM25, RAG Hybrid).
  • Groups not in the Run have no card or tab.

②Judgment criteria

TermMeaningDefault
CaseThe result of scoring one benchmark data item with one Metric-
Correctness ThresholdThe minimum score for a case to count as correct.
A lower score is Incorrect.		0.5
Failure rateThe share of Incorrect cases among a Metric's scored cases-
Fail ThresholdThe allowed failure rate set for each Metric.
If the failure rate exceeds this value, that Metric fails.		Varies by judgment group
  • The names differ in the Release Gate on the Criteria screen.
    • Set Correctness Threshold as Pass score, a value from 0.0 to 1.0.
    • Set Fail Threshold as Failure rate, a value from 0 to 100 (%).
  • You can see Incorrect cases in Metric Fail Rate Detail, sorted from lowest score.

③Snapshot time

Report values are fixed at the time the Report is generated. Changing settings later doesn't change Reports that already exist.

ValueWhen it's fixed
Application, Task, Metric, Benchmark SetWhen the Run starts
Correctness Threshold, Fail ThresholdWhen Report generation starts
  • Changed criteria apply starting with the next Run's Report.
  • Run Time is when the Run was executed, not when the Report was generated.

  1. 2-1Switch Runs

    1. Click the title Run #142 ⌄.
    2. Click the row of the Run you want to view.

    Runs that are in progress or have no Report are dimmed and can't be selected.

  2. 2-2Check evaluation details

    1. In Run Detail, check the number of Tasks, Metrics, and data items.
    2. Check the evaluated Application and Module versions.
    • Report values are the values at the time the Run and Report were generated.
    • They may differ from the current settings. See Snapshot time.

  3. 2-3Check the judgment criteria

    1. Click the Criteria name next to Criteria Detail.
    2. Check the Fail Threshold for each Metric. Judgment criteria explains the terms.

    Only the Metrics selected in the Run appear.

  4. 2-4Share the Report

    1. Click Share to copy the Report link.

    The Report of the Run you're currently viewing is shared.

3. Check the release verdict

①Judgment order

Judgment is made per Metric. A group card's result combines the judgments of the Metrics in it.

  1. Case judgment: If the score is lower than the Correctness Threshold, the case is Incorrect.
  2. Metric judgment: If the failure rate exceeds the Fail Threshold, the Metric fails.
  3. Group judgment: If all Metrics pass, the card is Pass. If any Metric fails, the card is Fail.

For example, take a Metric with a Fail Threshold of 10%. If 4 of 80 data items are Incorrect, the failure rate is 5% and the Metric passes. If 10 are Incorrect, the failure rate is 12.5% and the Metric fails.

  • If some cases errored, the failure rate is calculated without those cases.
  • Metrics where all cases errored are excluded from judgment.
  • The large % on a card is the failure rate of the whole group. It's a summary for reference, not the basis of the verdict.

  1. 3-1View the verdict

    1. Click the Conclusion Summary tab.
    2. Check Pass and Fail on the group cards for each version. See Judgment groups for the groups.
    • If any Metric fails, the card is Fail.
    • Judgment order explains the calculation with an example.

  2. 3-2Find the cause of a Fail

    1. Click N metrics not met on a Fail card.
    2. Check the Metrics that exceeded the threshold and their failure rates.
    3. Hover over the ⓘ next to the % to see why it was judged that way.

    The % on a card is a summary for reference. It's not the basis of the verdict.

4. Compare versions

①Version comparison

Version comparison compares two versions of the same Application or Module one-to-one. It shows how the two versions' verdicts diverged on the same benchmark data. Use it to decide whether to switch to a new version.

  • The left is the baseline version A, and the right is the comparison version B.
  • Cases are split into four quadrants based on the two versions' verdicts. Quadrant names include the version names.
QuadrantMeaningHow to read it
Only A CorrectCases A got right and B got wrongWhat you lose by switching to B
Only B CorrectCases B got right and A got wrongWhat you gain by switching to B
Both Version CorrectCases both versions got rightNo issue with either version
Both Version IncorrectCases both versions got wrongFailures that remain even if you switch versions
  • Cases where the two versions' verdicts differ are called Conflicting cases.
  • In Metric Comparison, the leading version is the one with the lower failure rate. If the failure rates are equal, it shows Tied.
  • You can't compare three or more versions at once. Keep A fixed and change only B.

  1. 4-1Select versions to compare

    1. Click a group tab, such as General Performance.
    2. On the left, select the baseline version A.
    3. On the right, select version B to compare.
    4. To swap A and B, click Swap versions.
    • A version selected on one side can't be selected on the other.
    • Set your current version as A and the candidate version as B for easier reading.
    • If this is your first time, read Version comparison first.

  2. 4-2Check cases with diverging verdicts

    1. Check the count for each Case in the quadrants on the left.
    2. Click the Case quadrant you want to view.
    3. In Metric Distribution, see which Metrics the failures concentrate in.
    4. On a case card, check both versions' scores and the difference.
    5. Click View all to see all of them in the Filtered Result panel.
    • Check Both Version Incorrect first. These failures remain even if you switch versions.
    • The score difference is A − B. A positive value means A scored higher.

  3. 4-3Compare by Metric

    1. In the card header, see the leading version and the number of cases it won.
    2. Compare Pass and Fail for the two versions in the version rows.
    3. Check the failure rate and the Fail Threshold line on the bar.
    4. In Reference Comparison, see the rank among all versions.

    Version rows are sorted by lowest failure rate, not by A and B order.

5. Analyze failure causes

①Failure analysis areas

In Version Detail at the bottom of a judgment group tab, see where failures concentrate.

AreaGrouping basisTargetWhen it appears
Query ClusterQuestion types formed by grouping similar questionsBoth correct and incorrect questionsWhen an LLM-as-a-judge evaluation was run
Score Reason ClusterReasons formed by grouping similar failure reasonsIncorrect cases onlyWhen the condition above is met and there are Incorrect cases
Metadata Incorrect RateValues of a Metadata KeyIncorrect dataWhen there's a Key that meets the conditions
  • A failure reason is the judgment rationale (Score Reason) a Metric leaves with its score.
  • Both clusters are generated by the Judge Model used most in the Run.
  • Tokens used to generate cluster names and descriptions count toward Trial token usage.
  • Metadata is the additional information columns of a Benchmark Set (e.g., difficulty, channel).

  1. 5-1Failures by question type

    1. In Version Detail, select the version to view.
    2. In Query Cluster, compare counts by question type. See Failure analysis areas.
    3. In Incorrect Rate By Query Cluster, find the types with high failure rates.
    4. Click a box or card to open the Filtered Result panel.
    • Box size is the number of questions, counting both correct and incorrect ones.
    • It appears only when an LLM-as-a-judge evaluation was run.

  2. 5-2Check failure reasons

    1. In Score Reason Cluster, compare failure counts by reason.
    2. In Fail Reason Breakdown, see each Metric's share of each reason.

    Only incorrect cases are grouped by reason.

  3. 5-3Failure rate by Metadata

    1. In Metadata Incorrect Rate, select a Key tab.
    2. Check the failure rate for each value.

    If there are more than 10 distinct values, only the 10 with the most failures appear.

  4. 5-4Failure cases by Metric

    1. Expand a Sub Group.
    2. Click the arrow on the right of a Metric card to expand it.
    3. Click View Detail to see all failure cases.

    Representative failure cases are sorted from lowest score.

  5. 5-5Module analysis

    1. Click the Module Performance tab.
    2. Click Select Module.
    3. Select the Module to view.
    • Changing the Module resets the version selection to the default.
    • Modules have no Sub Group, so Metrics appear directly under each Task.

6. View usage

Check how much cost, tokens, and time (Latency) one Run used. To see cost, set the token path and unit price for each Version. Application API integration explains how.

①See usage

  • Quick view: In the header's Run Detail, see the cost and token count by Judge Model and Version.
  • Detailed view: See the Usage tab.
Usage tab areaWhat it shows
Usage RankingVersions ranked by cost (or tokens, or time)
Distribution by Benchmark RowThe distribution of cost or time per data row. Orange marks rows far above the average.
Usage DetailToken, cost, and time totals and per-row averages by Version
App & Module TotalThe overall total
Judge Model Usage / Judge Model TotalUsage of the Judge Models used for scoring

②When usage doesn't appear

What you seeWhyWhat to do
Pending setupThe unit price for this Version isn't set yet.An Admin sets it with Set up Usage.
Not recordedThe Run was executed before usage tracking existed.Run again to see usage.
Couldn't calculate usageThe token path doesn't match the response.Fix the path in Set up Usage.
  • The Versions above are excluded from the totals. The number excluded appears under App & Module Total as Excludes ….
  • If all Versions are Pending setup, sample bars labeled SAMPLE appear instead of real data.

③Set up usage

You can set it up directly from the Run Report. Only Admins can do this. Other users see Contact admin to set up.

  1. In Run Detail, hover over a Pending setup Version and click Set up Usage
  2. Enter the token paths in Input Tokens and Output Tokens (you can also select them with Pick from response)
  3. Enter the input and output prices per 1M tokens
  4. Click Next if there are more Versions to set up, or Save on the last one

When you save, this Run's usage is recalculated with the new settings. The settings are saved to the Version and used for future Runs too.

For Versions shown as Pending setup ($), token counts are already recorded. You only need to enter the 2 prices.

Frequently asked questions

Why can't I see the Run Report?Because the Run is still in progress, or a Report can't be generated for it.
Cause
A Report is generated only after the Run finishes. It isn't generated if all of the Run's Metrics errored. Runs that finished before the Run Report feature was released don't have one either.
Solution
Wait until the Run finishes. If all Metrics errored, retry the Run. When you retry, the previous Report isn't kept and is replaced by a new one.
What should I do if Report generation fails?Click Retry on the Report Failed row in the Run switcher menu.
Cause
If an error occurs while generating a Report, it shows Report Failed. You can't select this Run.
Solution
Click Retry on the Report Failed row. When generation finishes, it changes to Report Completed.
Why is a card Fail when its % is below the threshold?Because the verdict is made per Metric, not by the card's %.
Cause
The card's % is a reference value summarizing the whole group. If even one Metric exceeds its own Fail Threshold, the card is Fail.
Solution
Expand N metrics not met to see the Metrics that exceeded the threshold. See Judgment order for how verdicts are made.
Why do the Report values differ from the current settings?Because the Report shows the values from when it was generated.
Cause
Application, Task, Metric, and data values are from when the Run started. Judgment criteria are from when Report generation started.
Solution
This is expected. Check the current settings on the Criteria screen. Changed criteria apply starting with the next Run. See Snapshot time.
Why is there no Query Cluster or Score Reason Cluster?Because they're created only when an LLM-as-a-judge evaluation was run.
Cause
They aren't generated if you ran only rule-based evaluations or the Judge Models include no LLM. If there are no incorrect cases, only the Score Reason Cluster is missing.
Solution
Add an LLM-as-a-judge evaluation to the Criteria and run the Run.
Why can't I see the Metadata failure rate area?Because no Metadata Key meets the conditions.
Cause
When incorrect data is grouped by value for each Key, the area appears only if there are 2 or more groups with 5 or more items each.
Solution
This is expected. It appears only when a Key meets the conditions.
Why does a Version's usage show Not recorded?Because the Run was executed before usage tracking existed.
Cause
Usage wasn't stored at that time, so you can't see it even if you set it up later.
Solution
Run again to see usage.
Why does Total cost look lower than it should?Because some Versions are excluded from the total.
Cause
Versions with no unit price set or no records aren't included in the total.
Solution
Check the number excluded in Excludes … under App & Module Total, and have an Admin set them up with Set up Usage.
Why do I see Couldn't calculate usage?Because the token path doesn't match the response.
Cause
The token count couldn't be found in the response using the saved path.
Solution
In Set up Usage, reselect the path with Pick from response and save.
Why can't I see Set up Usage?Because only Admins can set it up.
Cause
Other users see Contact admin to set up. Not recorded Versions can't be set up, so they have no button.
Solution
Ask your Workspace Admin to set it up.
I changed the unit price, and the cost of a past Run changed too. Why?Because saving from the Run Report recalculates that Run.
Cause
This Run's cost was recalculated with the new settings.
Solution
This is expected. The same unit price also applies to future Runs.

Last updated