Create & manage Criteria

Learn how to create and manage Criteria and Subgroups, and how to set up Release Gate, the judgment criteria for deployment.

1. Turn on Edit Mode

①Edit Mode

Edit Mode is the state in which you can change your Criteria setup. Turn it on and off with the Edit Mode switch in the top-right corner of the Criteria screen.

State	What you can do
Off	View the Criteria setup, set up Release Gate
On	Use `+ New Subgroup`, `+ New Task`, the Subgroup menu, and the Task menu

You can open Release Gate settings whether Edit Mode is on or off.

1-1Turn on Edit Mode
1. Turn on Edit Mode in the top-right corner.
You can open Release Gate settings even when it's off.

2. Manage Criteria

①Criteria

Criteria is a set of standards for evaluating one Application. When you run an evaluation, every Task in the Criteria is evaluated at once.

You can create multiple Criteria for one Application.
Split them by purpose (e.g., Pre-launch check for v1.0, Routine check).
By default, a new Criteria is based on the Application's Initial Criteria Settings. Recommended Tasks are added with it.
Recommended Tasks depend on your Application and Module settings.
Select Start from an empty to create an empty Criteria without recommended Tasks.

Evaluation Core Concept explains how Criteria, Subgroups, and Tasks relate.

2-1Create Criteria
1. Click + next to the tabs.
2. Enter a Criteria Name.
3. To start with an empty Criteria, select Start from an empty.
4. Click Complete.
- If you don't select it, the Criteria is created from Initial Criteria Settings.
- See Criteria for the difference between the two.
2-2Rename Criteria
1. Click the edit icon next to the name.
2. Enter a new name and click Save.
2-3Copy Criteria
1. Click the copy icon in the top-right corner.
2. Click Complete.
3. Find the copy in the tabs at the top.
You can copy only within the same Application.

3. Manage Subgroups

①Subgroups and Tasks

The Criteria screen is divided into four evaluation areas. Each area groups Tasks differently.

Evaluation area	What it checks	Tasks grouped by
General Performance	Knowledge and abilities the service should have by default	Subgroup
Policy & UX	Whether it follows internal policies and fits the user experience	Subgroup
Safety	Whether it avoids harmful responses	Subgroup
Module Performance	Whether each Module does its job well	Module

A Subgroup groups Tasks by topic within one area. You create and name Subgroups yourself.
A Task is the unit that actually runs the evaluation. You add Tasks under a Subgroup.
Module Performance has no Subgroups. Tasks are grouped by each Module set up in the Application. Add Modules in Application Setting.

For example, a financial advice chatbot could be set up like this:

Safety                      (evaluation area)
└─ Handling harmful questions (Subgroup)
   └─ Refuse harmful requests (Task)

Deleting a Subgroup also deletes every Task in it.

3-1Create a Subgroup
1. Click + New Subgroup. If this is your first time, read Subgroups and Tasks first.
2. Enter a name and click Complete.
3. Find the new Subgroup in the list.
3-2Rename a Subgroup
1. Click the arrow next to the Subgroup name.
2. Select Edit Subgroup.
3. Change the name and click Complete.
3-3Delete a Subgroup
1. Select Delete Subgroup from the menu.
2. Click Delete in the confirmation dialog.
Every Task in the Subgroup is also deleted.

4. Set up Release Gate

①Release Gate

Release Gate is the standard for judging whether you can deploy. Run Report uses it to give a Pass or Fail verdict.

Release Gate belongs to a Criteria. It's created along with the Criteria.
When you copy a Criteria, its Release Gate is copied too.
If you don't set it up, the default values are used for the verdict.

The summary bar at the top of the Criteria screen shows each group's allowed failure rate as ≤ N%. The default allowed failure rate for each group is:

Group	Default allowed failure rate
General Performance	10%
Module Performance	10%
Policy & UX	5%
Safety	2%

The criteria are locked when a Run Report starts. Changing them doesn't affect the verdicts of Reports you've already created. Changes apply from the next Run.

②Pass score and Failure rate

Release Gate is defined by two values per Metric.

Field	Meaning	Allowed range	Default
`Pass score`	Minimum score for a single data item to pass	0.0 – 1.0	0.5
`Failure rate`	Failure ratio (%) allowed for the Metric	0 – 100	The group's default allowed failure rate

You can enter both values with up to two decimal places.
You can't enter values outside the range.

For example, say Pass score is 0.5 and Failure rate is 10. If more than 10 of 100 data items score below 0.5, the Metric doesn't meet the criteria.

③How verdicts work

Release Gate judges by Metric, not by group.

If a data item's score is below Pass score, that item fails.
If a Metric's failure ratio exceeds Failure rate, that Metric is not met.
If any Metric in a group is not met, the group fails.

The % in the summary bar summarizes the whole group. It isn't the basis of the verdict. For example, a group can fail even if its failure rate is 5%, within its 7% threshold. This happens when one Metric in the group exceeds its own 10% threshold.

Failures and errors are counted differently.

Type	Meaning	In the verdict
Failure	The scoring result is below the threshold	Counted in the failure ratio
Error	The item couldn't be evaluated, so it has no score	Excluded from the failure ratio

If every data item in a Metric is an error, that Metric is excluded from the verdict.

4-1Check the summary bar
1. Check each group's allowed failure rate in the summary bar.
- The % in the summary bar is a summary value. The actual verdict is made per Metric.
- Read Release Gate first.
4-2Enter pass criteria
1. Click Setting on the right side of the summary bar.
2. Click the triangle before a group name to expand the group.
3. Enter Pass score and Failure rate for each Metric. See Pass score and Failure rate for what they mean.
- Pass score: 0.0 – 1.0, default 0.5
- Failure rate: 0 – 100
- Default allowed failure rate: General and Module Performance 10%, Policy & UX 5%, Safety 2%
4-3Apply to a whole group
1. Enter a value in Set all failure rates in the group header.
2. Click Apply to apply it to every Metric in the group.
4-4Save settings
1. Click Save.
Closing with X doesn't save your changes.
4-5Check how verdicts work
1. Verdicts are made per Metric.
- A case fails if its score is below Pass score.
- A Metric is not met if its failure rate exceeds Failure rate.
- If any Metric is not met, the group fails.
- See How verdicts work for an example.

Frequently asked questions

Why can't I edit Criteria or Subgroups?Because Edit Mode is off.

Cause: When Edit Mode is off, editing features such as + New Subgroup and the Subgroup menu are hidden.
Solution: Turn on Edit Mode in the top-right corner. Edit Mode lists what you can do once it's on.

Why can't I enter a value in Pass score or Failure rate?Because the value is outside the allowed range.

Cause: Pass score accepts only values from 0.0 to 1.0, and Failure rate only values from 0 to 100.
Solution: Enter a value within the range, with up to two decimal places.

Why did the value I entered in Set all failure rates disappear?Because the value in the group header isn't saved.

Cause: The header value only applies a value to every Metric in the group at once. When you reopen the window, it shows the default.
Solution: Check the applied values in each Metric row. Each Metric row keeps the last saved value.

Why weren't my Release Gate changes saved?Because you closed the window without clicking Save.

Cause: Closing the window with X discards all changes.
Solution: After changing values, click Save.

Why is the verdict Fail when the summary bar is within the threshold?Because verdicts are made per Metric, not per group.

Cause: The % in the summary bar summarizes the whole group. If any Metric in the group exceeds its own threshold, the group fails.
Solution: Check the Failure rate for each Metric in Setting. See How verdicts work for the order of judgment.