Review & run scenarios

Review the generated scenarios, select models to run the simulation, and track its progress.

1. Review scenarios

①Scenario

A Scenario is a test case that defines a conversation with a simulated user. IDs follow the S-01, S-02 format.

Card	Contents
`User Persona`	The simulated user's profile, emotion (`Emotion`), urgency (`Urgency`), and tone (`Tone`)
`Task Instructions`	The goal the simulated user pursues in the conversation and how they should behave
`Expected Actions`	The tools and arguments the agent must call
`Must Mention`	Values the agent must state in its responses
`Expected Behavior`	Behavior the agent must follow under the policy
`View JSON`	The scenario's raw JSON

Expected Actions appears only in Sets created with a Tool Schema.
You can't edit generated scenarios. To change them, update the policy document and create a new Set.

②Generation usage

Total Generation Usage on the left is the total usage spent generating scenarios. Next to the title on the right, you see the usage for the selected scenario.

Item	Meaning
`Tokens`	Number of tokens used
`Cost`	Cost incurred
`Latency`	Time taken to generate

You can see the same values in the Scenario tab of the results screen.

1-1Select a scenario
1. In the step indicator at the top, confirm that Scenario is Done.
2. Click a scenario in the list on the left.
1-2Review the scenario
1. Check the simulated user in User Persona.
2. Check the conversation goal in Task Instructions.
3. Check the tools to be called in Expected Actions.
4. Review Must Mention and Expected Behavior.
5. Expand View JSON if you need the raw data.
See Scenario for what each card contains.
1-3Review the policy document
1. Click View Detail.
2. Review the Policy and Tool Schema you entered.
1-4Edit Set information
1. Click the pencil icon next to the title.
2. Edit the name and description, then click Save.
1-5Delete a Set
1. Click Delete at the top right.
2. In the confirmation dialog, click Delete.
- This deletes all scenarios, conversations, and evaluation results.
- A deleted Set can't be restored.

2. Run the simulation

①Run models

When you start a simulation, you select three models.

Field	Role
`Application Version`	The agent to evaluate
`User LLM`	The model that plays the scenario's simulated user
`Judge Model`	The model that scores the conversation against `Must Mention` and `Expected Behavior`

Choose the Application Version from the Versions registered in your Application.
Choose the User LLM and Judge Model from the models registered in Gen & Judge.

You can't delete a Set while it's running. Wait until the run finishes.

2-1Open the run dialog
1. Click Run Simulation at the top right.
2-2Select models
1. In Application Version, select the agent to evaluate.
2. Select a User LLM.
3. Select a Judge Model.
See Run models for the role of each model.
2-3Start the run
1. Click Start Simulation.
2. The step indicator changes to Simulation Running.
If no models are registered, Start Simulation is disabled.

3. Track progress

①Scenario status

Each scenario runs its conversation and scoring separately. Scoring starts as soon as a scenario's conversation ends.

Status	Meaning
`Waiting`	Waiting for earlier scenarios to finish
`Running`	The simulated user and the agent are talking
`Evaluating`	The conversation has ended and is being scored
`Pass`	Passed every evaluation item
`Fail`	Failed one or more evaluation items
`Error`	An error occurred during the conversation or scoring

Turn N in the list is the number of conversation turns so far.
Evaluation results and Usage appear after the conversation ends.
When all scenarios finish, you move to the results screen.

②Fail vs. Error

Type	Meaning	What to do
`Fail`	The conversation finished but didn't meet the evaluation criteria	Check the cause on the results screen.
`Error`	The conversation or scoring couldn't finish	Rerun with `Retry All Errors`.

Errors during the conversation rerun from the conversation.
Errors during scoring rerun only the scoring.
You don't move to the results screen while any Error remains.

3-1Check overall progress
1. Check the overall Set status in Status.
2. Check each scenario's status and turn count in the list.
See Scenario status for what each status means.
3-2View the conversation
1. Watch the conversation in real time in the Result tab.
2. When scoring finishes, the cards on the right show Pass or Fail for each item.
3. Expected Actions isn't evaluated at this stage.
- Upload a Trajectory to evaluate Expected Actions.
- View results explains how.
3-3Rerun errors
1. Find scenarios with Error status.
2. Click Retry All Errors.
Conversation errors rerun from the conversation; scoring errors rerun only the scoring.

Frequently asked questions

Why can't I click Start Simulation?Because no Gen & Judge models are registered.

Cause: You can choose the User LLM and Judge Model only from the models registered in Gen & Judge.
Solution: Complete Register Gen & Judge models, then run it again.

Why don't I see the Expected Actions card?Because the Set was created without a Tool Schema.

Cause: Tool call criteria come from the Tool Schema.
Solution: To evaluate tool calls too, create a new Set with a Tool Schema.

Can I edit a scenario?No. You can't edit generated scenarios.

Cause: Scenarios are generated from the policy document. The screen is read-only.
Solution: Update the policy document and create a new Set. Adding a version to the name makes them easier to compare.

When can I delete a Set that's running?Delete it after the simulation finishes. Delete is off while it's running.

Cause: Delete is disabled during a run.
Solution: Delete the Set after the run finishes.

Why does Error keep showing up?Because some scenarios couldn't finish the conversation or scoring.

Cause: You don't move to the results screen while any Error remains.
Solution: Click Retry All Errors to rerun them. See Fail vs. Error for the difference.

How do I see the total cost of generating and testing a simulation?Check Cost in the Usage card on the results screen.

Cause: The Usage total includes Scenario Generation, LLM User, AI Agent, and Judge Model usage. If no AI Agent unit price is set, AI Agent cost is left out of the total, and scenarios still being scored or with scoring errors don't include Judge Model usage.
Solution: To see only the scenario generation cost, check Total Generation Usage on the left of this screen. For the total cost including conversations and scoring, check Cost in the Usage card on the results screen. To include AI Agent cost, have an Admin set unit prices in Set up Usage. For details, see Generation usage and View results.