Simulation

AI agent testing with simulated users: auto-generate scenarios from your policy document, run multi-turn conversations, and score each as Pass or Fail.

DATUMO Simulation evaluates whether an agent follows its policy during a conversation, through multi-turn conversations with a virtual user. You don't need to write test conversations yourself. Evaluation scenarios are generated automatically from your policy document.

In short

Add an Agent Policy (a policy document) to create a Simulation Set, and Scenarios are generated automatically from the policy. Each Scenario runs as a conversation between a virtual user and the agent, and the Judge Model scores it against evaluation items to give a Pass or Fail.

Simulation structure

Diagram

Unit	Role	Contains
Application	The agent to evaluate	Multiple Simulation Sets
Simulation Set	One simulation evaluation unit	1 Agent Policy, 0–1 Tool Schema, multiple Scenarios
Agent Policy	The rules the agent must follow. The source of Scenarios and scoring criteria	-
Tool Schema	Definitions of the tools the agent can use. If added, tool calls are evaluated too	-
Scenario	A test case that defines what the virtual user asks for	User Persona, Task Instructions, evaluation items
Evaluation items	The criteria the Judge Model uses to score the conversation	Must Mention, Expected Behavior, Expected Actions

See an example

Say you want to check, before release, that a credit card company's support agent follows its "credit card cancellation policy."

Stage	Example
Agent Policy	When a customer asks to cancel a card, verify their identity first, and before canceling, explain the annual fee refund and any remaining installments.
Tool Schema	`verify_identity(user_id)`, `cancel_card(card_id)`
Generated Scenarios	S-01 A busy office worker presses for cancellation in the app · S-02 A customer with remaining installments asks to cancel · S-03 A customer tries to cancel a family member's card on their behalf

Example of a single Scenario

S-02 A customer with remaining installments asks to cancel

Component	Content
`User Persona`	A self-employed person in their 40s. Wants to cancel because of the annual fee. Polite, but wants a quick resolution
`Task Instructions`	Ask to cancel the credit card. Don't mention the remaining installments first.
`Must Mention`	The remaining installment amount and how to pay after cancellation
`Expected Behavior`	Verify identity before processing the cancellation. Inform the customer that installments remain.
`Expected Actions`	Call `verify_identity`, then call `cancel_card`

The policy, Scenarios, and tool names in this example are made up for illustration. Actual Scenarios are generated from the policy you add.

Simulation Set: one evaluation unit

A Simulation Set bundles the policy document, generated Scenarios, run settings, and evaluation results into one unit. Each row in the list is one Set, and everything from creation to viewing results happens within the same Set.

When you revise the policy, create a new Set and compare its results with the previous Set.
Run the same Scenarios with a different Application Version to compare versions.

Agent Policy and Tool Schema: the basis of evaluation

Input	Required	What it does
Agent Policy	Required	A planning or policy document that describes the rules the agent must follow. Both Scenarios and scoring criteria come from this document. Upload a `.md` file or enter it directly.
Tool Schema	Optional	Definitions of the tools (functions) the agent can call. If you add it, `Expected Actions` are generated and tool call accuracy is evaluated too. If you don't, only the conversation content is evaluated.

The more specific your policy document, the more specific the Scenarios and scoring criteria. Write it so the order and conditions are clear, such as "Cancel after verifying identity."

Scenario: the conversation to evaluate

A Scenario is a test case that defines who the virtual user is and what they ask for in what situation. IDs use the format S-01, S-02, and so on.

Component	Description
`User Persona`	The virtual user's profile, emotions, and tone
`Task Instructions`	The goal to achieve in the conversation and behavior guidelines
Evaluation items	What the agent must do in the conversation. See the table below

Evaluation items

Evaluation item	What it checks	When it's generated
`Must Mention`	Whether the agent stated the values it must mention	Always
`Expected Behavior`	Whether the agent behaved as the policy requires	Always
`Expected Actions`	Whether the agent called the required tools with the correct arguments	When you add a Tool Schema

If a Scenario fails even one evaluation item, it's a Fail.
Generated Scenarios are read-only. To change them, revise the policy and create a new Set.

Simulation end-to-end flow

A simulation runs in this order: policy input → Scenario generation → conversation → scoring → analysis.

Diagram

Add an Agent Policy and Tool Schema to create a Set, and Scenarios are generated. The list status changes from Scenario Generating to Ready.
Review the generated Scenarios, then select an Application Version, User LLM, and Judge Model and run.
The User LLM plays the virtual user and converses with the agent. Conversations build up in real time.
The Judge Model scores each Scenario as soon as its conversation ends. The Scenario status changes from Waiting → Running → Evaluating → Pass / Fail.
In Results, check the Pass Rate and the result of each Scenario. If you added a Tool Schema, upload a Trajectory to evaluate tool calls too.

Model	Role
`User LLM`	Plays the virtual user in the Scenario.
`Judge Model`	Scores the conversation on `Must Mention` and `Expected Behavior`.

Select both models from the models registered in G-4. Register Gen & Judge models.

Fail and Error are different. A Fail means the conversation ended normally but didn't pass the evaluation criteria, and it is itself an evaluation result. An Error means the simulation itself failed due to a tool call failure or an API error. Errors must be rerun, and you can't move on to the next step while any remain.

Next steps

Document	What it covers
S-1. Create a Simulation Set	View the Set list, enter the policy document, generate Scenarios
S-2. Review & run scenarios	Review Scenarios, select models, run the simulation
S-3. View results	Results dashboard, view conversations, Trajectory evaluation