ChangeLog

Lists the new features added to DATUMO Platform in each version.

v1.6.0

2026.10.07

Multi-turn API registration: Register conversational APIs that exchange multiple turns in Application Setting.
Token usage and cost: Set token prices in Application Setting to see cost, token usage, and latency in the Run Report Usage tab and in Agent Simulation.
Unified Agent Simulation results screen: See conversation history, evaluation results, and usage on a single screen in the Result tab.
Google sign-in: Sign up and log in directly with your Google account.

2026.09.23

Agent Simulation (Simulation Studio): Upload your agent's planning and policy documents to automatically create evaluation scenarios, run conversations with virtual users, and score whether the agent followed its policies. Upload a Tool Schema and Trajectory as well to evaluate tool calls too.

Agent Simulation is available to some accounts first. We'll announce the general availability date separately.

2026.08.26

Run Report version comparison: Compare two versions of the same module side by side in Run Report. See on one screen what each version does better or worse.

2026.08.12

Observability: Connect the logs of your live service (AWS CloudWatch) to automatically score collected conversations with the Metrics you set and gather problematic turns as Fail Cases. Beyond pre-release evaluation, you can now monitor your live service continuously.

2026.08.12

Run Report: A Report is generated automatically when a Run finishes. Without building any charts yourself, you can immediately see whether you can ship (Pass / Fail) based on Datumo's judgment criteria. Set the judgment criteria in Release Gate in Create & manage Criteria.

2026.08.03

More red teaming algorithms and strategies: Run attacks with a wider range of red teaming algorithms and strategy groups beyond STAR-Teaming.
Custom seed datasets: Build your own dataset, use it as seeds, and run red teaming based on it.
Redesigned vulnerability analysis dashboard: We rebuilt the dashboard to make it easier to analyze vulnerabilities in red teaming results.

2026.07.01

2026.06.15

Custom Benchmark Set generation: Generate evaluation data with the AI Tool. Also added 45 data generation templates.

2026.05.29

Workspace- and Application-level evaluation: Set up and run evaluations at the Workspace and Application level.
Criteria-based batch evaluation: Group multiple evaluation metrics into Criteria and evaluate them all at once.
Custom dashboards: Build your own evaluation results dashboard with the layout you want.

2026.02.27

Self sign-up and automatic Workspace creation: Sign up directly with your email, and a Workspace is created automatically once you finish. Password recovery and reset are also supported.
Sample Workspace: After you sign up, explore a Workspace prefilled with sample data and evaluation results right away.
Trial accounts: Trial accounts can view and download results. If you need to generate data or run evaluations, reach out through Contact us on the screen.

2026.02.02

Vulnerability analysis dashboard: Visualize attack success rate (ASR) and scores by taxonomy and strategy group, and see the query and response of each attack attempt and the point at which an attack succeeded.
Improved strategy-based automated red teaming (STAR-Teaming): Strengthened strategy group descriptions and classification, and improved the scoring method. Set a maximum number of iterations (Max Loop, up to 20), and manage Attack Sets per Task.
Safety benchmark dataset: We provide our own Safety dataset of 10,908 items, all reviewed by humans.