Quickstart
Walks you through the shortest path from signing up to seeing your first evaluation results, in six steps.
This is the shortest path from signing up to seeing your first evaluation results. Each step links to a document with more detail. If you want the concepts first, read Get Start Core Concept and Evaluation Core Concept.
In short
Sign up and create an Organization and a Workspace → register a model for scoring → create the Application you want to evaluate and connect its API → set up evaluation criteria (Criteria) and run them → view the results as scores.
1. Sign up and create an Organization
When you sign up with a work email, your first Organization and first Workspace are created together. The person who signs up becomes the Owner.
- Invite teammates during sign-up, or later from Settings > Organization >
User Management. - If your team already uses an Organization, don't sign up. Join through the link in your invitation email instead.
Sign up & log in · Manage members
2. Register a Gen & Judge model
Evaluations often have an LLM score the answers (LLM-as-a-Judge). Register the model you'll use for scoring first.
- In Settings > Workspace >
Workspace Gen & Judge Model, click+ New Model. - For supported providers such as OpenAI and Anthropic, you only need to enter an API Key.
3. Create an Application and connect its API
Register the AI service you want to evaluate as an Application, and connect its API so DATUMO can call it.
- In the Application selector in the left menu, click
+ New Applicationand enter a name, type, and language. - In
Application Setting> Agent Connection, clickAPI Integrationon a Version, enter the API URL and request body, and pick the value in the response to evaluate (Output Mapping).
Create an Application · Integrate an Application API
4. Set up Criteria and Tasks
Criteria is a bundle of evaluation criteria, and each Task inside it is the unit that actually runs an evaluation. When you create an Application, Tasks that fit its type are recommended automatically from templates.
- Check the recommended Tasks in the
Criteriamenu. Template Tasks include evaluation data (Benchmark Set) and a scoring method (Metric), so you can run them right away. - To add questions tailored to your service, upload data to a Benchmark Set or generate it with AI.
Create & manage Criteria · Manage Tasks · Manage Benchmark Sets
5. Run an evaluation
On the Criteria screen, click Run Evaluation in the top right, choose the Tasks and Version to evaluate, and run it. Check the run status in the Runs menu.
6. View results
When a Run finishes, see the score for each Task and the scoring rationale for each data item. After you change a prompt or model, run it again as a new Version to compare before and after against the same criteria.
Eval Result: Scores per Task and results per data itemDashboard: Compare Versions and Runs in chartsRun Report: A verdict on whether you can ship (Pass / Fail)
Eval History · Dashboard · Eval Report
What to try next
| What you want to do | Feature | Document |
|---|---|---|
| Check through long conversations with virtual users whether your agent follows its policies | Simulation | Simulation Core Concept |
| Find out which attacks your service is vulnerable to with automated attacks | Red Teaming | Red Teaming Core Concept |
| Keep watching real conversations in your live service | Observability | Observability Core Concept |
| Red team events where people attack the service by hand | Evaluation Lab | Evaluation Lab Core Concept |