Observability
Find issues in real user conversations with your AI agent in production using LLM observability: set up integrations and review fail cases.
DATUMO Observability connects the real user logs of your AI service in production and continuously checks whether the service responds safely and in line with your policies. If Evaluation is a test run on data you prepared before launch, Observability is closer to reviewing actual conversations after launch.
In short
When you connect logs through an Integration, they are organized into Sessions and Turns. Turns selected at the Sampling Probability rate are evaluated by Metrics, and when a Rubric in a Metric is judged a violation, it becomes a Fail Case that you review in the list and in alert emails.
Observability structure
| Unit | Role | Contains |
|---|---|---|
| Integration | Settings that connect a log store and turn logs into Turns | Connection details, parser code |
| Session | One bundle from when a user starts a conversation to when it ends | Multiple Segments and Turns |
| Turn | One pair from User Input to App Output. The basic unit of evaluation | Multiple Spans |
| Metric | The unit that evaluates a Turn | 1 Judge Model, 1–10 Rubrics |
| Rubric | A check item judged separately within a Metric | - |
| Fail Case | A Rubric judged as a violation | - |
How it differs from Evaluation
| Category | Evaluation | Observability |
|---|---|---|
| Evaluation data | A Benchmark Set prepared in advance | Real user logs from production |
| When evaluation runs | When you execute a Run | Continuously, as logs come in |
| Evaluation unit | One data item | One Turn |
| Result | Score | Whether there's a violation (Fail Case) |
See an example
Say you run a financial advisory chatbot and want to watch for violations of investment solicitation rules.
Example of one session being evaluated
| Turn | User Input | App Output | Result |
|---|---|---|---|
| 1 | What's the interest rate on installment savings? | The current 12-month installment savings rate is 3.5% per year. | Pass |
| 2 | Then aren't stocks a better deal? | If you buy now, you're guaranteed a profit. Buy right away. | Fail Case: Advisory Policy: Investment Solicitation |
| 3 | Thank you. | I hope that helped. | Pass |
Turn 2 produces a Fail Case, so alert recipients get an email titled [DATUMO Fail Alert - Turn ID] - Advisory Policy. In the Fail Case list, you can view the full conversation of this Session.
The Metric names and conversation in this example are for illustration only.
Integration: connect logs
An Integration is the setting that brings your AI service's logs into DATUMO. Currently, it connects AWS CloudWatch logs.
- Enter the connection details (Region, Log Path, etc.) and verify the connection.
- Parser code finds the user questions and AI answers in the raw logs and groups them into Turns. Log formats differ by service, so you write the parser code yourself.
- After you connect, the Integration card shows the connection time, last log time, and total number of Turns.
- If your log format changes, switch to a new Integration. This increments the version.
- If you no longer need an Integration, disconnect it. Once disconnected, no new logs are collected.
- A disconnected Integration can't be turned back on. To use it again, create a new Integration.
- Evaluation results from before the disconnect, the sampling probability, and Metric settings stay as they are.
Log hierarchy: Session · Segment · Turn · Span
Connected logs are organized into the hierarchy below. Screens and evaluations work based on these units.
| Term | Plain English | Definition | Where it's used |
|---|---|---|---|
| Session | session | One bundle from the start of a conversation to its end | Card unit in the Fail Case list |
| Segment | segment | A part of a session split by topic | - |
| Turn | turn | One pair from User Input to App Output | Basic unit of evaluation, row in the Fail Case list |
| Span | span | Intermediate agent results produced on the way to a turn (Retriever, Router results, etc.) | - |
| User Input | user input | The question the user entered in a turn | Evaluation prompt variable user_input |
| App Output | app output | The final answer the AI service returned in a turn | Evaluation prompt variable app_output |
Sampling and Metrics: what to evaluate and how much
Sampling Probability is the percentage (0–100) of connected Turns to evaluate. Evaluating every Turn greatly increases cost and time, so you set a rate. A changed value applies only to Turns collected from then on.
A Metric is the unit that evaluates Turns, and each Metric consists of multiple Rubrics.
| Item | Description |
|---|---|
Judge Model | The model that judges the Metric. Choose from the models registered in G-4. Register Gen & Judge models. |
| Rubric | A check item. 1–10 per Metric. The Judge Model judges whether each Rubric is violated. |
Prompt Instruction | The judgment criteria for the Rubric |
- The Safety Metric is included by default in every Application (
DATUMO Default). You can change only its Judge Model, and you can't delete it. - When you turn off a Metric, Turns collected after that aren't evaluated. Past results remain.
How Metric, Rubric, and Fail Case relate
Each Rubric judged as a violation is one Fail Case. A single Turn can fail multiple Metrics, and a single Metric can produce multiple Fail Cases.
Fail Case: turns with a violation
The Fail Case list shows only Turns with at least one Fail Case. Turns from the same Session are grouped into one card, and each shows what was violated in the Metric name: Rubric name format.
- Open a row to view the full conversation of the Session that Turn belongs to.
- Set alert recipients for each Metric to receive an email when a Fail Case occurs.
Observability end-to-end flow
Production logs are processed in this order: collection → conversion → sampling → evaluation → Fail Case.
- Logs are continuously pulled from the log path connected through the Integration.
- Parser code converts the logs into Sessions and Turns.
- Turns are selected at the Sampling Probability rate as evaluation targets.
- For each enabled Metric, the Judge Model judges whether each Rubric is violated.
- Rubrics judged as violations become Fail Cases, which accumulate in the list and trigger alert emails.
If there's no Gen & Judge model, Metrics go into the Evaluating Paused state and evaluation stops. To resume evaluation, register a model and select the Metric's Judge Model again.
Next steps
| Document | What it covers |
|---|---|
| O-1. Connect an Integration | Connecting a log store, running parser code, managing and disconnecting Integrations |
| O-2. Evaluation settings | Sampling Probability, Metrics and Rubrics, alert recipients |
| O-3. Fail Case | Viewing the list of violating turns and Session conversations |