> For the complete documentation index, see [llms.txt](https://docs.datumo.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.datumo.com/documentation/documentation-en/readme.md). # Overview ## Overview > LLM-based services continue to change in quality even after deployment. > > Datumo Platform provides three tools to track and manage that change quantitatively. Evaluation measures performance, Redteaming uncovers vulnerabilities, and Observability detects anomalies during operation. *** ### What is Datumo Platform? Datumo Platform consists of three modules: Evaluation, Redteaming, and Observability. Each module operates independently, and issues found in Redteaming and Observability are reflected in Evaluation's evaluation set.

Datumo Platform is designed so that the three modules form a single feedback loop. Vulnerabilities found in Redteaming and anomaly patterns detected in Observability are automatically added to Evaluation's evaluation set. This allows the evaluation set to be continuously updated to reflect real operational data and adversarial scenarios. *** ### Introduction by module #### 1. Evaluation Evaluation is a module that measures the performance of an AI service against predefined criteria and quantitatively verifies whether it has improved.

Step	Content
Define	Establish evaluation criteria by designing domain metrics such as accuracy, tone, and safety.
Measure	Identify strengths and weaknesses in performance by measuring accuracy by category.
Analyze	Identify improvement points by clustering not only scores but also the causes of failures.
Improve	Quantitatively confirm performance changes by comparing before and after improvements under the same criteria.

> Evaluation supports over 100 evaluation templates, LLM-as-a-Judge automated scoring, Score Reason Clustering, and more. For details, see the [Evaluation](https://www.notion.so/evaluation/core-concept.md) section. *** #### 2. Redteaming Redteaming is a module that uncovers vulnerabilities in AI services against adversarial inputs and reflects discovered issues in the Evaluation evaluation set.

Step	Content
Categori ze	Define domain-specific threat categories such as harmfulness, bias, jailbreaks, and sensitive information.
Generate	Automatically generate adversarial prompts at scale, such as bypasses, transformations, and role-play.
Attack	Explore the points where the model fails by gradually increasing the intensity.
Feedback	Discovered vulnerabilities are automatically added to the Evaluation evaluation set and used for regression testing.

> Redteaming does not verify predefined scenarios; it works by automatically generating adversarial prompts to find the points where the model fails. For details, see the[ Redteaming](https://www.notion.so/redteaming/core-concept.md) section. *** #### 3. Observability Observability is a module that collects and analyzes user conversations during operation to detect anomalies and reflects discovered issues in the Evaluation evaluation set.

Step	Conten
Collect	Collects conversations, traces, and metadata in operation through the SDK.
Cluster	Automatically classifies collected conversations by topic, intent, and failure type.
Detect	Automatically detects unusual patterns such as changes in failure rate and emergence of new topics.
Loop Back	Detected issues are added to the Evaluation evaluation set and reflected in the next evaluation cycle.

> Pre-launch evaluation only covers predefined scenarios. Observability plays the role of feeding issues discovered in real operational data back into the evaluation framework. For details, see the[ Observability](https://www.notion.so/observability/dashboard.md) section. *** ### When using the three modules together Each module can be used independently, but when integrated, the following differences arise.

Category	When used separately	When integrated
Improvement verification	Relies on subjective judgment	Quantitatively verified with before-after using the same criteria
New risks	Only recognized after customer complaints come in	Preemptively uncovered by Redteaming and Observability
Evaluation framework	No changes after initial setup	Continuously updated as new issues are reflected
Operations-model team collaboration	Feedback is disconnected because data and criteria are separated	Collaboration based on the same data and criteria

*** ### Before you get started If you're using Datumo Platform for the first time, we recommend reading the documentation in the following order. 1. [**Get Started**](https://www.notion.so/get-start/core-concept.md) — Check the platform's core concepts, such as Workspace, User, Model, and Application, and how to set them up initially. 2. [**Evaluation**](https://www.notion.so/evaluation/core-concept.md) — Set evaluation criteria and run your first evaluation. 3. [**Redteaming**](https://www.notion.so/redteaming/core-concept.md) — After performance improvement is confirmed with Evaluation, further uncover vulnerabilities with adversarial inputs. 4. [**Observability**](https://www.notion.so/observability/dashboard.md) — Once the service starts receiving real traffic, reflect operational data in the evaluation set. > Regardless of which module you use, we recommend checking the Evaluation section first. *** ### Need help? * 📧 Contact: * 🌐 Website: [datumo.com](https://datumo.com/) * 🎬 Request a demo: [Request a demo](https://datumo.com/)