Register a Gen & Judge model

Shows how to register LLM and Embedding models for automation features at the Workspace or Organization level, activate and deactivate them, and assign replacement models.

1. Model list

①Model types

Gen & Judge models are the models DATUMO Platform uses for automation features such as data generation, LLM-as-a-Judge scoring, and simulation.

Model type	Used for
LLM	Data generation and LLM-as-a-Judge evaluation. LLM-as-a-Judge is an evaluation method in which an LLM scores the answers.
Decision Model	Decision Model Metrics. It doesn't generate text; it picks one of the options you define. Davi and Jev (Typesafe) are provided as DATUMO models.
Embedding	Embedding-based evaluation metrics.

②Model scope

Scope	Shown as	Managed in	Available in
Workspace model	`Workspace Model`	Settings > Workspace > `Workspace Gen & Judge Model`	That Workspace only
Organization model	`Organization Model`	Settings > Organization > `Organization Gen & Judge Model`	All Workspaces in the Organization
DATUMO model	`DATUMO Model`	Provided and managed by DATUMO	All Workspaces

Owners, Admins, and Workspace Admins manage Workspace models. The Owner and Admins manage Organization models.

1-1Open the list
1. Open Settings > Workspace > Workspace Gen & Judge Model.
1-2Groups by scope
1. Workspace Model is used only in this Workspace.
2. Organization Model is an Organization-wide model. You can't change it here.
3. DATUMO Model is managed by DATUMO.
1-3Model info
1. Each row shows the name, API Key, type, API URL, primary model, and last modified date.
2. For a Preset, click + n models to see all the models it includes.
Preset names are generated automatically as {Provider} ({last 4 characters of key}).

2. Add a new model

①Connection methods

Method	Description
Preset	For supported providers, just enter an API Key to connect. All of the provider's LLM models are added at once. Anthropic, Baseten, Cerebras, Cohere, Fireworks, Gemini, Groq, Lepton, Mistral, OpenAI, OpenRouter, Perplexity, Replicate, Together.ai, xAI
Custom Provider	Connect an OpenAI-compatible endpoint directly, such as vLLM, Ollama, or RunPod. LLMs support the Chat Completions API format, and Embedding models support the Embedding API format.

2-1Add with a Preset
1. Click + New Model.
2. Click a provider under Preset.
3. Enter the API Key.
4. Click Save.
If you don't have a key, follow the link to the provider's console to get one.
2-2Add a Custom Model
1. Click Custom Model under Custom Provider.
2. Choose LLM or Embedding in Model Type.
3. Enter the Model Name, API Request Model, API URL, and API Key.
4. Click Save.
API URL must start with https://.

3. Activate and deactivate

①Replacement models

You can't select an inactive model in evaluations or automation features. If the model is in use somewhere, choose a replacement model when you deactivate it.

Usages are grouped by Evaluation (Metric), Simulation, Red Teaming (Model Setting), Eval Lab, and Observability.
The replacement model applies only to Runs you start after the change. Runs you've already run and their results don't change.
When you reactivate the model, the usages that were left empty when you deactivated it are filled with this model again. Usages assigned a different model in the meantime stay as they are.
The same dialog opens when you delete a model. If the model is in use, delete it with Replace {n} & Delete.

3-1Deactivate (not in use)
1. Click a switch that's on.
2. If the model isn't in use, click Deactivate right away.
3-2Deactivate (in use)
1. Check how many places use it.
2. Choose one replacement model for all of them in Select Replacement Model.
3. To choose a different one for each usage, change it in the list below.
4. Click Replace {n} & Deactivate.
The replacement model applies only to future runs.
3-3Reactivate
1. Click a switch that's off.
2. In the confirmation dialog, click Activate.

4. Register an Organization model

The Owner and Admins can register models shared across all Workspaces in the Organization. Activation changes made here apply to all Workspaces immediately.

4-1Open the screen
1. Open Settings > Organization > Organization Gen & Judge Model.
4-2Add a model
1. Add one with + New Model. The steps are the same as in 2. Add a new model.
4-3Manage activation
1. Turn models on and off with the switch. When you deactivate one, choose a replacement model for each Workspace that uses it.
You can't change the DATUMO Model group at the bottom.

Frequently asked questions

Why can't I click the switch on a Organization Model?Because Organization models are managed only by the Owner and Admins on the shared model management screen.

Cause: The Owner and Admins manage them, only from the Organization Gen & Judge Model screen.
Solution: Ask the Owner or an Admin.

Why can't I click Save?Because a required field is empty, or API URL doesn't start with https://.

Cause: You can save only after you fill in all four fields.
Solution: Check the input fields.

Why can't I select a model in evaluations or automation features?Because the model is inactive.

Cause: Inactive models aren't available for use.
Solution: Turn on the switch on the model's row.

I deactivated a model. Did the model in my past Runs change too?No. Past Runs stay as they were.

Cause: The replacement model applies only to Runs you start after the change.
Solution: If you run it again, it runs with the replacement model.