Platform Docs
Try DatumoEval

Get Start

Get started with DATUMO Platform: sign up, set up your workspace, register Gen and Judge models, and connect your AI application by API.

Diagram

Organization

The top-level unit of the platform. One company or team uses one Organization.

  • An Organization can have multiple Workspaces.
  • One account can belong to multiple Organizations. Switch between them with the Organization Switcher, which opens when you click the Organization name at the top of the LNB.
  • Each account can create only one Organization of its own, as its Owner. You join other Organizations by invitation.

  1. 1-1Open the Switcher

    1. Click the Organization name at the top of the LNB.
    2. The Organization Switcher opens. Pick an Organization on the left and a Workspace on the right to move there. See Organization & Workspace for details.

Workspace

An independent workspace. You usually split Workspaces by service, team, or project.

  • Each Workspace has its own Applications, members, Gen & Judge models, and evaluation history.
  • Data and settings in one Workspace don't affect other Workspaces.

Application

The basic unit of evaluation and monitoring.

  • It maps to one AI service in production, such as a customer support chatbot or a document summarization service.
  • A Workspace can have multiple Applications.

Module

A pipeline component that makes up an Application, such as RAG, Retriever, Router, or Text to SQL.

Evaluation levelTargetDescription
Module-level evalIndividual ModuleMeasures each component, such as RAG retrieval quality or Router classification accuracy.
E2E evalEntire ApplicationValidates the quality of the whole pipeline, from user input to final output.

Run both levels together to pinpoint which Module causes a drop in overall quality.

Roles

There are two kinds of roles: Organization roles and Workspace roles.

Diagram

①Organization roles

RoleWhat it can do
OwnerThe top administrator of the Organization. Each Organization has exactly one.
Can transfer ownership, delete the Organization, and assign Admins.
AdminManages all Workspaces and members. Can invite members and create Workspaces.
MemberCan access only the Workspaces they're assigned to. Their permissions in each Workspace are set by their Workspace role.

Owners and Admins automatically get Workspace Admin permissions in every Workspace. These inherited permissions can't be changed at the Workspace level.

②Workspace roles

RoleWhat it can do
Workspace AdminManages all Workspace settings (Workspace info, members, models, Applications).
EditorEdits and runs everything except Settings.
ViewerCan only view and export results and Dashboards.

③Permission summary

ActionOwnerAdminMemberWorkspace
Admin		EditorViewer
Transfer ownership · Delete Organization●-----
Assign · Remove Admin●●----
Change member roles · Invite · Remove●●----
Workspace settings · Assign member roles◯◯WS role●--
Create · Delete Application◯◯WS role●--
Create · Run evaluation◯◯WS role●●-
View · Export results◯◯WS role●●●
Work in Annotation Queue◯◯WS role●●●

● : Allowed / - : Not allowed
◯ : Inherited from the Organization role
WS role : An Organization Member follows the role they have in each Workspace

Diagram

Screen layout

①Main screen

1. Tour the main screen

  1. 1-1Organization · Workspace

    1. The top of the LNB shows the Organization you're using. Click it to open the Organization Switcher.

  2. 1-2Select an Application

    1. Choose the Application to evaluate from the selector under Application.

  3. 1-3Menu

    1. The Evaluation, Simulation, Red Teaming, Playground, and Observability menus are grouped here.

  4. 1-4My account · Notifications

    1. Click your name at the bottom to open the My account menu, or click 🔔 to open notifications. See My account.

②Settings

You can open Settings from three places: Organization Setting in the Organization Switcher (Owner and Admins), the ⚙ on a Workspace row in the Switcher (Workspace Admins and above for that Workspace), and Application Setting in the left menu. The left menu of the Settings screen is divided into three areas.

AreaMenu
OrganizationGeneral Setting, User Management, Organization Gen & Judge Model
WorkspaceGeneral Setting, User Management, Workspace Gen & Judge Model
ApplicationAgent Connection, Observability Connection
  • The areas you see depend on where you came from. From the Organization, you see only the Organization area. From a Workspace, you see the Workspace and Organization areas. From an Application, you see all three.
  • Workspace Admins don't see the Organization area. Editors and Viewers can't open Settings.
  • Click Back in the top left to return to where you were.

1. Tour the Settings screen

  1. 1-1Organization area

    1. Settings that apply to the whole Organization.

  2. 1-2Workspace area

    1. Settings for the current Workspace.

  3. 1-3Application area

    1. API connection settings for the current Application.

  4. 1-4Go back

    1. Click Back to return to the screen you were on before opening Settings.

Last updated