DeepORG — The Organizational Agent Benchmark Can an AI agent handle the questions an organization actually asks? Explore DeepORG scores, model comparisons, safety, cost, and the meaning behind every full pass. benchmarks benchmark-overview benchmark benchmark/_index.md

Benchmarks

Last updated: 2026-08-14

DeepORG

The Organizational Agent Benchmark

Can an AI agent handle the questions an organization actually asks?

DeepORG tests models against live questions across real organizational systems. It measures what matters: correct answers, the right execution path, and zero unsafe effects.

82/100

Full pass score

Gemini 3.7 Flash achieved 82.3 out of 100 full passes across 113 live tasks with 3 attempts each.

Estimated range: 75–88What this means

0

unsafe effects

Zero unsafe effects across 339 attempts. No unintended writes, updates, or side effects.

Latency (p50)10.2sper attempt
Reliable passat list price
Scope113live tasks
Attempts3attempts each
Generation2028.2tested on

Model comparison

How this result breaks down

More model columns will appear automatically as comparable results are published.

DeepORG benchmark results by model
What is measuredGemini 3.7 FlashGemini · GraphJin 2e9e4a3d
Full passesPassed the complete four-part contract82.3%
Passed every attemptAll 3 tries earned a full pass70.8%
Correct answerMatched live, hidden ground truth83.5%
Right methodThe governed system did the real work94.2%
Expected behaviorFollowed the rules for the task96.5%
Unsafe effectsLower is better0
Median responseLower is better · per attempt10.2s
Cost per reliable passLower is better · provider list price

The score, in plain English

What 82 actually means

The agent fully succeeded on 93 of 113 real-work tasks. Each task was tried 3 times, and a pass had to meet all four checks.

  1. 1
    Correct answer

    It got the answer right.

  2. 2
    Right way of getting it

    The governed system did the real work; the model did not guess.

  3. 3
    Expected behavior

    It followed the rules for that task.

  4. 4
    No unsafe effects

    It made no unsafe or unauthorized change.

113 real-work tasks

What DeepORG actually tests

DeepORG uses everyday organizational work—not trivia questions. The ten technical task families are grouped here into six things anyone can recognize.

Find the right information

Locate the useful data without loading the whole company into the prompt.

19 tasks · discovery + saved metrics

Work out the answer

Calculate totals, rankings, and time windows in the database.

42 tasks · aggregates + rankings + windows

Follow the conversation

Use context from an earlier question in the next one.

7 tasks · multi-turn

Connect different systems

Combine databases, APIs, files, and code when the answer spans them.

8 tasks · cross-source

Act only when allowed

Make approved changes and refuse work that policy forbids.

25 tasks · actions + refusals

Notice what changed

Watch for new events, missing events, and conditions needing attention.

12 tasks · reactive

Illustrative example

From question to verified answer

DeepORG evaluates the complete agent workflow end to end, just like production.

  1. 1
    Ask

    A user asks a real question in natural language.

  2. 2
    Execute

    The agent selects and runs the governed method.

  3. 3
    Verify

    DeepORG checks the answer, method, behavior, and safety.

  4. 4
    Result

    Only all four together earn a full pass.

QuestionWhat was our ARR last month by region?
Method used (database-side)arr_by_region(as_of_date: "2026-07-31")
Answer (verified)
RegionARR (USD)
North America$12,431,000
EMEA$7,842,000
APAC$5,219,000
Latin America$1,908,000
Total$27,400,000
Full passCorrect answer · right method · expected behavior · zero unsafe effects

Illustrative flow—not a published test item.

Run DeepORG yourself

Test the frozen public benchmark with your own model and GraphJin build.

graphjin eval bench --public --yesRun it yourself