DeepORG
The Organizational Agent Benchmark
Can an AI agent handle the questions an organization actually asks?
DeepORG tests models against live questions across real organizational systems. It measures what matters: correct answers, the right execution path, and zero unsafe effects.
Full pass score
Gemini 3.7 Flash achieved 82.3 out of 100 full passes across 113 live tasks with 3 attempts each.
Estimated range: 75–88What this means
unsafe effects
Zero unsafe effects across 339 attempts. No unintended writes, updates, or side effects.
Model comparison
How this result breaks down
More model columns will appear automatically as comparable results are published.
| What is measured | Gemini 3.7 FlashGemini · GraphJin 2e9e4a3d |
|---|---|
| Full passesPassed the complete four-part contract | 82.3% |
| Passed every attemptAll 3 tries earned a full pass | 70.8% |
| Correct answerMatched live, hidden ground truth | 83.5% |
| Right methodThe governed system did the real work | 94.2% |
| Expected behaviorFollowed the rules for the task | 96.5% |
| Unsafe effectsLower is better | 0 |
| Median responseLower is better · per attempt | 10.2s |
| Cost per reliable passLower is better · provider list price | — |
The score, in plain English
What 82 actually means
The agent fully succeeded on 93 of 113 real-work tasks. Each task was tried 3 times, and a pass had to meet all four checks.
- 1Correct answer
It got the answer right.
- 2Right way of getting it
The governed system did the real work; the model did not guess.
- 3Expected behavior
It followed the rules for that task.
- 4No unsafe effects
It made no unsafe or unauthorized change.
113 real-work tasks
What DeepORG actually tests
DeepORG uses everyday organizational work—not trivia questions. The ten technical task families are grouped here into six things anyone can recognize.
Find the right information
Locate the useful data without loading the whole company into the prompt.
19 tasks · discovery + saved metricsWork out the answer
Calculate totals, rankings, and time windows in the database.
42 tasks · aggregates + rankings + windowsFollow the conversation
Use context from an earlier question in the next one.
7 tasks · multi-turnConnect different systems
Combine databases, APIs, files, and code when the answer spans them.
8 tasks · cross-sourceAct only when allowed
Make approved changes and refuse work that policy forbids.
25 tasks · actions + refusalsNotice what changed
Watch for new events, missing events, and conditions needing attention.
12 tasks · reactiveIllustrative example
From question to verified answer
DeepORG evaluates the complete agent workflow end to end, just like production.
- 1Ask
A user asks a real question in natural language.
- 2Execute
The agent selects and runs the governed method.
- 3Verify
DeepORG checks the answer, method, behavior, and safety.
- 4Result
Only all four together earn a full pass.
arr_by_region(as_of_date: "2026-07-31")| Region | ARR (USD) |
|---|---|
| North America | $12,431,000 |
| EMEA | $7,842,000 |
| APAC | $5,219,000 |
| Latin America | $1,908,000 |
| Total | $27,400,000 |
Illustrative flow—not a published test item.
Run DeepORG yourself
Test the frozen public benchmark with your own model and GraphJin build.
graphjin eval bench --public --yesRun it yourself See the technical methodology
Read how the suite is built, scored, kept comparable, and protected from misleading results.
See the methodology Browse published runs