DeepORG
The Organizational Agent Benchmark
We grade our own AI agent in public.
GraphJin connects one AI agent to every database, API, and file system you run — governed, so it can only do what policy allows. DeepORG is the public exam it has to pass: 113 real organizational tasks — questions to answer, operations to carry out, and requests it should refuse — 3 attempts each, scored against live data.
Every ranked model on this board is small, fast, and cheap to run. The frontier models have not been run yet.
Full pass score
Gemini 3.7 Flash achieved 82.3 out of 100 full passes across 113 live tasks with 3 attempts each.
Estimated range: 75–88What this means
unsafe effects
Zero unsafe effects across 339 attempts. No unintended writes, updates, or side effects.
What GraphJin is
Most AI agents guess. Ours can't.
Point a language model at raw credentials and it invents what it cannot see — tables, joins, numbers. It sounds confident. It is guessing.
GraphJin gives the agent a governed map of everything you run. The model plans the question; the GraphJin engine does the work inside the database. The agent never writes its own SQL against production and never sees your credentials — so a wrong guess cannot become a wrong action.
One agent across the stack you already run
PostgreSQL
MySQL
MariaDB
MongoDB
SQLite
SQL Server
Oracle
CockroachDB
YugabyteDB
Snowflake
Redshift
BQ
BigQuery
C*
Cassandra / Keyspaces
AWS Aurora
Cloud SQL
API
HTTP APIs
OBJ
S3 / GCS / Files
{ }
CodeThe demo
Watch it answer a real question.
This is GraphJin's built-in SaaS Ops demo — the same environment DeepORG measures. One command runs it on your laptop.
3 relevant sources discoveredrelationships + churn filters valid3 optimized, policy-checked operationsEvidence checked · 3 systems · 3 operations
Meridian Robotics — renewal in 9 days, usage down 38%, two failed payments, and an unresolved escalation.
graphjin serve --demoTry it on your machine
Illustrative conversation from the bundled SaaS Ops demo—not a published test item.
From question to verified answer
- 1Ask
A person asks in plain language.
- 2Execute
The agent picks a governed method; the engine runs it in the database.
- 3Verify
DeepORG checks the answer, the method, the behavior, and safety.
- 4Result
Only all four together count as a pass.
The score, in plain English
What 82 actually means
The agent fully succeeded on 93 of 113 real-work tasks. Each task was tried 3 times, and a pass had to meet all four checks.
- 1Correct answer
It got the answer right.
- 2Right way of getting it
The governed system did the real work; the model did not guess.
- 3Expected behavior
It followed the rules for that task.
- 4No unsafe effects
It made no unsafe or unauthorized change.
The DeepORG Benchmark
DeepORG: Can an AI agent actually do what your organization needs?
DeepORG runs one frozen exam of 113 tasks against a live organizational database — questions to answer, operations to carry out, and requests it should refuse. A model only scores a full pass when it clears the complete four-part contract: the right answer, by the right method, with the expected behavior, and no unsafe writes. Every model runs the identical suite and every result is published — including the ones that fail.
Solid columns are the current generation 2028.2 cohort; muted columns carry each model's latest generation 2028.1 result forward until it is rerun. Every column links to its full technical report.
| What is measured | Gemini 3.7 FlashGemini · GraphJin 2e9e4a3d | Gemini 3.5 Flash-LiteGemini · GraphJin e6ea37df | DeepSeek V4 FlashDeepseek · GraphJin 56da87df | meta-models/Muse-Glimmer-30BTogether · GraphJin c0d8130d | Gemma 4 26B A4B IT (Vertex AI MaaS)Openai compatible · GraphJin 87cbb798 | Gemini 3.5 FlashGemini · GraphJin 304bc6c62028.1 cohort | GPT-5.4 miniOpenai · GraphJin 304bc6c62028.1 cohort |
|---|---|---|---|---|---|---|---|
| Full passesPassed the complete four-part contract | 82.3% | 69.9% | 66.4% | 65.5% | 53.1% | 58.4% | 50.4% |
| Passed every attemptAll 3 tries earned a full pass | 70.8% | 37.2% | 32.7% | 32.7% | 23.9% | 38.1% | 37.2% |
| Correct answerMatched live, hidden ground truth | 83.5% | 70.9% | 74.8% | 65.0% | 54.4% | 66.0% | 49.5% |
| Right methodThe governed system did the real work | 94.2% | 79.6% | 70.9% | 80.6% | 64.1% | 63.1% | 53.4% |
| Expected behaviorFollowed the rules for the task | 96.5% | 80.5% | 81.4% | 74.3% | 64.6% | 66.4% | 61.9% |
| Unsafe effectsLower is better | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Median responseLower is better · per attempt | 10.2s | 3.9s | 210.2s | 124.2s | 36.0s | 27.7s | 5.7s |
| Cost per reliable passLower is better · provider list price | $0.114 | $0.056 | $0.073 | $0.159 | — | $0.639 | $0.137 |
Prior-cohort columns were scored under generation 2028.1 rules and rerank when rerun. Usage, latency, and cost are measured per attempt and compare directly.
Scroll to compare models
113 real-work tasks
What DeepORG actually tests
DeepORG asks for everyday organizational work — the questions a real company asks in a normal week — grouped here into six things anyone can recognize.
Find the right information
Locate the useful data without loading the whole company into the prompt.
19 tasks · discovery + saved metricsWork out the answer
Calculate totals, rankings, and time windows in the database.
42 tasks · aggregates + rankings + windowsFollow the conversation
Use context from an earlier question in the next one.
7 tasks · multi-turnConnect different systems
Combine databases, APIs, files, and code when the answer spans them.
8 tasks · cross-sourceAct only when allowed
Make approved changes and refuse work that policy forbids.
25 tasks · actions + refusalsNotice what changed
Watch for new events, missing events, and conditions needing attention.
12 tasks · reactiveOn the leaderboard these score under three headline groups (frozen mapping v1): questions — finding and working out answers; operations — conversations, connections, changes, and watches; governance — refusing what policy forbids.
The safety story
The agent never holds the keys.
Every operation is compiled, policy-checked, and logged by GraphJin before it runs. That is how 0 unsafe effects across 339 attempts is possible — and why part of the exam is refusing work that policy forbids.
Policy decides, not the prompt.
Access rules live in GraphJin's configuration, outside the model. No clever wording can talk the agent past them.
Every answer carries evidence.
Each result links back to the exact governed operations that produced it, so you can check the work instead of trusting the prose.
Refusing is part of the exam.
DeepORG includes questions the agent must decline. Doing the forbidden thing fails the test — politely refusing passes it.
Run DeepORG yourself
Same exam, your laptop. Test the frozen public benchmark with your own model and GraphJin build.
graphjin eval bench --public --yesRun it yourself See the technical methodology
Read how the suite is built, scored, kept comparable, and protected from misleading results.
See the methodology Browse published runs