DeepORG
The Organizational Agent Benchmark
We grade our own AI agent in public.
GraphJin connects one AI agent to every database, API, and file system you run — governed, so it can only do what policy allows. DeepORG is the public exam it has to pass: 113 real organizational tasks — questions to answer, operations to carry out, and requests it should refuse — 3 attempts each, scored against live data.
The board keeps the best trustworthy result for every model we've tested. Publishing a newer exam never makes a paid result disappear.
Full pass score
Gemini 3.8 Flash achieved 98.2 out of 100 full passes across 113 live tasks with 3 attempts each.
Estimated range: 96–100What this means
unsafe effects
Zero unsafe effects across 339 attempts. No unintended writes, updates, or side effects.
What GraphJin is
Most AI agents guess. Ours can't.
Point a language model at raw credentials and it invents what it cannot see — tables, joins, numbers. It sounds confident. It is guessing.
GraphJin gives the agent a governed map of everything you run. The model plans the question; the GraphJin engine does the work inside the database. The agent never writes its own SQL against production and never sees your credentials — so a wrong guess cannot become a wrong action.
One agent across the stack you already run
PostgreSQL
MySQL
MariaDB
MongoDB
SQLite
SQL Server
Oracle
CockroachDB
YugabyteDB
Snowflake
Redshift
BQ
BigQuery
C*
Cassandra / Keyspaces
AWS Aurora
Cloud SQL
API
HTTP APIs
OBJ
S3 / GCS / Files
{ }
CodeThe demo
Watch it answer a real question.
This is GraphJin's built-in SaaS Ops demo — the same environment DeepORG measures. One command runs it on your laptop.
3 relevant sources discoveredrelationships + churn filters valid3 optimized, policy-checked operationsEvidence checked · 3 systems · 3 operations
Meridian Robotics — renewal in 9 days, usage down 38%, two failed payments, and an unresolved escalation.
graphjin serve --demoTry it on your machine
Illustrative conversation from the bundled SaaS Ops demo—not a published test item.
From question to verified answer
- 1Ask
A person asks in plain language.
- 2Execute
The agent picks a governed method; the engine runs it in the database.
- 3Verify
DeepORG checks the answer, the method, the behavior, and safety.
- 4Result
Only all four together count as a pass.
The score, in plain English
What 98 actually means
The agent fully succeeded on 111 of 113 real-work tasks. Each task was tried 3 times, and a pass had to meet all four checks.
- 1Correct answer
It got the answer right.
- 2Right way of getting it
The governed system did the real work; the model did not guess.
- 3Expected behavior
It followed the rules for that task.
- 4No unsafe effects
It made no unsafe or unauthorized change.
The DeepORG Benchmark
DeepORG: Can an AI agent actually do what your organization needs?
DeepORG grades models against a live organizational system — questions to answer, operations to carry out, and requests they should refuse. A model only scores a full pass when it clears the complete four-part contract: the right answer, by the right method, with the expected behavior, and no unsafe writes.
Best published score for every model we've tested. The exam improves over time, but older trustworthy results stay visible until that model posts a higher score. Every column links to the exact test report.
| What is measured | Gemini 3.8 FlashGoogle gemini · best published · Sep 3, 2026 | gemini-3.7-flashGemini · best published · Aug 26, 2026 | gemini-3.5-flash-liteGemini · best published · Aug 30, 2026 | Gemma 4 31B (Cerebras)Cerebras · best published · Aug 26, 2026 | DeepSeek V4 FlashDeepseek · best published · Aug 19, 2026 | meta-models/Muse-Glimmer-30BTogether · best published · Aug 21, 2026 | Gemini 3.5 FlashGemini · best published · Aug 14, 2026 | Gemma 4 26B A4B IT (Vertex AI MaaS)Openai compatible · best published · Aug 17, 2026 | GPT-5.4 miniOpenai · best published · Aug 13, 2026 | OpenAI GPT-4o miniOpenai · best published · Aug 7, 2026 |
|---|---|---|---|---|---|---|---|---|---|---|
| Full passesPassed the complete four-part contract | 98.2% | 97.3% | 88.5% | 85.0% | 66.4% | 65.5% | 58.4% | 53.1% | 50.4% | 8.0% |
| Passed every attemptAll tries earned a full pass | 94.7% | 86.7% | 61.9% | 70.8% | 32.7% | 32.7% | 38.1% | 23.9% | 37.2% | 4.0% |
| Correct answerMatched live, hidden ground truth | 100.0% | 99.0% | 90.3% | 86.4% | 74.8% | 65.0% | 66.0% | 54.4% | 49.5% | 2.2% |
| Right methodThe governed system did the real work | 98.1% | 97.1% | 94.2% | 94.2% | 70.9% | 80.6% | 63.1% | 64.1% | 53.4% | 5.6% |
| Expected behaviorFollowed the rules for the task | 99.1% | 99.1% | 94.7% | 94.7% | 81.4% | 74.3% | 66.4% | 64.6% | 61.9% | 9.0% |
| Unsafe effectsLower is better | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Median responseLower is better · per attempt | 78.3s | 13.8s | 4.4s | 12.5s | 210.2s | 124.2s | 27.7s | 36.0s | 5.7s | 25.8s |
| Cost per reliable passLower is better · provider list price | — | $0.092 | $0.043 | $0.208 | $0.073 | $0.159 | $0.639 | — | $0.137 | $0.456 |
Scores come from each model's best trustworthy published run. Dates and full reports make the exact test version visible without removing older results.
Scroll to compare models
113 real-work tasks
What DeepORG actually tests
DeepORG asks for everyday organizational work — the questions a real company asks in a normal week — grouped here into six things anyone can recognize.
Find the right information
Locate the useful data without loading the whole company into the prompt.
19 tasks · discovery + saved metricsWork out the answer
Calculate totals, rankings, and time windows in the database.
42 tasks · aggregates + rankings + windowsFollow the conversation
Use context from an earlier question in the next one.
7 tasks · multi-turnConnect different systems
Combine databases, APIs, files, and code when the answer spans them.
8 tasks · cross-sourceAct only when allowed
Make approved changes and refuse work that policy forbids.
25 tasks · actions + refusalsNotice what changed
Watch for new events, missing events, and conditions needing attention.
12 tasks · reactiveOn the leaderboard these score under three headline groups (frozen mapping v1): questions — finding and working out answers; operations — conversations, connections, changes, and watches; governance — refusing what policy forbids.
The safety story
The agent never holds the keys.
Every operation is compiled, policy-checked, and logged by GraphJin before it runs. That is how 0 unsafe effects across 339 attempts is possible — and why part of the exam is refusing work that policy forbids.
Policy decides, not the prompt.
Access rules live in GraphJin's configuration, outside the model. No clever wording can talk the agent past them.
Every answer carries evidence.
Each result links back to the exact governed operations that produced it, so you can check the work instead of trusting the prose.
Refusing is part of the exam.
DeepORG includes questions the agent must decline. Doing the forbidden thing fails the test — politely refusing passes it.
Run DeepORG yourself
Same exam, your laptop. Test the frozen public benchmark with your own model and GraphJin build.
graphjin eval bench --public --yesRun it yourself See the technical methodology
Read how the suite is built, scored, kept comparable, and protected from misleading results.
See the methodology Browse published runs