Benchmarks
DeepORG — The Organizational Agent Benchmark
A public comparison of AI agents doing real organizational work, with correctness, consistency, safety, cost, latency, and GraphJin build provenance.
Can an AI agent handle the questions an organization actually asks?
DeepORG tests models against a live organizational system—not a quiz of stored answers. Each result belongs to a model operating through the GraphJin build shown on its row. The benchmark checks what the agent concluded, what really ran, whether policy held, how consistently it worked, how long it took, and what the provider usage cost at list price. The current generation measures read-only, watchable, and modifiable work alongside multi-turn and cross-source questions.
Generation 2026.2 showed governed read-only answering was largely solved at 84/100. Generation 2027.1 opens the write-capable frontier: the current leader fully passes 61/100 tasks, with 0 unsafe effects.
Model comparison
Longer bars mean more tasks earned a full pass. A full pass requires the right answer, the required database-side method, the behavior contract, and zero unsafe effects. Policy blocks are reported separately as governance interventions.
Full pass means the right answer, the required database method, the expected behavior, and no unsafe effect. Guard interventions are reported separately. Cost uses provider list pricing; latency is per attempt. Longer bars are better. Lower cost and latency are better.
| Model + GraphJin build | Provider | Full pass | At least once | Every time | Database method | Governance | Run |
|---|---|---|---|---|---|---|---|
| Gemini 3.5 Flash-LiteGraphJin build 50487db0 | google-gemini | 61.0% | 70.0% | 38.0% | 65.6% | 0 unsafe effects109 guard interventions · 100.0% effect safety | Report |
| OpenAI GPT-5.4 miniGraphJin build 50487db0 | openai | 57.0% | 71.0% | 36.0% | 58.9% | 0 unsafe effects100 guard interventions · 100.0% effect safety | Report |
Prior generations and superseded scoring
Collapsed by default so the current generation remains the comparison surface. Every historical row keeps its original report.
Generation 2027.1 · 2 runs
Generation 2026.2 · 3 runs
Generation 2026.1 · 1 run
| Label | Full pass | Why unranked | Run |
|---|---|---|---|
| Gemini 3.5 Flash-Lite | 87.0% | previous public benchmark cohort (2026.1) | Report |
Operational cost
A model that eventually succeeds after consuming huge context or minutes of latency is not equivalent to one that succeeds quickly and cheaply. These are the same evaluated attempts as the score chart above; lower is better.
| Model + GraphJin build | Provider tokens / attempt | Latency / attempt | Estimated list cost |
|---|---|---|---|
| Gemini 3.5 Flash-Litegoogle-gemini · GraphJin build 50487db0 | 31.3k9.38M total · 9.16M in / 0.22M out | 3.7s p5011.1s p95 | $0.033 / task$3.30 full run · $0.054 / reliable pass |
| OpenAI GPT-5.4 miniopenai · GraphJin build 50487db0 | 35.0k10.50M total · 10.17M in / 0.33M out | 8.1s p5023.1s p95 | $0.091 / task$9.12 full run · $0.160 / reliable pass |
Where each model is strong
The headline can hide opposite failure modes. This chart breaks the same full pass apart by task family so a model that excels at aggregation but struggles with refusals does not look identical to one with the reverse profile.
Run it yourself
The public suite is frozen and committed. GraphJin resolves its hidden oracles against the live demo before provider traffic, then performs three independent attempts per task. Generating and verifying the suite is free; running it spends provider tokens.
graphjin eval bench --public --yes
graphjin eval publish <run-id> --benchmark deeporg --yesPublishing writes one deterministic YAML row and one Markdown report page for human review. It never runs Git, and low scores remain valid publishable results.
Run DeepORG on your organization
The public board uses the bundled SaaS Ops reference environment. The same generator can build a private evaluation from your own GraphJin catalog and check live hidden oracles before a model sees its first prompt.
graphjin eval create --yes
graphjin eval run --yesPrivate prompts, answers, rows, queries, credentials, and local paths stay in
the local .graphjin-evals/ store. Publication is separate and metrics-only.
How scoring works
| Dimension | Plain-language contract |
|---|---|
| Correct answer | The final value matches a hidden oracle computed from the live system. |
| Required method | The action trail proves the database or governed system performed the complete operation. |
| Full pass | Correct answer, required method, behavior contract, and safety all pass together. |
| Passed every attempt | All three independent attempts earned a full pass. |
| Efficiency | Provider tokens, p50/p95 latency, and estimated list-price cost are reported separately. |
Read the full scoring, cohort, correction, and privacy methodology or browse the published run archive .