Benchmark Methodology

DeepORG, The Organizational Agent Benchmark built and published by GraphJin, asks models to work against a real, governed organizational schema. It measures the final answer and the path used to reach it instead of treating text-to-SQL as a single exact-match problem.

Every leaderboard score belongs to the submitted system: a model and provider operating through the recorded GraphJin commit and exact binary. It is not a context-free claim about the model. Material agent-runtime changes therefore require a new run and remain visible as a new GraphJin build rather than silently replacing history.

Test-version scope: 2026.1/2026.2 — governed read-only Q&A · 2027.1 — adds writes, alerts, follow-ups, multi-source · 2028.1 — intent-phrased tasks with execution twins · 2028.2 — cross-source ruler fix (v11) · 2028.3 — executable join shape for join-only tables · 2028.4 — tasks state what they grade; closable tickets documented. Every report keeps its exact test version, scorer contract, GraphJin build, and invalidation status even when it is not selected for the public board.

Frozen suite, live verification

The current 2028.4 test version uses generator contract graphjin.eval.generator/v12 and a committed, deterministic 113-task suite built from the bundled SaaS Ops demo at seed 23. Alongside organizational questions and governed refusals, it measures writes with post-state and collateral-safety checks, standing-watch definition and delivery, history-grounded follow-ups, and work that joins database evidence with file or API sources. Tasks are intent-phrased with hidden executable oracles, and execution twins isolate planning from execution. Publishing the suite makes a run reproducible. Later test versions can improve the exam without rewriting or hiding older trustworthy results.

Before any evaluated-agent traffic, GraphJin resolves every hidden oracle against the live instance. A broken oracle invalidates the suite and stops the run. Calendar-relative windows use a live anchor, so the expected answer stays aligned with the demo as its dates move forward.

What is scored

Each task runs three times. The report keeps the dimensions separate:

Recall is the fraction of tasks whose majority verdict passes. pass@3 asks whether at least one of the three rollouts passed; pass³ asks whether all three passed. The report also publishes a bootstrap confidence interval and per-tier and per-task-family results.

Safety and acceptance

Safety measures effects, not requests. Reports therefore separate governance interventions, forbidden attempts that GraphJin refused, and unsafe effects. Refused forbidden attempts fail behavior so refusal tasks remain failures; executed forbidden actions and collateral-state changes fail safety.

Safety is always a hard gate. Suite validity and environment health are also reported independently from answer quality. accepted records GraphJin’s local regression-gate result; it is not an admission threshold for this public board. Publish refuses incomplete, environment-failed, invalid-suite, and empty runs. It does not refuse a low score—a low score is a result.

Test versions and board eligibility

Ranked identity includes:

• mode and suite fingerprint;
• catalog and seed-manifest hashes;
• rollout seed, repeats, maximum steps, and temperature; and
• reward version.

Model, provider, GraphJin commit, and binary fingerprint are presentation axes, not test-version keys. Oracle value hash and data anchor are audit-only because the demo’s relative dates shift with the calendar.

A run outside the pinned cohort can be published only with an explicit --allow-off-suite; it remains in the archive with the mismatch reason and is not eligible for the main board.

The public board uses two explicit fields. model_key identifies the canonical provider/model route and normalizes known provider aliases such as gemini and google-gemini. board_eligible records the trust decision: official completed runs are eligible, while off-suite, broken-scorer, harness-defect, and retracted runs are not. The local accepted regression-gate result does not control board eligibility, so a valid failed or low-scoring run remains a real published result.

For each model_key, the board selects the eligible run with the highest full-pass recall; an exact tie selects the newer run. Models are then ordered by that score. This headline comparison can include different test versions, so the date and exact report are part of every selected row. Task-family charts remain limited to runs on the same current test version.

Scoring correction

An early public run was invalidated after its stale suite rejected a valid database aggregate dialect. The model’s answers were usually correct, but the published method score said otherwise. It remains in the archive with the invalidation reason and cannot become that model’s best result.

Three guards followed: a generator-version mismatch now makes the suite invalid before provider traffic starts; a large answer/method divergence marks the report as scoring-suspect and blocks publication; and publishing requires the run’s binary fingerprint to match the binary doing the publishing. Older valid reports remain eligible; invalidated results remain archive-only.

Reward contract graphjin.eval.reward/v4 also separates forbidden attempts from unsafe effects. A known broken scorer contract makes affected runs ineligible, while trustworthy results under older contracts retain their original scores and reports.

The same rule applied when the ruler itself was wrong: under generator v10, argument-free file reads did not count toward the method dimension, deflating the cross-source family to near zero for the entire 2028.1 cohort. Generator v11 restores that credit. The affected 2028.1 runs keep their reports, and the eligibility field records whether each original score remains trustworthy; published rows are never silently rescored in place.

Privacy

Published reports contain metrics, public task IDs and categories, fingerprints, provenance, and acceptance state. They never contain prompts, answers, database rows, executed queries, headers, credentials, task slugs, raw oracle errors, or local episode paths. The private JSON trajectories remain in the local .graphjin-evals/ store.