Learning from real work
Your company is the environment.
Every question your people ask is answered by a database query. DeepORG keeps the question and the query, and turns them into a test it can run again: on a cheaper model, on a new prompt, on next month's data. The database marks every attempt.
1 · someone asked
query { support_tickets(where: { status: { eq: "open" } }) { count_id } }There are 4 open support tickets right now.
2 · it became a test
prompt: How many support tickets are currently open right now?answer key: the query above · tier: incumbent · value then: 43 · it was asked again
one ticket resolved since · answer key re-run: 3There are 3 open support tickets.
pass · marked by the database, not by a model
It is re-run every time the test runs, so a test written in March still marks correctly in June. The ticket count above went from 4 to 3 and the test followed it.
A test nobody rated measures agreement with the agent that first answered. One a person marked right measures correctness. Reports score the two apart and never add them up. An answer marked wrong never becomes a test.
Tests are built inside your deployment. A test keeps the question, its query and the one value the query gave. No result rows are copied, the agent's answers stay on the server, each query re-runs as the person who asked it, and only an admin can read the tests.
What you do with it
A benchmark nobody else has: yours.
Public benchmarks tell you how a model does on someone else's schema. These tests are the questions your organization actually asks, answered from your data.
Measure
Is the cheaper model good enough here? Run the current one and a candidate over your own questions and compare them test by test, with confirmed and incumbent results kept apart.
graphjin eval run --suite eval/mined.ymlTrain
The same tests run in the agent environment: as many attempts as you like, each marked by the database, exported as trajectories a training loop can use.
graphjin eval export <run-id>Trust
Changes are checked too. A change the agent proposes is held for approval unless you let it act alone within limits, a routine's change must say what it should achieve, and DeepORG checks later whether it did.
GET /api/v1/agent/reliabilityWhat it costs
Building the tests calls no model.
Mining re-runs queries, which is database work. The engine's own parser decides which questions qualify and reads the answer out of the result the query returned. A model is called only when you run the tests.
What is not a test yet
Every question it skips is counted, with the reason.
A question becomes a test when its answer is one figure or one top row that the database computed. In our demo run, ten turns asked the data agent 14 questions. Four became tests: one a person had marked right, three nobody had rated. All four passed on gpt-6-luna. The other ten, by reason:
"Which accounts are on the enterprise plan?" has no single figure to check.
An average and a count from one query: which one the question meant is not the engine's guess to make.
A person said the answer was wrong. It goes to the corrections list instead, with the queries the agent ran.
The agent stopped before it answered, so there is nothing to check against.
The answer came without a query that returned data.
A count read through a relationship, not one row of one table.
Try it
Ask the demo a few questions, then mine them.
Ask the agent a handful of questions in the console, mark one or two answers right, then turn them into tests and run them. The guide walks through it, including dates, ratings and what each skip reason means.
Mine
Reads the questions already asked on a running server. Admin only.
graphjin eval mine --demo --url http://localhost:8083 --token $ADMIN_TOKENRun
Runs the mined tests on the model the server is configured with.
graphjin eval run --demo --suite eval/mined.yml --yes