Learning From Real Work Every question your people ask is answered by a database query. DeepORG keeps the question and the query and turns them into a test it can run again — on a cheaper model, a new prompt, or next month's data — with the database marking every attempt. environment overview experience experience/_index.md

Your company is the environment.

Every question your people ask is answered by a database query. DeepORG keeps the question and the query, and turns them into a test it can run again: on a cheaper model, on a new prompt, on next month's data. The database marks every attempt.

DeepORG · saas-ops demo

1 · someone asked

How many support tickets are open right now?
execute_graphql
query { support_tickets(where: { status: { eq: "open" } }) { count_id } }

There are 4 open support tickets right now.

2 · it became a test

graphjin eval mine
prompt: How many support tickets are currently open right now?answer key: the query above · tier: incumbent · value then: 4

3 · it was asked again

graphjin eval run --suite eval/mined.yml
one ticket resolved since · answer key re-run: 3

There are 3 open support tickets.

pass · marked by the database, not by a model

How does it know the right answer?The answer key is a query, not a number

It is re-run every time the test runs, so a test written in March still marks correctly in June. The ticket count above went from 4 to 3 and the test followed it.

What if the agent was wrong the first time?Agreement is not correctness

A test nobody rated measures agreement with the agent that first answered. One a person marked right measures correctness. Reports score the two apart and never add them up. An answer marked wrong never becomes a test.

Where does my data go?Nowhere

Tests are built inside your deployment. A test keeps the question, its query and the one value the query gave. No result rows are copied, the agent's answers stay on the server, each query re-runs as the person who asked it, and only an admin can read the tests.

A benchmark nobody else has: yours.

Public benchmarks tell you how a model does on someone else's schema. These tests are the questions your organization actually asks, answered from your data.

Measure

Is the cheaper model good enough here? Run the current one and a candidate over your own questions and compare them test by test, with confirmed and incumbent results kept apart.

graphjin eval run --suite eval/mined.yml

Train

The same tests run in the agent environment: as many attempts as you like, each marked by the database, exported as trajectories a training loop can use.

graphjin eval export <run-id>

Trust

Changes are checked too. A change the agent proposes is held for approval unless you let it act alone within limits, a routine's change must say what it should achieve, and DeepORG checks later whether it did.

GET /api/v1/agent/reliability

Building the tests calls no model.

Mining re-runs queries, which is database work. The engine's own parser decides which questions qualify and reads the answer out of the result the query returned. A model is called only when you run the tests.

A person asksin chat or a routineThe agent answerswith one database queryIt becomes a testthe query is the answer keyAsked againthe key re-runs that dayMeasure a modelTrain a model

Every question it skips is counted, with the reason.

A question becomes a test when its answer is one figure or one top row that the database computed. In our demo run, ten turns asked the data agent 14 questions. Four became tests: one a person had marked right, three nobody had rated. All four passed on gpt-6-luna. The other ten, by reason:

2Lists

"Which accounts are on the enterprise plan?" has no single figure to check.

2Several figures

An average and a count from one query: which one the question meant is not the engine's guess to make.

2Marked wrong

A person said the answer was wrong. It goes to the corrections list instead, with the queries the agent ran.

2Not answered

The agent stopped before it answered, so there is nothing to check against.

1No query

The answer came without a query that returned data.

1Nested

A count read through a relationship, not one row of one table.

Ask the demo a few questions, then mine them.

Ask the agent a handful of questions in the console, mark one or two answers right, then turn them into tests and run them. The guide walks through it, including dates, ratings and what each skip reason means.

Mine

Reads the questions already asked on a running server. Admin only.

graphjin eval mine --demo --url http://localhost:8083 --token $ADMIN_TOKEN

Run

Runs the mined tests on the model the server is configured with.

graphjin eval run --demo --suite eval/mined.yml --yes