Measure Your Agent On Your Own Questions Turn the questions people already asked the agent into tests whose answer key is the database query that answered them, then run those tests on any model. environment guide environment/your-own-work environment/your-own-work.md

Measure Your Agent On Your Own Questions

Turn the questions people already asked the agent into tests whose answer key is the database query that answered them, then run those tests on any model.

Your own graph builds tests from your schema. This builds them from your work: the questions people already asked the agent on a running server. The overview is at Learning from real work.

What becomes a test

Every turn on a thread is stored with the questions the lead asked the data agent, the answers, and the queries the data agent ran. A question becomes a test when the data agent’s last successful read was one of these:

  • One aggregate figure: query { accounts { sum_mrr_cents } }.
  • One top row: an order_by with limit: 1. When the row has a name, the answer must name it; when it has only an id, the figure itself is the answer (“the largest invoice”).
  • One looked-up value: any single row with limit: 1.

That query becomes the test’s answer key. It is not a stored number: the test re-runs it every time it runs, so the expected answer is always the one the data has that day.

The engine decides this, not a model. The query is parsed by GraphJin’s own parser, re-run against the database, and the answer is read from the result it returned.

Two more checks happen before a question is kept:

  • The query re-runs as the person who asked, never as the admin reading the tests, so it sees exactly the rows they saw.
  • The answer they were given must pass it. If it does not, the query was not where the answer came from, or the data has moved on, and the question is dropped.

Mine the questions

Mining reads every person’s questions, so it needs an admin. Against a server set up with graphjin cli setup:

Shell
graphjin eval mine --remote

Against a server you name, with an admin’s token:

Shell
graphjin eval mine --url https://graphjin.internal --token "$ADMIN_TOKEN"

The command reads GET /api/v1/agent/experience page by page, verifies every answer key against the evaluation target exactly as eval create verifies its own tasks, and writes eval/mined.yml:

text
Read 14 questions to the data agent from http://127.0.0.1:8191.
  4 became tests on the server (3 incumbent, 1 confirmed).
  not tests yet: 2 list, 2 not_answered, 2 rated_wrong, 2 several_figures, 1 no_query, 1 shape
Wrote 4 tests to eval/mined.yml (3 incumbent, 1 confirmed).
Run them with: graphjin eval run --suite eval/mined.yml

That is a real run on the saas-ops demo: ten console turns, fourteen questions to the data agent. --since and --origin chat|routine|continuation|outcome narrow what is read; --from reads a response you saved instead of fetching one.

Mining spends no model tokens. It costs one query per candidate on the database the questions were asked against.

Run them

Shell
graphjin eval run --suite eval/mined.yml --yes

Mined tests are a suite of their own, apart from eval/suite.yml: a generated suite samples its candidates, and a question somebody really asked should never lose that draw. A --suite run is compared with no baseline and promotes none.

In the run above, all four tests passed on gpt-6-luna, using 130,913 tokens. Then one open ticket was resolved and the open-tickets test ran again: its answer key now gave 3 instead of 4, the model answered “There are 3 open support tickets”, and the test passed.

To compare models, run the same suite with each and read the reports side by side. To train on the tests, run them in the environment and export the attempts with graphjin eval export.

Confirmed and incumbent

Every mined test has a tier, and the tiers are scored apart. A run of mined tests prints one line per tier, its reports have no combined score, and the JSON report keeps each tier under metrics.by_family. Read those: the JSON’s overall metrics.recall still counts every test together.

text
Confirmed tests: 1 of 1 passed (marked right by a person, so this is correctness).
Incumbent tests: 3 of 3 passed (never rated, so this is agreement with the agent that first answered).
TierWhere it comes fromWhat its score means
confirmedA person marked the answer right, on a turn that asked one questionCorrectness
incumbentNobody rated the answerAgreement with the agent that first answered

An incumbent test can be wrong: if the first answer was wrong and nobody said so, a better model scores lower on it. That is why the tiers stay apart. A thumbs-up on a turn that asked several questions does not say which answer was right, so those tests stay incumbent.

An answer a person marked wrong never becomes a test. It shows up in GET /api/v1/agent/feedback?format=corpus as a draft with the person’s correction and the queries the agent ran, waiting for someone to write the right answer key.

What is not a test yet

The server’s summary counts every question it skipped, by reason, and the full response lists each one under drops with the detail:

ReasonWhat it means
listThe answer was a list of rows
several_figuresThe query returned more than one figure, or more than one root
shapeThe result did not have one row of plain values, for example a nested relationship
datedThe question names a date, such as “November 2026”
uses_variablesThe query took variables, which the action trail does not keep
rated_wrongA person marked the answer wrong
not_answeredThe data agent did not answer
no_queryNo read returned data
unsafeThe run did something a guard did not stop
writeThe question changed data
query_failed, empty_resultThe answer key no longer runs, or returns nothing
answer_mismatchThe answer given does not pass the key
proseThe question names GraphJin’s own mechanism instead of a need
duplicateThe same question was already kept
asker_unknownA routine ran as the service identity, which has since been changed, so the rows it saw cannot be rebuilt

These counts are worth reading: they say which kinds of real questions your organization asks that cannot be tested yet.

Dates

A question like “which account renews soonest?” is often answered with a date in the query: renewal_date: { gte: "2026-11-19" }, the day it was asked. A test that kept that date would ask about renewals after November 19 forever. So a literal date becomes an offset from the day the question was asked, and the test resolves it against the day it runs: {{today}}, or {{today-30}} for a date thirty days earlier.

This is done only when the question names no date of its own. A question about “November 2026” or “11/19” means that day for good, and is skipped as dated rather than shifted.

A date more than about thirteen months from the day of asking is not shifted either. A query for an all-time total that starts at 2020-01-01 keeps that date as written, so its window does not creep forward with every run.

Docs