Evaluate the GraphJin Agent

An agent can return the right number for the wrong reason.

Imagine asking for total monthly recurring revenue. The agent fetches the first 10 subscriptions, adds those rows itself, and happens to land on the correct number because the demo account is small. The answer looks perfect. In production, the same method silently ignores page 2.

A normal answer check sees the right number and passes it. GraphJin Eval checks both the answer and how the agent obtained it, so this run fails until the database performs the complete aggregation.

That is the purpose of GraphJin Eval: prove that a model, prompt, skill, or GraphJin change still produces correct, governed answers on your catalog and your data.

What GraphJin checks

Every task can check five different parts of an agent run:

Ground truth, method, safety, and behavior decide whether a task passes. Efficiency is advisory: it helps you find expensive or wandering runs without turning a slow but correct answer into a false regression.

The evaluation loop

You do not write expected answers by hand. GraphJin samples the caller-visible catalog, creates tasks, and verifies each hidden GraphQL oracle before the agent receives any prompt. An oracle is simply a trusted read query plus the rule for extracting its expected value.

Each task runs three times because model output varies. Two passing runs form a majority. Safety is stricter: one unsafe run fails the task even when the other two pass.

flowchart LR
  A["Your catalog and data"] --> B["Verified tasks<br/>with hidden read-only oracles"]
  B --> C["Three independent<br/>agent runs per task"]
  C --> D["Ground truth · method<br/>safety · behavior · efficiency"]
  D --> E["Compare with the<br/>trusted baseline"]
  E -->|No regression| F["Accept the change"]
  E -->|Confirmed regression| G["Block and investigate"]

Before spending model tokens, GraphJin runs every oracle. If one no longer compiles, executes, or extracts a value, the suite stops with exit code 2 and sends no evaluated-agent traffic. A broken check is not reported as a model regression.

Your first evaluation in ten minutes

The built-in SaaS Ops demo uses SQLite in-process. It needs no Docker, database setup, or running GraphJin server.

1. Configure a model

The evaluated agent uses the provider configured under agent. For the demo, put one supported key in ./.env:

OPENAI_API_KEY=...
# or ANTHROPIC_API_KEY=...
# or GOOGLE_API_KEY=...

Creating and verifying the suite does not ask the model to answer its tasks. Running or extending the suite does. GraphJin prints the maximum expected call count before provider-backed traffic begins and asks for confirmation in an interactive terminal.

2. Create a verified demo suite

graphjin eval create --demo

GraphJin extracts or reuses the bundled SaaS Ops demo, starts an embedded loopback instance, inspects its public catalog, and writes eval/suite.yml under the selected demo project. Local and demo evaluation instances force the agent read-only and disable background watch work.

The default generator asks for at most 24 deterministic, deduplicated tasks. Small schemas may produce fewer tasks when they do not have enough safe, verifiable shapes. Every saved task already has a working oracle.

Do not edit eval/suite.yml by hand. Task IDs are content hashes, and the CLI protects the oracle and schema invariants that make comparisons meaningful.

3. Inspect what exists

graphjin eval --demo

The status shows the suite path, task count, catalog fingerprint, and whether a baseline exists. Immediately after create, seeing Baseline: none is normal.

4. Run the agent

graphjin eval run --demo

The traffic preview separates work already paid for from work still authorized:

Provider traffic preview: fresh run; <initial> initial slots remain, up to
<confirmation> confirmation slots, and at most <attempts> provider attempts
including one transient retry per pending slot.

Approve the prompt only when you are comfortable with that provider traffic. In a non-interactive shell, GraphJin refuses to continue unless you explicitly pass --yes.

The concise result has two lines:

Run <run-id>: recall <value>, ground truth <value>, method <value>, safety <value>
pass@3 <value>, pass^3 <value>; accepted=<true-or-false>

With no previous baseline, suite validity and perfect safety are the gates. The first safe run is promoted automatically at its observed recall and becomes the reference for later changes. This lets a real team start measuring from where its agent is today instead of requiring a stochastic suite to reach 100% first. Recall below 0.90 produces a prominent quality warning, not a failed command.

GraphJin writes a private run manifest before the first provider request and checkpoints it after every attempt and finalized slot. If the terminal or process stops, rerunning the identical command automatically resumes the newest strictly compatible incomplete run. Completed model failures count as finished slots; failed provider attempts do not.

Use a particular checkpoint or deliberately start over like this:

graphjin eval run --demo --resume <run-id>
graphjin eval run --demo --restart

--resume and --restart cannot be combined. graphjin eval --demo lists incomplete runs, progress, model, last update, and exact resume/restart commands. Compatibility is intentionally strict, so a changed suite, oracle values, dataset, binary/server fingerprint, provider/model, target, baseline, seed, or promotion intent starts a fresh run instead of silently mixing evidence.

5. Read the shareable report

Reports are written under the evaluated project:

.graphjin-evals/reports/<run-id>.json

Use --json when another program needs the complete shareable report:

graphjin eval run --demo --yes --json > graphjin-eval-report.json

The report contains verdicts, metrics, failure categories, fingerprints, model and prompt-registry provenance, and the acceptance decision. Provenance also includes binary_fingerprint, the exact CLI executable SHA-256. Compare it before treating two score changes as a same-build A/B. The report does not contain the task prompts, model answers, database rows, executed queries, headers, token contents, or secrets. It does include aggregate token-usage counts so you can compare cost and efficiency.

GraphJin reads usage from Ax after every agent run. The CLI and report show two views: finalized usage covers the episodes used for quality metrics, while actual provider usage also includes failed attempts and retries so the cost number stays honest. Usage from successful model calls is preserved even when the agent later ends with an error, including an actor-step exhaustion.

The report says provider usage accounting is complete when every provider attempt returned usage. A timeout or transport failure may return no usage; in that case provider_usage.complete is false, unknown_attempts says how many attempts are missing, and the recorded token total is a lower bound. GraphJin does not silently treat those unknown attempts as zero.

Against a compatible baseline, Eval shows whether total tokens and tokens per episode went up or down, with both the absolute and percentage change. Reports use graphjin.eval.report/v3 and a separate usage_accounting_version. Token percentages are disabled when the accounting version, provider, model, configured max_steps, suite shape, or finalized episode count differs, or when either run has unknown provider usage. Quality and safety comparisons are still valid.

Read the result without being a statistician

The overall reward is useful for optimization experiments, but the release gate uses the explicit checks above. A high average reward cannot excuse a safety failure or wrong answer.

Add a question your business actually cares about

Generated coverage is a starting point. The most valuable suite also contains the questions that would hurt if the agent answered them incorrectly.

graphjin eval add "Which customers are at churn risk?" --demo

GraphJin asks the configured model to interpret the question using the public catalog. It then shows you:

1. the plain-language interpretation;
2. the value produced by the proposed hidden oracle; and
3. the task ID that will represent this exact definition.

If the wording is ambiguous, the CLI asks for clarification instead of guessing a field, threshold, or time window. If it is clear, the task is saved only after you approve the interpretation and observed oracle value.

The command can incur one agent-authoring episode plus one or two read-only oracle queries. In non-interactive use, add --yes only after reviewing that traffic boundary:

graphjin eval add "Which customers are at churn risk?" --demo --yes --json

New task IDs are not in the old baseline, so they are advisory during ordinary comparison. Promote a new baseline only after the expanded suite passes and you intend those tasks to become release gates.

If an executable task is wrong for your business, remove it through the CLI instead of hand-editing the suite:

graphjin eval rm <task-id> --demo

GraphJin shows the task and asks for confirmation. Non-interactive use requires --yes. The normalized suite is saved and validated again; the last remaining task cannot be removed because a valid suite must contain at least one task.

Use the baseline as a release decision

The normal lifecycle is:

flowchart LR
  A["Create and verify suite"] --> B["Establish passing baseline"]
  B --> C["Change model, prompt,<br/>skills, or GraphJin"]
  C --> D["Run the same task IDs"]
  D -->|Accepted| E["Ship or continue testing"]
  D -->|Regression| F["Inspect failure and episodes"]
  F --> C
  E -->|Intentional improvement| G["Deliberately promote<br/>a new baseline"]
  G --> C

An ordinary run compares only task IDs present in both reports. This prevents a newly added task from masquerading as a regression before you deliberately make it part of the contract.

When an initial three-run result would regress, GraphJin performs one fresh three-run confirmation. Exit code 1 is returned only when the hard failure is confirmed (or safety fails). This absorbs one flaky sample without hiding a repeatable problem.

After an intentional accepted change, promote deliberately:

graphjin eval baseline --demo

Baseline promotion requires a hard pass: the suite must be valid, safety must be perfect, and no confirmed regression may remain. Recall does not need to be 1.0; a value below 0.90 remains visible as a quality warning.

Diagnose a failed run

Start with the process exit code, then inspect the report’s dominant failure category.

For deeper diagnosis, rerun with --debug. It prints the local episode paths:

graphjin eval run --demo --debug

Episodes contain the full prompt, answer, action trail, executed queries, oracle query and result, usage, timing, and seeds. Keep them private. Share the report unless someone explicitly needs the sensitive trajectory. Failed execution summaries include stable error_codes, recovery_codes, and recovery_tool fields, which make wrong-dialect and repair loops visible without relying only on an error count.

Interrupted and environment-failed provider calls are written separately under .graphjin-evals/attempts/<run-id>/. They are private too. A partial report has only status, progress, provenance, usage, and a safe environment code—never partial recall, task verdicts, or a baseline comparison.

GraphJin keeps the global agent limit at eight actor steps. If a model repeats an identical successful query, GraphJin does not hit the database again: it returns the cached governed result with a completion instruction and gives the model one grace turn to answer. A second repeated call can be completed from that evidence. If the limit is otherwise exhausted, the episode records agent_actor_steps_exhausted, keeps usage from the calls that already happened, and Eval classifies it as runaway. Raise a task-specific limit only when its trace shows distinct productive progress on every turn.

Choose the right target

GraphJin records a dataset fingerprint with each run. A catalog hash proves the visible schema shape, but it cannot prove that live row values stayed unchanged. Demo datasets can prove equality with the same catalog hash, data anchor, and deterministic seed manifest.

Local and remote projects get a second proof automatically. After oracle preflight, GraphJin computes one aggregate oracle_value_hash from the suite fingerprint and every resolved value and dimension in task-ID order. Matching digests mean the effective expected results for that exact suite are unchanged, so ground-truth regressions stay enabled without placing individual oracle values in the shareable report.

When both the dataset fingerprint and aggregate oracle hash differ, GraphJin says so and compares the intersecting tasks by method correctness instead. It still gates safety and behavior. This avoids reporting ordinary business-data movement as a model regression without pretending the old expected value is current.

Cost and privacy boundaries

Provider-backed evaluation is intentionally explicit:

• create uses catalog and read-only oracle traffic, not evaluated-agent runs.
• add previews its expected model and oracle traffic before continuing.
• run, baseline, and bench show reused slots, remaining initial slots, possible confirmation slots, and the retry-inclusive provider-attempt ceiling.
• interactive commands ask before spending model traffic;
• non-interactive commands require --yes.

Local state has owner-only permissions:

.graphjin-evals/
  baseline.json
  reports/<run-id>.json
  episodes/<run-id>/<task>-<repeat>.json
  attempts/<run-id>/<task>-attempt-<number>.json
  runs/<run-id>.json
  locks/<run-id>.lock

The baseline and reports use the sanitized report schema and are suitable for controlled sharing. Episodes and attempts are private trajectories. Private files may contain prompts, answers, rows, and queries, but even private files never contain credentials: GraphJin recursively redacts provider URL keys, authorization values, the configured provider secret, and recognizable key patterns before writing. Never upload the episodes/ or attempts/ directory as a routine CI artifact.

Add a minimal CI gate

Establish the baseline deliberately outside the candidate run:

graphjin eval baseline --yes

# The baseline is a sanitized report. Copy it; never edit it.
cp .graphjin-evals/baseline.json eval/baseline.json
git add eval/suite.yml eval/baseline.json

Then restore that known baseline before the CI run. Failing when the snapshot is missing is important: otherwise a first passing candidate could automatically become its own baseline.

set -eu
test -f eval/baseline.json
mkdir -p .graphjin-evals
cp eval/baseline.json .graphjin-evals/baseline.json

graphjin eval run --restart --yes --json > graphjin-eval-report.json

CI uses --restart because each candidate gate is intentionally fresh; local interactive work normally benefits from automatic resumption.

Configure the provider key as a CI secret. For a remote target, also configure the GraphJin CLI identity or set GRAPHJIN_EVAL_TOKEN, then add --remote.

Upload only graphjin-eval-report.json or .graphjin-evals/reports/. Keep .graphjin-evals/episodes/ and .graphjin-evals/attempts/ private. Exit 1 should block the candidate; exits 2, 3, and 130 should route to suite, infrastructure, or interrupted-job handling rather than be reported as model-quality regressions.

Run the frontier benchmark

Use the regular suite for release gates. Use bench when you want a broader, seeded distribution across discovery, aggregation, ranking, time windows, relationships, governed saved queries, annotations, permissions, and refusals:

graphjin eval bench --demo --scale 100 --seed 23

The benchmark still verifies every generated oracle before evaluated-agent traffic. It reports pass@3, pass³, bootstrap recall intervals, and metrics per difficulty tier. Fix the seed when comparing candidates so task generation is reproducible.

At 100 tasks and three initial repeats, the preview can be up to 300 provider-backed runs, plus confirmation runs for candidate regressions. Treat that as an intentional benchmark budget, not the default getting-started path.

The RL-ready boundary

Versioned tasks, private episodes, reward vectors, action trails, seeds, provenance, and environment interfaces make the evaluation engine usable as the foundation of a future rollout collector. That does not mean GraphJin Eval v1 is a reinforcement-learning trainer.

V1 is deliberately:

• read-only;
• sequential;
• single-turn;
• without mutation reset semantics; and
• without parallel instance-pool collection or trainer integration.

The reserved turns and mutation fields are rejected rather than silently approximated. InstancePool and ResettableInstance are future seams for safe, resettable rollout workers. See the compact technical reference when building against those interfaces.

The next milestone is environment diversity, not a more complicated reward: generate many distinct schemas and deterministic datasets, then run the existing task generator over each catalog. After that comes rollout throughput through per-worker SQLite file copies and ResettableInstance.Reset. Mutation tasks, multi-turn curricula, a headless batch API, and more process rewards come only after those two foundations.

Keep going

• Learn how the evaluated server-side agent discovers and executes governed answers.
• Add questions from the SaaS Ops demo to your suite.
• Use the installed graphjin-eval skill to let a supported coding agent run and diagnose the CLI without editing hidden oracles.