Eval Studio

The Eval Studio answers two questions in the dashboard:

1. Does my agent behave? It should use the right tool, in the right order, and actually answer instead of promising to answer later.
2. Which model should it run on? The same cases are replayed across candidate models, and ranked by what they did.

You will find it on an agent's Improve tab. The model dialog's Find best model button runs the same engine against a shortlist of candidates.

Both entry points work the same way. Run all cases and Find best model each create a run and open it at /improve/runs/:id, so the run has a URL that survives a reload and can be shared. The studio keeps showing the agent's latest run inline, with a link back to that page.

There is no CLI. Evals are product objects, stored per agent and edited in the dashboard.

How a case runs

A case runs as a real turn through the agent, with its own instructions, capabilities, delegation, step budgets and tracing. Only two things change:

• the model, so every candidate faces the same cases;
• the tool bodies, which are mocked.

Eval turns run in a playground. Outbound messages are blocked, usage is not billed, and the run stays out of your conversations, search, analytics and org-wide traces. The trace is kept, and every result links into the trace viewer.

Mock and live worlds

A mocked tool can also fail on purpose (fails), fail only the first few calls (failTimes), or take time (latencyMs). That is how the recovery cases and the "is it honest when a tool never works" cases are written.

Checks

Every check renders as its own pass or fail row, with the reason. A single score of 0.62 is not something you can act on.

Tool names are qualified by subagent (web_scraper.scrape_page) whenever the delegation layer is on, so a case can tell "the scraper subagent scraped" apart from "the orchestrator scraped it itself".

A case passes only when every check holds. The fractional score exists to rank candidates, not to call a failure a pass.

Code tasks

A code check asks whether the agent can actually fix a repo, which no substring or judge check can. It only runs in a live world, and turns the sandbox on for the case.

1. Before the turn, the case's repository is seeded into a fresh git repo in the case's sandbox, with one baseline commit.
2. The turn runs as usual. The case's prompt asks the agent to inspect the repo and reply with a unified diff.
3. Afterwards, the working tree is reset to that baseline, so only the patch the agent answered with counts, not anything it edited directly during the turn. The patch is applied, and the repo's test command runs. A zero exit passes the check.

A reply with no patch, a patch that does not apply, or a failing test each fail the check, with the reason shown inline.

Generating cases

When you open the studio for an agent that has no cases yet, Keiki drafts a suite from that agent's own spec. It writes a tool case, a repeat case, a compound lookup case, a self-heal case, a failing-tool case and a conversation-context case, plus channel-conformance and onboarding cases. Generation is a model call. There is no generic starter suite, and nothing is seeded when you read.

The cases are ordinary rows. Edit them, delete them, or add your own.

Model sweeps

Starting a run with candidates runs a sweep. Candidates and cases share one pool, with a bounded number of turns in flight overall and per vendor, so the measurement is the models and not a rate limiter. Sandbox cases have their own smaller limit.

Ranking is lexicographic, not blended: pass rate, then mean score, then judge score, then real cost, then median latency. A model that fails a delegation case is not saved by being cheap. Cost comes from the provider's own accounting of what was spent, not catalog pricing times tokens, and unknown cost stays unknown rather than showing as $0.

Keiki never picks a winner for you. The ranking has a Use this model button, and that is the only thing that changes the agent.

Limits

• A paid plan is required. A plan-less organization has its tools stripped and its model forced, so every candidate would score the same.
• One run per agent at a time, and at most 20 candidates per run.
• A run gives up after 15 minutes. Stop cancels it between cases.

API

These routes use a session cookie and are scoped to your organization. The driver surface under /api/v1/evals/channel/* is separate. See External-channel evals.