Replay

Replay answers a question the Eval Studio cannot: would this change have behaved worse on the turns that actually happened?

An eval case is a situation somebody imagined. A tape is a finished turn from production, frozen in place: the prompt it really ran with, the messages it answered, and every tool call it made with the exact output the world gave back. Replaying a tape runs the agent over that frozen world with your proposed prompt in place of the live one, so you compare against real traffic instead of a guess.

Turn it on

Capture is off by default, and you opt in per agent. When it is on, finished turns that passed the agent's quality review are saved as tapes. A turn that could not be reproduced is skipped rather than stored.

• A tape that is not pinned expires on its own. Pinning it into a corpus clears the expiry, so the evidence behind a decision is still there when someone reads it later.
• A tape is deleted only when you unpin it, and only once no corpus holds it.
• Each agent's automatic production corpus stops at a size ceiling. That ceiling is the cap on what a gate can cost.

Open an agent, go to Replay, and you can turn capture on, build corpora from its conversations, and start runs.

How a run decides

A run replays the corpus three ways. First it re-runs the prompt that is in production now, so the comparison is replay against replay rather than against the original reply. Then it runs your proposed prompt. Then it runs the production prompt once more on part of the corpus, to measure how often it disagrees with itself.

Each tape gets a verdict: unchanged, changed for the better, changed for the worse, unclear, diverged, or errored. The run as a whole concludes improved, regressed or inconclusive. Inconclusive is the default. A result has to clear the prompt's own run-to-run noise by a clear margin before it counts as an improvement.

A replay only answers tool calls the tape recorded. If the tape has no recording for a call, or the same tool is called with different arguments, that tape is marked as a divergence and left out of the comparison. Scoring a turn on a made-up tool output would measure nothing. Delivery wording is the exception: changing the words an agent sends is exactly what a prompt change is allowed to do.

The gate

When Keiki's quality pass proposes a fix, the change waits for a queued replay run if the agent has a production corpus worth gating on. Replay is a veto, never an approval. A change ships unless the run concludes that it regressed. An inconclusive, failed or cancelled run produced no evidence, so it does not hold a fix back.

API

All routes use a session and are scoped to your organization.