Skip to content
Get started
Back to writing
Guides9 min read

How to Test an AI Agent Before It Talks to Customers

How to test an AI agent before launch: write cases from real failures, mock tools, grade with checks, run on the real channel and replay real traffic.

Grainy film still of a dark craft rising into a cobalt sky on fanning beams of orange and blue light, with the words "How to test an AI agent before it talks to customers"

Short answer: To test an AI agent, write a small set of cases drawn from real conversations and failures, run each one as a real turn with mocked tools, and grade it with specific pass or fail checks instead of a single score. Run the cases several times and across models, then check them again on the real channel and against recorded production traffic before you ship a change.

In December 2023, people visiting a Chevrolet dealer's website found its new chatbot could be talked into almost anything. One of them got it to agree to sell a 2024 Tahoe for $1, as Business Insider reported. A month later, an update to the parcel company DPD's support bot made it swear at a customer and call DPD "the worst delivery firm in the world", and DPD disabled the AI part of its chat. In February 2024, a Canadian tribunal ordered Air Canada to pay a customer C$650.88 after its chatbot told him he could claim a bereavement fare after he flew.

None of these needed a new model to prevent. Each was one conversation that nobody had tried before customers did. That is what testing an agent is for: you have the bad conversation first, in private, and you find out what the agent does. It's also the step in launching a customer-facing AI agent that teams most often skip.

This guide covers how to do that. You'll learn:

  • Why testing an agent is different from testing ordinary software
  • How to turn real failures into test cases
  • How to grade an answer so a failure tells you what to fix
  • The eight steps to test an agent before and after it talks to customers

Why testing an agent is different

Ordinary code gives the same output for the same input. An agent doesn't. It reads a message, decides which tools to call, reads what they return and decides again. A change to the prompt or the model can fix one conversation and quietly break another.

Research shows how wide that gap can be. τ-bench, a 2024 benchmark that simulates customer service conversations with real policies and tools, found that state-of-the-art function-calling agents like GPT-4o succeeded on under half of its tasks. Run the same retail task eight times, and they got it right every time on under a quarter of them. The paper calls this metric pass^k: the chance that an agent succeeds on all k tries, not just one.

Anthropic's January 2026 guide, Demystifying evals for AI agents, describes what happens without tests. Teams catch problems only in production, "where fixing one failure creates others," and when users say the agent feels worse after a change, the team is "flying blind." OpenAI's evaluation best practices give the same warning a name: "vibe-based evals," where "it seems like it's working" is the whole test plan.

So a test for an agent has three jobs. It runs the agent the way production does. It checks behavior, not just wording. And it runs often enough to catch the agent being inconsistent.

What to test

Before writing any cases, decide what failure looks like for your agent. For most customer-facing agents it falls into four groups:

  • Wrong action. It called the wrong tool, called the right one twice, or skipped it and guessed.
  • Wrong words. The facts are right but the reply is wrong for the channel: Markdown that shows up as literal asterisks in a text message, or three bubbles where one would do.
  • Dishonesty under failure. A tool times out and the agent says it's done anyway, or promises to follow up and never schedules anything.
  • Saying yes to the wrong thing. The $1 Tahoe. The agent agrees to an offer, a refund or a policy it has no right to.

Each step below maps to one or more of these.

Step 1: Write down what a good answer looks like

A test needs a definition of passing. For each job your agent does, write the expected behavior as plain statements: "looks up the order before answering," "answers in one message," "never confirms a price it didn't get from the catalog." If two people on your team would grade the same reply differently, the statement isn't specific enough yet.

Anthropic's guide makes the same point: early on, evals force a team to say what success means. You'll reuse these statements as checks in Step 4.

Step 2: Turn real failures into test cases

You don't need hundreds of cases to start. Anthropic suggests that "20-50 simple tasks drawn from real failures is a great start," and that if you're already in production, your bug tracker and support queue are the place to look. OpenAI's advice is to "log everything" so you can mine your logs for good cases.

Good first cases come from:

  • Conversations a customer complained about
  • Questions your support team answers every day
  • Requests the agent should refuse, like a discount it can't give
  • Multi-turn conversations where the customer changes their mind halfway

On Keiki, the Eval Studio on an agent's Improve tab drafts a first suite from the agent's own spec: a tool case, a repeat case, a failing-tool case, a conversation-context case and channel-conformance cases. It can also suggest cases from your recent conversations, built from turns that went wrong. When Keiki's quality loop applies a fix for a recurring production failure, that failure is frozen as a regression case, so the suite remembers it happened.

Step 3: Mock the tools, including the ones that fail

A case should run as a real turn through your agent, with its real instructions and tools, but with tool results you control. That makes the run repeatable, and it lets you test the paths that are hard to trigger on purpose: a timeout, an error, an empty result.

Here is a case for a refund agent, in the shape the Eval Studio stores. The tool name is an example. Use your agent's own.

JSON
{
  "name": "Refund lookup fails twice, then works",
  "prompt": "Has my refund for order 4417 gone through?",
  "world": "mock",
  "channel": "sms",
  "stubs": {
    "get_refund_status": {
      "reply": "Order 4417: refunded, $42.00 to the original card",
      "failTimes": 2
    }
  },
  "checks": {
    "callsAtLeast": { "get_refund_status": 3 },
    "callsAtMost": { "get_refund_status": 4 },
    "mentions": ["42"],
    "absent": ["I'll get back to you"],
    "replies": 1,
    "noMarkup": true
  }
}

The stub fails the first two calls and then answers. The case passes only if the agent calls the tool at least three times (two failures, then the answer), stops before it loops, gives the real amount, doesn't stall, and answers in one plain-text message. A stub can also fail every call (fails) to test whether the agent admits it couldn't check, or add delay (latencyMs) to test what it does under a deadline.

Eval turns on Keiki run in a playground: outbound messages are blocked, usage isn't billed and the run stays out of your conversations. When you want to prove the real integrations work, switch the case to the live world, which runs the agent's actual capabilities. Keep mocks for comparing models, because a live run partly measures which candidate got the slower page.

Step 4: Grade with checks, not a single score

A score of 0.62 doesn't tell you what to fix. "Called the refund tool 0 times" does. Grade each case with separate checks, each with its own pass or fail and reason:

  • Tool use. Exact call counts, minimums, maximums and order.
  • Content. Text the reply must contain, and text it must not, like stall phrases or internal notes.
  • Format. No Markdown on channels that show it literally, no lists in a chat, and a limit on how many messages one turn may send.
  • Effects. State the turn had to leave behind, like a reminder it promised to schedule.
  • Judgment. For open-ended answers, a model grades the reply against criteria you write.

Use model grading only where a rule can't decide. OpenAI's guide recommends calibrating automated scoring against human judgment, so read a sample of judged replies yourself before you trust the grader.

In the Eval Studio, a case passes only when every check holds. The fractional score exists to rank candidates against each other, not to turn a failure into a pass.

Step 5: Run cases more than once, and across models

One passing run doesn't prove much when the agent is not deterministic. Run the suite again after every prompt or model change, and watch for cases that pass sometimes and fail other times. Those are the τ-bench problem in your own agent: the customer who gets the wrong answer on the third try.

The same suite is also how you pick a model. On Keiki, Find best model runs every case against a shortlist of up to 20 candidate models, so each one faces identical cases and checks. Ranking is in strict order: pass rate first, then mean score, judge score, real cost and median latency. A cheap model that fails a delegation case doesn't win on price. Keiki never switches the model for you. Use this model is the only thing that changes the agent.

Step 6: Test on the real channel

A reply can pass every check in process and still arrive wrong. The Markdown the model wrote reaches the customer as literal **, or one answer lands as three bubbles. Only a test over the real channel sees what a customer sees.

External-channel evals on Keiki grade the same cases on what was actually delivered. A driver sends each prompt as a real inbound message from a test contact's account, the agent answers over its real line, and the checks run against the messages as they arrived. For iMessage, the driver is a macOS app that drives Messages, and Keiki runs hosted drivers for some channels so you don't need a machine of your own. These are real turns with real tool calls and real usage, so send them from a designated test contact.

If you're building on iMessage, Building iMessage agents covers the channel rules these checks enforce.

Step 7: Replay real traffic before you change the prompt

Test cases are situations someone imagined. Before you ship a prompt change, you also want to know how it would have handled the conversations that actually happened.

Replay on Keiki saves finished production turns as tapes once you turn capture on for an agent: the prompt the turn ran with, the messages it answered and every tool call with its exact output. A replay run re-runs the current prompt, then your proposed one, then the current one again on part of the tapes to measure how often it disagrees with itself. A result counts as better or worse only if it clears that noise by a clear margin. Otherwise the run is inconclusive, which is the honest default.

Step 8: Keep testing after launch

Testing doesn't end at launch. Customers will find conversations your suite didn't. Two things help.

First, watch how customers react. Keiki's signals score each user message for mood and flag frustration, corrections, task failures and claims the agent made without a tool call to back them up. A sharp mood drop points to the reply that caused it, and that reply is your next test case.

Second, put a person in front of the actions a test can't fully cover. Approvals before refunds, and a handoff to your team when the agent is out of its depth, catch the conversation your suite missed before it reaches a tribunal. How to add human-in-the-loop to an iMessage agent shows how to set both up.

FAQ

What is an eval studio? An eval studio is a place to write test cases for an AI agent, run them as real turns and see which checks passed. Keiki's Eval Studio is on each agent's Improve tab. It runs cases with mocked or live tools, grades each check separately and can rank candidate models on the same cases.

What are AI agent evals? AI agent evals are automated tests for an agent's behavior. Each one gives the agent an input, often a multi-turn conversation, and grades what it did: which tools it called, what it said, and what state it left behind. Unlike a benchmark, your evals are built from your own agent's jobs and failures.

How do you test an AI agent? Define what a good answer looks like, write cases from real conversations and failures, run them with mocked tools and grade them with specific checks. Then run them repeatedly and across models, check them on the real channel, and replay recorded traffic before shipping a change.

How many test cases does an AI agent need? Fewer than most teams expect to start. Anthropic suggests 20 to 50 simple tasks drawn from real failures. Add a case every time the agent fails in production, so the same failure can't come back unnoticed.

Can I test an AI agent without sending real messages? Yes. Keiki runs eval turns in a playground where outbound messages are blocked and usage isn't billed. Use an external-channel run only when you need to see exactly what a customer would receive.

Sources