External-channel evals

An ordinary eval run asks "would the agent behave if a message arrived". It never leaves Keiki. The channel a case names is applied as a rule set, and the reply is graded in process.

An external-channel run grades the same cases on what a real contact actually received. The prompt arrives as a real inbound message over a real gateway, the agent answers over that gateway, and the checks run against the messages as delivered. How many bubbles arrived, whether the Markdown the model wrote reached the user as literal **, whether the answer came at all.

Nothing about the agent's production path changes. Keiki does not send the prompt. A driver does, from the account a real contact would use, and reports the transcript back for grading. The first driver is a macOS app that drives Messages.

plaintext

 driver (macOS)                        platform
 ─────────────                         ────────
 start a run          ──────────────▶  POST /runs
 reset conversation   ──────────────▶  POST /runs/:id/reset
 send iMessage ─▶ agent's line ─────▶  real inbound → steering → turn
 tail chat.db ◀─ Messages.app  ◀─────  real outbound over the gateway
 report transcript    ──────────────▶  POST /runs/:id/cases/:caseId → graded
 finish               ──────────────▶  POST /runs/:id/finish

The macOS driver's setup check confirms, in order, that the Messages log is readable and queryable (Full Disk Access), that the terminal may control Messages, that an iMessage service is available, and that Keiki knows the agent, driver, channel and target. A real send is the proof that it is set up.

Keiki also runs hosted drivers for some channels, so you can grade those without a machine of your own holding the sending account. Each hosted driver uses a shared Keiki line or inbox, and runs on it are serialized so conversations cannot interleave.

What is graded

The same checks run. Only the source of the observation changes.

	Internal run	External run
`replies`	The reply-tool calls the stubs counted	The messages that actually arrived on the channel
text	The turn's own answer	The delivered messages, joined
`completes`	The turn produced text	Something arrived before the collection window closed
`calls` / `callsAtLeast` / `callsAtMost` / `order`	In-process instrumentation	The real turn's trace, joined by conversation and time window
`mentions` / `absent` / `noMarkup` / `noLists` / `judge`	On the turn's text	On the delivered text
model	Swapped per candidate	The agent's own. A real turn cannot be told which model to use

Because the model cannot be swapped, an external run is never a sweep. It has exactly one candidate, the agent's configured model, and no ranking to read. It answers "does my agent behave on this channel", not "which model".

Which cases can run

A case that cannot run is excluded, with its reason returned when the run is created. It is not failed, because a case that cannot run is not a case that failed.

Mocked tools. Out here the tool bodies are the agent's real capabilities, so there is nothing to substitute. Cases whose whole point is a canned tool belong to internal runs.
Code checks. The patch is graded through the case's own eval sandbox, which an external turn does not have.
Seeded state and effect checks. Both need the eval harness rather than a live turn.
A different channel. A case named slack is not evidence about iMessage.

Everything else runs, including the channel-conformance cases. That is where an external run earns its keep. Those cases are capability-free, and replies, noMarkup and noLists are exactly the checks that used to be graded against an intercepted reply tool instead of a delivered message.

A case with history is replayed by sending its user turns as real messages before the graded one, so the conversation the agent answers in is one it really had. The conversation is reset between cases, so one case's messages never become the next one's context.

What it costs

An external run is a real turn on a real line: real capability calls, real provider spend, real usage counters, and real messages to whichever handle the driver sends from. Use a designated test contact. Between cases the driver asks Keiki to clear that conversation, but usage already metered cannot be un-metered.

A plan is required, for the same reason internal runs need one. A plan-less organization has its tools stripped and its model forced, so every check about tool use would measure the sandbox rather than the agent.

Driver API

The base path is /api/v1/evals/channel, and it uses a developer API key (Authorization: Bearer sk_…). An agent-bound key scopes every call to its agent automatically. An org-wide key names the agent with agent_id, in the query for requests without a body, and in the JSON body otherwise.

`GET /ping?driver=imessage`

A setup check. It returns { agent_id, driver, channel, target } and nothing else. It never claims a run.

`POST /runs`

Start a run. Body: { driver, case_ids? }, where driver is imessage, sms, slack, telegram, whatsapp, email or x.

JSON

{
  "run_id": "evalrun_…",
  "driver": "imessage",
  "target": { "channel": "sms", "handle": "+15550001111" },
  "cases": [
    {
      "id": "…",
      "name": "SMS: answers in one message",
      "prompt": "…",
      "history": [{ "role": "user", "content": "…" }]
    }
  ],
  "excluded": [{ "id": "…", "name": "Retries a failing tool", "reason": "mocked tools" }],
  "quiet_ms": 8000,
  "timeout_ms": 180000
}

target.handle is the agent's own address on that channel, which is what the driver sends to. Returns 409 when a channel run for the agent is already in flight, or when the agent has no active line on the channel. Returns 400 when no case is eligible.

`POST /runs/:runId/cases/:caseId`

Report one case and get its grade back.

JSON

{
  "identity": "+15559998888",
  "sent_at": "2026-08-25T03:00:00.000Z",
  "ended_at": "2026-08-25T03:00:24.000Z",
  "messages": [{ "text": "Sure, start with…", "at": "2026-08-25T03:00:12.000Z" }],
  "error": null
}

identity is the handle the driver sent from. It is how Keiki finds the turn's trace, so the tool-use checks depend on it being the address the agent saw. messages are the agent's messages in arrival order, because the count is the replies check. error reports a driver-side failure, which fails the case with that reason instead of scoring an empty transcript as a silent agent.

The response carries the graded result, with passed, score, judge_score, trace_id and checks as { label, passed, detail? } rows, so a driver can print it as it goes. Reporting a case again overwrites its result rather than adding a second one, so a retry after a network failure does not double-count.

`POST /runs/:runId/reset`

Body { identity }. Clears the driver conversation so the next case starts from nothing. Only the identity the run is being driven from, and only while the run is running.

`POST /runs/:runId/finish`

Body { identity, error? }. Totals the run and closes it. With no error it completes, and an error marks it failed. A run that is never finished times out on its own. It is scoped to the identity, so a second driver sharing the API key cannot end a run it is not driving.

Other channels

The protocol is channel-agnostic by design. A driver is anything that can send as a user and observe what comes back, and Keiki's half (eligibility, trace joining, grading and storage) is the same for Slack, Telegram, WhatsApp and email. What differs is who holds the sending account. iMessage has to be a client on a Mac. The others can be driven server-side.

On this page