How to Benchmark Language Models for Your Own Support Workload

Almost every model comparison you'll find has the same two problems. The numbers were true for about six weeks — providers ship new versions and change pricing constantly — and they were measured on a task that isn't yours.

Rather than quote figures that will be wrong by the time you read this, here's the method for producing numbers that are correct for your workload, on the day you need to decide. It takes about a day and it stays useful, because you can re-run it whenever a new model appears.

Measure four things

Time to first token (TTFT). How long until the reply starts appearing. On a streaming interface this is what users perceive as speed — total generation time matters far less than when text starts moving.

For a non-streaming channel like WhatsApp, where the message arrives complete, total response time is what matters instead. Measure the one that matches your channel.

Cost per conversation. Not per million tokens. Providers quote per-token pricing, but your actual unit is a conversation, and conversations vary enormously by how much history and context you send. Compute it from your own traffic.

Answer quality on your questions. The number that matters most and the only one requiring judgment.

Failure rate. How often the model returns malformed output, refuses inappropriately, or produces something you'd have to escalate. For a structured-output pipeline this can matter more than raw quality.

Build the test set first

This is the part that determines whether the exercise is worth anything.

Take 50–100 real messages from your actual inbox. Not invented examples — real ones, with their typos, their ambiguity, their code-switching, their missing context. Strip identifying details.

Composed test questions are systematically cleaner than real traffic in a way that hides exactly the failures you care about. This is the single most common flaw in home-grown model evaluations.

Include the awkward cases deliberately:

• Questions your documentation doesn't answer
• Ambiguous requests that could mean two things
• Messages in every language your customers use, including mixed-language ones
• Deliberately out-of-scope questions, to check whether the model escalates or invents
• The five questions you get most often

Write the expected outcome for each — not the exact wording, but what a good answer contains and whether it should escalate. Without this you're judging on vibes and you'll pick whichever output reads most confidently, which is not the same as most correct.

Measuring latency and cost

Run each test message through each candidate model, several times, and record per call.

async function measure(model: string, message: string) {
  const started = performance.now();
  const response = await callModel(model, message);
  const finished = performance.now();

  return {
    model,
    totalMs: finished - started,
    inputTokens: response.usage.inputTokens,
    outputTokens: response.usage.outputTokens,
  };
}

Points that change the answer:

Report percentiles, not averages. p50 and p95. The average hides the slow tail, and the slow tail is what users complain about. A model with a good average and an occasional eight-second response feels worse than a consistently mediocre one.

Run from where your code runs. Latency measured from your laptop is not latency from your production region. Run it from the same environment.

Repeat across the day. Provider latency varies with load. A single run at 3am is not representative.

Include your real prompt and context. A system prompt plus conversation history is dramatically more input than a bare question, and input volume drives both cost and latency. Benchmarking a naked question tells you nothing about your actual bill.

Then compute cost per conversation from your measured token counts and the provider's current published rates. Look those up on the day — this is exactly the number that goes stale.

Judging quality

The part that can't be automated away, and the part worth spending time on.

Blind the outputs. Strip the model names, shuffle the order, then rate. Knowing which model produced an answer biases the judgment substantially — including yours.

Score on specific criteria, not overall preference:

• Factually correct given your knowledge base
• Appropriately escalated when it should have been
• Right length for the channel
• Correct tone and register
• Right language and script

Have someone who knows the domain do the rating. For a multilingual product, a native speaker — checking whether it sounds natural, not just whether it's grammatically valid.

Consider an LLM-as-judge for scale, with caution. Useful for a first pass over hundreds of outputs, unreliable as the final word, and it exhibits known biases toward longer and more confident-sounding answers. Sample-check its judgments against human ratings before trusting it.

Reading the results

Fill in your own numbers:

Quality differences among current frontier-tier models are usually smaller than expected for straightforward support tasks. Where you'll often see meaningful separation is at the edges — ambiguous questions, non-English input, knowing when to say "I don't know."

If quality is close, the decision is cost and latency, and those differences can be large.

Weight by your actual volume. A cost difference that's trivial at a hundred conversations a month is decisive at fifty thousand.

Keep it re-runnable

The important output of this exercise isn't the answer — it's the harness.

Save the test set, the scripts, and the scoring rubric. When a new model ships, you re-run it in an hour and get a current answer instead of reading someone else's benchmark from last quarter.

Design your code so the model is a configuration value, not a hardcoded string. A single environment variable and a provider-agnostic call site means switching is a deploy rather than a project. Given how fast pricing and capability move, that flexibility is worth more than picking the optimal model today.

Want this done for your workload?

We build AI systems with model selection driven by measurement on real traffic, and structured so switching providers is trivial. Talk to us about AI chatbot development.