LLM Cost Optimization: Cutting Cost Per AI Interview

For most of a year we tracked model spend the way everybody does: one number per month, divided by requests. It told us nothing. The month the bill jumped, nothing in the per-request average had moved — we were simply running more interviews, and a few of them were running very long.

LLM cost optimization only started working for us when we changed the unit. Not cost per token, not cost per request: cost per completed interview, broken down by the stage that spent it. This is what that re-framing found, which levers paid, and how we check that a cheaper configuration still conducts a real interview.

Start LLM cost optimization at the unit, not the token

LLM cost optimization has a default unit, and the default unit is wrong. A token price is an input to a decision, never the decision. Anthropic's own cost guidance puts it bluntly: compare models on cost per completed task, not per token, because you pay for completed tasks. The same page makes the point that decides most of these arguments — "the bill is decided by the tasks the cheaper model fails, because a failed task still bills its tokens, then the retry, then whatever the failure costs downstream."

That is not theoretical for an interview product. If the interviewer loses the thread and the candidate restarts, we did not save money by running a cheaper model. We paid for a broken session, then paid again for the replacement, and lost the candidate.

What one interview actually spends money on

A Taqari interview is not one model call. It is a short pipeline, and the stages have wildly different cost profiles because they have wildly different latency requirements:

• Session mint — the backend issues a short-lived token with the interviewer prompt and tool declarations baked in. No model call, no model cost.
• Live voice interview — a realtime audio session for the length of the interview. This is the expensive stage, and the only one a human is waiting on.
• Code-run feedback — every time the candidate runs their solution, the judge result goes back into the conversation as text.
• Report generation — written after the candidate has left the page.
• Rubric scoring — structured scores over the finished transcript, also after the fact.

Three of those five run when nobody is waiting. That is the whole opportunity, and it is invisible if you only look at a monthly total.

Give each stage the cheapest model its latency allows

The useful question per stage is not "is this model good enough" but "how long may this stage take". Latency, not capability, is what forces you onto the expensive option.

The live stage: you are buying latency, so spend elsewhere

A spoken follow-up question has to land inside a natural pause. That rules out a slower, cheaper pipeline, and it rules out stitching transcription, text generation and speech synthesis together yourself. We pay the realtime premium here because the product is the conversation. Every other saving exists to fund it.

Everything after the call is a different purchasing decision

Once the tab closes, the constraint evaporates. The report and the scores can take ten seconds or ten minutes.

Report generation

The report reads a finished transcript and writes prose. There is no tool use, no multi-turn reasoning, no live audio. On the current published Anthropic price list, Claude Opus 5.5 is $4 per million input tokens and $20 per million output, while Claude Haiku 5.5 is $0.10 and $0.50 per million for prompts up to 100,000 tokens. That is a 40x spread on the same job. If the cheap model writes a report that passes review, paying 40x for prose is not a quality decision, it is an unexamined default.

Haiku's pricing does have a cliff worth knowing before you route long transcripts to it — a prompt over 100,000 tokens is billed at the higher rate for the whole request, which we wrote up separately in the 100K token cliff in Haiku 5.5 pricing.

Rubric scoring

Scoring is the stage where cheap models are most tempting and most dangerous, because a wrong score is invisible. It looks like a number either way. Our defence is that scoring is already pinned to a structured rubric rather than a free-form "rate this out of ten", which is what makes a model swap measurable at all — the reasoning behind that design is in our write-up on scoring answers reproducibly.

Because scoring has no latency requirement at all, it is also the one stage that can take the asynchronous discount. Anthropic's Batch API carries "a 50 percent discount on both input and output tokens", and that discount stacks with caching. A stage you were already willing to run overnight should not be paying interactive prices.

Cap output, because output is where the money is

Input and output are not priced alike. Across the current Claude lineup, output runs about five times input — $4 against $20 for Opus 5.5, $2 against $10 for Sonnet 5.5. A rambling model is not just a worse interviewer, it is a more expensive one.

So the live session config sets a hard ceiling rather than hoping the prompt keeps things short:

generationConfig: {
  responseModalities: ['AUDIO'],
  maxOutputTokens: 1024,
  temperature: 0.7,
  topP: 0.8,
}

1024 tokens is roughly a minute of speech. An interviewer that cannot exceed that per turn is forced into the behaviour we wanted anyway: ask a question, stop, listen. The cost ceiling and the product requirement turned out to be the same constraint, which is the most pleasant kind of optimisation there is.

The same instinct applies to the written stages. Ask for the fields you will actually render. A report schema with eight populated sections you display and three you do not is a 27% overcharge on the most expensive token class you buy.

Prompt caching only pays if the prefix stops moving

Of every LLM cost optimization lever on the list, prompt caching has the highest leverage for anything with a long, stable system prompt — and an interviewer prompt is exactly that. The rules are worth reading properly rather than assuming, because the failure mode is silent: you pay the write premium forever and never get a hit.

From Anthropic's prompt caching documentation:

• A five-minute cache write costs 1.25x base input; a one-hour write costs 2x.
• A cache hit costs 0.1x base input, with per-model exceptions down to 0.05x on Opus 5.5 and Sonnet 5.5.
• "Cache hits require 100 percent identical prompt segments, including all text and images up to and including the block marked with cache control."
• Shorter prompts cannot be cached at all — the minimum is 512 tokens on the current models — and "no error is returned" when you miss it.

The break-even falls out of those multipliers. At 1.25x to write and 0.1x to read, one read repays the write. At 2x for the one-hour TTL you need two. So the question is never "should we cache" but "will this prefix be read back at least twice before it changes".

What quietly breaks the prefix

Tool definitions

Tool names, descriptions and parameter schemas sit at the top of the hierarchy, so they invalidate everything. This is the argument for declaring tools server-side, once, and baking them into the session rather than assembling them per request from whatever the client thinks it supports. Our interview tools are declared in one place on the backend and the browser only holds the handlers — a split we chose for maintainability and which turns out to be the thing that keeps the cache prefix stable.

Images, thinking and toggles

Less obvious: "adding/removing images anywhere in the prompt affects message blocks", and thinking or effort parameters always invalidate messages. A debugging flag you flip per request, a timestamp you helpfully interpolate into the system prompt, a candidate name injected at position zero — each is a 100% cache miss rate wearing a disguise. Interpolate volatile values at the end of the prompt, after the breakpoint, or not at all.

One more mechanical limit worth designing around: you get up to four cache breakpoints, and a read "checks at most 20 positions for each breakpoint". A conversation that grows one message per turn will walk off the end of that window unless you place a breakpoint deliberately.

Trim context without losing the interview

The transcript is the context, and in a live session it grows every turn. Trimming it is where we found the savings that did not involve downgrading anything.

Summarise tool results instead of pasting them

When a candidate runs their code, the judge returns per-case results that can run to kilobytes of stdout, stderr and compiler noise. The interviewer does not need kilobytes. It needs to know whether the solution passed and, if not, what the first few failures look like. So the client formats a compact message rather than forwarding the payload, with explicit caps:

/** Maximum failing cases and characters per case in [TEST_RESULTS]. */
const MAX_FAILURES_REPORTED = 5;
const MAX_DETAIL_CHARS = 300;

Five failures is enough for an interviewer to say "your edge case at zero is wrong". The sixth failure tells it nothing new and costs input tokens on every subsequent turn, because once a message is in the conversation you keep paying for it. The summary also carries a line the model can act on — passed N of M, then the truncated failures — instead of making it parse judge output.

Send each state change exactly once

Two small guards do more for cost than they look like they should. The first is that the editor contents are re-synced to the model once after a whole batch of tool calls, not after each call — a model that writes code, switches language and inserts a helper in one batch produces one state message, not three. The second is a dedupe ref:

// Tell the model each run's outcome exactly once per runId.
if (lastReportedRunRef.current === leetCompilerRunId) return;
lastReportedRunRef.current = leetCompilerRunId;

Without it, any re-render or reconnect that re-evaluates the effect re-sends a result the model already has. It is a correctness fix first — a model told twice that the tests passed will sometimes congratulate the candidate twice — and a cost fix second. Most of our duplicated-token bugs were shaped like this, not like a badly chosen model.

Do not bill silence

In an audio session, the thing you are tokenising is sound, and a candidate thinking quietly for twenty seconds is still sound. The realtime input config decides whether that counts:

realtimeInputConfig: {
  automaticActivityDetection: {
    endOfSpeechSensitivity: 'END_SENSITIVITY_HIGH',
    prefixPaddingMs: 150,
    silenceDurationMs: 2000,
  },
  activityHandling: 'START_OF_ACTIVITY_INTERRUPTS',
  turnCoverage: 'TURN_INCLUDES_ONLY_ACTIVITY',
}

The last line is the one that matters here: the turn covers detected activity only, not the whole wall-clock window. In a coding interview, where long silences are the candidate working and are completely normal, the difference between billing the conversation and billing the clock is not marginal. The two-second silence threshold and the 150ms prefix padding are the counterweight: they exist so that choosing not to bill silence never clips the start of a sentence.

Measure quality first, or you are not cutting cost, you are degrading the product

Every lever above is reversible except this one: shipping a cheaper configuration that conducts slightly worse interviews, and not knowing. Nobody files a bug report saying "the interviewer asked marginally shallower follow-ups this week".

The method we borrowed is the one Anthropic documents, and it is four unglamorous steps: pull real tasks from production logs weighted like real traffic, write outcome checks for each, record cost per task beside the score, and "run the winner in shadow on a traffic slice before cutover, then keep the suite running".

Two refinements from that guidance changed how we read the results:

1. Price the tail, not the median. "Compare models on the hardest tenth of your tasks, not the typical one." A cheap model handles a clean two-pointer problem fine. It falls over on the candidate who misreads the problem, argues, and needs to be redirected — and that is the session that matters.
2. Sweep effort before you add a second model. The documented advice is to "tune one model before you add a second", and the measured gains from clever multi-model routing were "at the edge of run-to-run noise, for about 2.1 times the cost". Routing architecture is seductive. Turning down one dial is usually better.

Concretely, our check is: take a sample of finished sessions, re-run the candidate configuration against the same rubric, and compare the score distribution and the cost per session side by side. If the scores are flat and the cost drops, ship it behind a slice. If the scores drop at all on the hard tenth, the saving is not a saving.

The per-session cost ledger

None of this is measurable without recording the right fields, and the obvious ones are not enough. Per model call we keep the stage name, the model, input tokens, output tokens, cache-read tokens, cache-write tokens and TTL, and the retry count — then sum them per session rather than per request. The costly sessions are always the ones where a stage ran twice.

Two things deliberately stay out of the log: transcript text and candidate source code. Sizes and counts answer every cost question the payload would have answered, and the retention risk is not worth it. That line, and what else belongs in a real-time log, is the subject of what to log for SSE and voice sessions.

Cost reduction and cost control are also different problems, and solving one does not solve the other. Everything here lowers what a well-behaved session spends; none of it stops one user from starting forty sessions. That needs budgets and ceilings, which we covered in per-user budgets and cost ceilings for an AI product.

If you take one thing from this: the levers that paid best for us were not model downgrades. They were a hard output cap, a summarised tool result instead of a forwarded payload, a dedupe ref worth four lines, and not billing silence. Three of those four are bugs if you look at them from a different angle — which is probably the honest summary of most LLM cost optimization work.