LLM Rate Limiting: Per-User Budgets and Cost Ceilings

Rate limiting a CRUD API is a solved problem: count requests, reject the ones over the line, move on. LLM rate limiting is a different job, because the thing you are protecting is not a database — it is a bill. Two requests that look identical at the edge can differ by three orders of magnitude in cost, and the expensive one is usually the one an abuser sends.

This is how we think about budgets, ceilings and failure modes on Taqari, where one interview is a live audio session, a code-execution queue and a scoring pass stacked on top of each other. What follows is patterns and trade-offs. We do not publish our live limits, for reasons that become obvious about halfway down.

Why requests per minute is the wrong unit for LLM rate limiting

A request counter assumes requests are interchangeable. For an AI product they are not. A candidate asking the interviewer to repeat a question costs almost nothing. A twenty-eighth minute of live audio, carrying the whole accumulated transcript as context on every turn, costs a great deal more. Both increment the same counter by one.

The providers already concede this point in their own design. Anthropic's published rate limits are measured in three dimensions at once — requests per minute, input tokens per minute and output tokens per minute — and exceeding any one of them produces a 429. More tellingly, rate limits are only half the mechanism: spend limits are a separate control, with a monthly cap that pauses usage independently of how fast the requests arrived. Capacity and cost are two different problems, and one limiter cannot solve both.

Note also that the replenishment model is a token bucket rather than a fixed window. Capacity refills continuously up to a maximum instead of resetting on the minute, which is why a client that batches its retries into the top of each minute hurts itself.

What a single request can actually cost you

Before you can write a budget you have to decide what you are counting. We found four quantities worth metering, and only the first is visible to a conventional limiter.

• Requests — still useful, mostly as a cheap guard against scripted hammering.
• Tokens in and out — the real unit of spend. Input and output deserve separate counters because they are priced differently and behave differently; output is produced slowly and can be truncated, input arrives all at once and cannot.
• Wall-clock session time — for anything live, the connection itself is the cost. A voice session is billed by how long someone keeps it open, not by how many times they speak.
• Downstream compute — a submission that runs a dozen test cases in a sandbox is expensive in a way the model never sees. We treat the judge queue as its own budget, for the same reasons we wrote about when designing the sandbox that runs candidate code.

The trap specific to agentic and conversational products is that context grows while request counts stay flat. A fifteenth turn in a conversation resends everything that came before it. Request-per-minute graphs stay perfectly level while cost climbs, which is precisely the pattern a request limiter cannot see.

The three budgets LLM rate limiting actually needs

We ended up with three nested budgets. They exist for different reasons, they fail differently, and collapsing them into one number was the first design mistake we made.

Per-identity budget

The per-user budget is the one that maps to the product promise. A free tier means some bounded number of interviews in some window, and that bound belongs to an authenticated identity, not to a connection. This is the only budget whose numbers are a pricing decision rather than an engineering one, so it lives in server-side configuration, never in client code.

A useful detail borrowed from the providers: new accounts should not get the standard allowance immediately. Anthropic places new organisations in a lower starting tier and raises limits automatically as usage history accumulates, explicitly as a fraud and abuse measure. The same logic applies to a free consumer product, where the cheapest attack is a thousand fresh accounts. Giving a brand-new account a smaller allowance costs almost nothing in legitimate conversions and removes most of the upside from registration farming.

Per-IP budget, for the part before you have an identity

Everything in front of login needs a limiter keyed on something other than the user, because there is no user yet. Signup, token refresh and anything that sends mail or an OTP are all attackable without an account, and all of them cost real money.

Rate limiting keyed on an address is a blunt instrument, and worth being honest about. Shared mobile carrier NAT in India will put a great many unrelated users behind one address, so an IP limit tuned as though an address were a person will throttle a college hostel. We treat per-IP limits as a backstop against obvious floods, set loose enough to be embarrassing as a per-user limit, and never as the primary control on anything a signed-in user does.

Per-session cost ceiling

The third budget is the one that is specific to live AI features, and the one we would add first if we were starting again. A rate limit bounds how fast you may consume. It does not bound how much any single session may consume, and for an open-ended stream those are very different questions.

An interview session on Taqari carries a hard wall-clock cap — thirty minutes, enforced by a timer that ends the session rather than by trusting the client to stop asking. That single constant does more for cost predictability than any limiter, because it converts an unbounded stream into a bounded unit. Once a session cannot exceed a known maximum, a per-user budget expressed in sessions becomes a budget expressed in money.

Wall-clock cap

The simplest ceiling and the one users understand without being told, because a real interview also ends. Enforce it server-side; a client-side timer is a courtesy, not a control.

Context and turn caps

The subtler ceiling. Because every turn resends the transcript, a session's cost grows faster than its length. Trimming what you resend — summarising the early transcript rather than replaying it verbatim — bounds the growth. Caching the stable prefix helps too: Anthropic's prompt caching documentation notes that cache reads are billed at a fraction of base input and, on most models, do not count toward the input-token rate limit at all, so a stable system prompt and tool definitions are nearly free to resend once cached. The system prompt that defines our interviewer is long and completely static between turns, which makes it the ideal thing to cache — we covered how that prompt is built when writing about prompting an interviewer rather than a chatbot.

Put the gate at the session mint, not on the stream

The structural decision that matters most is where you check the budget, and it is the thing LLM rate limiting most often gets wrong. Our client cannot talk to a model directly on its own authority: it first asks the server to create a session, and that call is where every budget is evaluated. The server decides whether this user may start at all, which model they get, and what ceiling applies, then returns a session identifier. Only then does the client open the live connection.

That ordering is the whole design. Refusing a request before the first token is cheap and clean — one JSON response, a message the interface can explain. Interrupting a live session halfway through is expensive and has already spent most of the money. Enforcement belongs at the moment of authorisation, and the stream should carry only the consequences of a decision already made.

Returning the model name from that call, rather than hard-coding it in the client, turns out to be the single most useful cost lever we have. Changing which model serves which tier is a server-side configuration change, and no deploy of the frontend is involved.

Retryable versus terminal: the 429 you must never retry

Here is the bug that taught us the most, and it is a client bug rather than a server one. When a live model connection closes, the naive handler reconnects. If the reason it closed was that you are out of quota, reconnecting is the worst possible response: every attempt fails, and on a metered API every attempt may still cost something. A retry loop against an exhausted budget is a denial-of-service attack you run against yourself.

So the close handler has to classify before it acts. Ours checks for the resource-exhausted signal specifically and stops, and only treats other unexpected closes as reconnectable:

// A close is not automatically a reason to reconnect.
const reason = (event.reason || '').toLowerCase();
const isQuotaError = event.code === 1011 && (
  reason.includes('quota') ||
  reason.includes('exceeded') ||
  reason.includes('resource_exhausted')
);

if (isQuotaError) return showUnavailable();  // terminal: do not retry

if (event.code !== 1000 && attempts < maxReconnects) {
  setTimeout(reconnect, 2000 * Math.pow(2, attempts++));  // 2s, 4s, 8s
}

Three attempts with doubling delays, and then it stops and says so. The cap matters as much as the backoff: a client that retries forever with perfect exponential backoff is still a client that never tells the user anything is wrong.

This distinction is not our invention — it is in the provider contract, and reading it carefully is worth the ten minutes. A rate-limit 429 carries a retry-after header and will succeed later. A spend-cap rejection returns the same 429 status and the same rate_limit_error type but has no retry-after, and Anthropic's error documentation states plainly that retrying it, including the SDK's own automatic retries, fails until access resumes. A limit you set yourself comes back as a 400 instead. Same family of problem, three different correct responses.

Classify the failure, then pick the policy

We sort every downstream failure into one of three buckets, and the bucket determines the behaviour rather than the status code alone. Rate limiting on the way in is only half the system; this is the other half.

Transient

Connection resets, timeouts, a 5xx, an overloaded signal. Retry with exponential backoff and jitter, bounded attempts. Jitter matters more than it looks: without it, every client that failed in the same second retries in the same second.

Throttled with a hint

A 429 carrying a retry delay. Wait exactly that long rather than guessing, queue the work if it is not interactive, and tell the user it is queued. This is the only bucket where a retry is unambiguously correct.

Terminal for now

Quota exhausted, spend cap reached, a limit someone configured. No retry at any interval will help. Fail fast, say what happened, and make sure it reaches a dashboard, because this is the class that silently stops the product working while every graph still looks healthy.

Degrading gracefully instead of failing hard

Rate limiting that only ever says no is a blunt product decision. Hitting a limit should rarely produce nothing. The useful move is almost always to serve a smaller version of the same thing, and the options are more plentiful than they first appear.

1. Route to a cheaper model. Interactive turns need the capable model; a transcript summary does not. Picking the model server-side at session creation makes this a configuration change rather than a release.
2. Shrink what you resend. Summarise the early transcript instead of replaying it. Cost per turn stops growing with session length, and the quality loss is smaller than the saving.
3. Queue the non-interactive work. Scoring does not have to be synchronous. A report that arrives a minute later is fine; an interviewer that pauses for a minute is not. Keeping the two on separate budgets means a scoring backlog cannot starve a live session.
4. Shorten rather than refuse. A ten-minute session when the budget cannot fund thirty is a better product than an error page, provided you say so up front.

What never works is hiding the limit. A session that ends with a generic failure trains people to retry, which is exactly the behaviour you are trying to prevent. Our quota message names the cause plainly — the interviewer is temporarily unavailable because a usage limit was reached — because a user who understands a limit waits, and one who does not refreshes eleven times.

Abuse control that is not rate limiting

Some of the largest savings came from fixing things that were never limit problems at all. They were correctness bugs that happened to cost money.

• Duplicate session suppression. React effects in development, a double click, a reconnect racing a fresh start — each can open two sessions where the user wanted one, doubling cost for no benefit. Two guards, one for in-flight connects and one for completed initialisation, are worth more than any limiter.
• Idempotent creation. If a mint call is retried because its response was lost, it must return the existing session rather than start a second. The same reasoning we applied to payment callbacks that may arrive twice applies to anything expensive enough to meter.
• Concurrency guards on submission. One submission in flight per user, enforced by state the submit path actually checks, stops the queue being flooded by someone holding down a key.
• Input size caps. A length limit on submitted source and on any free-text field, checked server-side, closes the cheapest way to make one request cost ten.
• Abandoned work detection. A tab that closes mid-run leaves work nobody will read. Streaming results back over a connection that can be observed to drop gives you a signal to stop early — a side benefit of the design we described in streaming test results over SSE.

What to measure, and why the obvious metric is wrong

Requests per second tells you nothing about whether this is working. The metrics that actually predicted trouble for us were these four.

• Cost per completed interview. The denominator is the point. Cost per request falls reassuringly while cost per outcome rises, which is the exact shape of a context-growth problem.
• Share of sessions that hit a ceiling. Near zero means the ceiling is decoration. Uncomfortably high means it is a product constraint being sold as a safety limit, and someone should decide which it is.
• Rejections split by class. Transient, throttled and terminal counted separately. A rise in terminal rejections is a billing incident; a rise in throttled ones is a capacity conversation. Aggregated into one error rate, both are invisible.
• Retry amplification. Downstream attempts divided by user-initiated actions. Anything much above one means a client somewhere is retrying something it should not be, and that number climbs fastest precisely when things are already failing.

The uncomfortable conclusion, after all of this, is that cost control in an AI product is mostly not a limiter. It is a ceiling on the unit of work, a client that can tell a timing problem from a terminal one, and a server that decides every expensive thing before any of it starts. Rate limiting is the last line, and if it is the only line you will find out the expensive way.