Real-Time Observability: What to Log for SSE and Voice

The first time a Taqari interview broke in production, every dashboard was green. The API process was healthy, error rates were flat, p50 latency was fine. A candidate had still sat through ninety seconds of silence while the interviewer said nothing, and then the tab had quietly reconnected and started over.

That is the whole problem with real-time observability in one incident. Our monitoring was built around requests, and nothing about that failure was a request. This is what we log now for an SSE run stream and a live voice session, how we stitch one run back together across services, and which numbers actually move before a user has a bad experience.

Why a 200 tells you nothing about a stream

A request-shaped monitor watches a status code and a duration. Both are meaningless for a stream, because the status code is written before the work starts. The moment a server flushes headers for text/event-stream, it has committed to 200 — everything that can go wrong afterwards has to travel inside the body.

This is not a quirk of our stack. Anthropic documents the same hazard for its own streaming API: when receiving a response over server-sent events, an error can occur after the API returns a 200, and in that case error handling "doesn't follow these standard mechanisms". The failure arrives as an error event in the stream — an overloaded_error that would have been an HTTP 529 in a non-streaming call. A proxy log counts that as a success.

Real-time observability starts with the session, not the request

Once you accept that, the unit of instrumentation changes. We stopped asking "did this request succeed" and started asking "what happened over the life of this run". A run on Taqari is one code submission streaming test-case results back, or one voice interview lasting up to thirty minutes. Both are a sequence of events under a single identifier.

Mint the id before the work starts

Every stream on Taqari is keyed by an id the server mints when the work is accepted, not when the stream opens. Submitting code returns a runId, which the client immediately uses to subscribe:

const sseUrl = `${API_BASE_URL}${STREAM_PATH}/${runId}`;
eventSourceRef.current = new EventSource(sseUrl, { withCredentials: true });

That ordering matters for debugging as much as for correctness. Because the id exists before the stream does, a run that never gets a subscriber is still visible — and "submission accepted, nobody ever connected" is a real failure mode you cannot see if the id is born with the connection. The same design is what lets the stream survive a reload at all, which we covered when we wrote about streaming test results over Redis and SSE.

What belongs on every line

Four fields, on every log line a real-time feature emits: the run or session id, the attempt number, a monotonic timestamp, and the transport. Attempt number is the one people leave out and regret. A session on its third connection attempt behaves differently from a fresh one, and without the counter the two are indistinguishable in a log search. We keep it on a ref that survives re-renders:

connectionAttemptRef.current += 1;
const attemptNum = connectionAttemptRef.current;
console.log(`Connection attempt #${attemptNum}${isReconnect ? ' (reconnect)' : ''}`);

The upstream equivalent is worth capturing too. Every Claude API response carries a request-id header such as req_018EeWyXxfu5pfWkrYcMdjWG, and the same value appears in error bodies. Storing that id against our own session id is the difference between "the model call failed" and a support conversation that can actually be resolved.

What to log for an SSE run stream

Our judge stream emits typed messages — connected, case_update, completed — and the client dispatches each into Redux. The instinct is to log the messages. Resist it: on a hundred-test-case run that is a hundred log lines carrying candidate output, and it answers nothing you will actually ask at 2am.

The four lifecycle events that matter

• Opened. Subscriber attached, with the run id and the attempt number.
• First event. The moment the first payload reaches the client. This is the number users feel.
• Progress transitions. Not every case — just the count, so a stall is visible as a count that stops moving.
• Closed. With a reason: completed, client navigated away, error, or timed out.

Anthropic's stream format is a good model for why the boring events earn their place. Its event sequence runs message_start, content block deltas, message_delta, message_stop, and may include any number of ping events. Pings carry no data. They exist so that both ends — and anything in between — can tell a healthy idle stream from a dead one. The same docs add a line worth taping to the wall: new event types may be added, so handle unknown event types gracefully. We log an unknown type at warn level and move on rather than tearing down the stream:

default:
  console.warn('Unknown SSE message type:', message.type);

The fallback is the signal

When our EventSource errors, the client closes it and falls back to polling the run endpoint. From the user's side that is a graceful degradation and nothing visibly breaks. From an operator's side it is the single most valuable metric the feature emits, because a rising fallback rate means streaming is failing somewhere while every user-facing success metric stays flat.

Any silent fallback deserves this treatment. If the code path exists to hide a failure from the user, it has to be loud somewhere else — otherwise you have built a feature that degrades permanently and never tells you.

What to log for a voice session

A voice interview is a harder target. There is no request to measure, the transport is either WebRTC or a WebSocket, and the failure a candidate reports — "it stopped talking to me" — maps onto at least six distinct causes. We chose transports for reasons we wrote up separately; the observability consequence is that each one surfaces failure differently.

Close codes carry the diagnosis

For the WebSocket path, the close event is the richest diagnostic the session will ever produce, and it costs nothing to record properly:

ws.onclose = (event) => {
  console.log('Closed:', {
    code: event.code,
    reason: event.reason,
    wasClean: event.wasClean,
  });
};

Three structured fields, not a string. Once they are structured, the retry policy can read them — and the policy is where logging turns into behaviour.

Code 1000 — a clean close

Expected. The session ended or we closed it ourselves to reconnect. Never retry, and never alert.

Code 1011 with a quota reason

Terminal for now. Retrying a resource-exhausted error burns the quota it just failed on. We detect it and stop, telling the user plainly instead of looping. Anthropic draws the same distinction on its own limits: an ordinary 429 carries a retry-after header, while the spend-cap 429 never does and "keeps failing until access resumes". Two identical-looking rejections, opposite correct responses — which is exactly why the reason text has to survive into the log. We go further into the budget side of this in our notes on per-user budgets and cost ceilings.

Any other non-1000 close

Unexpected. Back off and reconnect, with a bounded number of attempts and exponential delay:

const backoffDelay = 2000 * Math.pow(2, reconnectAttempts - 1);

Log the attempt number and the delay alongside the code. A session that reconnected three times and succeeded is a success in every aggregate metric and a disaster for the person who sat through it.

ICE state changes are not errors until they are

On the WebRTC path the equivalent signal is the ICE connection state. Our handler treats failed and disconnected as a trigger to tear down and rebuild the peer connection. Worth knowing: disconnected is often transient and recovers on its own, so logging every transition at error level trains you to ignore the log. We record every transition, and alert only on the ones that did not recover inside a few seconds — the state machine is the data, not each edge.

Correlating one run across services

None of this helps if the signals live in four places. A single interview touches the browser, the API process, Redis, the judge workers and at least one model provider, and the only thing that makes those five streams one story is an id that travels through all of them.

Real-time observability depends on one unglamorous discipline: accept the id at every boundary, put it in every log line, and never let a service invent its own. When a candidate writes in with a timestamp, one search across services should return the whole run in order.

The metrics that actually predict a bad interview

We track plenty of things that have never once predicted a complaint — CPU, memory, average response time. Three metrics have.

Time to first event, at the tail

Not time to completion: time to the first thing the user sees. For a code run that is the first test-case result; for a voice session it is the first audio packet. And always at the 95th percentile, never the mean. Means are dominated by the sessions that worked. The tail is where the people who churn live.

Reconnects per session

The leading indicator for everything else. Reconnects rise before error rates do, because a reconnect is by definition a failure the system papered over. If this number drifts up, something upstream is degrading and the users are absorbing it on your behalf.

The silence histogram

For voice, the gap between a candidate finishing a sentence and the interviewer starting to speak. A mean of 800ms sounds excellent and is entirely compatible with one turn in twenty taking six seconds — which is the one the candidate remembers. We keep it as a histogram with buckets tight at the bottom, and we watch the top bucket, not the average. The related signal is how often a turn ends in a barge-in, which we dug into when writing about interruption handling in a voice agent.

What we deliberately do not log

Real-time features generate enormous, irresistible, highly sensitive payloads. The list of things that never reach a log: audio frames, transcript text, candidate source code, ephemeral keys, and model tokens. Not redacted — not collected.

It turns out we never needed them. Every operational question we have actually had was answerable from sizes, counts, durations and ids. When we sync editor state to the voice model, what gets logged is the length, not the code:

console.log('Editor state synced, length:', currentCode.length);

Two more rules we hold to. First, instrumentation must never be able to break the feature — our analytics helpers swallow every error on principle, because a dropped event is always preferable to a thrown exception in a click handler. Second, high-frequency signals get buffered and flushed in batches rather than sent per frame; our client-side proctoring loop buffers events and ships them periodically, because a detector running at animation frame rate would otherwise generate more network traffic than the interview itself.

The checklist we use

If you are adding real-time observability to a streaming feature, this is the short version:

1. Mint the correlation id server-side, before the connection exists.
2. Log lifecycle, never payload. Opened, first event, transitions, closed-with-reason.
3. Put the id, the attempt number and the transport on every line.
4. Record close codes as structured fields, and let the retry policy read them.
5. Make every silent fallback loudly visible in metrics.
6. Alert on tail latency, reconnect rate and the top silence bucket — not averages.
7. Store the upstream provider's request id against your own session id.
8. Handle unknown event types without tearing down the stream.

None of this required a new vendor. It required deciding that the session, not the request, is the thing being observed — and that a feature which hides its own failures is a feature you have stopped being able to operate. If you want to see what these streams look like from the other side, you can take a free AI mock interview on Taqari and watch the run results arrive one test case at a time.