Judge0 Batch Submission: Streaming Per-Test-Case Results

A candidate hits Run on a problem with twelve hidden test cases. The naive version of that is twelve HTTP requests to an execution service, each one blocking, each one waiting on the slowest. We replaced it with a single Judge0 batch submission plus per-test-case callbacks, and the interesting part turned out not to be the submit at all — it was assembling results that arrive out of order into a list the interface can render while it is still filling up.

This is the submission half of the pipeline. The other half, streaming those results to the browser over SSE, we wrote about separately; and the runner that actually executes the code is covered in how we run untrusted candidate code safely. Here we care about what happens between "one click" and "twelve ordered results".

Why N synchronous requests is the wrong shape

The obvious design is a loop: for each test case, POST the source and the input, wait for the verdict, render it. It works on a laptop with two test cases and falls apart everywhere else.

Run them serially and total latency is the sum of every execution, including the one pathological case that sits on the time limit. Run them in parallel instead and you now hold a dozen in-flight connections from a browser, which browsers throttle anyway, and you have made the client the coordinator of a distributed job. If the tab closes at case seven, nothing on the server knows the run is abandoned. If a request times out, you cannot tell whether the code ran and the response was lost or the code never ran, so a retry might double-execute.

There is also a scheduling cost that is easy to miss. Twelve independent requests look like twelve unrelated jobs to the execution queue. It cannot co-locate them, it cannot compile once, and it cannot decide that the whole run is already doomed because case one segfaulted. Batching hands the judge the information that these belong together.

And the thing users actually feel: with N synchronous requests there is no honest progress indicator. You either show a spinner until everything finishes, or you invent one.

What a Judge0 batch submission actually returns

Judge0 describes itself as a "robust, fast, scalable, and sandboxed open-source online code execution system for humans and AI", and its batch endpoint is the piece we care about. You POST to /submissions/batch with a body shaped like { "submissions": [ ... ] } and get back a 201.

Two properties of that response drive everything downstream.

Tokens come back positionally

The response is a JSON array with one entry per submission you sent, in the order you sent them, each carrying a token:

[
  { "token": "db54881d-bcf5-4c7b-a2e3-d33fe7e25de7" },
  { "token": "ecc52a9b-ea80-4a00-ad50-4ab6cc3bb2a1" },
  { "token": "1b35ec3b-5776-48ef-b646-d5522bdeb2cc" }
]

Index i in that array is test case i. That mapping is the only thing tying an opaque token back to "expected output for case 3", so it has to be persisted with the run before you do anything else with it.

Per-item errors ride in the same array

This is the part that bites. A batch is not all-or-nothing. If one submission fails validation, the response is still a 201 array — the bad entry is simply replaced by an error object in its own slot:

[
  { "token": "c2dd8881-644b-462d-b1f9-73dd3bb0118a" },
  { "language_id": ["language with id 123456789 doesn't exist"] },
  { "source_code": ["can't be blank"] }
]

If you map that array to tokens and filter out the falsy ones, you have just silently renumbered every test case after the first failure. Case 2 becomes case 1, and the candidate sees a diff against the wrong expected output. The safe move is to keep the array dense: walk it by index, record a token or a local failure for each slot, and never compact it.

Callbacks beat polling once the batch is real

With tokens in hand there are two ways to learn that a case finished. You can poll GET /submissions/batch with a comma-separated tokens list — and the fields parameter is worth using there, because trimming the response to what you render cuts a surprising amount of payload. But polling means choosing an interval, and every interval is wrong: too short and you hammer the judge for unchanged rows, too long and a case that finished in 40ms sits invisible for a second.

The alternative is the callback_url attribute on a submission. The Judge0 docs define it precisely: a "URL on which Judge0 will issue PUT request with the submission in a request body after submission has been done."

Three things in that sentence matter. It is a PUT, not a POST, so a receiver that only routes POST will silently 404 every result. The body is the submission itself, not an event envelope pointing at one, so there is no follow-up fetch. And it fires per submission — a batch of twelve produces twelve separate callbacks, not one consolidated batch callback.

That last point is sometimes reported as a limitation. For us it is the entire feature. Twelve independent completion events are exactly the granularity a progress bar needs; a single batch-level callback would put us right back at "spinner until done".

Reading a callback body without trusting its shape

A finished submission is not a clean typed object. Our client normalises three things on every single case update, and each one is there because it broke something first.

The status id may be a string

We run it through parseInt before comparing, because a strict === against a numeric literal fails quietly when the value arrives as "3". A quiet failure here means every passing test case renders as failed, which is about the worst possible bug in an interview product.

Output fields may be base64

Judge0 takes a base64_encoded flag in both directions — set it on the way in and you should encode source_code, stdin and expected_output; set it on the way out and the text fields come back encoded. Since the flag can differ between the path that submitted and the path that renders, our results component decodes defensively and falls back to the raw string when decoding throws, rather than showing a candidate a wall of base64.

Expected output arrives under two names

Depending on which layer normalised the payload, the field is expectedOutput or expected_output. We read both. It is not elegant, but a one-line coalesce at the boundary is cheaper than a naming migration across two services.

Status ids are not a boolean

Judge0 publishes a status table, and only one value in it is a pass:

• 1 In Queue, 2 Processing
• 3 Accepted — the only pass
• 4 Wrong Answer
• 5 Time Limit Exceeded
• 6 Compilation Error
• 7 onwards: runtime errors, broken out by signal — SIGSEGV, SIGXFSZ, SIGFPE, SIGABRT, and a non-zero exit code case

Collapsing that to statusId === 3 ? "passed" : "failed" is fine for the pass/fail pill, and that is what we store. But it is not fine for the message. A segfault, an infinite loop and a genuinely wrong answer are three different lessons, and an interview product that tells all three candidates "Wrong Answer" has thrown away the most useful thing it knows. We keep the raw status id and its description on every case so the detail view can say what actually happened.

Compilation errors deserve special handling too: a compile failure fails every case identically, so our results view detects the presence of compiler output and surfaces it once at the top rather than repeating the same stack twelve times.

Assembling partial results in order

Here is the actual sequence on the client. When the stream opens, the first message carries the total case count, and before a single result exists we build the whole array:

const initialResults = Array(message.totalCases)
  .fill(null)
  .map((_, index) => ({
    caseIndex: index,
    status: 'pending',
    statusId: null,
    stdout: null,
    stderr: null,
    time: null,
    memory: null,
  }));

Every later update is an indexed write into that array, not a push:

updateCompilerTestResult(state, action) {
  const { caseIndex, result } = action.payload;
  if (state.testResults[caseIndex]) {
    state.testResults[caseIndex] = {
      ...state.testResults[caseIndex],
      ...result,
    };
  }
}

That reducer is four lines and it buys three properties at once. It is order-independent, because case 9 landing before case 2 writes to slot 9 and nothing shifts. It is idempotent, because a redelivered callback merges the same fields into the same slot. And it is a constant-time write, so a batch of a hundred cases does not turn into a hundred array scans.

The progress counter is the other half of that discipline. We take finished and total from the server message rather than counting non-pending entries on the client, precisely because a duplicate delivery would inflate a local count while leaving the server's number correct.

The data flow, end to end

Notice what the diagram does not show: any arrow from the browser to the execution service, and any arrow from the public internet to the callback receiver. The receiver lives on the private side of our network and accepts work only from the execution layer. A browser never addresses either one — it talks to our API and reads the stream, and that is the whole surface it gets.

What still goes wrong

The callback that never arrives

Webhooks are best-effort by nature. A receiver restart, a network blip, a bug in our own handler — any of those loses an update, and a candidate is left with one case stuck on "processing" forever. The fix is not cleverness, it is redundancy: our completion message carries the full results array as a summary, and the client overwrites its whole list from that summary when the run finishes. Every incremental update is an optimisation; the final message is the source of truth.

The transport dies but the run does not

If the event stream errors, the client closes it and falls back to polling the run by its id, retrying on a short interval and backing off on failure. This works only because run state is held on the server keyed by the run id rather than accumulated in the browser — the same reason a mid-run tab refresh can rejoin and see cases that already finished. Transport is replaceable; state is not.

Arrival order is not case order

It bears repeating because it is the mistake everyone makes once. Case 5 frequently finishes before case 2, because test inputs are not the same size. If any part of your pipeline appends in arrival order — the receiver, the store, the list component — the candidate sees results attributed to the wrong inputs. Keying by case index everywhere, and never rendering from an append-ordered list, is what prevents it.

What we would tell anyone building this

• Submit once and get an identifier back immediately. The client should never be the coordinator of a multi-part job.
• Treat the batch response as positional. Never compact it, never filter it, and persist the index-to-token map before you do anything else.
• Use callbacks for latency and polling for certainty. They are not competing designs; the second one is the floor under the first.
• Size the result array from the declared total, not from what has arrived.
• Make every write idempotent and indexed, so a duplicate delivery costs you nothing.
• Keep the raw status id. Pass and fail is what you render first; what actually failed is what makes the product useful.
• Send a final summary that can rebuild the entire run from scratch, and have the client trust it over everything it accumulated.

None of this is exotic. It is the same shape as any long-running job behind a REST API — return an id, stream progress, guarantee a terminal state — applied to a workload where the user is watching the progress bar with real stakes attached. You can see the end result on a live problem in a Taqari mock interview, where the per-case pills fill in one at a time while the run is still going.

The reference for everything protocol-specific above is the Judge0 CE API documentation, and the feature list in the Judge0 project README is a good map of what the execution layer will and will not do for you. What it will not do is assemble your batch — that part is yours.