What is browser-based proctoring?

Browser-based proctoring runs presence checks on the candidate's own device instead of streaming video to a server. A model in the page reads webcam frames, derives signals like face count and head direction, and sends only small event records. The video never leaves the browser.

Can MediaPipe Face Landmarker detect where someone is looking?

Partly. The refined face mesh includes iris landmarks, so you can estimate horizontal gaze from the iris position inside the eye socket. It is a rough direction, not a calibrated screen coordinate, and it degrades badly in low light or behind glasses. Treat it as a weak signal.

Why use the CPU delegate instead of GPU for MediaPipe in the browser?

GPU inference is faster but depends on a healthy WebGL context. Tab backgrounding and driver resets can lose that context mid-session and kill the detector. CPU inference is roughly 30% slower, which is invisible at proctoring frame rates, and it survives those events.

How many landmarks does the MediaPipe face mesh return?

MediaPipe's face mesh estimates 468 3D face landmarks. The attention-mesh variant adds refined points around the lips, eyes and irises, which is where iris indices such as 468 come from. Always check that an index exists before using it, because the extra points are not guaranteed.

Does browser-based proctoring prove that someone cheated?

No. A camera signal can tell you a face left the frame; it cannot tell you why. Lighting, a second monitor, a disability, a doorbell, and an actual lookup all produce the same event. We surface the signals alongside the technical score and let a human read them in context.

How do you stop per-frame detection from re-rendering React?

Keep everything the loop touches in refs and never call setState inside it. The detector instance, the event buffer, timers and the violation state machine all live in useRef, so a 30-frames-per-second loop produces zero renders. Only coarse lifecycle flags belong in state.

Browser-Based Proctoring: Face and Gaze Signals in JS

Every mock interview on Taqari can run with the camera on. That raises an obvious question: what do you do with the camera feed? Our answer is browser-based proctoring — the frames never leave the candidate's machine, a face model runs in the page, and the only thing we transmit is a short list of timestamped events.

This post is about how that actually works: the signals we compute, the ones we deliberately refuse to compute, how a 30-frames-per-second loop coexists with React without triggering a single re-render, and why we label all of it advisory rather than accusatory.

Why browser-based proctoring instead of server-side video

The obvious design is to upload the webcam stream and analyse it on a server. We rejected it on three counts.

The first is bandwidth. A candidate solving a hard problem is already holding a WebRTC audio session with the interviewer and pushing code to a judge queue. Adding a video uplink competes with the thing they are actually being assessed on, and it degrades worst on exactly the connections that are already weak.

The second is storage. Recorded interview video is a liability with a retention policy, an access-control story and a breach radius. Not having it is cheaper than securing it. The same instinct drives how we think about the rest of the platform — see our write-up on running untrusted candidate code safely, where the cheapest way to secure a capability was to not grant it.

The third is honesty about what the signal is worth. If we are only ever going to produce soft, advisory indicators, it makes no sense to pay for a video pipeline to produce them.

What we decided not to build

We do not do identity verification, emotion inference, or any attempt to classify a person. We do not record frames, keep thumbnails, or ship a single image off the device. We do not watch the screen, enumerate displays, or inspect other windows. The feature answers one narrow question — was a single attentive face in front of the camera — and stops there.

The signals we actually compute

Everything our browser-based proctoring reads comes from one model: MediaPipe's Face Landmarker, loaded as a WASM task into the page. MediaPipe's face mesh estimates 468 3D face landmarks, and the attention-mesh variant refines the points around the lips, eyes and irises. Those refined iris points are what make a gaze estimate possible at all.

Face presence and face count

The cheapest two signals come straight out of the result object. An empty landmark array means FACE_MISSING. More than one entry means MULTIPLE_FACES. We configure the task with numFaces: 2 rather than the documented default of 1, for a specific reason: with a cap of one, a second person in frame is indistinguishable from one person, and the most interesting event becomes invisible. Two is enough to detect "more than one" while keeping inference cost flat.

Eye aspect ratio as a gate, not a signal

Before looking at gaze we compute an eye aspect ratio: two vertical distances across the eyelid divided by twice the horizontal eye width. It collapses toward zero when the eye closes.

const getEAR = (landmarks, indices) => {
  const [p1, p2, p3, p4, p5, p6] = indices.map(i => landmarks[i]);
  if (!p1 || !p2 || !p3 || !p4 || !p5 || !p6) return 1.0;
  const dist = (pA, pB) => Math.hypot(pA.x - pB.x, pA.y - pB.y);
  return (dist(p2, p6) + dist(p3, p5)) / (2.0 * dist(p1, p4));
};

We are not detecting blinks or drowsiness with this. It exists purely as a gate: if the ratio sits below roughly 0.2 the eye is closed or nearly closed, the iris position is meaningless, and we skip gaze entirely for that frame. Without the gate, every blink reads as a gaze excursion, because a closed eyelid pushes the apparent iris centre wherever it likes.

Note the return 1.0 on the missing-landmark path. A missing point returns a ratio that is unambiguously "open", which makes the gate fail open. Failing closed would have been worse — a dropped landmark would manufacture a violation.

Gaze from the iris, head turn as the fallback

Horizontal gaze is the iris centre expressed as a fraction of the distance between the inner and outer eye corners. Dead centre is 0.5; we treat below 0.40 and above 0.60 as looking away. Vertical gaze gets a much cruder test, because the eyelid aperture changes shape as you look up and the landmark geometry stops being a clean ratio.

Head turn is computed independently and does not need iris points at all: take the nose tip, take the midpoint between the two outer eye corners, and divide the horizontal offset by the eye-corner distance. Normalising by face width is what makes the threshold hold whether the candidate is leaning into the laptop or sitting back.

Why gaze can silently fail

The refined iris landmarks are not guaranteed to be present. Depending on the model asset and the frame, the array can come back without them. Our first version returned 'CENTER' in that case, which is a quiet disaster: gaze appears to be working, reports "looking at the screen" forever, and nothing ever trips.

The fix is a distinct third state. The gaze helper returns 'NO_IRIS' when the iris index is absent, and the caller treats that as "no gaze signal available" rather than "gaze is fine", falling through to head turn:

const gaze = getGazeDirection(landmarks);
if (gaze !== 'CENTER' && gaze !== 'NO_IRIS') {
  currentViolation = gaze;
} else {
  const headTurn = getHeadTurn(landmarks);
  if (headTurn !== 'CENTER') currentViolation = headTurn;
}

The general lesson outlived this function: when a detector can be unavailable, "unavailable" must be its own return value. Folding it into the happy path makes a broken signal indistinguishable from a clean one.

Wiring the detector into a React app without re-rendering it

The loop runs on requestAnimationFrame and calls detectForVideo on each eligible frame. At that rate, anything that touches React state is a non-starter.

Keeping every frame out of React state

The detector instance, the event buffer, the last-warning timestamp, the last-batch timestamp, the monotonic frame clock and the violation state machine all live in refs. The only piece of actual state the hook exposes is a coarse isTracking boolean that flips twice per session. The live props — the session id and whether proctoring is active — are also mirrored into refs, so the frame callback created once inside the effect always reads current values without the effect re-mounting and tearing down the model. That is the same discipline we described in what we keep out of Redux during a live session: high-frequency data belongs in refs, not in a store.

The guards before detectForVideo

A video element can report readyState >= 2 and still not be paintable. During camera warm-up, or just after a tab returns to the foreground, detectForVideo throws on a zero-area region of interest. We check non-zero dimensions, not paused, not ended, and currentTime > 0 — the last one being the cheapest proof that at least one frame has actually advanced.

Monotonic timestamps

The video-mode entry point takes a frame timestamp in milliseconds. In our experience it must strictly increase across calls, and performance.now() is not a safe source on its own — clamped timer resolution can hand you the same value twice. We force the invariant ourselves:

let ts = performance.now();
if (ts <= lastTimestampRef.current) ts = lastTimestampRef.current + 1;
lastTimestampRef.current = ts;

The ref also resets to zero on cleanup, so a remount starts a fresh clock rather than inheriting a future timestamp from the previous session.

Backing off instead of swallowing errors

The subtlest bug in this file had nothing to do with correctness. A thrown frame was caught and logged quietly, but the WASM runtime prints its own diagnostics before the exception reaches JavaScript — so suppressing our log changed nothing, and in development the error overlay kept appearing. The only fix that works is to stop calling the detector for a short cooldown (we use about 400 ms) so the throw never happens. Catching an error you cannot silence is not the same as handling it; the handling is the back-off.

From frames to events

Per-frame classification is far too noisy to send anywhere. A small state machine turns it into a handful of durable events.

Sustained violations and periodic flushes

The machine holds one current violation type with the time it started. A new label resets it; the same label accumulates duration. Each type gets its own minimum duration before anything is reported, because they are not equally ambiguous:

Multiple faces — no delay. A second face is unambiguous and worth recording immediately.
Face missing — around 1.5 seconds. Turning to think, adjusting a chair, or a frame the model simply failed to resolve all look identical for a moment.
Gaze and head turn — a shorter window, under a second, since the feedback is a gentle nudge rather than a mark against anyone.

The first version only wrote an event when a violation ended, which left an obvious hole: a candidate who walked away and never came back produced nothing at all, because the recovery branch never ran. Now a long-running violation also flushes a partial record every ten seconds, tagged as ongoing, and the recovery branch writes only the tail since the last flush. Durations add up correctly either way, and nothing depends on the candidate coming back.

Batching and requeueing on failure

Events accumulate in a ref and go out roughly every three seconds as one batch, scoped to the interview session. On a network failure the batch is prepended back onto the buffer rather than dropped, so a flaky connection delays the record instead of losing it. Each event carries its type, a severity, start and end timestamps, a duration and a small metadata object — and nothing else. There is no image payload, because there is no image.

Candidate-facing feedback is separate and deliberately gentler: a toast with a five-second cooldown, so a candidate who keeps glancing away gets one reminder rather than a stream of them.

The frame-rate and battery trade-off

Running inference on every animation frame is the wrong default, and we are honest that our current loop is more generous than it needs to be. Two choices keep the cost tolerable.

The first is the CPU delegate. GPU inference is faster, but it depends on a live WebGL context, and a backgrounded tab or a driver reset can take that context away mid-interview — which means losing proctoring exactly when a candidate has switched away from the tab. CPU inference costs roughly 30% more time per frame and is imperceptible at the rate that matters here, because nothing browser-based proctoring measures changes meaningfully between 15 and 60 frames per second.

The second is that the expensive paths are gated. Blendshape output and the facial transformation matrix are both switched off, since neither feeds a signal we use. The guards short-circuit before inference whenever the video is not paintable, which means a backgrounded tab costs almost nothing — requestAnimationFrame stops firing there anyway, and that is a feature, not a gap: a tab nobody is looking at needs no analysis.

The honest improvement we have not shipped is decoupling detection from the frame clock entirely — sampling at a fixed 10 to 15 Hz regardless of display refresh. On a 120 Hz laptop we currently do up to eight times the work needed to notice that someone left the frame two seconds ago.

Why these signals stay advisory

The server turns events into an integrity score out of 100, and the report shows it beside the technical evaluation with colour bands rather than a pass or fail. That framing is the whole design.

A camera signal can tell you that a face left the frame. It cannot tell you why. Poor lighting, a second monitor, a screen reader, a toddler, a doorbell, and genuinely looking something up all produce identical events. Any browser-based proctoring system that converts those events into an accusation is asserting something its inputs do not support — so ours reports what it saw, next to the score, and leaves the judgement to a person.

This is not only an ethics position, it is an accuracy one. Webcam gaze estimation without per-user calibration is weak, and it is weakest under exactly the conditions that correlate with having worse equipment and a worse room to work in. A threshold tuned on a well-lit desk setup will fire more often for someone on a cheap laptop in a shared room. Building an automatic verdict on that would mostly automate a bias.

So the signals are visible to the candidate while the interview runs, the score is explained rather than enforced, and a low score routes to review rather than to rejection. If you are preparing with the camera on, you can see how a Taqari mock interview runs end to end.

What we would change

Three things, in the order we would do them.

Decouple the sample rate from the frame clock. A fixed low-frequency timer with explicit frame skipping, instead of inference on every animation frame.
Self-host the model and the WASM runtime. Loading them from a public CDN at runtime means a third party is in the critical path of a feature that must work offline-ish and consistently, and it couples us to whatever that URL serves. Pinning the loader and the runtime to the same version is necessary but not sufficient.
Collapse the thresholds into one declarative table. The per-type minimum durations, the warning cooldown and the flush interval are currently constants spread across a file; they belong in one object you can tune and, eventually, test against recorded landmark traces rather than against a live face.

The pattern worth stealing from all of this is narrower than the feature: decide what the smallest sufficient signal is, compute it where the data already lives, and transmit the conclusion rather than the evidence. We reached the same place when we built the voice transport for live interviews — the right answer was usually to move less data, not to move it faster.