A model drawing a box around an object is immediately satisfying. It looks like understanding, and sometimes it is useful enough that the distinction barely matters. Then somebody changes the lighting and the box develops a philosophical objection to the object.
A useful local-vision experiment starts with a narrow task and a way to recognize failure. It does not begin by promising that the browser can inspect every machine in the workshop.
Pick one task
Choose image classification or object detection for a first workbench, not both plus segmentation, tracking, and a conversational explanation of the failure. Supported pipelines and model artifacts must match the task. Transformers.js documents its pipeline interface, while ONNX Runtime documents browser execution paths and their constraints. Hugging Face ONNX Runtime
For a first evaluation, choose a small, clearly licensed set of still images. Record expected labels or boxes manually and preserve the annotation method. If the images are synthetic fixtures, label them as synthetic. Do not call a demonstration set a representative industrial test set.
Establish the execution boundary
“Local” should describe where inference executes, not imply that nothing was downloaded. A browser application may need runtime and model artifacts before it can process an image. Show the intended download and cache behavior before starting, and identify any network requests.
WebGPU availability and the selected model’s supported operators affect feasibility. A WebGPU property existing in the browser is not proof that the workload will run. If offering another backend, test it and report its actual behavior rather than assuming equivalent speed. Hugging Face
Let visitors choose a file explicitly. Keep private images out of URL parameters and analytics. A failure should not trigger an undisclosed remote-inference fallback.
Make failure visible
For detection, show the image, predicted box, score, and expected annotation when one exists. Include a text list of detections for keyboard and screen-reader users. A score should be described as the model’s output score, not a calibrated probability of being correct.
A basic box-overlap exercise can use intersection over union: overlapping area divided by the area covered by either box. Two identical boxes have an overlap ratio of 1; disjoint boxes have 0. That geometric calculation alone is not a complete detection benchmark, especially with multiple objects, incorrect classes, and unmatched detections.
Use a deliberately simple example: an expected 10 × 10 box and a prediction covering exactly half that box with no area outside it produce a ratio of 0.5. These are synthetic coordinates for testing the calculation. They do not describe a model’s performance.
Keep a small error gallery
Retain missed objects, extra detections, and incorrect labels. Group them by conditions worth investigating: scale, occlusion, lighting, background, and camera angle. The group names are hypotheses about failure patterns, not automatic causal explanations.
When changing a model or threshold, rerun the same held-out examples before celebrating. A threshold can change which outputs are displayed without improving the underlying model. Keep the previous results so the comparison remains inspectable.
Record enough to repeat it
Export the model identifier and revision, runtime version, backend, preprocessing choices, threshold, image identifiers, expected annotations, and raw predictions. Record timings as observed timings on that environment; avoid a universal performance promise.
The first successful demo earns further investigation. A repeatable set of successes and failures earns trust. That distinction is what turns an entertaining rectangle generator into a useful instrument.
Next: Tokens per second need footnotes applies the same discipline to speed claims.
Sources
- The pipeline API · Hugging Face — Hugging Face. Reviewed 15 September 2026.
- Web | onnxruntime — ONNX Runtime. Reviewed 15 September 2026.
- Running models on WebGPU · Hugging Face — Hugging Face. Reviewed 15 September 2026.
Put this to work
- Memory Workbench — Calculate a full-attention decoder’s raw cache storage; keep unknown runtime allocations visible.