Scene keyframes

Tested browsers and formats

From automated browser tests on one machine (macOS 26, Apple M1), run on 9 October 2026 (library 1.1.1, ONNX Runtime Web 1.30.0). Browser automation on one machine doesn't show that other GPUs, drivers or devices work. Report: reports/compatibility/2026-10-09-20-06.json.

Formats a browser can't decode fail cleanly with a "playback" error rather than wrong results; the tests check that too.

The sample-video checks compare keyframe counts with what Chrome gave (±2). Chromium and WebKit decode a little differently and come out 3 off on one clip each, so those checks fail there; decoding itself works (see the format table).

How it works

        your video file  (stays in the browser, nothing is uploaded)
              |
              v
        1. SAMPLE    seek a hidden <video> to 2 frames per second
                     (max 180), draw each frame once:
                       - ~224x224 px copy for the model (aspect kept)
                       - sharpness + brightness score
              |
              v
        2. EMBED     DINOv2-small (fp16) via ONNX Runtime Web,
                     on WebGPU or the WASM fallback:
                     each frame -> 384 numbers describing what it shows
              |      runs while the next frame is being seeked
              v
        3. SELECT    re-runs instantly when you move the slider
              |        a. drop dark, overexposed and blurry frames
              |        b. mean-center the embeddings for this clip
              |        c. group consecutive frames of the same view
              |        d. keep the sharp frame nearest each group's centroid
              |        e. drop views the camera returns to later
              v
        4. SHOW      keyframe timestamps in time order; thumbnails
                     are captured on demand; click one to seek

        [frame][frame][frame][frame][frame][frame][frame][frame]
         \_____ view A _____/ \__ view B __/ \_____ view C ____/
                |                   |                 |
             keyframe            keyframe          keyframe

Video credits

The sample videos are clips from Apple's ARKitScenes dataset (iPhone captures of real homes), used under the ARKitScenes license. They have been trimmed to 25 seconds and re-encoded (main camera, 1280 px wide, no audio). This demo is not affiliated with or endorsed by Apple.

Dataset: G. Baruch, Z. Chen, A. Dehghan, T. Dimry, Y. Feigin, P. Fu, T. Gebauer, B. Joffe, D. Kurz, A. Schwartz and E. Shulman, "ARKitScenes: A Diverse Real-World Dataset for 3D Indoor Scene Understanding Using Mobile RGB-D Data", NeurIPS 2021 Datasets and Benchmarks Track.