Winner · Gaussian Splat · HCI · 2026

SyncScape

Aligning AI-generated virtual rooms with physical space in MR — walk freely, no motion sickness.

Team: Qinchuan Zhang · Weibo Xu
Type: Research · XR
Award: Winner · Worlds in Action Hack [01-SF]
Stack: Quest 3 · Unity MRUK · WorldLabs · Open3D
Date: 2026

Overview

SyncScape is an application framework that seamlessly bridges the virtual and real worlds in Mixed Reality. The system reconstructs the user's physical room as a stylized virtual environment inside the headset while maintaining spatial congruence between virtual and real objects — users can walk freely without colliding with physical obstacles, and because visual movement stays consistent with physical motion, vestibular–visual conflict and motion sickness are significantly reduced.

How it works

The room is captured with Quest 3's spatial scanning, then semantically abstracted in Unity: MRUK effect meshes cover every real object's bounding box with uniform white material, keeping geometry but stripping semantics. A panoramic capture of this abstract layout is sent to the WorldLabs model with a text prompt, which generates a stylized Gaussian Splatting room that preserves the identical spatial layout.

Automatic alignment

Point clouds are sampled from both the real-world bounding-box surfaces and the AI-generated splat scene, and a rigid transformation is computed with Open3D to register the virtual room against the physical environment. A hand-tracked refinement pass follows: single-hand pinch translates the room, bimanual pinch adjusts rotation and scale, with scene opacity dropped to 30% for precise visual registration.

Current research

Uniform surface sampling biases alignment because generated scenes are noisy in occluded regions and drift far from the capture camera. Ongoing work replaces it with camera-centric 360° ray sampling plus distance/occlusion-weighted matching, panoramic feature matching, and semantic color-coding of the model input. Next steps: Vision Pro support, dynamic object representation, outdoor scenes, and user studies for gaming, education, and therapy.