Winner · HCI · ML · Hardware · XR

HandSpeaker

Real-time sign language translation on AR glasses — understanding motion as language.

Team: Qinchuan Zhang · Weibo Xu
Type: Research · HCI
Award: Winner · Shift SC Hack
Stack: AR Spectacles · Hand Tracking · WebSocket
Focus: Spatiotemporal gesture understanding

Overview

Sign language is inherently spatiotemporal — meaning depends not just on hand shape but on motion, trajectory, and timing. Most digital solutions stop at static gesture classification. HandSpeaker is a real-time sign language translation system built on AR hand tracking that treats recognition as a sequence problem: understanding motion as language.

How it works

Instead of RGB images or depth sensors, the system operates directly on 3D hand joint transforms streamed from AR Spectacles. Motion is decomposed into spatial features (joint positions), motion features (trajectory and velocity), and structural features (joint angles). Static and dynamic gestures are unified as temporal sequences — short for poses, longer for trajectories — classified by a lightweight pipeline and pushed over WebSocket to text and speech output with demo-grade latency.

Data pipeline

No dataset exists for 3D hand-joint trajectories, so the project includes its own: trigger-based 3-second recordings flow through upload, timeline playback, manual labeling, and dataset management in a web interface — turning every user interaction into training-ready data for the next stage: LSTM/Transformer models, sentence-level translation, and continuous unsegmented signing.