GenAI · 4DGS · 2026

4DGS World Lab

An AI-generated panoramic video lifted frame-by-frame into Gaussian splats — a dynamic 3D world you can fly through in the browser.

Team: Qinchuan Zhang · Wanhe Jiang
Type: Research · GenAI × 3D
Stack: Seedance 2 · Marble 1.1 API · Three.js · Spark
Live: 4dgsworldlab.vercel.app
Date: Jul 2026

Turning AI Video into a Walkable 4D World: A Seedance × World Labs Pipeline

3D Gaussian Splatting gives us photoreal, explorable scenes, but they are frozen in time. Video models give us motion, but trap it on a flat screen. This project connects the two: generate a panoramic video with Seedance 2, extract every frame, lift each frame into a 3D Gaussian splat with World Labs' Marble, then play the splats back frame by frame — producing a "4DGS" effect: a dynamic 3D world you can look and move around in while it animates, with no training involved. The pipeline sounds simple, but almost every step fails in a non-obvious way if done naively — each stage below exists to route around one of those failures.

Step 1 — The first panorama: why not just ask for one?

Everything starts from a single equirectangular panorama that seeds the whole scene, and the two obvious approaches both fail. Asking Seedance 2 to generate a panoramic video directly doesn't work: panoramic footage is a tiny fraction of any video model's training data, and the output drifts away from a valid equirectangular projection almost immediately. Asking GPT Image 2 for a panorama produces images that look panoramic but aren't — the left and right edges don't wrap seamlessly, and objects show severe distortion in a 360° viewer. My read is that this is a training-data problem, not a prompting problem. What worked is World Labs' Marble: it is built around spatial 3D generation, and its panoramas are properly wrap-consistent. So the first frame comes from Marble.

Step 2 — Constraining the video with a matching last frame

Since Seedance can't be trusted to stay panoramic over 5 seconds on its own, I constrain it from both ends: give it a first frame and a last frame with the same layout, and let it interpolate the motion between them. The catch is that Marble can't generate the last frame — it has no conversation or context understanding, so you can't tell it "same scene, but changed like this." The two keyframes therefore come from different models, each chosen for what it's good at: Marble for the geometrically correct first frame, GPT Image 2 for the last — it can understand "take this exact image and change X while keeping the layout" precisely because it has the multimodal context understanding Marble lacks. Its projection flaws matter less here, since the first frame anchors the geometry and Seedance interpolates toward the edit.

Step 3 — The aspect-ratio squeeze

Equirectangular panoramas are 2:1, and Seedance 2 doesn't support 2:1 input or output. Cropping to 16:9 would destroy the layout — you'd lose part of the 360° sphere. The fix is low-tech but effective: squash, don't crop. Both keyframes are resized from 2:1 to 16:9 by compressing the long edge, so the layout survives horizontally squeezed. Seedance takes the squeezed keyframes plus a prompt describing the scene change and returns a 16:9 "panoramic" video, which is then stretched back to 2:1. The round trip costs some horizontal resolution, but the equirectangular structure — the thing that actually matters — survives intact.

Step 4 — 120 frames, 120 splats: via API, not UI

The video is 5 seconds at 24 fps: 120 frames. Reconstructing them one at a time in Marble Studio would take forever, so this stage runs entirely through the Marble 1.1 API — extract every frame, submit each as a panorama-to-splat generation job, download 120 .spz splat files. One knob matters here: fixing the random seed across all 120 generations. Each frame is reconstructed independently — the model has no idea frame 47 should agree with frame 46 — so a fixed seed is the only consistency lever available. It helps, with limits (see below).

Web playback

The result plays back in a Three.js viewer built on Spark, swapping which splat is mounted in the scene graph each tick — 24 splat switches per second at full quality, with WASD fly controls. Because shipping 120 splats means ~864 MB of downloads, the viewer defaults to a sparse 6-frame preview (~43 MB) covering the same 5-second timeline, and lets users opt into 24/60/120-frame tiers — each labeled with its size, a download-time estimate from measured bandwidth, and a hardware recommendation. Device detection locks the heavy tiers on phones and standalone headsets, with a manual unlock for anyone who wants to gamble.

Honest drawbacks

Two problems remain, and both are fundamental to independent per-frame reconstruction. Performance: playing 4DGS as 120 discrete splats is brute force — every switch remounts a 500k-splat mesh and triggers a re-sort, and memory holds the whole sequence. This is a flipbook of 3D scenes, with a flipbook's costs, not a true 4DGS representation with a shared canonical space. Temporal consistency: even with a fixed seed, each splat is generated independently, so geometry that should be static shimmers locally between frames — surfaces breathe, thin structures flicker. Both point at the same future fix: reconstruction that is temporally aware, either true 4DGS training across the sequence or per-frame generation conditioned on the previous frame's splat.

Takeaway

No single model can do this today. Marble can't animate, Seedance can't do panoramas unaided, GPT Image 2 can't do valid projections. But each one is excellent at exactly the thing the others can't do — and a pipeline that squeezes an aspect ratio here and fixes a seed there is enough to get a generated, animated, explorable 3D world running in a browser tab.