SplatXR: A Toolkit for Multi-Layered Extended Reality via On-Headset 3D Gaussian Splatting

Abstract

Recent advances in Gaussian splatting, 3D foundation models, and video-passthrough mixed reality offer unprecedented fidelity for capturing and blending physical and virtual worlds. However, established XR methods rely on polygon-based pipelines that treat physical and digital environments as separate domains, making seamless integration across recent novel representations difficult. Because splats lack explicit surface geometry, they impede standard graphics techniques necessary for cross-reality interactions, such as shadow or collision handling. Furthermore, while 3D foundation models can reconstruct physical objects on-the-fly into digital splats, current XR systems fall short of leveraging this capability for user experiences that transition seamlessly between physicality and virtuality. To address this, we organize XR as a multi-layered reality of interacting primitives, letting pixels and objects move across physical and digital boundaries. We contribute SplatXR, a Unity toolkit that operationalizes this structure as building blocks for interactive, immersive apps leveraging 3D Gaussian splatting: a standalone renderer, splat-to-mesh reconstruction, splat physics, splat blending, and offloaded splat object reconstruction via on-headset capture. SplatXR unites vision and graphics techniques to make splat-based XR development broadly available, and we validate it through demonstrative applications and a small developer study.