paper

KinectFusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth Camera*

  • Authors:
Read the paper

📜 Abstract

KinectFusion enables a user holding and moving a standard Kinect camera to rapidly create detailed 3D reconstructions of an indoor scene. Only the depth data from Kinect is used to track the 3D pose of the sensor and reconstruct, geometrically precise, 3D models of the physical scene in real-time. The capabilities of KinectFusion, as well as the novel GPU-based pipeline are described in full. We show uses of the core system for low-cost handheld scanning, and geometry-aware augmented reality and physics-based interactions. Novel extensions to the core GPU pipeline demonstrate object segmentation and user interaction directly in front of the sensor, without degrading camera tracking or reconstruction. These extensions are used to enable real-time multi-touch interactions anywhere, allowing any planar or non-planar reconstructed physical surface to be appropriated for touch.

✨ Summary

KinectFusion introduced a real-time pipeline for dense RGB-D reconstruction using a moving commodity depth camera and GPU computation. Its central design combines dense point-to-plane ICP tracking, volumetric truncated signed-distance-function integration, and GPU raycasting. The paper also demonstrated applications including handheld scanning, geometry-aware augmented reality, physics simulation, foreground segmentation, dynamic-object reconstruction, and touch interaction on planar and non-planar surfaces. (microsoft.com)

  • Research influence: Later systems explicitly identify KinectFusion as a major basis for volumetric RGB-D reconstruction. InfiniTAM extends this approach with replaceable tracking and integration components, scalable volumetric representations, hashed voxel blocks, and mechanisms for globally consistent reconstruction. (arxiv.org)
  • Dense SLAM: ElasticFusion cites KinectFusion as prior work and develops an alternative real-time dense SLAM system that avoids a pose graph, illustrating continued research into improving the scalability and global consistency of dense scene mapping. (journals.sagepub.com)
  • Research infrastructure: Microsoft’s 7-Scenes RGB-D dataset used a KinectFusion implementation to generate dense scene models and ground-truth camera tracks, supporting subsequent evaluation of dense tracking, mapping, and relocalization methods. (microsoft.com)
  • Industry and mixed reality: A Microsoft REF 2021 impact submission reports that KinectFusion was important to the environment-mapping approach used in HoloLens 1 and 2. Microsoft’s later mixed-reality documentation describes spatial mapping as a detailed representation of real-world surfaces used for stable hologram placement, occlusion, raycasting, and collision-related applications. (ref2021-resultsapp-live.azurewebsites.net)

Overall, the documented impact is the normalization of real-time volumetric depth fusion as a practical foundation for dense SLAM, 3D scanning, spatial mapping, and geometry-aware mixed-reality interaction.