Digital Video Stabilization and Rolling Shutter Correction using Gyroscopes
📜 Abstract
In this paper we present a robust, real-time video stabilization and rolling shutter correction technique based on commodity gyroscopes. First, we develop a unified algorithm for modeling camera motion and rolling shutter warping. We then present a novel framework for automatically calibrating the gyroscope and camera outputs from a single video capture. This calibration allows us to use only gyroscope data to effectively correct rolling shutter warping and to stabilize the video. Using our algorithm, we show results for videos featuring large moving foreground objects, parallax, and low-illumination. We also compare our method with commercial image-based stabilization algorithms. We find that our solution is more robust and computationally inexpensive. Finally, we implement our algorithm directly on a mobile phone. We demonstrate that by using the phone’s inbuilt gyroscope and GPU, we can remove camera shake and rolling shutter artifacts in real-time.
✨ Summary
The paper presents a real-time method for video stabilization and rolling-shutter correction using commodity MEMS gyroscopes. It models camera motion as rotation, integrates gyroscope measurements to estimate orientation, and accounts for the fact that different image rows are captured at different times. Stabilization and rolling-shutter correction are expressed as a single warp from the observed camera motion to a synthetic camera with smoothed motion and a global shutter.
The method automatically estimates camera focal length, rolling-shutter duration, gyroscope-to-camera timing delay, gyroscope drift, and axis correspondence from one approximately ten-second calibration recording. Feature matching and RANSAC provide image correspondences for optimizing these parameters, after which the correction can operate using gyroscope data alone. The image warp is implemented as a piecewise-linear mesh, with approximately ten vertical subdivisions typically sufficient for the reported videos.
Experiments on iPhone 4 footage show robustness in scenes with moving foreground objects, parallax, poor illumination, and lens flare. The authors report better behavior than the tested commercial image-based systems, particularly when feature tracking is unreliable or rolling-shutter motion is nonlinear. A GPU implementation produces a corrected and stabilized viewfinder at 30 frames per second on the iPhone 4.
The principal limitations are the rotation-only motion model, loss of field of view caused by cropping empty regions after warping, and residual motion blur in low-light footage. The paper’s approach became a documented reference point for subsequent inertial video-stabilization and rolling-shutter systems: later patents explicitly describe or incorporate its unified gyro-based warping and calibration concepts, while subsequent academic work cites it as prior work for gyroscope-based stabilization and rolling-shutter correction. (patents.google.com)