Learning-based Sampling for Natural Image Matting

Abstract

📜 Abstract

The goal of natural image matting is the estimation of opacities of a user-defined foreground object that is essential in creating realistic composite imagery. Natural matting is a challenging process due to the high number of unknowns in the mathematical modeling of the problem, namely the opacities as well as the foreground and background layer colors, while the original image serves as the single observation. In this paper, we propose the estimation of the layer colors through the use of deep neural networks prior to the opacity estimation. The layer color estimation is a better match for the capabilities of neural networks, and the availability of these colors substantially increase the performance of opacity estimation due to the reduced number of unknowns in the compositing equation. A prominent approach to matting in parallel to ours is called sampling-based matting, which involves gathering color samples from known-opacity regions to predict the layer colors. Our approach outperforms not only the previous hand-crafted sampling algorithms, but also current data-driven methods. We hence classify our method as a hybrid sampling- and learning-based approach to matting, and demonstrate the effectiveness of our approach through detailed ablation studies using alternative network architectures.

Description

✨ Summary

The paper proposes decomposing natural image matting into layer-color estimation followed by alpha-opacity estimation. Instead of asking one network to infer the alpha matte while simultaneously accounting for unknown foreground and background colors, the method first predicts the hidden background and foreground colors in the unknown-opacity region. A final matting network then uses the input image, trimap, and predicted layer colors to estimate the alpha matte.

The background is estimated with a modified image-inpainting network. The input image is provided as guidance, and the background loss is evaluated only in the unknown-opacity region, encouraging the network to recover the actual obscured background rather than hallucinate merely plausible content. The foreground sampling network uses the predicted background as an additional input and is trained with both an L1 foreground-color loss and a compositional loss that measures reconstruction of the observed image.

The final matting network combines alpha and compositional losses with an alpha-gradient loss. The gradient term is intended to preserve sharp, high-frequency transitions that can be blurred when only pixelwise alpha and compositional losses are used. The system also employs extensive dynamic augmentation: foreground layering, random trimap dilation, horizontal flipping, randomly selected MS COCO backgrounds, scale variation, and color transformations of foreground images.

Experiments on the Adobe matting dataset and the standard alpha-matting benchmark show that predicted background colors provide the largest ablation improvement, while predicted foreground colors provide an additional but smaller improvement. Dynamic augmentation improves both SAD and MSE, and the gradient loss reduces undesirable blur. The sampling predictions also improve performance when used with more than one final matting architecture, suggesting that the main contribution is not tied to a single alpha-prediction network.

On the benchmark reported in the paper, the proposed method achieved the strongest overall results among the compared methods at the time of submission and required approximately 0.6 seconds per benchmark image. Because foreground colors are estimated before alpha prediction, the output can also be used for compositing without a separate foreground-color post-processing step.

The method can fail when its background or foreground color predictions are inaccurate. In such cases, supplying erroneous color estimates to the final matting network may produce a worse matte than a model that does not use sampling predictions. The authors identify broader and more diverse training data for the sampling networks as a possible way to reduce these failures.

The paper is cited in later image-matting surveys and research as an example of learning-based or hybrid sampling-based matting, placing it within the transition from hand-crafted color-sampling methods toward neural methods that exploit image structure and texture. (mdpi.com) A related patent family describes an implementation using background-sampling, foreground-sampling, and matting networks, and explicitly documents applications of the same general pipeline to image matting. (patents.google.com) The available evidence supports academic continuation and technology disclosure; it does not establish a specific commercial product deployment attributable to this paper. The CVF record identifies the paper as a CVPR 2019 publication by the listed authors, with the conference publication occurring in June 2019. (openaccess.thecvf.com)