Deep Photo Style Transfer
📜 Abstract
This paper introduces a deep-learning approach to photographic style transfer that handles a large variety of image content while faithfully transferring the reference style. Our approach builds upon the recent work on painterly transfer that separates style from the content of an image by considering different layers of a neural network. However, as is, this approach is not suitable for photorealistic style transfer. Even when both the input and reference images are photographs, the output still exhibits distortions reminiscent of a painting. Our contribution is to constrain the transformation from the input to the output to be locally affine in colorspace, and to express this constraint as a custom fully differentiable energy term. We show that this approach successfully suppresses distortion and yields satisfying photorealistic style transfers in a broad variety of scenarios, including transfer of the time of day, weather, season, and artistic edits.
✨ Summary
Summary
The paper extends neural style transfer to photographic editing by combining three objectives: content preservation, semantic style matching, and a photorealism regularizer based on the Matting Laplacian. The regularizer constrains the input-to-output transformation to be locally affine in color space, reducing geometric distortions while permitting spatially varying color changes. Semantic segmentation masks guide style matching between corresponding regions, reducing artifacts caused by unrelated content being matched across the images. The method was evaluated on changes in illumination, time of day, weather, season, and artistic appearance. The reported user studies found that the method was judged photorealistic and was preferred for style faithfulness in 86.8% of comparisons against the evaluated global color-transfer methods. (openaccess.thecvf.com)
Subsequent research explicitly used this work as a reference point for photorealistic style transfer. Later methods, including PhotoWCT and WCT², pursued feed-forward or more efficient alternatives to the paper’s iterative optimization and addressed reconstruction artifacts and runtime limitations. WCT², for example, compared itself directly with Deep Photo Style Transfer and reported substantially faster high-resolution processing. (openaccess.thecvf.com) Other follow-up work adapted the approach for automated semantic segmentation and for portrait-specific transfer, indicating that the paper’s locally affine photorealism constraint and semantic-region guidance became established design ideas in subsequent research. (arxiv.org)