RiCS: A 2D Self-Occlusion Map for Harmonizing Volumetric Objects

by   Yunseok Jang, et al.

There have been remarkable successes in computer vision with deep learning. While such breakthroughs show robust performance, there have still been many challenges in learning in-depth knowledge, like occlusion or predicting physical interactions. Although some recent works show the potential of 3D data in serving such context, it is unclear how we efficiently provide 3D input to the 2D models due to the misalignment in dimensionality between 2D and 3D. To leverage the successes of 2D models in predicting self-occlusions, we design Ray-marching in Camera Space (RiCS), a new method to represent the self-occlusions of foreground objects in 3D into a 2D self-occlusion map. We test the effectiveness of our representation on the human image harmonization task by predicting shading that is coherent with a given background image. Our experiments demonstrate that our representation map not only allows us to enhance the image quality but also to model temporally coherent complex shadow effects compared with the simulation-to-real and harmonization methods, both quantitatively and qualitatively. We further show that we can significantly improve the performance of human parts segmentation networks trained on existing synthetic datasets by enhancing the harmonization quality with our method.


page 1

page 3

page 5

page 7


Blocks World Revisited: The Effect of Self-Occlusion on Classification by Convolutional Neural Networks

Despite the recent successes in computer vision, there remain new avenue...

Seeing Behind Dynamic Occlusions with Event Cameras

Unwanted camera occlusions, such as debris, dust, rain-drops, and snow, ...

RobustFusion: Robust Volumetric Performance Reconstruction under Human-object Interactions from Monocular RGBD Stream

High-quality 4D reconstruction of human performance with complex interac...

Model-based disentanglement of lens occlusions

With lens occlusions, naive image-to-image networks fail to learn an acc...

Mixed Reality Depth Contour Occlusion Using Binocular Similarity Matching and Three-dimensional Contour Optimisation

Mixed reality applications often require virtual objects that are partly...

Future Localization from an Egocentric Depth Image

This paper presents a method for future localization: to predict a set o...

Bicameral Structuring and Synthetic Imagery for Jointly Predicting Instance Boundaries and Nearby Occlusions from a Single Image

Oriented boundary detection is a challenging task aimed at both delineat...

Please sign up or login with your details

Forgot password? Click here to reset