4D Imaging through Spray-On Optics

Julian Iseringhausen, Bastian Goldl├╝cke, Nina Pesheva, Stanimir Iliev, Alexander Wender, Martin Fuchs, Matthias B. Hullin
ACM Transactions on Graphics 36(4) (Proc. SIGGRAPH), 35:1--35:11, 2017.


Light fields are a powerful concept in computational imaging and a mainstay in image-based rendering; however, so far their acquisition required either carefully designed and calibrated optical systems (micro-lens arrays), or multi-camera/multi-shot settings. Here, we show that fully calibrated light field data can be obtained from a single ordinary photograph taken through a partially wetted window. Each drop of water produces a distorted view on the scene, and the challenge of recovering the unknown mapping from pixel coordinates to refracted rays in space is a severely underconstrained problem. The key idea behind our solution is to combine ray tracing and low-level image analysis techniques (extraction of 2D drop contours and locations of scene features seen through drops) with state-of-the-art drop shape simulation and an iterative refinement scheme to enforce photo-consistency across features that are seen in multiple views. This novel approach not only recovers a dense pixel-to-ray mapping, but also the refractive geometry through which the scene is observed, to high accuracy. We therefore anticipate that our inherently self-calibrating scheme might also find applications in other fields, for instance in materials science where the wetting properties of liquids on surfaces are investigated.


BibTeX Citation

@article{IseringhausenSIG2017, author = {Julian Iseringhausen and Bastian Goldl"ucke and Nina Pesheva and Stanimir Iliev and Alexander Wender and Martin Fuchs and Matthias B. Hullin}, title = {{4D} Imaging through Spray-On Optics}, journal = {ACM Trans. Graph. (Proc. SIGGRAPH 2017)}, volume = {36}, number = {4}, year = {2017}, pages = {35:1--35:11}, }