Stereo to Multi-View Conversion

Project Members

Nikolce Stefanoski (Disney Research Zurich)
Manuel Lang (Disney Research Zurich/ETH Zurich)
Oliver Wang (Disney Research Zurich)
Simon Heinzle (Disney Research Zurich)
Aljoscha Smolic (Disney Research Zurich)

 

PROJECT_StereoMultiViewConversion_teaser

A transmission and view synthesis system, which we proposed for standardization to MPEG, consisting of an encoder and decoder for stereoscopic video and a stereo to multi-view conversion module which enables glasses-free stereoscopic viewing on an autostereoscopic display.

Content creation for autostereoscopic displays is an unresolved problem. Typical methods rely on view synthesis based on depth image based rendering (DIBR). DIBR also forms the core of a corresponding standardization activity in MPEG, which correspondingly aims at efficient coding and transmission of multiview video plus depth (MVD) data to support MADs. However, this approach relies on depth estimation, which is an ill-posed and unresolved task so far. It is highly questionable if automatic depth estimation can be resolved with sufficient accuracy, reliability and robustness in the near future. Our method applies purely image domain warping instead. Input video is analyzed and information about sparse disparity, vertical edges and saliency is extracted. A constrained energy minimization problem is formulated and efficiently solved. The resulting image warping functions are used to synthesize novel views. Our approach is fully automatic, accurate, and reliable. Disocclusions and related artifacts are avoided due to smooth, saliency-driven warping functions. Our method also works well for extrapolation of views in a limited range, thus supporting multiview creation from stereo input, which is the most relevant use case scenario.

Our method was used as part of a proposal to MPEG for a standard on 3D Video Technology. There it was evaluated in the scope of a subjective testing procedure, which was conducted anonymously, and in which more than 600 subjects participated. Our proposal was always ranked among the four best performing approaches where the concrete rank depends on the testing category

Publications

PROJECT_Automatic-View-Synthesis-by-Image-Domain-Warping_IEEE2013_THUMBNAIL (1)

Automatic View Synthesis by Image-Domain-Warping
June 6, 2013
IEEE Transactions on Image Processing 2013
Paper File [pdf, 1.99 MB]

Copyright Notice

The documents contained in these directories are included by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.