Chinese
Adv Search
Home | Accepted | Article In Press | Current Issue | Archive | Special Issues | Collections | Featured Articles | Statistics

2020, 2(3): 247-260 Published Date:2020-6-20

DOI: 10.1016/j.vrih.2020.04.004

Interactive free-viewpoint video generation

Full Text: PDF (9) HTML (87)

Export: EndNote | Reference Manager | ProCite | BibTex | RefWorks

Abstract:

Background
Free-viewpoint video (FVV) is processed video content in which viewers can freely select the viewing position and angle. FVV delivers an improved visual experience and can also help synthesize special effects and virtual reality content. In this paper, a complete FVV system is proposed to interactively control the viewpoints of video relay programs through multimedia terminals such as computers and tablets.
Methods
The hardware of the FVV generation system is a set of synchronously controlled cameras, and the software generates videos in novel viewpoints from the captured video using view interpolation. The interactive interface is designed to visualize the generated video in novel viewpoints and enable the viewpoint to be changed interactively.
Results
Experiments show that our system can synthesize plausible videos in intermediate viewpoints with a view range of up to 180°.
Keywords: Free-viewpoint video ; View interpolation ; Interactive interface

Cite this article:

Yanru WANG, Zhihao HUANG, Hao ZHU, Wei LI, Xun CAO, Ruigang YANG. Interactive free-viewpoint video generation. Virtual Reality & Intelligent Hardware, 2020, 2(3): 247-260 DOI:10.1016/j.vrih.2020.04.004

1. Tanimoto M, Tehrani M, Fujii T, Yendo T. Free-viewpoint TV. IEEE Signal Processing Magazine, 2011, 28(1): 67–76 DOI:10.1109/msp.2010.939077

2. Seitz S M, Curless B, Diebel J, Scharstein D, Szeliski R. A comparison and evaluation of multi-view stereo reconstruction algorithms. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York, NY, USA, IEEE, 2006, 519–528 DOI:10.1109/cvpr.2006.19

3. Zhu H, Nie Y M, Yue T, Cao X. The role of prior in image based 3D modeling: a survey. Frontiers of Computer Science, 2017, 11(2): 175–191 DOI:10.1007/s11704-016-5520-8

4. Seitz S M. Photorealistic scene reconstruction by voxel coloring. IEEE Conference on Computer Vision and Pattern Recognition Conference, 1997

5. Chen J, Watanabe R, Nonaka K, Konno T, Sankoh H, Naito S. Fast free-viewpoint video synthesis algorithm for sports scenes. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Macau, China, IEEE, 2019 DOI:10.1109/iros40897.2019.8967584

6. Miller G, Hilton A, Starck J. Interactive free-viewpoint video. IEEE European Conference on Visual Media Production, 2005

7. Shum H, Kang S B. Review of image-based rendering techniques. Visual Communications and Image Processing, 2000

8. McMillan L, Bishop G. Plenoptic modeling. In: Proceedings of the 22nd annual conference on Computer graphics and interactive techniques. New York, USA, ACM Press, 1995 DOI:10.1145/218380.218398

9. Levoy M, Hanrahan P. Light field rendering. ACM Transactions on Graphics, 1996

10. Hedman P, Kopf J. Instant 3D photography. ACM Transactions on Graphics, 2018, 37(4): 1–12 DOI:10.1145/3197517.3201384

11. Hedman P, Philip J, Price T, Frahm J M, Drettakis G, Brostow G. Deep blending for free-viewpoint image-based rendering. ACM Transactions on Graphics, 2019, 37(6): 1–15 DOI:10.1145/3272127.3275084

12. Chaurasia G, Duchene S, Sorkine-Hornung O, Drettakis G. Depth synthesis and local warps for plausible image-based navigation. ACM Transactions on Graphics, 2013, 32(3): 1–12 DOI:10.1145/2487228.2487238

13. Hedman P, Ritschel T, Drettakis G, Brostow G. Scalable inside-out image-based rendering. ACM Transactions on Graphics, 2016, 35(6): 1–11 DOI:10.1145/2980179.2982420

14. Zitnick C L, Kang S B, Uyttendaele M, Winder S, Szeliski R. High-quality video view interpolation using a layered representation. ACM Transactions on Graphics, 2004, 23(3): 600 DOI:10.1145/1015706.1015766

15. Zhu H, Zuo X X, Wang S, Cao X, Yang R G. Detailed human shape estimation from a single image by hierarchical mesh deformation. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA, IEEE, 2019 DOI:10.1109/cvpr.2019.00462

16. Flynn J, Neulander I, Philbin J, Snavely N. Deep stereo: learning to predict new views from the world's imagery. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA, IEEE, 2016 DOI:10.1109/cvpr.2016.595

17. Zhou T H, Tucker R, Flynn J, Fyffe G, Snavely N. Stereo magnification: learning view synthesis using multiplane images. 2018

18. Penner E, Zhang L. Soft 3D reconstruction for view synthesis. ACM Transactions on Graphics, 2017, 36(6): 1–11 DOI:10.1145/3130800.3130855

19. Smolic A. 3D video and free viewpoint video: From capture to display. Pattern Recognition, 2011, 44(9): 1958–1968 DOI:10.1016/j.patcog.2010.09.005

20. Zhu H, Su H, Wang P, Cao X, Yang R G. View extrapolation of human body from a single image. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, IEEE, 2018 DOI:10.1109/cvpr.2018.00468

21. Collet A, Chuang M, Sweeney P, Gillett D, Evseev D, Calabrese D, Hoppe H, Kirk A, Sullivan S. High-quality streamable free-viewpoint video. ACM Transactions on Graphics, 2015, 34(4): 1–13 DOI:10.1145/2766945

22. Debevec P E, Taylor C J, Malik J. Modeling and rendering architecture from photographs. In: Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. New York, USA, ACM Press, 1996 DOI:10.1145/237170.237191

23. Montemerlo M, Thrun S, Koller D, Wegbreit B. A factored solution to the simultaneous localization and mapping problem. Conference on Artificial Intelligence, 2002

24. Sturm P, Triggs B. A factorization based algorithm for multi-image projective structure and motion//Lecture Notes in Computer Science. Berlin, Heidelberg, Springer Berlin Heidelberg, 1996, 709–720 DOI:10.1007/3-540-61123-1_183

25. Tan F, Zhu H, Cui Z. Self-supervised human depth estimation from monocular videos. IEEE Conference on Computer Vision and Pattern Recognition, 2020

26. Yang H, Zhu H, Wang Y, Huang M, Shen Q, Yang R, Cao X. FaceScape: a large-scale high quality 3D face dataset and detailed riggable 3D face prediction. IEEE Conference on Computer Vision and Pattern Recognition, 2020

27. Bosc E, Pepion R, Le Callet P, Koppel M, Ndjiki-Nya P, Pressigout M, Morin L. Towards a new quality metric for 3-D synthesized view assessment. IEEE Journal of Selected Topics in Signal Processing, 2011, 5(7): 1332–1343 DOI:10.1109/jstsp.2011.2166245

28. Ceulemans B, Lu S P, Lafruit G, Munteanu A. Robust multiview synthesis for wide-baseline camera arrays. IEEE Transactions on Multimedia, 2018, 20(9): 2235–2248 DOI:10.1109/tmm.2018.2802646

29. Niklaus S, Mai L, Yang J M, Liu F. 3D Ken Burns effect from a single image. ACM Transactions on Graphics, 2019, 38(6): 1–15 DOI:10.1145/3355089.3356528

30. Regmi K, Borji A. Cross-view image synthesis using conditional GANs. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA, IEEE, 2018, 3501–3510 DOI:10.1109/cvpr.2018.00369

31. Lu Y L, Sun T F, Jiang X H, Xu K, Zhu B. Frontal view synthesis based on a novel GAN with global and local discriminators. In: 2019 12th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics. Suzhou, China, IEEE, 2019, 1–5 DOI:10.1109/cisp-bmei48845.2019.8965829

32. Wang Y L, Liu F, Wang Z L, Hou G Q, Sun Z N, Tan T N. End-to-end view synthesis for light field imaging with pseudo 4DCNN//Computer Vision–ECCV 2018. Cham: Springer International Publishing, 2018, 340–355 DOI:10.1007/978-3-030-01216-8_21

33. Niklaus S, Mai L, Liu F. Video frame interpolation via adaptive separable convolution. In: 2017 IEEE International Conference on Computer Vision (ICCV). Venice, IEEE, 2017, 261–270 DOI:10.1109/iccv.2017.37

34. Liu Z W, Yeh R A, Tang X O, Liu Y M, Agarwala A. Video frame synthesis using deep voxel flow. In: 2017 IEEE International Conference on Computer Vision. Venice, IEEE, 2017, 4463–4471 DOI:10.1109/iccv.2017.478

35. Niklaus S, Liu F. Context-aware synthesis for video frame interpolation. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, IEEE, 2018, 1701–1710 DOI:10.1109/cvpr.2018.00183

36. Zhou T H, Tulsiani S, Sun W L, Malik J, Efros A A. View synthesis by appearance flow//Computer Vision–ECCV 2016. Cham: Springer International Publishing, 2016, 286–301 DOI:10.1007/978-3-319-46493-0_18

37. Park E, Yang J M, Yumer E, Ceylan D, Berg A C. Transformation-grounded image generation network for novel 3D view synthesis. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, IEEE, 2017 DOI:10.1109/cvpr.2017.82

38. Kalantari N K, Wang T C, Ramamoorthi R. Learning-based view synthesis for light field cameras. ACM Transactions on Graphics, 2016, 35(6): 1–10 DOI:10.1145/2980179.2980251

39. Ji D H, Kwon J, McFarland M, Savarese S. Deep view morphing. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, IEEE, 2017 DOI:10.1109/cvpr.2017.750

40. Zhou T H, Tucker R, Flynn J, Fyffe G, Snavely N. Stereo magnification: learning view synthesis using multiplane images. 2018

41. Lam E Y. Combining gray world and retinex theory for automatic white balance in digital photography. In: Proceedings of the Ninth International Symposium on Consumer Electronics. Macau SAR, IEEE, 2005 DOI:10.1109/isce.2005.1502356

42. Johnson J, Alahi A, Li F F. Perceptual losses for real-time style transfer and super-resolution//Computer Vision–ECCV 2016. Cham: Springer International Publishing, 2016, 694–711 DOI:10.1007/978-3-319-46475-6_43

43. Furukawa Y, J.Accurate Ponce, dense, and robust multiview stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(8): 1362–1376 DOI:10.1109/tpami.2009.161

44. Pérez P, Gangnet M, Blake A. Poisson image editing. ACM Transactions on Graphics, 2003, 22(3): 313 DOI:10.1145/882262.882269

45. Bleyer M, Gelautz M, Rother C, Rhemann C. A stereo approach that handles the matting problem via image warping. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL, IEEE, 2009 DOI:10.1109/cvpr.2009.5206656

46. Sun D Q, Roth S, Black M J. Secrets of optical flow estimation and their principles. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Francisco, CA, USA, IEEE, 2010, 2432–2439 DOI:10.1109/cvpr.2010.5539939

47. Bao W B, Lai W S, Ma C, Zhang X Y, Gao Z Y, Yang M H. Depth-aware video frame interpolation. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA, IEEE, 2019, 3703–3712 DOI:10.1109/cvpr.2019.00382

email E-mail this page

Articles by authors

VRIH