Chinese
Adv Search
Home | Accepted | Article In Press | Current Issue | Archive | Special Issues | Collections | Featured Articles | Statistics

2019, 1(5): 483-499 Published Date:2019-10-20

DOI: 10.1016/j.vrih.2019.09.003

Survey of 3D modeling using depth cameras

Full Text: PDF (21) HTML (702)

Export: EndNote | Reference Manager | ProCite | BibTex | RefWorks

Abstract:

Three-dimensional (3D) modeling is an important topic in computer graphics and computer vision. In recent years, the introduction of consumer-grade depth cameras has resulted in profound advances in 3D modeling. Starting with the basic data structure, this survey reviews the latest developments of 3D modeling based on depth cameras, including research works on camera tracking, 3D object and scene reconstruction, and high-quality texture reconstruction. We also discuss the future work and possible solutions for 3D modeling based on the depth camera.
Keywords: 3D Modeling ; Depth camera ; Camera tracking ; Signed distance function ; Surfel

Cite this article:

Hantong XU, Jiamin XU, Weiwei XU. Survey of 3D modeling using depth cameras. Virtual Reality & Intelligent Hardware, 2019, 1(5): 483-499 DOI:10.1016/j.vrih.2019.09.003

1. Seitz S M, Curless B, Diebel J, Scharstein D, Szeliski R. A comparison and evaluation of multi-view stereo reconstruction algorithms. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Volume 1 (CVPR'706), New York, NY, USA, 519–528 DOI:10.1109/cvpr.2006.19

2. Davison. Real-time simultaneous localisation and mapping with a single camera. In: Proceedings of Ninth IEEE International Conference on Computer Vision. Nice, France, IEEE, 2003 DOI:10.1109/iccv.2003.1238654

3. Klein G, Murray D. Parallel tracking and mapping for small AR workspaces. In: 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality. Nara, Japan, IEEE, 2007 DOI:10.1109/ismar.2007.4538852

4. Rusinkiewicz S, Hall-Holt O, Levoy M. Real-time 3D model acquisition. ACM Transactions on Graphics, 2002, 21(3): 438–446 DOI:10.1145/566654.566600

5. Izadi S, Kim D, Hilliges O, Molyneaux D, Newcombe R, Kohli P, Shotton J, Hodges S, Freeman D, Davison A, Fitzgibbon A. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In:Proceedings of the 24th annual ACM symposium on User interface software and technology, 2011, 559–568 DOI:10.1145/2047196.2047270

6. Newcombe R A, Davison A J, Izadi S, Kohli P, Hilliges O, Shotton J, Molyneaux D, Hodges S, Kim D, Fitzgibbon A. KinectFusion: Real-time dense surface mapping and tracking. In: 2011 10th IEEE International Symposium on Mixed and Augmented Reality. Basel, New York, USA, IEEE, 2011, 127–136 DOI:10.1109/ismar.2011.6092378

7. Curless B, Levoy M. A volumetric method for building complex models from range images. In: Proceedings of the 23rd annual conference on Computer graphics and interactive techniques-SIGGRAPH '96. New York, USA, ACM Press, 1996, 303–312 DOI:10.1145/237170.237269

8. Whelan T, Kaess M, Fallon M, Johannsson H, Leonard J, McDonald J. Kintinuous: Spatially extended KinectFusion. RSS Workshop on RGB-D: Advanced Reasoning with Depth Cameras, 2012

9. Roth H, Vona M. Moving volume KinectFusion. In: Procedings of the British Machine Vision Conference. British Machine Vision Association, 2012, 1–11 DOI:10.5244/c.26.112

10. Frisken S F, Perry R N, Rockwood A P, Jones T R. Adaptively sampled distance Fields. In: Proceedings of the 27th annual conference on Computer graphics and interactive techniques-SIGGRAPH '00. New York, USA, ACM Press, 2000, 249–254 DOI:10.1145/344779.344899

11. Sun X, Zhou K, Stollnitz E, Shi J Y, Guo B N. Interactive relighting of dynamic refractive objects. ACM Transactions on Graphics, 2008, 27(3): 1–9 DOI:10.1145/1360612.1360634

12. Zhou K, Gong M M, Huang X, Guo B N. Data-parallel octrees for surface reconstruction. IEEE Transactions on Visualization and Computer Graphics, 2011, 17(5): 669–681 DOI:10.1109/tvcg.2010.75

13. Zeng M, Zhao F K, Zheng J X, Liu X G. Octree-based fusion for realtime 3D reconstruction. Graphical Models, 2013, 75(3): 126–136 DOI:10.1016/j.gmod.2012.09.002

14. Chen J W, Bautembach D, Izadi S. Scalable real-time volumetric surface reconstruction. ACM Transactions on Graphics, 2013, 32(4): 1–16 DOI:10.1145/2461912.2461940

15. Steinbrucker F, Sturm J, Cremers D. Volumetric 3D mapping in real-time on a CPU. In: 2014 IEEE International Conference on Robotics and Automation (ICRA). Hong Kong, China, IEEE, 2014, 2021–2028 DOI:10.1109/icra.2014.6907

16. Nießner M, Zollhöfer M, Izadi S, Stamminger M. Real-time 3D reconstruction at scale using voxel hashing. ACM Transactions on Graphics, 2013, 32(6): 1–11 DOI:10.1145/2508363.2508374

17. Kahler O, Adrian Prisacariu V, Yuheng Ren C, Sun X, Torr P, Murray D. Very high frame rate volumetric integration of depth images on mobile devices. IEEE Transactions on Visualization and Computer Graphics, 2015, 21(11): 1241–1250 DOI:10.1109/tvcg.2015.2459891

18. Dryanovski I, Klingensmith M, Srinivasa S S, Xiao J Z. Large-scale, real-time 3D scene reconstruction on a mobile device. Autonomous Robots, 2017, 41(6): 1423–1445 DOI:10.1007/s10514-017-9624-2

19. Pfister H, Zwicker M, van Baar J, Gross M.Surfels: Surface elements as rendering primitives. In: Proceedings of the 27th annual conference on Computer graphics and interactive techniques-SIGGRAPH '00. ACM Press, 2000, 335–342 DOI:10.1145/344779.344936

20. Keller M, Lefloch D, Lambers M, Izadi S, Weyrich T, Kolb A. Real-time 3D reconstruction in dynamic scenes using point-based fusion. In: 2013 International Conference on 3D Vision. Seattle, WA, USA, IEEE, 2013, 1–8 DOI:10.1109/3dv.2013.9

21. Stuckler J, Behnke S. Integrating depth and color cues for dense multi-resolution scene mapping using RGB-D cameras. In: 2012 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI). Hamburg, Germany, IEEE, 2012 DOI:10.1109/mfi.2012.6343050

22. Stückler J, Behnke S. Multi-resolution surfel maps for efficient dense 3D modeling and tracking. Journal of Visual Communication and Image Representation, 2014, 25(1): 137–147 DOI:10.1016/j.jvcir.2013.02.008

23. Mallick T, Das P P, Majumdar A K. Characterizations of noise in kinect depth images: A review. IEEE Sensors Journal, 2014, 14(6): 1731–1740 DOI:10.1109/jsen.2014.2309987

24. Tomasi C, Manduchi R. Bilateral filtering for gray and color images. In: Proceedding of IEEE international conference on computer vision, 1998, 839–846 DOI:10.1109/iccv.1998.710815

25. Salas-Moreno R F, Glocken B, Kelly P H J, Davison A J. Dense planar SLAM. In: 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Munich, Germany, IEEE, 2014 DOI:10.1109/ismar.2014.6948422

26. Besl P J, McKay N D. A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence,1992, 14(2): 239–256 DOI:10.1109/34.121791

27. Chen Y, Medioni G. Object modeling by registration of multiple range images. In: Proceedings of 1991 IEEE International Conference on Robotics and Automation. Sacramento, CA, IEEE Comput. Soc. Press, 2724–2729 DOI:10.1109/robot.1991.132043

28. Kerl C, Sturm J, Cremers D. Dense visual SLAM for RGB-D cameras. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems. Tokyo, IEEE, 2013 DOI:10.1109/iros.2013.6696650

29. Henry P, Krainin M, Herbst E, Ren X F, Fox D. RGB-D mapping: using depth cameras for dense 3D modeling of indoor environments//Experimental Robotics. Berlin, Heidelberg, Springer Berlin Heidelberg, 2014, 477–491 DOI:10.1007/978-3-642-28572-1_33

30. Fischler M A, Bolles R C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 1981, 24(6): 381–395 DOI:10.1145/358669.358692

31. Triggs B, McLauchlan P F, Hartley R I, Fitzgibbon A W. Bundle adjustment—A modern synthesis//Vision Algorithms: Theory and Practice. Berlin, Heidelberg, Springer Berlin Heidelberg, 2000, 298–372 DOI:10.1007/3-540-44480-7_21

32. Lourakis M., Argyros A. SBA: A software package for generic sparse bundle adjustment. ACM Transactions on Mathematical Software, 36, 2009, 1–30

33. Konolige K. Sparse sparse bundle adjustment. In: Procedings of the British Machine Vision Conference. Aberystwyth, British Machine Vision Association, 2010 DOI:10.5244/c.24.102

34. Zhou Q Y, Miller S, Koltun V. Elastic fragments for dense scene reconstruction. In: 2013 IEEE International Conference on Computer Vision. Sydney, Australia, IEEE, 2013 DOI:10.1109/iccv.2013.65

35. Zhou Q Y, Koltun V. Dense scene reconstruction with points of interest. ACM Transactions on Graphics, 2013, 32(4):1–8 DOI:10.1145/2461912.2461919

36. Whelan T, Leutenegger S, Salas Moreno R, Glocker B, Davison A. ElasticFusion: dense SLAM without A pose graph. In: Robotics: Science and Systems XI, Robotics: Science and Systems Foundation, 2015 DOI:10.15607/rss.2015.xi.001

37. Glocker B, Shotton J, Criminisi A, Izadi S. Real-time RGB-D camera relocalization via randomized ferns for keyframe encoding. IEEE Transactions on Visualization and Computer Graphics, 2015, 21(5): 571–583 DOI:10.1109/tvcg.2014.2360403

38. Sumner R W, Schmid J, Pauly M. Embedded deformation for shape manipulation. ACM Transactions on Graphics, 2007, 26(3): 80 DOI:10.1145/1276377.1276478

39. Lowe D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2): 91–110 DOI:10.1023/b:visi.0000029664.99615.94

40. Bay H, Ess A, Tuytelaars T, van Gool L. Speeded-up robust features (SURF). Computer Vision and Image Understanding, 2008, 110(3): 346–359 DOI:10.1016/j.cviu.2007.09.014

41. Endres F, Hess J, Engelhard N, Sturm J, Cremers D, Burgard W. An evaluation of the RGB-D SLAM system. In: 2012 IEEE International Conference on Robotics and Automation. St Paul, MN, USA, IEEE, 2012 DOI:10.1109/icra.2012.6225199

42. Zhou Q Y, Koltun V. Depth camera tracking with contour cues. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA, IEEE, 2015, 632–638 DOI:10.1109/cvpr.2015.7298662

43. Mehta D, Sridhar S, Sotnychenko O, Rhodin H, Shafiei M, SeidelH-P, Xu W, Casas D, Theobalt C. VNect: real-time 3D human pose estimation with a single RGB camera. ACM Transactions on Graphics, 2017, 36(4):1–14 DOI:10.1145/3072959.3073596

44. Blais G, Levine M D. Registering multiview range data to create 3D computer objects. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1995, 17(8): 820–824 DOI:10.1109/34.400574

45. Steinbrucker F, Kerl C, Cremers D. Large-scale multi-resolution surface reconstruction from RGB-D sequences. In: 2013 IEEE International Conference on Computer Vision. Sydney, Australia, IEEE, 2013, 3264–3271 DOI:10.1109/iccv.2013.405

46. Lefloch D, Kluge M, Sarbolandi H, Weyrich T, Kolb A. Comprehensive use of curvature for robust and accurate online surface reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12): 2349–2365 DOI:10.1109/tpami.2017.2648803

47. Newcombe R A, Fox D, Seitz S M. DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA, IEEE, 2015 DOI:10.1109/cvpr.2015.7298631

48. Orts-Escolano S, Rhemann C, Fanello S, Chang W, Kowdle A, Degtyarev Y, Kim D, Davidson P L, Khamis S, Dou M, Tankovich V, Loop C, Cai Q, Chou P A, Mennicken S, Valentin J, Pradeep V, Wang S, Kang S B, Kohli P, Lutchyn Y, Keskin C, Izadi S. Holoportation: Virtual 3D Teleportation in Real-time. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology. Tokyo, Japan, ACM, 2016, 741–754 DOI:10.1145/2984511.2984517.

49. Yu T, Zheng Z R, Guo K W, Zhao J H, Dai Q H, Li H, Pons-Moll G, Liu Y B. DoubleFusion: real-time capture of human performances with inner body shapes from a single depth sensor. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, IEEE, 2018 DOI:10.1109/cvpr.2018.00761

50. Loper M, Mahmood N, Romero J, Pons-Moll G, Black M J. Smpl. ACM Transactions on Graphics, 2015, 34(6): 1–16 DOI:10.1145/2816795.2818013

51. Guo K W, Xu F, Yu T, Liu X Y, Dai Q H, Liu Y B. Real-time geometry, albedo and motion reconstruction using a single RGBD camera. ACM Transactions on Graphics, 2017, 36(4): 1 DOI:10.1145/3072959.3126786

52. Runz M, Agapito L. Co-fusion: Real-time segmentation, tracking and fusion of multiple objects. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). Singapore, IEEE, 2017, 4471–4478 DOI:10.1109/icra.2017.7989518

53. Gao W, Tedrake R. SurfelWarp: efficient non-volumetric single view dynamic reconstruction. In: Robotics: Science and Systems XIV. Robotics: Science and Systems Foundation, 2018

54. Runz M, Buffier M, Agapito L. MaskFusion: real-time recognition, tracking and reconstruction of multiple moving objects. In: 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Munich, Germany, IEEE, 2018 DOI:10.1109/ismar.2018.00024

55. Zhang Y Z, Xu W W, Tong Y Y, Zhou K. Online structure analysis for real-time indoor scene reconstruction. ACM Transactions on Graphics, 2015, 34(5): 1–13 DOI:10.1145/2768821

56. Dzitsiuk M, Sturm J, Maier R, Ma L N, Cremers D. De-noising, stabilizing and completing 3D reconstructions on-the-go using plane priors. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). Singapore, IEEE, 2017 DOI:10.1109/icra.2017.7989457

57. Shi Y F, Xu K, Nießner M, Rusinkiewicz S, Funkhouser T. PlaneMatch: patch coplanarity prediction for robust RGB-D reconstruction//Computer Vision–ECCV 2018. Cham: Springer International Publishing, 2018, 767–784 DOI:10.1007/978-3-030-01237-3_46

58. Salas-Moreno R F, Newcombe R A, Strasdat H, Kelly P H J, Davison A J. SLAM++: simultaneous localisation and mapping at the level of objects. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland, OR, USA, IEEE, 2013, 1352–1359 DOI:10.1109/cvpr.2013.178

59. Tateno K, Tombari F, Navab N. When 2.5D is not enough: Simultaneous reconstruction, segmentation and recognition on dense SLAM. In: 2016 IEEE International Conference on Robotics and Automation (ICRA). Stockholm, Sweden, IEEE, 2016, 2295–2302 DOI:10.1109/icra.2016.7487378

60. Aldoma A, Tombari F, Rusu R B, Vincze M. OUR-CVFH–oriented, unique and repeatable clustered viewpoint feature histogram for object recognition and 6DOF pose estimation//Lecture Notes in Computer Science. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, 113–122 DOI:10.1007/978-3-642-32717-9_12

61. Xu K, Shi Y F, Zheng L T, Zhang J Y, Liu M, Huang H, Su H, Cohen-Or D, Chen B Q. 3D attention-driven depth acquisition for object identification. ACM Transactions on Graphics, 2016, 35(6): 1–14 DOI:10.1145/2980179.2980224

62. Huang J W, Dai A, Guibas L, Niessner M. 3Dlite: towards commodity 3D scanning for content creation. ACM Transactions on Graphics, 2017, 36(6): 1–14 DOI:10.1145/3130800.3130824

63. Dong S Y, Xu K, Zhou Q, Tagliasacchi A, Xin S Q, Nießner M, Chen B Q. Multi-robot collaborative dense scene reconstruction. ACM Transactions on Graphics, 2019, 38(4): 1–16 DOI:10.1145/3306346.3322942

64. McCormac J, Handa A, Davison A, Leutenegger S. SemanticFusion: Dense 3D semantic mapping with convolutional neural networks. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). Singapore, IEEE, 2017, 4628–4635 DOI:10.1109/icra.2017.7989538

65. Noh H, Hong S, Han B. Learning deconvolution network for semantic segmentation. In: 2015 IEEE International Conference on Computer Vision (ICCV). Santiago, Chile, IEEE, 2015, 1520–1528 DOI:10.1109/iccv.2015.178

66. He K M, Gkioxari G, Dollar P, Girshick R. Mask R-CNN. In: 2017 IEEE International Conference on Computer Vision (ICCV). Venice, IEEE, 2017, 2980–2988 DOI:10.1109/iccv.2017.322

67. Hu R Z, Wen C, van Kaick O, Chen L M, Lin D, Cohen-Or D, Huang H. Semantic object reconstruction via casual handheld scanning. ACM Transactions on Graphics, 2019, 37(6): 1–12 DOI:10.1145/3272127.3275024

68. Schonberger J L, Pollefeys M, Geiger A, Sattler T. Semantic visual localization. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, IEEE, 2018

69. Bernardini F, Martin I M, Rushmeier H. High-quality texture reconstruction from multiple scans. IEEE Transactions on Visualization and Computer Graphics, 2001, 7(4): 318–332 DOI:10.1109/2945.965346

70. Pulli K, Shapiro L G. Surface reconstruction and display from range and color data. Graphical Models, 2000, 62(3): 165–201 DOI:10.1006/gmod.1999.0519

71. Lensch H P A, Heidrich W, Seidel H P. A silhouette-based algorithm for texture registration and stitching. Graphical Models, 2001, 63(4): 245–262 DOI:10.1006/gmod.2001.0554

72. Stamos I, Allen P K. Geometry and texture recovery of scenes of large scale. Computer Vision and Image Understanding, 2002, 88(2): 94–118 DOI:10.1006/cviu.2002.0963

73. Corsini M, Dellepiane M, Ponchio F, Scopigno R. Image-to-geometry registration: a mutual information method exploiting illumination-related geometric properties. Computer Graphics Forum, 2009, 28(7): 1755–1764 DOI:10.1111/j.1467-8659.2009.01552.x

74. Corsini M, Dellepiane M, Ganovelli F, Gherardi R, Fusiello A, Scopigno R. Fully automatic registration of image sets on approximate geometry. International Journal of Computer Vision, 2013, 102(1/2/3): 91–111 DOI:10.1007/s11263-012-0552-5

75. Zhou Q Y, Koltun V. Color map optimization for 3D reconstruction with consumer depth cameras. ACM Transactions on Graphics, 2014, 33(4): 1–10 DOI:10.1145/2601097.2601134

76. Whelan T, Kaess M, Johannsson H, Fallon M, Leonard J J, McDonald J. Real-time large-scale dense RGB-D SLAM with volumetric fusion. The International Journal of Robotics Research, 2015, 34(4/5): 598–626 DOI:10.1177/0278364914551008

77. Bi S, Kalantari N K, Ramamoorthi R. Patch-based optimization for image-based texture mapping. ACM Transactions on Graphics, 2017, 36(4): 1–11 DOI:10.1145/3072959.3073610

78. Whelan T, Salas-Moreno R F, Glocker B, Davison A J, Leutenegger S. ElasticFusion: Real-time dense SLAM and light source estimation. The International Journal of Robotics Research, 2016, 35(14): 1697–1716 DOI:10.1177/0278364916669237

79. Meilland M, Barat C, Comport A. 3D High Dynamic Range dense visual SLAM and its application to real-time object re-lighting. In: 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Adelaide, Australia, IEEE, 2013, 143–152 DOI:10.1109/ismar.2013.6671774

80. Li S D, Handa A, Zhang Y, Calway A. HDRFusion: HDR SLAM using a low-cost auto-exposure RGB-D sensor. In: 2016 Fourth International Conference on 3D Vision (3DV). CA, USA, IEEE, 2016, 314–322 DOI:10.1109/3dv.2016.40

81. Alexandrov S V, Prankl J, Zillich M, Vincze M. Towards dense SLAM with high dynamic range colors. In Computer Vision Winter Workshop (CVWW), 2017

82. Xu L, Su Z, Han L, Yu T, Liu Y B, Fang L. Unstructured Fusion: realtime 4D geometry and texture reconstruction using Commercial RGBD cameras. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 1 DOI:10.1109/tpami.2019.2915229

83. Xu K, Zheng L T, Yan Z H, Yan G H, Zhang E, Niessner M, Deussen O, Cohen-Or D, Huang H. Autonomous reconstruction of unknown indoor scenes guided by time-varying tensor Fields. ACM Transactions on Graphics, 2017, 36(6): 1–15 DOI:10.1145/3130800.3130812

84. Peng Y, Deng B L, Zhang J Y, Geng F Y, Qin W J, Liu L G. Anderson acceleration for geometry optimization and physics simulation. ACM Transactions on Graphics, 2018, 37(4): 1–14 DOI:10.1145/3197517.3201290

email E-mail this page

Articles by authors

VRIH