Home About the Journal Latest Work Current Issue Archive Special Issues Editorial Board
<< Previous Next >>

2019, 1(6): 597-610

Published Date:2019-12-20 DOI: 10.1016/j.vrih.2019.10.002

Digital assembly technology based on augmented reality and digital twins: a review

Abstract

Product assembly simulation is considered as one of the key technologies in the process of complex product design and manufacturing. Virtual assembly realizes the assembly process design, verification, and optimization of complex products in the virtual environment, which plays an active and effective role in improving the assembly quality and efficiency of complex products. In recent years, augmented reality (AR) and digital twin (DT) technology have brought new opportunities and challenges to the digital assembly of complex products owing to their characteristics of virtual reality fusion and interactive control. This paper expounds the concept and connotation of AR, enumerates a typical AR assembly system structure, analyzes the key technologies and applications of AR in digital assembly, and notes that DT technology is the future development trend of intelligent assembly research.

Keyword

Augmented reality ; Digital twin ; Digital assembly ; Virtual simulation ; Assembly analysis

Cite this article

Chan QIU, Shien ZHOU, Zhenyu LIU, Qi GAO, Jianrong TAN. Digital assembly technology based on augmented reality and digital twins: a review. Virtual Reality & Intelligent Hardware, 2019, 1(6): 597-610 DOI:10.1016/j.vrih.2019.10.002

References

1. Carmigniani J, Furht B, Anisetti M, Ceravolo P, Damiani E, Ivkovic M. Augmented reality technologies, systems and applications. Multimedia Tools and Applications, 2011, 51(1): 341–377 DOI:10.1016/j.procir.2015.07.044

2. Azuma R T. A survey of augmented reality. Presence: Teleoperators and Virtual Environments, 1997, 6(4): 355–385 DOI:10.1162/pres.1997.6.4.355

3. van Doormaal T P C, van Doormaal J A M, Mensink T. Clinical accuracy of holographic navigation using point-based registration on augmented-reality glasses. Operative Neurosurgery, 2019, 1–6 DOI:10.1093/ons/opz094

4. Nee A Y C, Ong S K, Chryssolouris G, Mourtzis D. Augmented reality applications in design and manufacturing. CIRP Annals, 2012, 61(2): 657–679 DOI:10.1016/j.cirp.2012.05.010

5. Dini G, Mura M D. Application of augmented reality techniques in through-life engineering services. Procedia CIRP, 2015, 38: 14–23 DOI:10.1016/j.procir.2015.07.044

6. Didier J-Y, Roussel D, Mallem M, Otmane S, Naudet S, Pham Q-C, Bourgeois S, Mégard C, Leroux C, Hocquard A. AMRA: Augmented Reality Assistance for Train Maintenance Tasks. In: Workshop Industrial Augmented Reality, 4th ACM/IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2005). Vienna, Austria, IEEE, 2005

7. Haritos T, MacChiarella N D. A mobile application of augmented reality for aerospace maintenance training. In: 24th Digital Avionics Systems Conference. Washington, DC, USA, IEEE, 2005DOI:10.1109/dasc.2005.1563376

8. De Crescenzio F, Fantini M, Persiani F, Di Stefano L, Azzari P, Salti S. Augmented Reality for Aircraft Maintenance Training and Operations Support. IEEE Computer Graphics and Applications, 2011, 31(1): 96–101 DOI:10.1109/MCG.2011.4

9. Dong S Y, Feng C, Kamat V R. Sensitivity analysis of augmented reality-assisted building damage reconnaissance using virtual prototyping. Automation in Construction, 2013, 33: 24–36 DOI:10.1016/j.autcon.2012.09.005

10. Grieves M. Digital Twin: Manufacturing excellence through virtual factory replication. Whitepaper, 2015

11. Tao F, Liu W, Liu J, Liu X, Liu Q, Qu T, Xiang F. Digital twin and its potential application exploration. Computer Integrated Manufacturing Systems, 2018, 24: 1–18 DOI:10.13196/j.cims.2018.01.001

12. Li C Z, Mahadevan S, Ling Y, Choze S, Wang L P. Dynamic Bayesian network for aircraft wing health monitoring digital twin. AIAA Journal, 2017, 55(3): 930–941 DOI:10.2514/1.j055201

13. Revetria R, Tonelli F, Damiani L, Demartini M, Bisio F, Peruzzo N. A real-time mechanical structures monitoring system based on digital twin, iot and augmented reality. In: 2019 Spring Simulation Conference (SpringSim). Tucson, AZ, USA, IEEE, 2019DOI:10.23919/springsim.2019.8732917

14. KOng S, LYuan M, CNee A Y. Augmented reality applications in manufacturing: a survey. International Journal of Production Research, 2008, 46(10): 2707–2742 DOI:10.1080/00207540601064773

15. Nee A Y C, Ong S K, Chryssolouris G, Mourtzis D. Augmented reality applications in design and manufacturing. CIRP Annals, 2012, 61(2): 657–679 DOI:10.1016/j.cirp.2012.05.010

16. Wang X, Ong S K, Nee A Y C. A comprehensive survey of augmented reality assembly research. Advances in Manufacturing, 2016, 4(1): 1–22 DOI:10.1007/s40436-015-0131-4

17. Henderson S J, Feiner S K. Augmented reality in the psychomotor phase of a procedural task. In: Mixed and Augmented Reality (ISMAR), 2011 10th IEEE International Symposium. Basel, Switzerland, IEEE, 2011, 191–200 DOI: 10.1109/ISMAR.2011.6092386

18. Zhu J, Ong S K, Nee A Y C. An authorable context-aware augmented reality system to assist the maintenance technicians. The International Journal of Advanced Manufacturing Technology, 2013, 66(9/11/12): 1699–1714 DOI:10.1007/s00170-012-4451-2

19. Benbelkacem S, Belhocine M, Bellarbi A, Zenatihenda N, Tadjine M. Augmented reality for photovoltaic pumping systems maintenance tasks. Renewable Energy, 2013, 55(55): 428–437 DOI:10.1016/j.renene.2012.12.043

20. Radkowski R, Oliver J H. Natural Feature Tracking Augmented Reality for On-Site Assembly Assistance Systems. International Conference on Virtual, Augmented and Mixed Reality, 2013, 8022: 281–290 DOI:10.1007/978-3-642-39420-1_30

21. Petersen N, Pagani A, Stricker D. Real-time modeling and tracking manual workflows from first-person vision. In: 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Adelaide, SA, Australia, IEEE, 2013, 117–124

22. Wang Z B, Ong S K, Nee A Y C. Augmented reality aided interactive manual assembly design. The International Journal of Advanced Manufacturing Technology, 2013, 69(5/6/7/8): 1311–1321 DOI:10.1007/s00170-013-5091-x

23. Wang Z B, Ng L X, Ong S K, Nee A Y C. Assembly planning and evaluation in an augmented reality environment. International Journal of Production Research, 2013, 51(23/24): 7388–7404 DOI:10.1080/00207543.2013.837986

24. Khuong B M, Kiyokawa K, Miller A, La Viola J J, Mashita T, Takemura H. The effectiveness of an AR-based context-aware assembly support system in object assembly. In: 2014 IEEE Virtual Reality (VR). Minneapolis, MN, USA, IEEE, 2014DOI:10.1109/vr.2014.6802051

25. OpenCV. http://opencv.willowgarage.com/wiki/

26. Damen D M, Gee A, Mayol-Cuevas W, Calway A. Egocentric Real-time Workspace Monitoring using an RGB-D camera. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. Vilamoura-Algarve, Portugal, IEEE, 2012DOI:10.1109/iros.2012.6385829

27. Henderson S J, Feiner S K. Augmented reality in the psychomotor phase of a procedural task. In: 2011 10th IEEE International Symposium on Mixed and Augmented Reality. Basel, Switzerland, IEEE, 2011DOI:10.1109/ismar.2011.6162888

28. Wang Y T, Shen Y, Liu D C, Wei S, Zhu C G. Key technique of assembly system in an augmented reality environment. In: 2010 Second International Conference on Computer Modeling and Simulation. Sanya, China, IEEE, 2010DOI:10.1109/iccms.2010.201

29. Andersen M, Andersen R, Larsen C, Moeslund T B, Madsen O. Interactive assembly guide using augmented reality. Advances in Visual Computing. Berlin, Heidelberg, Springer, 2009, 999–1008 DOI:10.1007/978-3-642-10331-5_93

30. Yuan M L, Ong S K, Nee A Y C. Augmented reality for assembly guidance using a virtual interactive tool. International Journal of Production Research, 2008, 46(7): 1745–1767 DOI:10.1080/00207540600972935

31. Charoenseang S, Panjan S. 5-finger exoskeleton for assembly training in augmented reality. Virtual and Mixed Reality- New Trends. Berlin, Heidelberg, Springer, 2011, 30–39 DOI:10.1007/978-3-642-22021-0_4

32. Sukan M, Elvezio C, Oda O, Feiner S, Tversky B. ParaFrustum: visualization techniques for guiding a user to a constrained set of viewing positions and orientations. In: Proceedings of the 27th annual ACM symposium on User interface software and technology. Honolulu, Hawaii, USA, ACM, 2014: 331–340

33. Wang X, Ong S, Nee A. Augmented reality interfaces for industrial assembly design and planning. In: Proceedings of Eighth International Conference on Interfaces and Human Computer Interaction. Lisbon, 2014, 83–90

34. Raghavan V, Molineros J, Sharma R. Interactive evaluation of assembly sequences using augmented reality. IEEE Transactions on Robotics and Automation, 1999, 15(3): 435–449 DOI:10.1109/70.768177

35. Henderson S, Feiner S. Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE Transactions on Visualization and Computer Graphics, 2011, 17(10): 1355–1368 DOI:10.1109/tvcg.2010.245

36. Hou L, Wang X Y, Bernold L, Love P E D. Using animated augmented reality to cognitively guide assembly. Journal of Computing in Civil Engineering, 2013, 27(5): 439–451 DOI:10.1061/(asce)cp.1943-5487.0000184

37. Gavish N, Gutiérrez T, Webel S, Rodríguez J, Peveri M, Bockholt U, Tecchia F. Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks. Interactive Learning Environments, 2015, 23(6): 778–798 DOI:10.1080/10494820.2013.815221

38. Zhou Y, Yan D Y, Wang Y T, Chang H, Xu T. A registration method in augmented reality. Journal of Image and Graphics, 2000, 5(5): 430–433(in Chinese)

39. Li S Q, Xu C. Efficient lookup table-based camera pose estimation for augmented reality. Computer Animation and Virtual Worlds, 2011, 22(1): 47–58 DOI:10.1002/cav.385

40. Leu M C, ElMaraghy H A, Nee A Y C, Ong S K, Lanzetta M, Putz M, Zhu W J, Bernard A. CAD model based virtual assembly simulation, planning and training. CIRP Annals, 2013, 62(2): 799–822 DOI:10.1016/j.cirp.2013.05.005

41. Zhang Y, Li M, Zeng B. The 3D registration based on the single circle of image and application in augmented assembly. Mechanical Science And Technology for Aerospace Engineering, 2014, 33(6): 881–885

42. Hořejší P. Augmented reality system for virtual training of parts assembly. Procedia Engineering, 2015, 100: 699–706 DOI:10.1016/j.proeng.2015.01.422

43. Kim D, Park J, Ko K H. Development of an AR based method for augmentation of 3D CAD data onto a real ship block image. Computer-Aided Design,2018, 98: 1–11

44. Wang Y, Zhang S, He W P, Bai X L. Model-based marker-less 3D tracking approach for augmented reality[J]. Journal of Shanghai Jiaotong University, 2018, 52(1):83–89 (in Chinese) DOI:10.16183/j.cnki.jsjtu.2018.01.013

45. Rentzos L, Papanastasiou S, Papakostas N, Chryssolouris G. Augmented reality for human-based assembly: using product and process semantics. IFAC Proceedings Volumes, 2013, 46(15): 98–101 DOI:10.3182/20130811-5-us-2037.00053

46. Zhu Z W, Branzoi V, Wolverton M, Murray G, Vitovitch N, Yarnall L, Acharya G, Samarasekera S, Kumar R. AR-mentor: Augmented reality based mentoring system. In: 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Munich, Germany, IEEE, 2014DOI:10.1109/ismar.2014.6948404

47. Chen C J, Hong J, Wang S F. Automated positioning of 3D virtual scene in AR-based assembly and disassembly guiding system. The International Journal of Advanced Manufacturing Technology, 2015, 76: 753–764 DOI:10.1007/s00170-014-6321-6

48. Vignais N, Miezal M, Bleser G, Mura K, Gorecky D, Marin F. Innovative system for real-time ergonomic feedback in industrial manufacturing. Applied Ergonomics, 2013, 44(4): 566–574 DOI:10.1016/j.apergo.2012.11.008

49. Wang X, Ong S K, Nee A Y C. Augmented reality interfaces for industrial assembly design and planning. In: Proceedings of the Eighth International Conference on Interfaces and Human Computer Interaction. Lisbon, Portugal, 2014, 83–90

50. Wang X, Ong S K, Nee A Y C. Real-virtual components interaction for assembly simulation and planning. Robotics and Computer-Integrated Manufacturing, 2016, 41, 102–114

51. Söderberg R, Wärmefjord K, Carlson J S, Lindkvist L. Toward a Digital Twin for real-time geometry assurance in individualized production. CIRP Annals, 2017, 66(1): 137–140 DOI:10.1016/j.cirp.2017.04.038

52. Liu R, Fan X, Yin X Y, Wang J K, Yang X. Research on combinational pose estimation of base part and assembly part for augmented reality aided assembly. Machine Design and Research, 2018, 34(6): 119–125(in Chinese)

53. Wang X, Dunston P S. Design, strategies, and issues towards an augmented reality-based construction training platform. Journal of Information Technology in Construction (ITcon), 2007, 12(25): 363–380

54. Webel S, Bockholt U, Engelke T, Gavish N, Olbrich M, Preusche C. An augmented reality training platform for assembly and maintenance skills. Robotics and Autonomous Systems, 2013, 61(4): 398–403 DOI:10.1016/j.robot.2012.09.013

55. Damiani L, Demartini M, Guizzi G, Revetria R, Tonelli F. Augmented and virtual reality applications in industrial systems: A qualitative review towards the industry 4.0 era. IFAC-PapersOnLine, 2018, 51(11): 624–630 DOI:10.1016/j.ifacol.2018.08.388

56. Hu J Z. Design of auxiliary learning software of augmented reality technique based on image recognition. Mechanical Engineering & Automation, 2018(3), 38–40

57. Liverani A, Amati G, Caligiana G. A CAD-augmented reality integrated environment for assembly sequence check and interactive validation. Concurrent Engineering, 2004, 12(1): 67–77 DOI:10.1177/1063293X04042469

58. Ong S K, Shen Y. A mixed reality environment for collaborative product design and development. CIRP Annals, 2009, 58(1): 139–142 DOI:10.1016/j.cirp.2009.03.020

59. Oda O, Feiner S. 3D referencing techniques for physical objects in shared augmented reality. 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Atlanta, GA, USA, IEEE, 2012, 207–215 DOI:10.1109/ISMAR.2012.6402558

60. Ranatunga D, Feng D, Adcock M, Thomas B. Towards object based manipulation in remote guidance. In: 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Adelaide, Australia, IEEE, 2013DOI:10.1109/ismar.2013.6671839

61. Gartner Top 10 Strategic Technology Trends for 2019. https://www.gartner.com/smarterwithgartner/gartner-top-10-strategic-technology-trends-for-2019/

62. Tao F, Zhang M. Digital twin shop-floor: A new shop-floor paradigm towards smart manufacturing. IEEE Access, 2017, 5: 20418–20427 DOI:10.1109/access.2017.2756069

63. Tao F, Zhang M, Cheng J F, Qi Q L. Digital twin workshop: a new paradigm for future workshop. Computer Integrated Manufacturing Systems, 2017, 23(1): 1–9 (in Chinese) DOI:10.13196/j.cims.2017.01.001

64. Damiani L, Demartini M, Giribone P, Maggiani M, Revetria R, Tonelli F. Simulation and Digital Twin Based Design of a Production Line: A Case Study. In: Proceedings of the International MultiConference of Engineers and Computer Scientists. 2018

65. Wärmefjord K, Söderberg R, Lindkvist L, Lindau B, Carlson J S. Inspection Data to Support a Digital Twin for Geometry Assurance. In: ASME 2017 International Mechanical Engineering Congress and Exposition. American Society of Mechanical Engineers, 2017

66. Schleich B, Anwer N, Mathieu L, Wartzack S. Shaping the digital twin for design and production engineering. CIRP Annals, 2017, 66(1): 141–144

67. Wang L. Research on the docking technology of final installation for aeroengine low pressure turbine unit based on digital twin. Computer Measurement & Control, 2018, 26(10): 286–290, 303(in Chinese)

68. Zhang Y L, Zhang J P, Wang X D, Chen X B. Digital twin technology for spacecraft on-orbit assembly. Navigation and Control, 2018, 17(3): 75–82(in Chinese)

Related

1. Shiguang QIU, Shuntao LIU, Deshuai KONG, Qichang HE, Three-dimensional virtual-real mapping of aircraft autom-atic spray operation and online simulation monitoring Virtual Reality & Intelligent Hardware 2019, 1(6): 611-621

2. Mengting XIAO, Zhiquan FENG, Xiaohui YANG, Tao XU, Qingbei GUO, Multimodal interaction design and application in augmented reality for chemical experiment Virtual Reality & Intelligent Hardware 2020, 2(4): 291-304

3. Yuxiang ZHAO, Xiaowei AN, Nongliang SUN, Virtual simulation experiment of the design and manufacture of a beer bottle-defect detection system Virtual Reality & Intelligent Hardware 2020, 2(4): 354-367

4. Yonghang TAI, Junsheng SHI, Junjun PAN, Aimin HAO, Victor CHANG, Augmented reality-based visual-haptic modeling for thoracoscopic surgery training systems Virtual Reality & Intelligent Hardware 2021, 3(4): 274-286

5. Mingxuan CHEN, Ping ZHANG, Zebo WU, Xiaodan CHEN, A multichannel human-swarm robot interaction system in augmented reality Virtual Reality & Intelligent Hardware 2020, 2(6): 518-533

6. Yukang YAN, Xin YI, Chun YU, Yuanchun SHI, Gesture-based target acquisition in virtual and augmented reality Virtual Reality & Intelligent Hardware 2019, 1(3): 276-289

7. Yuan GAO, Le XIE, A review on the application of augmented reality in craniomaxillofacial surgery Virtual Reality & Intelligent Hardware 2019, 1(1): 113-120

8. Jinyu LI, Bangbang YANG, Danpeng CHEN, Nan WANG, Guofeng ZHANG, Hujun BAO, Survey and evaluation of monocular visual-inertial SLAM algorithms for augmented reality Virtual Reality & Intelligent Hardware 2019, 1(4): 386-410

9. Xiaomei ZHAO, Fulin TANG, Yihong WU, Real-time human segmentation by BowtieNet and a SLAM-based human AR system Virtual Reality & Intelligent Hardware 2019, 1(5): 511-524

10. Wang LI, Junfeng WANG, Sichen JIAO, Meng WANG, Shiqi LI, Research on the visual elements of augmented reality assembly processes Virtual Reality & Intelligent Hardware 2019, 1(6): 622-634

11. Pengfei HAN, Gang ZHAO, A review of edge-based 3D tracking of rigid objects Virtual Reality & Intelligent Hardware 2019, 1(6): 580-596

12. Shenze WANG, Kaikai DU, Ningfang SONG, Dongfeng ZHAO, Di FENG, Zhengqian TU, Study on the adaptability of augmented reality smartglasses for astigmatism based on holographic waveguide grating Virtual Reality & Intelligent Hardware 2020, 2(1): 79-85

13. Xiang ZHOU, Liyu TANG, Ding LIN, Wei HAN, Virtual & augmented reality for biological microscope in experiment education Virtual Reality & Intelligent Hardware 2020, 2(4): 316-329

14. Lingfei ZHU, Qi CAO, Yiyu CAI, Development of augmented reality serious games with a vibrotactile feedback jacket Virtual Reality & Intelligent Hardware 2020, 2(5): 454-470

15. Yun-Han LEE, Tao ZHAN, Shin-Tson WU, Prospects and challenges in augmented reality displays Virtual Reality & Intelligent Hardware 2019, 1(1): 10-20

16. Zike YAN, Hongbin ZHA, Flow-based SLAM: From geometry computation to learning Virtual Reality & Intelligent Hardware 2019, 1(5): 435-460