Adv Search
Home | Accepted | Article In Press | Current Issue | Archive | Special Issues | Collections | Featured Articles | Statistics

2020, 2(5): 454-470 Published Date:2020-10-20

DOI: 10.1016/j.vrih.2020.05.005

Development of augmented reality serious games with a vibrotactile feedback jacket

Full Text: PDF (4) HTML (88)

Export: EndNote | Reference Manager | ProCite | BibTex | RefWorks


In the past few years, augmented reality (AR) has rapidly advanced and has been applied in different fields. One of the successful AR applications is the immersive and interactive serious games, which can be used for education and learning purposes.
In this project, a prototype of an AR serious game is developed and demonstrated. Gamers utilize a head-mounted device and a vibrotactile feedback jacket to explore and interact with the AR serious game. Fourteen vibration actuators are embedded in the vibrotactile feedback jacket to generate immersive AR experience. These vibration actuators are triggered in accordance with the designed game scripts. Various vibration patterns and intensity levels are synthesized in different game scenes. This article presents the details of the entire software development of the AR serious game, including game scripts, game scenes with AR effects design, signal processing flow, behavior design, and communication configuration. Graphics computations are processed using the graphics processing unit in the system.
Results /Conclusions
The performance of the AR serious game prototype is evaluated and analyzed. The computation loads and resource utilization of normal game scenes and heavy computation scenes are compared. With 14 vibration actuators placed at different body positions, various vibration patterns and intensity levels can be generated by the vibrotactile feedback jacket, providing different real-world feedback. The prototype of this AR serious game can be valuable in building large-scale AR or virtual reality educational and entertainment games. Possible future improvements of the proposed prototype are also discussed in this article.
Keywords: Augmented reality ; AR serious games ; Vibrotactile feedback jacket ; Game scenes

Cite this article:

Lingfei ZHU, Qi CAO, Yiyu CAI. Development of augmented reality serious games with a vibrotactile feedback jacket. Virtual Reality & Intelligent Hardware, 2020, 2(5): 454-470 DOI:10.1016/j.vrih.2020.05.005

1. Cai Y. 3D Immersive Interactive Learning. Singapore: Springer Verlag, 2013 DOI:10.1007/978-981-4021-90-6

2. Xie Y, Zhang Y Z, Cai Y Y. Virtual reality engine disassembly simulation with natural hand-based interaction. In: VR, Simulations and Serious Games for Education. Singapore: Springer Singapore, 2018, 121–128 DOI:10.1007/978-981-13-2844-2_11

3. Orozco M, Silva J, El A, Petriu E. The role of haptics in games. In: Haptics Rendering and Applications. InTech, 2012, 217–234 DOI:10.5772/32809

4. Barajas A O, Al Osman H, Shirmohammadi S. A Serious Game for children with Autism Spectrum Disorder as a tool for play therapy. In: 2017 IEEE 5th International Conference on Serious Games and Applications for Health (SeGAH). Perth, WA, Australia, IEEE, 2017, 1–7 DOI:10.1109/segah.2017.7939266

5. DiPietro J, Kelemen A, Liang Y L, Sik-Lanyi C. Computer- and robot-assisted therapies to aid social and intellectual functioning of children with autism spectrum disorder. Medicina, 2019, 55(8): 440 DOI:10.3390/medicina55080440

6. Tsikinas S, Xinogalos S. Studying the effects of computer serious games on people with intellectual disabilities or autism spectrum disorder: a systematic literature review. Journal of Computer Assisted Learning, 2019, 35(1): 61–73 DOI:10.1111/jcal.12311

7. Lu A, Chan S, Cai Y Y, Huang L H, Nay Z T, Goei S L. Learning through VR gaming with virtual pink dolphins for children with ASD. Interactive Learning Environments, 2018, 26(6): 718–729 DOI:10.1080/10494820.2017.1399149

8. Cai Y Y, Chiew R, Nay Z T, Indhumathi C, Huang L H. Design and development of VR learning environments for children with ASD. Interactive Learning Environments, 2017, 25(8): 1098–1109 DOI:10.1080/10494820.2017.1282877

9. Tang J S Y, Falkmer M, Chen N T M, Bӧlte S, Girdler S. Designing a serious game for youth with ASD: perspectives from end-users and professionals. Journal of Autism and Developmental Disorders, 2019, 49(3): 978–995 DOI:10.1007/s10803-018-3801-9

10. Borresen A, Wolfe C, Lin C, Tian Y, Raghuraman S, Nahrstedt K, Prabhakaran B, Annaswamy T M. Usability of an immersive augmented reality based telerehabilitation system with haptics (ARTESH) for synchronous remote musculoskeletal examination. International Journal of Telerehabilitation, 2019, 11(1): 23–32 DOI:10.5195/ijt.2019.6275

11. Culbertson H, Kuchenbecker K J. Ungrounded haptic augmented reality system for displaying roughness and friction. IEEE/ASME Transactions on Mechatronics, 2017, 22(4): 1839–1849 DOI:10.1109/tmech.2017.2700467

12. Maisto M, Pacchierotti C, Chinello F, Salvietti G, de Luca A, Prattichizzo D. Evaluation of wearable haptic systems for the fingers in augmented reality applications. IEEE Transactions on Haptics, 2017, 10(4): 511–522 DOI:10.1109/toh.2017.2691328

13. Pridhvi Krishna M V, Mehta S, Verma S, Rane S. Mixed reality in smart computing education system. In: 2018 International Conference on Smart Systems and Inventive Technology (ICSSIT). Tirunelveli, India, IEEE, 2018, 72–75 DOI:10.1109/icssit.2018.8748813

14. Bhargava A, Bertrand J W, Gramopadhye A K, Madathil K C, Babu S V. Evaluating multiple levels of an interaction fidelity continuum on performance and learning in near-field training simulations. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(4): 1418–1427 DOI:10.1109/tvcg.2018.2794639

15. Johnson C I, Bailey S K T, van Buskirk W L. Designing effective feedback messages in serious games and simulations: A research review. In: Instructional Techniques to Facilitate Learning and Motivation of Serious Games. Cham: Springer International Publishing, 2016, 119–140 DOI:10.1007/978-3-319-39298-1_7

16. Menelas B A, Benaoudia R. Use of haptics to promote learning outcomes in serious games. Multimodal Technologies and Interaction, 2017, 1(4): 31 DOI:10.3390/mti1040031

17. Vaquero-Melchor D, Bernardos A M. Enhancing interaction with augmented reality through mid-air haptic feedback: architecture design and user feedback. Applied Sciences, 2019, 9(23): 5123 DOI:10.3390/app9235123

18. Lee J, Kim Y, Kim G J. Effects of visual feedback on out-of-body illusory tactile sensation when interacting with augmented virtual objects. IEEE Transactions on Human-Machine Systems, 2017, 47(1): 101–112 DOI:10.1109/thms.2016.2599492

19. de Jesus Oliveira V A, Brayda L, Nedel L, Maciel A. Designing a vibrotactile head-mounted display for spatial awareness in 3D spaces. IEEE Transactions on Visualization and Computer Graphics, 2017, 23(4): 1409–1417 DOI:10.1109/tvcg.2017.2657238

20. Garciavalle G, Ferre M, Brenosa J, Vargas D. Evaluation of presence in virtual environments: haptic vest and user's haptic skills. IEEE Access, 2018, 6: 7224–7233 DOI:10.1109/access.2017.2782254

21. Rantala J, Majaranta P, Kangas J, Isokoski P, Akkil D, Špakov O, Raisamo R. Gaze interaction with vibrotactile feedback: review and design guidelines. Human-Computer Interaction, 2020, 35(1): 1–39 DOI:10.1080/07370024.2017.1306444

22. Shim Y A, Lee J, Lee G. Exploring multimodal watch-back tactile display using wind and vibration. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Montreal QC, Canada, New York, ACM Press, 2018, 1–12 DOI:10.1145/3173574.3173706

23. Kaul O B, Rohs M. HapticHead: 3D guidance and target acquisition through a vibrotactile grid. In: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. California, USA, New York, ACM Press, 2016, 2533–2539 DOI:10.1145/2851581.2892355

24. Young E M, Memar A H, Agarwal P, Colonnese N. Bellowband: a pneumatic wristband for delivering local pressure and vibration. In: 2019 IEEE World Haptics Conference (WHC). Tokyo, Japan, IEEE, 2019, 55–60 DOI:10.1109/whc.2019.8816075

25. Wolf D, Rietzler M, Hnatek L, Rukzio E. Face/on: multi-modal haptic feedback for head-mounted displays in virtual reality. IEEE Transactions on Visualization and Computer Graphics, 2019, 25(11): 3169–3177 DOI:10.1109/tvcg.2019.2932215

26. Louison C, Ferlay F, Mestre D R. Spatialized vibrotactile feedback contributes to goal-directed movements in cluttered virtual environments. In: 2017 IEEE Symposium on 3D User Interfaces (3DUI). Los Angeles, CA, USA, IEEE, 2017, 99–102 DOI:10.1109/3dui.2017.7893324

27. Delazio A, Nakagaki K, Klatzky R L, Hudson S E, Lehman J F, Sample A P. Force jacket: pneumatically-actuated jacket for embodied haptic experiences. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Montreal QC, Canada, New York, ACM Press, 2018 DOI:10.1145/3173574.3173894

28. Kishishita Y, Das S, Ramirez A V, Thakur C, Tadayon R, Kurita Y. Muscleblazer: force-feedback suit for immersive experience. In: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). Osaka, Japan, IEEE, 2019, 1813–1818 DOI:10.1109/vr.2019.8797962

29. Rognon C, Koehler M, Duriez C, Floreano D, Okamura A M. Soft haptic device to render the sensation of flying like a drone. IEEE Robotics and Automation Letters, 2019, 4(3): 2524-2531 DOI:10.1109/LRA.2019.2907432

30. Ouyang Q Q, Wu J, Wu M. Vibrotactile display of flight attitude with combination of multiple coding parameters. Applied Sciences, 2017, 7(12): 1291 DOI:10.3390/app7121291

31. Günther S, Makhija M, Müller F, Schön D, Mühlhäuser M, Funk M. PneumAct: pneumatic kinesthetic actuation of body joints in virtual reality environments. In: Proceedings of the 2019 on Designing Interactive Systems Conference. San Diego, CA, USA, ACM, 2019, 227–240 DOI:10.1145/3322276.3322302

32. ImmerzInc, product of KOR-FX [2020-02-20].

33. bHapticsInc, product of TACTOT, haptic vest for torso [2020-02-20].

34. USAInc Woojer, product of Vest Pro [2020-02-20].

35. Ltd VRElectronics, product of TESLASUIT [2020-02-20].

36. Zhang Z. A Flexible New Technique for Camera Calibration (Technical report). Microsoft Research. MSR-TR-98-71, 1998

37. Reaction Time Statistics (2020) [2020-02-20].

38. Tang Z, Lin Y, Lee K, Hwang J, Chuang J. ESTHER: joint camera self-calibration and automatic radial distortion correction from tracking of walking humans. IEEE Access, 2019, 7: 10754–10766 DOI:10.1109/access.2019.2891224

39. Zhang Z D, Matsushita Y, Ma Y. Camera calibration with lens distortion from low-rank textures. In: CVPR 2011. Providence, RI, USA, IEEE, 2011, 2321–2328 DOI:10.1109/cvpr.2011.5995548

40. Tehrani M A, Beeler T, Grundhöfer A. A practical method for fully automatic intrinsic camera calibration using directionally encoded light. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA, IEEE, 2017, 125–133 DOI:10.1109/cvpr.2017.21

41. Kakani V, Kim H, Lee J, Ryu C, Kumbham M. Automatic distortion rectification of wide-angle images using outlier refinement for streamlining vision tasks. Sensors, 2020, 20(3): 894 DOI:10.3390/s20030894

email E-mail this page

Articles by authors