Adv Search
Home | Accepted | Article In Press | Current Issue | Archive | Special Issues | Collections | Featured Articles | Statistics

2020, 2(5): 421-442 Published Date:2020-10-20

DOI: 10.1016/j.vrih.2020.07.008

Multimodal teaching, learning and training in virtual reality: a review and case study

Full Text: PDF (9) HTML (122)

Export: EndNote | Reference Manager | ProCite | BibTex | RefWorks


It is becoming increasingly prevalent in digital learning research to encompass an array of different meanings, spaces, processes, and teaching strategies for discerning a global perspective on constructing the student learning experience. Multimodality is an emergent phenomenon that may influence how digital learning is designed, especially when employed in highly interactive and immersive learning environments such as Virtual Reality (VR). VR environments may aid students’ efforts to be active learners through consciously attending to, and reflecting on, critique leveraging reflexivity and novel meaning-making most likely to lead to a conceptual change. This paper employs eleven industrial case-studies to highlight the application of multimodal VR-based teaching and training as a pedagogically rich strategy that may be designed, mapped and visualized through distinct VR-design elements and features. The outcomes of the use cases contribute to discern in-VR multimodal teaching as an emerging discourse that couples system design-based paradigms with embodied, situated and reflective praxis in spatial, emotional and temporal VR learning environments.
Keywords: Virtual reality ; Multimodality ; Training ; Teaching and learning ; Semiotic resources

Cite this article:

Stéphanie PHILIPPE, Alexis D. SOUCHET, Petros LAMERAS, Panagiotis PETRIDIS, Julien CAPORAL, Gildas COLDEBOEUF, Hadrien DUZAN. Multimodal teaching, learning and training in virtual reality: a review and case study. Virtual Reality & Intelligent Hardware, 2020, 2(5): 421-442 DOI:10.1016/j.vrih.2020.07.008

1. Reigeluth C M, Beatty B J, Myers R D. Instructional-design theories and models. The Learner-Centered Paradigm of Education. Routledge, 2016

2. Deci E L, Ryan R M. Intrinsic motivation. In: The Corsini Encyclopedia of Psychology. American Cancer Society, 2010 DOI:10.1002/9780470479216.corpsy0467

3. Carbonneau N, Vallerand R J, Lafrenière M A K. Toward a tripartite model of intrinsic motivation. Journal of Personality, 2012, 80(5): 1147–1178 DOI:10.1111/j.1467-6494.2011.00757.x

4. Veenman M V J, Hout-Wolters B H A M, Afflerbach P. Metacognition and learning: conceptual and methodological considerations. Metacognition and Learning, 2006, 1(1): 3–14 DOI:10.1007/s11409-006-6893-0

5. Donker A, de Boer H, Kostons D, van Ewijk C C D, van der Werf M P C. Effectiveness of learning strategy instruction on academic performance: a meta-analysis. Educational Research Review, 2014, 11: 1–26 DOI:10.1016/j.edurev.2013.11.002

6. Prince M J. Does active learning work? A review of the research. Journal of Engineering Education, 2004, 93(3): 223–231 DOI:10.1002/j.2168-9830.2004.tb00809.x

7. Vernon D T, Blake R L. Does problem-based learning work? A meta-analysis of evaluative research. Academic Medicine, 1993, 68(7): 550–563 DOI:10.1097/00001888-199307000-00015

8. Dochy F, Segers M, van den Bossche P, Gijbels D. Effects of problem-based learning: a meta-analysis. Learning and Instruction, 2003, 13(5): 533–568 DOI:10.1016/s0959-4752(02)00025-7

9. Walker A, Leary H. A problem based learning meta analysis: differences across problem types, implementation types, disciplines, and assessment levels. Interdisciplinary Journal of Problem-based Learning, 2009, 3(1): 6–28 DOI:10.7771/1541-5015.1061

10. Hess T, Gunter G A. Serious game-based and nongame-based online courses: learning experiences and outcomes. British Journal of Educational Technology, 2013, 44(3): 372–385 DOI:10.1111/bjet.12024

11. Lameras P, Philippe, S, Oertel L. A serious game for amplifying awareness on multimodal teaching: game design and usability study. In: Advances in Intelligent Systems and Computing. Springer, 2019

12. Zyda M. From visual simulation to virtual reality to games. Computer, 2005, 38(9): 25–32 DOI:10.1109/mc.2005.297

13. Alamri A, Hassan M M, Hossain M A, Al-Qurishi M, Aldukhayyil Y, Hossain M S. Evaluating the impact of a cloud-based serious game on obese people. Computers in Human Behavior, 2014, 30: 468–475 DOI:10.1016/j.chb.2013.06.021

14. Clark D B, Tanner-Smith E E, Killingsworth S S. Digital games, design, and learning: a systematic review and meta-analysis. Review of Educational Research, 2016, 86(1): 79–122 DOI:10.3102/0034654315582065

15. Ritterfeld U, Cody M, Vorderer P. Serious Games: mechanisms and effects. Routledge, 2009

16. Djaouti D, Alvarez J, Jessel J P, Rampnoux O. Origins of serious games. In: Serious Games and Edutainment Applications. London, Springer, 2011, 25–43 DOI:10.1007/978-1-4471-2161-9_3

17. Jewitt C. Multimodality and literacy in school classrooms. Review of Research in Education, 2008, 32(1): 241–267 DOI:10.3102/0091732x07310586

18. Kress G, van Leeuwen T. Multimodal discourse: the modes and media of contemporary communication.

19. Hassett D D, Curwood J S. Theories and practices of multimodal education: the instructional dynamics of picture books and primary classrooms. The Reading Teacher, 2009, 63(4): 270–282 DOI:10.1598/rt.63.4.2

20. Souchet A. Visual fatigue impacts on learning via serious game in virtual reality. 2020

21. Yang K C C. Reality-creating technologies as a global phenomenon. In: Advances in Multimedia and Interactive Technologies. IGI Global, 2019, 1–18 DOI:10.4018/978-1-5225-5912-2.ch001

22. Anthes C, García-Hernández R J, Wiedemann M, Kranzlmüller D. State of the art of virtual reality technology. In: 2016 IEEE Aerospace Conference. Big Sky, MT, USA, IEEE, 2016, 1–19 DOI:10.1109/aero.2016.7500674

23. Arnaldi B, Cotin S, Couture N, Dautin J L, Gouranton V, Gruson F, Lourdeaux D. New applications. In: Virtual Reality and Augmented Reality. Hoboken, NJ, USA, John Wiley & Sons, Inc., 2018, 1–71 DOI:10.1002/9781119341031.ch1

24. Öman A, Sofkova Hashemi S. Design and redesign of a multimodal classroom task–implications for teaching and learning. Journal of Information Technology Education: Research, 2015, 14: 139–159 DOI:10.28945/2127

25. Alaker M, Wynn G R, Arulampalam T. Virtual reality training in laparoscopic surgery: a systematic review & meta-analysis. International Journal of Surgery (London, England), 2016, 29: 85–94 DOI:10.1016/j.ijsu.2016.03.034

26. Freina L, Ott M. A literature review on immersive virtual reality in education-state of the art and perspectives. In: Proceedings of eLearning and Software for Education (eLSE). Bucharest, Romania, 2015 DOI:10.12753/2066-026X-15-020

27. Kavanagh S, Luxton-Reilly A, Wuensche B, Plimmer B. A systematic review of virtual reality in education. Themes in Science and Technology Education, 2017,10(2): 85-119

28. Mikropoulos T A, Natsis A. Educational virtual environments: a ten-year review of empirical research (1999–2009). Computers & Education, 2011

29. Slater M, Sanchez-Vives M V. Enhancing our lives with immersive virtual reality. Frontiers in Robotics and AI, 2016, 3: 74 DOI:10.3389/frobt.2016.00074

30. Kao C L, Liao C Y, Lan Y J. Listening strategy applications by learners under the context of multimodality. In: 2017 IEEE 17th International Conference on Advanced Learning Technologies (ICALT). Timisoara, Romania, IEEE, 2017, 354–356 DOI:10.1109/icalt.2017.71

31. Kumar P, Agrawal A, Prasad S. Multimodal interface for temporal pattern based interactive large volumetric visualization. In: TENCON 2017-2017 IEEE Region 10 Conference. Penang, Malaysia, IEEE, 2017, 1239–1244 DOI:10.1109/tencon.2017.8228047

32. Godec M, Sternig S, Roth P M, Bischof H. Context-driven clustering by multi-class classification in an active learning framework. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops. San Francisco, CA, USA, IEEE, 2010, 19–24 DOI:10.1109/cvprw.2010.5543886

33. Zhou X K, Liang W, Wang K I K, Shimizu S. Multi-modality behavioral influence analysis for personalized recommendations in health social media environment. IEEE Transactions on Computational Social Systems, 2019, 6(5): 888–897 DOI:10.1109/tcss.2019.2918285

34. Kuang Q, Jin X, Zhao Q P, Zhou B. Deep multimodality learning for UAV video aesthetic quality assessment. IEEE Transactions on Multimedia, 2020 DOI:10.1109/tmm.2019.2960656

35. Gee J P. What video games have to teach us about learning and literacy. 2004

36. Norris S. Modal density and modal configurations. In: The Routledge Handbook of Multimodal Analysis. Routledge, 2009

37. Wandera D B. Teaching poetry through collaborative art: an analysis of multimodal ensembles for transformative learning. Journal of Transformative Education, 2016, 14(4): 305–326 DOI:10.1177/1541344616650749

38. Gilakjani A P, Ismail H N, Ahmadi S M. The effect of multimodal learning models on language teaching and learning. Theory and Practice in Language Studies, 2011, 1(10): 1321–1327 DOI:10.4304/tpls.1.10.1321-1327

39. Farías M, Obilinovic K, Orrego R. Implications of Multimodal Learning Models for foreign language teaching and learning. Colombian Applied Linguistics Journal, 2011(9): 174 DOI:10.14483/22487085.3150

40. Miller S M, McVee M B, Thompson M K, Lauricella A M, Boyd F B. A Literacy Pedagogy for Multimodal Composing: Transforming Learning and Teaching. 2013 DOI:10.4324/9780203804032

41. Sun M W. Application of multimodal learning in online English teaching. International Journal of Emerging Technologies in Learning, 2015, 10(4): 54–58 DOI:10.3991/ijet.v10i4.4697

42. Blondell M, Miller S M. Engaging literature through digital video composing: a teacher′s journey to “meaning that matters”. 2013 DOI:10.4324/9780203804032

43. Schmerbeck N, Lucht F. Creating meaning through multimodality: multiliteracies assessment and photo projects for online portfolios. Die Unterrichtspraxis/teaching German, 2017, 50(1): 32–44 DOI:10.1111/tger.12020

44. White C. The distance learning of foreign languages: a research agenda. Language Teaching, 2014, 47(4): 538–553 DOI:10.1017/s0261444814000196

45. Barber W, King S, Buchanan S. Authentic assessment in online learning: moving beyond text to celebrate multimodal measures of student achievement. In: Proceedings of the International Conference on e-Learning. 2014

46. Greenhow C, Robelia B, Hughes J E. Learning, teaching, and scholarship in a digital age web 2.0 and classroom research: what path should we take now? Educational Researcher, 2009, 38(4): 246–259 DOI:10.3102/0013189x09336671

47. Sharples M, de Rooc R, Ferguson R. Innovating pedagogy 2016: open university innovation report 5. Milton Keynes: The Open University. 2016

48. Mayer R E. Multimedia learning. Cambridge University Press, 2001 DOI:10.1017/CBO9781139164603

49. Hazari S, North A, Moreland D. Journal of information systems education. Investigating Pedagogical Value of Wiki Technology, 2006, 20(2): 187–198

50. Shah P, Freedman E G. Visuospatial cognition in electronic learning. Journal of Educational Computing Research, 2003, 29(3): 315–324 DOI:10.2190/qyvj-q59l-ve7c-ehuv

51. Parong J, Mayer R E. Learning science in immersive virtual reality. Journal of Educational Psychology, 2018, 110(6): 785–797 DOI:10.1037/edu0000241

52. Ogbuanya T C, Onele N O. Investigating the effectiveness of desktop virtual reality for teaching and learning of electrical/electronics technology in universities. Computers in The Schools, 2018, 35(3): 226–248 DOI:10.1080/07380569.2018.1492283

53. King D, Tee S, Falconer L, Angell C, Holley D, Mills A. Virtual health education: scaling practice to transform student learning: using virtual reality learning environments in healthcare education to bridge the theory/practice gap and improve patient safety. Nurse Education Today, 2018, 71: 7–9 DOI:10.1016/j.nedt.2018.08.002

54. Keskitalo T. Teachers′ conceptions and their approaches to teaching in virtual reality and simulation-based learning environments. Teachers and Teaching, 2011, 17(1): 131–147 DOI:10.1080/13540602.2011.538503

55. Gamage V, Tretiakov A, Crump B. Teacher perceptions of learning affordances of multi-user virtual environments. Computers & Education, 2011, 57(4): 2406–2413 DOI:10.1016/j.compedu.2011.06.015

56. Goehle G. Teaching with virtual reality: crafting a lesson and student response.(report). The International Journal for Technology in Mathematics Education, 2018, 25(1): 35 DOI:10.1564/tme_v25.1.04

57. Huang H M, Rauch U, Liaw S S. Investigating learners′ attitudes toward virtual reality learning environments: based on a constructivist approach. Computers & Education, 2010, 55(3): 1171–1182 DOI:10.1016/j.compedu.2010.05.014

58. Esteves M, Fonseca B, Morgado L, Martins P. Improving teaching and learning of computer programming through the use of the Second Life virtual world. British Journal of Educational Technology, 2011, 42(4): 624–637 DOI:10.1111/j.1467-8535.2010.01056.x

59. Ke F F, Lee S, Xu X H. Teaching training in a mixed-reality integrated learning environment. Computers in Human Behavior, 2016, 62: 212–220 DOI:10.1016/j.chb.2016.03.094

60. Nicely S, Farra S. Fostering learning through interprofessional virtual reality simulation development. Nursing Education Perspectives, 1900, 36(5): 335–336

61. Aiello P, Delia F, di Tore S, Sibilio M. A constructivist approach to virtual reality for experiential learning. E-learning and Digital Media, 2012, 9(3): 317–324 DOI:10.2304/elea.2012.9.3.317

62. Lindgren R, Johnson-Glenberg M C. Emboldened by embodiment: six precepts for research on embodied learning and mixed reality. Educational Researcher, 2013, 42(8): 445–452 DOI:10.3102/0013189x13511661

63. Tennyson R D, Volk A. Learning theories and educational paradigms. In: International Encyclopedia of the Social & Behavioral Sciences. Amsterdam, Elsevier, 2015, 699–711 DOI:10.1016/b978-0-08-097086-8.92036-1

64. Sherman W R, Craig A B. Understanding virtual reality: interface, application, and design. Presence: Teleoperators and Virtual Environments, 2003, 12(4): 441–442 DOI:10.1162/105474603322391668

65. Jensen L, Konradsen F. A review of the use of virtual reality head-mounted displays in education and training. Education and Information Technologies, 2018, 23(4): 1515–1529 DOI:10.1007/s10639-017-9676-0

66. Lin T J, Lan Y J. Language learning in virtual reality environments: past, present, and future. Educational Technology & Society, 2015, 18(4): 486–497

67. Innocenti E D, Geronazzo M, Vescovi D, Nordahl R, Serafin S, Ludovico L A, Avanzini F. Mobile virtual reality for musical genre learning in primary education. Computers & Education, 2019, 139: 102–117 DOI:10.1016/j.compedu.2019.04.010

68. Gorisse G, Christmann O, Amato E A, Richir S. First- and third-person perspectives in immersive virtual environments: presence and performance analysis of embodied users. Frontiers in Robotics and AI, 2017, 4: 33 DOI:10.3389/frobt.2017.00033

69. Cooper G, Thong L. Implementing virtual reality in the classroom: envisaging possibilities in STEM Education. Leiden, Koninklijke Brill NV, 2018

70. Slater M, Sanchez-Vives M V. Enhancing our lives with immersive virtual reality. Frontiers in Robotics and AI, 2016, 3: 74 DOI:10.3389/frobt.2016.00074

71. Kavanagh S, Luxton-Reilly A, Wuensche B, Plimmer B. A systematic review of virtual reality in education. Themes in Science and Technology Education, 2017, 10(2): 85–119

72. Velev D, Zlateva P. Virtual reality challenges in education and training. International Journal of Learning and Teaching. 2017, 3(1): 33–37 DOI:10.18178/ijlt.3.1.33-37

73. Selzer M N, Gazcon N F, Larrea M L. Effects of virtual presence and learning outcome using low-end virtual reality systems. Displays, 2019, 59: 9–15 DOI:10.1016/j.displa.2019.04.002

74. Nersesian E, Spryszynski A, Lee M J. Integration of virtual reality in secondary STEM education. In: 2019 IEEE Integrated STEM Education Conference (ISEC). Princeton, NJ, USA, IEEE, 2019, 83–90 DOI:10.1109/isecon.2019.8882070

75. Mikropoulos T A, Natsis A. Educational virtual environments: a ten-year review of empirical research (1999–2009). Computers & Education, 2011, 56(3): 769–780 DOI:10.1016/j.compedu.2010.10.020

76. Alaker M, Wynn G R, Arulampalam T. Virtual reality training in laparoscopic surgery: a systematic review & meta-analysis. International Journal of Surgery (London, England), 2016, 29: 85–94 DOI:10.1016/j.ijsu.2016.03.034

77. Lanier M, Waddell T F, Elson M, Tamul D J, Ivory J D, Przybylski A. Virtual reality check: statistical power, reported results, and the validity of research on the psychology of virtual reality and immersive environments. Computers in Human Behavior, 2019, 100: 70–78 DOI:10.1016/j.chb.2019.06.015

78. Dickey M D. Teaching in 3D: pedagogical affordances and constraints of 3D virtual worlds for synchronous distance learning. Distance Education, 2003, 24(1): 105–121 DOI:10.1080/01587910303047

79. Doumanis I, Economou D, Sim G R, Porter S. The impact of multimodal collaborative virtual environments on learning: a gamified online debate. Computers & Education, 2019, 130: 121–138 DOI:10.1016/j.compedu.2018.09.017

80. Al Zayer M, MacNeilage P, Folmer E. Virtual locomotion: a survey. IEEE Transactions on Visualization and Computer Graphics, 2020, 26(6): 2315–2334 DOI:10.1109/tvcg.2018.2887379

81. Lave J, Wenger E. Situated learning: legitimate peripheral participation. Cambridge University Press, 1991

82. Marky K, Müller F, Funk M, Geiß A, Günther S, Schmitz M, Riemann J, Mühlhäuser M. Teachyverse: collaborative E-learning in virtual reality lecture halls. In: Proceedings of Mensch und Computer 2019 on-MuC'19. Hamburg, Germany, New York, ACM Press, 2019, 831–834 DOI:10.1145/3340764.3344917

83. Churchill E F, Snowdon D. Collaborative virtual environments: an introductory review of issues and systems. Virtual Reality, 1998, 3(1): 3–15 DOI:10.1007/bf01409793

84. Dehaene S. Apprendre ! les talents du cerveau, le défi des machines.

85. Weiss P L, Jessel A S. Virtual reality applications to work. Work, 1998, 11(3): 277–293 DOI:10.3233/wor-1998-11305

86. Suh A, Prophet J. The state of immersive technology research: a literature analysis. Computers in Human Behavior, 2018, 86: 77–90 DOI:10.1016/j.chb.2018.04.019

87. Freina L, Ott M. A literature review on immersive virtual reality in education: state of the art and perspectives. In: eLearning and Software for Education (eLSE). Bucharest, Romania, 2015

88. Slater M. Implicit learning through embodiment in immersive virtual reality. In: Smart Computing and Intelligence. Singapore: Springer Singapore, 2017, 19–33. DOI:10.1007/978-981-10-5490-7_2

89. Duncan I, Miller A, Jiang S Y. A taxonomy of virtual worlds usage in education. British Journal of Educational Technology, 2012, 43(6): 949–964 DOI:10.1111/j.1467-8535.2011.01263.x

90. Fuchs P. Virtual reality headsets-a theoretical and pragmatic approach. CRC Press, 2017

91. Cummings J J, Bailenson J N. How immersive is enough? A meta-analysis of the effect of immersive technology on user presence. Media Psychology, 2016, 19(2): 272–309 DOI:10.1080/15213269.2015.1015740

92. Murcia López M. The effectiveness of training in virtual environments. 2018

93. Lamb R, Annetta L, Firestone J B, Etopio E. A meta-analysis with examination of moderators of student cognition, affect, and learning outcomes while using serious educational games, serious games, and simulations. Computers in Human Behavior, 2018, 80: 158–167 DOI:10.1016/j.chb.2017.10.040

94. Menin A, Torchelsen R, Nedel L. An analysis of VR technology used in immersive simulations with a serious game perspective. IEEE Computer Graphics and Applications, 2018, 38(2): 57–73 DOI:10.1109/mcg.2018.021951633

95. Zheng L, Xie T, Liu G P. Affordances of virtual reality for collaborative learning. In: 2018 International Joint Conference on Information, Media and Engineering (ICIME). Osaka, Japan, IEEE, 2018, 6–10 DOI:10.1109/icime.2018.00011

96. Souchet A D, Granier de Cassagnac R, Maurice E,et al. Virtual classroom’s quality of experience: a collaborative VR platform tested. In: ConVRgence (VRIC) Virtual Reality International Conference Proceedings, 2020

97. Milgram P, Takemura H, Utsumi A, Kishino F. Augmented reality: a class of displays on the reality-virtuality continuum. In: Proc SPIE 2351, Telemanipulator and Telepresence Technologies, 1995, 2351: 282–292 DOI:10.1117/12.197321

98. Varjo. product xr-1.

99. Lynx product.

100. Wigdor D, Wixon D. Brave NUI world: designing natural user interfaces for touch and gesture. Morgan Kaufmann, 2011

101. Shadow.

102. IDATE. Le Marché Des Technologies Immersives (VR/AR/MR). 2019

103. SourceHTC VIVE China. HTC conducts its first virtual "VIVE Ecosystem Conference" (V²EC) fully in VR allowing global attendees to join virtual sessions. 2020

104. Laval Virtual 2020.

105. IBC 2019.

106. UNESCO & COVID19.

107. Marie-Christine Levet Interview.