Adv Search
Home | Accepted | Article In Press | Current Issue | Archive | Special Issues | Collections | Featured Articles | Statistics

2019, 1(1): 84-112 Published Date:2019-2-20

DOI: 10.3724/SP.J.2096-5796.2018.0006

Gesture interaction in virtual reality

Full Text: PDF (112) HTML (940)

Export: EndNote | Reference Manager | ProCite | BibTex | RefWorks


With the development of virtual reality (VR) and human-computer interaction technology, how to use natural and efficient interaction methods in the virtual environment has become a hot topic of research. Gesture is one of the most important communication methods of human beings, which can effectively express users’ demands. In the past few decades, gesture-based interaction has made significant progress. This article focuses on the gesture interaction technology and discusses the definition and classification of gestures, input devices for gesture interaction, and gesture interaction recognition technology. The application of gesture interaction technology in virtual reality is studied, the existing problems in the current gesture interaction are summarized, and the future development is prospected.
Keywords: Virtual reality ; Gesture interaction ; Gesture recognition

Cite this article:

Yang LI, Jin HUANG, Feng TIAN, Hong-An WANG, Guo-Zhong DAI. Gesture interaction in virtual reality. Virtual Reality & Intelligent Hardware, 2019, 1(1): 84-112 DOI:10.3724/SP.J.2096-5796.2018.0006

1. Pantic M, Nijholt A, Pentland A, Huang T S. Human-Centred Intelligent Human Computer Interaction (HCI²): how far are we from attaining it? International Journal of Autonomous and Adaptive Communications Systems,2008, 1(2): 168−187 DOI:10.1504/IJAACS.2008.019799

2. Karam M. A framework for research and design of gesture-based human-computer interactions. Doctoral. University of Southampton, 2006

3. Dam A V. Post-WIMP user interfaces. Communications of the ACM, 1997, 40(2): 63−67 DOI:10.1145/253671.253708

4. Green M, Jacob R. Software architectures and metaphors for non-wimp user interfaces. ACM SIGGRAPH Computer Graphics, 1991, 25(3): 229−235 DOI:10.1145/126640.126677

5. Mitra S, Acharya T. Gesture Recognition: A Survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 2007, 37(3): 311−324 DOI:10.1109/TSMCC.2007.893280

6. Cerney M M, Vance J M. Gesture recognition in virtual environments: a review and framework for future development. Iowa State University Human Computer Interaction Technical Report ISU-HCI-2005-01. 2005

7. Parvini F, Shahabi C. An algorithmic approach for static and dynamic gesture recognition utilising mechanical and biomechanical characteristics. International Journal of Bioinformatics Research and Applications, 2007, 3(1): 4−23 DOI:10.1504/ijbra.2007.011832

8. Rogers Y, Sharp H, Preece J. Interaction design: beyond human-computer interaction. John Wiley Sons, 2011

9. Rautaray S S, Agrawal A. Vision based hand gesture recognition for human computer interaction: a survey. Artificial Intelligence Review, 2015, 43(1): 1−54 DOI:10.1007/s10462-012-9356-9

10. Pavlovic V, Sharma R, Huang T S. Visual interpretation of hand gestures for human-computer interaction: a review. IEEE Transactions on Pattern Analysis Machine Intelligence, 1997, 19(7): 677−695 DOI:10.1109/34.598226

11. Ottenheimer H J. The anthropology of language: an introduction to linguistic anthropology. Wadsworth Publishing. 2005

12. McNeill D. Hand and mind: What gestures reveal about thought. Chicago, USA: University of Chicago Press. 1992

13. Kanniche M B. Gesture recognition from video sequences. PhD Thesis, University of Nice. 2009

14. International standards: ISO/IEC 30113-11: 2017(E)

15. Nishikawa A, Hosoi T, Koara K, Negoro D, Hikita A, Asano S, Kakutani H, Miyazaki F, Sekimoto M, Yasui M, Miyake Y, Takiguchi S, Monden M. FAce MOUSe: A novel human-machine interface for controlling the position of a laparoscope. IEEE Transactions on Robotics and Automation, 2003, 19(5): 825−841 DOI:10.1109/TRA.2003.817093

16. Tarchanidis K N, Lygouras J N. Data glove with a force sensor. IEEE Transactions on Instrumentation and Measurement, 2003, 52(3): 984−989 DOI:10.1109/TIM.2003.809484

17. Temoche P, Esmitt R J, Rodríguez O. A low-cost data glove for virtual reality. Xi International Congress of Numerical Methods in Engineering and Applied Sciences. 2012, TCG31−36

18. Furness T A. The Super Cockpit and its Human Factors Challenges. Proceedings of the Human Factors Society Annual Meeting, 1986, 30(1): 48−52 DOI:10.1177/154193128603000112

19. Rekimoto J. GestureWrist and GesturePad: unobtrusive wearable interaction devices. In: Proceedings Fifth International Symposium on Wearable Computers, 2001, 21−27 DOI:10.1109/ISWC.2001.962092

20. Baek J, Jang I J, Park K, Kang H S, Yun B J. Human Computer Interaction for the Accelerometer-Based Mobile Game. In: Embedded and Ubiquitous Computing. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006, 509−518 DOI:10.1007/11802167_52

21. Webster J G. Medical Instrumentation-Application and Design. 1978, 3(3): 306

22. Jain A, Bhargava D B, Rajput A. Touch-screen technology. International Journal of Advanced Research in Computer Science and Electronics Engineering, 2013, 2(1)

23. Subrahmonia J, Zimmerman T. Pen computing: challenges and applications. In: Proceedings 15th International Conference on Pattern Recognition ICPR-2000. Barcelona, Spain, 2000, 60−66 DOI:10.1109/ICPR.2000.906018

24. Freeman W T Roth M. Orientation histograms for hand gesture recognition. In: International workshop on automatic face and gesture recognition. 1995, 12: 296−301

25. Zhang L G, Wu J Q, Gao W, Yao H X. Hand gesture recognition based on Hausdorff Distance. Journal of Image and Graphics, 2002, 7(11): 1144−1150 DOI:10.11834/jig.2002011341

26. Meng C N, Lv J P, Chen X H. Gesture recognition based on universal infrared camera. Computer Engineering and Applications. 2015, 51(16): 17−22

27. Weichert F, Bachmann D, Rudak B, Fisseler D. Analysis of the Accuracy and Robustness of the Leap Motion Controller. Sensors, 2013, 13(5): 6380−6393 DOI:10.3390/s130506380

28. Chen Y, Ding Z, Chen Y, Wu X. Rapid recognition of dynamic hand gestures using leap motion. In: 2015 IEEE International Conference on Information and Automation. 2015, 1419−1424 DOI:10.1109/ICInfA.2015.7279509

29. Anonymous. USens CTO Detailed Human-Computer Interactive Tracking Technology. Computer Telecommunica-tion, 2016(4): 12−13

30. Anonymous. Virtual Reality Technology Trends to “Bare Hand Manipulation”. Machine Tool Hydraulics, 2017(8): 38−38

31. Gu Y, Do H, Ou Y, Sheng W. Human gesture recognition through a Kinect sensor. In: 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO). 2012, 1379–1384 DOI:10.1109/ROBIO.2012.6491161

32. Ren Z, Yuan J, Meng J, Zhang Z. Robust Part-Based Hand Gesture Recognition Using Kinect Sensor. IEEE Transactions on Multimedia. 2013, 15(5): 1110–1120 DOI:10.1109/TMM.2013.2246148

33. Raheja J L, Chaudhary A, Singal K. Tracking of Fingertips and Centers of Palm Using KINECT. In: 2011 Third International Conference on Computational Intelligence, Modelling Simulation, 2011, 248−252 DOI:10.1109/CIMSim.2011.51

34. Ren Z, Meng J, Yuan J, Zhang Z: Robust hand gesture recognition with kinect sensor. In: Proceedings of the 19th ACM international conference on Multimedia. Scottsdale, Arizona, USA, ACM, 2011: 759−760 DOI:10.1145/2072298.2072443

35. Han Y. A low-cost visual motion data glove as an input device to interpret human hand gestures. IEEE Transactions on Consumer Electronics. 2010, 56(2): 501−509 DOI:10.1109/TCE.2010.5505962

36. Mistry P, Maes P, Chang L. WUW - wear Ur world: a wearable gestural interface. In: CHI '09 Extended Abstracts on Human Factors in Computing Systems. Boston. MA, USA, ACM, 2009: 4111−4116 DOI:10.1145/1520340.1520626

37. Rautaray S S, Agrawal A. Vision based hand gesture recognition for human computer interaction: a survey. Artificial Intelligence Review, 2015, 43(1): 1−54 DOI:10.1007/s10462-012-9356-9

38. Mehdi S A, Khan Y N. Sign language recognition using sensor gloves. In: Proceedings of the 9th International Conference on Neural Information Processing. 2002, 2204−2206 DOI:10.1109/ICONIP.2002.1201884

39. Shi J F, Chen Y, Zhao H M. Node-Pair BP Network Based Gesture Recognition by Data Glove. System Simulation Technology, 2008, 4(3): 154−157 DOI:10.3969/j.issn.1673-1964.2008.03.003

40. Rung-Huei L, Ming O. A real-time continuous gesture recognition system for sign language. In: Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition. 1998, 558−567 DOI:10.1109/AFGR.1998.671007

41. Liu M T, Lei Y. Chinese finger Alphabet flow recognition system based on data glove. Computer Engineering, 2011, 37(22): 168−170

42. Wu J Q, Gao W, Chen L X. A system recognizing Chinese finger-spelling alphabets based on data-glove input. Pattern Recognition and Artificial Intelligence, 1999(1): 74−78

43. Weissmann J, Salomon R. Gesture recognition for virtual reality applications using data gloves and neural networks. In: IJCNN'99 International Joint Conference on Neural Networks Proceedings. 1999, 2043−2046 DOI:10.1109/IJCNN.1999.832699

44. Xu Y H, Li J R. Research and implementation of virtual hand interaction in virtual mechanical assembly. Machinery Design Manufacture, 2014(5): 262−266

45. Mirabella O, Brischetto M, Mastroeni G. MEMS based gesture recognition. In: 3rd International Conference on Human System Interaction. 2010, 599−604 DOI:10.1109/HSI.2010.5514506

46. Kela J, Korpipää P, Mäntyjärvi J, Kallio S, Savino G, Jozzo L, Marca D. Accelerometer-based gesture control for a design environment. Personal Ubiquitous Computing, 2006, 10(5): 285−299 DOI:10.1007/s00779-005-0033-8

47. Xu J, Liu C H, Meng Y X. Gesture recognition base on wearable controller. Application of Electronic Technique, 2016, 42(7): 68−71

48. He Z Y, Jin L W, Zhen L X, Huang J C. Gesture recognition based on 3D accelerometer for cell phones interaction. In: APCCAS2008 - 2008IEEE Asia Pacific Conference on Circuits and Systems. 2008, 217−220 DOI:10.1109/APCCAS.2008.4745999

49. Schlömer T, Poppinga B, Henze N, Boll S. Gesture recognition with a Wii controller. In: Proceedings of the 2nd international conference on Tangible and embedded interaction. Bonn, Germany, ACM, 2008: 11−14 DOI:10.1145/1347390.1347395

50. Du Y, Jin W, Wei W, Hu Y, Geng W. Surface EMG-Based Inter-Session Gesture Recognition Enhanced by Deep Domain Adaptation. 2017, 17(3): 458 DOI:10.3390/s17030458

51. Kim J, Mastnik S, André E. EMG-based hand gesture recognition for realtime biosignal interfacing. In: Proceedings of the 13th international conference on Intelligent user interfaces. Gran Canaria, Spain, ACM, 2008: 30−39 DOI:10.1145/1378773.1378778

52. Madhavan G. Electromyography: physiology, engineering and non-invasive applications. Annals of Biomedical Engineering, 2005, 33(11): 1671

53. Saponas T S, Tan D S, Morris D, Balakrishnan R. Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Florence, Italy, ACM, 2008: 515−524 DOI:10.1145/1357054.1357138

54. Saponas T S, Tan D S, Morris D, Balakrishnan R, Turner J, Landay J A. Enabling always-available input with muscle-computer interfaces. In: Proceedings of the 22nd annual ACM symposium on User interface software and technology. Victoria, BC, Canada, ACM, 2009: 167−176 DOI:10.1145/1622176.1622208

55. Saponas T S, Tan D S, Morris D, Turner J, Landay J A. Making muscle-computer interfaces more practical. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Atlanta, Georgia, USA, ACM, 2010: 851−854 DOI:10.1145/1753326.1753451

56. Amma C, Krings T, Böer J, Schultz T. Advancing Muscle-Computer Interfaces with High-Density Electromyography. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. Seoul, Republic of Korea, ACM, 2015: 929−938 DOI:10.1145/2702123.2702501

57. Huang D, Zhang X, Saponas T S, Fogarty J, Gollakota S. Leveraging Dual-Observable Input for Fine-Grained Thumb Interaction Using Forearm EMG. In: Proceedings of the 28th Annual ACM Symposium on User Interface Software Technology. Charlotte, NC, USA, ACM, 2015: 523−528 DOI:10.1145/2807442.2807506

58. Rubine D H. The automatic recognition of gestures. Carnegie Mellon University, 1992

59. Wobbrock J O, Wilson A D, Li Y. Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. In: Proceedings of the 20th annual ACM symposium on User interface software and technology. Newport, Rhode Island, USA, ACM, 2007: 159−168 DOI:10.1145/1294211.1294238

60. Anthony L, Wobbrock J O. A lightweight multistroke recognizer for user interface prototypes. In: Proceedings of Graphics Interface 2010. Ottawa, Ontario, Canada, Canadian Information Processing Society, 2010: 245−252

61. VatavuR-D, Anthony L, Wobbrock J O. Gestures as point clouds: a $P recognizer for user interface prototypes. In: Proceedings of the 14th ACM international conference on Multimodal interaction. Santa Monica, California, USA, ACM, 2012: 273−280 DOI:10.1145/2388676.2388732

62. Hackenberg G, McCall R, Broll W. Lightweight palm and finger tracking for real-time 3D gesture control. In: 2011 IEEE Virtual Reality Conference. 2011, 19−26 DOI:10.1109/VR.2011.5759431

63. Freeman W T, Weissman C D. Television control by hand gestures. International Workshop on Automatic Face Gesture Recognition, 1995: 179−183

64. Kaufmann B, Louchet J, Lutton E. Hand Posture Recognition Using Real-Time Artificial Evolution. In: Applications of Evolutionary Computation. Berlin, Heidelberg, Springer Berlin Heidelberg, 2010, 251−260 DOI:10.1007/978-3-642-12239-2_26

65. Flasiński M, Myśliński S. On the use of graph parsing for recognition of isolated hand postures of Polish Sign Language. Pattern Recognition, 2010, 43(6): 2249−2264 DOI:10.1016/j.patcog.2010.01.004

66. Bergh M V d, Gool L V. Combining RGB and ToF cameras for real-time 3D hand gesture interaction. In: Proceedings of the 2011 IEEE Workshop on Applications of Computer Vision (WACV). IEEE Computer Society, 2011: 66−72 DOI:10.1109/WACV.2011.5711485

67. Jones M J, Rehg J M. Statistical Color Models with Application to Skin Detection. International Journal of Computer Vision, 2002, 46(1): 81−96 DOI:10.1023/A:1013200319198

68. Weng C, Li Y, Zhang M, Guo K, Tang X, Pan Z. Robust Hand Posture Recognition Integrating Multi-cue Hand Tracking. In: Entertainment for Education Digital Techniques and Systems. Berlin, Heidelberg, Springer Berlin Heidelberg, 2010, 497−508 DOI:10.1007/978-3-642-14533-9_51

69. Ju S X, Black M J, Yacoob Y. Cardboard People: A Parameterized Model of Articulated Image Motion. In: Proceedings of the 2nd International Conference on Automatic Face and Gesture Recognition. IEEE Computer Society; 1996: 38

70. Kervrann C, Heitz F. Learning structure and deformation modes of nonrigid objects in long image sequences. 1995

71. Ren H B, Xu G H, Lin X Y. Hand gesture recognition based on characteristic curves. Journal of Software, 2002, 13(5): 987−993

72. Priyal P S, Bora P K. A robust static hand gesture recognition system using geometry based normalizations and Krawtchouk moments. Pattern Recognition, 2013, 46(8): 2202−2219 DOI:10.1016/j.patcog.2013.01.033

73. Ju S X, Black M J, Minneman S, Kimber D. Analysis of Gesture and Action in Technical Talks for Video Indexing. In: Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, 1997: 595

74. Luo Q, Kong X, Zeng G, Fan J. Human action detection via boosted local motion histograms. Machine Vision and Applications, 2010, 21(3): 377−389 DOI:10.1007/s00138-008-0168-5

75. Shotton J, Fitzgibbon A, Cook M, Sharp T, Finocchio M, Moore R, Kipman A, Blake A. Real-time human pose recognition in parts from single depth images. In: CVPR 2011, 2011, 1297−1304 DOI:10.1109/CVPR.2011.5995316

76. Kass M, Witkin A, TerzopoulosD. Snakes. Active contour models. International Journal of Computer Vision, 1988, 1(4): 321−331 DOI:10.1007/BF00133570

77. Lu W-L, Little J J. Simultaneous Tracking and Action Recognition using the PCA-HOG Descriptor. In: Proceedings of the The 3rd Canadian Conference on Computer and Robot Vision. IEEE Computer Society, 2006: 6 DOI:10.1109/CRV.2006.66

78. Moni M A, Ali A B M S. HMM based hand gesture recognition. A review on techniques and approaches. In: 2009 2nd IEEE International Conference on Computer Science and Information Technology. 2009, 433−437 DOI:10.1109/ICCSIT.2009.5234536

79. Keskin C, Erkan A, Akarun L. Real time hand tracking and 3D gesture recognition for interactive interfaces using HMM. In: Proceedings of International Conference on Artificial Neural Networks. 2003

80. Chai X, Liu Z, Yin F, Liu Z, Chen X. Two streams Recurrent Neural Networks for Large-Scale Continuous Gesture Recognition. In: 2016 23rd International Conference on Pattern Recognition, 2016, 31−36 DOI:10.1109/ICPR.2016.7899603

81. Tsironi E, Barros P, Wermter S. Gesture recognition with a convolutional long short-term memory recurrent neural network. In: Proceedings of the European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN). Bruges, Belgium, 2016, 213−218

82. Bolt R A. “Put-that-there”: Voice and gesture at the graphics interface. Acm Siggraph Computer Graphics, 1980, 14(3): 262−270

83. Cohen P R, Johnston M, McGee D, Oviatt S, Pittman J, Smith I, Chen L, Clow J. QuickSet: multimodal interaction for distributed applications. In: Proceedings of the fifth ACM international conference on Multimedia. Seattle, Washington, USA, ACM, 1997: 31−40

84. Chatterjee I, Xiao R, Harrison C. Gaze+Gesture: Expressive, Precise and Targeted Free-Space Interactions. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. Seattle, Washington, USA, ACM, 2015: 131−138 DOI:10.1145/2818346.2820752

85. Velloso E, Turner J, Alexander J, Bulling A, Gellersen H. An Empirical Investigation of Gaze Selection in Mid-Air Gestural 3D Manipulation. In: Human-Computer Interaction – INTERACT 2015. Cham: Springer International Publishing, 2015, 315−330 DOI:10.1007/978-3-319-22668-2_25

86. Zhang F J, Dai G Z, Peng X L. A survey on human-computer interaction in virtual reality. Scientia Sinica (Informationis), 2016(12): 1711−1736 DOI:10.1360/N112016-00252

87. Wu L. Oviatt S L. Cohen P R. Multimodal integration-a statistical view. IEEE Transactions on Multimedia, 1999, 1(4): 334−341 DOI:10.1109/6046.807953

88. Sim K C. Speak-as-you-swipe (SAYS): a multimodal interface combining speech and gesture keyboard synchronously for continuous mobile text entry. In: Proceedings of the 14th ACM international conference on Multimodal interaction. Santa Monica, California, USA, ACM, 2012: 555−560 DOI:10.1145/2388676.2388793

89. Kopp S, Tepper P, Cassell J. Towards integrated microplanning of language and iconic gesture for multimodal output. In: Proceedings of the 6th International Conference on Multimodal Interfaces. New York, NY, ACM Press, 2004: 97−104

90. Ruppert G C S, Reis L O, Amorim P H J, de Moraes T F, da Silva J V L. Touchless gesture user interface for interactive image visualization in urological surgery. World Journal of Urology, 2012, 30(5): 687−691 DOI:10.1007/s00345-012-0879-0

91. Wachs J P, Stern H I, Edan Y, Gillam M. , Handler J. , Feied C. , Smith M. A gesture-based tool for sterile browsing of radiology images. J Am Med Inform Assoc, 2008, 15(3): 321−323 DOI:10.1197/jamia.M2410

92. Keskin C, Balci K, Aran O, Sankur B, Akarun L. A Multimodal 3D Healthcare Communication System. In: 2007 3DTV Conference, 2007, 1−4 DOI:10.1109/3DTV.2007.4379488

93. Phelan I, Arden M, Garcia C, Roast C. Exploring virtual reality and prosthetic training. In: 2015 IEEE Virtual Reality (VR), 2015, 353−354 DOI:10.1109/VR.2015.7223441

94. Moustakas K, Nikolakis G, Tzovaras D, Strintzis M G. A geometry education haptic VR application based on a new virtual hand representation. In: IEEE Proceedings VR 2005 Virtual Reality, 2005, 249−252 DOI:10.1109/VR.2005.1492782

95. Vrellis I, Moutsioulis A, Mikropoulos T A. Primary School Students’ Attitude towards Gesture Based Interaction: A Comparison between Microsoft Kinect and Mouse. In: 2014 IEEE 14th International Conference on Advanced Learning Technologies, 2014, 678−682 DOI:10.1109/ICALT.2014.199

96. Zhang X, Chen X, Li Y, Lantz V, Wang K, Yang J. A Framework for Hand Gesture Recognition Based on Accelerometer and EMG Sensors. IEEE Transactions on Systems, Man, and Cybernetics―Part A: Systems and Humans, 2011, 41(6): 1064−1076 DOI:10.1109/TSMCA.2011.2116004

97. Zhang F, Chu S, Pan R, Ji N, Xi L. Double hand-gesture interaction for walk-through in VR environment. In: 2017 IEEE/ACIS 16th International Conference on Computer and Information Science (ICIS). Wuhan, IEEE, 2017, 539−544 DOI:10.1109/JCSSE.2015.7219818

98. Khundam C. First person movement control with palm normal and hand gesture interaction in virtual reality. In: 2015 12th International Joint Conference on Computer Science and Software Engineering (JCSSE). Songkhla, IEEE, 2015, 325−330 DOI:10.1109/JCSSE.2015.7219818

99. Park H, Jeong S, Kim T, Youn D and Kim K. Visual representation of gesture interaction feedback in virtual reality games. In: International Symposium on Ubiquitous Virtual Reality. Nara, IEEE, 2017, 20−23 DOI:10.1109/ISUVR.2017.14

100. Latoschik M E: A gesture processing framework for multimodal interaction in virtual reality. In: Proceedings of the 1st international conference on Computer graphics, virtual reality and visualisation. Cape Town, ACM, 2001: 95−100 DOI:10.1145/513867.513888

101. Chun L M, Arshad H, Piumsomboon T, Billinghurst M. A combination of static and stroke gesture with speech for multimodal interaction in a virtual environment. In: 2015 International Conference on Electrical Engineering and Informatics (ICEEI). Denpasar, IEEE, 2015: 59−64 DOI:10.1109/ICEEI.2015.7352470

email E-mail this page

Articles by authors