Chinese
Adv Search
Home | Accepted | Article In Press | Current Issue | Archive | Special Issues | Collections | Featured Articles | Statistics

2019, 1(3): 276-289 Published Date:2019-6-20

DOI: 10.3724/SP.J.2096-5796.2019.0007

Gesture-based target acquisition in virtual and augmented reality

Full Text: PDF (24) HTML (89)

Export: EndNote | Reference Manager | ProCite | BibTex | RefWorks

Abstract:

Background
Gesture is a basic interaction channel that is frequently used by humans to communicate in daily life. In this paper, we explore to use gesture-based approaches for target acquisition in virtual and augmented reality. A typical process of gesture-based target acquisition is: when a user intends to acquire a target, she performs a gesture with her hands, head or other parts of the body, the computer senses and recognizes the gesture and infers the most possible target.
Methods
We build mental model and behavior model of the user to study two key parts of the interaction process. Mental model describes how user thinks up a gesture for acquiring a target, and can be the intuitive mapping between gestures and targets. Behavior model describes how user moves the body parts to perform the gestures, and the relationship between the gesture that user intends to perform and signals that computer senses.
Results
In this paper, we present and discuss three pieces of research that focus on the mental model and behavior model of gesture-based target acquisition in VR and AR.
Conclusions
We show that leveraging these two models, interaction experience and performance can be improved in VR and AR environments.
Keywords: Gesture-based interaction ; Mental model, Behavior model ; Virtual reality ; Augmented reality

Cite this article:

Yukang YAN, Xin YI, Chun YU, Yuanchun SHI. Gesture-based target acquisition in virtual and augmented reality. Virtual Reality & Intelligent Hardware, 2019, 1(3): 276-289 DOI:10.3724/SP.J.2096-5796.2019.0007

1. Geoffrey M. Davis. Virtual reality game method and apparatus. US Patent, 5423554, 1995-06-13

2. Gallagher A G, Ritter E M, Champion H, Higgins G, Fried M P, Moses G, Smith C D, Satava R M. Virtual reality simulation for the operating room: Proficiency-based training as a paradigm shift in surgical skills training. Annals of Surgery, 2005, 241(2): 364–372

3. Kaufmann H, Schmalstieg D, Wagner M. Construct 3D: A virtual reality application for mathematics and geometry education. Education and Information Technologies, 2000, 5(4): 263–276 DOI:10.1023/A:1012049406877

4. Bowman D A, Wingrave C A. Design and evaluation of menu systems for immersive virtual environments. In: Proceedings IEEE Virtual Reality 2001, Yokohama, Japan, 2001, 149–156 DOI:10.1109/VR.2001.913781

5. Baudel T, Beaudouin-Lafon M. Charade: remote control of objects using free-hand gestures. Communications of the ACM, 1993, 36(7): 28–35 DOI:10.1145/159544.159562

6. Nacenta M A, Kamber Y, Qiang Y, Kristensson P O. Memorability of pre-designed and user-defined gesture sets. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Paris, France: ACM, 2013, 1099–1108 DOI:10.1145/2470654.2466142

7. Wagner J, Lecolinet E, Selker T. Multi-finger chords for hand-held tablets: recognizable and memorable. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Toronto, Canada, ACM, 2014, 2883–2892 DOI:10.1145/2556288.2556958

8. Kulshreshth A, Joseph J. LaViola J. Exploring the usefulness of finger-based 3D gesture menu selection. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Toronto, Canada, ACM, 2014, 1093–1102 DOI:10.1145/2556288.2557122

9. Wobbrock J O, Morris M R, Wilson A D. User-defined gestures for surface computing. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Boston, USA, ACM, 2009, 1083–1092 DOI:10.1145/1518701.1518866

10. Bragdon A, KoH-S. Gesture select: acquiring remote targets on large displays without pointing. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Vancouver, Canada, ACM, 2011, 187–196 DOI:10.1145/1978942.1978970

11. Carter M, Velloso E, Downs J, Sellen A, O'Hara K, Vetere F. PathSync. Multi-User Gestural Interaction with Touchless Rhythmic Path Mimicry. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. San Jose, USA, ACM, 2016, 3415–3427 DOI:10.1145/2858036.2858284

12. Esteves A, Velloso E, Bulling A, Gellersen H. Orbits: Gaze Interaction for Smart Watches using Smooth Pursuit Eye Movements. In: Proceedings of the 28th Annual ACM Symposium on User Interface Software&Technology. Charlotte, USA, ACM, 2015, 457–466 DOI:10.1145/2807442.2807499

13. Yatani K, Partridge K, Bern M, Newman M W. Escape: a target selection technique using visually-cued gestures. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Florence, Italy, ACM, 2008, 285–294 DOI:10.1145/1357054.1357104.

14. Shimon S S A, Lutton C, Xu Z, Morrison-Smith S, Boucher C, Ruiz J. Exploring Non-touchscreen Gestures for Smartwatches. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. San Jose, USA, ACM, 2016, 3822–3833 DOI:10.1145/2858036.2858385

15. Kray C, Nesbitt D, Dawson J, Rohs M. User-defined gestures for connecting mobile phones, public displays, and tabletops. In: Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. Lisbon, Portugal, ACM, 2010, 239–248 DOI:10.1145/1851600.1851640

16. Ruiz J, Li Y, Lank E. User-defined motion gestures for mobile interaction. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Vancouver, Canada, ACM, 2011, 197–206 DOI:10.1145/1978942.1978971

17. Vatavu R-D: User-defined gestures for free-hand TV control. In: Proceedings of the 10th European Conference on Interactive TV and Video. Berlin, Germany, ACM, 2012, 45–48 DOI:10.1145/2325616.2325626

18. Cockburn A, Quinn P, Gutwin C, Ramos G, Looser J. Air pointing: Design and evaluation of spatial target acquisition with and without visual feedback. International Journal of Human-Computer Studies, 2011, 69(6): 401–414 DOI:10.1016/j.ijhcs.2011.02.005

19. Johnson E A. A study of the effects of immersion on short-term spatial memory. Purdue Polytechnic Masters Theses. Purdue University. 2010

20. Cockburn A, McKenzie B. Evaluating the effectiveness of spatial memory in 2D and 3D physical and virtual environments. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Minneapolis, USA, ACM, 2002, 203–210DOI:10.1145/503376.503413

21. Gutwin C, Cockburn A, Gough N. A Field Experiment of Spatially-Stable Overviews for Document Navigation. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. Denver, USA, ACM, 2017, 5905–5916 DOI:10.1145/3025453.3025905

22. Li F C Y, Dearman D, Truong K N. Leveraging proprioception to make mobile phones more accessible to users with visual impairments. In: Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility. Orlando, USA, ACM, 2010, 187–194 DOI:10.1145/1878803.1878837

23. Gugenheimer J, Dobbelstein D, Winkler C, Haas G, Rukzio E. FaceTouch: Enabling Touch Interaction in Display Fixed UIs for Mobile Virtual Reality. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology. Tokyo, Japan, ACM, 2016, 49–60 DOI:10.1145/2984511.2984576

24. Li F C Y, Dearman D, Truong K N. Virtual shelves: interactions with orientation aware devices. In: Proceedings of the 22nd annual ACM symposium on User interface software and technology. Victoria, Canada, ACM, 2009, 125–128 DOI:10.1145/1622176.1622200

25. Lopes P, Ion W A, Mueller D, Hoffmann P, Jonell P, Baudisch P. Proprioceptive interaction. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. New York, USA, ACM, 2015, 939–948 DOI:10.1145/2702123.2702461

26. Scott J, Dearman D, Yatani K, Truong K N. Sensing foot gestures from the pocket. In: Proceedings of the 23nd annual ACM symposium on User interface software and technology. New York, USA, ACM, 2010, 199–208 DOI:10.1145/1866029.1866063

27. Yan Y, Yu C, Ma X, Yi X, Sun K, Shi Y. VirtualGrasp: Leveraging Experience of Interacting with Physical Objects to Facilitate Digital Object Retrieval. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Montreal QC, Canada, ACM, 2018, 1–13 DOI:10.1145/3173574.3173652

28. Yan Y, Yu C, Ma X, Huang S, Iqbal H, Shi Y. Eyes-Free Target Acquisition in Interaction Space around the Body for Virtual Reality. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Montreal QC, Canada, ACM, 2018, 1–13 DOI:10.1145/3173574.3173616

29. LoPresti E, Brienza D M, Angelo J, Gilbertson L, Sakai J. Neck range of motion and use of computer head controls. In: Proceedings of the Fourth International ACM Conference on Assistive Technologies. ACM, New York, USA, 2000, 121–128 DOI:10.1145/354324.354352

30. Jia P, Hu H H, Lu T, Yuan K. Head gesture recognition for hands-free control of an intelligent wheelchair. Industrial Robot: an International Journal, 2007, 34(1): 60–68 DOI:10.1108/01439910710718469

31. Craig D A, Nguyen H T. Wireless real-time head movement system using a personal digital assistant (PDA) for control of a power wheelchair. 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference. Shanghai, China, 2006, 6235–6238 DOI:10.1109/IEMBS.2005.1615921

32. Gorodnichy D O, Roth G. Nouse ‘use your nose as a mouse’ perceptual vision technology for hands-free games and interfaces. Image and Vision Computing, 2004, 22(12): 931–942 DOI:10.1016/j.imavis.2004.03.021

33. Varona J, Manresa-Yee C, Perales F J. Hands-free vision-based interface for computer accessibility. Journal of Network and Computer Applications, 2008, 31(4): 357–374 DOI:10.1016/j.jnca.2008.03.003

34. Crossan A, McGill M, Brewster S, Murray-Smith R. Head tilting for interaction in mobile contexts. In: Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services. Bonn, Germany, ACM, 2009, 1–10 DOI:10.1145/1613858.1613866

35. Piumsomboon T, Clark A, Billinghurst M, Cockburn A. User-Defined Gestures for Augmented Reality. In: Human-Computer Interaction–INTERACT 2013. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, 282–299 DOI:10.1007/978-3-642-40480-1_18

36. Edge D, Blackwell A F. Peripheral tangible interaction by analytic design. In: Proceedings of the 3rd International Conference on Tangible and Embedded Interaction. Cambridge, United Kingdom, ACM, 2009, 69–76 DOI:10.1145/1517664.1517687

email E-mail this page

Articles by authors

VRIH