Chinese
Adv Search
Home | Accepted | Article In Press | Current Issue | Archive | Special Issues | Collections | Featured Articles | Statistics

2020, 2(3): 276-289 Published Date:2020-6-20

DOI: 10.1016/j.vrih.2020.05.001

Neural hand reconstruction using an RGB image

Full Text: PDF (14) HTML (168)

Export: EndNote | Reference Manager | ProCite | BibTex | RefWorks

Abstract:

Background
This study presents a neural hand reconstruction method for monocular 3D hand pose and shape estimation.
Methods
Alternate to directly representing hand with 3D data, a novel UV position map is used to represent a hand pose and shape with 2D data that maps 3D hand surface points to 2D image space. Furthermore, an encoder-decoder neural network is proposed to infer such UV position map from a single image. To train this network with inadequate ground truth training pairs, we propose a novel MANOReg module that employs MANO model as a prior shape to constrain high-dimensional space of the UV position map.
Results
The quantitative and qualitative experiments demonstrate the effectiveness of our UV position map representation and MANOReg module.
Keywords: Hand reconstruction ; Convolution neural network ; Single image ; Motion capture

Cite this article:

Mengcheng LI, Liang AN, Tao YU, Yangang WANG, Feng CHEN, Yebin LIU. Neural hand reconstruction using an RGB image. Virtual Reality & Intelligent Hardware, 2020, 2(3): 276-289 DOI:10.1016/j.vrih.2020.05.001

1. Cheng Y W, Li G F, Li J H, Sun Y, Jiang G Z, Zeng F, Zhao H Y, Chen D S. Visualization of activated muscle area based on sEMG. Journal of Intelligent & Fuzzy Systems, 2020, 38(3): 2623–2634 DOI:10.3233/jifs-179549

2. Qi J X, Jiang G Z, Li G F, Sun Y, Tao B. Intelligent human-computer interaction based on surface EMG gesture recognition. IEEE Access, 2019, 7: 61378–61387 DOI:10.1109/access.2019.2914728

3. Li J, Wang J X, Ju Z J. A novel hand gesture recognition based on high-level features. International Journal of Humanoid Robotics, 2017, 15(2): 1750022 DOI:10.1142/s0219843617500220

4. Park G, Argyros A, Lee J, Woo W. 3D hand tracking in the presence of excessive motion blur. IEEE Transactions on Visualization and Computer Graphics, 2020, 26(5): 1891–1901 DOI:10.1109/tvcg.2020.2973057

5. Tian J R, Cheng W T, Sun Y, Li G F, Jiang D, Jiang G Z, Tao B, Zhao H Y, Chen D S. Gesture recognition based on multilevel multimodal feature fusion. Journal of Intelligent & Fuzzy Systems, 2020, 38(3): 2539–2550 DOI:10.3233/jifs-179541

6. Oikonomidis I, Kyriazis N, Argyros A. Efficient model-based 3D tracking of hand articulations using Kinect. In: Procedings of the British Machine Vision Conference 2011. Dundee, British Machine Vision Association, 2011, 1(2): 3 DOI:10.5244/c.25.101

7. Sridhar S, Oulasvirta A, Theobalt C. Interactive markerless articulated hand motion tracking using RGB and depth data. 2013 IEEE International Conference on Computer Vision, 2013, 2456–2463 DOI:10.1109/iccv.2013.305

8. He K M, Zhang X Y, Ren S Q, Sun J. Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA, IEEE, 2016, 770–778 DOI:10.1109/cvpr.2016.90

9. Romero J, Tzionas D, Black M J. Embodied hands: modeling and capturing hands and bodies together. ACM Transactions on Graphics, 2017, 36(6): 245 DOI:10.1145/3130800.3130883

10. Boukhayma A, de Bem R, Torr P H S. 3D hand shape and pose from images in the wild. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA, USA, IEEE, 2019, 10835–10844 DOI:10.1109/cvpr.2019.01110

11. Cai Y J, Ge L H, Cai J F, Yuan J S. Weakly-supervised 3D hand pose estimation from monocular RGB images. Computer Vision-ECCV 2018, 666‒682 DOI:10.1007/978-3-030-01231-1_41

12. Ge L H, Ren Z, Li Y C, Xue Z H, Wang Y Y, Cai J F, Yuan J S. 3D hand shape and pose estimation from a single RGB image. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA, USA, IEEE, 2019, 10833‒10842 DOI:10.1109/cvpr.2019.01109

13. Panteleris P, Oikonomidis I, Argyros A. Using a single RGB frame for real time 3D hand pose estimation in the wild. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). Lake Tahoe, NV, USA, IEEE, 2018, 436–445 DOI:10.1109/wacv.2018.00054

14. Feng Y, Wu F, Shao X H, Wang Y F, Zhou X. Joint 3D face reconstruction and dense alignment with position map regression network. In: Computer Vision–ECCV 2018. Cham: Springer International Publishing, 2018, 557–574 DOI:10.1007/978-3-030-01264-9_33

15. Wan C D, Probst T, van Gool L, Yao A. Self-supervised 3D hand pose estimation through training by fitting. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA, USA, IEEE, 2019, 10845–10854 DOI:10.1109/cvpr.2019.01111

16. Khamis S, Taylor J, Shotton J, Keskin C, Izadi S, Fitzgibbon A. Learning an efficient model of hand shape variation from depth images. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA, IEEE, 2015, 2540–2548 DOI:10.1109/cvpr.2015.7298869

17. Taylor J, Stebbing R, Ramakrishna V, Keskin C, Shotton J, Izadi S, Hertzmann A, Fitzgibbon A. User-specific hand modeling from monocular depth sequences. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH, USA, IEEE, 2014, 644–651 DOI:10.1109/cvpr.2014.88

18. Mueller F, Bernard F, Sotnychenko O, Mehta D, Sridhar S, Casas D, Theobalt C. GANerated hands for real-time 3D hand tracking from monocular RGB. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA, IEEE, 2018, 49–59 DOI:10.1109/cvpr.2018.00013

19. Kulon D, Wang H, Güler R A, Bronstein M, Zafeiriou S. Single image 3D hand reconstruction with mesh convolutions. 2019

20. Rother C, Kolmogorov V, Blake A. “GrabCut”: interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics (TOG), 2004, 23(3): 309–314 DOI:10.1145/1015706.1015720

21. Kato H, Ushiku Y, Harada T. Neural 3D mesh renderer. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA, IEEE, 2018, 3907–3916 DOI:10.1109/cvpr.2018.00411

22. Zhang J, Jiao J, Chen M, Qu L, Xu X, Yang Q. A hand pose tracking benchmark from stereo matching. In: 2017 IEEE International Conference on Image Processing (ICIP). 2017, IEEE, 982–986

23. Zimmermann C, Brox T. Learning to estimate 3D hand pose from single RGB images. In: 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy, IEEE, 2017, 4913–4921 DOI:10.1109/iccv.2017.525

24. Simon T, Joo H, Matthews I, Sheikh Y. Hand keypoint detection in single images using multiview bootstrapping. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA, IEEE, 2017, 4645–4653 DOI:10.1109/cvpr.2017.494

25. Zimmermann C, Ceylan D, Yang J M, Russell B, Argus M J, Brox T. FreiHAND: a dataset for markerless capture of hand pose and shape from single RGB images. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South), IEEE, 2019, 813–822 DOI:10.1109/iccv.2019.00090

26. Iqbal U, Molchanov P, Breuel T, Gall J, Kautz J. Hand pose estimation via latent 2.5D heatmap regression. In: Computer Vision–ECCV 2018. Cham: Springer International Publishing, 2018, 125–143 DOI:10.1007/978-3-030-01252-6_8

27. Spurr A, Song J, Park S, Hilliges O. Cross-modal deep variational hand pose estimation. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA, IEEE, 2018, 89–98 DOI:10.1109/cvpr.2018.00017

28. Zhang J, Jiao J, Chen M, Qu L, Xu X, Yang Q. 3D hand pose tracking and estimation using stereo matching. 2016

email E-mail this page

Articles by authors

VRIH