2021, 3(1): 33-42

Published Date：2021-2-20 DOI: 10.1016/j.vrih.2020.10.002

NAS-HR: Neural architecture search for heart rate estima-tion from face videos

Hao LU ¹ , Hu HAN ^2,³

1. School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China

2. Key Laboratory of Intelligent Information Processing, Chinese Academy of Sciences(CAS ), Institute of Computing Technology, Beijing 100080, China

3. Peng Cheng Laboratory, Shenzhen 518000, China

Abstract

Background

In anticipation of its great potential application to natural human-computer interaction and health monitoring, heart-rate (HR) estimation based on remote photoplethysmography has recently attracted increasing research attention. Whereas the recent deep-learning-based HR estimation methods have achieved promising performance, their computational costs remain high, particularly in mobile-computing scenarios.

Methods

We propose a neural architecture search approach for HR estimation to automatically search a lightweight network that can achieve even higher accuracy than a complex network while reducing the computational cost. First, we define the regions of interests based on face landmarks and then extract the raw temporal pulse signals from the R, G, and B channels in each ROI. Then, pulse-related signals are extracted using a plane-orthogonal-to-skin algorithm, which are combined with the R and G channel signals to create a spatial-temporal map. Finally, a differentiable architecture search approach is used for the network-structure search.

Results

Compared with the state-of-the-art methods on the public-domain VIPL-HR and PURE databases, our method achieves better HR estimation performance in terms of several evaluation metrics while requiring a much lower computational cost¹.

Keyword

Heart-rate estimation ; Plane-orthogonal-to-skin STMap ; Differentiable architecture search

Cite this article

Hao LU, Hu HAN. NAS-HR: Neural architecture search for heart rate estima-tion from face videos. Virtual Reality & Intelligent Hardware, 2021, 3(1): 33-42 DOI:10.1016/j.vrih.2020.10.002

References

1. Chen X, Cheng J, Song R C, Liu Y, Ward R, Wang Z J. Video-based heart rate measurement: recent advances and future prospects. IEEE Transactions on Instrumentation and Measurement, 2019, 68(10): 3600–3615 DOI:10.1109/tim.2018.2879706

2. Wang C, Pun T, Chanel G. A comparative survey of methods for remote heart rate detection from frontal face videos. Frontiers in Bioengineering and Biotechnology, 2018, 6: 33

3. Li X, Han H, Lu H, Niu X, Yu Z, Dantcheva A, Zhao G, Shan S. The 1st challenge on remote physiological signal sensing (RePSS). In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. IEEE, 2020

4. Li X, Chen J, Zhao G, Pietikainen M. Remote heart rate measurement from face videos under realistic situations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2014

5. Qiu Y, Liu Y, Arteaga-Falconi J, Dong H W, Saddik A E. EVM-CNN: real-time contactless heart rate estimation from facial video. IEEE Transactions on Multimedia, 2019, 21(7): 1778–1787

6. Kopeliovich M. On indirect assessment of heart rate in video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. IEEE, 2020

7. Mironenko Y. Remote photoplethysmography: rarely considered factors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. IEEE, 2020

8. Poh M Z, McDuff D J, Picard R W. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Optics Express, 2010, 18(10): 10762–10774

9. Balakrishnan G, Durand F, Guttag J. Detecting pulse from head motions in video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2013

10. de Haan G, Jeanne V. Robust pulse rate from chrominance-based rPPG. IEEE Transactions on Biomedical Engineering, 2013, 60(10): 2878–2886

11. de Haan G, van Leest A. Improved motion robustness of remote-PPG by using the blood volume pulse signature. Physiological Measurement, 2014, 35(9): 1913–1926

12. Wang W J, Stuijk S, de Haan G. A novel algorithm for remote photoplethysmography: spatial subspace rotation. IEEE Transactions on Biomedical Engineering, 2016, 63(9): 1974–1984

13. Wang W J, den Brinker A C, Stuijk S, de Haan G. Algorithmic principles of remote PPG. IEEE Transactions on Biomedical Engineering, 2017, 64(7): 1479–1491

14. Chen W X, McDuff D. DeepPhys: video-based physiological measurement using convolutional attention networks. In: Computer Vision – ECCV 2018. Springer International Publishing, 2018, 356–373

15. Wang Z K, Kao Y, Hsu C T. Vision-based heart rate estimation via a two-stream cnn. In: IEEE International Conference on Image Processing. IEEE, 2019

16. Liu H X, Simonyan K, Yang Y M. DARTS: differentiable architecture search. 2018

17. Yu Z, Li X, Zhao G. Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks. In: British Machine Vision Conference. 2019

18. Hsu G S, Ambikapathi A M, Chen M S. Deep learning with time-frequency representation for pulse estimation from facial videos. In: IEEE International Joint Conference on Biometrics. IEEE, 2017

19. Niu X S, Shan S G, Han H, Chen X L. RhythmNet: end-to-end heart rate estimation from face via spatial-temporal representation. IEEE Transactions on Image Processing, 2020, 29: 2409–2423

20. Niu X, Zhao X, Han H, Das A, Chen X. Robust remote heart rate estimation from face utilizing spatial-temporal attention. In: IEEE International Conference on Automatic Face & Gesture Recognition. IEEE, 2019

21. Baker B, Gupta O, Naik N, Raskar R. Designing neural network architectures using reinforcement learning. 2016

22. Zoph B, Le Q V. Neural architecture search with reinforcement learning. 2016

23. Real E. Large-scale evolution of image classifiers. In: Proceedings of the 34th International Conference on Machine Learning. JMLR, 2017

24. Chen X, Xie L X, Wu J, Tian Q. Progressive differentiable architecture search: bridging the depth gap between search and evaluation. 2019

25. Xu Y, Xie L, Zhang X, Chen X, Qi G, Tian Q, Xiong H. PC-DARTS: partial channel connections for memory-efficient architecture search. In: International Conference on Learning Representations. New Orleans, USA, 2019

26. Verkruysse W, Svaasand L O, Stuart Nelson J. Remote plethysmographic imaging using ambient light. Optics Express, 2008, 16(26): 21434–21445

27. Wang W J, den Brinker A C, de Haan G. Single-element remote-PPG. IEEE Transactions on Biomedical Engineering, 2019, 66(7): 2032–2043

28. Niu X S, Han H, Shan S G, Chen X L. VIPL-HR: A multi-modal database for pulse estimation from less-constrained face video. In: Computer Vision–ACCV 2018. Springer International Publishing, 2019, 562–576

29. Špetlík R, Franc V, Matas J. Visual heart rate estimation with convolutional neural network. In: Proceedings of British Machine Vision Conference. 2018

30. Stricker R, Müller S, Gross H M. Non-contact video-based pulse rate measurement on a mobile service robot. In: The 23rd IEEE International Symposium on Robot and Human Interactive Communication. 2014

31. Carreira J, Zisserman A. Quo vadis, action recognition? a new model and the kinetics dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2017

32. Niu X, Han H, Shan S, Chen X. Synrhythm: learning a deep heart rate estimator from general to specifific. In: International Conference on Pattern Recognition. 2018

33. Tulyakov S, Alameda-Pineda X, Ricci E, Yin L, Cohn J F, Sebe N. Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2016

34. Yu Z T, Li X B, Niu X S, Shi J G, Zhao G Y. AutoHR: a strong end-to-end baseline for remote heart rate measurement with neural searching. IEEE Signal Processing Letters, 2020, 27: 1245–1249

35. Ma N N, Zhang X Y, Zheng H T, Sun J. ShuffleNet V2: practical guidelines for efficient CNN architecture design. In: Computer Vision – ECCV 2018. Springer International Publishing, 2018, 122–138

36. Han H, Shan S G, Chen X L, Lao S H, Gao W. Separability oriented preprocessing for illumination-insensitive face recognition. In: Computer Vision – ECCV 2012. Berlin, Heidelberg, Springer Berlin Heidelberg, 2012, 307–320

37. Niu X, Han H, Yang S, Huang Y, Shan S. Local relationship learning with person-specific shape regularization for facial action unit detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2019

Full Text

20 PDF

471 HTML

EXPORT

EndNote ProCite Reference Manager BibTex RefWorks

SHARE ON

ARTICLES BY AUTHORS

VRIH

BAIDU SCHOLAR

WANFANG DATA

CNKI