Home About the Journal Latest Work Current Issue Archive Special Issues Editorial Board
<< Previous Next >>

2021, 3(1): 1-17

Published Date:2021-2-20 DOI: 10.1016/j.vrih.2020.10.003

Review of micro-expression spotting and recognition in video sequences

Abstract

Facial micro-expressions are short and imperceptible expressions that involuntarily reveal the true emotions that a person may be attempting to suppress, hide, disguise, or conceal. Such expressions can reflect a person's real emotions and have a wide range of application in public safety and clinical diagnosis. The analysis of facial micro-expressions in video sequences through computer vision is still relatively recent. In this research, a comprehensive review on the topic of spotting and recognition used in micro-expression analysis databases and methods, is conducted, and advanced technologies in this area are summarized. In addition, we discuss challenges that remain unresolved alongside future work to be completed in the field of micro-expression analysis.

Keyword

Facial expression ; Micro-expression spotting ; Micro-expression recognition ; Database ; Review

Cite this article

Hang PAN, Lun XIE, Zhiliang WANG, Bin LIU, Minghao YANG, Jianhua TAO. Review of micro-expression spotting and recognition in video sequences. Virtual Reality & Intelligent Hardware, 2021, 3(1): 1-17 DOI:10.1016/j.vrih.2020.10.003

References

1. Mehrabian A. Communication without words. Psychological Today, 1968, 2(6): 53–55

2. Haggard E A, Isaacs K S. Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. In: Methods of Research in Psychotherapy. Boston, MA, Springer US, 1966, 154–165 DOI:10.1007/978-1-4684-6045-2_14

3. Ekman P, Friesen W V. Nonverbal leakage and clues to deception. Psychiatry 1969, 32(1):88–106 DOI:10.1080/00332747.1969.11023575

4. Ekman P. Lie catching and microexpressions. The Philosophy of Deception, 2009, 1(2): 5

5. Frank M, Maccario C, Govindaraju V. Protecting airline passengers in the age of terrorism. ABC-CLIO, Santa Barbara, 2009

6. O'Sullivan M, Frank M G, Hurley C M, Tiwana J. Police lie detection accuracy: the effect of lie scenario. Law and Human Behavior, 2009, 33(6): 530

7. Matsumoto D, Hwang H S. Evidence for training the ability to read microexpressions of emotion. Motivation and Emotion, 2011, 35(2): 181–191

8. Turner J H. The evolution of emotions: the nonverbal basis of human social organization. Lawrence Erlbaum Associates, Mahwah, New Jersey, 1997

9. Frank M, Herbasz M, Sinuk K, Keller A, Nolan C. I see how you feel: training laypeople and professionals to recognize fleeting emotions. In: The Annual Meeting of the International Communication Association. Sheraton New York, New York City, 2009, 1–35

10. Ekman P. Telling lies: clues to deceit in the marketplace, politics, and marriage (revised edition). WW Norton & Company, 2009

11. Ekman P, Friesen W V. Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 1971, 17(2): 124–129 DOI:10.1037/h0030377

12. Yan W J, Wu Q, Liang J, Chen Y H, Fu X L. How fast are the leaked facial expressions: the duration of micro-expressions. Journal of Nonverbal Behavior, 2013, 37(4): 217–230 DOI:10.1007/s10919-013-0159-8

13. Porter S, ten Brinke L. Reading between the lies: identifying concealed and falsified emotions in universal facial expressions. Psychological Science, 2008, 19(5): 508–514 DOI:10.1111/j.1467-9280.2008.02116.x

14. Ekman P. Microexpression training tool (METT). University of California, San Francisco, 2002

15. Frank M G, Maccario C J, Govindaraju V. Behavior and security. Protecting airline passengers in the age of terrorism, 2009, 86–106

16. Zhao G Y, Pietikainen M. Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6): 915–928 DOI:10.1109/tpami.2007.1110

17. Liu M, Shan S, Wang R, Chen X. Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2014, 1749–1756

18. Jung H, Lee S, Yim J, Park S, Kim J. Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE international conference on computer vision. 2015, 2983–2991

19. Zeng N Y, Zhang H, Song B Y, Liu W B, Li Y R, Dobaie A M. Facial expression recognition via learning deep sparse autoencoders. Neurocomputing, 2018, 273: 643–649 DOI:10.1016/j.neucom.2017.08.043

20. Li S, Deng W H. Deep facial expression recognition: a survey. IEEE Transactions on Affective Computing, 2020: 1 DOI:10.1109/taffc.2020.2981446

21. Tornincasa S, Vezzetti E, Moos S, Violante M, Marcolin F, Dagnes N, Ulrich L, Tregnaghi G. 3D facial action units and expression recognition using a crisp logic. Computer-Aided Design and Applications, 2018, 16(2): 256–268 DOI:10.14733/cadaps.2019.256-268

22. Merghani W, Davison A, Yap M. Facial Micro-expressions Grand Challenge 2018: evaluating spatio-temporal features for classification of objective classes. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 2018, 662–666

23. Yap M H, See J, Hong X, Wang S J. Facial micro-expressions grand challenge 2018 summary. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 2018, 675–678

24. See J, Yap M H, Li J, Hong X, Wang S J. Megc 2019–the second facial micro-expressions grand challenge. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, 2019, 1–5

25. Li J, Wang S, Yap M H, See J, Hong X, Li X. MEGC2020-the third facial micro-expression grand challenge. In: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)(FG). 2020, 234–237

26. Polikovsky S, Kameda Y, Ohta Y. Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor. In: International Conference on Imaging for Crime Detection and Prevention. IET, 2009, 1–6

27. Shreve M, Godavarthy S, Goldgof D, Sarkar S. Macro-and micro-expression spotting in long videos using spatio-temporal strain. In: IEEE international conference and workshops on automatic face and gesture recognition. IEEE, 2011, 51–56

28. Warren G, Schertler E, Bull P. Detecting deception from emotional and unemotional cues. Journal of Nonverbal Behavior, 2009, 33(1): 59–69 DOI:10.1007/s10919-008-0057-7

29. Pfister T, Li X, Zhao G, Pietikäinen M. Recognising spontaneous facial micro-expressions. In: IEEE International Conference on Computer Vision. IEEE, 2011, 1449–1456

30. Li X, Pfister T, Huang X, Zhao G, Pietikäinen M. A spontaneous micro-expression database: Inducement, collection and baseline. In: IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. IEEE, 2013,1–6

31. Yan W J, Wu Q, Liu Y J, Wang S J, Fu X. CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces. In: IEEE International Conference And Workshops on Automatic Face and Gesture Recognition. IEEE, 2013, 1–7

32. Yan W J, Li X B, Wang S J, Zhao G Y, Liu Y J, Chen Y H, Fu X L. CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS One, 2014, 9(1): e86041 DOI:10.1371/journal.pone.0086041

33. Qu F B, Wang S J, Yan W J, Li H, Wu S H, Fu X L. CAS(ME): a database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Transactions on Affective Computing, 2018, 9(4): 424–436 DOI:10.1109/taffc.2017.2654440

34. Davison A K, Lansley C, Costen N, Tan K, Yap M H. SAMM: a spontaneous micro-facial movement dataset. IEEE Transactions on Affective Computing, 2018, 9(1): 116–129 DOI:10.1109/taffc.2016.2573832

35. Yap C H, Kendrick C, Yap M H. Samm long videos: a spontaneous facial micro-and macro-expressions datase. 2019

36. Hess U, Kleck R E. Differentiating emotion elicited and deliberate emotional facial expressions. European Journal of Social Psychology, 1990, 20(5): 369–385 DOI:10.1002/ejsp.2420200502

37. Friesen E, Ekman P. Facial action coding system: a technique for the measurement of facial movement. Palo Alto, 1978

38. Valstar M F, Pantic M. Fully automatic recognition of the temporal phases of facial actions. IEEE Transactions on Systems, Man, and Cybernetics. Part B, Cybernetics, 2012, 42(1): 28–43 DOI:10.1109/tsmcb.2011.2163710

39. Hong X, Tran T K, Zhao G. Micro-expression spotting: a benchmark. 2017

40. Polikovsky S, Kameda Y, Ohta Y. Facial micro-expression detection in hi-speed video based on facial action coding system (FACS). IEICE transactions on information and systems, 2013, 96(1): 81–92

41. Moilanen A, Zhao G, Pietikäinen M. Spotting rapid facial movements from videos using appearance-based feature difference analysis. In: 2014 22nd international conference on pattern recognition. IEEE, 2014,1722–1727

42. Davison A K, Yap M H, Lansley C. Micro-facial movement detection using individualised baselines and histogram-based descriptors. In: 2015 IEEE International Conference on Systems, Man, and Cybernetics. IEEE, 2015, 1864–1869

43. Patel D, Zhao G, Pietikäinen M. Spatiotemporal integration of optical flow vectors for micro-expression detection. In: International conference on advanced concepts for intelligent vision systems. Springer, 2015, 369–380

44. Xia Z, Feng X, Peng J, Peng X, Zhao G. Spontaneous micro-expression spotting via geometric deformation modeling. Computer Vision and Image Understanding. 2016, 147: 87–94

45. Li X, Hong X, Moilanen A, Huang X, Pfister T, Zhao G, Pietikäinen M. Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Transactions on Affective Computing, 2017, 9(4): 563–577

46. Wang S J, Wu S, Qian X, Li J, Fu X. A main directional maximal difference analysis for spotting facial movements from long-term videos. Neurocomputing, 2017, 230: 382–389

47. Davison A, Merghani W, Lansley C, Ng C C, Yap M H. Objective micro-facial movement detection using facs-based regions and baseline evaluation. In: IEEE International Conference on Automatic Face and Gesture Recognition (FG). IEEE, 2018, 642–649

48. Li J, Soladie C, Seguier R, Wang S J, Yap M H. Spotting micro-expressions on long videos sequences. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, 2019, 1–5

49. Tran T K, Hong X, Zhao G. Sliding window based micro-expression spotting: a benchmark. In: International Conference on Advanced Concepts for Intelligent Vision Systems. Springer, 2017, 542–553

50. Davison A, Merghani W, Lansley C, Ng C C, Yap M H. Objective micro-facial movement detection using facs-based regions and baseline evaluation. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 2018, 642–649

51. Li J, Soladie C, Seguier R. Ltp-ml: micro-expression detection by recognition of local temporal pattern of facial movements. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 2018, 634–641

52. Ojala T, Pietikäinen M, Harwood D. A comparative study of texture measures with classification based on featured distributions. Pattern recognition, 1996, 29(1): 51–59

53. Zhou Z, Zhao G, Pietikäinen M. Towards a practical lipreading system. In: CVPR 2011, IEEE, 2011, 137–144

54. Pfister T, Li X, Zhao G, Pietikäinen M. Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops). IEEE, 2011, 868–875

55. Huang X, Zhao G, Hong X, Zheng W, Pietikäinen M. Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing, 2016, 175: 564–578

56. Wang Y, See J, Phan R C W, Oh Y H. Lbp with six intersection points: Reducing redundant information in lbp-top for micro-expression recognition. In: Asian conference on computer vision. Springer, 2014, 525–537

57. Wang Y, See J, Phan R C W, Oh Y H. Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition. PloS One, 2015, 10(5): e0124674

58. Huang X, Wang S J, Zhao G, Piteikainen M. Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection. In: Proceedings of the IEEE international conference on computer vision workshops. 2015, 1–9

59. Wang S J, Yan W J, Li X, Zhao G, Fu X. Micro-expression recognition using dynamic textures on tensor independent color space. In: 2014 22nd international conference on pattern recognition. IEEE, 2014, 4678–4683

60. Le Ngo A C, Liong S T, See J, Phan R C W. Are subtle expressions too sparse to recognize? In: 2015 IEEE International Conference on Digital Signal Processing (DSP). IEEE, 2015, 1246–1250

61. Huang X, Zhao G. Spontaneous facial micro-expression analysis using spatiotemporal local radon-based binary pattern. In: 2017 International Conference on the Frontiers and Advances in Data Science (FADS). IEEE, 2017, 159–164

62. Ben X, Jia X, Yan R, Zhang X, Meng W. Learning effective binary descriptors for micro-expression recognition transferred by macro-information. Pattern Recognition Letters, 2018, 107: 50–58

63. Niu M, Li Y, Tao J, Wang S J. Micro-expression recognition based on local two-order gradient pattern. In: 2018 First Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia), IEEE, 2018, 1–6

64. Niu M, Tao J, Li Y, Huang J, Lian Z. Discriminative video representation with temporal order for micro-expression recognition. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, 2112–2116

65. Horn B K, Schunck B G. Determining optical flow. In: Techniques and Applications of Image Understanding. International Society for Optics and Photonics, 1981, 319–331

66. Xu F, Zhang J, Wang J Z. Microexpression identification and categorization using a facial dynamics map. IEEE Transactions on Affective Computing, 2017, 8(2): 254–267

67. Allaert B, Bilasco I M, Djeraba C. Consistent optical flow maps for full and micro facial expression recognition. In: International Conference on Computer Vision Theory and Applications (VISAPP). Spirnger, 2017

68. Liong S T, Phan R C W, See J, Oh Y H, Wong K. Optical strain based recognition of subtle emotions. In: 2014 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS). IEEE, 2014, 180–184

69. Liong S T, See J, Phan R C W, Le Ngo A C, Oh Y H, Wong K. Subtle expression recognition using optical strain weighted features. In: Asian Conference on Computer Vision. Springer, 2014, 644–657

70. Liong S T, See J, Wong K, Phan R C W. Less is more: micro-expression recognition from video using apex frame. Signal Processing: Image Communication, 2018, 62: 82–92

71. Zhang S, Feng B, Chen Z, Huang X. Micro-expression recognition by aggregating local spatio-temporal patterns. In: International Conference on Multimedia Modeling. Springer, 2017, 638–648

72. Happy S, Routray A. Recognizing subtle micro-facial expressions using fuzzy histogram of optical flow orientations and feature selection methods. Computational Intelligence for Pattern Recognition, 2018, 341–368

73. Liu Y J, Zhang J K, Yan W J, Wang S J, Zhao G, Fu X. A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Transactions on Affective Computing, 2015, 7(4): 299–310

74. LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86(11): 2278–2324

75. Krizhevsky A, Sutskever I, Hinton G E. Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems. 2012, 1097–1105

76. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. 2014

77. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A. Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2015, 1–9

78. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 770–778

79. Kim D H, Baddar W J, Ro Y M. Micro-expression recognition with expression-state constrained spatio-temporal feature representations. In: Proceedings of the 24th ACM International Conference on Multimedia. 2016, 382–386

80. Gan Y, Liong S T, Yau W C, Huang Y C, Tan L K. Off-apexnet on micro-expression recognition system. Signal Processing: Image Communication, 2019, 74:129–139

81. Van Quang N, Chun J, Tokuyama T. Capsulenet for micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, 2019, 1–7

82. Zhou L, Mao Q, Xue L. Dual-inception network for cross-database micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, 2019, 1–5

83. Liong S T, Gan Y, See J, Khor H Q, Huang Y C. Shallow triple stream three-dimensional cnn (ststnet) for micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, 2019, 1–5

84. Liu Y, Du H, Zheng L, Gedeon T. A neural micro-expression recognizer. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, 2019,1–4

85. Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, Marchand M, Lempitsky V. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 2016, 17(1): 2096–2030