Review of micro-expression spotting and recognition in video sequences
1. School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100083, China
2. National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
Abstract
Keywords: Facial expression ; Micro-expression spotting ; Micro-expression recognition ; Database ; Review
Content

Dataset | Dataset-Sub | Subjects | Samples | FACS | FPS | Classes | Resolution | Frame annotations | Type |
---|---|---|---|---|---|---|---|---|---|
Polikovsky[26] | 11 | 42 | No | 200 | 6 | 640 × 480 | - | Posed | |
USF-HD[27] | - | 100 | No | 29.7 | 4 | 1280 × 720 | - | Posed | |
YorkDDT[28] | 9 | 18 | No | 25 | 2 | 320 × 240 | - | Posed | |
SMIC-sub[29] | 6 | 77 | No | 100 | 3 | 640 × 480 | - | Spontaneous | |
SMIC[30] | HS | 16 | 164 | No | 100 | 3 | 640 × 480 | - | Spontaneous |
VIS | 8 | 71 | No | 25 | 3 | 640 × 480 | - | Spontaneous | |
NIR | 8 | 71 | No | 25 | 3 | 640 × 480 | - | Spontaneous | |
E-HS | 16 | 157 | No | 100 | 3 | 640 × 480 | Onset, offset | Spontaneous | |
E-VIS | 8 | 71 | No | 25 | 3 | 640 × 480 | - | Spontaneous | |
E-NIR | 8 | 71 | No | 25 | 3 | 640 × 480 | - | Spontaneous | |
CASME[31] | Section A | 7 | 96 | Yes | 60 | 7 | 1280 × 720 | - | Spontaneous |
Section B | 12 | 101 | Yes | 60 | 7 | 640 × 480 | - | Spontaneous | |
CASME II[32] | 35 | 247 | Yes | 200 | 5 | 640 × 480 | Onset, offset, apex | Spontaneous | |
CAS(ME)2[33] | 22 | 357 | No | 30 | 4 | 640 × 480 | Onset, offset, apex | Spontaneous | |
SAMM[34] | 32 | 159 | Yes | 200 | 7 | 2040 × 1088 | Onset, offset, apex | Spontaneous | |
SAMM Long Videos[35] | 32 | 502 | Yes | 200 | 2 | 2040 × 1088 | Onset, offset, apex | Spontaneous |
Dataset | Positive | Negative | Surprise | Total |
---|---|---|---|---|
HS | 51 | 70 | 43 | 164 |
VIS | 28 | 23 | 20 | 71 |
NIR | 28 | 23 | 20 | 71 |

Dataset | Positive | Negative | Surprise | Others | Total | |||||
---|---|---|---|---|---|---|---|---|---|---|
Happiness | Despise | Disgust | Fear | Repression | Sad | Tension | ||||
CAS(ME)2 | 8 | 21 | 9 | 19 | 57 | |||||
CASME | 9 | 1 | 44 | 2 | 38 | 6 | 69 | 20 | 189 | |
CASME II | 32 | - | 63 | 2 | 27 | 7 | - | 25 | 99 | 255 |
Dataset | Happiness | Surprise | Angry | Disgust | Sad | Sad | Despise | Others | Total |
---|---|---|---|---|---|---|---|---|---|
SAMM | 26 | 15 | 57 | 9 | 6 | 8 | 12 | 26 | 159 |
Work | Feature | Spotting method | Datasets |
---|---|---|---|
Polikovsky et al., 2009[26] | 3D-HOG | K means | Polikovsky |
Polikovsky et al., 2013[40] | 3D-HOG | K means | Polikovsky |
Moilanen et al., 2014[41] | LBP | Threshold technique | CASME、SMIC |
Davison et al., 2015[42] | HOG | Threshold technique | SAMM |
Patel et al., 2015[43] | Optical flow | Threshold technique | SMIC |
Xia et al., 2016[44] | Geometrical motion | Random walk model | CASME、SMIC |
Li et al., 2017[45] | HOOF, LBP | Threshold technique | CASME II、SMIC |
Wang et al., 2017[46] | MDMD | Threshold technique | CAS(ME)2 |
Davison et al., 2018[47] | 3DHOG, LBP, OF | Threshold technique | SAMM、CASME II |
Li et al., 2019[48] | LBP-χ2 | Threshold technique | SAMM、CASME II |
LBP-TOP Series | Accuracy | F1-Score | ||
---|---|---|---|---|
SMIC | CASME II | SMIC | CASME II | |
LBP-TOP[29] | 48.78 | - | - | - |
CLBP-TOP[54] | 78.2 | - | - | - |
STCLQP[55] | 64.02 | 58.39 | 0.6381 | 0.5836 |
LBP-SIP[56] | 44.51 | 46.56 | 0.4492 | 0.4480 |
LBP-MOP[57] | 50.61 | 44.13 | - | - |
STLBP-IP[58] | 57.93 | 59.51 | 0.5800 | 0.5700 |
TICS[59] | - | 61.47 | - | - |
DMDSP[60] | 58.00 | 49.00 | 0.6000 | 0.5100 |
STRBP[61] | 60.98 | 64.37 | - | - |
HWP-TOP[62] | 64.80 | - | - | |
LTOGP[64] | - | 66.00 | - | - |
Deep Learning | Unweighted F1-score (UF1) | Unweighted Average Recall (UAR) | ||||
---|---|---|---|---|---|---|
SMIC | CASME II | SAMM | SMIC | CASME II | SAMM | |
OFF-Apex[80] | 0.6817 | 0.8764 | 0.5409 | 0.6695 | 0.8681 | 0.5392 |
CapsuleNet[81] | 0.5820 | 0.7068 | 0.6209 | 0.5877 | 0.7018 | 0.5989 |
DINet[82] | 0.6645 | 0.8621 | 0.5868 | 0.6726 | 0.8560 | 0.5663 |
STSTNet[83] | 0.6801 | 0.8382 | 0.6588 | 0.7013 | 0.8686 | 0.6810 |
PB-DNN[84] | 0.7461 | 0.8293 | 0.7754 | 0.7530 | 0.8209 | 0.7152 |
Reference
Mehrabian A. Communication without words. Psychological Today, 1968, 2(6): 53–55
Haggard E A, Isaacs K S. Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. In: Methods of Research in Psychotherapy. Boston, MA, Springer US, 1966, 154–165 DOI:10.1007/978-1-4684-6045-2_14
Ekman P, Friesen W V. Nonverbal leakage and clues to deception. Psychiatry 1969, 32(1):88–106 DOI:10.1080/00332747.1969.11023575
Ekman P. Lie catching and microexpressions. The Philosophy of Deception, 2009, 1(2): 5
Frank M, Maccario C, Govindaraju V. Protecting airline passengers in the age of terrorism. ABC-CLIO, Santa Barbara, 2009
O'Sullivan M, Frank M G, Hurley C M, Tiwana J. Police lie detection accuracy: the effect of lie scenario. Law and Human Behavior, 2009, 33(6): 530
Matsumoto D, Hwang H S. Evidence for training the ability to read microexpressions of emotion. Motivation and Emotion, 2011, 35(2): 181–191
Turner J H. The evolution of emotions: the nonverbal basis of human social organization. Lawrence Erlbaum Associates, Mahwah, New Jersey, 1997
Frank M, Herbasz M, Sinuk K, Keller A, Nolan C. I see how you feel: training laypeople and professionals to recognize fleeting emotions. In: The Annual Meeting of the International Communication Association. Sheraton New York, New York City, 2009, 1–35
Ekman P. Telling lies: clues to deceit in the marketplace, politics, and marriage (revised edition). WW Norton & Company, 2009
Ekman P, Friesen W V. Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 1971, 17(2): 124–129 DOI:10.1037/h0030377
Yan W J, Wu Q, Liang J, Chen Y H, Fu X L. How fast are the leaked facial expressions: the duration of micro-expressions. Journal of Nonverbal Behavior, 2013, 37(4): 217–230 DOI:10.1007/s10919-013-0159-8
Porter S, ten Brinke L. Reading between the lies: identifying concealed and falsified emotions in universal facial expressions. Psychological Science, 2008, 19(5): 508–514 DOI:10.1111/j.1467-9280.2008.02116.x
Ekman P. Microexpression training tool (METT). University of California, San Francisco, 2002
Frank M G, Maccario C J, Govindaraju V. Behavior and security. Protecting airline passengers in the age of terrorism, 2009, 86–106
Zhao G Y, Pietikainen M. Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6): 915–928 DOI:10.1109/tpami.2007.1110
Liu M, Shan S, Wang R, Chen X. Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2014, 1749–1756
Jung H, Lee S, Yim J, Park S, Kim J. Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE international conference on computer vision. 2015, 2983–2991
Zeng N Y, Zhang H, Song B Y, Liu W B, Li Y R, Dobaie A M. Facial expression recognition via learning deep sparse autoencoders. Neurocomputing, 2018, 273: 643–649 DOI:10.1016/j.neucom.2017.08.043
Li S, Deng W H. Deep facial expression recognition: a survey. IEEE Transactions on Affective Computing, 2020: 1 DOI:10.1109/taffc.2020.2981446
Tornincasa S, Vezzetti E, Moos S, Violante M, Marcolin F, Dagnes N, Ulrich L, Tregnaghi G. 3D facial action units and expression recognition using a crisp logic. Computer-Aided Design and Applications, 2018, 16(2): 256–268 DOI:10.14733/cadaps.2019.256-268
Merghani W, Davison A, Yap M. Facial Micro-expressions Grand Challenge 2018: evaluating spatio-temporal features for classification of objective classes. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 2018, 662–666
Yap M H, See J, Hong X, Wang S J. Facial micro-expressions grand challenge 2018 summary. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 2018, 675–678
See J, Yap M H, Li J, Hong X, Wang S J. Megc 2019–the second facial micro-expressions grand challenge. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, 2019, 1–5
Li J, Wang S, Yap M H, See J, Hong X, Li X. MEGC2020-the third facial micro-expression grand challenge. In: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)(FG). 2020, 234–237
Polikovsky S, Kameda Y, Ohta Y. Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor. In: International Conference on Imaging for Crime Detection and Prevention. IET, 2009, 1–6
Shreve M, Godavarthy S, Goldgof D, Sarkar S. Macro-and micro-expression spotting in long videos using spatio-temporal strain. In: IEEE international conference and workshops on automatic face and gesture recognition. IEEE, 2011, 51–56
Warren G, Schertler E, Bull P. Detecting deception from emotional and unemotional cues. Journal of Nonverbal Behavior, 2009, 33(1): 59–69 DOI:10.1007/s10919-008-0057-7
Pfister T, Li X, Zhao G, Pietikäinen M. Recognising spontaneous facial micro-expressions. In: IEEE International Conference on Computer Vision. IEEE, 2011, 1449–1456
Li X, Pfister T, Huang X, Zhao G, Pietikäinen M. A spontaneous micro-expression database: Inducement, collection and baseline. In: IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. IEEE, 2013,1–6
Yan W J, Wu Q, Liu Y J, Wang S J, Fu X. CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces. In: IEEE International Conference And Workshops on Automatic Face and Gesture Recognition. IEEE, 2013, 1–7
Yan W J, Li X B, Wang S J, Zhao G Y, Liu Y J, Chen Y H, Fu X L. CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS One, 2014, 9(1): e86041 DOI:10.1371/journal.pone.0086041
Qu F B, Wang S J, Yan W J, Li H, Wu S H, Fu X L. CAS(ME): a database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Transactions on Affective Computing, 2018, 9(4): 424–436 DOI:10.1109/taffc.2017.2654440
Davison A K, Lansley C, Costen N, Tan K, Yap M H. SAMM: a spontaneous micro-facial movement dataset. IEEE Transactions on Affective Computing, 2018, 9(1): 116–129 DOI:10.1109/taffc.2016.2573832
Yap C H, Kendrick C, Yap M H. Samm long videos: a spontaneous facial micro-and macro-expressions datase. 2019
Hess U, Kleck R E. Differentiating emotion elicited and deliberate emotional facial expressions. European Journal of Social Psychology, 1990, 20(5): 369–385 DOI:10.1002/ejsp.2420200502
Friesen E, Ekman P. Facial action coding system: a technique for the measurement of facial movement. Palo Alto, 1978
Valstar M F, Pantic M. Fully automatic recognition of the temporal phases of facial actions. IEEE Transactions on Systems, Man, and Cybernetics. Part B, Cybernetics, 2012, 42(1): 28–43 DOI:10.1109/tsmcb.2011.2163710
Hong X, Tran T K, Zhao G. Micro-expression spotting: a benchmark. 2017
Polikovsky S, Kameda Y, Ohta Y. Facial micro-expression detection in hi-speed video based on facial action coding system (FACS). IEICE transactions on information and systems, 2013, 96(1): 81–92
Moilanen A, Zhao G, Pietikäinen M. Spotting rapid facial movements from videos using appearance-based feature difference analysis. In: 2014 22nd international conference on pattern recognition. IEEE, 2014,1722–1727
Davison A K, Yap M H, Lansley C. Micro-facial movement detection using individualised baselines and histogram-based descriptors. In: 2015 IEEE International Conference on Systems, Man, and Cybernetics. IEEE, 2015, 1864–1869
Patel D, Zhao G, Pietikäinen M. Spatiotemporal integration of optical flow vectors for micro-expression detection. In: International conference on advanced concepts for intelligent vision systems. Springer, 2015, 369–380
Xia Z, Feng X, Peng J, Peng X, Zhao G. Spontaneous micro-expression spotting via geometric deformation modeling. Computer Vision and Image Understanding. 2016, 147: 87–94
Li X, Hong X, Moilanen A, Huang X, Pfister T, Zhao G, Pietikäinen M. Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Transactions on Affective Computing, 2017, 9(4): 563–577
Wang S J, Wu S, Qian X, Li J, Fu X. A main directional maximal difference analysis for spotting facial movements from long-term videos. Neurocomputing, 2017, 230: 382–389
Davison A, Merghani W, Lansley C, Ng C C, Yap M H. Objective micro-facial movement detection using facs-based regions and baseline evaluation. In: IEEE International Conference on Automatic Face and Gesture Recognition (FG). IEEE, 2018, 642–649
Li J, Soladie C, Seguier R, Wang S J, Yap M H. Spotting micro-expressions on long videos sequences. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, 2019, 1–5
Tran T K, Hong X, Zhao G. Sliding window based micro-expression spotting: a benchmark. In: International Conference on Advanced Concepts for Intelligent Vision Systems. Springer, 2017, 542–553
Davison A, Merghani W, Lansley C, Ng C C, Yap M H. Objective micro-facial movement detection using facs-based regions and baseline evaluation. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 2018, 642–649
Li J, Soladie C, Seguier R. Ltp-ml: micro-expression detection by recognition of local temporal pattern of facial movements. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 2018, 634–641
Ojala T, Pietikäinen M, Harwood D. A comparative study of texture measures with classification based on featured distributions. Pattern recognition, 1996, 29(1): 51–59
Zhou Z, Zhao G, Pietikäinen M. Towards a practical lipreading system. In: CVPR 2011, IEEE, 2011, 137–144
Pfister T, Li X, Zhao G, Pietikäinen M. Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops). IEEE, 2011, 868–875
Huang X, Zhao G, Hong X, Zheng W, Pietikäinen M. Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing, 2016, 175: 564–578
Wang Y, See J, Phan R C W, Oh Y H. Lbp with six intersection points: Reducing redundant information in lbp-top for micro-expression recognition. In: Asian conference on computer vision. Springer, 2014, 525–537
Wang Y, See J, Phan R C W, Oh Y H. Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition. PloS One, 2015, 10(5): e0124674
Huang X, Wang S J, Zhao G, Piteikainen M. Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection. In: Proceedings of the IEEE international conference on computer vision workshops. 2015, 1–9
Wang S J, Yan W J, Li X, Zhao G, Fu X. Micro-expression recognition using dynamic textures on tensor independent color space. In: 2014 22nd international conference on pattern recognition. IEEE, 2014, 4678–4683
Le Ngo A C, Liong S T, See J, Phan R C W. Are subtle expressions too sparse to recognize? In: 2015 IEEE International Conference on Digital Signal Processing (DSP). IEEE, 2015, 1246–1250
Huang X, Zhao G. Spontaneous facial micro-expression analysis using spatiotemporal local radon-based binary pattern. In: 2017 International Conference on the Frontiers and Advances in Data Science (FADS). IEEE, 2017, 159–164
Ben X, Jia X, Yan R, Zhang X, Meng W. Learning effective binary descriptors for micro-expression recognition transferred by macro-information. Pattern Recognition Letters, 2018, 107: 50–58
Niu M, Li Y, Tao J, Wang S J. Micro-expression recognition based on local two-order gradient pattern. In: 2018 First Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia), IEEE, 2018, 1–6
Niu M, Tao J, Li Y, Huang J, Lian Z. Discriminative video representation with temporal order for micro-expression recognition. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, 2112–2116
Horn B K, Schunck B G. Determining optical flow. In: Techniques and Applications of Image Understanding. International Society for Optics and Photonics, 1981, 319–331
Xu F, Zhang J, Wang J Z. Microexpression identification and categorization using a facial dynamics map. IEEE Transactions on Affective Computing, 2017, 8(2): 254–267
Allaert B, Bilasco I M, Djeraba C. Consistent optical flow maps for full and micro facial expression recognition. In: International Conference on Computer Vision Theory and Applications (VISAPP). Spirnger, 2017
Liong S T, Phan R C W, See J, Oh Y H, Wong K. Optical strain based recognition of subtle emotions. In: 2014 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS). IEEE, 2014, 180–184
Liong S T, See J, Phan R C W, Le Ngo A C, Oh Y H, Wong K. Subtle expression recognition using optical strain weighted features. In: Asian Conference on Computer Vision. Springer, 2014, 644–657
Liong S T, See J, Wong K, Phan R C W. Less is more: micro-expression recognition from video using apex frame. Signal Processing: Image Communication, 2018, 62: 82–92
Zhang S, Feng B, Chen Z, Huang X. Micro-expression recognition by aggregating local spatio-temporal patterns. In: International Conference on Multimedia Modeling. Springer, 2017, 638–648
Happy S, Routray A. Recognizing subtle micro-facial expressions using fuzzy histogram of optical flow orientations and feature selection methods. Computational Intelligence for Pattern Recognition, 2018, 341–368
Liu Y J, Zhang J K, Yan W J, Wang S J, Zhao G, Fu X. A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Transactions on Affective Computing, 2015, 7(4): 299–310
LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86(11): 2278–2324
Krizhevsky A, Sutskever I, Hinton G E. Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems. 2012, 1097–1105
Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. 2014
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A. Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2015, 1–9
He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 770–778
Kim D H, Baddar W J, Ro Y M. Micro-expression recognition with expression-state constrained spatio-temporal feature representations. In: Proceedings of the 24th ACM International Conference on Multimedia. 2016, 382–386
Gan Y, Liong S T, Yau W C, Huang Y C, Tan L K. Off-apexnet on micro-expression recognition system. Signal Processing: Image Communication, 2019, 74:129–139
Van Quang N, Chun J, Tokuyama T. Capsulenet for micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, 2019, 1–7
Zhou L, Mao Q, Xue L. Dual-inception network for cross-database micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, 2019, 1–5
Liong S T, Gan Y, See J, Khor H Q, Huang Y C. Shallow triple stream three-dimensional cnn (ststnet) for micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, 2019, 1–5
Liu Y, Du H, Zheng L, Gedeon T. A neural micro-expression recognizer. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, 2019,1–4
Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, Marchand M, Lempitsky V. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 2016, 17(1): 2096–2030