CN104933416A - Micro expression sequence feature extracting method based on optical flow field - Google Patents
Micro expression sequence feature extracting method based on optical flow field Download PDFInfo
- Publication number
- CN104933416A CN104933416A CN201510360969.4A CN201510360969A CN104933416A CN 104933416 A CN104933416 A CN 104933416A CN 201510360969 A CN201510360969 A CN 201510360969A CN 104933416 A CN104933416 A CN 104933416A
- Authority
- CN
- China
- Prior art keywords
- optical flow
- micro
- flow field
- micro expression
- expression
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 230000014509 gene expression Effects 0.000 title claims abstract description 40
- 230000003287 optical effect Effects 0.000 title claims abstract description 34
- 230000033001 locomotion Effects 0.000 claims abstract description 36
- 239000013598 vector Substances 0.000 claims abstract description 27
- 238000000605 extraction Methods 0.000 claims description 7
- 239000011159 matrix material Substances 0.000 claims description 4
- 238000012935 Averaging Methods 0.000 claims description 3
- 230000008921 facial expression Effects 0.000 claims description 3
- 241000764238 Isis Species 0.000 claims description 2
- 238000006073 displacement reaction Methods 0.000 claims description 2
- 238000013519 translation Methods 0.000 abstract description 7
- 238000011161 development Methods 0.000 abstract description 3
- 230000004438 eyesight Effects 0.000 abstract description 3
- 238000005516 engineering process Methods 0.000 abstract description 2
- 238000002474 experimental method Methods 0.000 description 9
- OUXCBPLFCPMLQZ-WOPPDYDQSA-N 4-amino-1-[(2r,3s,4s,5r)-4-hydroxy-5-(hydroxymethyl)-3-methyloxolan-2-yl]-5-iodopyrimidin-2-one Chemical compound C[C@H]1[C@H](O)[C@@H](CO)O[C@H]1N1C(=O)N=C(N)C(I)=C1 OUXCBPLFCPMLQZ-WOPPDYDQSA-N 0.000 description 7
- 206010028813 Nausea Diseases 0.000 description 5
- 238000001514 detection method Methods 0.000 description 5
- 230000008693 nausea Effects 0.000 description 5
- 238000012706 support-vector machine Methods 0.000 description 5
- 238000012937 correction Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 210000003205 muscle Anatomy 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 230000008451 emotion Effects 0.000 description 1
- 238000010195 expression analysis Methods 0.000 description 1
- 230000001939 inductive effect Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000006996 mental state Effects 0.000 description 1
- 230000036651 mood Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention belongs to the technical field of computer visions, in particular a micro expression sequence feature extracting method based on an optical flow field. The method comprises the following steps: firstly extracting a dense optical flow field between adjacent frames in the premise that the frame number of the micro expression is fixed; eliminating the influence of the human face translation to the micro expression recognition through fine alignment; and then segmenting the aligned optical flow fields into a series of space-time sub-blocks, extracting a main direction in each space-time sub-block to represent the motion pattern of majority points in the sub-block; quantizing and splicing main directions in all sub-blocks, and expressing in a vector form, namely obtaining the designed micro expression sequence feature. The novel feature based on the motion description provided by the invention can be used for the micro expression recognition. Through the adoption of the method disclosed by the invention, the comprehensive indexes such as accuracy, precision and recalling rate are superior to other existing methods, and the further development of the micro expression recognition technology is promoted; and meanwhile, the method is capable of depicting the dynamic pattern of the micro expression and providing deeper understanding for the analysis of the micro expression.
Description
Technical Field
The invention belongs to the technical field of computer vision, and particularly relates to a micro-expression sequence feature extraction method based on an optical flow field.
Background
At present, the micro expression recognition has a plurality of difficulties, and a practical method and a theoretical framework are not formed at present. The difficulty is mainly expressed in feature extraction. The features currently used are often generic video feature expressions, which are not optimized for the application of micro expressions, nor do they provide a deep understanding of micro expressions.
Micro-expression was originally discovered in 1969 and was discovered by psychologists by observing video recordings of conversations with depression patients [1 ]. Patients in video often show a normal smile, however, an abnormally painful expression of several frames can be found. Psychologists name this as micro-expression.
Unlike conventional expressions, micro-expression is a micro-expression that a person cannot subjectively control. Therefore, observing the micro-expressions to determine the real mental state has potential and important application value in the fields of public security inquiries, psychological disease diagnosis and treatment, business negotiation and the like, and has received considerable attention at present.
However, the recognition of micro-expressions is not easy, and the main difficulties are that 1) the duration is short, and 2) the motion amplitude is small. Even the trained personnel have low recognition accuracy. Therefore, the automatic identification algorithm based on computer vision can improve the identification stability, greatly save labor and has strong application value. The technical field related to the method mainly comprises: face detection, face key point positioning, face alignment, image preprocessing, feature extraction, machine learning and the like.
Although the micro expression recognition development is not perfect, it is still studied by a large number of scholars, and representative documents are listed below.
There are currently two major groups of studies that continue to delve into the identification of microexpressions internationally. Starting from space-time textures, the university of Oulu in Finland tries to apply general video features to micro-expressions and extract effective expressions to identify the micro-expressions. Local Binary patterns (Local Binary patterns) are extracted on Three Planes X-Y, X-T, Y-T, collectively used for micro-representation [2], as characterized by the tri-Orthogonal plane Local Binary patterns (LBP-TOP) used by Pfister. And for each pixel point, the local binary pattern uses a binary number to encode the value size relation between the pixel point and surrounding pixels. The binary-coded distribution histogram is then counted as a feature expression. However, the micro-expression analysis has high requirements on fine alignment of the human face, and the method cannot well deal with the problem.
The Susbania researchers at the psychological institute of Chinese academy of sciences starts from the machine learning theory, each micro-expression image sequence is regarded as a third-order Tensor, and then a group of Subspace mappings is learned through Discriminant Tensor Subspace Analysis (DTSA), so that the distances among the tensors in the same category are as small as possible, and the distances among the tensors in different categories are as large as possible. The mapped microexpression tensor is then identified by an Extreme Learning Machine (Extreme Learning Machine) [5], which is essentially a Machine Learning algorithm and does not provide an in-depth understanding of the microexpression at the feature expression level.
Cited documents:
[1] Ekman P, Friesen W V. Nonverbal leakage and clues to deception. Psychiatry, vol.32, no.1, pp.88-106, 1969.
[2] T. Pfister, X. Li, G. Zhao, and M. Pietikainen. Recognising spontaneous facial micro-expressions. CVPR, 2011.
[3] M. Shreve, S. Godavarthy, V. Manohar, D. Goldgof, and S. Sarkar. Towards macro- and micro-expression spotting in video using strain patterns. IEEE Workshop on Applications of Computer Vision, 2009.
[4] M. Shreve, S. Godavarthy, D. Goldgof, and S. Sarkar. Macro-and micro-expression spotting in long videos using spatio-temporal strain, AFGR, 2011.
[5] S.-J.Wang, H.-L.Chen, W.-J.Yan, Y.-H.Chen, and X.Fu, Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine, Neural Processing Letters, vol.39, no.1, pp. 25–43, 2014.
[6] X.Li, T.Pfister, X. Huang, G. Zhao, and M. Pietikainen. A spontaneous micro-expression database: Inducement, collection and baseline, AFGR, 2013.
[7] W.-J. Yan, Q. Wu, Y.-J. Liu, S.-J. Wang, and X. Fu, CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces, AFGR, 2013
[8] W.-J.Yan, X.Li, S.-J.Wang, G.Zhao,Y.-J.Liu,Y.-H.Chen, and X.Fu, CASME II: an improved spontaneous micro-expression database and the baseline evaluation, PLoS ONE, vol.9, no.1, p.e86041, 2014.
[9] Wu Q, Shen X, Fu X. The machine knows what you are hiding: an automatic micro-expression recognition system. Affective Computing and Intelligent Interaction. Springer Berlin Heidelberg, pp.152-162, 2011.。
disclosure of Invention
The invention aims to provide an effective method for extracting the characteristics of a micro expression sequence.
Firstly, extracting a dense optical flow field between adjacent frames on the premise of a certain micro expression frame number; fine alignment is carried out through a simple method on the basis of a dense optical flow field, and the influence of face translation on micro expression recognition is eliminated; then, dividing the aligned optical flow field into a series of space-time blocks, and extracting a main direction from each space-time block to represent the motion mode of most points in the block; and quantizing and splicing the main directions in all the blocks, and expressing the main directions into a vector form to obtain the designed micro expression sequence characteristics. FIG. 1 is a flow chart of the present invention.
The invention provides a micro-expression sequence feature extraction method, which comprises the following specific steps:
1. giving a segment of facial expression sequenceAligning video to specified frame number by interpolationTo obtain. Wherein the interpolation method may be a linear interpolation method, or a manifold interpolation method described by Pfister [2]]。
2. In the micro-expression sequence with determined length, a Horn-Schunck method is used for estimating a dense optical flow field. Wherein,is thatAndthe formula expression of the optical flow field is as follows:
，
to representIn the first placeGo to the firstThe pixel values of the columns are selected,andare respectivelyAndin the first placeGo to the firstElements of a column;referred to as the motion vector for that location. In practical problems, the above formula is not strictly true, and only approximate, so that certain errors exist. Therefore, the present inventionAn iterative method for estimating the principal direction will be described in step 4.
3. And eliminating the whole face displacement by using a refined alignment algorithm. With a horizontal componentFor example, for eachCalculating a histogram。Equal to the median of the horizontal components of the optical flow fieldThe number of the cells. Order to，
Namely, it isIs the inverse of the level component value that occurs the most frequently. Order toAll values in (A) plusSo as to obtain a horizontal optical flow field after accurate alignment,
，
in the formulaIs andhave the same dimensions and the elements are allA matrix of (a);
the refined alignment of the vertical component V is similar:
，。
4. to achieve compact representation, the aligned optical flow field is divided into time-space blocks, each having a size ofIn each space-time block, a principal direction is sought describing the space-time block. The algorithm flow is as follows:
(a) initializing the principal direction estimate P as a two-dimensional unit vector: p = (1, 0);
(b) each plane coordinate in the block，Finding a time coordinateSo that the motion vector at the positionAndmaximum inner product of (d):
；
(c) averaging the found motion vectors, normalizing the motion vectors, and taking the normalized motion vectors as an updated value of P:
(d) repeating steps (a) - (c) until P converges or exceeds a maximum number of steps limit.
5. By the above steps, a main direction is sought in each space-time partition, each direction is quantized to several intervals, and the main direction is represented by the number of the intervals, for example, fig. 2 describes a strategy for quantizing the main direction to 10 intervals. And pulling and connecting the main directions in all the blocks to obtain the description characteristics of the whole sequence.
The features resulting from the above steps may be used to describe a micro-expression. And (3) learning the micro-expressions with the labels in the data set by using a supervised learning method to obtain a trained classifier (such as a Support Vector Machine). By extracting the above features from the unlabeled micro-expression sequences, the classifier can be used to predict the tags.
The key point of the invention is step 3 and step 4, which are also main contributions of the invention: refining an alignment method; a fast iterative-based principal direction estimation method. The following are detailed separately:
fine alignment method
In step 3, it is necessary to eliminate the influence of the translational motion of the entire face of the subject on the feature extraction. The optical flow field extracted by the invention contains the integral translation movement of the faceThe motion also includes the local motion of the micro expression. Since the micro-expression involves only a local part of the face, the overall optical flow field should have a value of 0 at most locations. Therefore, the invention divides the integral translation movement into horizontal translation and vertical translation, and respectively searches a correction quantity for the horizontal component and the vertical component of the optical flow fieldThe horizontal component and the vertical component of the corrected optical flow field are respectively。
For eachCalculatingHistogram of (1)Equal to the median of the horizontal components of the optical flow fieldThe number of the cells. Order toI.e. byIs the inverse of the level component value that occurs the most frequently. To pairApplying correction to all values inNamely:
in the formulaIs andhave the same dimensions and the elements are allOf the matrix of (a). The horizontal component thus obtainedThe values at most of the positions are 0, and the number is the original one。
To pairIs similar for eachCalculatingHistogram of (1)Equal to the median of the horizontal components of the optical flow fieldThe number of the cells. Order toTo, forApplying correction to all values inNamely:
in the formulaIs andhave the same dimensions and the elements are allOf the matrix of (a).
Fast iteration-based principal direction estimation method
For facial expressions, there are two reasonable assumptions: limited to muscle dimensions, on a small area of the face, the directions of motion are convergent; due to the muscle movement speed, the movement direction is convergent in a very short time window.
After obtaining the corrected optical flow field, the optical flow field is divided into space-time blocks, and according to the assumption of micro expression, the motion vectors in the space-time blocks should be convergent, so that the space-time blocks can be characterized by a main direction. One of the simplest ways is to take the average, which however will take the error of the optical flow field into account also in the main direction, to a certain extent affecting the correctness of the features. Therefore, the invention designs an iterative algorithm, which comprises the following processes:
(a) initializing the principal direction estimate P as a two-dimensional unit vector: p = (1,0)
(b) Each plane coordinate in the block，Finding a time coordinateSo that the motion vector at the positionAndmaximum inner product of (d):
(c) averaging the found motion vectors, normalizing the motion vectors, and taking the normalized motion vectors as an updated value of P:
(d) repeating steps (a) - (c) until P converges or exceeds a maximum number of steps limit.
In the case where the number of correctly estimated motion vectors is large, the method can ignore the case of a small amount of optical flow errors and converge quickly.
The principal direction obtained is quantized into a plurality of sections, the principal direction is represented by the number of the section, and all the principal directions are pulled up, so that the characteristic of a micro expression sequence is obtained.
The novel characteristic based on the motion description provided by the invention can be used for micro-expression recognition, and the method can perform fine alignment operation on the micro-expression sequence, so that the subsequent analysis is more reasonable. The experimental results show that the method is superior to other existing methods in the comprehensive indexes of accuracy, precision and recall rate, and further development of the micro-expression recognition technology is promoted. Meanwhile, the method can depict the dynamic mode of the micro expression, and provides deeper understanding for the analysis of the micro expression.
The experimental effects of the present invention will be described in detail below.
In the case of the experiment 1,two comparison methods are used, namely a method based on a tri-orthogonal plane local binary pattern (LBP-TOP) and a method based on Discriminant Tensor Subspace Analysis (DTSA). The experiment was performed on four datasets, CASME I, CASME II, SMIC2, where SMIC2 contains three subdata sets: HS, VIS, NIR.
Among them, CASME I contains 8 classes, which are kept away from sight (contentt), nausea (distorst), fear (fear), happiness (happy), depression (depression), sadness (sadness), surprise (surrise), and tension (tense), respectively. The frame rate of CASME I is 60 frames/second.
CASME II contains 7 classes, nausea (distust), fear (fear), happiness (happy), depression (depression), sadness (sadness), surprise (surrise) and others (other), respectively. The frame rate of CASME II is 200 frames/second.
Table 1 shows the number of samples in each category in CASME I and CASME II.
In CASME I and CASME II, to obtain effective micro-expressions, participants need to watch mood-inducing video while trying not to make expressions, otherwise the reward for participation will be reduced.
The three subdata sets of both the SMIC and the SMIC2 contain two types of tasks, detection and classification respectively. In the task of detection, given a face sequence, it is required whether the sequence contains micro-expression. In the task of classification, given a sequence of micro-expressions, it is necessary to indicate to which micro-expressions it belongs.
For classification tasks, there are only two categories of SMICs, positive (positive) and negative (negative); SMIC2 contains three categories, positive (positive), negative (negative) and surprise (surrise), respectively.
The frame rate of SMIC and SMIC2-HS is 100 frames/sec; both SMIC2-VIS and SMIC2-NIR have frame rates of 25 frames/sec.
Table 2 shows the number of instances of each category in SMIC and SMIC 2.
Similar to CASME I/II, the micro-expression inducement methods of SMIC and SMIC2 require participants to watch emotionally induced video while suppressing their expressions, otherwise a lengthy questionnaire is filled in as a penalty.
Fig. 3 illustrates some examples of such data sets.
The experiment uses two respective measures, respectively the accuracyAnd. Wherein the accuracy rateThe definition of (A) is:
is defined as
In the above-mentioned definition of the invention,a positive example of a correct classification is indicated,a negative example of a correct classification is indicated,indicating that the error is divided into negative examples of positive examples,indicating a positive case where the error is divided into negative cases. SubscriptIs expressed asThe samples of the classes are positive examples, and the samples of the remaining classes are negative examples of settings.
Tables 3 and 4 show the results of the three methods on six data sets. It can be seen that the method of the invention achieves optimal results in all problems. It should be noted that the results of the Discriminant Tensor Subspace Analysis (DTSA) based method on CASME I and CASME II are not as good as in the original paper, because the original paper experiment does not use the class with a smaller number of samples, and we use the complete data set to do the experiment.
Therefore, we add a set of experiments for CASME I and CASME II, using only the more sample-numerous classes. In particular, only four emotions of nausea, depression, surprise and tension are used in CASME I; only nausea, happiness, depression, surprise and other five expressions were used in CASME II. The results of the experiment are shown in tables 5 and 6. It can be seen that we have still achieved the best performance.
In the case of the experiment 2, the reaction was,to verify the effect of the refined alignment of the present invention, we compared the gap in results between using the refined alignment process and not using this process. Results as shown in table 7, a number of experimental results indicate that fine alignment has a positive impact on the experimental results.
In the case of experiment 3, the reaction was,the principal direction iterative solution method cannot theoretically guarantee convergence, a maximum iteration period needs to be set, and once the maximum iteration period exceeds the maximum iteration period, the iteration process is ended. Thus taking into account the convergence speed of the algorithm in practical applications. Fig. 4 shows the relationship of the iteration period to the convergence block weight. It can be seen that after three iterations have been completed, 90% of the principal directions in the tiles have converged. With the value of t in the third, the main direction in some blocks cannot converge, and this ratio is 0.500% at t =2, 0.833% at t =3, and 0.834% at t = 4. Such a small scale is not sufficient to affect the efficiency of the algorithm and the correctness of the features.
Drawings
Fig. 1 is a flow chart of a micro expression recognition method based on an optical flow field.
FIG. 2 is a diagram illustrating principal direction quantization. The left image is the motion vector in a space-time block, and the right image is the result of quantizing the estimated main direction into 10 blocks.
Fig. 3 data set sample. Wherein the first row is from SMIC2-VIS, a negative micro expression; the second line, from SMIC2-NIR, is an active micro-expression, the original sample contains 13 pictures, here the first 8 are shown; the third row, from SMIC2-HS, is a surprising micro-expression, originally containing 25 pictures, here shown with 8 equidistant (1 in every 3); the fourth line comes from the SMIC, which is not a non-micro-expression sample for the detection task, the original sample containing 34 pictures, of which 8 are shown at equal distances (1 in every 4); the fifth row, from CASME I, is a nausea micro-expression, originally containing 10, here the first 8 shown; the sixth line, from CASME II, is a vanishing micro-expression, as is, consisting of 66 pictures, here shown with equidistant 8 (1 in each 8).
Fig. 4 shows the convergence speed at different values of t. The horizontal axis is the number of iterations, and the vertical axis is the proportion of the spatio-temporal patches whose principal directions converge to the total number of patches at the number of iterations.
Detailed Description
The invention provides a method for describing characteristics of micro-expressions, which is used for identifying and classifying the micro-expressions. The following examples illustrate the operation of the present invention.
In practical use, it is necessary to divide a video sequence from a long-time video sequence in advance. The segmentation can be performed by using a time window with a fixed length, or can be performed by matching a specific pattern. The present invention does not relate to segmentation techniques and therefore the following is only exemplified by a simple time window technique.
Human face video is taken using a high speed camera (50-200 fps) where micro-expression sequences are detected and classified. Conventional 25fps cameras can also capture micro-expressions, however, some very short micro-expressions may be missed. In addition, even for captured micro-expressions, temporally similar information cannot be provided as with a high-speed camera.
Studies have demonstrated that micro-expression typically lasts between 0.05 seconds and 0.2 seconds [9 ]. To this end, we maintain a time window of 0.2 seconds in length, and we can always acquire the past 0.2 seconds of video sequence at each time.
Face detection is performed in this 0.2 second video sequence, resulting in uniformly sized boxes that surround the face. The other part of the video frame is discarded resulting in a face sequence of 0.2 seconds.
And linear interpolation is carried out to obtain a face sequence with the fixed length of 20 frames. Specifically, for each plane position, the pixel value of all frames at this position is regarded as the value of a function at a fixed sampling interval. The video of 0.2 second is divided into 19 sections with fixed length, the pixel values of the left and right nearest neighbors are obtained at each interval point, and linear interpolation is carried out. Thereby resulting in a unified 20-frame face sequence.
In the 20-frame face sequence, a Horn-Schunck method is used to calculate a dense optical flow field between two adjacent frames.
To eliminate the effect of translation on subsequent features, fine alignment based on the optical flow field is required. Specifically, for the horizontal and vertical components of each optical flow fieldAndcalculating a histogramAndwhereinReturning a horizontal component ofThe number of motion vectors of (a) is,returning a horizontal component ofThe number of motion vectors of (2). Computing。
Then order. This completes the fine alignment process.
The micro-expression spatiotemporal sequence is divided into smaller spatiotemporal partitions. In each space-time block, the motion mode in the space-time block can be represented by only finding out one motion principal direction. For this reason, the principal direction is assumed to be P, and initialized to be a unit vector P = (1, 0). At each planar coordinate, a time coordinate is found such that the inner product of the motion vector at that location with P is maximized. The motion vectors found on all horizontal coordinates are averaged and normalized as a new estimate of P. This iterates until P converges. This algorithm does not guarantee convergence and therefore a maximum number of iterations 20 is set and once the number of iterations exceeds this maximum, the iteration is ended.
Thus, the main directions of all the spatio-temporal patches are obtained, and these directions are discretized into 10 directions, which are respectively denoted by 1 and … 10. And (5) connecting the main directions in all the space-time blocks in a pulling mode to obtain the final characteristics of the micro expression sequence.
The characteristics of all micro expressions in a database are calculated, and an SVM (support vector machine) based on a Radial Basis Function Kernel (RBF Kernel) is used for training to obtain a trained SVM classification. For any micro expression sequence, firstly, the micro expression sequence is interpolated into 20 frames by linear interpolation, and the main direction feature is extracted. And (4) judging the expression type by using the trained SVM classifier.
TABLE 1 number of instances of each class of SMIC and SMIC2
TABLE 2 number of CASME I and CASME II Categories
TABLE 3 results of classification
TABLE 4 accuracy of classification results
TABLE 5 results of CASME I/II classification excluding the less sample class
TABLE 6 accuracy of CASME I/II classification results excluding the few samples of the class
| Data set | The invention | LBP-top | DTSA |
| CASME I | 56.14% | 40.35% | 46.20% |
| CASME II | 45.93% | 40.65% | 36.18% |
TABLE 7 Classification Performance enhancements from refined alignment
Claims (1)
1. A micro-expression sequence feature extraction method based on an optical flow field is characterized by comprising the following specific steps:
(1) giving a segment of facial expression sequenceAligning video to specified frame number by interpolationTo obtain；
(2) In the micro-expression sequence with determined length, a Horn-Schunck method is used for estimating a dense optical flow field(ii) a Wherein,is thatAndthe formula expression of the optical flow field is as follows:
，
to representIn the first placeGo to the firstThe pixel values of the columns are selected,andare respectivelyAndin the first placeGo to the firstElements of a column;a motion vector referred to as the location;
(3) eliminating face global displacement using refined alignment algorithm for horizontal componentEach of which isCalculating a histogram，Equal to the median of the horizontal components of the optical flow fieldThe number of (2); order to，
Namely, it isIs the most frequent occurrenceThe inverse of the multiple level component values; order toAll values in (A) plusAnd then obtaining a horizontal optical flow field after accurate alignment:
，
in the formula,is andhave the same dimensions and the elements are allA matrix of (a);
the refined alignment of the vertical component V is similar:
，
；
(4) cutting the aligned optical flow field into space-time blocks, wherein the size of each space-time block is set asAnd seeking a main direction in each space-time block to describe the space-time block, wherein the algorithm flow is as follows:
(a) initializing the principal direction estimate P as a two-dimensional unit vector: p = (1, 0);
(b) each plane coordinate in the block，Finding a time coordinateSo that the motion vector at the positionAndmaximum inner product of (d):
；
(c) averaging the found motion vectors, normalizing the motion vectors, and taking the normalized motion vectors as an updated value of P:
(d) repeating steps (a) - (c) until P converges or exceeds a maximum number of steps limit;
(5) through the steps, a main direction is obtained in each space-time block, each direction is quantized to a plurality of intervals, the main direction is represented by the number of the intervals, and the main directions in all the blocks are connected in a pulling mode, so that the description characteristics of the whole sequence are obtained.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510360969.4A CN104933416B (en) | 2015-06-26 | 2015-06-26 | Micro- expression sequence characteristic extracting method based on optical flow field |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510360969.4A CN104933416B (en) | 2015-06-26 | 2015-06-26 | Micro- expression sequence characteristic extracting method based on optical flow field |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104933416A true CN104933416A (en) | 2015-09-23 |
| CN104933416B CN104933416B (en) | 2018-11-02 |
Family
ID=54120576
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510360969.4A Expired - Fee Related CN104933416B (en) | 2015-06-26 | 2015-06-26 | Micro- expression sequence characteristic extracting method based on optical flow field |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104933416B (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105608440A (en) * | 2016-01-03 | 2016-05-25 | 复旦大学 | Minimum -error-based feature extraction method for face microexpression sequence |
| CN105913038A (en) * | 2016-04-26 | 2016-08-31 | 哈尔滨工业大学深圳研究生院 | Video based dynamic microexpression identification method |
| CN106096537A (en) * | 2016-06-06 | 2016-11-09 | 山东大学 | A kind of micro-expression automatic identifying method based on multi-scale sampling |
| CN106897671A (en) * | 2017-01-19 | 2017-06-27 | 山东中磁视讯股份有限公司 | A kind of micro- expression recognition method encoded based on light stream and FisherVector |
| CN107358206A (en) * | 2017-07-13 | 2017-11-17 | 山东大学 | Micro- expression detection method that a kind of Optical-flow Feature vector modulus value and angle based on area-of-interest combine |
| CN107403142A (en) * | 2017-07-05 | 2017-11-28 | 山东中磁视讯股份有限公司 | A kind of detection method of micro- expression |
| CN108040217A (en) * | 2017-12-20 | 2018-05-15 | 深圳岚锋创视网络科技有限公司 | A kind of decoded method, apparatus of video and camera |
| CN109034126A (en) * | 2018-08-31 | 2018-12-18 | 上海理工大学 | A kind of micro- expression recognition method based on light stream principal direction |
| CN109398310A (en) * | 2018-09-26 | 2019-03-01 | 深圳万智联合科技有限公司 | A kind of pilotless automobile |
| CN109922310A (en) * | 2019-01-24 | 2019-06-21 | 北京明略软件系统有限公司 | The monitoring method of target object, apparatus and system |
| CN110717418A (en) * | 2019-09-25 | 2020-01-21 | 北京科技大学 | Method and system for automatically identifying favorite emotion |
| CN111461021A (en) * | 2020-04-01 | 2020-07-28 | 中国科学院心理研究所 | Micro-expression detection method based on optical flow |
| CN111626179A (en) * | 2020-05-24 | 2020-09-04 | 中国科学院心理研究所 | Micro-expression detection method based on optical flow superposition |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070091085A1 (en) * | 2005-10-13 | 2007-04-26 | Microsoft Corporation | Automatic 3D Face-Modeling From Video |
| CN103440509A (en) * | 2013-08-28 | 2013-12-11 | 山东大学 | Effective micro-expression automatic identification method |
| CN103971137A (en) * | 2014-05-07 | 2014-08-06 | 上海电力学院 | Three-dimensional dynamic facial expression recognition method based on structural sparse feature study |
-
2015
- 2015-06-26 CN CN201510360969.4A patent/CN104933416B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070091085A1 (en) * | 2005-10-13 | 2007-04-26 | Microsoft Corporation | Automatic 3D Face-Modeling From Video |
| CN103440509A (en) * | 2013-08-28 | 2013-12-11 | 山东大学 | Effective micro-expression automatic identification method |
| CN103971137A (en) * | 2014-05-07 | 2014-08-06 | 上海电力学院 | Three-dimensional dynamic facial expression recognition method based on structural sparse feature study |
Non-Patent Citations (2)
| Title |
|---|
| FENG XU 等: "Microexpression Identification and Categorization using a Facial Dynamics Map", 《IEEE TRANSACTION ON AFFECTIVE COMPUTING》 * |
| 贲晛烨 等: "微表情自动识别综述", 《计算机辅助设计与图形学学报》 * |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105608440B (en) * | 2016-01-03 | 2019-05-31 | 复旦大学 | Based on the micro- expression sequence signature abstracting method of face for minimizing error |
| CN105608440A (en) * | 2016-01-03 | 2016-05-25 | 复旦大学 | Minimum -error-based feature extraction method for face microexpression sequence |
| CN105913038A (en) * | 2016-04-26 | 2016-08-31 | 哈尔滨工业大学深圳研究生院 | Video based dynamic microexpression identification method |
| CN105913038B (en) * | 2016-04-26 | 2019-08-06 | 哈尔滨工业大学深圳研究生院 | A kind of micro- expression recognition method of dynamic based on video |
| CN106096537A (en) * | 2016-06-06 | 2016-11-09 | 山东大学 | A kind of micro-expression automatic identifying method based on multi-scale sampling |
| CN106897671A (en) * | 2017-01-19 | 2017-06-27 | 山东中磁视讯股份有限公司 | A kind of micro- expression recognition method encoded based on light stream and FisherVector |
| CN107403142A (en) * | 2017-07-05 | 2017-11-28 | 山东中磁视讯股份有限公司 | A kind of detection method of micro- expression |
| CN107403142B (en) * | 2017-07-05 | 2018-08-21 | 山东中磁视讯股份有限公司 | A kind of detection method of micro- expression |
| CN107358206A (en) * | 2017-07-13 | 2017-11-17 | 山东大学 | Micro- expression detection method that a kind of Optical-flow Feature vector modulus value and angle based on area-of-interest combine |
| CN107358206B (en) * | 2017-07-13 | 2020-02-18 | 山东大学 | Micro-expression detection method based on region-of-interest optical flow features |
| CN108040217A (en) * | 2017-12-20 | 2018-05-15 | 深圳岚锋创视网络科技有限公司 | A kind of decoded method, apparatus of video and camera |
| CN108040217B (en) * | 2017-12-20 | 2020-01-24 | 深圳岚锋创视网络科技有限公司 | Video decoding method and device and camera |
| CN109034126B (en) * | 2018-08-31 | 2021-09-28 | 上海理工大学 | Micro-expression recognition method based on optical flow main direction |
| CN109034126A (en) * | 2018-08-31 | 2018-12-18 | 上海理工大学 | A kind of micro- expression recognition method based on light stream principal direction |
| CN109398310A (en) * | 2018-09-26 | 2019-03-01 | 深圳万智联合科技有限公司 | A kind of pilotless automobile |
| CN109922310A (en) * | 2019-01-24 | 2019-06-21 | 北京明略软件系统有限公司 | The monitoring method of target object, apparatus and system |
| CN110717418A (en) * | 2019-09-25 | 2020-01-21 | 北京科技大学 | Method and system for automatically identifying favorite emotion |
| CN111461021A (en) * | 2020-04-01 | 2020-07-28 | 中国科学院心理研究所 | Micro-expression detection method based on optical flow |
| CN111626179A (en) * | 2020-05-24 | 2020-09-04 | 中国科学院心理研究所 | Micro-expression detection method based on optical flow superposition |
| CN111626179B (en) * | 2020-05-24 | 2023-04-28 | 中国科学院心理研究所 | Micro-expression detection method based on optical flow superposition |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104933416B (en) | 2018-11-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104933416B (en) | Micro- expression sequence characteristic extracting method based on optical flow field | |
| Fan et al. | Fusing dynamic deep learned features and handcrafted features for facial expression recognition | |
| Lin et al. | A heat-map-based algorithm for recognizing group activities in videos | |
| CN109685045B (en) | Moving target video tracking method and system | |
| Zhang et al. | Person re-identification with triplet focal loss | |
| CN110097115B (en) | Video salient object detection method based on attention transfer mechanism | |
| AU2014368997A1 (en) | System and method for identifying faces in unconstrained media | |
| Li et al. | Facial micro-expression recognition based on the fusion of deep learning and enhanced optical flow | |
| CN110827312B (en) | Learning method based on cooperative visual attention neural network | |
| Chen et al. | Person re-identification by exploiting spatio-temporal cues and multi-view metric learning | |
| Yao et al. | Improving long range and high magnification face recognition: Database acquisition, evaluation, and enhancement | |
| Wu et al. | An end-to-end heterogeneous restraint network for RGB-D cross-modal person re-identification | |
| CN105608440B (en) | Based on the micro- expression sequence signature abstracting method of face for minimizing error | |
| Ming et al. | A unified 3D face authentication framework based on robust local mesh SIFT feature | |
| Lu et al. | Micro-expression recognition by regression model and group sparse spatio-temporal feature learning | |
| Abdulghafoor et al. | Real-time object detection with simultaneous denoising using low-rank and total variation models | |
| Zhang et al. | Lightweight mobile network for real-time violence recognition | |
| Luo et al. | CI-Net: Appearance-Based Gaze Estimation via Cooperative Network | |
| CN110222599B (en) | Gait recognition method based on Gaussian mapping | |
| Almalki et al. | Characterizing scattered occlusions for effective dense-mode crowd counting | |
| CN107273873A (en) | Pedestrian based on irregular video sequence recognition methods and system again | |
| Jiang et al. | CE-PeopleSeg: Real-time people segmentation with 10% CPU usage for video conference | |
| Cai | Intelligent automatic processing of sports image based on deep learning algorithm | |
| Jabr | Novel deep learning system for person re-identification using sequence frames of motion. | |
| Duan | An Automatic Extraction Method for Binocular Stereo Colour Vision Image. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20181102 |