CN104933416A - Micro expression sequence feature extracting method based on optical flow field - Google Patents

Abstract

The invention belongs to the technical field of computer vision, and specifically relates to a micro-expression sequence feature extraction method based on an optical flow field. The present invention first extracts the dense optical flow field between adjacent frames under the premise of a certain number of micro-expression frames; then eliminates the influence of face translation on micro-expression recognition through fine alignment; then, the aligned light The flow field is divided into a series of space-time blocks, and the main direction is extracted in each space-time block to characterize the motion mode of most points in the block; the main directions in all blocks are quantified, spliced, and expressed Reach the form of vector, that is, get the designed micro-expression sequence features. The above-mentioned novel feature based on motion description proposed by the present invention can be used for micro-expression recognition. The method of the invention is superior to other existing methods in the comprehensive index of accuracy rate, precision and recall rate, and promotes the further development of micro-expression recognition technology. At the same time, the method can characterize the dynamic patterns of micro-expressions, which provides a deeper understanding for the analysis of micro-expressions.

Description

Micro-expression Sequence Feature Extraction Method Based on Optical Flow Field

technical field

The invention belongs to the technical field of computer vision, and in particular relates to a micro-expression sequence feature extraction method based on an optical flow field.

Background technique

At present, there are many difficulties in micro-expression recognition, and practical methods and theoretical frameworks have not yet been formed at this stage. Its difficulty is mainly manifested in feature extraction. The currently used features are often general-purpose video feature expressions, which are not optimized for the application of micro-expressions, nor can they provide a deep understanding of micro-expressions. the

Microexpressions were first discovered in 1969 when psychologists observed videotaped conversations with depressed patients [1]. The patients in the video often show a normal smile, however, a few frames of abnormally distressed expressions can be found. Psychologists call them microexpressions.

Unlike regular expressions, micro-expressions are tiny expressions that cannot be controlled subjectively. Therefore, observing micro-expressions to judge the real psychological state has potential and important application value in the fields of public security interrogation, diagnosis and treatment of mental illness, business negotiation, etc., and has received considerable attention.

However, the recognition of micro-expression is not easy, the main difficulty lies in its 1) short duration and 2) small range of motion. Even with professionally trained personnel, the recognition accuracy is not high. Therefore, an automatic recognition algorithm based on computer vision can improve the stability of recognition, and can greatly save manpower, which has strong application value. The technical fields involved mainly include: face detection, face key point positioning, face alignment, image preprocessing, feature extraction, machine learning, etc.

Although the development of micro-expression recognition is not perfect, there are still a large number of scholars who have studied it, and the representative literature is listed below.

At present, there are two main research groups that continue to study micro-expression recognition in the world. Starting from the spatio-temporal texture, Oulu University in Finland tried to apply general video features to micro-expressions, extract effective expressions, and recognize micro-expressions. For example, the Local Binary Patten on Three Orthogonal Planes (LBP-TOP) feature used by Pfister extracts the local binary pattern (Local Binary Pattern) on the three planes of X-Y, X-T, and Y-T. In micro-expression description [2]. Among them, for each pixel in the local binary mode, a binary number is used to encode its relationship with the value of surrounding pixels. Then count the distribution histogram of the binary code and express it as a feature. However, micro-expression analysis has high requirements for fine alignment of faces, and this method cannot handle this problem well.

Researcher Wang Sujing from the Institute of Psychology, Chinese Academy of Sciences started with machine learning theory, regarded each micro-expression image sequence as a third-order tensor, and then learned a set of subspaces through discriminative tensor subspace analysis (DTSA) Mapping, so that the distance between tensors of the same category is as small as possible, and the distance between tensors of different categories is as large as possible. Then, the micro-expression tensor after mapping is identified by Extreme Learning Machine [5], which is essentially a machine learning algorithm and does not provide a deep understanding of micro-expressions at the level of feature expression.

Citation:

[1] Ekman P, Friesen W V. Nonverbal leakage and clues to deception. Psychiatry, vol.32, no.1, pp.88-106, 1969.

[2] T. Pfister, X. Li, G. Zhao, and M. Pietikainen. Recognizing spontaneous facial micro-expressions. CVPR, 2011.

[3] M. Shreve, S. Godavarthy, V. Manohar, D. Goldgof, and S. Sarkar. Towards macro- and micro-expression spotting in video using strain patterns. IEEE Workshop on Applications of Computer Vision, 2009.

[4] M. Shreve, S. Godavarthy, D. Goldgof, and S. Sarkar. Macro-and micro-expression spotting in long videos using spatio-temporal strain, AFGR, 2011.

[5] S.-J.Wang, H.-L.Chen, W.-J.Yan, Y.-H.Chen, and X.Fu, Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine, Neural Processing Letters, vol.39, no.1, pp. 25–43, 2014.

[6] X.Li, T.Pfister, X. Huang, G. Zhao, and M. Pietikainen. A spontaneous micro-expression database: Inducement, collection and baseline, AFGR, 2013.

[7] W.-J. Yan, Q. Wu, Y.-J. Liu, S.-J. Wang, and X. Fu, CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces, AFGR, 2013

[8] W.-J.Yan, X.Li, S.-J.Wang, G.Zhao, Y.-J.Liu, Y.-H.Chen, and X.Fu, CASME II: an improved spontaneous micro-expression database and the baseline evaluation, PLoS ONE, vol.9, no.1, p.e86041, 2014.

[9] Wu Q, Shen X, Fu X. The machine knows what you are hiding: an automatic micro-expression recognition system. Effective Computing and Intelligent Interaction. Springer Berlin Heidelberg, pp.152-162, 2011.

Contents of the invention

The purpose of the present invention is to provide an effective micro-expression sequence feature extraction method.

The micro-expression sequence feature extraction method proposed by the present invention, at first, extracts the dense optical flow field between adjacent frames under the premise of a certain number of micro-expression frames; Fine alignment eliminates the impact of face translation on micro-expression recognition; then, the aligned optical flow field is divided into a series of spatiotemporal blocks, and the main direction is extracted in each spatiotemporal block to represent the block The motion patterns of most of the points in the block; the main directions in all the blocks are quantified, concatenated, and expressed in the form of vectors, that is, the designed micro-expression sequence features are obtained. Fig. 1 is a flow diagram of the present invention.

The micro-expression sequence feature extraction method that the present invention proposes, concrete steps are:

1. Given a sequence of facial expressions , align the video to the specified number of frames by interpolation ,get . Wherein, the interpolation method may be a linear interpolation method, or a manifold interpolation method described by Pfister [2].

2. In a micro-expression sequence of a certain length, use the Horn-Schunck method to estimate the dense optical flow field . in, yes and The optical flow field between , its formula expression is:

,

express on the row number the pixel value of the column, and respectively and on the row number elements of the column; is called the motion vector for that location. In practical problems, the above formula cannot be strictly established, but can only be approximated, so there is a certain error. Therefore, the present invention will introduce an iterative method in step 4 to estimate the main direction.

3. Use the refined alignment algorithm to eliminate the overall displacement of the face. in horizontal components For example, for each , computing the histogram . Equal to the median value of the horizontal component of the optical flow field quantity. make ,

Right now is the inverse of the most frequently occurring horizontal component value. make All values in plus , then the precisely aligned horizontal optical flow field is obtained,

,

In the formula yes and have the same dimension, and the elements are matrix;

The refined alignment for the vertical component V is similar:

, .

4. In order to obtain a compact expression, the aligned optical flow field is divided into space-time blocks, so that the size of each space-time block is , in each space-time block seek a main direction to describe the space-time block. Its algorithm flow is:

(a) Initialize the main direction estimation value P as a two-dimensional unit vector: P=(1,0);

(b) Each plane coordinate in the block , , looking for a time coordinate , so that the motion vector at this position and has the largest inner product:

;

(c) Average and normalize the motion vectors found above, and use it as the update value of P:

(d) Repeat steps (a)-(c) until P converges or exceeds the maximum number of steps.

5. Through the above steps, seek a main direction in each space-time block, quantize each direction into several intervals, and use the number of the interval to represent the main direction. For example, Figure 2 describes a method to quantize the main direction into 10 interval strategies. Connect the main directions in all the blocks to obtain the descriptive features of the entire sequence.

The features obtained by the above steps can be used to describe a micro-expression. Use the supervised learning method to learn the labeled micro-expression in the data set, and get the trained classifier (such as support vector machine, Support Vector Machine). By extracting the above features from the unlabeled micro-expression sequence, the classifier can be used to predict its label.

The key of the present invention lies in steps 3 and 4, which are also the main contributions of the present invention: a refined alignment method; a fast iterative-based main direction estimation method. The following are detailed introductions:

Refined Alignment Method

In the above step 3, it is necessary to eliminate the impact of the overall translational motion of the subject's face on the feature extraction. The optical flow field extracted by the present invention not only includes the overall translational motion of the face, but also includes the local motion of micro-expressions. Since the micro-expressions only involve local parts of the face, the overall optical flow field should have a value of 0 in most positions. Therefore, the present invention splits the overall translation motion into horizontal translation and vertical translation, and finds a correction amount for the horizontal component and vertical component of the optical flow field respectively , the horizontal and vertical components of the corrected optical flow field are respectively .

for each ,calculate Histogram of Equal to the median value of the horizontal component of the optical flow field quantity. make ,Right now is the inverse of the most frequently occurring horizontal component value. right Apply corrections to all values in ,Right now:

 

In the formula yes and have the same dimension, and the elements are matrix. The resulting horizontal component The value in most positions is 0, and its number is the original .

right The treatment is similar, for each ,calculate Histogram of Equal to the median value of the horizontal component of the optical flow field quantity. make ,right Apply corrections to all values in ,Right now:

In the formula yes and have the same dimension, and the elements are matrix.

A Fast Iterative Based Principal Direction Estimation Method

For facial expressions, there are two reasonable assumptions: limited to the muscle scale, the direction of motion is convergent on a small area of the face; due to the speed of muscle movement, the direction of motion is convergent in a very short time window .

After obtaining the corrected optical flow field, we divide the optical flow field into spatio-temporal blocks. According to the assumption of micro-expressions, the motion vectors in these spatio-temporal blocks should be convergent, so they can be represented by a main direction. One of the simplest ways is to take the average, but taking the average will also consider the error of the optical flow field in the main direction, which affects the correctness of the features to a certain extent. The present invention has designed a kind of iterative algorithm for this reason, and its flow process is:

(a) Initialize the main direction estimation value P as a two-dimensional unit vector: P=(1,0)

(b) Each plane coordinate in the block , , looking for a time coordinate , so that the motion vector at this position and has the largest inner product:

(c) Average and normalize the motion vectors found above, and use it as the update value of P:

 

(d) Repeat steps (a)-(c) until P converges or exceeds the maximum number of steps.

In the case where correctly estimated motion vectors are in the majority, the method can ignore a small amount of optical flow errors and converge quickly.

Quantize the main direction obtained above into several intervals, use the number of the interval to represent the main direction, and connect all the main directions to obtain the characteristics of a micro-expression sequence.

The above-mentioned new features based on motion description proposed by the present invention can be used for micro-expression recognition. This method can perform fine alignment operations on micro-expression sequences, making subsequent analysis more reasonable. From the experimental results, it can be found that the method is superior to other existing methods in the comprehensive indicators of accuracy, precision and recall rate, which promotes the further development of micro-expression recognition technology. At the same time, the method can characterize the dynamic patterns of micro-expressions, which provides a deeper understanding for the analysis of micro-expressions.

The experimental effects of the present invention will be described in detail below.

In experiment 1, two comparison methods were used, namely the method based on the local binary pattern of three orthogonal surfaces (LBP-TOP) and the method based on discriminant tensor space analysis (DTSA). The experiment is carried out on four datasets of CASME I, CASME II, SMIC, and SMIC2, where SMIC2 contains three sub-datasets: HS, VIS, and NIR.

Among them, CASME I contains 8 categories, which are contempt, disgust, fear, happiness, repression, sadness, surprise and tension. The frame rate of CASME I is 60 frames per second.

CASME II contains 7 categories, namely nausea (disgust), fear (fear), happiness (happiness), depression (repression), sadness (sadness), surprise (surprise) and other (other). The frame rate of CASME II is 200 frames per second.

Table 1 shows the number of samples in each category in CASME I and CASME II.

In CASME I and CASME II, in order to acquire effective microexpressions, participants need to watch emotion-evoking videos while trying not to make expressions, otherwise the reward for participation will be reduced.

The three sub-datasets of SMIC and SMIC2 all contain two types of tasks, namely detection and classification. In the detection task, given a face sequence, it is necessary whether the sequence contains micro-expressions. In the classification task, given a micro-expression sequence, it is necessary to indicate which micro-expression it belongs to.

For classification tasks, SMIC has only two categories, namely positive and negative; SMIC2 contains three categories, namely positive, negative and surprise.

The frame rate of SMIC and SMIC2-HS is 100 frames per second; the frame rate of SMIC2-VIS and SMIC2-NIR is 25 frames per second.

Table 2 shows the number of samples in each category in SMIC and SMIC2.

The micro-expression induction method of SMIC and SMIC2 is similar to CASME I/II. Participants need to watch the video that induces emotions while restraining their expressions, otherwise they will have to fill out a lengthy questionnaire as a punishment.

Figure 3 shows some samples of the above datasets.

The experiment uses two separate measurement indicators, namely the accuracy rate and . Among them, the accuracy rate The definition is:

is defined as

In the above definition, Denotes positive examples that are correctly classified, represents correctly classified negative examples, Indicates that the error is classified as a negative example of a positive example, Represents positive examples that are incorrectly classified as negative examples. subscript means the first The samples of one class are positive examples, and the samples of other classes are negative examples.

Tables 3 and 4 show the results of the three methods on six datasets. It can be seen that the method of the present invention has achieved optimal results on all problems. It should be noted that the results of CASME I and CASME II based on discriminant tensor space analysis (DTSA) are not as good as those in the original paper, because the experiments in the original paper did not use categories with a small number of samples , while we used the full dataset for experiments.

Therefore, for CASME I and CASME II, we add a set of experiments using only categories with a large number of samples. Specifically, only nausea, depression, surprise and tension are used in CASME I; only nausea, happiness, depression, surprise and other five expressions are used in CASME II. The experimental results are shown in Table 5 and Table 6. It can be seen that we still achieved the best results.

In experiment 2, in order to verify the effect of the refined alignment of the present invention, we compared the result gap between using the refined alignment process and not using such a process. The results are shown in Table 7, and a large number of experimental results show that fine alignment has a positive impact on the experimental results.

In experiment 3, the iterative solution method in the main direction cannot theoretically guarantee convergence. It is necessary to set the maximum iteration period, and the iteration process will end once it exceeds. Therefore, consider the convergence speed of the algorithm in practical applications. Figure 4 shows the relationship between the iteration period and the proportion of converged blocks. It can be seen that after completing three iterations, the main directions in 90% of the blocks have converged. Under the three values of t, there will be some main directions in the blocks that cannot converge. This ratio is 0.500% when t=2, 0.833% when t=3, and 0.834% when t=4 . Such a small ratio is not enough to affect the efficiency of the algorithm and the correctness of the features.

Description of drawings

Figure 1 is a flow chart of the micro-expression recognition method based on optical flow field.

Figure 2 is a schematic diagram of main direction quantization. Among them, the left picture is the motion vector in a space-time block, and the right picture is the result of quantizing the estimated main direction to 10 partitions.

Figure 3 Sample dataset. Among them, the first line is from SMIC2-VIS, which is a negative micro-expression; the second line is from SMIC2-NIR, which is a positive micro-expression. The original sample contains 13 pictures, and the first 8 pictures are shown here; the third line is from SMIC2-HS, which is a surprised microexpression, the original sample contains 25 images, 8 of which are shown here at an equidistant distance (1 for every 3 images); the fourth row is from SMIC, which is not a non-microexpression sample, For detection tasks, the original sample contains 34 pictures, and here are 8 of them in the middle distance (1 for every 4 pictures); the fifth line is from CASME I, which is a disgusting micro-expression, the original sample contains 10 pictures, here The first 8 pictures are shown here; the sixth line is from CASME II, which is a depressed micro-expression. The original sample contains 66 pictures, and 8 of them are shown here at a medium distance (1 picture for every 8 pictures).

Figure 4 Convergence speed at different t values. The horizontal axis is the number of iterations, and the vertical axis is the ratio of the space-time blocks that converge in the main direction to the total number of blocks under the number of iterations.

Detailed ways

The present invention provides a method for describing the features of micro-expressions, and uses such features for identification and classification of micro-expressions. The following examples illustrate the application of the present invention.

In practical application, it is necessary to segment the sequence from the long-term video sequence in advance. Segmentation can use a fixed-length time window, or match a specific pattern for segmentation. The present invention does not involve segmentation technology, therefore, only a simple time window technology is used as an example below.

A high-speed camera (50-200fps) is used to capture human facial video, in which micro-expression sequences are detected and classified. Conventional 25fps cameras can also capture micro-expressions, but some extremely short micro-expressions may be missed. In addition, even for captured micro-expressions, it cannot provide temporally similar information as high-speed cameras can.

Studies have shown that micro-expressions usually last between 0.05 seconds and 0.2 seconds [9]. To this end, we maintain a time window of length 0.2 seconds, and at each moment we always have access to the video sequence 0.2 seconds past.

Face detection is performed in this 0.2-second video sequence, and a box of uniform size is obtained to surround the face. Discard other parts of the video frame to obtain a 0.2-second face sequence.

Perform linear interpolation to obtain a face sequence with a fixed length of 20 frames. Specifically, for each plane position, the pixel values of all frames at this position are regarded as the value of a function at a fixed sampling interval. Divide the 0.2-second video into 19 fixed-length intervals, obtain the left and right nearest neighbor pixel values at each interval point, and perform linear interpolation. A unified 20-frame face sequence is thus obtained.

In the 20-frame face sequence, the Horn-Schunck method is used to calculate the dense optical flow field between two adjacent frames.

In order to eliminate the influence of translation on subsequent features, fine alignment based on optical flow field is required. Specifically, for each horizontal and vertical component of the optical flow field and , computing the histogram and ,in Returns the horizontal component as The number of motion vectors, Returns the horizontal component as The number of motion vectors. calculate .

Then order . This completes the process of fine alignment.

Divide the micro-expression spatio-temporal sequence into smaller spatio-temporal blocks. In each spatio-temporal block, as long as a main motion direction is found, the motion pattern in this spatio-temporal block can be represented. For this, assume that the main direction is P, and initialize it to a unit vector P=(1,0) . On each plane coordinate, find a time coordinate so that the inner product of the motion vector and P at this position is the largest. The motion vectors found on all horizontal coordinates are averaged and normalized as a new estimate of P. This iterates until P converges. This algorithm does not guarantee convergence. Therefore, a maximum number of iterations is set to 20, and once the number of iterations exceeds this maximum value, the iteration is terminated.

In this way, the main directions of all space-time blocks can be obtained, and these directions are discretized into 10 directions, denoted by 1,...10 respectively. The main directions in all spatio-temporal blocks are connected to obtain the final features of the micro-expression sequence.

We calculate the characteristics of all the micro-expressions in the database, and use the SVM based on the Radial Basis Function Kernel (RBF Kernel) for training to obtain a trained SVM classification. For any micro-expression sequence, first use linear interpolation to interpolate it to 20 frames, and extract the main direction features. Use the trained SVM classifier to discriminate the type of expression.

Table 1 The number of samples in each category of SMIC and SMIC2

Table 2 The number of samples in each category of CASME I and CASME II

Table 3 Classification results of

Table 4 Accuracy of classification results

Table 5 CASME I/II classification results after excluding categories with a small number of samples

Table 6 The accuracy rate of CASME I/II classification results after excluding categories with a small number of samples

data set this invention LBP-top DTSA CASME I 56.14% 40.35% 46.20% Casme II 45.93% 40.65% 36.18%

Table 7 Classification performance improvement brought about by fine alignment

Claims (1)

1. A micro-expression sequence feature extraction method based on optical flow field, characterized in that the specific steps are: (1) Given a sequence of facial expressions , align the video to the specified number of frames by interpolation ,get ; (2) In a micro-expression sequence of a certain length, use the Horn-Schunck method to estimate the dense optical flow field ;in, yes and The optical flow field between , its formula expression is: , express on the row number the pixel value of the column, and respectively and on the row number elements of the column; is called the motion vector for that position; (3) Use the refined alignment algorithm to eliminate the overall displacement of the face, for the horizontal component ,Every , computing the histogram , Equal to the median value of the horizontal component of the optical flow field the number of orders , Right now is the opposite number of the horizontal component value that occurs most frequently; let All values in plus , then the precisely aligned horizontal optical flow field is obtained: , In the formula, yes and have the same dimension, and the elements are matrix; The refined alignment for the vertical component V is similar: , ; (4) Divide the aligned optical flow field into space-time blocks, so that the size of each space-time block is , seek a main direction in each space-time block to describe the space-time block, and its algorithm flow is: (a) Initialize the main direction estimation value P as a two-dimensional unit vector: P=(1,0); (b) Each plane coordinate in the block , , looking for a time coordinate , so that the motion vector at this position and has the largest inner product: ; (c) Average and normalize the motion vectors found above, and use it as the update value of P: (d) Repeat steps (a)-(c) until P converges or exceeds the maximum number of steps; (5) Through the above steps, seek a main direction in each space-time block, quantize each direction into several intervals, and use the number of the interval to represent the main direction, and connect the main directions in all blocks, That is, the descriptive features of the entire sequence are obtained.