CN112766159A - Cross-database micro-expression identification method based on multi-feature fusion - Google Patents
Cross-database micro-expression identification method based on multi-feature fusion Download PDFInfo
- Publication number
- CN112766159A CN112766159A CN202110073665.5A CN202110073665A CN112766159A CN 112766159 A CN112766159 A CN 112766159A CN 202110073665 A CN202110073665 A CN 202110073665A CN 112766159 A CN112766159 A CN 112766159A
- Authority
- CN
- China
- Prior art keywords
- micro
- optical flow
- frame
- feature
- face
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 230000004927 fusion Effects 0.000 title claims abstract description 15
- 230000003287 optical effect Effects 0.000 claims abstract description 95
- 238000010586 diagram Methods 0.000 claims abstract description 19
- 210000000003 hoof Anatomy 0.000 claims abstract description 16
- 230000001815 facial effect Effects 0.000 claims abstract description 11
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 8
- 238000013528 artificial neural network Methods 0.000 claims abstract description 7
- 230000014509 gene expression Effects 0.000 claims description 36
- 238000010606 normalization Methods 0.000 claims description 19
- 238000012545 processing Methods 0.000 claims description 18
- 239000013598 vector Substances 0.000 claims description 18
- 230000006870 function Effects 0.000 claims description 14
- 238000011176 pooling Methods 0.000 claims description 11
- 230000008569 process Effects 0.000 claims description 11
- 230000008451 emotion Effects 0.000 claims description 8
- 238000012549 training Methods 0.000 claims description 7
- 238000006073 displacement reaction Methods 0.000 claims description 5
- 230000001419 dependent effect Effects 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 230000017105 transposition Effects 0.000 claims description 2
- 238000013135 deep learning Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 230000008921 facial expression Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000002996 emotional effect Effects 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000012952 Resampling Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000008602 contraction Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 210000001097 facial muscle Anatomy 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 210000003205 muscle Anatomy 0.000 description 1
- 210000000653 nervous system Anatomy 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
- 230000001755 vocal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention belongs to the field of computer vision, and particularly relates to a cross-database micro-expression identification method based on multi-feature fusion, which comprises the following steps: acquiring a start frame and a vertex frame in two data sets; cutting the face area of the initial frame and the vertex frame to obtain a face area image; calculating a human face key point feature map and optical flow features according to the preprocessed initial frame and the preprocessed top frame, and calculating the direction, the amplitude and the optical strain of the optical flow features; weighting the obtained optical flow amplitude characteristic diagram and direction characteristic diagram to obtain a HOOF characteristic diagram; forming a three-dimensional feature block by the face key point feature map, the optical strain feature map and the HOOF feature map, inputting the three-dimensional feature block into a three-dimensional convolution neural network for feature learning, and obtaining a micro-expression recognition result; the method well utilizes the shallow convolutional neural network to effectively learn the micro-expression characteristics fused with the multiple characteristics, thereby improving the accuracy and efficiency of the facial micro-expression recognition.
Description
Technical Field
The invention belongs to the field of computer vision, and particularly relates to a cross-database micro-expression identification method based on multi-feature fusion.
Background
Human facial expressions are mainly classified into macro expressions and micro expressions. Facial expressions are a form of non-verbal communication, resulting from contraction of facial muscles in emotional states. Different muscle movements and patterns ultimately reflect different types of emotions. However, a macro expression does not accurately imply an emotional state of a person because it is easily forged. The micro expression is a transient facial expression which cannot be controlled by a nervous system, can reflect the real emotion of a human and cannot be hidden. Micro-expression has many potential applications in psychology, public safety, clinical medicine, criminal interrogation, etc. But the duration is only about 0.04 s-0.2 s, the intensity is weak, the micro expression is difficult to be recognized by human eyes, the recognition rate of the micro expression is not high even for a person who is trained professionally, and a large amount of time cost is spent. Research on computer-based automatic recognition of micro-expressions has become a focus.
At present, the traditional machine learning method and the deep learning method are mainly used for automatic micro-expression recognition. The traditional micro expression recognition algorithm mainly utilizes different feature extractors to extract micro expression features, then inputs the extracted features into a classifier, and utilizes the classifier to learn a training sample to recognize micro expressions. However, the traditional method has a complex feature extraction mode, a feature descriptor needs to be designed manually, and because the motion of the micro expression is local, the traditional feature extraction method is difficult to completely capture the weak change of the face, meanwhile, the calculation amount of the features is large, and the selection of a classifier has a great influence on the classification performance. Therefore, the traditional micro expression recognition method has certain limitations.
In recent years, the deep neural network has made remarkable progress in the face micro-expression recognition. The robustness of deep learning brings more promising performance than traditional manual methods. However, deep learning requires a large amount of data, and micro-expression data is relatively lacking, which causes a problem of difficult recognition.
Disclosure of Invention
In order to solve the problems in the prior art, the invention provides a cross-database micro-expression identification method based on multi-feature fusion, which comprises the following steps: acquiring a human face micro expression image, and preprocessing the human face micro expression image; calculating three characteristic graphs according to the preprocessed face image, and fusing the three characteristic graphs; inputting the fused feature diagram into a trained micro-expression recognition model to obtain a micro-expression recognition result;
the process of training the micro-expression recognition model comprises the following steps:
s1: respectively acquiring a starting frame and a vertex frame of a CASMEII data set and a SAMM data set and an emotion label corresponding to each data;
s2: cutting the face regions of the initial frame and the vertex frame to obtain an image only containing the face region;
s3: carrying out normalization processing on the face region image; the normalization processing comprises scale normalization processing and gray level normalization processing;
s4: calculating a face key point feature map LMF according to the initial frame and the vertex frame of the normalized face region image;
s5: calculating the optical flow characteristics of the facial region image according to the initial frame and the vertex frame of the facial region image after normalization processing; calculating an optical flow direction feature map p, an amplitude feature map m and an optical strain feature map epsilon according to the optical flow features; the optical flow features comprise horizontal optical flow features and vertical optical flow features;
s6: weighting the optical flow direction characteristic diagram p and the amplitude characteristic diagram m to obtain a HOOF characteristic diagram;
s7: forming a three-dimensional feature block by using the face key point feature map LMF, the optical strain feature map epsilon and the HOOF feature map, and inputting the three-dimensional feature block into a three-dimensional convolution neural network for feature learning to obtain a micro-expression recognition result;
s8: calculating a loss function of the micro expression recognition model according to the micro expression recognition result; and continuously adjusting parameters of the loss function, and finishing the training of the model when the value of the loss function is minimum.
Preferably, the process of acquiring the start frame and the vertex frame of the casseii dataset and the SAMM dataset includes: giving out the index of the initial frame and the index of the vertex frame of the micro-expression video sequence in the data set, and directly obtaining two frames of pictures and emotion labels according to the indexes; the starting frame is the frame of image where the observed person starts to appear the micro expression, and the top frame is the frame of image where the micro expression intensity of the observed person is most significant.
Preferably, the clipping the face regions of the start frame and the vertex frame includes detecting the face regions of the start frame and the vertex frame by using a face detector of a dlib library, detecting 68 face key points of the start frame and the vertex frame by using a face landmark detector, and obtaining the position of the clipping region and the size of the clipping region according to the face key points to obtain an image only including the face regions.
Preferably, the formula for calculating the face key point feature map LMF is as follows:
LMFt(i,j)＝||p(i,t)-p(j,t)||2-||p(i,t-Δt)-p(j,t-Δt)||2
preferably, the process of calculating the optical flow features and the respective feature maps of the face region image includes:
step 1: calculating facial area images of a start frame and a vertex frame by adopting a TV-L1 approximate optical flow method to obtain a horizontal optical flow and a vertical optical flow of an optical flow field;
step 2: converting the horizontal optical flow and the vertical optical flow into optical flow vectors p ═ u, v, converting the coordinates of the optical flow vectors into polar coordinates, and calculating the direction θ of the optical flow motion from the polar coordinatesx,yAnd intensity ρx,y；
And step 3: calculating an optical strain characteristic map epsilon according to the optical flow vector p ═ (u, v);
and 4, step 4: and calculating an amplitude characteristic map m according to the optical strain characteristic map epsilon.
Further, the direction of the optical flow movement θx,yAnd intensity ρx,yThe formula of (1) is:
further, the formula for calculating the optical strain characteristic map epsilon is as follows:
preferably, the process of obtaining the HOOF feature map includes: equally dividing the optical flow amplitude feature map and the optical flow direction feature map into 5-by-5 non-overlapping blocks; to the direction thetax,y∈[-π,π]Bin processing is performed and dependent on the amplitude ρx,0Weighting the sizes of the two to obtain a HOOF characteristic diagram; wherein the range of each histogram bin is:
preferably, the three-dimensional convolutional neural network comprises: the three-dimensional convolutional neural network comprises three parallel streams, a convolutional layer, a maximum pooling layer, an FC full-link layer and a softmax layer; each of the three parallel flows consists of a convolution layer and a maximum pooling layer, the number of convolution kernels of the convolution layer of each flow is 3, 5 and 8 respectively, and the sizes of the convolution kernels are 3 x 3; adopting same padding to keep the input and output sizes the same, and stride being 1; inputting the three-dimensional characteristic block of 28 multiplied by 3 into a three-dimensional convolution neural network, wherein the outputs obtained by the three parallel streams are respectively data of 28 multiplied by 3, 28 multiplied by 5 and 28 multiplied by 8; the data output by the convolution layer respectively passes through the maximum pooling layers of 2 × 2 and stride 2, and data of 14 × 14 × 3, 14 × 14 × 5 and 14 × 14 × 8 are output; connecting the outputs of the three streams to form a 14 × 14 × 16 feature block, and outputting a 7 × 16-dimensional feature through a maximum pooling layer of 2 × 2 and stride 2, and flattening the feature into a 784 × 1-dimensional feature vector; and (4) subjecting the feature vector of 784 x 1 dimension to softmax function to micro-expression recognition classification.
Preferably, the penalty function of the micro-expression recognition model is:
aiming at the problems of short duration, weak intensity and small sample data amount of micro expression, the invention provides a traditional micro expression recognition method combined with a deep learning technology for micro expression recognition on the basis of a cross-database experiment. A plurality of feature graphs of the micro expression are obtained through calculation by a traditional method, and the plurality of features are fused and sent to a convolutional neural network for further feature learning, so that the defect that the micro expression is recognized by directly using an original micro expression sequence is overcome. The method effectively improves the accuracy and efficiency of micro-expression recognition.
Drawings
FIG. 1 is a general flow diagram of the present invention;
FIG. 2 is a sample view of the present invention;
FIG. 3 is a picture of the present invention containing only facial regions;
FIG. 4 is a graph of the calculated optical flow characteristics of the present invention;
FIG. 5 is a graph of calculated optical strain characteristics according to the present invention;
FIG. 6 is a HOOF feature diagram example;
fig. 7 is a 3D convolutional neural network designed by the present invention.
Detailed Description
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings, and the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Other embodiments obtained by persons of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the protection scope of the present invention.
A cross-database micro-expression identification method based on multi-feature fusion comprises the following steps: acquiring a human face micro expression image, and preprocessing the human face micro expression image; calculating three characteristic graphs according to the preprocessed face image, and fusing the three characteristic graphs; inputting the fused feature diagram into a trained micro-expression recognition model to obtain a micro-expression recognition result; the process of training the micro expression recognition model is shown in fig. 1, and the specific process includes:
s1: respectively acquiring a starting frame and a vertex frame of a CASMEII data set and a SAMM data set and an emotion label corresponding to each data;
s2: cutting the face regions of the initial frame and the vertex frame by adopting a face detector of a dlib library to obtain an image only containing the face regions;
s3: carrying out normalization processing on the face region image; the normalization processing comprises scale normalization processing and gray level normalization processing;
s4: calculating a face key point feature map LMF according to the initial frame and the vertex frame of the normalized face region image;
s5: calculating the optical flow characteristics of the facial region image according to the initial frame and the vertex frame of the facial region image after normalization processing; calculating an optical flow direction feature map p, an amplitude feature map m and an optical strain feature map epsilon according to the optical flow features; the optical flow features comprise horizontal optical flow features and vertical optical flow features;
s6: the optical flow direction feature map p and the amplitude feature map m are equally divided into 5 × 5 blocks which are not overlapped, and the direction is binned in each block. Weighting the direction according to the moving amplitude of each pixel to obtain a HOOF characteristic diagram;
s7: forming a three-dimensional feature block by using the face key point feature map LMF, the optical strain feature map epsilon and the HOOF feature map, and inputting the three-dimensional feature block into a three-dimensional convolution neural network for feature learning to obtain a micro-expression recognition result;
s8: calculating a loss function of the micro expression recognition model according to the micro expression recognition result; and continuously adjusting parameters of the loss function, and finishing the training of the model when the value of the loss function is minimum.
As shown in FIG. 2, the a image in FIG. 2 is the start frame of the Happoess sample, and the b image in FIG. 2 is the top frame of the Happoess sample. Since the study of micro-expressions is video-based and not a single picture, the start and vertex frames of micro-expressions and corresponding emotion labels can be selected directly from the index for a given sample sequence.
And respectively extracting face region images from the initial frame and the vertex frame, and then performing scale normalization and gray scale normalization. The method comprises the following specific steps:
step 1: using a face detector and a face key point detector included in the dlib library to respectively detect face regions of the start frame and the vertex frame, and performing 68 face key point detections, and obtaining the position of the cropping region and the size of the cropping region according to the 68 face key points detected, to obtain an image only including the face region, as shown in fig. 3, where the image a in fig. 3 is a face region image of the start frame, and the image b in fig. 3 is a face region image of the vertex frame.
Step 2: and carrying out geometric normalization and gray level normalization operation on the image only containing the face area of the human face.
And calculating a face key point feature map LMF according to the preprocessed initial frame and the preprocessed top frame, wherein the key points are used for having the advantage of not being influenced by characters such as the shape, sex, age and brightness of the face in the input video or background features, and the attention points are put on the face motion. The formula for calculating the face key point feature map LMF is as follows:
LMFt(i,j)＝||p(i,t)-p(j,t)||2-||p(i,t-Δt)-p(j,t-Δt)||2
where p (i, t) is the ith personal face key point in the tth frame, where i is 1,2 … 68, the start frame corresponds to the frame at time t- Δ t, and the vertex frame corresponds to the frame at time t.
Preferably, as the face has 68 key points, 68 key points are collected for the face image, and the size of the face key point feature map LMF is 68 × 68.
The process of calculating the optical flow characteristics of the face area image and each characteristic graph according to the preprocessed initial frame and the preprocessed vertex frame comprises the following steps:
step 1: calculating facial area images of a start frame and a vertex frame by adopting a TV-L1 approximate optical flow method to obtain a horizontal optical flow and a vertical optical flow of an optical flow field; as shown in fig. 4, the image a in fig. 4 is a horizontal light flow diagram, and the image b in fig. 4 is a vertical light flow diagram. Assuming that the intensity of the image pixel at the point (x, y, t) is I (x, y, t), the pixel point moves to the position of the point (x + Δ x, y + Δ y, t + Δ t) after the time Δ t, and the intensity of the pixel is I (x + Δ x, y + Δ y, t + Δ t); from the conservation of brightness:
I(x,y,t)＝I(x+Δx,y+Δy,t+Δt)
wherein Δ x ═ ux,yΔt，Δy＝vx,yΔt，ux,yAnd vx,yRespectively, a horizontal optical flow and a vertical optical flow to be estimated, Δ x represents a displacement of a pixel in a horizontal direction, and Δ y represents a displacement of a pixel in a vertical direction.
Step 2: converting the horizontal optical flow and the vertical optical flow into optical flow vectors p ═ u, v, converting the coordinates of the optical flow vectors into polar coordinates, and calculating the direction θ of the optical flow motion from the polar coordinatesx,yAnd intensity ρx,y(ii) a The calculation formula is as follows:
where ρ isx,yTheta being the magnitude of the pixel shiftx,yU (x, y) and v (x, y) are directions in which the pixels move.
And step 3: calculating an optical strain characteristic map epsilon according to the optical flow vector p ═ (u, v); optical strain is the derivative of optical flow that correctly characterizes the minute movements of a variable object between successive frames of motion, as shown in FIG. 5. The optical strain ε is defined as:
wherein,
means to derive a displacement vector, T means transposition; u ═ u, v]TRepresenting the displacement vector, the optical strain epsilon can also be expressed as:
wherein a diagonal strain component (epsilon)xx，εyy) Is the normal strain component (ε)xy，εyx) Is the tangential strain component. The normal strain component measures the change in optical flow in the x and y dimensions, while the tangential strain component measures the angular change caused by deformation along two axes.
And 4, step 4: and calculating an amplitude characteristic map m according to the optical strain characteristic map epsilon. The sum of the squares of the normal and tangential strain components can be used to calculate the magnitude of the optical strain for each pixel by the formula:
equally dividing the optical flow amplitude feature map and the optical flow direction feature map into 5-by-5 non-overlapping blocks, and aligning the direction thetax,y∈[-π,π]Bin processing is performed and dependent on the corresponding amplitude ρx,yIs weighted to obtain the HOOF feature, as shown in fig. 6. The range of each histogram bin is:
where bin C is e {1,2 … C }, and C represents the total number of histogram bins.
As shown in FIG. 7, the three-dimensional convolutional neural network of the present invention is a small and shallow convolutional neural network; the network has three parallel flows, each flow consists of a convolution layer and a pooling layer, each flow has different convolution kernel numbers of 3, 5 and 8, the convolution kernel sizes are all 3 × 3, same input and output sizes are kept by same stride is 1. In a three-dimensional feature block theta (namely { LMF, epsilon, HOOF } obtained by resampling the face key point feature map LMF, the optical strain epsilon and the HOOF feature map to 28 × 28 × 3, the outputs obtained by three parallel streams are respectively 28 × 28 × 3, 28 × 28 × 5 and 28 × 28 × 8; then, the samples were passed through the maximum pooling layers of 2 × 2 and stride 2, and the outputs were 14 × 14 × 3, 14 × 14 × 5, and 14 × 14 × 8, respectively; and connecting the outputs of the three streams to form a 14 × 14 × 16 feature block, outputting a 7 × 16-dimensional feature through a maximum pooling layer of 2 × 2 and stride 2, flattening the feature into a 784 × 1-dimensional feature vector, connecting the feature vector with a full-connection layer of 400 nodes, and finally performing micro-expression recognition and classification through a softmax function.
Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by instructions associated with hardware via a program, which may be stored in a computer-readable storage medium, and the storage medium may include: ROM, RAM, magnetic or optical disks, and the like.
The above-mentioned embodiments, which further illustrate the objects, technical solutions and advantages of the present invention, should be understood that the above-mentioned embodiments are only preferred embodiments of the present invention, and should not be construed as limiting the present invention, and any modifications, equivalents, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A cross-database micro-expression identification method based on multi-feature fusion is characterized by comprising the following steps: acquiring a human face micro expression image, and preprocessing the human face micro expression image; calculating three characteristic graphs according to the preprocessed face image, and fusing the three characteristic graphs; inputting the fused feature diagram into a trained micro-expression recognition model to obtain a micro-expression recognition result;
the process of training the micro-expression recognition model comprises the following steps:
s1: respectively acquiring a starting frame and a vertex frame of a CASMEII data set and a SAMM data set and an emotion label corresponding to each data;
s2: cutting the face regions of the initial frame and the vertex frame to obtain an image only containing the face region;
s3: carrying out normalization processing on the face region image; the normalization processing comprises scale normalization processing and gray level normalization processing;
s4: calculating a face key point feature map LMF according to the initial frame and the vertex frame of the normalized face region image;
s5: calculating the optical flow characteristics of the facial region image according to the initial frame and the vertex frame of the facial region image after normalization processing; calculating an optical flow direction feature map p, an amplitude feature map m and an optical strain feature map epsilon according to the optical flow features; the optical flow features comprise horizontal optical flow features and vertical optical flow features;
s6: weighting the optical flow direction characteristic diagram p and the amplitude characteristic diagram m to obtain a HOOF characteristic diagram;
s7: forming a three-dimensional feature block by using the face key point feature map LMF, the optical strain feature map epsilon and the HOOF feature map, and inputting the three-dimensional feature block into a three-dimensional convolution neural network for feature learning to obtain a micro-expression recognition result;
s8: calculating a loss function of the micro expression recognition model according to the micro expression recognition result; and continuously adjusting parameters of the loss function, and finishing the training of the model when the value of the loss function is minimum.
2. The method for cross-database micro-expression recognition based on multi-feature fusion, according to claim 1, wherein the process of obtaining the start frame and the vertex frame of the CASMEII dataset and the SAMM dataset comprises: giving out the index of the initial frame and the index of the vertex frame of the micro-expression video sequence in the data set, and directly obtaining two frames of pictures and emotion labels according to the indexes; the starting frame is the frame of image where the observed person starts to appear the micro expression, and the top frame is the frame of image where the micro expression intensity of the observed person is most significant.
3. The method for cross-database micro-expression recognition based on multi-feature fusion of claim 1, wherein the cropping of the face region of the start frame and the vertex frame comprises detecting the face region of the start frame and the vertex frame by a face detector of a dlib library, detecting 68 face key points of the start frame and the vertex frame by a face landmark detector, and obtaining the position of the cropping region and the size of the cropping region according to the face key points to obtain an image only containing the face region.
4. The cross-database micro-expression recognition method based on multi-feature fusion as claimed in claim 1, wherein the formula for calculating the face key point feature map LMF is as follows:
LMFt(i，j)＝||p(i，t)-p(j，t)||2-||p(i，t-Δt)-p(j，t-Δt)||2
where p (i, t) represents the ith personal face keypoint in the tth frame, and Δ t represents a time interval.
5. The method for cross-database micro-expression recognition based on multi-feature fusion, according to claim 1, wherein the process of calculating the optical flow features of the face area image and each feature map comprises:
step 1: calculating facial area images of a start frame and a vertex frame by adopting a TV-L1 approximate optical flow method to obtain a horizontal optical flow and a vertical optical flow of an optical flow field;
step 2: converting the horizontal optical flow and the vertical optical flow into optical flow vectors p ═ u, v, converting the coordinates of the optical flow vectors into polar coordinates, and calculating the direction θ of the optical flow motion from the polar coordinatesx，yAnd intensity ρx，y；
And step 3: calculating an optical strain characteristic map epsilon according to the optical flow vector p ═ (u, v);
and 4, step 4: and calculating an amplitude characteristic map m according to the optical strain characteristic map epsilon.
6. The method of claim 5, wherein the method comprises a step of optically identifying the micro-expression across databases based on multi-feature fusionDirection of flow motion thetax，yAnd intensity ρx，yThe formula of (1) is:
where ρ isx，yTheta being the magnitude of the pixel shiftx，yIn the direction of pixel movement, u (x, y) represents a horizontal optical flow component, and v (x, y) represents a vertical optical flow component.
7. The method for cross-database micro-expression recognition based on multi-feature fusion according to claim 5, wherein the formula for calculating the optical strain characteristic map epsilon is as follows:
wherein u is [ u, v ]]TRepresenting a displacement vector and T representing a transposition.
8. The method for cross-database micro-expression recognition based on multi-feature fusion according to claim 1, wherein the process of obtaining the HOOF feature map comprises: equally dividing the optical flow amplitude feature map and the optical flow direction feature map into 5-by-5 non-overlapping blocks; to the direction thetax，y∈[-π，π]Bin processing is performed and dependent on the amplitude ρx，yWeighting the sizes of the two to obtain a HOOF characteristic diagram; wherein the range of each histogram bin is:
wherein bin C is in {1, 2.. C }, and C represents the total number of histogram bins，θx，yIndicating the direction of the optical flow motion.
9. The cross-database micro-expression recognition method based on multi-feature fusion is characterized in that the three-dimensional convolutional neural network comprises three parallel streams, a convolutional layer, a max pooling layer, an FC full connection layer and a softmax layer; each of the three parallel flows consists of a convolution layer and a maximum pooling layer, the number of convolution kernels of the convolution layer of each flow is 3, 5 and 8 respectively, and the sizes of the convolution kernels are 3 x 3; adopting same padding to keep the input and output sizes the same, and stride being 1; inputting the three-dimensional characteristic block of 28 multiplied by 3 into a three-dimensional convolution neural network, wherein the outputs obtained by the three parallel streams are respectively data of 28 multiplied by 3, 28 multiplied by 5 and 28 multiplied by 8; the data output by the convolution layer respectively passes through the maximum pooling layers of 2 × 2 and stride 2, and data of 14 × 14 × 3, 14 × 14 × 5 and 14 × 14 × 8 are output; connecting the outputs of the three streams to form a 14 × 14 × 16 feature block, and outputting a 7 × 16-dimensional feature through a maximum pooling layer of 2 × 2 and stride 2, and flattening the feature into a 784 × 1-dimensional feature vector; and (4) subjecting the feature vector of 784 x 1 dimension to softmax function to micro-expression recognition classification.
10. The cross-database micro expression recognition method based on multi-feature fusion as claimed in claim 1, wherein the loss function of the micro expression recognition model is:
where N is the number of samples, M is the number of data set categories, yicTo indicate a variable, 1 if the class is the same as that of sample i, otherwise 0, picIs the predicted probability that sample i belongs to class c.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110073665.5A CN112766159A (en) | 2021-01-20 | 2021-01-20 | Cross-database micro-expression identification method based on multi-feature fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110073665.5A CN112766159A (en) | 2021-01-20 | 2021-01-20 | Cross-database micro-expression identification method based on multi-feature fusion |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112766159A true CN112766159A (en) | 2021-05-07 |
Family
ID=75703436
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110073665.5A Pending CN112766159A (en) | 2021-01-20 | 2021-01-20 | Cross-database micro-expression identification method based on multi-feature fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112766159A (en) |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113361415A (en) * | 2021-06-08 | 2021-09-07 | 浙江工商大学 | Micro-expression data set collection method based on crowdsourcing |
| CN113469153A (en) * | 2021-09-03 | 2021-10-01 | 中国科学院自动化研究所 | Multi-modal emotion recognition method based on micro-expressions, limb actions and voice |
| CN113537008A (en) * | 2021-07-02 | 2021-10-22 | 江南大学 | Micro-expression identification method based on adaptive motion amplification and convolutional neural network |
| CN113901895A (en) * | 2021-09-18 | 2022-01-07 | 武汉未来幻影科技有限公司 | Door opening action recognition method and device for vehicle and processing equipment |
| CN113902774A (en) * | 2021-10-08 | 2022-01-07 | 无锡锡商银行股份有限公司 | Method for detecting facial expression of dense optical flow characteristics in video |
| CN113920571A (en) * | 2021-11-06 | 2022-01-11 | 北京九州安华信息安全技术有限公司 | Micro-expression identification method and device based on multi-motion feature fusion |
| CN114005157A (en) * | 2021-10-15 | 2022-02-01 | 武汉烽火信息集成技术有限公司 | Micro-expression recognition method of pixel displacement vector based on convolutional neural network |
| CN114333002A (en) * | 2021-12-27 | 2022-04-12 | 南京邮电大学 | Micro-expression recognition method based on deep learning of image and three-dimensional reconstruction of human face |
| CN114913461A (en) * | 2022-05-19 | 2022-08-16 | 广东电网有限责任公司 | Emotion recognition method and device, terminal equipment and computer readable storage medium |
| CN115661850A (en) * | 2022-12-29 | 2023-01-31 | 武汉大学 | Seal identification method integrating multiple characteristics |
| CN116311472A (en) * | 2023-04-07 | 2023-06-23 | 湖南工商大学 | Micro-expression recognition method and device based on multi-level graph convolution network |
| CN116824280A (en) * | 2023-08-30 | 2023-09-29 | 安徽爱学堂教育科技有限公司 | Psychological early warning method based on micro-expression change |
| CN116825365A (en) * | 2023-08-30 | 2023-09-29 | 安徽爱学堂教育科技有限公司 | Mental health analysis method based on multi-angle micro-expression |
| CN117392727A (en) * | 2023-11-02 | 2024-01-12 | 长春理工大学 | Facial micro-expression recognition method based on contrast learning and feature decoupling |
| CN118015686A (en) * | 2024-04-10 | 2024-05-10 | 济南超级计算技术研究院 | Emotion recognition device and method based on full-simulation photonic neural network |
| CN118314617A (en) * | 2024-06-11 | 2024-07-09 | 东北大学 | Micro-expression recognition method based on segmented multi-scale convolutional neural network |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110516571A (en) * | 2019-08-16 | 2019-11-29 | 东南大学 | Inter-library micro- expression recognition method and device based on light stream attention neural network |
| AU2020102556A4 (en) * | 2020-10-01 | 2020-11-19 | Ci, Yuming Mr | Psychological state analysis method based on facial micro-expression |
| CN112115796A (en) * | 2020-08-21 | 2020-12-22 | 西北大学 | Attention mechanism-based three-dimensional convolution micro-expression recognition algorithm |
-
2021
- 2021-01-20 CN CN202110073665.5A patent/CN112766159A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110516571A (en) * | 2019-08-16 | 2019-11-29 | 东南大学 | Inter-library micro- expression recognition method and device based on light stream attention neural network |
| CN112115796A (en) * | 2020-08-21 | 2020-12-22 | 西北大学 | Attention mechanism-based three-dimensional convolution micro-expression recognition algorithm |
| AU2020102556A4 (en) * | 2020-10-01 | 2020-11-19 | Ci, Yuming Mr | Psychological state analysis method based on facial micro-expression |
Non-Patent Citations (4)
| Title |
|---|
| RIZWAN CHAUDHRY等: "Histograms of Oriented Optical Flow and Binet-Cauchy Kernels on Nonlinear Dynamical Systems for the Recognition of Human Actions", 《2009 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION》 * |
| 梁正友 等: "一种用于微表情自动识别的三维卷积神经网络进化方法", 《计算机科学》 * |
| 苏育挺 等: "基于多运动特征融合的微表情识别算法", 《激光与光电子学进展》 * |
| 赵突: "基于视频的人脸微表情识别关键技术研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113361415A (en) * | 2021-06-08 | 2021-09-07 | 浙江工商大学 | Micro-expression data set collection method based on crowdsourcing |
| CN113537008A (en) * | 2021-07-02 | 2021-10-22 | 江南大学 | Micro-expression identification method based on adaptive motion amplification and convolutional neural network |
| CN113537008B (en) * | 2021-07-02 | 2024-03-29 | 江南大学 | Micro expression recognition method based on self-adaptive motion amplification and convolutional neural network |
| CN113469153A (en) * | 2021-09-03 | 2021-10-01 | 中国科学院自动化研究所 | Multi-modal emotion recognition method based on micro-expressions, limb actions and voice |
| CN113469153B (en) * | 2021-09-03 | 2022-01-11 | 中国科学院自动化研究所 | Multi-modal emotion recognition method based on micro-expressions, limb actions and voice |
| CN113901895A (en) * | 2021-09-18 | 2022-01-07 | 武汉未来幻影科技有限公司 | Door opening action recognition method and device for vehicle and processing equipment |
| CN113901895B (en) * | 2021-09-18 | 2022-09-27 | 武汉未来幻影科技有限公司 | Door opening action recognition method and device for vehicle and processing equipment |
| CN113902774A (en) * | 2021-10-08 | 2022-01-07 | 无锡锡商银行股份有限公司 | Method for detecting facial expression of dense optical flow characteristics in video |
| CN113902774B (en) * | 2021-10-08 | 2024-04-02 | 无锡锡商银行股份有限公司 | Facial expression detection method of thick and dense optical flow characteristics in video |
| CN114005157A (en) * | 2021-10-15 | 2022-02-01 | 武汉烽火信息集成技术有限公司 | Micro-expression recognition method of pixel displacement vector based on convolutional neural network |
| CN114005157B (en) * | 2021-10-15 | 2024-05-10 | 武汉烽火信息集成技术有限公司 | Micro-expression recognition method for pixel displacement vector based on convolutional neural network |
| CN113920571A (en) * | 2021-11-06 | 2022-01-11 | 北京九州安华信息安全技术有限公司 | Micro-expression identification method and device based on multi-motion feature fusion |
| CN114333002A (en) * | 2021-12-27 | 2022-04-12 | 南京邮电大学 | Micro-expression recognition method based on deep learning of image and three-dimensional reconstruction of human face |
| CN114913461A (en) * | 2022-05-19 | 2022-08-16 | 广东电网有限责任公司 | Emotion recognition method and device, terminal equipment and computer readable storage medium |
| CN115661850A (en) * | 2022-12-29 | 2023-01-31 | 武汉大学 | Seal identification method integrating multiple characteristics |
| CN116311472A (en) * | 2023-04-07 | 2023-06-23 | 湖南工商大学 | Micro-expression recognition method and device based on multi-level graph convolution network |
| CN116311472B (en) * | 2023-04-07 | 2023-10-31 | 湖南工商大学 | Micro-expression recognition method and device based on multi-level graph convolution network |
| CN116824280B (en) * | 2023-08-30 | 2023-11-24 | 安徽爱学堂教育科技有限公司 | Psychological early warning method based on micro-expression change |
| CN116825365B (en) * | 2023-08-30 | 2023-11-28 | 安徽爱学堂教育科技有限公司 | Mental health analysis method based on multi-angle micro-expression |
| CN116825365A (en) * | 2023-08-30 | 2023-09-29 | 安徽爱学堂教育科技有限公司 | Mental health analysis method based on multi-angle micro-expression |
| CN116824280A (en) * | 2023-08-30 | 2023-09-29 | 安徽爱学堂教育科技有限公司 | Psychological early warning method based on micro-expression change |
| CN117392727A (en) * | 2023-11-02 | 2024-01-12 | 长春理工大学 | Facial micro-expression recognition method based on contrast learning and feature decoupling |
| CN117392727B (en) * | 2023-11-02 | 2024-04-12 | 长春理工大学 | Facial micro-expression recognition method based on contrast learning and feature decoupling |
| CN118015686A (en) * | 2024-04-10 | 2024-05-10 | 济南超级计算技术研究院 | Emotion recognition device and method based on full-simulation photonic neural network |
| CN118314617A (en) * | 2024-06-11 | 2024-07-09 | 东北大学 | Micro-expression recognition method based on segmented multi-scale convolutional neural network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112766159A (en) | Cross-database micro-expression identification method based on multi-feature fusion | |
| Jain et al. | Extended deep neural network for facial emotion recognition | |
| Mahmood et al. | WHITE STAG model: Wise human interaction tracking and estimation (WHITE) using spatio-temporal and angular-geometric (STAG) descriptors | |
| CN110427867B (en) | Facial expression recognition method and system based on residual attention mechanism | |
| Happy et al. | Fuzzy histogram of optical flow orientations for micro-expression recognition | |
| CN106960202B (en) | Smiling face identification method based on visible light and infrared image fusion | |
| Youssif et al. | Arabic sign language (arsl) recognition system using hmm | |
| CN105574510A (en) | Gait identification method and device | |
| CN110796101A (en) | Face recognition method and system of embedded platform | |
| Maruyama et al. | Word-level sign language recognition with multi-stream neural networks focusing on local regions | |
| CN111666845A (en) | Small sample deep learning multi-mode sign language recognition method based on key frame sampling | |
| Paul et al. | Extraction of facial feature points using cumulative histogram | |
| Littlewort et al. | Fully automatic coding of basic expressions from video | |
| CN114550270A (en) | Micro-expression identification method based on double-attention machine system | |
| Chen et al. | A multi-scale fusion convolutional neural network for face detection | |
| Assiri et al. | Face emotion recognition based on infrared thermal imagery by applying machine learning and parallelism | |
| Mosayyebi et al. | Gender recognition in masked facial images using EfficientNet and transfer learning approach | |
| Elbarawy et al. | Facial expressions recognition in thermal images based on deep learning techniques | |
| CN117636436A (en) | Multi-person real-time facial expression recognition method and system based on attention mechanism | |
| Sajid et al. | Facial asymmetry-based feature extraction for different applications: a review complemented by new advances | |
| Kumar et al. | Facial emotion recognition and detection using cnn | |
| Jindal et al. | Sign Language Detection using Convolutional Neural Network (CNN) | |
| Mohan | Object detection in images by components | |
| Khubchandani et al. | Sign Language Recognition | |
| Bhardwaj et al. | A review: facial expression detection with its techniques and application |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210507 |