CN111950373A - Method for recognizing micro-expressions through transfer learning based on optical flow input - Google Patents
Method for recognizing micro-expressions through transfer learning based on optical flow input Download PDFInfo
- Publication number
- CN111950373A CN111950373A CN202010666988.0A CN202010666988A CN111950373A CN 111950373 A CN111950373 A CN 111950373A CN 202010666988 A CN202010666988 A CN 202010666988A CN 111950373 A CN111950373 A CN 111950373A
- Authority
- CN
- China
- Prior art keywords
- expression
- micro
- optical flow
- layer
- transfer learning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000003287 optical effect Effects 0.000 title claims abstract description 40
- 238000013526 transfer learning Methods 0.000 title claims abstract description 27
- 238000000034 method Methods 0.000 title claims abstract description 19
- 230000014509 gene expression Effects 0.000 claims abstract description 74
- 230000006870 function Effects 0.000 claims abstract description 19
- 238000012549 training Methods 0.000 claims abstract description 17
- 230000008921 facial expression Effects 0.000 claims abstract description 12
- 238000010606 normalization Methods 0.000 claims description 13
- 239000013598 vector Substances 0.000 claims description 12
- 230000002123 temporal effect Effects 0.000 claims description 11
- 230000001815 facial effect Effects 0.000 claims description 8
- 230000004913 activation Effects 0.000 claims description 3
- 238000011176 pooling Methods 0.000 claims description 3
- 230000009466 transformation Effects 0.000 claims description 3
- 238000013527 convolutional neural network Methods 0.000 description 13
- 238000013135 deep learning Methods 0.000 description 3
- 230000008451 emotion Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000005034 decoration Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000007935 neutral effect Effects 0.000 description 2
- 238000003759 clinical diagnosis Methods 0.000 description 1
- 238000011840 criminal investigation Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a method for recognizing micro-expressions by transfer learning based on optical flow input, which comprises the following steps: step 1: downloading a micro expression data set, and aligning and normalizing the micro expression data set; step 2: calculating the optical flow estimation of each micro-expression video in the micro-expression data set processed in the step 1 to obtain an optical flow sequence; and step 3: and (3) performing transfer learning from macro expression to micro expression by using a CNN model based on facial expression, inputting the optical flow sequence obtained in the step (2), outputting the optical flow sequence as a time-space characteristic, and training a network to finally realize a micro expression recognition function. The method solves the problem of overfitting caused by small data scale in micro expression recognition, and meanwhile, the optical flow has higher hierarchical characteristics than the original data, so that the performance of the model is further improved.
Description
Technical Field
The invention belongs to the field of computer vision, and particularly relates to a micro-expression recognition method realized by deep learning.
Background
Micro-expression recognition is a challenging task in the field of computer vision because it has the characteristics of suppressed facial expression and short duration. In recent years, due to the potential application in many fields such as clinical diagnosis, criminal investigation, safety systems and the like, the experts in different disciplines are receiving more and more attention. Micro-expression is a special facial expression defined as a rapid facial action that is not controlled by human consciousness, can represent a real emotion, and can reveal the real emotion that a person is trying to hide. It is possible to deceive others with facial expressions, but micro-expressions are not possible.
Although micro-expressions are visually similar to facial expressions, micro-expressions are short and suppressed, which makes micro-expression recognition more challenging than facial expression recognition. The seven categories of micro-expressions were trained using the micro-expression training tool (METT) in a psychological experiment with an average micro-expression recognition rate of 50%. The development of deep learning makes it possible to learn hierarchical features from multiple labeled images using Convolutional Neural Networks (CNNs). However, deep learning requires a large amount of data, and it is not feasible to train a Convolutional Neural Network (CNN) model directly from micro-expression data due to the lack of micro-expression data. According to the problem of small scale.
Disclosure of Invention
The invention aims to provide a method for micro expression recognition based on optical flow input transfer learning, which aims to solve the over-fitting problem caused by small data scale of micro expression recognition directly performed by a micro expression database
In order to achieve the purpose, the invention adopts the technical scheme that:
a method for micro-expression recognition based on optical flow input transfer learning, comprising the steps of:
step 1: downloading a micro expression data set, and aligning and normalizing the micro expression data set;
step 2: calculating the optical flow estimation of each micro-expression video in the micro-expression data set processed in the step 1 to obtain an optical flow sequence;
and step 3: and (3) performing transfer learning from macro expression to micro expression by using a CNN model based on facial expression, inputting the optical flow sequence obtained in the step (2), outputting the optical flow sequence as a time-space characteristic, and training a network to finally realize a micro expression recognition function.
In the step 1, 68 facial markers are detected in the first frame of each micro expression video in the micro expression data set by using the active shape model during alignment, then the first frame of each micro expression video is normalized according to a comparison template, and the subsequent frames in each micro expression video are compared with the first frame through local weighted mean transformation; normalization includes spatial domain normalization, which cuts all images into 96 × 112 pixels in the face region, and temporal domain normalization, which uses a linear interpolation method to obtain a sufficient number of frames.
In step 2, in a segment of micro-expression video, the value of the point (x, y, t) is I (x, y, t), the time interval t reaches the next frame, the pixel moves by (x + x, y + y, t + t), the intensity is I (x + x, y + y, t + t), and the intensity is obtained based on the invariance of the brightness in the small period
I(x,y,t)＝I(x+x,y+y,t+t) (1)
Where x ═ ut, y ═ vt, u (x, y) and v (x, y) are the horizontal and vertical components to be estimated in the optical flow field, and assuming that the pixel values in the micro-expression video are continuous functions of their positions and times, the expansion is based on taylor series, the right part of the above function is represented as:
wherein, the unbiased estimation quantity of the second order and above time t, when t tends to be infinitesimal small, the two sides of the equation (2) are divided by the time t and the equation (1), and then the optical flow equation is obtained as follows:
that is to say that the first and second electrodes,
in the step 3, a network for transfer learning from the macro expression to the micro expression is designed to realize the micro expression recognition function:
the source label space for transfer learning is:
yS＝{Neutral,Angry,Contempt,Disgust,Fear,Happy,Sad,Surprise}
the target label space is:
yT＝{Positive,Negative,Surprise,Others}
wherein, Positive is { Happy }, Negative is { Afraid, Angry, dispost, Sad, front }, surpriise is { surpride }, and the unclear facial movements belong to other;
the overall network structure is as follows:
input->conv_1->max-pool_1->conv_2->max-pool_2->conv_3->max-pool_3->fc_1->fc_2->lstm_1->lstm_2->lstm_3->fc_3->spatial_temporal feature
wherein, input is the optical flow sequence obtained in step 2, conv _ i { i ═ 1, 2, 3} represents the ith convolution layer, and batch normalization is adopted after the convolution operation of conv1 is removed; max-pool _ i { i ═ 1, 2, 3} represents the ith largest pooling layer; fc _ i { i ═ 1, 2, 3} represents the ith fully-connected layer, extracting spatial feature representations from the fc2 layer; LSTM _ i { i ═ 1, 2, 3} represents the ith LSTM layer; spatial _ temporal feature represents a space-time feature vector finally obtained through transfer learning; the output of each volume and full-connection layer adopts a ReLU nonlinear layer as an activation function to restrict the output; after the first and second fully connected layers, there is a dropout layer to mitigate overfitting to the feature vectors;
the objective function of learning the spatial feature representation is as follows:
wherein,
a true value representing the ith sample, 1 if k is the correct class, 0 otherwise,
representing the predicted probability of the expression category k calculated on the full connection layer; target item L1Enabling samples with different expression types to be separable in a functional space;
wherein f isc，p，iRepresenting a spatial feature representation vector of the ith training sample of the c types and the p expression state extracted from the last layer; m iscRepresenting a mean feature vector of the class c training samples;
when j ≠ c, mcAnd mjHalf of the minimum distance therebetween; target item L2Reducing the influence of intra-class changes in the same expression class caused by factors such as the appearance of a subject;
the operation of learning the LSTM layer of temporal feature representation is as follows:
gin，t (l)＝sigm(Win (l)[ht-1 (l)，ht (l-1)]+bin (l))，
gf，t (l)＝sigm(Wf (l)[ht-1 (l)，ht (l-1)]+bf (l))，
go，t (l)＝sigm(Wo (l)[ht-1 (l)，ht (l-1)]+bo (l))，
wherein, W* (l)And bx (l)Respectively representing the weight and the deviation of the first LSTM layer, wherein subscripts in, f, o and cell respectively represent input, forget, output and a memory unit; gin，t (l)The representation input gate determines how much input of the network at the current moment t is stored in the unit state; gf，t (l)The forgetting gate is shown, and the number of unit states at the previous moment is determined to be stored to the current moment t; go，t (l)The output gate is shown and determines how much of the unit state is output to the current output value of the LSTM; cellt (l)Representing the unit state input at the current time t; h ist (l)The output of the l-th LSTM layer given the t-th input is represented.
Has the advantages that: unlike the prior art, the present invention provides innovation in two ways. On one hand, network input is not a micro expression data sequence subjected to simple alignment and normalization processing, but optical flow estimation is carried out on an original micro expression data concentrated sequence, so that fine actions in a micro expression video are more obvious, and high-level features are obtained; on the other hand, the convolutional neural network can learn the space image characteristics of representative expression state frames, but the sample size of micro expression data is very small, and direct training is easy to overfit, so that the method adopts a CNN model based on facial expressions to perform transfer learning, uses pre-trained ImageNet _ CNN to train on a facial expression data set CK +, and the trained network contains some expression information shared with the micro expressions, and the information is transmitted to an LSTM training network to learn spatiotemporal characteristics. Therefore, the overfitting problem caused by small scale of micro-expression identification data directly carried out by using the micro-expression database is solved.
Detailed Description
The present invention is explained further below.
A method for micro-expression recognition based on optical flow input transfer learning, comprising the steps of:
step 1, downloading a micro expression data set, and aligning and normalizing the micro expression data set; and acquiring CASMEII as a data set of the training model. Detecting 68 facial markers in a first frame of each micro expression video in the micro expression data set by using an Active Shape Model (ASM) during alignment, normalizing the first frame of each micro expression video according to a comparison template, and comparing subsequent frames in each micro expression video with the first frame through Local Weighted Mean (LWM) transformation; normalization includes spatial domain normalization, which cuts all images into 96 × 112 pixels in the face region, and temporal domain normalization, which uses a linear interpolation method to obtain a sufficient number of frames.
Step 2: calculating the optical flow estimation of each micro-expression video in the micro-expression data set processed in the step 1 to obtain an optical flow sequence; in order to ensure that the proposed architecture is able to obtain high-level features, the data input to the network is not raw data, but optical flow, which has higher level features than the raw data, has proven to be effective in micro-expression recognition.
In a segment of micro-expression video, the value of the point (x, y, t) is I (x, y, t), the time interval t is to the next frame, the pixel moves by (x + x, y + y, t + t), the intensity is I (x + x, y + y, t + t), and the micro-expression video is obtained based on the invariance of the brightness in a small period
I(x，y，t)＝I(x+x，y+y，t+t) (1)
Where x ═ ut, y ═ vt, u (x, y) and v (x, y) are the horizontal and vertical components to be estimated in the optical flow field, and assuming that the pixel values in the micro-expression video are continuous functions of their positions and times, the expansion is based on taylor series, the right part of the above function is represented as:
wherein, the unbiased estimation quantity of the second order and above time t, when t tends to be infinitesimal small, the two sides of the equation (2) are divided by the time t and the equation (1), and then the optical flow equation is obtained as follows:
that is to say that the first and second electrodes,
and step 3: and (3) performing transfer learning from macro expression to micro expression by using a CNN model based on facial expression, inputting the optical flow sequence obtained in the step (2), outputting the optical flow sequence as a time-space characteristic, and training a network to finally realize a micro expression recognition function.
Expressions and micro-expressions have common knowledge that they are similar when expressing emotion and therefore have similar texture information. As dynamic facial movements, they have the same temporal pattern, onset period, apex period, and offset period. Similar texture and temporal patterns enable learning from expressions to micro-expressions.
The goal of transfer learning is to transfer knowledge between related source and target domains.
Designing a network for transfer learning from macro expression to micro expression to realize the micro expression recognition function:
the source label space for transfer learning is:
ys＝{Neutral，Angry，Contempt，Disgust，Fear，Happy，Sad，Surprise}
the target label space is:
yT＝{Positive，Negative，Surprise，Others}
wherein, Positive is { Happy }, Negative is { Afraid, Angry, dispost, Sad, front }, surpriise is { surpride }, and the unclear facial movements belong to other; thus, the source task and the target task are related, which further improves the performance of the migration learning.
Deep CNN requires a large number of annotated samples, and the sample size of micro-expression data is very small compared to existing expression data. Deep CNN does not guarantee good performance for such small size data. Thus, using pre-trained ImageNet _ CNN trained on the facial expression dataset CK +, the trained network contains some expression information shared with the micro-expressions, which is transmitted to the LSTM training network to learn spatiotemporal features.
The overall network structure is as follows:
input->conv_1->max-pool_1->conv_2->max-pool_2->conv_3->max-pool_3->fc_1->fc_2->lstm_1->lstm_2->lstm_3->fc_3->spatial_temporalfeature
wherein, input is the optical flow sequence obtained in step 2, conv _ i { i ═ 1, 2, 3} represents the ith convolution layer, and batch normalization is adopted after the convolution operation of conv1 is removed; max-pool _ i { i ═ 1, 2, 3} represents the ith largest pooling layer; fc _ i { i ═ 1, 2, 3} represents the ith fully-connected layer, and spatial feature representations are extracted from fc _2 layer; LSTM _ i { i ═ 1, 2, 3} represents the ith LSTM layer; spatial _ temporal feature represents a space-time feature vector finally obtained through transfer learning; the output of each volume and full-connection layer adopts a ReLU nonlinear layer as an activation function to restrict the output; after the first and second fully connected layers, there is a dropout layer to mitigate overfitting to the feature vectors;
the objective function of learning the spatial feature representation is as follows:
wherein,
a true value representing the ith sample, 1 if k is the correct class, 0 otherwise,
representing the predicted probability of the expression category k calculated on the full connection layer; target item L1Enabling samples with different expression types to be separable in a functional space;
wherein f isc，p，iRepresenting a spatial feature representation vector of the ith training sample of the c types and the p expression state extracted from the last layer; m iscRepresenting a mean feature vector of the class c training samples;
when j ≠ c, mcAnd mjHalf of the minimum distance therebetween; target item L2Reducing the influence of intra-class changes in the same expression class caused by factors such as the appearance of a subject;
the operation of learning the LSTM layer of temporal feature representation is as follows:
gin，t (l)＝sigm(Win (l)[ht-1 (l)，ht (l-1)]+bin (l))，
gf，t (l)＝sigm(Wf (l)[ht-1 (l)，ht (l-1)]+bf (l))，
go，t (l)＝sigm(Wo (l)[ht-1 (l)，ht (l-1)]+bo (l))，
wherein, W* (l)And b* (l)Respectively representing the weight and the deviation of the first LSTM layer, wherein subscripts in, f, o and cell respectively represent input, forget, output and a memory unit; gin，t (l)The representation input gate determines how much input of the network at the current moment t is stored in the unit state; gf，t (l)The forgetting gate is shown, and the number of unit states at the previous moment is determined to be stored to the current moment t; go，t (l)The output gate is shown and determines how much of the unit state is output to the current output value of the LSTM; cellt (l)Representing the unit state input at the current time t; h ist (l)The output of the l-th LSTM layer representing the given t-th input; .
The method carries out optical flow estimation on the video sequence in the original micro-expression data set, so that the subtle actions in the video are more obvious, and high-level features are obtained; in the aspect of network structure, the invention uses the transfer learning from macro expression to micro expression, uses the pre-trained deep CNN on ImageNet and uses the expression data of a large sample to pre-train the deep CNN, the trained network comprises some expression information shared with the micro expression, and the information is transmitted to the LSTM training network to carry out micro expression recognition, thereby accelerating the network training speed. The problem of creating an overfitting fit to such small size data and not guaranteeing good performance because the convolutional neural network requires a large number of annotated samples is solved.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (5)
1. A method for micro-expression recognition based on optical flow input transfer learning is characterized in that: the method comprises the following steps:
step 1: downloading a micro expression data set, and aligning and normalizing the micro expression data set;
step 2: calculating the optical flow estimation of each micro-expression video in the micro-expression data set processed in the step 1 to obtain an optical flow sequence;
and step 3: and (3) performing transfer learning from macro expression to micro expression by using a CNN model based on facial expression, inputting the optical flow sequence obtained in the step (2), outputting the optical flow sequence as a time-space characteristic, and training a network to finally realize a micro expression recognition function.
2. The method for micro-expression recognition based on optical flow input transfer learning of claim 1, wherein: in the step 1, 68 facial markers are detected in the first frame of each micro expression video in the micro expression data set by using the active shape model during alignment, then the first frame of each micro expression video is normalized according to a comparison template, and the subsequent frames in each micro expression video are compared with the first frame through local weighted mean transformation; normalization includes spatial domain normalization, which cuts all images into 96 × 112 pixels in the face region, and temporal domain normalization, which uses a linear interpolation method to obtain a sufficient number of frames.
3. The method for micro-expression recognition based on optical flow input transfer learning according to claim 1 or 2, wherein: in the step 1, the micro-expression dataset is CASMEII.
4. The method for micro-expression recognition based on optical flow input transfer learning of claim 1, wherein: in step 2, in a segment of micro-expression video, the value of the point (x, y, t) is I (x, y, t), the time interval t is to the next frame, the pixel moves by (x + x, y + y, t + t), the intensity is I (x + x, y + y, t + t), and based on the invariance of the brightness in the small period, the following results are obtained:
I(x，y，t)＝I(x+x，y+y，t+t) (1)
where x ═ ut, y ═ vt, u (x, y) and v (x, y) are the horizontal and vertical components to be estimated in the optical flow field, and assuming that the pixel values in the micro-expression video are continuous functions of their positions and times, the expansion is based on taylor series, the right part of the above function is represented as:
wherein, the unbiased estimation quantity of the second order and above time t, when t tends to be infinitesimal small, the two sides of the equation (2) are divided by the time t and the equation (1), and then the optical flow equation is obtained as follows:
that is to say that the first and second electrodes,
5. the method for micro-expression recognition based on optical flow input transfer learning of claim 1, wherein: in the step 3, a network for transfer learning from the macro expression to the micro expression is designed to realize the micro expression recognition function:
the source label space for transfer learning is:
the target label space is:
wherein, Positive is { Happy }, Negative is { Afraid, Angry, dispost, Sad, front }, surpriise is { surpride }, and the unclear facial movements belong to other;
the overall network structure is as follows:
input-＞conv_1-＞max-pool_1-＞conv_2-＞max-pool_2-＞conv_3-＞max-pool_3-＞fc_1-＞fc_2-＞lstm_1-＞lstm_2-＞lstm_3-＞fc_3-＞spatial_temporal feature
wherein, input is the optical flow sequence obtained in step 2, conv _ i { i ═ 1, 2, 3} represents the ith convolution layer, and batch normalization is adopted after the convolution operation of conv1 is removed; max-pool _ i { i ═ 1, 2, 3} represents the ith largest pooling layer; fc _ i { i ═ 1, 2, 3} represents the ith fully-connected layer, and spatial feature representations are extracted from fc _2 layer; LSTM _ i { i ═ 1, 2, 3} represents the ith LSTM layer; spatial _ temporal feature represents a space-time feature vector finally obtained through transfer learning; the output of each volume and full-connection layer adopts a ReLU nonlinear layer as an activation function to restrict the output; after the first and second fully connected layers, there is a dropout layer to mitigate overfitting to the feature vectors;
the objective function of learning the spatial feature representation is as follows:
wherein,
a true value representing the ith sample, 1 if k is the correct class, 0 otherwise,
representing the predicted probability of the expression category k calculated on the full connection layer; target item L1Enabling samples with different expression types to be separable in a functional space;
wherein f isc，p，iRepresenting a spatial feature representation vector of the ith training sample of the c types and the p expression state extracted from the last layer; m iscRepresenting a mean feature vector of the class c training samples;
when j ≠ c, mcAnd mjHalf of the minimum distance therebetween; target item L2The influence of intra-class change in the same expression class caused by factors such as the appearance of a subject is reduced;
the operation of learning the LSTM layer of temporal feature representation is as follows:
gin，t (l)＝sigm(Win (l)[ht-1 (l)，ht (l-1)]+bin (l))，
gf，t (l)＝sigm(Wf (l)[ht-1 (l)，ht (l-1)]+bf (l))，
go，t (l)＝sigm(Wo (l)[ht-1 (l)，ht (l-1)]+bo (l))，
wherein, W* (l)And b* (l)Respectively representing the weight and the deviation of the first LSTM layer, wherein subscripts in, f, o and cell respectively represent input, forget, output and a memory unit; gin，t (l)The representation input gate determines how much input of the network at the current moment t is stored in the unit state; gf，t (l)The forgetting gate is shown, and the number of unit states at the previous moment is determined to be stored to the current moment t; go，t (l)The output gate is shown and determines how much of the unit state is output to the current output value of the LSTM; cellt (l)Representing the unit state input at the current time t; h ist (l)The output of the l-th LSTM layer given the t-th input is represented.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010666988.0A CN111950373B (en) | 2020-07-13 | 2020-07-13 | Method for micro expression recognition based on transfer learning of optical flow input |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010666988.0A CN111950373B (en) | 2020-07-13 | 2020-07-13 | Method for micro expression recognition based on transfer learning of optical flow input |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111950373A true CN111950373A (en) | 2020-11-17 |
| CN111950373B CN111950373B (en) | 2024-04-16 |
Family
ID=73340475
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010666988.0A Active CN111950373B (en) | 2020-07-13 | 2020-07-13 | Method for micro expression recognition based on transfer learning of optical flow input |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111950373B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112381036A (en) * | 2020-11-26 | 2021-02-19 | 厦门大学 | Micro expression and macro expression fragment identification method applied to criminal investigation |
| CN113870315A (en) * | 2021-10-18 | 2021-12-31 | 南京硅基智能科技有限公司 | Training method of action migration model and action migration method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107194347A (en) * | 2017-05-19 | 2017-09-22 | 深圳市唯特视科技有限公司 | A kind of method that micro- expression detection is carried out based on Facial Action Coding System |
| CN108710829A (en) * | 2018-04-19 | 2018-10-26 | 北京红云智胜科技有限公司 | A method of the expression classification based on deep learning and the detection of micro- expression |
| CN110287805A (en) * | 2019-05-31 | 2019-09-27 | 东南大学 | Micro- expression recognition method and system based on three stream convolutional neural networks |
| CN110348271A (en) * | 2018-04-04 | 2019-10-18 | 山东大学 | A kind of micro- expression recognition method based on long memory network in short-term |
-
2020
- 2020-07-13 CN CN202010666988.0A patent/CN111950373B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107194347A (en) * | 2017-05-19 | 2017-09-22 | 深圳市唯特视科技有限公司 | A kind of method that micro- expression detection is carried out based on Facial Action Coding System |
| CN110348271A (en) * | 2018-04-04 | 2019-10-18 | 山东大学 | A kind of micro- expression recognition method based on long memory network in short-term |
| CN108710829A (en) * | 2018-04-19 | 2018-10-26 | 北京红云智胜科技有限公司 | A method of the expression classification based on deep learning and the detection of micro- expression |
| CN110287805A (en) * | 2019-05-31 | 2019-09-27 | 东南大学 | Micro- expression recognition method and system based on three stream convolutional neural networks |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112381036A (en) * | 2020-11-26 | 2021-02-19 | 厦门大学 | Micro expression and macro expression fragment identification method applied to criminal investigation |
| CN113870315A (en) * | 2021-10-18 | 2021-12-31 | 南京硅基智能科技有限公司 | Training method of action migration model and action migration method |
| CN113870315B (en) * | 2021-10-18 | 2023-08-25 | 南京硅基智能科技有限公司 | Multi-algorithm integration-based action migration model training method and action migration method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111950373B (en) | 2024-04-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Wang et al. | MESNet: A convolutional neural network for spotting multi-scale micro-expression intervals in long videos | |
| CN113496217B (en) | Method for identifying human face micro expression in video image sequence | |
| Shan et al. | Automatic facial expression recognition based on a deep convolutional-neural-network structure | |
| CN107609460B (en) | Human body behavior recognition method integrating space-time dual network flow and attention mechanism | |
| Wang et al. | Trajectory predictor by using recurrent neural networks in visual tracking | |
| CN106919903B (en) | robust continuous emotion tracking method based on deep learning | |
| CN112800903B (en) | Dynamic expression recognition method and system based on space-time diagram convolutional neural network | |
| CN112307958A (en) | Micro-expression identification method based on spatiotemporal appearance movement attention network | |
| CN112560810B (en) | Micro-expression recognition method based on multi-scale space-time characteristic neural network | |
| CN110399821B (en) | Customer satisfaction acquisition method based on facial expression recognition | |
| CN107679526A (en) | A kind of micro- expression recognition method of face | |
| CN108509839A (en) | One kind being based on the efficient gestures detection recognition methods of region convolutional neural networks | |
| CN112464865A (en) | Facial expression recognition method based on pixel and geometric mixed features | |
| Borgalli et al. | Deep learning for facial emotion recognition using custom CNN architecture | |
| Xu et al. | Face expression recognition based on convolutional neural network | |
| CN112597873A (en) | Dual-channel facial expression recognition method based on deep learning | |
| CN111950373B (en) | Method for micro expression recognition based on transfer learning of optical flow input | |
| Podder et al. | Time efficient real time facial expression recognition with CNN and transfer learning | |
| Seyedarabi et al. | Recognition of six basic facial expressions by feature-points tracking using RBF neural network and fuzzy inference system | |
| Nie | Research on facial expression recognition of robot based on CNN convolution neural network | |
| CN113723287B (en) | Micro-expression recognition method, device and medium based on bidirectional circulating neural network | |
| Hassan et al. | Enhanced dynamic sign language recognition using slowfast networks | |
| Liu et al. | An optimized Capsule-LSTM model for facial expression recognition with video sequences | |
| Dembani et al. | UNSUPERVISED FACIAL EXPRESSION DETECTION USING GENETIC ALGORITHM. | |
| CN115359522A (en) | Elderly health monitoring method and system based on expression emotion calculation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |