CN109190582B - Novel micro-expression recognition method - Google Patents
Novel micro-expression recognition method Download PDFInfo
- Publication number
- CN109190582B CN109190582B CN201811085510.8A CN201811085510A CN109190582B CN 109190582 B CN109190582 B CN 109190582B CN 201811085510 A CN201811085510 A CN 201811085510A CN 109190582 B CN109190582 B CN 109190582B
- Authority
- CN
- China
- Prior art keywords
- frame
- expression
- channel value
- pixel
- classification result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000000034 method Methods 0.000 title claims abstract description 64
- 230000014509 gene expression Effects 0.000 claims abstract description 267
- 238000012549 training Methods 0.000 claims description 25
- 238000004364 calculation method Methods 0.000 claims description 18
- 230000009467 reduction Effects 0.000 claims description 16
- 238000012545 processing Methods 0.000 claims description 15
- 238000013389 whole blood assay Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 description 21
- 238000002790 cross-validation Methods 0.000 description 6
- 238000012706 support-vector machine Methods 0.000 description 5
- 230000008921 facial expression Effects 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 239000013598 vector Substances 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000006698 induction Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000036651 mood Effects 0.000 description 2
- OUXCBPLFCPMLQZ-WOPPDYDQSA-N 4-amino-1-[(2r,3s,4s,5r)-4-hydroxy-5-(hydroxymethyl)-3-methyloxolan-2-yl]-5-iodopyrimidin-2-one Chemical compound C[C@H]1[C@H](O)[C@@H](CO)O[C@H]1N1C(=O)N=C(N)C(I)=C1 OUXCBPLFCPMLQZ-WOPPDYDQSA-N 0.000 description 1
- 206010011469 Crying Diseases 0.000 description 1
- 206010027951 Mood swings Diseases 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 150000001875 compounds Chemical class 0.000 description 1
- 230000008602 contraction Effects 0.000 description 1
- 230000000994 depressogenic effect Effects 0.000 description 1
- 230000008451 emotion Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000001815 facial effect Effects 0.000 description 1
- 210000001097 facial muscle Anatomy 0.000 description 1
- 230000005764 inhibitory process Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001629 suppression Effects 0.000 description 1
- 230000001755 vocal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Human Computer Interaction (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The application provides a micro-expression recognition method, which comprises the following steps: extracting each frame of the face video; sequentially comparing the difference between each frame except the first frame and the last frame and the previous frame; sequentially determining the difference between the difference of each frame except the first frame and the last frame and the difference between the frame and the previous frame as the difference value of the frame; determining a micro-expression frame in each frame except the first frame and the last frame; and extracting the expression characteristics of the micro expression frame through a pre-trained micro expression recognition model, reducing the dimension of the expression characteristics, and recognizing the reduced dimension characteristics to obtain a recognition result. According to the method, the difference between each frame and the next frame and the difference between each frame and the previous frame are compared to obtain the difference value of the frame, and the micro expression frame is determined according to the difference value of each frame.
Description
Technical Field
The application relates to the technical field of artificial intelligence, in particular to a micro-expression identification method.
Background
Micro-expressions are non-verbal behaviors that can reveal a person's own emotions.
Most of the current concerns are mainly common expressions, and besides the common facial expressions, the facial expressions also have micro-expressions generated by uncontrolled contraction of facial muscles in a psychological inhibition state.
The duration of micro expression is short and the amplitude of action is very small. There is considerable difficulty in correctly observing and identifying. The success rate of accurately capturing and identifying the micro-expressions with the naked eye is low. After professional training, the recognition rate can only reach 47%.
Therefore, the recognition method of micro-expressions is receiving increasing attention from researchers.
Disclosure of Invention
In order to solve the above problem, an embodiment of the present application provides a micro expression recognition method.
The method comprises the following steps:
acquiring a face video;
extracting each frame of the face video;
sequentially comparing the difference between each frame except the first frame and the last frame and the previous frame;
sequentially determining the difference between the difference of each frame except the first frame and the last frame and the difference between the frame and the previous frame as the difference value of the frame;
selecting frames with difference values of which the absolute values are not 0 and are smaller than a first preset threshold value from the frames except the first frame and the last frame;
determining the mark of each selected frame;
determining frames with continuous marks as micro-expression frames in the selected frames;
and extracting the expression characteristics of the micro expression frame through a pre-trained micro expression recognition model, reducing the dimension of the expression characteristics, and recognizing the reduced dimension characteristics to obtain a recognition result.
Optionally, the sequentially comparing differences between each frame except the first frame and the last frame and the next frame thereof includes:
for any frame i other than the first and last frames,
acquiring a red channel value, a green channel value and a blue channel value of each pixel in any frame i;
acquiring a red channel value, a green channel value and a blue channel value of each pixel in a frame i +1 subsequent to the frame i;
determining the difference between any frame i and the subsequent frame i +1 according to the red channel value, the green channel value and the blue channel value of each pixel in any frame i, and the red channel value, the green channel value and the blue channel value of each pixel in the subsequent frame i +1 of any frame i;
the sequentially comparing the difference between each frame except the first frame and the last frame and the previous frame comprises:
for any frame i except the first frame and the last frame;
acquiring a red channel value, a green channel value and a blue channel value of each pixel in any frame i;
acquiring a red channel value, a green channel value and a blue channel value of each pixel in a previous frame i-1 of any frame i;
and determining the difference between any frame i and the previous frame i-1 thereof according to the red channel value, the green channel value and the blue channel value of each pixel in any frame i and the red channel value, the green channel value and the blue channel value of each pixel in the previous frame i-1 of any frame i.
Optionally, the determining a difference between the any frame i and the subsequent frame i +1 according to the red channel value, the green channel value, and the blue channel value of each pixel in the subsequent frame i +1 of the any frame i, and the red channel value, the green channel value, and the blue channel value of each pixel in the subsequent frame i +1 of the any frame i, includes:
calculating the difference between any frame i and the subsequent frame i +1 by the following formula:
wherein,
is the difference between any frame i and the subsequent frame i +1, j is a pixel identifier, j is more than or equal to 1 and less than or equal to M, M is the total number of pixels of any frame of the face video, WRAs a red channel weight, WGAs a green channel weight, WBAs a blue channel weight, Ri,jIs the red channel value, G, of pixel j in said any frame ii,jIs the green channel value, B, of pixel j in said any frame ii,jIs the blue channel value, R, of the pixel j in said any frame ii+1,jIs the red channel value, G, of pixel j in the subsequent frame i +1i+1,jIs the green channel value, B, of pixel j in the subsequent frame i +1i+1,jIs the blue channel value of the pixel j in the next frame i + 1;
determining a difference between any frame i and a previous frame i-1 thereof according to the red channel value, the green channel value, and the blue channel value of each pixel in any frame i, and the red channel value, the green channel value, and the blue channel value of each pixel in a previous frame i-1 of any frame i, includes:
calculating the difference between any frame i and the previous frame i-1 by the following formula:
wherein,
is the difference between any frame i and its previous frame i-1, Ri-1,jIs the red channel value, G, of the pixel j in the frame i-1 preceding said any frame ii-1,jIs the green channel value, B, of a pixel j in a frame i-1 preceding said any frame ii-1,jIs the blue channel value of the pixel j in the frame i-1 preceding said any frame i.
Optionally, the WRThe determination method comprises the following steps:
in the face video, any frame comprising a face is obtained;
identifying a face region in any frame of the face;
determining a red channel value of the skin color of the human face according to the human face area;
WRred channel value/255 for face skin color;
the W isGThe determination method comprises the following steps:
in the face video, any frame comprising a face is obtained;
identifying a face region in any frame of the face;
determining a green channel value of the skin color of the human face according to the human face area;
WGgreen channel value/255 for face skin color;
the W isBThe determination method comprises the following steps:
in the face video, any frame comprising a face is obtained;
identifying a face region in any frame of the face;
determining a blue channel value of the skin color of the human face according to the human face area;
WBblue channel value/255 for face skin tone.
Optionally, before obtaining the recognition result, the method further includes:
training a micro expression recognition model;
the training micro-expression recognition model comprises:
acquiring a plurality of sample videos;
for each sample video, extracting corresponding expression features by adopting a local binary pattern;
performing dimensionality reduction processing according to the expression features of all sample videos;
and carrying out recognition training on each sample video based on the features after the dimension reduction processing to form a micro expression recognition model.
Optionally, the performing, according to the expression features of all sample videos, the dimension reduction processing includes:
determining the information increment of each expression characteristic according to the following formula:
determining the expression features with the information increment larger than a second preset threshold value as the features after dimension reduction processing;
wherein s is an expression feature identifier, and s is more than or equal to 1 and less than or equal to NA，NAFor the total number of expressive features of all sample videos, Gsis the information increment of the expression feature s, E () represents entropy, E(s) is the entropy of the expression feature s, v is the sample video identification containing the expression feature s, v is more than or equal to 1 and less than or equal to C, C is the total number of all sample videos containing the expression feature s, E (v) is the entropy of the sample video v containing the expression feature s, NvAnd obtaining the number of the expression features corresponding to the sample video v containing the expression features s.
Alternatively, the calculation formula of E () is:
E()＝-p+log2p+-p-log2p-；
wherein p is+To classify the probability of correctness, p-Is the probability of classification error.
Optionally, the calculation formula of e(s) is:
E(s)＝-p+(s)log2p+(s)-p-(s)log2p-(s)；
wherein p is+(s) probability of correct classification of expression features s, p-(s) is the probability of misclassification of the expression features s;
said p is+(s) and p-The calculation method of(s) is as follows:
s1-1, taking q as 0;
s1-2, sequentially selecting q expression features as first auxiliary expression features from the expression features except the expression feature S;
classifying each sample video based on the expression feature s and the first auxiliary expression feature to obtain a first classification result of each sample video after the first auxiliary expression feature is selected;
obtaining a first standard classification result of each sample video;
comparing the first classification result of each sample video with the first standard classification result, and determining the number N of the sample videos with the first classification result consistent with the first standard classification result+(s) number of sample videos N inconsistent with the first classification result and the first standard classification result-(s)；
Determining the probability of correctness
Determining error probability
S1-3, after the q expression features are sequentially selected, determining whether q +1 is N or notAIf q +1 is not NAThen S1-4 is executed if q +1 is NAThen execution of S1-5;
s1-4, q ═ q +1, S1-2 and S1-3 were repeatedly performed;
s1-5, mixing all
Is determined as p+(s) mixing all
Is determined as p-(s)。
Optionally, the calculation formula of e (v) is:
E(v)＝-p+(v)log2p+(v)-p-(v)log2p-(v)；
wherein p is+(v) Probability of correctly classifying v sample video, p-(v) Probability of classification error for sample video v;
said p is+(v) And p-(v) The calculation method comprises the following steps:
s2-1, taking y as 1;
s2-2, sequentially selecting y expression features as second auxiliary expression features from the expression features corresponding to the sample video v;
every time a second auxiliary expression feature is selected, x1 is x1+1, and the sample video v is classified based on the second auxiliary expression feature to obtain a second classification result of the sample video v; wherein x1 is a first counting mark, and x1 is an initial value of 0;
acquiring a second standard classification result of the sample video v;
comparing the second classification result with the second standard classification result, and if the second classification result is consistent with the second standard classification result, x2 is x2+ 1; if the second classification result is inconsistent with the second standard classification result, x3 is x3+ 1; wherein x2 is the second counting mark, x2 is 0 in initial value, x3 is the third counting mark, and x3 is 0 in initial value;
s2-3, after the q expression features are sequentially selected, determining whether y +1 is larger than N or notvIf y +1 is not greater than NvThen S2-4 is performed if y +1 is greater than NvThen execution of S2-5;
s2-4, y +1, repeating S2-2 and S2-3;
s2-5, determining the value of x2/x1 as p+(s) determining the value of x3/x1 as p-(s)。
The beneficial effects are as follows:
the difference between each frame and the next frame and the difference between the frame and the previous frame are compared to obtain the difference value of the frame, and the micro expression frame is determined according to the difference value of each frame.
Drawings
Specific embodiments of the present application will be described below with reference to the accompanying drawings, in which:
fig. 1 is a schematic diagram illustrating LBP descriptor calculation according to an embodiment of the present application;
FIG. 2 illustrates a schematic diagram of feature extraction provided by an embodiment of the present application;
fig. 3 is a schematic flow chart illustrating a micro expression recognition method according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions and advantages of the present application more apparent, the following further detailed description of the exemplary embodiments of the present application with reference to the accompanying drawings makes it clear that the described embodiments are only a part of the embodiments of the present application, and not an exhaustive list of all embodiments. And the embodiments and features of the embodiments in the present description may be combined with each other without conflict.
Due to the short duration of micro-expression and the very small amplitude of the motion. There is considerable difficulty in correctly observing and identifying. Based on the above, the application provides a micro expression frame identification method, the method compares the difference between each frame and the next frame and the difference between each frame and the previous frame to obtain the difference value of the frame, and the micro expression frame is determined according to the difference value of each frame.
The expression recognition method provided by the application comprises 2 major processes, wherein the first major process is a micro expression recognition model training process, and the other major process is an actual micro expression recognition process based on a trained micro expression recognition model.
The micro expression recognition model training process is not a process which is executed every time the expression recognition method provided by the application is executed, only when the expression recognition method provided by the application is executed for the first time, or the expression recognition scene changes, or the recognition result of the actual micro expression recognition process based on the trained micro expression recognition model is not ideal, or other reasons exist, the micro expression recognition model training process is executed, so that the accuracy of the micro expression recognition model is improved, and the accuracy of the result of the actual micro expression recognition process based on the trained micro expression recognition model is improved.
The method and the device do not limit the execution triggering conditions of the process of training the micro-expression recognition model.
The specific implementation method for the process of training the micro-expression recognition model is as follows:
The sample video may be obtained from an existing microexpression dataset.
Since micro-expressions are the tiny facial movements that a person makes when trying to mask his mood. In a strict sense, the micro expression that people subjectively simulate cannot be called as the micro expression, so the induction mode of the micro expression determines the reliability degree of the data.
This step may obtain multiple sample videos from one or 2 of the following 2 existing micro-expression datasets:
the micro-expression dataset SMIC, established by the university of orlu, finland, requires the subject to watch a video with large mood swings and to try to mask his mood from being exposed, and the recorder observes the subject's expression without watching the video. If the recorder observes the facial expression of the subject, the subject gets a penalty. Under the induction mechanism, 164 video sequences of 16 persons are formed, the micro-expression categories are 3, namely positive (positive), surprise (negative) and negative (negative), and the number of the video sequences is 70, 51 and 43 respectively.
The micro-expression data set CASME2, established by the psychological institute of Chinese academy of sciences, employs a similar inducement mechanism to ensure the reliability of the data, but is rewarded if the subject successfully suppresses his facial expression and is not discovered by the recorder. The data set is 5 micro-expression categories consisting of 247 video sequences of 26 individuals, respectively happy (happy), dislike (disgust), surprised (surrise), depressed (suppression), other (other), the number of video sequences 32, 64, 25, 27, 99.
And 2, for each sample video, extracting corresponding expression features by adopting a local binary pattern.
A Local Binary Pattern (LBP) descriptor is defined on a central pixel and its surrounding rectangular neighborhood, as shown in fig. 1, with the gray value of the central pixel as a threshold, Binary quantizing the neighborhood pixels around the central pixel, and the code greater than or equal to the central pixel value is 1, and if smaller, the code is 0, and a Local Binary Pattern is formed.
And (4) connecting the binary mode in series in a clockwise direction by taking the upper left corner as a starting point to obtain a string of binary digits, wherein the corresponding decimal digits can uniquely identify the central pixel point. In this way, each pixel in the image can be computed using a local binary pattern.
As shown in fig. 1, the center pixel value in the left table is 178, the upper left corner is 65, 65<178, so the corresponding value is 0, 188>178, and so the corresponding value is 1. By analogy, the table on the right side of fig. 1 is obtained, and then the binary pattern value is 01000100.
In addition, the extension of the LBP static texture descriptor in the time-space domain can also form 2-dimensional local binary patterns on 3 orthogonal planes. As shown in fig. 2, LBP features of video sequences in three orthogonal planes XY, XT, and YT are extracted, and feature vectors in each orthogonal plane are concatenated to form an LBP-TOP feature vector. The local texture information of the image is considered, and the change situation of the video along with the time is described.
However, the vector dimension of LBP-TOP is 3X 2LAnd L is the number of the field points. If the expression features extracted in the step 2 are directly used for modeling, the model training efficiency is low and the effect is poor due to large feature dimension. Therefore, after the expression features are extracted in the step 2, the step 3 is executed to reduce the dimension of the practical training model which is the considered expression features, and improve the training efficiency of the model.
And 3, performing dimension reduction processing according to the expression characteristics of all sample videos.
According to the method, the information increment of each expression feature is used as a dimension screening basis, the expression features of which the information increment is not larger than a second preset threshold are not considered, and the expression features of which the information increment is larger than the second preset threshold are determined as the features after dimension reduction processing.
It should be noted that, in the present application, since a preset threshold is also involved in a subsequent actual micro expression recognition process based on a trained micro expression recognition model, and there is no association relationship between the two preset thresholds, in order to distinguish the preset threshold from a subsequent actual micro expression recognition process based on the trained micro expression recognition model, the preset threshold in the micro expression recognition model training process is referred to as a second preset threshold, and the preset threshold in the actual micro expression recognition process based on the trained micro expression recognition model is referred to as a first preset threshold, where the first and second thresholds are only different stages of distinguishing, and do not have any actual significance.
Specifically, the information increment of each expression feature is determined according to the following formula:
and determining the expression features of which the information increment is larger than a second preset threshold value as the features after the dimension reduction processing.
Wherein s is an expression feature identifier, and s is more than or equal to 1 and less than or equal to NA，NAThe total number of the expressive features of all the sample videos (to improve the dimension reduction effect of the expressive features, the expressive features of all the sample videos here may be non-repetitive expressive features), GsIs the information increment of the expression feature s, E () represents entropy, E(s) is the entropy of the expression feature s, v is the sample video identification containing the expression feature s, v is more than or equal to 1 and less than or equal to C, C is the total number of all sample videos containing the expression feature s, E (v) is the entropy of the sample video v containing the expression feature s, NvAnd obtaining the number of the expression features corresponding to the sample video v containing the expression features s.
Wherein, the calculation formula of E () is: e () ═ p+log2p+-p-log2p-。
p+To classify the probability of correctness, p-Is the probability of classification error.
Based on the E () calculation formula, the entropy of the expressive features can be calculated, as well as the entropy of the sample video.
● for calculating the entropy of the expressive feature, taking the expressive feature s as an example, the calculation formula is:
E(s)＝-p+(s)log2p+(s)-p-(s)log2p-(s)；
wherein p is+(s) probability of correct classification of expression features s, p-(s) is the probability of misclassification of the expression features s;
p+(s) and p-The calculation method of(s) is as follows:
s1-1, taking q as 0;
s1-2, sequentially selecting q expression features as first auxiliary expression features from the expression features except the expression feature S;
classifying each sample video based on the expression feature s and the first auxiliary expression feature to obtain a first classification result of each sample video after the first auxiliary expression feature is selected;
obtaining a first standard classification result of each sample video;
comparing the first classification result of each sample video with the first standard classification result, and determining the number N of the sample videos with the first classification result consistent with the first standard classification result+(s) number of sample videos N inconsistent with the first classification result and the first standard classification result-(s)；
Determining the probability of correctness
Determining error probability
Wherein C is the total number of sample videos containing the expression feature s.
S1-3, after the q expression features are sequentially selected, determining whether q +1 is N or notAIf q +1 is not NAThen S1-4 is executed if q +1 is NAThen execution of S1-5;
s1-4, q ═ q +1, S1-2 and S1-3 were repeatedly performed;
s1-5, mixing all
Is determined as p+(s) mixing all
Is determined as p-(s)。
It should be noted that, since the present application further relates to an auxiliary expression feature, a classification result, and a standard classification result in the subsequent calculation of e (v) (the entropy of the sample video v including the expression feature s), there is no association between two auxiliary expression features, there is no association between classification results, and there is no association between two standard classification results. In order to distinguish the auxiliary expression features, the classification results and the standard classification results in the subsequent E (v) calculation process, the auxiliary expression features, the classification results and the standard classification results in the E(s) (entropy of the expression features s) calculation process are respectively called as first auxiliary expression features, first classification results and first standard classification results, the auxiliary expression features, the classification results and the standard classification results in the E (v) calculation process are respectively called as second auxiliary expression features, second classification results and second standard classification results, and the first and second features are only distinguishing functions and do not have any practical significance.
The implementation scheme is described by taking as an example that 3 sample videos are respectively a sample video 1 and a sample video 2, the total number of expression features of all the sample videos is 3, the expression features are respectively an expression feature 1, an expression feature 2 and an expression feature 3, the expression feature s is an expression feature 3, and both the sample video 1 and the sample video 2 contain the expression feature 3.
(1) Taking q as 0, sequentially selecting 0 expression features from the expression features (expression features 1 and 2) except the expression feature s (expression feature 3) as first auxiliary expression features (namely, not selecting other expression features).
At this time, each sample video (sample video 1, sample video 2) is classified based on the expression feature s (expression feature 3) only, and the first classification result of the sample video 1 and the first classification result of the sample video 2 are obtained.
And acquiring a first standard classification result of the sample video 1 and a first standard classification result of the sample video 2.
The first classification result of the sample video 1 is compared with the first standard classification result of the sample video 1, and the first classification result of the sample video 2 is compared with the first standard classification result of the sample video 2. If the first classification result of the sample video 1 is inconsistent with the first standard classification result of the sample video 1, the first classification result of the sample video 2 is inconsistent with the first standard classification result of the sample video 2. Determining the number of sample videos N for which the first classification result is consistent with the first standard classification result+(s) < 0 > and samples whose first classification result is inconsistent with the first standard classification resultNumber of videos N-(s)＝2。
Since both sample video 1 and sample video 2 contain expressive feature 3, C ═ 2.
Determining the probability of correctness
Determining error probability
After 0 expression features are sequentially selected, whether 0+1 is N or not is determinedA(NAThe total number of the expressive features of all the sample videos is 3 in this example), if 0+ 1-1 is not 3, then q-0 + 1-1, and enter (2).
(2) At this time, q is 1, and 1 of the expression features (expression feature 1, expression feature 2) other than the expression feature s (expression feature 3) is sequentially selected as the first auxiliary expression feature.
Selecting expression feature 1 as a first auxiliary expression feature
At this time, each sample video (sample video 1, sample video 2) is classified based on the expression feature s (expression feature 3) and the first auxiliary expression feature (expression feature 1), and a first classification result of the sample video 1 and a first classification result of the sample video 2 are obtained.
And acquiring a first standard classification result of the sample video 1 and a first standard classification result of the sample video 2.
The first classification result of the sample video 1 is compared with the first standard classification result of the sample video 1, and the first classification result of the sample video 2 is compared with the first standard classification result of the sample video 2. If the first classification result of the sample video 1 is consistent with the first standard classification result of the sample video 1, the first classification result of the sample video 2 is consistent with the first standard classification result of the sample video 2. Determining the number of sample videos N for which the first classification result is consistent with the first standard classification result+(s) — 2 and the number of sample videos N for which the first classification result and the first standard classification result do not agree-(s)＝0。
Determining the probability of correctness
Determining error probability
Selecting expression feature 2 as a first auxiliary expression feature
At this time, each sample video (sample video 1, sample video 2) is classified based on the expression feature s (expression feature 3) and the first auxiliary expression feature (expression feature 2), and a first classification result of the sample video 1 and a first classification result of the sample video 2 are obtained.
And acquiring a first standard classification result of the sample video 1 and a first standard classification result of the sample video 2.
The first classification result of the sample video 1 is compared with the first standard classification result of the sample video 1, and the first classification result of the sample video 2 is compared with the first standard classification result of the sample video 2. If the first classification result of the sample video 1 is consistent with the first standard classification result of the sample video 1, the first classification result of the sample video 2 is inconsistent with the first standard classification result of the sample video 2. Determining the number of sample videos N for which the first classification result is consistent with the first standard classification result+(s) ═ 1 and number of sample videos N for which the first classification result and the first standard classification result do not agree-(s)＝1。
Determining the probability of correctness
Determining error probability
When 1 expression feature is selected in sequence, if it is determined that 1+ 1-2 is not 3, then q is 1+ 1-2, and (3) is entered.
(3) At this time, q is 2, of the expression features (expression feature 1, expression feature 2) other than the expression feature s (expression feature 3), 2 expression features (only 2 remaining expression features) are sequentially selected as the first auxiliary expression feature.
At this time, the sample videos (sample video 1 and sample video 2) are classified based on the expression feature s (expression feature 3) and the first auxiliary expression features (expression feature 1 and expression feature 2), so that a first classification result of the sample video 1 and a first classification result of the sample video 2 are obtained.
And acquiring a first standard classification result of the sample video 1 and a first standard classification result of the sample video 2.
The first classification result of the sample video 1 is compared with the first standard classification result of the sample video 1, and the first classification result of the sample video 2 is compared with the first standard classification result of the sample video 2. If the first classification result of the sample video 1 is consistent with the first standard classification result of the sample video 1, the first classification result of the sample video 2 is consistent with the first standard classification result of the sample video 2. Determining the number of sample videos N for which the first classification result is consistent with the first standard classification result+(s) — 2 and the number of sample videos N for which the first classification result and the first standard classification result do not agree-(s)＝0。
Determining the probability of correctness
Determining error probability
When 2 expression features are sequentially selected, if 2+1 is determined to be 3, all the expression features are selected
Is determined as p+(s) mixing all
Is determined as p-(s)。
That is to say in (1)
(2) In (1)
(case) &
(case), (3)
Is determined as p+(s). Such as (0+1+0.5+ 1)/4-2.5/4-0.625.
The compound of (1)
(2) In (1)
(case) &
Case), (3) of
Is determined as p-(s). Such as (1+0+0.5+ 0)/4-1.5/4-0.375.
It should be noted that the above numbers are merely examples, and actual data is the standard in specific implementation.
● for calculating the entropy of the sample video, taking the sample video v as an example, the calculation formula is:
E(v)＝-p+(v)log2p+(v)-p-(v)log2p-(v)；
wherein p is+(v) Probability of correctly classifying v sample video, p-(v) Probability of classification error for sample video v;
p+(v) and p-(v) The calculation method comprises the following steps:
s2-1, taking y as 1;
s2-2, sequentially selecting y expression features as second auxiliary expression features from the expression features corresponding to the sample video v;
every time the second auxiliary expression feature is selected, x1 is x1+1, the sample video v is classified based on the second auxiliary expression feature, and a second classification result of the sample video v is obtained; wherein x1 is a first counting mark, and x1 is an initial value of 0;
acquiring a second standard classification result of the sample video v;
comparing the second classification result with the second standard classification result, and if the second classification result is consistent with the second standard classification result, x2 is x2+ 1; if the second classification result is inconsistent with the second standard classification result, x3 is x3+ 1; wherein x2 is the second counting mark, x2 is 0 in initial value, x3 is the third counting mark, and x3 is 0 in initial value;
s2-3, after the q expression features are sequentially selected, determining whether y +1 is larger than N or notvIf y +1 is not greater than NvThen S2-4 is performed if y +1 is greater than NvThen execution of S2-5;
s2-4, y +1, repeating S2-2 and S2-3;
s2-5, determining the value of x2/x1 as p+(s) determining the value of x3/x1 as p-(s)。
The implementation scheme is described by taking an example that the sample video v corresponds to 2 expression features, namely expression feature 1 and expression feature 2.
1) And taking y as 1, and sequentially selecting y (1) expression features as second auxiliary expression features from the expression features corresponding to the sample video v.
Selecting expression feature 1 as a first auxiliary expression feature
At this time, x1 is x1+1 (since x1 is the first count flag, and the initial value of x1 is 0, at this time, x1 is 0+ 1), and the sample video v is classified based on the second auxiliary expression feature (expression feature 1), so as to obtain a second classification result of the sample video v;
acquiring a second standard classification result of the sample video v;
comparing the second classification result with the second standard classification result, and if the second classification result is not consistent with the second standard classification result, x3 is equal to x3+1 (x 3 is equal to 0+1 because x3 is the third counter mark and the x3 initial value is 0).
Selecting expression feature 2 as a first auxiliary expression feature
At this time, x1 is 1+1 is 2, and the sample video v is classified based on the second auxiliary expression feature (expression feature 2), so as to obtain a second classification result of the sample video v;
acquiring a second standard classification result of the sample video v;
comparing the second classification result with the second standard classification result, and if the second classification result matches the second standard classification result, x2 is x2+1 (x 2 is 0+1 since x2 is the second count flag and x2 is the initial value 0).
After 1 expression feature is sequentially selected, whether y +1(1+1 ═ 2) is larger than N is determinedv(NvThe number of the expression features corresponding to the sample video v containing the expression features s is 2), 2 is not greater than 2, and then y +1+ 2 enters 2).
2) At this time, y is 2, and 2 expression features (only 2 expression features) are sequentially selected as the second auxiliary expression feature from the expression features corresponding to the sample video v. Namely, the expressive features 1 and 2 are selected as the first auxiliary expressive feature.
At this time, x1 is 2+1 is 3, and the sample video v is classified based on the second auxiliary expression features (expression features 1 and 2), so as to obtain a second classification result of the sample video v;
acquiring a second standard classification result of the sample video v;
and comparing the second classification result with the second standard classification result, and if the second classification result is consistent with the second standard classification result, x2 is 1+1 is 2.
When 2 expression features are sequentially selected, determining that 3+1 is 3 and 3>2, determining x2/x1 as p 2/3+(s) determining x3/x1 as p 2/3-(s)。
And 4, carrying out recognition training on each sample video based on the features after the dimension reduction processing to form a micro expression recognition model.
The training method in this step can be various, for example, any clustering algorithm is adopted to cluster each sample video based on the features after the dimension reduction processing, and the micro-expression class to which each sample video belongs is formed. And adjusting parameters in the clustering algorithm according to a second standard classification result of each sample video, and repeatedly clustering each sample video based on the characteristics after the dimension reduction processing to form a micro-expression class to which each sample video belongs. And then, according to the second standard classification result of each sample video, adjusting parameters in the clustering algorithm to complete training and form a micro expression recognition model.
The micro expression recognition model in the application is a classifier.
For example: a Support Vector Machine (SVM) method is employed. The key of the SVM is the kernel function, and different SVM classification results can be obtained by adopting different kernel functions.
For example, the following kernel functions may be employed: there are Linear kernels (Linear kernels), Chi-square kernels (Chi-square kernels), and Histogram Intersection kernels (Histogram Intersection kernels).
In addition, in order to improve the classification recognition rate of the finally trained classification model, Cross Validation (Cross Validation) can be adopted to verify the performance of the micro expression recognition model. Specifically, all sample videos are divided into two subsets, one subset for training the classifier is called a training set, and the other subset for verifying the effectiveness of the analysis classifier is called a test set. The classifier obtained by training is tested by using the test set, and the test set is used as the performance index of the classifier. Common methods are simple cross validation, K-fold cross validation and leave-one-cross validation.
The leave-one-out cross validation method is used for performing micro-expression classification training on SVM classifiers with different kernel functions. And (3) selecting all video sequences of one subject as a test sample and all video sequences of the other I subjects as training samples, repeating the experiment for I +1 times, and calculating the average classification recognition rate for I +1 times.
Based on the method, the training of the micro expression recognition model is completed, and the micro expression recognition model improves the recognition rate and the recognition accuracy rate of the micro expression recognition model due to the fact that dimension reduction processing is carried out on the expression characteristics.
The implementation method of the actual micro expression recognition process based on the trained micro expression recognition model is shown in fig. 3:
101, acquiring a face video.
Because the duration of the micro expression is short and the action amplitude is very small, the face video image in the step only needs to include the face in each frame, and the precise corresponding video of the micro expression is not needed.
And 102, extracting each frame of the face video.
103, sequentially comparing the difference between each frame except the first frame and the last frame and the next frame, and sequentially comparing the difference between each frame except the first frame and the last frame and the previous frame.
Since the first frame and the last frame generally do not contain valid information, the first frame and the last frame are not considered in the present application.
In particular, the method comprises the following steps of,
(1) comparing the difference between each frame except the first frame and the last frame and the next frame in turn, including:
for any frame i other than the first and last frames,
the red, green and blue channel values for each pixel in any frame i are obtained.
The red channel value, the green channel value and the blue channel value of each pixel in a frame i +1 subsequent to any frame i are obtained.
And determining the difference between any frame i and the subsequent frame i +1 according to the red channel value, the green channel value and the blue channel value of each pixel in any frame i and the red channel value, the green channel value and the blue channel value of each pixel in the subsequent frame i +1 of any frame i.
For example, the difference between any frame i and the subsequent frame i +1 is calculated by the following formula:
wherein,
is the difference between any frame i and the subsequent frame i +1, j is a pixel identifier, j is more than or equal to 1 and less than or equal to M, M is the total number of pixels of any frame of the face video, WRAs a red channel weight, WGAs a green channel weight, WBAs a blue channel weight, Ri,jIs the red channel value, G, of pixel j in any frame ii,jIs the green channel value, B, of pixel j in any frame ii,jIs the blue channel value, R, of pixel j in any frame ii+1,jIs the red channel value, G, of pixel j in the subsequent frame i +1i+1,jIs the green channel value, B, of pixel j in the subsequent frame i +1i+1,jIs the blue channel value of pixel j in the next frame i + 1.
Wherein,
①WRthe determination method comprises the following steps:
in the face video, any frame including a face is acquired (as long as one frame image including a face is acquired). Face regions in any frame of a face are identified. And determining a red channel value of the skin color of the human face according to the human face area. WRRed channel value/255 for face skin tone.
②WGThe determination method comprises the following steps:
in a face video, acquiring any frame including a face; identifying a face region in any frame of a face; determining a green channel value of the skin color of the face according to the face area; wGGreen channel value of face skin/255.
③WBThe determination method comprises the following steps:
in a face video, acquiring any frame including a face; identifying a face region in any frame of a face; determining a blue channel value of the skin color of the human face according to the human face area; wBBlue channel value/255 for face skin tone.
In particular implementations, W may be determined simultaneouslyR、WG、WB. For example, in a face video, any frame including a face is acquired; identifying a face region in any frame of a face; determining a red channel value, a green channel value and a blue channel value of the skin color of the face according to the face region; wRRed channel value/255, W for face skin colorGGreen channel value/255, W for face skin colorBBlue channel value/255 for face skin tone.
(2) Comparing the difference between each frame except the first frame and the last frame and the previous frame in turn, including:
for any frame i except the first and last frames.
Acquiring a red channel value, a green channel value and a blue channel value of each pixel in any frame i;
the red, green and blue channel values are obtained for each pixel in a frame i-1 prior to any frame i.
The difference between any frame i and its previous frame i-1 is determined according to the red channel value, the green channel value, and the blue channel value of each pixel in any frame i, and the red channel value, the green channel value, and the blue channel value of each pixel in the previous frame i-1 of any frame i.
For example, determining the difference between any frame i and its previous frame i-1 according to the red channel value, the green channel value, and the blue channel value of each pixel in any frame i, and the red channel value, the green channel value, and the blue channel value of each pixel in the previous frame i-1 of any frame i comprises:
the difference between any frame i and its previous frame i-1 is calculated by the following formula:
wherein,
is the difference between any frame i and its previous frame i-1, Ri-1,jThe red channel value, G, of pixel j in frame i-1 preceding any frame ii-1,jThe green channel value, B, of pixel j in frame i-1 preceding any frame ii-1,jThe blue channel value of pixel j in frame i-1, which is the frame immediately preceding any frame i.
The difference between the two frames is calculated in the step, not only the difference of the actual pixels is considered, but also the skin color of a person is considered, and the pixel difference is finely adjusted according to the difference of the skin color of the person, so that the difference in the step is more consistent with the time condition, and the micro expression is more accurately reflected.
And 104, sequentially determining the difference between the difference of each frame except the first frame and the last frame and the difference between the frame and the frame before as the difference value of the frame.
For example,
and 105, determining the micro expression frame according to the difference value of the frames except the first frame and the last frame.
Since the content of the face video in 101 is not limited in the present application, the face video in 101 is likely to include non-micro expression video segments, such as non-expressive face video segments, or segments of regular expressions (such as laughing, crying, etc.), although expressions, but not micro expressions, or non-expressive face video segments and regular expressions.
In the step, the start frame and the end frame of the micro expression can be accurately found in the face video in the 101, so that the frame only containing the micro expression is extracted, and the frame only containing the micro expression is identified subsequently, so that the identification efficiency can be improved.
Specifically, in each frame except for the first frame and the last frame, selecting a frame of which the absolute value of the difference value is not 0 and is smaller than a first preset threshold value; determining the mark of each selected frame; and in each selected frame, determining the frames with continuous marks as the micro expression frames.
Taking 10 frames of the face video obtained in step 101, which are respectively recorded as frame 1, frame 2, frame 3, frame 4, frame 5, frame 6, frame 7, frame 8, frame 9, and frame 10 in time sequence as an example, except for the first frame (frame 1) and the last frame (frame 10), the remaining 8 frames are respectively frame 2, frame 3, frame 4, frame 5, frame 6, frame 7, frame 8, and frame 9. If the difference value of frame 2 obtained in step 104 is 0.9, the difference value of frame 3 is 0.01, the difference value of frame 4 is 0, the difference value of frame 5 is 0.02, the difference value of frame 6 is 0.01, the difference value of frame 7 is 0, the difference value of frame 8 is 0.01, and the difference value of frame 9 is 0.01. When the first preset threshold is 0.03, selecting frames (frame 3, frame 5, frame 6, frame 8 and frame 9) with the absolute value of the difference value being not 0 and being less than 0.03, and determining the frames (frame 5, frame 6, frame 8 and frame 9) with continuous marks as the microexpression frames.
It is likely that frame 5, frame 6, and frame 8, frame 9, represent a micro expression at this time.
The foregoing is merely exemplary and does not represent actual situations.
The first preset threshold value ensures the difference size, the difference amplitude of different micro expressions is different, and the first preset threshold value is used for carrying out adaptive selection according to different application fields of the method, so that the universality of the expression recognition method provided by the application is ensured.
In addition, the present application does not limit "continuous" but only non-individual frames. For example, if there are 2 frames with consecutive marks, the 2 frames with consecutive marks are all determined as the micro expression frames. For another example, if there are 3 frames with consecutive marks, all the 3 frames with consecutive marks are determined as the microexpressing frames.
And 106, extracting the expression characteristics of the micro expression frame through a pre-trained micro expression recognition model, reducing the dimension of the expression characteristics, and recognizing the reduced dimension characteristics to obtain a recognition result.
The micro expression recognition model in this step is the micro expression recognition model obtained before 101. In this step, the micro-expression frame obtained in 105 is input into the micro-expression recognition model, and the micro-expression recognition model can classify the micro-expressions to obtain the micro-expression result corresponding to the face video obtained in 101.
In the process that the trained micro expression recognition model carries out actual micro expression recognition, the micro expression frame is intelligently extracted, and micro expression recognition is carried out only on the basis of the micro expression frame, so that the micro expression recognition efficiency is improved, and the micro expression recognition quality is also improved.
It should be noted that "first", "second", and the like in this embodiment and the subsequent embodiments are only used for distinguishing preset thresholds, classification results, standard classification results, and the like in different steps, and do not have any other special meanings.
Has the advantages that:
the difference between each frame and the next frame and the difference between the frame and the previous frame are compared to obtain the difference value of the frame, and the micro expression frame is determined according to the difference value of each frame.
Claims (6)
1. A micro-expression recognition method, the method comprising:
acquiring a face video;
extracting each frame of the face video;
sequentially comparing the difference between each frame except the first frame and the last frame and the previous frame;
sequentially determining the difference between the difference of each frame except the first frame and the last frame and the difference between the frame and the previous frame as the difference value of the frame;
selecting frames with difference values of which the absolute values are not 0 and are smaller than a first preset threshold value from the frames except the first frame and the last frame;
determining the mark of each selected frame;
determining frames with continuous marks as micro-expression frames in the selected frames;
training a micro expression recognition model;
extracting expression characteristics of the micro expression frame through a trained micro expression recognition model, reducing the dimension of the expression characteristics, and recognizing the reduced dimension characteristics to obtain a recognition result;
wherein, the comparing the difference between each frame except the first frame and the last frame and the next frame in turn comprises:
for any frame i other than the first and last frames,
acquiring a red channel value, a green channel value and a blue channel value of each pixel in any frame i;
acquiring a red channel value, a green channel value and a blue channel value of each pixel in a frame i +1 subsequent to the frame i;
determining the difference between any frame i and the subsequent frame i +1 according to the red channel value, the green channel value and the blue channel value of each pixel in any frame i, and the red channel value, the green channel value and the blue channel value of each pixel in the subsequent frame i +1 of any frame i;
the sequentially comparing the difference between each frame except the first frame and the last frame and the previous frame comprises:
for any frame i except the first frame and the last frame;
acquiring a red channel value, a green channel value and a blue channel value of each pixel in any frame i;
acquiring a red channel value, a green channel value and a blue channel value of each pixel in a previous frame i-1 of any frame i;
determining the difference between any frame i and the previous frame i-1 thereof according to the red channel value, the green channel value and the blue channel value of each pixel in any frame i, and the red channel value, the green channel value and the blue channel value of each pixel in the previous frame i-1 of any frame i;
the training micro-expression recognition model comprises:
acquiring a plurality of sample videos;
for each sample video, extracting corresponding expression features by adopting a local binary pattern;
performing dimensionality reduction processing according to the expression features of all sample videos;
based on the features after the dimension reduction processing, carrying out recognition training on each sample video to form a micro-expression recognition model;
the dimension reduction processing according to the expression features of all sample videos comprises the following steps:
determining the information increment of each expression characteristic according to the following formula:
determining the expression features with the information increment larger than a second preset threshold value as the features after dimension reduction processing;
wherein s is an expression feature identifier, and s is more than or equal to 1 and less than or equal to NA，NATotal number of expressive features, G, for all sample videossIs the information increment of the expression feature s, E () represents entropy, E(s) is the entropy of the expression feature s, v is the sample video identification containing the expression feature s, v is more than or equal to 1 and less than or equal to C, C is the total number of all sample videos containing the expression feature s, E (v) is the entropy of the sample video v containing the expression feature s, NvAnd obtaining the number of the expression features corresponding to the sample video v containing the expression features s.
2. The method according to claim 1, wherein the determining the difference between the any frame i and the subsequent frame i +1 according to the red channel value, the green channel value and the blue channel value of each pixel in the any frame i and the red channel value, the green channel value and the blue channel value of each pixel in the subsequent frame i +1 of the any frame i comprises:
calculating the difference between any frame i and the subsequent frame i +1 by the following formula:
wherein,
is the difference between any frame i and the subsequent frame i +1, j is a pixel identifier, j is more than or equal to 1 and less than or equal to M, M is the total number of pixels of any frame of the face video, WRAs a red channel weight, WGAs a green channel weight, WBAs a blue channel weight, Ri,jIs the red channel value, G, of pixel j in said any frame ii,jIs the green channel value, B, of pixel j in said any frame ii,jIs the blue channel value, R, of the pixel j in said any frame ii+1,jIs the red channel value, G, of pixel j in the subsequent frame i +1i+1,jIs the green channel value, B, of pixel j in the subsequent frame i +1i+1,jIs the blue channel value of the pixel j in the next frame i + 1;
determining a difference between any frame i and a previous frame i-1 thereof according to the red channel value, the green channel value, and the blue channel value of each pixel in any frame i, and the red channel value, the green channel value, and the blue channel value of each pixel in a previous frame i-1 of any frame i, includes:
calculating the difference between any frame i and the previous frame i-1 by the following formula:
wherein,
is the difference between any frame i and its previous frame i-1, Ri-1,jIs the red channel value, G, of the pixel j in the frame i-1 preceding said any frame ii-1,jIs the green channel value, B, of a pixel j in a frame i-1 preceding said any frame ii-1,jIs the blue channel value of the pixel j in the frame i-1 preceding said any frame i.
3. The method of claim 2, wherein W isRThe determination method comprises the following steps:
in the face video, any frame comprising a face is obtained;
identifying a face region in any frame of the face;
determining a red channel value of the skin color of the human face according to the human face area;
WRred channel value/255 for face skin color;
the W isGThe determination method comprises the following steps:
in the face video, any frame comprising a face is obtained;
identifying a face region in any frame of the face;
determining a green channel value of the skin color of the human face according to the human face area;
WGgreen channel value/255 for face skin color;
the W isBThe determination method comprises the following steps:
in the face video, any frame comprising a face is obtained;
identifying a face region in any frame of the face;
determining a blue channel value of the skin color of the human face according to the human face area;
WBblue channel value/255 for face skin tone.
4. The method of claim 1, wherein the formula for E () is:
E()＝-p+log2p+-p-log2p-；
wherein p is+To classify the probability of correctness, p-Is the probability of classification error.
5. The method of claim 4, wherein the formula for E(s) is:
E(s)＝-p+(s)log2p+(s)-p-(s)log2p-(s)；
wherein p is+(s) probability of correct classification of expression features s, p-(s) is the probability of misclassification of the expression features s;
said p is+(s) and p-The calculation method of(s) is as follows:
s1-1, taking q as 0;
s1-2, sequentially selecting q expression features as first auxiliary expression features from the expression features except the expression feature S;
classifying each sample video based on the expression feature s and the first auxiliary expression feature to obtain a first classification result of each sample video after the first auxiliary expression feature is selected;
obtaining a first standard classification result of each sample video;
comparing the first classification result of each sample video with the first standard classification result, and determining the number N of the sample videos with the first classification result consistent with the first standard classification result+(s) and first classificationNumber N of sample videos whose result is inconsistent with the first standard classification result-(s)；
Determining the probability of correctness
Determining error probability
S1-3, after the q expression features are sequentially selected, determining whether q +1 is N or notAIf q +1 is not NAThen S1-4 is executed if q +1 is NAThen execution of S1-5;
s1-4, q ═ q +1, S1-2 and S1-3 were repeatedly performed;
s1-5, mixing all
Is determined as p+(s) mixing all
Is determined as p-(s)。
6. The method of claim 4, wherein the formula for E (v) is:
E(v)＝-p+(v)log2p+(v)-p-(v)log2p-(v)；
wherein p is+(v) Probability of correctly classifying v sample video, p-(v) Probability of classification error for sample video v;
said p is+(v) And p-(v) The calculation method comprises the following steps:
s2-1, taking y as 1;
s2-2, sequentially selecting y expression features as second auxiliary expression features from the expression features corresponding to the sample video v;
every time a second auxiliary expression feature is selected, x1 is x1+1, and the sample video v is classified based on the second auxiliary expression feature to obtain a second classification result of the sample video v; wherein x1 is a first counting mark, and x1 is an initial value of 0;
acquiring a second standard classification result of the sample video v;
comparing the second classification result with the second standard classification result, and if the second classification result is consistent with the second standard classification result, x2 is x2+ 1; if the second classification result is inconsistent with the second standard classification result, x3 is x3+ 1; wherein x2 is the second counting mark, x2 is 0 in initial value, x3 is the third counting mark, and x3 is 0 in initial value;
s2-3, after the q expression features are sequentially selected, determining whether y +1 is larger than N or notvIf y +1 is not greater than NvThen S2-4 is performed if y +1 is greater than NvThen execution of S2-5;
s2-4, y +1, repeating S2-2 and S2-3;
s2-5, determining the value of x2/x1 as p+(s) determining the value of x3/x1 as p-(s)。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811085510.8A CN109190582B (en) | 2018-09-18 | 2018-09-18 | Novel micro-expression recognition method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811085510.8A CN109190582B (en) | 2018-09-18 | 2018-09-18 | Novel micro-expression recognition method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109190582A CN109190582A (en) | 2019-01-11 |
| CN109190582B true CN109190582B (en) | 2022-02-08 |
Family
ID=64911648
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811085510.8A Expired - Fee Related CN109190582B (en) | 2018-09-18 | 2018-09-18 | Novel micro-expression recognition method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109190582B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109800771B (en) * | 2019-01-30 | 2021-03-05 | 杭州电子科技大学 | Spontaneous micro-expression positioning method of local binary pattern of mixed space-time plane |
| US11954905B2 (en) | 2019-06-28 | 2024-04-09 | Hewlett-Packard Development Company, L.P. | Landmark temporal smoothing |
| CN113076813B (en) * | 2021-03-12 | 2024-04-12 | 首都医科大学宣武医院 | Training method and device for mask face feature recognition model |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPS6248866A (en) * | 1985-08-28 | 1987-03-03 | Fuji Photo Film Co Ltd | Read condition deciding method for radiographic information |
| CN101567043A (en) * | 2009-05-31 | 2009-10-28 | 中山大学 | Face tracking method based on classification and identification |
| WO2011065952A1 (en) * | 2009-11-30 | 2011-06-03 | Hewlett-Packard Development Company, L.P. | Face recognition apparatus and methods |
| CN104123543A (en) * | 2014-07-23 | 2014-10-29 | 泰亿格电子（上海）有限公司 | Eyeball movement identification method based on face identification |
| CN105913038A (en) * | 2016-04-26 | 2016-08-31 | 哈尔滨工业大学深圳研究生院 | Video based dynamic microexpression identification method |
| CN107346417A (en) * | 2017-06-13 | 2017-11-14 | 浪潮金融信息技术有限公司 | Method for detecting human face and device |
| CN107358206A (en) * | 2017-07-13 | 2017-11-17 | 山东大学 | Micro- expression detection method that a kind of Optical-flow Feature vector modulus value and angle based on area-of-interest combine |
| CN107403142A (en) * | 2017-07-05 | 2017-11-28 | 山东中磁视讯股份有限公司 | A kind of detection method of micro- expression |
| CN107808113A (en) * | 2017-09-13 | 2018-03-16 | 华中师范大学 | A kind of facial expression recognizing method and system based on difference depth characteristic |
| KR20180043937A (en) * | 2016-10-21 | 2018-05-02 | 삼성전자주식회사 | Method and apparatus for recognizing facial expression |
-
2018
- 2018-09-18 CN CN201811085510.8A patent/CN109190582B/en not_active Expired - Fee Related
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPS6248866A (en) * | 1985-08-28 | 1987-03-03 | Fuji Photo Film Co Ltd | Read condition deciding method for radiographic information |
| CN101567043A (en) * | 2009-05-31 | 2009-10-28 | 中山大学 | Face tracking method based on classification and identification |
| WO2011065952A1 (en) * | 2009-11-30 | 2011-06-03 | Hewlett-Packard Development Company, L.P. | Face recognition apparatus and methods |
| CN104123543A (en) * | 2014-07-23 | 2014-10-29 | 泰亿格电子（上海）有限公司 | Eyeball movement identification method based on face identification |
| CN105913038A (en) * | 2016-04-26 | 2016-08-31 | 哈尔滨工业大学深圳研究生院 | Video based dynamic microexpression identification method |
| KR20180043937A (en) * | 2016-10-21 | 2018-05-02 | 삼성전자주식회사 | Method and apparatus for recognizing facial expression |
| CN107346417A (en) * | 2017-06-13 | 2017-11-14 | 浪潮金融信息技术有限公司 | Method for detecting human face and device |
| CN107403142A (en) * | 2017-07-05 | 2017-11-28 | 山东中磁视讯股份有限公司 | A kind of detection method of micro- expression |
| CN107358206A (en) * | 2017-07-13 | 2017-11-17 | 山东大学 | Micro- expression detection method that a kind of Optical-flow Feature vector modulus value and angle based on area-of-interest combine |
| CN107808113A (en) * | 2017-09-13 | 2018-03-16 | 华中师范大学 | A kind of facial expression recognizing method and system based on difference depth characteristic |
Non-Patent Citations (2)
| Title |
|---|
| Recognising spontaneous facial micro-expressions;Tomas Pfister;《2011 International Conference on Computer Vision》;20120212;page1449-1457 * |
| 基于全局光流特征的微表情识别;张轩阁;《模式识别与人工智能》;20160830;第29卷(第8期);第760-769页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109190582A (en) | 2019-01-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Shan | Smile detection by boosting pixel differences | |
| CN106548149B (en) | Method for identifying human face micro-expression image sequence in monitoring video sequence | |
| Liu et al. | Learning expressionlets via universal manifold model for dynamic facial expression recognition | |
| Hassanat et al. | Color-based object segmentation method using artificial neural network | |
| CN109614927B (en) | Micro expression recognition based on difference of front and rear frames and characteristic dimension reduction | |
| CN109190582B (en) | Novel micro-expression recognition method | |
| Vishwakarma et al. | An efficient interpretation of hand gestures to control smart interactive television | |
| CN112364801A (en) | Dynamic threshold face recognition method | |
| Duan et al. | Face verification with local sparse representation | |
| Taha et al. | Iris features extraction and recognition based on the local binary pattern technique | |
| CN112183504A (en) | Video registration method and device based on non-contact palm vein image | |
| CN115862120A (en) | Separable variation self-encoder decoupled face action unit identification method and equipment | |
| Tee et al. | Facial recognition using enhanced facial features k-nearest neighbor (k-NN) for attendance system | |
| Aslam et al. | Gender classification based on isolated facial features and foggy faces using jointly trained deep convolutional neural network | |
| CN109800657A (en) | A kind of convolutional neural networks face identification method for fuzzy facial image | |
| De Siqueira et al. | Adaptive detection of human skin in color images | |
| Abbes et al. | Age-API: are landmarks-based features still distinctive for invariant facial age recognition? | |
| Alrikabi et al. | Deep Learning-Based Face Detection and Recognition System | |
| Ekinci et al. | Kernel Fisher discriminant analysis of Gabor features for online palmprint verification | |
| Chai et al. | Towards contactless palm region extraction in complex environment | |
| Choras | Facial feature detection for face authentication | |
| Gao et al. | Enhancing action recognition in low-resolution videos using Dempster-Shafer's model | |
| Hanumanthaiah et al. | Challenge Responsive Multi Modal Biometric Authentication Resilient to Presentation Attacks. | |
| Kumar et al. | An Advance Approach of Face Recognition using PCA and Region Base Color Segmentation | |
| Chen et al. | A VQ-based fast face recognition algorithm using optimized codebook |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20220208 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |