CN109614927A - Micro- Expression Recognition based on front and back frame difference and Feature Dimension Reduction - Google Patents
Micro- Expression Recognition based on front and back frame difference and Feature Dimension Reduction Download PDFInfo
- Publication number
- CN109614927A CN109614927A CN201811499959.9A CN201811499959A CN109614927A CN 109614927 A CN109614927 A CN 109614927A CN 201811499959 A CN201811499959 A CN 201811499959A CN 109614927 A CN109614927 A CN 109614927A
- Authority
- CN
- China
- Prior art keywords
- frame
- value
- difference
- color
- face
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/232—Non-hierarchical techniques
- G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
- G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Probability & Statistics with Applications (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
This application provides a kind of micro- expression recognition methods, carry out recognition of face to each frame in video, extract human face region；Extract pixel number, the background color, face brightness of each frame in video；A non-first frame is successively chosen, calculates that the frame of selection and the human face region difference in areas of its former frame, pixel number is poor, background color is poor, face luminance difference；Calculate the difference value of each non-first frame；Difference value is greater than to the frame of preset threshold, and, the first frame of video is determined as candidate frame；In candidate frame, continuous frame will be identified and be determined as micro- expression frame；The expressive features for extracting micro- expression frame carry out dimension-reduction treatment to expressive features by dimensionality reduction model trained in advance, identify to the feature after dimensionality reduction, obtain recognition result.The application according to human face region difference in areas, pixel number is poor, background color is poor, face luminance difference selects micro- expression frame to identify, can accurately extract frame relevant to micro- expression in face video, promote the recognition efficiency and accuracy of micro- expression frame.
Description
Technical field
This application involves field of artificial intelligence more particularly to micro- expression recognition methods.
Background technique
Micro- expression is a kind of nonverbal behavior, can show the own emotions of people.
Pay close attention at present mostly based on generic expression, other than face generic expression, there are also under psychological holddown, face
Micro- expression of the uncontrolled contraction of muscle and generation.
The duration of micro- expression is short, and movement range is very small.Therefore it correctly observes and has identified comparable difficulty.With
Naked eye accurately captures and identifies that micro- expression success rate is very low.After professional training, discrimination is also only capable of reaching 47%.
Therefore, the recognition methods of micro- expression receives the concern of more and more researchers.
Summary of the invention
To solve the above problems, the embodiment of the present application proposes a kind of micro- expression recognition method.
Obtain face video；
Recognition of face is carried out to each frame in the video, extracts human face region；
Extract the pixel number of each frame in the video, background color, face brightness；
A non-first frame is successively chosen, the frame of selection is calculated and the human face region difference in areas of its former frame, pixel number is poor, background color
Difference, face luminance difference；
Calculate the difference value of each non-first frame, wherein difference value=(human face region difference in areas * face luminance difference+background color
Difference) ^ pixel number is poor；
Difference value is greater than to the frame of preset threshold, and, the first frame of the video is determined as candidate frame；
In candidate frame, continuous frame will be identified and be determined as micro- expression frame；
The expressive features for extracting micro- expression frame carry out dimension-reduction treatment to expressive features by dimensionality reduction model trained in advance, to drop
Feature after dimension is identified, recognition result is obtained.
Optionally, the background color of each frame in the video is extracted, comprising:
For any frame in the video,
Region non-face in any frame is determined as background area；
Determine the RGB color value of each pixel in any frame background area, the RGB color value include red color value,
Green color value, Blue value；
The RGB color mean value of any frame background area is calculated by following formula, the RGB color mean value includes red face
Color mean value, green color mean value, Blue mean value:
Wherein j is the pixel point identification of any frame background area,Red color for any frame background area is equal
Value,For the green color mean value of any frame background area,Blue for any frame background area is equal
Value, c1jFor the red color value of any frame background area jth pixel, c2jFor any frame background area jth pixel
The green color value of point, c3jFor the Blue value of any frame background area jth pixel, n1For any frame background
Pixel total quantity in region；
The RGB color mean square deviation of any frame background area is calculated, the RGB color mean square deviation includes that red color is square
Difference, green color mean square deviation, Blue mean square deviation:
Wherein, σ11For red color mean square deviation, σ21For green color mean square deviation, σ31For Blue mean square deviation；
Determine that the RGB color section of any frame background area, the RGB color section include red color sectionGreen color sectionBlue section
In all pixels point of any frame background area, determine that red color value is located at RGB color area in RGB color value
Between in middle red color section, and green color value is located in green color section, and Blue value is located at Blue area
Interior pixel quantity n2；
The background color of any frame is determined according to n2.
Optionally, the background color is indicated by RGB color value；
The background color that any frame is determined according to n2, comprising:
Calculate the pixel number ratio n of any frame3=n2/n1；
The red color value of the background color of any frame isGreen color value is
Blue value is
Optionally, the face brightness of each frame in the video is extracted, comprising:
For any frame in the video,
The brightness value of each pixel in the human face region of any frame is determined by following formula:
Wherein, k is the pixel point identification of any frame human face region, hkFor the kth pixel of any frame human face region
Brightness value, RkFor the red color value in the RGB color value of kth pixel, GkIt is green in the RGB color value of kth pixel
Color color value, BkFor the Blue value in the RGB color value of kth pixel；
In the brightness value of all pixels point of any frame human face region, maximum brightness value and minimum luminance value are determined；
Calculate the luminance mean value of any frame human face regionWherein, n4For the human face region of any frame
Middle pixel total quantity；
According to maximum brightness value, minimum luminance value andDetermine the face brightness of any frame in the video.
Optionally, it is described according to maximum brightness value, minimum luminance value andDetermine the face brightness of any frame in the video,
Include:
Calculate the first difference d1=maximum brightness value-minimum luminance value；
Calculate the second difference
Calculate third difference
Calculate brightness ratio d4=| d1-d2 |/| d1-d3 |；
Calculate the brightness mean square deviation of any frame human face region
The face brightness of any frame is in the video
Optionally, before the difference value for calculating each non-first frame, further includes:
According to the human face region difference in areas of each non-first frame, pixel number is poor, background color is poor, face luminance difference carries out just non-first frame
Sieve；
The difference value for calculating each non-first frame, comprising:
The difference value of each frame after calculating primary dcreening operation.
Optionally, the human face region difference in areas according to each non-first frame, pixel number is poor, background color is poor, face luminance difference pair
Non- head frame carries out primary dcreening operation, comprising:
For any non-first frame,
If the human face region difference in areas of any non-first frame is not more than the first value, and, pixel number difference is not more than second value, and,
Background color difference is not more than third value, and, face luminance difference is not more than the 4th value, then any non-first frame passes through primary dcreening operation；Or
Person,
If the human face region difference in areas of any non-first frame is not more than the first value, but pixel number is poor, background color is poor, face is bright
Spending difference is 0, then any non-first frame passes through primary dcreening operation；Alternatively,
If the face luminance difference of any non-first frame is not more than the 4th value, but human face region difference in areas, pixel number are poor, background face
Color difference is 0, then any non-first frame passes through primary dcreening operation；
Wherein, the first value is (the sum of human face region difference in areas of all non-first frames+first frame human face region area-avg1)/institute
State the totalframes of face video, second value is (the sum of pixel number difference of all non-first frames+first frame pixel number-avg2)/described
The totalframes of face video, third value are (the sum of background color difference of all non-first frames+first frame background color-avg3)/institute
State the totalframes of face video, the 4th value be (the sum of face luminance difference of all non-first frames+first frame face brightness-avg4)/
The totalframes of the face video, the sum of human face region area of all frames of avg1=/face video totalframes, avg2
The sum of pixel number of=all frames/face video totalframes, the sum of background color of all frames of the avg3=/face
The totalframes of video, the sum of face brightness of all frames of avg4=/face video totalframes.
Optionally, it is described dimension-reduction treatment is carried out to expressive features by dimensionality reduction model trained in advance before, further includes:
Obtain sample set X, wherein total sample number is m in X, and each sample includes multiple expressive features, and each sample belongs to one
Classification；
Category classifies all samples；
Calculate all kinds of mean vectorsWherein, i is class mark, μiFor the mean vector of the i-th class, biIt is i-th
The sample size of class, j are sample identification, xijThe vector formed for the expressive features of j-th of sample of the i-th class；
According to all kinds of mean vectors, grand mean vector is determinedWherein, μ0For grand mean
Vector, E are different classes of sum belonging to sample in X；
According to grand mean vector, inter-class variance vector sum variance within clusters vector is calculated；
Expressive features after determining dimensionality reduction according to the inter-class variance vector sum variance within clusters vector form dimensionality reduction model.
Optionally, described according to grand mean vector, calculate inter-class variance vector sum variance within clusters vector, comprising:
Wherein, SwFor inter-class variance vector, SbFor variance within clusters vector, XiFor the set of the i-th class sample composition.
Optionally, the expressive features determined according to the inter-class variance vector sum variance within clusters vector after dimensionality reduction, comprising:
Calculate the weight vectors W=diag (S being made of each expressive features weightb·/Sw), wherein diag () is function, described
Function is used to take the element on diagonal of a matrix ,/it is operator, the operator is used for SwAnd SbCorresponding element be divided by；
By the sequence of each expressive features weight from big to small, expressive features are ranked up；
Expressive features after the forward expressive features of preset quantity sequence to be determined as to dimensionality reduction.
It has the beneficial effect that:
According to human face region difference in areas, pixel number is poor, background color is poor, face luminance difference selects micro- expression frame to be identified, can
Accurately to extract frame relevant to micro- expression in face video, the recognition efficiency and accuracy of micro- expression frame are promoted.
Detailed description of the invention
The specific embodiment of the application is described below with reference to accompanying drawings, in which:
Fig. 1 shows a kind of dimensionality reduction modular concept schematic diagram for being divided into 2 classes of one embodiment of the application offer；
Fig. 2 shows a kind of micro- expression recognition method flow diagrams that one embodiment of the application provides；
A kind of LBP description that Fig. 3 shows the offer of one embodiment of the application calculates schematic diagram；
Fig. 4 shows a kind of feature extraction schematic diagram of one embodiment of the application offer.
Specific embodiment
In order to which technical solution and the advantage of the application is more clearly understood, below in conjunction with attached drawing to the exemplary of the application
Embodiment is described in more detail, it is clear that and described embodiment is only a part of the embodiment of the application, rather than
The exhaustion of all embodiments.And in the absence of conflict, the feature in the embodiment and embodiment in this explanation can be mutual
It combines.
Since the duration of micro- expression is short, and movement range is very small.Therefore it correctly observes and has identified comparable difficulty
Degree.Based on this, the application provides a kind of micro- expression recognition method, the difference of this method more each frame and its a later frame, and
With the difference of its former frame, the difference value of the frame is obtained, micro- expression frame is determined according to the difference value of each frame, this method can be accurate
Extraction face video in frame relevant to micro- expression, promote the recognition efficiency and accuracy of micro- expression frame.
Expression recognition method provided by the present application includes 2 big processes, and first big process is trained dimensionality reduction model process, another
A big process is that practical micro- Expression Recognition process is carried out based on trained dimensionality reduction model.
Training dimensionality reduction model process be not execute every time expression recognition method provided by the present application be intended to execute process, only when
Expression recognition method provided by the present application is executed for the first time, alternatively, Expression Recognition scene changes, alternatively, based on training
Dimensionality reduction model when carrying out practical micro- Expression Recognition, when expressive features dimensionality reduction effect is undesirable or other reasons, can just hold
Row training dimensionality reduction model process to promote expressive features dimensionality reduction effect, and then promotes the accuracy of practical micro- Expression Recognition result.
The application is not defined the execution trigger condition of training dimensionality reduction model process.
The concrete methods of realizing of training dimensionality reduction model process is as follows:
Step 1, sample set X is obtained.
Wherein, total sample number is m in X, and each sample includes multiple expressive features, and each sample belongs to a classification.
For example, if sample belongs to E different classifications, respectively the 1st class, the 2nd class ... ..., the i-th class ... ... E class altogether in X.
There is b in 1st class1A sample, b1The collection of a sample composition is combined into X1, have b in the 2nd class2A sample, b2The collection of a sample composition is combined into
X2... ....
Step 2, category classifies all samples.
By taking column in step 1 as an example, all samples are divided into E class by this step, and the sample for belonging to the 1st class is divided into 1 class, belong to
The sample of 2 classes is divided into 1 class ... ....
Step 3, all kinds of mean vectors is calculated.
Specifically, mean vector is calculated by following formula for any sort (such as the i-th class):
Wherein, i is class mark, μiFor the mean vector of the i-th class, biFor the sample size of the i-th class, j is sample identification, xijIt is
The vector of the expressive features composition of j-th of sample of i class.
Step 4, according to all kinds of mean vectors, grand mean vector is determined.
Specifically, determining grand mean vector by following formula:
Wherein, μ0For grand mean vector, E is different classes of sum belonging to sample in X.
Step 5, according to grand mean vector, inter-class variance vector sum variance within clusters vector is calculated.
Specific formula for calculation is as follows:
Wherein, SwFor inter-class variance vector, SbFor variance within clusters vector, XiFor the set of the i-th class sample composition.
Step 6, the expressive features after dimensionality reduction are determined according to inter-class variance vector sum variance within clusters vector, form dimensionality reduction model.
Circular is as follows:
1) the weight vectors W=diag (S being made of each expressive features weight is calculatedb·/Sw)。
Wherein, diag () is function, which is used to take the element on diagonal of a matrix ,/it is operator, which uses
In by SwAnd SbCorresponding element be divided by.
2) sequence of each expressive features weight from big to small is pressed, expressive features are ranked up.
3) the forward expressive features of preset quantity sequence are determined as to the expressive features after dimensionality reduction.
Expressive features after dimensionality reduction can form character subset F.Weight is bigger, and the characteristic component corresponding to the weight the suitable micro-
Expression classification.
Output obtains character subset, forms a dimensionality reduction model.
Fig. 1 shows the dimensionality reduction modular concept schematic diagram for being divided into 2 classes.
The implementation method for carrying out practical micro- Expression Recognition process based on trained dimensionality reduction model is as shown in Figure 2:
S101 obtains face video.
Because the duration of micro- expression is short, and movement range is very small, therefore, as long as the every frame of the facial video image of this step
Include face, the video of micro- expression must not corresponded to accurately.
S102 carries out recognition of face to each frame in video, extracts human face region.
The present embodiment is not defined the extracting method of human face region, existing extracting method.
S103 extracts pixel number, the background color, face brightness of each frame in video.
The pixel value for the video file that different configuration of video capture device obtains is different, and the pixel number of former frame and a later frame
In various degree, micro- Expression Recognition can be had an impact, therefore, this motion can extract the pixel number of each frame in video.
Pixel number can be indicated with a number, such as " 0.3 million pixel " digital camera, it has specified 300,000 pixel；It can also
With with a pair of of digital representation, such as " 640*480 display ", it indicates that (such as VGA is shown for laterally 640 pixels and longitudinal 480 pixels
Device).And a pair of of number can also be changed into a number, if the pixel of 640*480 display is 640*480=307200 pixel.
The pixel number of each frame in this step can be calculated for the total quantity of pixel in the frame by the resolution ratio of image.Such as one
The image resolution ratio of frame is 1280*960, pixel number=1280*960=1228800 of the frame.
The present embodiment is not defined the extracting method of pixel number, existing extracting method.
For extracting the implementation method of the background color of each frame in video, including but not limited to:
For any frame in video,
Step 1.1, region non-face in any frame is determined as background area.
Step 1.2, the RGB color value of each pixel in any frame background area is determined.
Wherein, RGB color value includes red color value, green color value, Blue value.
Step 1.3, the RGB color mean value of any frame background area is calculated by following formula.
Wherein, RGB color mean value includes red color mean value, green color mean value, Blue mean value:
J is the pixel point identification of any frame background area,For the red color mean value of any frame background area,It is any
The green color mean value of frame background area,For the Blue mean value of any frame background area, c1jFor any frame background area
The red color value of jth pixel, c2jFor the green color value of any frame background area jth pixel, c3jFor any frame background
The Blue value of region jth pixel, n1For pixel total quantity in any frame background area.
Step 1.4, the RGB color mean square deviation of any frame background area is calculated.
Wherein, RGB color mean square deviation includes red color mean square deviation, green color mean square deviation, Blue mean square deviation:
σ11For red color mean square deviation, σ21For green color mean square deviation, σ31For Blue mean square deviation.
Step 1.5, the RGB color section of any frame background area is determined.
Wherein, RGB color section includes red color sectionGreen color sectionBlue section
Step 1.6, in all pixels point of any frame background area, determine that red color value is located at RGB face in RGB color value
In color section in red color section, and green color value is located in green color section, and Blue value is located at blue face
Pixel quantity n in color section2。
Step 1.7, according to n2Determine the background color of any frame.
Wherein, background color is indicated by RGB color value, and RGB color value includes red color value, green color value and Blue
Value.
Specifically, calculating the pixel number ratio n of any frame3=n2/n1；The red color value of the background color of any frame isGreen color value isBlue value is
Background color extracting method provided in this embodiment is not simply by each face in pixel RGB color value each in background
The mean value of chrominance channel is as background color, but according to the distribution situation of Color Channel respective value each in each pixel RGB color value
Dynamically mean value is adjusted, value adjusted is regard as background color, so that the determination of background color is more in line with reality
Situation.
For extracting the implementation of the face brightness of each frame in video, including but not limited to:
For any frame in video,
Step 2.1, the brightness value of each pixel in the human face region of any frame is determined by following formula.
Wherein, k is the pixel point identification of any frame human face region, hkFor the brightness value of the kth pixel of any frame human face region,
RkFor the red color value in the RGB color value of kth pixel, GkFor the green color value in the RGB color value of kth pixel,
BkFor the Blue value in the RGB color value of kth pixel.
Step 2.2, in the brightness value of all pixels point of any frame human face region, maximum brightness value and minimum brightness are determined
Value.
Step 2.3, the luminance mean value of any frame human face region is calculated
Wherein, n4For pixel total quantity in the human face region of any frame.
Step 2.4, according to maximum brightness value, minimum luminance value andDetermine the face brightness of any frame in video.
Specifically,
1) the first difference d1=maximum brightness value-minimum luminance value is calculated.
2) the second difference is calculated
3) third difference is calculated
4) brightness ratio d4=is calculated | d1-d2 |/| d1-d3 |.
5) the brightness mean square deviation of any frame human face region is calculated
6) the face brightness of any frame is in video
Face brightness extracting method provided in this embodiment is not simply to make the mean value of pixel intensity each in human face region
For face brightness, but dynamically mean value is adjusted according to the gap between each pixel intensity and maximum brightness and minimum brightness
It is whole, value adjusted is regard as face brightness, so that the determination of face brightness is more in line with actual conditions.
S104 successively chooses a non-first frame, calculates that the frame of selection and the human face region difference in areas of its former frame, pixel number are poor, carry on the back
Scape colour-difference, face luminance difference.
Terminate since the second frame to last frame, successively select a frame, by the face area of the frame of selection and the former frame of the frame
The difference of domain area is determined as human face region difference in areas, the difference of pixel number is determined as that pixel number is poor, background color difference is true
It is set to that background color is poor, difference of face brightness is determined as face luminance difference.
For example, human face region difference in areas=selection frame human face region area-its former frame human face region area.Pixel number is poor
The pixel number of its former frame of the pixel number-of=selection frame.The background of its former frame of background color difference=background color-of selection frame
Color.Face brightness-its former frame face brightness of face luminance difference=selection frame.
S105 calculates the difference value of each non-first frame.
Wherein, difference value=(human face region difference in areas * face luminance difference+background color is poor) ^ pixel number is poor.
^ is power operation symbol.
In addition, executing speed to promote scheme provided in this embodiment, before the difference value for calculating each non-first frame, may be used also
To carry out primary dcreening operation to non-first frame, propose that obvious is not same people, hence it is evident that be not belonging to the frame of micro- expression.
That is the specific implementation procedure of S105 are as follows: according to the human face region difference in areas of each non-first frame, pixel number is poor, background color is poor, people
Face luminance difference carries out primary dcreening operation to non-first frame, calculates the difference value of each frame after primary dcreening operation.
According to the human face region difference in areas of each non-first frame, pixel number is poor, background color is poor, face luminance difference carries out just non-first frame
The scheme of sieve, including but not limited to:
For any non-first frame, if the human face region difference in areas of any non-first frame is not more than the first value, and, pixel number difference is not more than
Second value, and, background color difference is not more than third value, and, face luminance difference is not more than the 4th value, then any non-first frame passes through just
Sieve；Alternatively, if the human face region difference in areas of any non-first frame is not more than the first value, but pixel number is poor, background color is poor, face is bright
Spending difference is 0, then any non-first frame passes through primary dcreening operation；Alternatively, if the face luminance difference of any non-first frame is not more than the 4th value, but people
Face area surface product moment, pixel number are poor, background color difference is 0, then any non-first frame passes through primary dcreening operation.
Wherein, the first value is (the sum of human face region difference in areas of all non-first frames+first frame human face region area-avg1)/institute
State the totalframes of face video, second value is (the sum of pixel number difference of all non-first frames+first frame pixel number-avg2)/described
The totalframes of face video, third value are (the sum of background color difference of all non-first frames+first frame background color-avg3)/institute
State the totalframes of face video, the 4th value be (the sum of face luminance difference of all non-first frames+first frame face brightness-avg4)/
The totalframes of the face video, the sum of human face region area of all frames of avg1=/face video totalframes, avg2
The sum of pixel number of=all frames/face video totalframes, the sum of background color of all frames of the avg3=/face
The totalframes of video, the sum of face brightness of all frames of avg4=/face video totalframes.
Difference value is greater than the frame of preset threshold by S106, and, the first frame of video is determined as candidate frame.
Preset threshold ensure that the size of difference, and different micro- expression differential magnitudes is different, by preset threshold according to this method
Application field is different, carries out adaptability and selects, and then guarantees the versatility of expression recognition method provided by the present application.
S107 will identify continuous frame and be determined as micro- expression frame in candidate frame.
For example, candidate frame is frame 3, frame 5, frame 6, frame 8, frame 9, then it will identify continuous frame (frame 5, frame 6, frame 8, frame 9) and determine
For micro- expression frame.
It is likely to frame 5 at this time, frame 6 indicates a micro- expression, frame 8, the one micro- expression of expression of frame 9.
It above are only example, do not represent actual conditions.
The application is not defined " continuous ", as long as non-individual frame.For example, there is 2 continuous frames of mark then will
This 2 continuous frames of mark are determined as micro- expression frame.For another example there is 3 continuous frames of mark then that this 3 marks are continuous
Frame is determined as micro- expression frame.
S108 extracts the expressive features of micro- expression frame, carries out dimension-reduction treatment to expressive features by dimensionality reduction model trained in advance,
Feature after dimensionality reduction is identified, recognition result is obtained.
This step can train micro- Expression Recognition model in advance, then extract the expressive features of micro- expression frame, pass through training in advance
After dimensionality reduction model carries out dimension-reduction treatment to expressive features, the feature after dimensionality reduction is identified using micro- Expression Recognition model,
Obtain recognition result.
Wherein, the training process of micro- Expression Recognition model, including but not limited to:
Step 3.1 obtains multiple Sample videos.
Sample video can be obtained from existing micro- expression data concentration.
Since micro- expression is the small face action that people generates when attempting and covering up oneself mood.People master is said in a strict sense
The small expression for seeing simulation cannot be known as micro- expression, therefore the degree of reliability of the induction mode determination data of micro- expression.
The one or 2 multiple Sample videos of acquisition that this step can be concentrated from following 2 kinds existing micro- expression datas:
Micro- expression data collection SMIC, by Oulu, Finland, university is established, it is desirable that and subject watches the video for having big mood swing, and
The mood of try to cover up oneself is not exposed, and keeper observes the expression of subject in the case where not watching video.If record
Person observes the facial expression of subject, then subject will be punished.Constitute 16 people's under this abduction mechanism
164 video sequences, micro- expression classification have 3 kinds, respectively actively (positive), surprised (surprise), passiveness
(negative), video sequence number is respectively 70,51,43.
Micro- expression data collection CASME2, is established by Institute of Developed Organisms, Academia Sinica, ensures to count using similar abduction mechanism
According to reliability can be obtained only if facial expression that subject successfully holds in and if not being recorded person's discovery
To corresponding reward.5 kinds of micro- expression classifications that the data set is made of 247 video sequences of 26 people are glad respectively
(happiness), detest (disgust), surprised (surprise), constrain (repression), other (other), it is of all categories
Corresponding video sequence number is respectively 32,64,25,27,99.
Step 3.2, for each Sample video, extract corresponding expressive features using local binary patterns.
Local binary patterns (Local Binary Pattern, LBP) description is defined in center pixel and its surrounding rectangle
On neighborhood, as shown in figure 3, two-value quantifies the neighborhood territory pixel around center pixel using the gray value of center pixel as threshold value, it is greater than
Or it is encoded to 1 equal to center pixel value, less than being then encoded to 0, and form a local binary pattern.
It using the upper left corner is starting point according to being connected to obtain a string of binary digits clockwise by the binary mode, it is right
The ten's digit answered can uniquely identify central pixel point.According to the method, each of image pixel is ok
It is calculated with a local binary pattern.
As shown in figure 3, the center pixel value in the table of the left side is 178, the value in the upper left corner is 65,65 < 178, so, corresponding value
It is 0,188 > 178, so corresponding value is 1.And so on, the table on the right side Fig. 3 is obtained, and then obtain binary mode value and be
01000100。
Furthermore it is also possible to which the extension by LBP static state texture descriptor in time-space domain, forms 2 dimension parts two on 3 orthogonal planes
Value mode.As shown in figure 4, the LBP feature in tri- orthogonal plane video sequences of XY, XT and YT is extracted respectively, it will be each orthogonal
Feature vector in plane is connected, and LBP-TOP feature vector is formed.This method had both considered the local grain letter of image
The case where ceasing, and changing over time to video is described.
But the vector dimension of LBP-TOP is 3 × 2L, L is field point number.If the expression directly extracted with step 3.2 is special
Sign carries out modeling can be ineffective because intrinsic dimensionality causes greatly model training speed slower.Therefore, the application is executing step
After 3.2 extract expressive features, step 3.3 can be executed, to reduce the dimension that hands-on model is considered expressive features,
Lift scheme training effectiveness.
Step 3.3 carries out recognition training to each Sample video, forms micro- Expression Recognition model.
The training method of this step can there are many, the present embodiment provides use following training method:
3.3.1, expressive features are based on using any clustering algorithm (such as k-means algorithm) to gather each Sample video
Class forms micro- expression class belonging to each Sample video.
3.3.2, the parameter in clustering algorithm is adjusted according to the second criteria classification result of each Sample video.
Since each Sample video obtains the label per se with the label for identifying its micro- expression classification, in this step, as
Second criteria classification result of each Sample video.
3.3.3,3.3.1 and 3.3.2 is repeated, training is completed, forms micro- Expression Recognition model.
Micro- Expression Recognition model in the application is a classifier.
Such as: use support vector machines (Support Vector Machine, SVM) method.The key of SVM is kernel function,
Different svm classifier effects is just had using different kernel functions.
Such as following kernel function can be used: linear kernel (Linear Kernel), card side's core (Chi-square Kernel), straight
Side's figure intersects core (Histogram Intersection kernel).
In addition, cross validation (Cross can also be used in order to promote the Classification and Identification rate of final trained disaggregated model
Validation the performance of micro- Expression Recognition model) is examined.Specifically, all Sample videos are divided into two subsets, one
For subset for training classifier to be known as training set, the validity that another subset is used to verify analysis classifier is known as test set.
Trained obtained classifier is tested using test set, in this, as the performance indicator of classifier.Common method has simply
Cross validation, K roll over cross validation and stay a cross validation.
For with staying a cross validation method to carry out micro- expression classification training to the SVM classifier of different kernel functions.Every time
Select all video sequences of a subject as test sample, all video sequences of remaining I subject are as training
Sample repeats I+1 experiment, calculates I+1 average classification discrimination.
Based on this, the training of a micro- Expression Recognition model is completed.
After training micro- Expression Recognition model, dimension-reduction treatment is carried out to expressive features by dimensionality reduction model trained in advance, is adopted
The feature after dimensionality reduction is identified with micro- Expression Recognition model, obtains recognition result.
It, can be with due to having carried out dimension-reduction treatment to expressive features before carrying out micro- Expression Recognition by micro- Expression Recognition model
Promote the recognition rate and recognition accuracy of micro- Expression Recognition model.
It should be noted that " first " in the present embodiment and subsequent embodiment, " second ", " third " etc. are only used for distinguishing different
Preset threshold, classification results, criteria classification result in step etc. do not have other any particular meanings.
The utility model has the advantages that
According to human face region difference in areas, pixel number is poor, background color is poor, face luminance difference selects micro- expression frame to be identified, can
Accurately to extract frame relevant to micro- expression in face video, the recognition efficiency and accuracy of micro- expression frame are promoted.
Claims (10)
1. a kind of micro- expression recognition method, which is characterized in that the described method includes:
Obtain face video；
Recognition of face is carried out to each frame in the video, extracts human face region；
Extract the pixel number of each frame in the video, background color, face brightness；
A non-first frame is successively chosen, the frame of selection is calculated and the human face region difference in areas of its former frame, pixel number is poor, background color
Difference, face luminance difference；
Calculate the difference value of each non-first frame, wherein difference value=(human face region difference in areas * face luminance difference+background color
Difference) ^ pixel number is poor；
Difference value is greater than to the frame of preset threshold, and, the first frame of the video is determined as candidate frame；
In candidate frame, continuous frame will be identified and be determined as micro- expression frame；
The expressive features for extracting micro- expression frame carry out dimension-reduction treatment to expressive features by dimensionality reduction model trained in advance, to drop
Feature after dimension is identified, recognition result is obtained.
2. the method according to claim 1, wherein extracting the background color of each frame in the video, comprising:
For any frame in the video,
Region non-face in any frame is determined as background area；
Determine the RGB color value of each pixel in any frame background area, the RGB color value include red color value,
Green color value, Blue value；
The RGB color mean value of any frame background area is calculated by following formula, the RGB color mean value includes red face
Color mean value, green color mean value, Blue mean value:
Wherein j is the pixel point identification of any frame background area,Red color for any frame background area is equal
Value,For the green color mean value of any frame background area,Blue for any frame background area is equal
Value, c1jFor the red color value of any frame background area jth pixel, c2jFor any frame background area jth pixel
The green color value of point, c3jFor the Blue value of any frame background area jth pixel, n1For any frame background
Pixel total quantity in region；
The RGB color mean square deviation of any frame background area is calculated, the RGB color mean square deviation includes that red color is square
Difference, green color mean square deviation, Blue mean square deviation:
Wherein, σ11For red color mean square deviation, σ21For green color mean square deviation, σ31For Blue mean square deviation；
Determine that the RGB color section of any frame background area, the RGB color section include red color sectionGreen color sectionBlue section
In all pixels point of any frame background area, determine that red color value is located at RGB color area in RGB color value
Between in middle red color section, and green color value is located in green color section, and Blue value is located at Blue area
Interior pixel quantity n2；
The background color of any frame is determined according to n2.
3. according to the method described in claim 2, it is characterized in that, the background color is indicated by RGB color value；
The background color that any frame is determined according to n2, comprising:
Calculate the pixel number ratio n of any frame3=n2/n1；
The red color value of the background color of any frame isGreen color value isIt is blue
Color color value is
4. the method according to claim 1, wherein extracting the face brightness of each frame in the video, comprising:
For any frame in the video,
The brightness value of each pixel in the human face region of any frame is determined by following formula:
Wherein, k is the pixel point identification of any frame human face region, hkFor the kth pixel of any frame human face region
Brightness value, RkFor the red color value in the RGB color value of kth pixel, GkFor the green in the RGB color value of kth pixel
Color value, BkFor the Blue value in the RGB color value of kth pixel；
In the brightness value of all pixels point of any frame human face region, maximum brightness value and minimum luminance value are determined；
Calculate the luminance mean value of any frame human face regionWherein, n4For the human face region of any frame
Middle pixel total quantity；
According to maximum brightness value, minimum luminance value andDetermine the face brightness of any frame in the video.
5. according to the method described in claim 4, it is characterized in that, it is described according to maximum brightness value, minimum luminance value andReally
The face brightness of any frame in the fixed video, comprising:
Calculate the first difference d1=maximum brightness value-minimum luminance value；
Calculate the second difference
Calculate third difference
Calculate brightness ratio d4=| d1-d2 |/| d1-d3 |；
Calculate the brightness mean square deviation of any frame human face region
The face brightness of any frame is in the video
6. the method according to claim 1, wherein also being wrapped before the difference value for calculating each non-first frame
It includes:
According to the human face region difference in areas of each non-first frame, pixel number is poor, background color is poor, face luminance difference carries out just non-first frame
Sieve；
The difference value for calculating each non-first frame, comprising:
The difference value of each frame after calculating primary dcreening operation.
7. according to the method described in claim 6, it is characterized in that, the human face region difference in areas according to each non-first frame, as
Prime number is poor, background color is poor, face luminance difference carries out primary dcreening operation to non-first frame, comprising:
For any non-first frame,
If the human face region difference in areas of any non-first frame is not more than the first value, and, pixel number difference is not more than second value, and,
Background color difference is not more than third value, and, face luminance difference is not more than the 4th value, then any non-first frame passes through primary dcreening operation；Or
Person,
If the human face region difference in areas of any non-first frame is not more than the first value, but pixel number is poor, background color is poor, face is bright
Spending difference is 0, then any non-first frame passes through primary dcreening operation；Alternatively,
If the face luminance difference of any non-first frame is not more than the 4th value, but human face region difference in areas, pixel number are poor, background face
Color difference is 0, then any non-first frame passes through primary dcreening operation；
Wherein, the first value is (the sum of human face region difference in areas of all non-first frames+first frame human face region area-avg1)/institute
State the totalframes of face video, second value is (the sum of pixel number difference of all non-first frames+first frame pixel number-avg2)/described
The totalframes of face video, third value are (the sum of background color difference of all non-first frames+first frame background color-avg3)/institute
State the totalframes of face video, the 4th value be (the sum of face luminance difference of all non-first frames+first frame face brightness-avg4)/
The totalframes of the face video, the sum of human face region area of all frames of avg1=/face video totalframes, avg2
The sum of pixel number of=all frames/face video totalframes, the sum of background color of all frames of the avg3=/face
The totalframes of video, the sum of face brightness of all frames of avg4=/face video totalframes.
8. according to claim 1 to method described in 7 any claims, which is characterized in that described to pass through drop trained in advance
Dimension module carries out expressive features before dimension-reduction treatment, further includes:
Obtain sample set X, wherein total sample number is m in X, and each sample includes multiple expressive features, and each sample belongs to one
Classification；
Category classifies all samples；
Calculate all kinds of mean vectorsWherein, i is class mark, μiFor the mean vector of the i-th class, biFor the i-th class
Sample size, j is sample identification, xijThe vector formed for the expressive features of j-th of sample of the i-th class；
According to all kinds of mean vectors, grand mean vector is determinedWherein, μ0For grand mean
Vector, E are different classes of sum belonging to sample in X；
According to grand mean vector, inter-class variance vector sum variance within clusters vector is calculated；
Expressive features after determining dimensionality reduction according to the inter-class variance vector sum variance within clusters vector form dimensionality reduction model.
9. according to the method described in claim 8, it is characterized in that, described according to grand mean vector, calculating inter-class variance vector
With variance within clusters vector, comprising:
Wherein, SwFor inter-class variance vector, SbFor variance within clusters vector, XiFor the set of the i-th class sample composition.
10. according to the method described in claim 9, it is characterized in that, described according to the inter-class variance vector sum variance within clusters
Vector determines the expressive features after dimensionality reduction, comprising:
Calculate the weight vectors W=diag (S being made of each expressive features weightb·/Sw), wherein diag () is function, described
Function is used to take the element on diagonal of a matrix ,/it is operator, the operator is used for SwAnd SbCorresponding element be divided by；
By the sequence of each expressive features weight from big to small, expressive features are ranked up；
Expressive features after the forward expressive features of preset quantity sequence to be determined as to dimensionality reduction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811499959.9A CN109614927B (en) | 2018-12-10 | 2018-12-10 | Micro expression recognition based on difference of front and rear frames and characteristic dimension reduction |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811499959.9A CN109614927B (en) | 2018-12-10 | 2018-12-10 | Micro expression recognition based on difference of front and rear frames and characteristic dimension reduction |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109614927A true CN109614927A (en) | 2019-04-12 |
| CN109614927B CN109614927B (en) | 2022-11-08 |
Family
ID=66007965
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811499959.9A Active CN109614927B (en) | 2018-12-10 | 2018-12-10 | Micro expression recognition based on difference of front and rear frames and characteristic dimension reduction |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109614927B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110991458A (en) * | 2019-11-25 | 2020-04-10 | 创新奇智(北京)科技有限公司 | Artificial intelligence recognition result sampling system and sampling method based on image characteristics |
| CN112528945A (en) * | 2020-12-24 | 2021-03-19 | 上海寒武纪信息科技有限公司 | Method and device for processing data stream |
| CN113076813A (en) * | 2021-03-12 | 2021-07-06 | 首都医科大学宣武医院 | Mask face feature recognition model training method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015109693A1 (en) * | 2014-01-22 | 2015-07-30 | 中兴通讯股份有限公司 | Method and system for image color calibration |
| CN105139039A (en) * | 2015-09-29 | 2015-12-09 | 河北工业大学 | Method for recognizing human face micro-expressions in video sequence |
| CN108268859A (en) * | 2018-02-08 | 2018-07-10 | 南京邮电大学 | A kind of facial expression recognizing method based on deep learning |
-
2018
- 2018-12-10 CN CN201811499959.9A patent/CN109614927B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015109693A1 (en) * | 2014-01-22 | 2015-07-30 | 中兴通讯股份有限公司 | Method and system for image color calibration |
| CN105139039A (en) * | 2015-09-29 | 2015-12-09 | 河北工业大学 | Method for recognizing human face micro-expressions in video sequence |
| CN108268859A (en) * | 2018-02-08 | 2018-07-10 | 南京邮电大学 | A kind of facial expression recognizing method based on deep learning |
Non-Patent Citations (2)
| Title |
|---|
| 孔健等: "基于多频域LBP-TOP的人脸表情识别", 《计算机工程》 * |
| 马浩原等: "平均光流方向直方图描述的微表情识别", 《信号处理》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110991458A (en) * | 2019-11-25 | 2020-04-10 | 创新奇智(北京)科技有限公司 | Artificial intelligence recognition result sampling system and sampling method based on image characteristics |
| CN112528945A (en) * | 2020-12-24 | 2021-03-19 | 上海寒武纪信息科技有限公司 | Method and device for processing data stream |
| CN112528945B (en) * | 2020-12-24 | 2024-04-26 | 上海寒武纪信息科技有限公司 | Method and device for processing data stream |
| CN113076813A (en) * | 2021-03-12 | 2021-07-06 | 首都医科大学宣武医院 | Mask face feature recognition model training method and device |
| CN113076813B (en) * | 2021-03-12 | 2024-04-12 | 首都医科大学宣武医院 | Training method and device for mask face feature recognition model |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109614927B (en) | 2022-11-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107742107B (en) | Facial image classification method, device and server | |
| CN105160317B (en) | One kind being based on area dividing pedestrian gender identification method | |
| CN104732200B (en) | A kind of recognition methods of skin type and skin problem | |
| Mahmood et al. | EAR: Enhanced augmented reality system for sports entertainment applications | |
| CN106326874A (en) | Method and device for recognizing iris in human eye images | |
| CN110348319A (en) | A kind of face method for anti-counterfeit merged based on face depth information and edge image | |
| CN107180234A (en) | The credit risk forecast method extracted based on expression recognition and face characteristic | |
| CN111931758B (en) | Face recognition method and device combining facial veins | |
| CN104036278B (en) | The extracting method of face algorithm standard rules face image | |
| CN105426829B (en) | Video classification methods and device based on facial image | |
| CN105825183B (en) | Facial expression recognizing method based on partial occlusion image | |
| CN108229458A (en) | A kind of intelligent flame recognition methods based on motion detection and multi-feature extraction | |
| CN105354554A (en) | Color and singular value feature-based face in-vivo detection method | |
| CN106599854A (en) | Method for automatically recognizing face expressions based on multi-characteristic fusion | |
| CN107871101A (en) | A kind of method for detecting human face and device | |
| CN109815874A (en) | A kind of personnel identity recognition methods, device, equipment and readable storage medium storing program for executing | |
| CN109614927A (en) | Micro- Expression Recognition based on front and back frame difference and Feature Dimension Reduction | |
| CN109753950A (en) | Dynamic human face expression recognition method | |
| CN111563417A (en) | Pyramid structure convolutional neural network-based facial expression recognition method | |
| CN107909067A (en) | The examination system of intelligence generation answering card and mobile terminal Automated Assessment | |
| CN109740572A (en) | A kind of human face in-vivo detection method based on partial color textural characteristics | |
| CN108805094A (en) | Data enhancement methods based on artificial face | |
| CN107785061A (en) | Autism-spectrum disorder with children mood ability interfering system | |
| CN101383008A (en) | Image classification method based on visual attention model | |
| CN107633229A (en) | Method for detecting human face and device based on convolutional neural networks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |