CN109753950A - Dynamic human face expression recognition method - Google Patents
Dynamic human face expression recognition method Download PDFInfo
- Publication number
- CN109753950A CN109753950A CN201910109704.5A CN201910109704A CN109753950A CN 109753950 A CN109753950 A CN 109753950A CN 201910109704 A CN201910109704 A CN 201910109704A CN 109753950 A CN109753950 A CN 109753950A
- Authority
- CN
- China
- Prior art keywords
- human face
- face expression
- delta
- expression
- shaped region
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000008921 facial expression Effects 0.000 title claims abstract description 620
- 238000000034 method Methods 0.000 title claims abstract description 83
- 238000012549 training Methods 0.000 claims abstract description 54
- 238000000605 extraction Methods 0.000 claims abstract description 49
- 238000001514 detection method Methods 0.000 claims abstract description 10
- 230000014509 gene expression Effects 0.000 claims description 46
- 238000012360 testing method Methods 0.000 claims description 31
- 239000000284 extract Substances 0.000 claims description 24
- 239000011159 matrix material Substances 0.000 claims description 24
- 230000007935 neutral effect Effects 0.000 claims description 22
- 210000001508 eye Anatomy 0.000 claims description 15
- 210000000088 lip Anatomy 0.000 claims description 15
- 210000004709 eyebrow Anatomy 0.000 claims description 14
- 210000001331 nose Anatomy 0.000 claims description 12
- 238000002474 experimental method Methods 0.000 claims description 10
- 230000006870 function Effects 0.000 claims description 9
- 230000001174 ascending effect Effects 0.000 claims description 7
- NAWXUBYGYWOOIX-SFHVURJKSA-N (2s)-2-[[4-[2-(2,4-diaminoquinazolin-6-yl)ethyl]benzoyl]amino]-4-methylidenepentanedioic acid Chemical compound C1=CC2=NC(N)=NC(N)=C2C=C1CCC1=CC=C(C(=O)N[C@@H](CC(=C)C(O)=O)C(O)=O)C=C1 NAWXUBYGYWOOIX-SFHVURJKSA-N 0.000 claims description 6
- 238000002790 cross-validation Methods 0.000 claims description 5
- 230000008451 emotion Effects 0.000 claims description 5
- 210000005252 bulbus oculi Anatomy 0.000 claims description 3
- 238000010276 construction Methods 0.000 claims description 3
- 210000000056 organ Anatomy 0.000 claims description 3
- 210000000887 face Anatomy 0.000 claims description 2
- 230000007547 defect Effects 0.000 abstract description 11
- 230000000694 effects Effects 0.000 abstract description 8
- 238000005286 illumination Methods 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 14
- 238000013135 deep learning Methods 0.000 description 9
- 239000012141 concentrate Substances 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000009467 reduction Effects 0.000 description 6
- 238000013527 convolutional neural network Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 238000006073 displacement reaction Methods 0.000 description 4
- 238000010606 normalization Methods 0.000 description 4
- 238000011160 research Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 230000004927 fusion Effects 0.000 description 3
- 239000000203 mixture Substances 0.000 description 3
- 238000010219 correlation analysis Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000001537 neural effect Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010195 expression analysis Methods 0.000 description 1
- 230000001815 facial effect Effects 0.000 description 1
- 238000009472 formulation Methods 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Abstract
Dynamic human face expression recognition method of the present invention is related to the method for the characteristics of image of figure or characteristic for identification, is a kind of dynamic human face expression recognition method based on geometrical characteristic and semantic feature, step is: the pretreatment of dynamic human face image sequence；The detection of human face expression frame and the characteristic point of human face expression gray level image mark；The calibration of human face expression delta-shaped region on human face expression gray level image；The extraction of the geometrical characteristic of human face expression delta-shaped region on human face expression gray level image；The analysis and extraction of semantic feature on human face expression gray level image；SVM classifier training simultaneously obtains classification results；Complete the identification of dynamic human face expression.That the present invention overcomes the generally existing real-times of the prior art is poor, high vulnerable to illumination effect, intrinsic dimensionality and time complexity and then influence the satisfactory defect of facial expression recognition rate.
Description
Technical field
Technical solution of the present invention is related to the method for the characteristics of image of figure or characteristic for identification, specifically dynamic
Facial expression recognizing method.
Background technique
Human face expression is most effective way in human emotion's exchange, and with the development of computer technology, human face expression is known
There is not important application in the field for being related to NI Vision Builder for Automated Inspection and pattern-recognition, such as psychological study, video conference, intelligence
It can human-computer interaction, affection computation and medical industry.With the development in an all-round way of human-computer interaction technology, how research makes computer automatic
Perception human emotion be artificial intelligence focus.
The facial expression recognizing method of early stage concentrates on the human face expression feature in research still image.However, based on quiet
Motion information of the facial expression recognition of state image due to lacking expression, cannot reflect the space-time characteristic of expression.Human face expression is made
For a dynamic change procedure, space-time characteristic plays an important role.Dynamic human face expression recognition method can be mentioned comprehensively
The space-time characteristic of human face expression is got, the variation of human face expression itself is reflected with this, to improve the robust of facial expression recognition
Property and accuracy.
Following some document reports research of existing dynamic human face expression recognition method: document " Recognition
of facial expressions based on salient geometric features and support vector
machines”(Ghimire D,Lee J,Li Z N,et al.Recognition of facial expressions
based on salient geometric features and support vector machines[J].Multimedia
Tools&Applications, 2016:1-26.) it proposes to initialize face feature point by Elastic Bunch Graph Matching, it uses
Kanade-Lucas-Tomaci (KLT) tracker tracks characteristic point, then by using ELM as Weak Classifier
Multistage AdaBoost classifier carrys out selected characteristic, this method since the number of ELM Weak Classifier reaches 23426, feature selecting
Time loss is long.CN108256426A discloses a kind of facial expression recognizing method based on convolutional neural networks, first passes through people
The key point of face calibrates dynamic human face image sequence, is identified using convolutional neural networks to human face expression, the party
For method since the convolutional neural networks frame number of plies is too deep, the time complexity for extracting human face expression feature is high.CN108921042A is public
A kind of face sequence expression recognition method based on deep learning has been opened, has proposed to extract dynamic human face figure by deep learning frame
As the Analysis On Multi-scale Features of sequence, facial expression recognition is completed with this, this method is different resolutions due to Multi resolution feature extraction
The convolutional space-time feature of rate image sequence leads to the overlong time for extracting feature consumption.Document " Facial Expression
Recognition from Video Sequences Based on Spatial-Temporal Motion Local
Binary Pattern and Gabor Multiorientation Fusion Histogram”(Zhao L,Wang Z,
Zhang G.Facial Expression Recognition from Video Sequences Based on Spatial-
Temporal Motion Local Binary Pattern and Gabor Multiorientation Fusion
Histogram[J].Mathematical Problems in Engineering,2017,(2017-02-19),2017,
It 2017:1-12.) proposes to combine using dynamic and static information in dynamic human face image sequence and carries out facial expression analysis
It is special to extract texture using the method that both spatiotemporal motion local binary patterns (STM-LBP) and Gabor filter combine for method
Sign, it is big, high vulnerable to illumination interference and intrinsic dimensionality all to there is computation complexity in STM-LBP and Gabor algorithm itself therein
Defect.CN105139004A discloses the facial expression recognizing method based on video sequence, proposes and utilizes three-dimensional orthogonal
The center Haar-like binary pattern (HCBP-TOP) extracts the textural characteristics of dynamic human face image sequence, and this method is due to HCBP-
TOP textural characteristics are obtained in layered piecemeal treated sub-block, and sub-block quantity leads to intrinsic dimensionality height more.
CN104036255A discloses a kind of facial expression recognizing method, by comparing the people formed by characteristic point position information vector difference
The similarity of face expressive features library and expressive features vector Euclidean distance to be measured completes facial expression recognition, and this method is due to defeated
The quantity of the Facial Expression Image entered and the human face expression characteristic point of calibration is more, leads to the intrinsic dimensionality of human face expression feature vector
It is high.CN103971137B discloses the three-dimensional dynamic human face expression recognition method based on the study of structural sparse features, extracts dynamic
Training sample of the LBP-TOP textural characteristics of three-dimensional module as encoder dictionary in state human face image sequence, and using PCA to it
Feature input condition random field models after dimensionality reduction are completed training and identified by dimensionality reduction, and this method is special due to LBP-TOP texture
Sign is to extract to obtain on the three-dimensional module for dividing dynamic human face image sequence, and the quantity of three-dimensional module is more and textural characteristics
Itself high defect of existing characteristics dimension, even if intrinsic dimensionality is still excessively high after PCA dimensionality reduction.Document " Dynamic image
sequences expression recognition based on active appearance model and optical
flow”(Shao Hong,Wang Yang,Wang Wei.Dynamic image sequences expression
recognition based on active appearance model and optical flow[J].Computer
Engineering and Design, 2017,38 (6): 1642-1646.) it reports and is moved using active appearance models AAM positioning
In state human face image sequence in initial frame Facial Expression Image 68 characteristic points, recycle gaussian pyramid track these features
Point, using the difference of the peak value frame of human face expression in dynamic human face image sequence and the characteristic point coordinate of neutral frame as human face expression spy
Sign, this method because of real-time difference due to that can cause tracking effect to be deteriorated in the excessive situation of dynamic human face image sequence frame number.
CN106934375A discloses the facial expression recognizing method based on the description of characteristic point motion profile, by tracking dynamic human face figure
As characteristic point in the slope variation of interframe extracts human face expression feature in sequence, and it is inputted RBF neural and is identified with this
Human face expression, for this method since the frame number of tracked dynamic human face image sequence and the quantity of characteristic point are more, it is poor that there are real-times
Defect.CN101908149A discloses a kind of method that countenance is identified from human face image sequence, and proposition passes through tracking
The change in displacement of 20 characteristic points obtains geometrical characteristic on dynamic human face image sequence, and by canonical correlation analysis to feature into
Row analysis completes facial expression recognition with this, this method be only extracted characteristic point displacement and length these two types geometrical characteristic and not
Consider textural characteristics or semantic feature, there is a problem of that discrimination is low.Document " Spatio-temporal convolutional
features with nested LSTM for facial expression recognition”(Zhenbo Yu,
Guangcan Liu,Qingshan Liu,Jiankang Deng.Spatio-temporal convolutional
features with nested LSTM for facial expression recognition[J]
.Neurocomputing, 2018:50-57) report the multilayer apperance feature for learning human face expression using deep learning frame
With time multidate information, the problem of this method, is time complexity height.CN106980811A discloses facial expression recognition side
Method and facial expression recognition device carry out facial expression recognition, this method using the training pattern containing more deep learning frames
Since the deep learning frame used is more, have the defects that time complexity is high.
In short, tracking characteristics point effect is poor, leads to dynamic human face since the feature based on dynamic human face image sequence is complicated
Expression recognition method research the generally existing real-time of the prior art it is poor, vulnerable to illumination effect, intrinsic dimensionality and time complexity
It is high to influence the satisfactory defect of facial expression recognition rate in turn.
Summary of the invention
It is a kind of special based on geometry the technical problems to be solved by the present invention are: providing dynamic human face expression recognition method
Sign and semantic feature dynamic human face expression recognition method, overcome the generally existing real-time of the prior art it is poor, vulnerable to illumination shadow
It rings, intrinsic dimensionality and time complexity height influence the satisfactory defect of facial expression recognition rate in turn.
The present invention solves technical solution used by the technical problem: dynamic human face expression recognition method, is a kind of base
In the dynamic human face expression recognition method of geometrical characteristic and semantic feature, the specific steps are as follows:
The first step, the pretreatment of dynamic human face image sequence:
First every frame Facial Expression Image in the dynamic human face image sequence of input is carried out size to be normalized to size being M
× N pixel, then every frame Facial Expression Image in the dynamic human face image sequence inputted using following formula (1) by
Rgb space is transformed into gray space, obtains every frame human face expression gray level image Igray_tn,
Igray_tn=0.299IR+0.587IG+0.114IB(1),
In formula (1), IR、IG、IBIt is every frame Facial Expression Image in inputted dynamic human face image sequence respectively
Red, green and blue three channel components, retain every frame human face expression gray level image Igray_tn, for people in following second step
The detection of face expression frame is used with characteristic point mark；
Second step, the detection of human face expression frame and the characteristic point of human face expression gray level image mark:
Every frame human face expression that the above-mentioned first step is obtained using the Multiview_Reinforce interface in the library LibFace
Gray level image Igray_tnThe detection of human face expression frame is carried out, and characteristic point mark, this 68 spies are carried out to 68 characteristic points therein
Shown in the following formula of total coordinate vector (2) for levying point,
X=((x1,y1),(x2,y2),...,(xk,yk),...,(x68,y68))T(2),
In formula (2), xk, ykIn respectively every frame human face expression gray level image abscissa corresponding to k-th of characteristic point and
Ordinate, k ∈ [1,68]；
Third step, the calibration of human face expression delta-shaped region on human face expression gray level image:
The every frame human face expression gray level image I marked from above-mentioned second stepgray_tnIn 68 characteristic points in select eyebrow,
30 characteristic points on eyes, nose and mouth carry out the calibration of human face expression delta-shaped region on human face expression gray level image,
10 human face expression delta-shaped regions of composition calibration altogether, the vector combination TR that human face expression delta-shaped region is consequently formed is such as
Shown in lower formula (3),
TR={ tr1,tr2,...,tri,...,tr10(3),
In formula (3), tri={ X_SI, 1,X_SI, 2,X_SI, 3, i ∈ [1,10], triFor i-th of human face expression triangle
Region, X_SI, 1、X_SI, 2、X_SI, 3The vertex on the 1st, 2,3 vertex in respectively i-th of human face expression delta-shaped region is sat
Mark,
The 10 human face expression delta-shaped regions demarcated include: left eye and lip, eyes and eyebrow, in lip and eyebrow
Heart point, eyebrow and nose, nose and lip, eyebrow, eyes and nose, nose central point and the corners of the mouth, right eye eyeball and lip, pure mouth
Lip, pure eyes；
4th step, the extraction of the geometrical characteristic of human face expression delta-shaped region on human face expression gray level image:
The human face expression of every frame human face expression gray level image calibration in the dynamic human face image sequence in above-mentioned third step
It is calculated on the vector combination TR of delta-shaped region, the specific steps are as follows:
4.1st step, the human face expression delta-shaped region top of every frame human face expression gray level image in dynamic human face image sequence
The extraction of distance feature between point:
By the human face expression triangle of frame human face expression gray level image every in the dynamic human face image sequence in above-mentioned third step
I-th of human face expression delta-shaped region tr in the vector combination TR in shape regioniIn three vertex transverse and longitudinal coordinate groups be combined into for
X_SI, 1(xI, 1, yI, 1)、X_SI, 2(xI, 2, yI, 2)、X_SI, 3(xI, 3, yI, 3), respectively with following formula (4), formula (5), formula
(6) the human face expression delta of every frame human face expression gray level image in the dynamic human face image sequence in above-mentioned third step is calculated
I-th of human face expression delta-shaped region tr in the vector combination TR in domainiIn the Euclidean distance between vertex two-by-two,
Vertex X_SI, 1With vertex X_SI, 2Between Euclidean distance dI, 1The following formula of calculating (4) shown in:
Vertex X_SI, 1With vertex X_SI, 3Between Euclidean distance dI, 2The following formula of calculating (5) shown in:
Vertex X_SI, 2With vertex X_SI, 3Between Euclidean distance di,3The following formula of calculating (6) shown in:
Formula (4), (5), in (6), xi,1, xi,2, xI, 3In respectively i-th of human face expression delta-shaped region the 1st, 2,3
The abscissa on a vertex, yI, 1, yI, 2, yI, 3The the 1st, 2,3 vertex in respectively i-th of human face expression delta-shaped region it is vertical
Coordinate, i ∈ [1,10],
Thus the human face expression delta-shaped region top of every frame human face expression gray level image in dynamic human face image sequence is completed
The extraction of distance feature between point；
4.2nd step, the human face expression delta-shaped region top of every frame human face expression gray level image in dynamic human face image sequence
The extraction of the angle character of point:
Calculate the human face expression three of every frame human face expression gray level image in the dynamic human face image sequence in above-mentioned third step
I-th of human face expression delta-shaped region tr in the vector combination TR of angular domainiIn three apex coordinates angle character,
triThree vertex transverse and longitudinal coordinate groups be combined into as X_SI, 1(xI, 1, yI, 1)、X_SI, 2(xI, 2, yI, 2)、X_SI, 3(xI, 3, yI, 3), it uses
X_Si,1、X_Si,2、X_Si,3The coordinate on these three vertex calculates the angle character r on corresponding 1st vertexi,1, the 2nd top
The angle character r of pointI, 2, the 3rd vertex angle character rI, 3Formula (7), formula (8), formula (9) as follows,
In formula (7), formula (8) and formula (9), xi,1、xi,2、xI, 3In respectively i-th of human face expression delta-shaped region
The the 1st, 2,3 vertex abscissa, yI, 1, yI, 2, yI, 3The the 1st, 2,3 in respectively i-th of human face expression delta-shaped region
The ordinate on vertex, i ∈ [1,10]；
4.3rd step, the human face expression delta-shaped region of every frame human face expression gray level image in dynamic human face image sequence
The extraction of geometrical characteristic:
The geometrical characteristic is made of distance feature and angle character, the dynamic human face figure that above-mentioned 4.1st step is obtained
As the vector of the human face expression delta-shaped region of the human face expression gray level image of the neutral frame in sequence combines i-th of face in TR
Expression delta-shaped region triDistance feature and the neutral frame in the obtained dynamic human face image sequence of above-mentioned 4.2nd step people
I-th of human face expression delta-shaped region tr in the vector combination TR of the human face expression delta-shaped region of face expression gray level imagei's
Angle character is worth i.e. for six totally: di,1, di,2, di,3, ri,1, rI, 2,ri,3It is stored in vector hiIn, h herei={ di,1,di,2,di,3,
ri,1,ri,2,ri,3, wherein [1,10] i ∈, by the people of the peak value frame with above-mentioned neutral frame in same dynamic human face image sequence
I-th of human face expression delta-shaped region tr in the vector combination TR of the human face expression delta-shaped region of face expression gray level imagei's
Distance feature and angle character are worth i.e. for six totally: di,1, di,2, di,3, ri,1, ri,2, ri,3It is stored in vector wiIn, wi={ di,1,di,2,
di,3,ri,1,ri,2,ri,3, wherein [1,10] i ∈, by vector wiWith vector hiCharacteristic Ratios be put into array z, z=
{z(i-1)×6+j, z(i-1)×6+j=w(i-1),j/h(i-1),j, wherein h is the face table of human face expression gray level image in above-mentioned neutral frame
The distance feature of all delta-shaped regions and the total value of angle character in the vector combination TR of feelings delta-shaped region, w is above-mentioned peak
It is worth the distance of all delta-shaped regions in the vector combination TR of the human face expression delta-shaped region of human face expression gray level image in frame
The total value of feature and angle character, z are the human face expression delta-shaped region of human face expression gray level image in peak value frame and neutral frame
Vector combination TR in the distance feature of all delta-shaped regions and the ratio of angle character value, i ∈ [1,10], j ∈ [0,5],
So far it is fully completed the human face expression delta-shaped region combination TR of the every frame human face expression gray level image of this dynamic human face image sequence
Extraction of Geometrical Features；
The extraction of the geometrical characteristic of all dynamic human face image sequences in 4.4th step, training set and test set:
Circulation executes the operation of above-mentioned the 4.1st step to the 4.3rd step, i.e. by six class human face expressions in training set: surprised, evil
Use of the array z storage that fearness, glad, sad, detest and angry corresponding each dynamic human face image sequence obtain to training set
The human face expression geometrical characteristic vector f of SVM classifier is trained, six different human face expressions being obtained by six class human face expressions
Geometrical characteristic vector f forms six class human face expression collection, then i.e. by six class human face expressions in test set: it is surprised, fear, be glad,
The array z storage that sad, detest and angry corresponding each dynamic human face image sequence obtain is used to test SVM to test set
The human face expression geometrical characteristic vector te of disaggregated model, the six different human face expression geometry obtained by six class human face expressions are special
Vector te is levied, also forms six class human face expression collection, above-mentioned two place refers to that six class human face expression collection are the six class people for being combined into one
Face expression collection so far completes training until all dynamic human face image sequences circulation in training set and test set executes completion
The Extraction of Geometrical Features of all dynamic human face image sequences in collection and test set；
5th step, the analysis and extraction of the semantic feature on human face expression gray level image:
The human face expression of human face expression geometrical characteristic vector f and test set to training set obtained in above-mentioned 4th step is several
What feature vector te carries out semantic analysis, to realize the analysis and extraction of the semantic feature on human face expression gray level image, specifically
Steps are as follows:
5.1st step constructs human face expression semantic characteristics description set:
Defining human face expression semanteme is a kind of natural language description to human face expression feature, including to human face expression feature
In geometric shape, these dominant and recessive attributes of the relative position and emotion of Different Organs explanation.
IfGather for one, when human face expression has A human face expression
Feature, a are wherein a-th of human face expression feature, and when a-th of human face expression feature is made of U attribute, u is u kind therein
Attribute, when u attribute is made of the different strength grade of B kind, b is b kind grade therein, then claimsFor a face
Expression semanteme, Μ are a face expression semanteme description collection.
Formulate it is following it is small, in, the strength grade of big three kinds of descriptions human face expression semantic feature, human face expression semanteme is abbreviated
For ma,b, wherein a is a-th of human face expression semantic feature, and a ∈ [1,59], b are strength grade, b ∈ [1,3], wherein 1,2,3 point
Do not represent it is small, in, big strength grade,
On above-mentioned 4th step human face expression gray level image after the completion of the extraction of the geometrical characteristic of human face expression delta-shaped region,
Utilize face in the human face expression delta-shaped region vector combination TR divided in frame human face expression every in dynamic human face image sequence
The side length and angle of expression delta-shaped region construct human face expression semantic characteristics description set YU, shown in following formula (10),
YU={ yu1,yu2,...,yun,...,yu591≤n≤59 (10),
In formula (10), YU is the conjunction of face expression semanteme characteristic descriptor set, yunFor the conjunction of face expression semanteme characteristic descriptor set
In to human face expression delta-shaped region vector combination TR in n-th of side length of human face expression delta-shaped region and the language of angle character
Justice description, n are the quantity of face expression semanteme feature description；
It include human face expression delta-shaped region vector described in above-mentioned third step in human face expression semantic feature set YU
It combines in 10 human face expression delta-shaped regions in TR according to six of each delta-shaped region of above-mentioned 4th step away from walk-off angle
Degree feature carries out the human face expression semantic feature of 60 face expressive features that semantic description obtains, due to wherein two face tables
The description of feelings semantic feature is identical, therefore omits 1 face expression semanteme feature；
5.2nd step formulates semantic feature strength grade decision rule:
All semantic feature values in the human face expression geometrical characteristic vector f of training set in above-mentioned 4.4th step are risen
Sequence sequence, obtains semantic feature range set PF, PF={ pf1,pf2,...,pfv,...pf59, 1 <=<=59 v wherein pfv
For each semantic description yunCorresponding characteristic range, by pfvSemantic feature strength grade is divided according to the size of f/3, it is semantic special
Sign strength grade defines identical with the 5.1st step；
5.3rd step, to all human face expression triangles in frame human face expression gray level image every in dynamic human face image sequence
Region carries out semantic analysis:
Calculate every frame human face expression grayscale image in each dynamic human face image sequence in the training set in above-mentioned 4.4th step
I-th of human face expression delta-shaped region tr in the vector combination TR of the human face expression delta-shaped region of pictureiHuman face expression is semantic special
Mean value TR_AVG and standard deviation TR_SD is levied,
Shown in the following formula of the calculating of mean value TR_AVG (11):
TR_AVG=(d0+i×6+d1+i×6+d2+i×6+r3+i×6+r4+i×6+r5+i×6)/6 (11),
Shown in the following formula of the calculating of standard deviation TR_SD (12):
D in formula (11), (12)j+i×6For the people of every frame human face expression gray level image in each dynamic human face image sequence
I-th of human face expression delta-shaped region tr in the vector combination TR of face expression delta-shaped regioniJ-th interior of distance feature value,
rm+i×6For the Vector Groups of the human face expression delta-shaped region of every frame human face expression gray level image in each dynamic human face image sequence
Close i-th of human face expression delta-shaped region tr in TRiM-th interior of angle character value, wherein setting i ∈ [1,10], j ∈ [0,
2], thus all human face expression triangles in every frame human face expression gray level image are completed in dynamic human face image sequence in [3,5] m ∈
The semantic analysis in shape region；
5.4th step obtains the optimal human face expression delta-shaped region combination of six class human face expression collection:
Human face expression triangle in each dynamic human face image sequence is concentrated to six class human face expressions in the 4.4th step first
Region vector combines 10 human face expression delta-shaped region number consecutivelies in TR, the standard deviation then obtained according to above-mentioned 5.3 step
TR_SD carries out ascending sort to 10 human face expression delta Field Numbers in the TR, selects 5 before ranking human face expressions three
Then angular domain number counts before the six class human face expressions concentration ranking in above-mentioned 4.4th step 5 number quantity,
It obtains 5 most numbers of shared number quantity, ascending sort is finally carried out to this 5 numbers according to standard deviation TR_SD, is obtained
The optimal human face expression delta-shaped region of six class human face expression collection combines；
5.5th step extracts the final semantic feature of six class human face expression collection:
According to the above-mentioned prepared semantic feature strength grade decision rule of 5.2nd step, to six of the acquisition in the 5.4th step
The semantic feature intensity of human face expression delta-shaped region in the optimal human face expression delta-shaped region combination of class human face expression collection
Grade is determined, and counts each semantic feature strength grade quantity shared by six class human face expression collection, selects shared quantity
Most strength grade, as corresponding semantic description yunCorresponding strength grade, final circulation executes aforesaid operations, to six classes
All semantic description yu in human face expression collectionnThe shared quantity of corresponding strength grade is counted, and then obtains six class faces
All semantic description yu in expression collectionnCorresponding strength grade extracts the final semantic feature of six class human face expression collection with this；
So far the analysis of the semantic feature on human face expression gray level image all terminates with extraction；
6th step, SVM classifier training simultaneously obtain classification results:
By the analysis of the semantic feature on above-mentioned 5th step human face expression gray level image and extract obtained semantic feature
Data input SVM classifier is trained and predicts, judges that the dynamic human face image sequence inputted in the above-mentioned first step belongs to
Which class human face expression takes the average result of experiment as final facial expression recognition rate using ten times of cross-validation methods, specific to grasp
It is as follows to make process:
(6.1) by the analysis of the semantic feature on above-mentioned 5th step human face expression gray level image and the obtained semanteme of extraction
Characteristic inputs SVM classifier training, is constructed according to the human face expression geometrical characteristic vector f of the training set of above-mentioned 4.4th step
The semantic feature matrix of training sample out, further according to the human face expression geometrical characteristic vector te structure of the test set of above-mentioned 4.4th step
The semantic feature matrix of test sample is produced, then according to its corresponding trained classification of the semantic feature matrix construction of training sample
Sample matrix, the value in the training sample classification matrix are human face expression classification；
(6.2) linear kernel function is used, the type of stopping criterion for iteration 100, SVM classifier uses C_SVC, first will instruction
The semantic feature matrix, training classification sample matrix and parameter for practicing sample are sent into the train function of SVM classifier, are classified
Model, then will be predicted in the predict function of the semantic feature Input matrix of test sample to the disaggregated model, it is thus complete
At SVM classifier training and obtain classification results, then in the library CK+ and the library MMI experiment obtain it is surprised, fear, be glad, sad,
Detest the classification results with angry six kinds of human face expressions；
Thus the identification of dynamic human face expression is completed.
Above-mentioned dynamic human face expression recognition method, wherein the gray scale normalization algorithm used, geometrical normalization algorithm,
Libface Face datection and characteristic point dimensioning algorithm and SVM classifier are all well-known in the art.
The invention has the advantages that compared with prior art, remarkable result of the invention and superiority are as follows:
(1) dynamic human face expression recognition method of the present invention is a kind of dynamic human face table based on geometrical characteristic and semantic feature
Feelings recognition methods, by marking the characteristic point of face key position, in every frame human face expression gray scale of dynamic human face image sequence
The combination of expression human face expression delta-shaped region is demarcated on image, and distance is then extracted respectively to each human face expression delta-shaped region
Then feature and angle character are carried out using the distance feature and angle character ratio of neutral frame and peak value frame as geometrical characteristic
Semantic analysis finally extracts semantic feature.Based on the geometrical characteristic of human face expression frame, for dynamic human face image sequence face
Key position carries out multidate information extraction, can not only reduce time complexity, moreover it is possible to improve for the face as caused by age difference
Portion's scale, size, the robustness of cephalad direction and texture variations, extracted semantic feature, to the robust of facial expression recognition
Property is good, not only reduces intrinsic dimensionality and time complexity and also improves facial expression recognition rate.
(2) the selected geometrical characteristic based on human face expression delta-shaped region of the present invention is by dynamic human face image sequence
It is every in distance feature and dynamic human face image sequence between the human face expression delta-shaped region vertex of every frame human face expression gray level image
The angle character on the human face expression delta-shaped region vertex of frame human face expression gray level image is constituted, by calculating dynamic human face image
In sequence between the human face expression delta-shaped region vertex of every frame human face expression gray level image in distance and dynamic human face image sequence
The tracking result that the difference of the angle on the human face expression delta-shaped region vertex of every frame human face expression gray level image obtains, for by
The variation of face scale caused by age is different, size, cephalad direction and texture is robust, and two kinds of the method for the present invention calculating is several
What feature, that is, distance feature and angle character step are easy, and operation time is short.
(3) human face expression geometrical characteristic proposed by the present invention is the spy of dynamic human face image sequence kind peak value frame and neutral frame
Ratio is levied, feature only has 60 dimensions, and dimension reduces half again after by semantic analysis, reduces intrinsic dimensionality.
(4) present invention carries out semantic description to geometrical characteristic, and counts to the geometrical characteristic after semantic description, makes
The decision rule of semantic feature strength grade is determined；Semantic analysis is carried out to the geometrical characteristic after semantic description, obtains area
Optimal human face expression delta-shaped region combination in domain, and according to the decision rule of formulation to human face expression triangle optimal in region
The strength grade of the semantic feature of region combination is determined, final semantic feature is obtained, experiments verify that the present invention can be into one
Step improves dynamic human face Expression Recognition rate.
(5) present invention is compared with CN108256426A: CN108256426A is by the key point of face to human face expression figure
Human face expression is identified using convolutional neural networks after being calibrated as sequence.Due to convolutional neural networks frame number of plies mistake
It is deep, cause the time complexity for extracting face characteristic high, and the geometrical characteristic that the present invention extracts is according to peak value frame and neutral frame
Characteristic Ratios obtain, computation complexity is low.
(6) present invention is compared with CN108921042A: CN108921042A extracts dynamic human face by deep learning frame
The Analysis On Multi-scale Features of image sequence complete facial expression recognition with this.What it is due to Multi resolution feature extraction is different resolution figure
As the convolutional space-time feature of sequence, algorithm elapsed time itself is long to cause time complexity high, and the face table that the present invention extracts
The geometrical characteristic of feelings delta-shaped region combination only calculates in frame distance and angle character value in human face expression delta-shaped region, is not required to
Convolutional space-time feature is extracted, time loss is reduced, CN108921042A is lower than on time complexity.
(7) present invention is compared with CN105139004A: CN105139004A passes through the center three-dimensional orthogonal Haar-like two-value
Mode (HCBP-TOP) extracts the textural characteristics of human face expression sequence, since HCBP-TOP textural characteristics are at layered piecemeal
It is obtained in sub-block after reason, sub-block quantity causes the intrinsic dimensionality extracted high more, the human face expression triangle that the present invention extracts
The intrinsic dimensionality of the geometrical characteristic of region combination only has 60 dimensions, and only original in the semantic feature obtained after semantic analysis
The half of geometrical characteristic dimension, therefore, the method for the present invention reduces intrinsic dimensionality.
(8) present invention is compared with CN104036255A: CN104036255A is by comparison by dynamic human face image sequence
The expressive features library and test the similarity of expressive features vector Euclidean distance to complete that characteristic point position information vector difference is formed
Facial expression recognition, but since the quantity of the human face characteristic point of the facial expression image and calibration of input is excessive, cause expressive features to
The intrinsic dimensionality of amount is high, and the present invention only keeps track peak value frame and human face expression triangle in neutral frame in dynamic human face image sequence
The ratio of the angle and distance of region combination, and the intrinsic dimensionality extracted only has 60 dimensions to obtain after further semantic analysis
Semantic feature be only original half, further reduced intrinsic dimensionality.
(9) present invention is compared with CN106934375A: CN106934375A passes through special on tracking dynamic human face image sequence
Sign point extracts expressive features in the slope variation of interframe, and is inputted RBF neural to identify human face expression, due to tracking
The frame number of dynamic human face image sequence and the quantity of characteristic point are excessive, there is a problem of real-time difference, and the present invention not only with
CN106934375A is less than on the frame number of the facial image of track, in the characteristic point quantity of selected human face expression delta-shaped region
Also well below CN106934375A, the type for the aspect ratio CN106934375A that the present invention extracts is more, including distance, angle and
Semantic feature, therefore the present invention ensure that real-time while improving the effect of tracking dynamic human face image sequence.
(10) present invention is compared with CN101908149A: CN101908149A is proposed through tracking dynamic human face image sequence
The change in displacement of upper 20 characteristic points extracts geometrical characteristic, and is analyzed it by canonical correlation analysis and complete face with this
Expression Recognition, due to this method be only extracted characteristic point displacement and both geometrical characteristics of length and do not consider textural characteristics or
Semantic feature causes this method to there is a problem of that discrimination is low, and the present invention not only allows for human face expression delta-shaped region group
Two kinds of geometrical characteristics of angle and distance in conjunction, have also extracted advanced semantic feature, have improved the discrimination of human face expression.
(11) present invention is compared with CN106980811A: CN106980811A uses the instruction containing multiclass deep learning frame
Practice model to carry out facial expression recognition, since the deep learning frame used is excessive, there is a problem of that time complexity is high, this
The semantic feature dimension for the human face expression delta-shaped region combination that invention is extracted only 30 is tieed up, and is calculated simple and is not needed to establish deep layer
Deep learning frame identify human face expression, the recognition effect that need to only use SVM classifier that can just be got well reduces the time
Complexity.
(12) CN103971137B extracts the LBP-TOP textural characteristics of three-dimensional module in dynamic human face image sequence as volume
The training sample of code word allusion quotation, and using PCA to its dimensionality reduction.By the feature input condition random field models after dimensionality reduction to complete to train
And identification.Since LBP-TOP textural characteristics are to extract to obtain on the three-dimensional module for dividing dynamic human face image sequence, three
The defect that the quantity of dimension module is more and textural characteristics existing characteristics dimension itself is high, even if intrinsic dimensionality is still after PCA dimensionality reduction
It is excessively high, and the geometrical characteristic dimension for the dynamic human face image sequence that the present invention extracts only 60 is tieed up, and is being mentioned after semantic analysis
The semantic feature taken reduces the geometrical characteristic dimension of half, therefore, the defect that the method for the present invention can overcome intrinsic dimensionality high.
(13) a kind of small sample face identification method of CN106529447A, face of the CN105139004A based on video sequence
A kind of recognition methods of human face expression of expression recognition method, CN105069447B, CN105139039B human face in video frequency sequence are micro-
The classification and recognition methods of the human face expression of recognition methods, CN106127196A based on dynamic texture feature of expression,
The recognition methods of the micro- facial expression image sequence of face and CN106599854A are based on mostly special in CN106548149A monitor video sequence
The human face expression automatic identifying method of sign fusion is the previous patented technology of the present inventor team, in practice also
It is poor that there are robustness, and intrinsic dimensionality and time complexity are higher, the lower defect of facial expression recognition rate, in order to overcome these
Defect, the present inventor team continue the face identification method technology for researching and developing update with great concentration, just by creative labor
Develop dynamic human face expression recognition method of the invention.It is obtained on the basis of six patented technologies of above-mentioned previous application
The claimed technical solution of the method for the present invention is not that those skilled in the art can obtain easily.
Detailed description of the invention
Present invention will be further explained below with reference to the attached drawings and examples.
Fig. 1 is the schematic process flow diagram of the method for the present invention.
Fig. 2 is the schematic diagram for carrying out characteristic point mark in the method for the present invention using Libface on human face expression frame.
Fig. 3 is that characteristic point annotation process and warp are first on two surprised Facial Expression Images in data set in the method for the present invention
The schematic diagram of the characteristic point of beginningization.
Fig. 4 be in the method for the present invention human face expression delta-shaped region calibration after the completion of result and extract human face expression triangle
The process schematic of geometrical characteristic in region.
Quantity accounting shared by human face expression delta-shaped region Happy semantic analysis ranking feature of the Fig. 5 (a) for before ranking 5
Histogram.
Quantity accounting shared by human face expression delta-shaped region Angry semantic analysis ranking feature of the Fig. 5 (b) for before ranking 5
Histogram.
Quantity shared by human face expression delta-shaped region Disgust semantic analysis ranking feature of the Fig. 5 (c) for before ranking 5 accounts for
Compare histogram.
Quantity accounting column shared by human face expression delta-shaped region Sad semantic analysis ranking feature of the Fig. 5 (d) for before ranking 5
Shape figure.
Quantity accounting column shared by human face expression delta-shaped region Fear semantic analysis ranking feature of the Fig. 5 (e) for before ranking 5
Shape figure.
Quantity shared by human face expression delta-shaped region Surprise semantic analysis ranking feature of the Fig. 5 (f) for before ranking 5 accounts for
Compare histogram.
Fig. 6 is that the method for the present invention divides geometrical characteristic and semantic feature extraction method through SVM on CK+ and MMI data set
Recognition effect comparison diagram after class.
Specific embodiment
Embodiment illustrated in fig. 1 shows that the process of the method for the present invention is:
Embodiment illustrated in fig. 2, which is shown in the present invention, carries out the inspection of human face expression frame to human face expression sequence using Libface
After survey and mark the schematic diagram of characteristic point.
Embodiment illustrated in fig. 3, which is shown, carries out characteristic point mark to two surprised human face expression sequences that data are concentrated, and
Therefrom observe the variation of characteristic point in face facial expression image, and the characteristic point schematic diagram by initialization.I.e. pure
The characteristic point marked under white background, the diagram can intuitively show that characteristic point changes the best part in surprised expression.
After embodiment illustrated in fig. 4 shows characteristic point mark, 10 human face expressions three being demarcated on Facial Expression Image
The schematic diagram of angular domain and human face expression delta-shaped region distance and angle character, wherein A, B, C are respectively delta
The vertex in domain, AB, AC and BC are respectively the side length calculated distance feature of vertex A, B, C two-by-two between vertex, and α, β and γ
The angle character that the interior angle of respectively vertex A, B, C is calculated.
Embodiment illustrated in fig. 5 show the method for the present invention in six class human face expression frames human face expression delta-shaped region it is several
After what feature carries out semantic analysis, each human face expression delta Field Number exists in human face expression delta-shaped region vector combination TR
Human face expression delta-shaped region quantity accounting histogram in standard deviation ranking TOP V is counted most by shared quantity
Excellent human face expression delta-shaped region combination.
Fig. 5 (a) illustrated embodiment is that Happy expression sequence human face expression delta-shaped region vector after semantic analysis combines
Each human face expression delta Field Number 5 quantity accounting schematic diagrames before standard deviation ranking in TR, according to human face expression three
Each human face expression delta Field Number quantity accounting is ranked up in angular domain vector combination TR, last Happy expression sequence
Optimal human face expression delta-shaped region combination is arranged as tr from small to large in column9, tr1, tr5, tr2, tr6, wherein tr0-tr9Respectively
Indicate the corresponding number of 10 delta-shaped regions, the numerical value in ordinate indicates each delta Field Number in Happy face table
Feelings concentrate shared quantitative value, first five ranking of the standard deviation that abscissa indicates, each delta-shaped region is compiled in each standard deviation ranking
Quantity shared by number sums to the population size of happy human face expression collection.
Fig. 5 (b) illustrated embodiment is that Angry expression sequence human face expression delta-shaped region vector after semantic analysis combines
Each human face expression delta Field Number 5 quantity accounting schematic diagrames before standard deviation ranking in TR, according to human face expression three
Each human face expression delta Field Number quantity accounting is ranked up in angular domain vector combination TR, last Angry expression sequence
Optimal human face expression delta-shaped region combination is arranged as tr from small to large in column1, tr2, tr7, tr6, tr3, wherein tr0-tr9Respectively
Indicate the corresponding number of 10 delta-shaped regions, the numerical value in ordinate indicates each delta Field Number in Angry face table
Feelings concentrate shared quantitative value, first five ranking of the standard deviation that abscissa indicates, each delta-shaped region is compiled in each standard deviation ranking
Quantity shared by number sums to the population size of Angry human face expression collection.
Fig. 5 (c) illustrated embodiment is Disgust expression sequence human face expression delta-shaped region Vector Groups after semantic analysis
Each human face expression delta Field Number 5 quantity accounting schematic diagrames before standard deviation ranking in TR are closed, according to human face expression
Each human face expression delta Field Number quantity accounting is ranked up in delta-shaped region vector combination TR, last Disgust table
Optimal human face expression delta-shaped region combination is arranged as tr from small to large in feelings sequence8, tr6, tr2, tr3, tr4, wherein tr0-tr9
The corresponding number of 10 delta-shaped regions is respectively indicated, the numerical value in ordinate indicates each delta Field Number in Disgust
Human face expression concentrates shared quantitative value, first five ranking of the standard deviation that abscissa indicates, each triangle in each standard deviation ranking
Quantity shared by zone number sums to the population size of Disgust human face expression collection.
Fig. 5 (d) illustrated embodiment is that Sad expression sequence human face expression delta-shaped region vector after semantic analysis combines TR
Interior each human face expression delta Field Number 5 quantity accounting schematic diagrames before standard deviation ranking, according to human face expression triangle
Each human face expression delta Field Number quantity accounting is ranked up in shape region vector combination TR, in last Sad expression sequence
Optimal human face expression delta-shaped region combination is arranged as tr from small to large2, tr1, tr6, tr3, tr9, wherein tr0-tr9It respectively indicates
The corresponding number of 10 delta-shaped regions, the numerical value in ordinate indicate that each delta Field Number is concentrated in Sad human face expression
Shared quantitative value, standard deviation first five the ranking that abscissa indicates, in each standard deviation ranking shared by each delta Field Number
Quantity sums to the population size of Sad human face expression collection.
Fig. 5 (e) illustrated embodiment is that Fear expression sequence human face expression delta-shaped region vector after semantic analysis combines
Each human face expression delta Field Number 5 quantity accounting schematic diagrames before standard deviation in TR, according to human face expression triangle
Each human face expression delta Field Number quantity accounting is ranked up in region vector combination TR, in last Fear expression sequence most
Excellent human face expression delta-shaped region combination is arranged as tr from small to large9, tr8, tr1, tr2, tr6, wherein tr0-tr9Respectively indicate 10
The corresponding number of a delta-shaped region, the numerical value in ordinate indicate that each delta Field Number concentrates institute in Fear human face expression
Account for quantitative value, first five ranking of the standard deviation that abscissa indicates, number shared by each delta Field Number in each standard deviation ranking
Amount sums to the population size of Fear human face expression collection.
Fig. 5 (f) illustrated embodiment is Surprise expression sequence human face expression delta-shaped region vector after semantic analysis
Each human face expression delta Field Number 5 quantity accounting schematic diagrames before standard deviation ranking in TR are combined, according to face table
Each human face expression delta Field Number quantity accounting is ranked up in feelings delta-shaped region vector combination TR, last Surprise
Optimal human face expression delta-shaped region combination is arranged as tr from small to large in expression sequence1, tr5, tr2, tr3, tr9, wherein tr0-
tr9The corresponding number of 10 delta-shaped regions is respectively indicated, the numerical value in ordinate indicates that each delta Field Number exists
Surprise human face expression concentrates shared quantitative value, first five ranking of the standard deviation that abscissa indicates, in each standard deviation ranking
Quantity shared by each delta Field Number sums to the population size of Surprise human face expression collection.
Embodiment illustrated in fig. 6 shows that the method for the present invention is special by geometrical characteristic method and semanteme on CK+ and MMI data set
Levy effect contrast figure of the extracting method through SVM training and classifying, wherein the numerical value of ordinate indicate whether or not there is semantic analysis in CK+ and
Discrimination on MMI data set, abscissa indicate the number of 10 experiments, are divided into 1 between abscissa, are divided between ordinate
5%, the minimum discrimination of ordinate is 50%.
Embodiment 1
A kind of dynamic human face expression recognition method based on geometrical characteristic and semantic feature of the present embodiment, specific steps are such as
Under:
The first step, the pretreatment of dynamic human face image sequence:
Every frame Facial Expression Image progress size in the dynamic human face image sequence of input, which is first normalized to size, is
Then every frame Facial Expression Image in the dynamic human face image sequence inputted is used following formula by 640 × 480 pixels
(1) gray space is transformed by rgb space, obtains every frame human face expression gray level image Igray_tn,
Igray_tn=0.299IR+0.587IG+0.114IB(1),
In formula (1), IR、IG、IBIt is every frame Facial Expression Image in inputted dynamic human face image sequence respectively
Red, green and blue three channel components, retain every frame human face expression gray level image Igray_tn, for people in following second step
The detection of face expression frame is used with characteristic point mark；
Second step, the detection of human face expression frame and the characteristic point of human face expression gray level image mark:
Every frame human face expression that the above-mentioned first step is obtained using the Multiview_Reinforce interface in the library LibFace
Gray level image Igray_tnThe detection of human face expression frame is carried out, and characteristic point mark, this 68 spies are carried out to 68 characteristic points therein
Shown in the following formula of total coordinate vector (2) for levying point,
X=((x1,y1),(x2,y2),...,(xk,yk),...,(x68,y68))T(2),
In formula (2), xk, ykIn respectively every frame human face expression gray level image abscissa corresponding to k-th of characteristic point and
Ordinate, k ∈ [1,68]；
Third step, the calibration of human face expression delta-shaped region on human face expression gray level image:
The every frame human face expression gray level image I marked from above-mentioned second stepgray_tnIn 68 characteristic points in select eyebrow,
30 characteristic points on eyes, nose and mouth carry out the calibration of human face expression delta-shaped region on human face expression gray level image,
10 human face expression delta-shaped regions of composition calibration altogether, the vector combination TR that human face expression delta-shaped region is consequently formed is such as
Shown in lower formula (3),
TR={ tr1,tr2,...,tri,...,tr10(3),
In formula (3), tri={ X_SI, 1,X_SI, 2,X_SI, 3, i ∈ [1,10], triFor i-th of human face expression triangle
Region, X_SI, 1、X_SI, 2、X_SI, 3The vertex on the 1st, 2,3 vertex in respectively i-th of human face expression delta-shaped region is sat
Mark,
The 10 human face expression delta-shaped regions demarcated include: left eye and lip, eyes and eyebrow, in lip and eyebrow
Heart point, eyebrow and nose, nose and lip, eyebrow, eyes and nose, nose central point and the corners of the mouth, right eye eyeball and lip, pure mouth
Lip, pure eyes；
4th step, the extraction of the geometrical characteristic of human face expression delta-shaped region on human face expression gray level image:
The human face expression of every frame human face expression gray level image calibration in the dynamic human face image sequence in above-mentioned third step
It is calculated on the vector combination TR of delta-shaped region, the specific steps are as follows:
4.1st step, the human face expression delta-shaped region top of every frame human face expression gray level image in dynamic human face image sequence
The extraction of distance feature between point:
By the human face expression triangle of frame human face expression gray level image every in the dynamic human face image sequence in above-mentioned third step
I-th of human face expression delta-shaped region tr in the vector combination TR in shape regioniIn three vertex transverse and longitudinal coordinate groups be combined into for
X_SI, 1(xI, 1, yI, 1)、X_SI, 2(xI, 2, yI, 2)、X_SI, 3(xI, 3, yI, 3), respectively with following formula (4), formula (5), formula
(6) the human face expression delta of every frame human face expression gray level image in the dynamic human face image sequence in above-mentioned third step is calculated
I-th of human face expression delta-shaped region tr in the vector combination TR in domainiIn the Euclidean distance between vertex two-by-two,
Vertex X_SI, 1With vertex X_SI, 2Between Euclidean distance dI, 1The following formula of calculating (4) shown in:
Vertex X_SI, 1With vertex X_SI, 3Between Euclidean distance dI, 2The following formula of calculating (5) shown in:
Vertex X_SI, 2With vertex X_SI, 3Between Euclidean distance di,3The following formula of calculating (6) shown in:
Formula (4), (5), in (6), xi,1, xi,2, xI, 3In respectively i-th of human face expression delta-shaped region the 1st, 2,3
The abscissa on a vertex, yI, 1, yI, 2, yI, 3The the 1st, 2,3 vertex in respectively i-th of human face expression delta-shaped region it is vertical
Coordinate, i ∈ [1,10],
Thus the human face expression delta-shaped region top of every frame human face expression gray level image in dynamic human face image sequence is completed
The extraction of distance feature between point；
4.2nd step, the human face expression delta-shaped region top of every frame human face expression gray level image in dynamic human face image sequence
The extraction of the angle character of point:
Calculate the human face expression three of every frame human face expression gray level image in the dynamic human face image sequence in above-mentioned third step
I-th of human face expression delta-shaped region tr in the vector combination TR of angular domainiIn three apex coordinates angle character,
triThree vertex transverse and longitudinal coordinate groups be combined into as X_SI, 1(xI, 1, yI, 1)、X_SI, 2(xI, 2, yI, 2)、X_SI, 3(xI, 3, yI, 3), it uses
X_Si,1、X_Si,2、X_Si,3The coordinate on these three vertex calculates the angle character r on corresponding 1st vertexi,1, the 2nd top
The angle character r of pointI, 2, the 3rd vertex angle character rI, 3Formula (7), formula (8), formula (9) as follows,
In formula (7), formula (8) and formula (9), xi,1、xi,2、xI, 3In respectively i-th of human face expression delta-shaped region
The the 1st, 2,3 vertex abscissa, yI, 1, yI, 2, yI, 3The the 1st, 2,3 in respectively i-th of human face expression delta-shaped region
The ordinate on vertex, i ∈ [1,10]；
4.3rd step, the human face expression delta-shaped region of every frame human face expression gray level image in dynamic human face image sequence
The extraction of geometrical characteristic:
The geometrical characteristic is made of distance feature and angle character, the dynamic human face figure that above-mentioned 4.1st step is obtained
As the vector of the human face expression delta-shaped region of the human face expression gray level image of the neutral frame in sequence combines i-th of face in TR
Expression delta-shaped region triDistance feature and the neutral frame in the obtained dynamic human face image sequence of above-mentioned 4.2nd step people
I-th of human face expression delta-shaped region tr in the vector combination TR of the human face expression delta-shaped region of face expression gray level imagei's
Angle character is worth i.e. for six totally: di,1, di,2, di,3, ri,1, ri,2, ri,3It is stored in vector hiIn, h herei={ di,1,di,2,di,3,
ri,1,ri,2,ri,3, wherein [1,10] i ∈, by the people of the peak value frame with above-mentioned neutral frame in same dynamic human face image sequence
I-th of human face expression delta-shaped region tr in the vector combination TR of the human face expression delta-shaped region of face expression gray level imagei's
Distance feature and angle character are worth i.e. for six totally: di,1, di,2, di,3, ri,1, ri,2, ri,3It is stored in vector wiIn, wi={ di,1,di,2,
di,3,ri,1,ri,2,ri,3, wherein [1,10] i ∈, by vector wiWith vector hiCharacteristic Ratios be put into array z, z=
{z(i-1)×6+j, z(i-1)×6+j=w(i-1),j/h(i-1),j, wherein h is the face table of human face expression gray level image in above-mentioned neutral frame
The distance feature of all delta-shaped regions and the total value of angle character in the vector combination TR of feelings delta-shaped region, w is above-mentioned peak
It is worth the distance of all delta-shaped regions in the vector combination TR of the human face expression delta-shaped region of human face expression gray level image in frame
The total value of feature and angle character, z are the human face expression delta-shaped region of human face expression gray level image in peak value frame and neutral frame
Vector combination TR in the distance feature of all delta-shaped regions and the ratio of angle character value, i ∈ [1,10], j ∈ [0,5],
So far it is fully completed the human face expression delta-shaped region combination TR of the every frame human face expression gray level image of this dynamic human face image sequence
Extraction of Geometrical Features；
The extraction of the geometrical characteristic of all dynamic human face image sequences in 4.4th step, training set and test set:
Circulation executes the operation of above-mentioned the 4.1st step to the 4.3rd step, i.e. by six class human face expressions in training set: surprised, evil
Use of the array z storage that fearness, glad, sad, detest and angry corresponding each dynamic human face image sequence obtain to training set
The human face expression geometrical characteristic vector f of SVM classifier is trained, six different human face expressions being obtained by six class human face expressions
Geometrical characteristic vector f forms six class human face expression collection, then i.e. by six class human face expressions in test set: it is surprised, fear, be glad,
The array z storage that sad, detest and angry corresponding each dynamic human face image sequence obtain is used to test SVM to test set
The human face expression geometrical characteristic vector te of disaggregated model, the six different human face expression geometry obtained by six class human face expressions are special
Vector te is levied, also forms six class human face expression collection, above-mentioned two place refers to that six class human face expression collection are the six class people for being combined into one
Face expression collection so far completes training until all dynamic human face image sequences circulation in training set and test set executes completion
The Extraction of Geometrical Features of all dynamic human face image sequences in collection and test set；
5th step, the analysis and extraction of the semantic feature on human face expression gray level image:
The human face expression of human face expression geometrical characteristic vector f and test set to training set obtained in above-mentioned 4th step is several
What feature vector te carries out semantic analysis, to realize the analysis and extraction of the semantic feature on human face expression gray level image, specifically
Steps are as follows:
5.1st step constructs human face expression semantic characteristics description set:
Defining human face expression semanteme is a kind of natural language description to human face expression feature, including to human face expression feature
In geometric shape, these dominant and recessive attributes of the relative position and emotion of Different Organs explanation.
IfGather for one, when human face expression has A human face expression
Feature, a are wherein a-th of human face expression feature, and when a-th of human face expression feature is made of U attribute, u is u kind therein
Attribute, when u attribute is made of the different strength grade of B kind, b is b kind grade therein, then claimsFor a face
Expression semanteme, Μ are a face expression semanteme description collection.
Formulate it is following it is small, in, the strength grade of big three kinds of descriptions human face expression semantic feature, human face expression semanteme is abbreviated
For ma,b, wherein a is a-th of human face expression semantic feature, and a ∈ [1,59], b are strength grade, b ∈ [1,3], wherein 1,2,3 point
Do not represent it is small, in, big strength grade,
On above-mentioned 4th step human face expression gray level image after the completion of the extraction of the geometrical characteristic of human face expression delta-shaped region,
Utilize face in the human face expression delta-shaped region vector combination TR divided in frame human face expression every in dynamic human face image sequence
The side length and angle of expression delta-shaped region construct human face expression semantic characteristics description set YU, shown in following formula (10),
YU={ yu1,yu2,...,yun,...,yu591≤n≤59 (10),
In formula (10), YU is the conjunction of face expression semanteme characteristic descriptor set, yunFor the conjunction of face expression semanteme characteristic descriptor set
In to human face expression delta-shaped region vector combination TR in n-th of side length of human face expression delta-shaped region and the language of angle character
Justice description, specific statement provide in the following Table 1, and n is the quantity of face expression semanteme feature description；
It include human face expression delta-shaped region vector described in above-mentioned third step in human face expression semantic feature set YU
It combines in 10 human face expression delta-shaped regions in TR according to six of each delta-shaped region of above-mentioned 4th step away from walk-off angle
Degree feature carries out the human face expression semantic feature of 60 face expressive features that semantic description obtains, due to wherein two face tables
The description of feelings semantic feature is identical, therefore omits 1 face expression semanteme feature, i.e. n ∈ [1,59]；
1. semantic characteristics description collection YU of table
5.2nd step formulates semantic feature strength grade decision rule:
All semantic feature values in the human face expression geometrical characteristic vector f of training set in above-mentioned 4.4th step are risen
Sequence sequence, obtains semantic feature range set PF, PF={ pf1,pf2,...,pfv,...pf59, 1 <=<=59 v wherein pfv
For each semantic description yunCorresponding characteristic range, by pfvSemantic feature strength grade is divided according to the size of f/3, it is semantic special
Sign strength grade defines identical with the 5.1st step, and the decision rule of semantic feature strength grade is as shown in Table 2 below:
The decision rule of 2. semantic feature strength grade of table
5.3rd step, to all human face expression triangles in frame human face expression gray level image every in dynamic human face image sequence
Region carries out semantic analysis:
The vector of the human face expression delta-shaped region of every frame human face expression gray level image in each dynamic human face image sequence
Combine i-th of human face expression delta-shaped region tr in TRiHuman face expression semantic characteristics description combination, as shown in Table 3 below:
3. human face expression delta-shaped region tr of tableiSemantic description combination
∧ in table 3 is in order that logical "and" symbol in discrete mathematics；
Calculate every frame human face expression grayscale image in each dynamic human face image sequence in the training set in above-mentioned 4.4th step
I-th of human face expression delta-shaped region tr in the vector combination TR of the human face expression delta-shaped region of pictureiHuman face expression is semantic special
Mean value TR_AVG and standard deviation TR_SD is levied,
Shown in the following formula of the calculating of mean value TR_AVG (11):
TR_AVG=(d0+i×6+d1+i×6+d2+i×6+r3+i×6+r4+i×6+r5+i×6)/6 (11),
Shown in the following formula of the calculating of standard deviation TR_SD (12):
D in formula (11), (12)j+i×6For the people of every frame human face expression gray level image in each dynamic human face image sequence
I-th of human face expression delta-shaped region tr in the vector combination TR of face expression delta-shaped regioniJ-th interior of distance feature value,
rm+i×6For the Vector Groups of the human face expression delta-shaped region of every frame human face expression gray level image in each dynamic human face image sequence
Close i-th of human face expression delta-shaped region tr in TRiM-th interior of angle character value, wherein setting i ∈ [1,10], j ∈ [0,
2], thus all human face expression triangles in every frame human face expression gray level image are completed in dynamic human face image sequence in [3,5] m ∈
The semantic analysis in shape region；
5.4th step obtains the optimal human face expression delta-shaped region combination of six class human face expression collection:
Human face expression triangle in each dynamic human face image sequence is concentrated to six class human face expressions in the 4.4th step first
Region vector combines 10 human face expression delta-shaped region number consecutivelies in TR, the standard deviation then obtained according to above-mentioned 5.3 step
TR_SD carries out ascending sort to 10 human face expression delta Field Numbers in the TR, selects 5 before ranking human face expressions three
Then angular domain number counts before the six class human face expressions concentration ranking in above-mentioned 4.4th step 5 number quantity,
It obtains 5 most numbers of shared number quantity, ascending sort is finally carried out to this 5 numbers according to standard deviation TR_SD, is obtained
The optimal human face expression delta-shaped region of six class human face expression collection combines；
The optimal human face expression delta-shaped region combination of six class human face expressions is as shown in Table 4 below:
The optimal human face expression delta-shaped region of 4. 6 class human face expression of table combines
| Expression classification | The combination of human face expression optimum triangular shape region |
| Surprise | tr1∧tr5∧tr2∧tr3∧tr9 |
| Fear | tr9∧tr8∧tr1∧tr2∧tr6 |
| Happy | tr9∧tr1∧tr5∧tr2∧tr6 |
| Sad | tr2∧tr1∧tr6∧tr3∧tr4 |
| Disgust | tr8∧tr6∧tr2∧tr3∧tr4 |
| Angry | tr1∧tr2∧tr7∧tr6∧tr3 |
∧ in table 4 is in order that logical "and" symbol in discrete mathematics；
5.5th step extracts the final semantic feature of six class human face expression collection:
According to the above-mentioned prepared semantic feature strength grade decision rule of 5.2nd step, to six of the acquisition in the 5.4th step
The semantic feature intensity of human face expression delta-shaped region in the optimal human face expression delta-shaped region combination of class human face expression collection
Grade is determined, and counts each semantic feature strength grade quantity shared by six class human face expression collection, selects shared quantity
Most strength grade, as corresponding semantic description yunCorresponding strength grade, final circulation executes aforesaid operations, to six classes
All semantic description yu in human face expression collectionnThe shared quantity of corresponding strength grade is counted, statistical conditions such as the following table 5,
6, shown in 7,8,9,10:
Table 5.Surprise expression delta-shaped region combines interior semantic feature strength grade quantity statistics
Table 6.Fear expression delta-shaped region combines interior semantic feature strength grade quantity statistics
Table 7.Sad expression delta-shaped region combines interior semantic feature strength grade quantity statistics
Table 8.Disgust expression delta-shaped region combines interior semantic feature strength grade quantity statistics
Table 9.angry expression delta-shaped region combines interior semantic feature strength grade quantity statistics
Table 10.happy expression delta-shaped region combines interior semantic feature strength grade quantity statistics
According to all semantic description yu in six class human face expression collection in above-mentioned table 5-10nCorresponding semantic feature strength grade
Quantity situation, statistics provide the strength grade that each expression delta-shaped region combines interior semantic feature, extract six class faces with this
The final semantic feature of expression collection；
So far the analysis of the semantic feature on human face expression gray level image all terminates with extraction；
The final semantic feature of six class human face expression collection is as shown in table 11,12,13,14,15,16；
The combination of table 11.Surprise expression semanteme feature
| Delta-shaped region | Semantic feature combination |
| tr1 | m7,3∧m8,3∧m9,2∧m10,1∧m11,1∧m12,3 |
| tr5 | m31,3∧m32,3∧m3,1∧m33,1∧m34,3∧m35,3 |
| tr2 | m13,3∧m14,3∧m15,3∧m16,1∧m17,1∧m18,3 |
| tr3 | m19,3∧m20,3∧m21,3∧m22,1∧m23,3∧m24,3 |
| tr9 | m54,3∧m55,3∧m56,1∧m57,1∧m58,3∧m59,3 |
The combination of table 12.Fear expression semanteme feature
| Delta-shaped region | Semantic feature combination |
| tr9 | m54,2∧m55,2∧m56,3∧m57,2∧m58,2∧m59,2 |
| tr8 | m48,3∧m49,2∧m50,3∧m51,3∧m52,2∧m53,2 |
| tr1 | m7,3∧m8,3∧m9,3∧m10,3∧m11,2∧m12,2 |
| tr2 | m13,3∧m14,3∧m15,2∧m16,3∧m17,2∧m18,2 |
| tr6 | m36,3∧m37,2∧m38,3∧m39,3∧m40,1∧m41,2 |
The combination of table 13.Happy expression semanteme feature
| Delta-shaped region | Semantic feature combination |
| tr9 | m54,3∧m55,3∧m56,3∧m57,3∧m58,1∧m59,1 |
| tr1 | m7,2∧m8,2∧m9,3∧m10,3∧m11,2∧m12,1 |
| tr5 | m31,1∧m32,3∧m3,3∧m33,3∧m34,1∧m35,1 |
| tr2 | m13,2∧m14,3∧m15,3∧m16,3∧m17,2∧m18,1 |
| tr6 | m36,2∧m37,2∧m38,1∧m39,3∧m40,1∧m41,1 |
The combination of table 14.Sad expression semanteme feature
| Delta-shaped region | Semantic feature combination |
| tr2 | m13,2∧m14,3∧m15,1∧m16,1∧m17,3∧m18,3 |
| tr1 | m7,3∧m8,3∧m9,1∧m10,1∧m11,3∧m12,3 |
| tr6 | m36,2∧m37,2∧m38,1∧m39,1∧m40,3∧m41,3 |
| tr3 | m19,2∧m20,3∧m21,1∧m22,2∧m23,2∧m24,3 |
| tr9 | m54,1∧m55,1∧m56,1∧m57,1∧m58,3∧m59,3 |
The combination of table 15.Disgust expression semanteme feature
| Delta-shaped region | Semantic feature combination |
| tr8 | m48,1∧m49,1∧m50,2∧m51,2∧m52,2∧m53,3 |
| tr6 | m36,1∧m37,1∧m38,2∧m39,2∧m40,2∧m41,2 |
| tr2 | m13,1∧m14,1∧m15,1∧m16,2∧m17,3∧m18,1 |
| tr3 | m19,1∧m20,1∧m21,1∧m22,3∧m23,1∧m24,1 |
| tr4 | m25,1∧m26,1∧m27,1∧m28,3∧m29,3∧m30,1 |
The combination of table 16.Angry expression semanteme feature
| Delta-shaped region | Semantic feature combination |
| tr1 | m7,1∧m8,1∧m9,1∧m10,2∧m11,3∧m12,1 |
| tr2 | m13,1∧m14,2∧m15,1∧m16,2∧m17,3∧m18,1 |
| tr7 | m42,1∧m43,1∧m44,1∧m45,3∧m46,1∧m47,1 |
| tr6 | m36,1∧m37,1∧m38,2∧m39,2∧m40,2∧m41,2 |
| tr3 | m19,1∧m20,1∧m21,1∧m22,3∧m23,1∧m24,1 |
∧ in above-mentioned table 11-16 is in order that logical "and" symbol in discrete mathematics；
6th step, SVM classifier training simultaneously obtain classification results:
By the analysis of the semantic feature on above-mentioned 5th step human face expression gray level image and extract obtained semantic feature
Data input SVM classifier is trained and predicts, judges that the dynamic human face image sequence inputted in the above-mentioned first step belongs to
Which class human face expression takes the average result of experiment as final facial expression recognition rate using ten times of cross-validation methods, specific to grasp
It is as follows to make process:
(6.1) by the analysis of the semantic feature on above-mentioned 5th step human face expression gray level image and the obtained semanteme of extraction
Characteristic inputs SVM classifier training, is constructed according to the human face expression geometrical characteristic vector f of the training set of above-mentioned 4.4th step
The semantic feature matrix of training sample out, further according to the human face expression geometrical characteristic vector te structure of the test set of above-mentioned 4.4th step
The semantic feature matrix of test sample is produced, then according to its corresponding trained classification of the semantic feature matrix construction of training sample
Sample matrix, the value in the training sample classification matrix are human face expression classification；
(6.2) linear kernel function is used, the type of stopping criterion for iteration 100, SVM classifier uses C_SVC, first will instruction
The semantic feature matrix, training classification sample matrix and parameter for practicing sample are sent into the train function of SVM classifier, are classified
Model, then will be predicted in the predict function of the semantic feature Input matrix of test sample to the disaggregated model, it is thus complete
At SVM classifier training and obtain classification results, then in the library CK+ and the library MMI experiment obtain it is surprised, fear, be glad, sad,
Detest the classification results with angry six kinds of human face expressions；
Thus the identification of dynamic human face expression is completed.
Embodiment 2
The present embodiment is to carry out experimental verification to dynamic human face expression recognition method of the invention.
A. 262 dynamic human face image sequences are chosen in CK+ data set, each dynamic human face image sequence includes 2 width
Image, i.e., neutral frame and peak value frame, totally 524 Facial Expression Image frames are tested.
By the Extraction of Geometrical Features method TGF and geometry semantic feature extraction method SA- in the present invention in CK+ data set
TGF, in the document 1, document 2, document 3, document 4 in the discrimination and background technique obtained after ten times of cross-validation experiments
Discrimination comparison it is as shown in table 17:
Discrimination compares on table 17.CK+ data set
B. 208 dynamic human face image sequences are chosen in MMI data set, each dynamic human face image sequence includes 2 width
Image, i.e., neutral frame and peak value frame, totally 416 Facial Expression Image frames are tested.
By the Extraction of Geometrical Features method TGF and geometry semantic feature extraction method SA- in the present invention in MMI data set
TGF, the identification in document 1, document 2, document 4 in the discrimination and background technique obtained after ten times of cross-validation experiments
Rate comparison is as shown in table 18:
Discrimination compares on table 18.MMI data set
In above-described embodiment, gray scale normalization algorithm, geometrical normalization algorithm, the Libface Face datection point mark of use
It infuses algorithm and SVM classifier is all well-known in the art.
Claims (1)
1. dynamic human face expression recognition method, it is characterised in that: be a kind of dynamic human face based on geometrical characteristic and semantic feature
Expression recognition method, the specific steps are as follows:
The first step, the pretreatment of dynamic human face image sequence:
First every frame Facial Expression Image in the dynamic human face image sequence of input is carried out size to be normalized to size being M × N
Pixel, it is then that every frame Facial Expression Image in the dynamic human face image sequence inputted is empty by RGB using following formula (1)
Between be transformed into gray space, obtain every frame human face expression gray level image Igray_tn,
Igray_tn=0.299IR+0.587IG+0.114IB(1),
In formula (1), IR、IG、IBIt is the red of every frame Facial Expression Image in inputted dynamic human face image sequence respectively
Three channel components of color, green and blue retain every frame human face expression gray level image Igray_tn, for face in following second step
The detection of expression frame is used with characteristic point mark；
Second step, the detection of human face expression frame and the characteristic point of human face expression gray level image mark:
Every frame human face expression gray scale that the above-mentioned first step is obtained using the Multiview_Reinforce interface in the library LibFace
Image Igray_tnThe detection of human face expression frame is carried out, and characteristic point mark, this 68 characteristic points are carried out to 68 characteristic points therein
The following formula of total coordinate vector (2) shown in,
X=((x1,y1),(x2,y2),...,(xk,yk),...,(x68,y68))T(2),
In formula (2), xk, ykAbscissa corresponding to k-th of characteristic point and vertical seat in respectively every frame human face expression gray level image
Mark, k ∈ [1,68]；
Third step, the calibration of human face expression delta-shaped region on human face expression gray level image:
The every frame human face expression gray level image I marked from above-mentioned second stepgray_tnIn 68 characteristic points in select eyebrow, eyes,
30 characteristic points on nose and mouth carry out the calibration of human face expression delta-shaped region on human face expression gray level image, form altogether
10 human face expression delta-shaped regions of calibration, the vector combination TR that human face expression delta-shaped region is consequently formed is following formula
(3) shown in,
TR={ tr1,tr2,...,tri,...,tr10(3),
In formula (3), tri={ X_SI, 1,X_SI, 2,X_SI, 3, i ∈ [1,10], triFor i-th of human face expression delta-shaped region,
X_SI, 1、X_SI, 2、X_SI, 3The apex coordinate on the 1st, 2,3 vertex in respectively i-th of human face expression delta-shaped region,
The 10 human face expression delta-shaped regions demarcated include: left eye and lip, eyes and eyebrow, lip and eyebrow center
Point, eyebrow and nose, nose and lip, eyebrow, eyes and nose, nose central point and the corners of the mouth, right eye eyeball and lip, pure mouth
Lip, pure eyes；
4th step, the extraction of the geometrical characteristic of human face expression delta-shaped region on human face expression gray level image:
The human face expression triangle of every frame human face expression gray level image calibration in the dynamic human face image sequence in above-mentioned third step
It is calculated on the vector combination TR in shape region, the specific steps are as follows:
4.1st step, in dynamic human face image sequence between the human face expression delta-shaped region vertex of every frame human face expression gray level image
The extraction of distance feature:
By the human face expression delta of frame human face expression gray level image every in the dynamic human face image sequence in above-mentioned third step
I-th of human face expression delta-shaped region tr in the vector combination TR in domainiIn three vertex transverse and longitudinal coordinate groups be combined into as X_SI, 1
(xI, 1, yI, 1)、X_SI, 2(xI, 2, yI, 2)、X_SI, 3(xI, 3, yI, 3), it is calculated respectively with following formula (4), formula (5), formula (6)
In dynamic human face image sequence in above-mentioned third step the human face expression delta-shaped region of every frame human face expression gray level image to
I-th of human face expression delta-shaped region tr in amount combination TRiIn the Euclidean distance between vertex two-by-two,
Vertex X_SI, 1With vertex X_SI, 2Between Euclidean distance dI, 1The following formula of calculating (4) shown in:
Vertex X_SI, 1With vertex X_SI, 3Between Euclidean distance dI, 2The following formula of calculating (5) shown in:
Vertex X_SI, 2With vertex X_SI, 3Between Euclidean distance di,3The following formula of calculating (6) shown in:
Formula (4), (5), in (6), xi,1, xi,2, xI, 3The the 1st, 2,3 top in respectively i-th of human face expression delta-shaped region
The abscissa of point, yI, 1, yI, 2, yI, 3The ordinate on the 1st, 2,3 vertex in respectively i-th of human face expression delta-shaped region,
I ∈ [1,10],
Thus it completes in dynamic human face image sequence between the human face expression delta-shaped region vertex of every frame human face expression gray level image
The extraction of distance feature；
4.2nd step, the human face expression delta-shaped region vertex of every frame human face expression gray level image in dynamic human face image sequence
The extraction of angle character:
Calculate the human face expression triangle of every frame human face expression gray level image in the dynamic human face image sequence in above-mentioned third step
I-th of human face expression delta-shaped region tr in the vector combination TR in regioniIn three apex coordinates angle character, tri's
Three vertex transverse and longitudinal coordinate groups are combined into as X_SI, 1(xI, 1, yI, 1)、X_SI, 2(xI, 2, yI, 2)、X_SI, 3(xI, 3, yI, 3), use X_Si,1、
X_Si,2、X_Si,3The coordinate on these three vertex calculates the angle character r on corresponding 1st vertexi,1, the 2nd vertex angle
Spend feature rI, 2, the 3rd vertex angle character rI, 3Formula (7), formula (8), formula (9) as follows,
In formula (7), formula (8) and formula (9), xi,1、xi,2、xI, 3In respectively i-th of human face expression delta-shaped region
1, the abscissa on 2,3 vertex, yI, 1, yI, 2, yI, 3The the 1st, 2,3 vertex in respectively i-th of human face expression delta-shaped region
Ordinate, i ∈ [1,10]；
4.3rd step, the geometry of the human face expression delta-shaped region of every frame human face expression gray level image in dynamic human face image sequence
The extraction of feature:
The geometrical characteristic is made of distance feature and angle character, the dynamic human face image sequence that above-mentioned 4.1st step is obtained
I-th of human face expression in the vector combination TR of the human face expression delta-shaped region of the human face expression gray level image of neutral frame in column
Delta-shaped region triDistance feature and the neutral frame in the obtained dynamic human face image sequence of above-mentioned 4.2nd step face table
I-th of human face expression delta-shaped region tr in the vector combination TR of the human face expression delta-shaped region of feelings gray level imageiAngle
Feature is worth i.e. for six totally: di,1, di,2, di,3, ri,1, ri,2, ri,3It is stored in vector hiIn, h herei={ di,1,di,2,di,3,ri,1,
ri,2,ri,3, wherein [1,10] i ∈, by the face table of the peak value frame with above-mentioned neutral frame in same dynamic human face image sequence
I-th of human face expression delta-shaped region tr in the vector combination TR of the human face expression delta-shaped region of feelings gray level imageiDistance
Feature and angle character are worth i.e. for six totally: di,1, di,2, di,3, ri,1, ri,2, ri,3It is stored in vector wiIn, wi={ di,1,di,2,di,3,
ri,1,ri,2,ri,3, wherein [1,10] i ∈, by vector wiWith vector hiCharacteristic Ratios be put into array z, z=
{z(i-1)×6+j, z(i-1)×6+j=w(i-1),j/h(i-1),j, wherein h is the face table of human face expression gray level image in above-mentioned neutral frame
The distance feature of all delta-shaped regions and the total value of angle character in the vector combination TR of feelings delta-shaped region, w is above-mentioned peak
It is worth the distance of all delta-shaped regions in the vector combination TR of the human face expression delta-shaped region of human face expression gray level image in frame
The total value of feature and angle character, z are the human face expression delta-shaped region of human face expression gray level image in peak value frame and neutral frame
Vector combination TR in the distance feature of all delta-shaped regions and the ratio of angle character value, i ∈ [1,10], j ∈ [0,5],
So far it is fully completed the human face expression delta-shaped region combination TR of the every frame human face expression gray level image of this dynamic human face image sequence
Extraction of Geometrical Features；
The extraction of the geometrical characteristic of all dynamic human face image sequences in 4.4th step, training set and test set:
Circulation executes the operation of above-mentioned the 4.1st step to the 4.3rd step, i.e. by six class human face expressions in training set: it is surprised, fear,
The array z storage that glad, sad, detest and angry corresponding each dynamic human face image sequence obtain is used to instruct to training set
Practice the human face expression geometrical characteristic vector f of SVM classifier, the six different human face expression geometry obtained by six class human face expressions
Feature vector f forms six class human face expression collection, then i.e. by six class human face expressions in test set: surprised, fear, is glad, wound
The array z storage that the heart, detest and angry corresponding each dynamic human face image sequence obtain is used to test SVM points to test set
The human face expression geometrical characteristic vector te of class model, the six different human face expression geometrical characteristics obtained by six class human face expressions
Vector te, also forms six class human face expression collection, and above-mentioned two place refers to that six class human face expression collection are the six class faces for being combined into one
Expression collection so far completes training set until all dynamic human face image sequences circulation in training set and test set executes completion
With the Extraction of Geometrical Features of dynamic human face image sequences all in test set；
5th step, the analysis and extraction of the semantic feature on human face expression gray level image:
The human face expression geometry of human face expression geometrical characteristic vector f and test set to training set obtained in above-mentioned 4th step is special
It levies vector te and carries out semantic analysis, to realize the analysis and extraction of the semantic feature on human face expression gray level image, specific steps
It is as follows:
5.1st step constructs human face expression semantic characteristics description set:
Defining human face expression semanteme is a kind of natural language description to human face expression feature, including in human face expression feature
The explanation of these dominant and recessive attributes of geometric shape, the relative position of Different Organs and emotion.
IfGather for one, when human face expression has A face expressive features,
A is wherein a-th of human face expression feature, and when a-th of human face expression feature is made of U attribute, u is u attribute therein,
When u attribute is made of the different strength grade of B kind, b is b kind grade therein, then claimsFor a human face expression language
Justice, Μ are a face expression semanteme description collection.
Formulate it is following it is small, in, the strength grade of big three kinds of descriptions human face expression semantic feature, human face expression semanteme is abbreviated as
ma,b, wherein a is a-th of human face expression semantic feature, and a ∈ [1,59], b are strength grade, b ∈ [1,3], wherein 1,2,3 difference
Represent it is small, in, big strength grade,
On above-mentioned 4th step human face expression gray level image after the completion of the extraction of the geometrical characteristic of human face expression delta-shaped region, utilize
Human face expression in the human face expression delta-shaped region vector combination TR divided in every frame human face expression in dynamic human face image sequence
The side length and angle of delta-shaped region construct human face expression semantic characteristics description set YU, shown in following formula (10),
YU={ yu1,yu2,...,yun,...,yu591≤n≤59 (10),
In formula (10), YU is the conjunction of face expression semanteme characteristic descriptor set, yunIt is right in the conjunction of face expression semanteme characteristic descriptor set
N-th of side length of human face expression delta-shaped region and the semanteme of angle character are retouched in human face expression delta-shaped region vector combination TR
It states, n is the quantity of face expression semanteme feature description；
It include the combination of human face expression delta-shaped region vector described in above-mentioned third step in human face expression semantic feature set YU
It is special according to six distances of each delta-shaped region of above-mentioned 4th step and angle in 10 human face expression delta-shaped regions in TR
Sign carries out the human face expression semantic feature of 60 face expressive features that semantic description obtains, due to wherein two human face expression languages
The description of adopted feature is identical, therefore omits 1 face expression semanteme feature；
5.2nd step formulates semantic feature strength grade decision rule:
Ascending order row is carried out to semantic feature values all in the human face expression geometrical characteristic vector f of the training set in above-mentioned 4.4th step
Sequence obtains semantic feature range set PF, PF={ pf1,pf2,...,pfv,...pf59, 1 <=<=59 v wherein pfvIt is every
A semantic description yunCorresponding characteristic range, by pfvSemantic feature strength grade is divided according to the size of f/3, semantic feature is strong
Degree grade defines identical with the 5.1st step；
5.3rd step, to all human face expression delta-shaped regions in frame human face expression gray level image every in dynamic human face image sequence
Carry out semantic analysis:
Calculate every frame human face expression gray level image in each dynamic human face image sequence in the training set in above-mentioned 4.4th step
I-th of human face expression delta-shaped region tr in the vector combination TR of human face expression delta-shaped regioniHuman face expression semantic feature is equal
Value TR_AVG and standard deviation TR_SD,
Shown in the following formula of the calculating of mean value TR_AVG (11):
TR_AVG=(d0+i×6+d1+i×6+d2+i×6+r3+i×6+r4+i×6+r5+i×6)/6 (11),
Shown in the following formula of the calculating of standard deviation TR_SD (12):
D in formula (11), (12)j+i×6For the human face expression of every frame human face expression gray level image in each dynamic human face image sequence
I-th of human face expression delta-shaped region tr in the vector combination TR of delta-shaped regioniJ-th interior of distance feature value, rm+i×6For
In each dynamic human face image sequence in the vector combination TR of the human face expression delta-shaped region of every frame human face expression gray level image
I-th of human face expression delta-shaped region triM-th interior of angle character value, wherein setting i ∈ [1,10], j ∈ [0,2], m ∈
[3,5] thus complete in dynamic human face image sequence all human face expression delta-shaped regions in every frame human face expression gray level image
Semantic analysis；
5.4th step obtains the optimal human face expression delta-shaped region combination of six class human face expression collection:
Human face expression delta-shaped region in each dynamic human face image sequence is concentrated to six class human face expressions in the 4.4th step first
Vector combines 10 human face expression delta-shaped region number consecutivelies in TR, the standard deviation TR_SD then obtained according to above-mentioned 5.3 step
Ascending sort is carried out to 10 human face expression delta Field Numbers in the TR, selects 5 before ranking human face expression triangles
Then zone number counts before the six class human face expressions concentration ranking in above-mentioned 4.4th step 5 number quantity, obtains
This 5 numbers are finally carried out ascending sort according to standard deviation TR_SD, obtain six classes by 5 most numbers of shared number quantity
The optimal human face expression delta-shaped region of human face expression collection combines；
5.5th step extracts the final semantic feature of six class human face expression collection:
According to the above-mentioned prepared semantic feature strength grade decision rule of 5.2nd step, to six class people of the acquisition in the 5.4th step
The semantic feature strength grade of human face expression delta-shaped region in the optimal human face expression delta-shaped region combination of face expression collection
Determined, and count each semantic feature strength grade quantity shared by six class human face expression collection, selects shared quantity most
Strength grade, as corresponding semantic description yunCorresponding strength grade, final circulation executes aforesaid operations, to six class faces
All semantic description yu in expression collectionnThe shared quantity of corresponding strength grade is counted, and then obtains six class human face expressions
All semantic description yu in collectingnCorresponding strength grade extracts the final semantic feature of six class human face expression collection with this；
So far the analysis of the semantic feature on human face expression gray level image all terminates with extraction；
6th step, SVM classifier training simultaneously obtain classification results:
By the analysis of the semantic feature on above-mentioned 5th step human face expression gray level image and extract obtained semantic feature data
Input SVM classifier is trained and predicts, judges which class the dynamic human face image sequence inputted in the above-mentioned first step belongs to
Human face expression takes the average result of experiment as final facial expression recognition rate, concrete operations stream using ten times of cross-validation methods
Journey is as follows:
(6.1) by the analysis of the semantic feature on above-mentioned 5th step human face expression gray level image and the obtained semantic feature of extraction
Data input SVM classifier training, construct instruction according to the human face expression geometrical characteristic vector f of the training set of above-mentioned 4.4th step
The semantic feature matrix for practicing sample, constructs further according to the human face expression geometrical characteristic vector te of the test set of above-mentioned 4.4th step
The semantic feature matrix of test sample, then its corresponding trained classification sample according to the semantic feature matrix construction of training sample
Matrix, the value in the training sample classification matrix are human face expression classification；
(6.2) linear kernel function is used, the type of stopping criterion for iteration 100, SVM classifier uses C_SVC, first will training sample
This semantic feature matrix, training classification sample matrix and parameter are sent into the train function of SVM classifier, obtain disaggregated model,
It will be predicted in the predict function of the semantic feature Input matrix of test sample to the disaggregated model again, thus complete SVM
Classifier training simultaneously obtains classification results, then in the library CK+ and the library MMI experiment obtain it is surprised, fear, be glad, sad, detest and
The classification results of angry six kinds of human face expressions；
Thus the identification of dynamic human face expression is completed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910109704.5A CN109753950B (en) | 2019-02-11 | 2019-02-11 | Dynamic facial expression recognition method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910109704.5A CN109753950B (en) | 2019-02-11 | 2019-02-11 | Dynamic facial expression recognition method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109753950A true CN109753950A (en) | 2019-05-14 |
| CN109753950B CN109753950B (en) | 2020-08-04 |
Family
ID=66407447
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910109704.5A Expired - Fee Related CN109753950B (en) | 2019-02-11 | 2019-02-11 | Dynamic facial expression recognition method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109753950B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110647856A (en) * | 2019-09-29 | 2020-01-03 | 大连民族大学 | Method for recognizing facial expressions based on theory of axiomatic fuzzy set |
| CN111652073A (en) * | 2020-05-08 | 2020-09-11 | 腾讯科技（深圳）有限公司 | Video classification method, device, system, server and storage medium |
| CN112287524A (en) * | 2020-10-13 | 2021-01-29 | 泉州津大智能研究院有限公司 | Emotion classification method and device based on sparse Gaussian conditional random field |
| WO2021027553A1 (en) * | 2019-08-15 | 2021-02-18 | 深圳壹账通智能科技有限公司 | Micro-expression classification model generation method, image recognition method, apparatus, devices, and mediums |
| CN112613416A (en) * | 2020-12-26 | 2021-04-06 | 中国农业银行股份有限公司 | Facial expression recognition method and related device |
| CN112733616A (en) * | 2020-12-22 | 2021-04-30 | 北京达佳互联信息技术有限公司 | Dynamic image generation method and device, electronic equipment and storage medium |
| CN112766145A (en) * | 2021-01-15 | 2021-05-07 | 深圳信息职业技术学院 | Method and device for identifying dynamic facial expressions of artificial neural network |
| CN114578968A (en) * | 2022-03-09 | 2022-06-03 | 润芯微科技(江苏)有限公司 | Switching method for 3D/2D display state of instrument |
| CN115665399A (en) * | 2022-10-21 | 2023-01-31 | 人民百业科技有限公司 | Liquid crystal grating-based 3D display switching method |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101059836A (en) * | 2007-06-01 | 2007-10-24 | 华南理工大学 | Human eye positioning and human eye state recognition method |
| CN103854306A (en) * | 2012-12-07 | 2014-06-11 | 山东财经大学 | High-reality dynamic expression modeling method |
| KR101510798B1 (en) * | 2008-12-10 | 2015-04-10 | 광주과학기술원 | Portable Facial Expression Training System and Methods thereof |
| CN106228145A (en) * | 2016-08-04 | 2016-12-14 | 网易有道信息技术（北京）有限公司 | A kind of facial expression recognizing method and equipment |
| CN107358169A (en) * | 2017-06-21 | 2017-11-17 | 厦门中控智慧信息技术有限公司 | A kind of facial expression recognizing method and expression recognition device |
| CN107704810A (en) * | 2017-09-14 | 2018-02-16 | 南京理工大学 | A kind of expression recognition method suitable for medical treatment and nursing |
| CN108108677A (en) * | 2017-12-12 | 2018-06-01 | 重庆邮电大学 | One kind is based on improved CNN facial expression recognizing methods |
| CN109034099A (en) * | 2018-08-14 | 2018-12-18 | 华中师范大学 | A kind of expression recognition method and device |
-
2019
- 2019-02-11 CN CN201910109704.5A patent/CN109753950B/en not_active Expired - Fee Related
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101059836A (en) * | 2007-06-01 | 2007-10-24 | 华南理工大学 | Human eye positioning and human eye state recognition method |
| KR101510798B1 (en) * | 2008-12-10 | 2015-04-10 | 광주과학기술원 | Portable Facial Expression Training System and Methods thereof |
| CN103854306A (en) * | 2012-12-07 | 2014-06-11 | 山东财经大学 | High-reality dynamic expression modeling method |
| CN106228145A (en) * | 2016-08-04 | 2016-12-14 | 网易有道信息技术（北京）有限公司 | A kind of facial expression recognizing method and equipment |
| CN107358169A (en) * | 2017-06-21 | 2017-11-17 | 厦门中控智慧信息技术有限公司 | A kind of facial expression recognizing method and expression recognition device |
| CN107704810A (en) * | 2017-09-14 | 2018-02-16 | 南京理工大学 | A kind of expression recognition method suitable for medical treatment and nursing |
| CN108108677A (en) * | 2017-12-12 | 2018-06-01 | 重庆邮电大学 | One kind is based on improved CNN facial expression recognizing methods |
| CN109034099A (en) * | 2018-08-14 | 2018-12-18 | 华中师范大学 | A kind of expression recognition method and device |
Non-Patent Citations (3)
| Title |
|---|
| CHUNNA TIAN等: "Multiview face recognition:from tensorface to v-tensorface and k-tensorface", 《IEEE》 * |
| MIRRIRYUCHEN: "libfacedetection（1）：最简单粗暴的配置方法", 《CSDN》 * |
| 于明等: "基于LGBP特征和稀疏表示的人脸表情识别", 《计算机工程与设计》 * |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021027553A1 (en) * | 2019-08-15 | 2021-02-18 | 深圳壹账通智能科技有限公司 | Micro-expression classification model generation method, image recognition method, apparatus, devices, and mediums |
| CN110647856A (en) * | 2019-09-29 | 2020-01-03 | 大连民族大学 | Method for recognizing facial expressions based on theory of axiomatic fuzzy set |
| CN110647856B (en) * | 2019-09-29 | 2023-04-18 | 大连民族大学 | Method for recognizing facial expressions based on theory of axiomatic fuzzy set |
| CN111652073B (en) * | 2020-05-08 | 2023-02-28 | 腾讯科技（深圳）有限公司 | Video classification method, device, system, server and storage medium |
| CN111652073A (en) * | 2020-05-08 | 2020-09-11 | 腾讯科技（深圳）有限公司 | Video classification method, device, system, server and storage medium |
| CN112287524A (en) * | 2020-10-13 | 2021-01-29 | 泉州津大智能研究院有限公司 | Emotion classification method and device based on sparse Gaussian conditional random field |
| CN112733616A (en) * | 2020-12-22 | 2021-04-30 | 北京达佳互联信息技术有限公司 | Dynamic image generation method and device, electronic equipment and storage medium |
| CN112733616B (en) * | 2020-12-22 | 2022-04-01 | 北京达佳互联信息技术有限公司 | Dynamic image generation method and device, electronic equipment and storage medium |
| CN112613416A (en) * | 2020-12-26 | 2021-04-06 | 中国农业银行股份有限公司 | Facial expression recognition method and related device |
| CN112766145A (en) * | 2021-01-15 | 2021-05-07 | 深圳信息职业技术学院 | Method and device for identifying dynamic facial expressions of artificial neural network |
| CN112766145B (en) * | 2021-01-15 | 2021-11-26 | 深圳信息职业技术学院 | Method and device for identifying dynamic facial expressions of artificial neural network |
| CN114578968B (en) * | 2022-03-09 | 2022-09-23 | 润芯微科技(江苏)有限公司 | Switching method for 3D/2D display state of instrument |
| CN114578968A (en) * | 2022-03-09 | 2022-06-03 | 润芯微科技(江苏)有限公司 | Switching method for 3D/2D display state of instrument |
| CN115665399A (en) * | 2022-10-21 | 2023-01-31 | 人民百业科技有限公司 | Liquid crystal grating-based 3D display switching method |
| CN115665399B (en) * | 2022-10-21 | 2024-02-06 | 人民百业科技有限公司 | 3D display switching method based on liquid crystal grating |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109753950B (en) | 2020-08-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109753950A (en) | Dynamic human face expression recognition method | |
| CN106960202B (en) | Smiling face identification method based on visible light and infrared image fusion | |
| CN110532900B (en) | Facial expression recognition method based on U-Net and LS-CNN | |
| CN106599854B (en) | Automatic facial expression recognition method based on multi-feature fusion | |
| Jung et al. | Deep temporal appearance-geometry network for facial expression recognition | |
| CN105139004B (en) | Facial expression recognizing method based on video sequence | |
| Torralba | Contextual priming for object detection | |
| CN108268859A (en) | A kind of facial expression recognizing method based on deep learning | |
| CN104281853B (en) | A kind of Activity recognition method based on 3D convolutional neural networks | |
| Zhu et al. | Learning a hierarchical deformable template for rapid deformable object parsing | |
| CN105825183B (en) | Facial expression recognizing method based on partial occlusion image | |
| CN108280397A (en) | Human body image hair detection method based on depth convolutional neural networks | |
| CN108830237B (en) | Facial expression recognition method | |
| CN106778852A (en) | A kind of picture material recognition methods for correcting erroneous judgement | |
| Neche et al. | Arabic handwritten documents segmentation into text-lines and words using deep learning | |
| CN105825168A (en) | Golden snub-nosed monkey face detection and tracking algorithm based on S-TLD | |
| CN108710916A (en) | The method and device of picture classification | |
| CN109033978A (en) | A kind of CNN-SVM mixed model gesture identification method based on error correction strategies | |
| CN107818299A (en) | Face recognition algorithms based on fusion HOG features and depth belief network | |
| CN110837777A (en) | Partial occlusion facial expression recognition method based on improved VGG-Net | |
| Ke et al. | Weakly supervised fine-grained image classification via two-level attention activation model | |
| US11521427B1 (en) | Ear detection method with deep learning pairwise model based on contextual information | |
| CN112598056A (en) | Software identification method based on screen monitoring | |
| CN113096079A (en) | Image analysis system and construction method thereof | |
| CN111444860A (en) | Expression recognition method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20200804 |