CN106980811A - Facial expression recognizing method and expression recognition device - Google Patents

Facial expression recognizing method and expression recognition device Download PDF

Info

Publication number
CN106980811A
CN106980811A CN201610921132.7A CN201610921132A CN106980811A CN 106980811 A CN106980811 A CN 106980811A CN 201610921132 A CN201610921132 A CN 201610921132A CN 106980811 A CN106980811 A CN 106980811A
Authority
CN
China
Prior art keywords
image
human
expression
frame
facial
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201610921132.7A
Other languages
Chinese (zh)
Inventor
金啸
胡晨晨
旷章辉
张伟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sensetime Group Ltd
Original Assignee
Sensetime Group Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sensetime Group Ltd filed Critical Sensetime Group Ltd
Priority to CN201610921132.7A priority Critical patent/CN106980811A/en
Publication of CN106980811A publication Critical patent/CN106980811A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06KRECOGNITION OF DATA; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K9/00Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints
    • G06K9/00221Acquiring or recognising human faces, facial parts, facial sketches, facial expressions
    • G06K9/00302Facial expression recognition
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06KRECOGNITION OF DATA; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K9/00Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints
    • G06K9/62Methods or arrangements for recognition using electronic means
    • G06K9/6217Design or setup of recognition systems and techniques; Extraction of features in feature space; Clustering techniques; Blind source separation
    • G06K9/6256Obtaining sets of training patterns; Bootstrap methods, e.g. bagging, boosting
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06NCOMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computer systems based on biological models
    • G06N3/02Computer systems based on biological models using neural network models
    • G06N3/08Learning methods
    • G06N3/084Back-propagation

Abstract

The invention discloses a kind of facial expression recognizing method and expression recognition device, wherein, the facial expression recognizing method includes:Human face image sequence to be identified is obtained, the human face image sequence includes single frames or two frame above facial images;Each frame facial image in the human face image sequence is pre-processed respectively;The training pattern that pretreated each frame facial image input has been trained carries out Expression Recognition, obtains the Expression Recognition result of the human face image sequence;Wherein, the input of the training pattern is built by convolutional neural networks model, long short-term memory Recognition with Recurrent Neural Network model, the first pond layer and Logic Regression Models successively to output end, and sequential frame image set training of the training pattern by marking classification of expressing one's feelings is obtained.The technical scheme that the present invention is provided can effectively lift the recognition performance of human face expression.

Description

Facial expression recognizing method and expression recognition device
Technical field
The present invention relates to image identification technical field, and in particular to a kind of facial expression recognizing method and expression recognition Device.
Background technology
Expression recognition technology refers to expression classification specified to given facial image, including:Indignation, detests, Happily, it is sad, it is frightened, it is surprised etc..At present, expression recognition technology is in man-machine interaction, clinical diagnosis, long-distance education and investigation The fields such as hearing gradually show wide application prospect, are the popular research directions of computer vision and artificial intelligence.
It presently, there are a kind of facial expression recognizing method based on depth convolutional neural networks, the facial expression recognizing method After being detected, calibrating by facial image, the depth convolutional neural networks that the facial image input after calibration has been trained are carried out Expression Recognition.In above-mentioned facial expression recognizing method, depth convolutional neural networks are obtained by single-frame images training, and due to The expression of face is close with scene contextual relation, and highly dependent with the neutral expression of object, therefore, passes through above-mentioned face table Feelings recognition methods is difficult to centering expression and accurately identified, and the recognition performance of human face expression is poor.
The content of the invention
The present invention provides a kind of facial expression recognizing method and expression recognition device, the knowledge for lifting human face expression Other performance.
First aspect present invention provides a kind of facial expression recognizing method, including:
Human face image sequence to be identified is obtained, the human face image sequence includes single frames or two frame above facial images;
Each frame facial image in the human face image sequence is pre-processed respectively;
The training pattern that pretreated each frame facial image input has been trained carries out Expression Recognition, obtains the people The Expression Recognition result of face image sequence;
Wherein, the input of the training pattern is followed by convolutional neural networks model, long short-term memory successively to output end Ring neural network model, the first pond layer and Logic Regression Models are built, and the training pattern is by marking classification of expressing one's feelings Sequential frame image set training is obtained.
Second aspect of the present invention provides a kind of expression recognition device, including:
Image acquisition unit, the human face image sequence to be identified for obtaining, the human face image sequence comprising single frames or Two frame above facial images;
Image pre-processing unit, for being pre-processed respectively to each frame facial image in the human face image sequence;
Identifying processing unit, for the training pattern carry out table for having trained pretreated each frame facial image input Feelings are recognized, obtain the Expression Recognition result of the human face image sequence;
Wherein, the input of the training pattern is followed by convolutional neural networks model, long short-term memory successively to output end Ring neural network model, the first pond layer and Logic Regression Models are built, and the training pattern is by marking classification of expressing one's feelings Sequential frame image set training is obtained.
Therefore, long short-term memory Recognition with Recurrent Neural Network (LSTM-RNN, Long Short Term are based in the present invention Memory-Recurrent Neural Networks) model construction training pattern, and by sequential frame image set (for example regarding Frequently inputted) as the training of the training pattern, the training pattern can be made to make full use of the multidate information of countenance change certainly Mapping relations between the neutral expression of dynamic study identification object and different posture expressive features, so as to improve the training pattern Precision of prediction and robustness, and then lifted human face expression recognition performance.
Brief description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing There is the accompanying drawing used required in technology description to be briefly described, it should be apparent that, drawings in the following description are only this Some embodiments of invention, for those of ordinary skill in the art, without having to pay creative labor, may be used also To obtain other accompanying drawings according to these accompanying drawings.
A kind of facial expression recognizing method one embodiment schematic flow sheet that Fig. 1-a provide for the present invention;
Fig. 1-b are implemented for a kind of training pattern applied to facial expression recognizing method shown in Fig. 1-a that the present invention is provided Example structural representation;
A kind of sequential processing flow direction of the training pattern shown in Fig. 1-b that Fig. 1-c provide for the present invention under application scenarios Schematic diagram;
Sequential processing stream of the training pattern shown in Fig. 1-b that Fig. 1-d provide for the present invention under another application scenarios To schematic diagram;
Fig. 1-e show for a kind of LSTM-RNN model structures for being applied to the training pattern shown in Fig. 1-b that the present invention is provided It is intended to;
A kind of CNN model structures schematic diagram for being applied to the training pattern shown in Fig. 1-b that Fig. 1-f provide for the present invention;
A kind of expression recognition device one embodiment structural representation that Fig. 2 provides for the present invention.
Embodiment
To enable goal of the invention, feature, the advantage of the present invention more obvious and understandable, below in conjunction with the present invention Accompanying drawing in embodiment, the technical scheme in the embodiment of the present invention is clearly and completely described, it is clear that described reality It is only a part of embodiment of the invention to apply example, and not all embodiments.Based on the embodiment in the present invention, the common skill in this area The every other embodiment that art personnel are obtained under the premise of creative work is not made, belongs to the model that the present invention is protected Enclose.
Embodiment one
Present example provides a kind of facial expression recognizing method.As shown in Fig. 1-a, the face table in the embodiment of the present invention Feelings recognition methods includes:
Step 101, acquisition human face image sequence to be identified;
Wherein, above-mentioned human face image sequence includes single frames or two frame above facial images.That is, in the embodiment of the present invention Continuous multiframe facial image (such as video) can be identified for facial expression recognizing method, meanwhile, it is also compatible with to single frames The identification of facial image.
In a step 101, human face image sequence to be identified can be obtained in real time by camera, or, it can also lead to The mode for receiving the human face image sequence from external equipment is crossed, human face image sequence to be identified is obtained, or, can also base Human face image sequence to be identified is obtained in selection of the user in existing image data base or video database, is not made herein Limit.
Step 102, each frame facial image in above-mentioned human face image sequence is pre-processed respectively;
After obtaining human face image sequence to be identified in step 101, respectively to each frame in above-mentioned human face image sequence Facial image is pre-processed, to enable pretreated facial image to be more suitable for follow-up Expression Recognition, specifically, Under different application scenarios, the pretreatment to facial image can also use corresponding processing method.
For example, in one embodiment, it is above-mentioned that each frame facial image in above-mentioned human face image sequence is carried out in advance respectively Processing can specifically include following two steps:
Step 1, in above-mentioned each frame facial image every frame facial image carry out Face datection, determine human face region. The process of above-mentioned Face datection can be realized using a variety of Face datection algorithms, such as based on Haar-Like features Adaboost Face datection algorithms etc.., can be with appropriately sized window and appropriate step scan based on Face datection algorithm Input picture (namely above-mentioned per frame facial image), until determine the human face region in the facial image (human face region namely Region where face).
Key feature points in step 2, the above-mentioned human face region of detection, and based on the key feature points detected to corresponding Facial image carries out alignment.On the basis of Face datection, further determine that key feature points in human face region (for example Eyes, eyebrow, nose, face, face's outline etc.) position., can according to the key feature points detected in human face region Alignment is carried out to corresponding facial image by rigid body translation so that the position base of face each key feature points in the picture This is consistent.In embodiments of the present invention, specifically the alignment of facial image can be carried out using landmark methods.Separately Outside, during alignment is carried out to facial image, key feature points can also be carried out according to preset faceform Positioning adjustment.
Further, it is above-mentioned respectively to above-mentioned facial image sequence in order to avoid image size disunity influences the result of identification Each frame facial image in row, which carries out pretreatment, to be comprised the following steps:Facial image after step 2 alignment is pressed Editing and processing is carried out according to default template, to obtain the facial image of unified size, wherein, above-mentioned editing and processing is included as next Plant or two or more:Shear treatment, scaling processing.For example, during above-mentioned editing and processing, based on the human face region detected In key feature points, corresponding facial image is cut out coming by uniform template, and facial image is zoomed into unified size.
If it should be noted that above-mentioned human face image sequence include single frames facial image, it is above-mentioned respectively to above-mentioned face Each frame facial image progress pretreatment in image sequence is actually appeared to be pre-processed to the single frames facial image;If above-mentioned Human face image sequence includes two frame above facial images, then above-mentioned respectively to each frame facial image in above-mentioned human face image sequence Carry out pre-processing to actually appear pre-processing each frame facial image in above-mentioned two frames above facial image respectively.
Step 103, the training pattern for having trained pretreated each frame facial image input carry out Expression Recognition, obtain To the Expression Recognition result of above-mentioned human face image sequence;
In step 103, the training pattern pretreated each frame facial image input of step 102 trained is carried out Expression Recognition, obtains the Expression Recognition result of above-mentioned human face image sequence.Above-mentioned Expression Recognition result may indicate that above-mentioned face figure Expression classification as belonging to sequence, wherein, the expression classification existed may include but be not limited to:It is angry, tranquil, puzzled, detest, it is fast It is happy, sad, fear, surprised, strabismus and scream.
During the present invention is implemented, as shown in Fig. 1-b, the input of above-mentioned training pattern is to output end successively by convolutional Neural net Network (CNN, Convolutional Neural Network) model, long short-term memory Recognition with Recurrent Neural Network model (i.e. LSTM- RNN models), the first pond layer and Logic Regression Models build.Also, above-mentioned training pattern is by marking the continuous of classification of expressing one's feelings Two field picture set training is obtained.Because above-mentioned training pattern is trained by the sequential frame image set of mark expression classification Arrive, therefore, on the one hand, above-mentioned training pattern can learning time yardstick automatically dependence, make full use of countenance to change Multidate information, contact expression present frame front and rear frame information so that Expression Recognition has more robustness;On the other hand, Ke Yijing Really neutral expression is defined to eliminate expression tension force brought influence different from intensity etc. between different objects, and lifting identification is accurate Rate;Another further aspect, by each two field picture in sequential frame image set and the expression classification that is marked have strong correlation, because This, there is distortion distortion even if the image sequence of input can also realize Expression Recognition.
Optionally, above-mentioned first pond layer can be average pond layer or maximum pond layer or other types of pond Layer, is not construed as limiting herein.
Optionally, if above-mentioned human face image sequence includes two frame above facial images, it is above-mentioned by pretreated each frame The training pattern that facial image input has been trained carries out recognition of face, including:By above-mentioned first pond layer to above-mentioned length When memory Recognition with Recurrent Neural Network mode input above-mentioned each frame facial image face feature vector it is unified carry out dimension-reduction treatment, obtain Face feature vector after to dimension-reduction treatment;To above-mentioned Logic Regression Models export the face characteristic after above-mentioned dimension-reduction treatment to Amount.Below to the training pattern by taking sequential frame image (human face image sequence inputted includes two frame above facial images) as an example Sequential processing flow direction be described, the sequential processing of training pattern as shown in fig 1-c flows to schematic diagram, wherein, X0, X1 ..., Xn are each two field picture for the video that length is n frames, the face feature vector that each two field picture is extracted through CNN modules LSTM modules are sequentially input sequentially in time, the face feature vector not exported in the same time that will be obtained through LSTM resume modules H0, h1 ..., hn carry out dimension-reduction treatment by the first pond layer is unified, obtain the face feature vector h for expression classification, most Face feature vector h input logics regression model is subjected to logistic regression processing afterwards, the Expression Recognition of the sequential frame image is obtained As a result.When the human face image sequence of input is single frames facial image (i.e. above-mentioned n=1), training pattern shown in Fig. 1-c when Sequence processing, which flows to schematic diagram and can be reduced to the sequential processing of the training pattern as shown in Fig. 1-d, flows to schematic diagram.
In the embodiment of the present invention, the structures of the LSTM-RNN models that above-mentioned training pattern is included can as shown in Fig. 1-e, Including:Input gate (i.e. input gate), forgetting door (i.e. forget gate), out gate (i.e. output gate), state list First (i.e. cell) and LSTM-RNN model output results.
When the human face image sequence of input includes two frame above facial images, above-mentioned input gate, above-mentioned forgetting The processing procedure of door, above-mentioned out gate, above-mentioned state cell and above-mentioned LSTM-RNN models output result can respectively by with Lower formula is realized:
it=σ (Wixxt+Wimmt-1+Wicct-1+bi);
ft=σ (Wfxxt+Wfmmt-1+Wfcct-1+bf);
ct=ft⊙ct-1+it⊙σ(Wcxxt+Wcmmt-1+bc);
ot=σ (Woxxt+Wommt-1+Wocct-1+bo);
mt=ot⊙h(ct)。
Wherein, in above-mentioned formula, xtIt is expressed as the face feature vector of t input;W (i.e. Wix、Wim、Wic、Wfx、 Wfm、Wfc、Wcx、Wcm、Wox、WomAnd Woc) it is default weight matrix, the element for representing each door is by the data of correspondence dimension Obtain, that is to say, that do not interfere with each other between the node of different dimensions;B (i.e. bi、bf、bc、bo) represent default bias vector, it、 ft、ot、ct、mtThe above-mentioned input gate of t, above-mentioned forgetting door, above-mentioned out gate, above-mentioned state cell and above-mentioned are represented respectively The state of LSTM-RNN model output results, ⊙ is dot product, and σ () is sigmoid functions, and h () is the output of above-mentioned state cell Activation primitive, the output activation primitive is specifically as follows tanh functions.
Optionally, when the human face image sequence of input includes single frames facial image, above-mentioned input gate, above-mentioned something lost Forgetting the processing procedure of door, above-mentioned out gate, above-mentioned state cell and above-mentioned LSTM-RNN models output result can also be reduced to Equation below is realized:
it=σ (Wixxt+consatant1);
ft=σ (Wfxxt+consatant2);
ct=ft⊙ct-1+it⊙σ(Wcxxt+consatant3);
ot=σ (Woxxt+Wommt-1+consatant4);
mt=ot⊙h(ct)。
Wherein, in above-mentioned formula, xtIt is expressed as the face feature vector of t input;W (i.e. Wix、Wim、Wic、Wfx、 Wfm、Wfc、Wcx、Wcm、Wox、WomAnd Woc) it is default weight matrix, the element for representing each door is by the data of correspondence dimension Obtain, that is to say, that do not interfere with each other between the node of different dimensions;Consatant (i.e. consatant1、consatant2、 consatant3And consatant4) it is default constant, it、ft、ot、ct、mtRespectively represent t above-mentioned input gate, on The state for forgeing door, above-mentioned out gate, above-mentioned state cell and above-mentioned LSTM-RNN models output result is stated, ⊙ is dot product, σ () is sigmoid functions, and h () is the output activation primitive of above-mentioned state cell, and the output activation primitive is specifically as follows tanh Function.
Optionally, as shown in Fig. 1-f, the inputs of above-mentioned CNN models is to output end successively by the first convolutional layer, the second pond Change layer, the second convolutional layer and the 3rd pond layer building.The above-mentioned instruction for having trained pretreated each frame facial image input Practice model and carry out recognition of face, including:The people obtained after being handled through above-mentioned 3rd pond layer is exported to above-mentioned LSTM-RNN models Face characteristic vector.Wherein, above-mentioned second pond layer and the 3rd pond layer can for average pond layer or maximum pond layer or its The pond layer of its type, is not construed as limiting herein.Certainly, in other embodiments, above-mentioned CNN models can also be with reference to existing CNN model constructions, are not construed as limiting herein.
The mistake being trained below to the sequential frame image set above by mark expression classification to above-mentioned training pattern Cheng Jing explanations, specifically can be as follows:1st, collecting one or more sequential frame image set, (above-mentioned sequential frame image set can be comprising company Continuous two field picture (such as video)) and each expression classification (same sequential frame image set belonging to sequential frame image set In each image belonging to expression classification it is identical), by the expression classification belonging to each sequential frame image set be labeled as expect The expression classification exported by above-mentioned training pattern.In the embodiment of the present invention, a variety of expression classifications can be preset and (for example given birth to Gas, calmness, puzzlement, detest, it is happy, sad, fear, surprised, strabismus and scream), it is every kind of expression classification correspondence one mapping value. 2nd, the image in above-mentioned sequential frame image set is pre-processed (process of pretreatment is referred to the description in step 102, Here is omitted).3rd, pretreated image is inputted in above-mentioned training pattern, and based on back-propagation algorithm to the training Model is trained, to cause the value exported and classification of being expressed one's feelings belonging to the image after the image of input is handled through above-mentioned training pattern Mapping value deviation in default allowed band.Certainly, the training process to training pattern can also be with reference to other existing Technical scheme realize, be not construed as limiting herein.
It should be noted that the facial expression recognizing method in the embodiment of the present invention can be held by expression recognition device OK, above-mentioned expression recognition device can be integrated in robot, monitor terminal or other terminals, be not construed as limiting herein.
Therefore, the facial expression recognizing method in the embodiment of the present invention is based on LSTM-RNN model constructions and trains mould Type, and inputted sequential frame image set (such as video) as the training of the training pattern, the training pattern can be made abundant The multidate information changed using countenance is learnt between neutral expression and the different posture expressive features of identification object automatically Mapping relations, so as to improve the precision of prediction and robustness of the training pattern, and then lift the recognition performance of human face expression.
Embodiment two
Present example provides a kind of expression recognition device, as shown in Fig. 2 the human face expression in the embodiment of the present invention Identifying device 200 includes:
Image acquisition unit 201, the human face image sequence to be identified for obtaining, the human face image sequence includes single frames Or two frame above facial images;
Image pre-processing unit 202, for being located in advance to each frame facial image in the human face image sequence respectively Reason;
Identifying processing unit 203, the training pattern for pretreated each frame facial image input have been trained is entered Row Expression Recognition, obtains the Expression Recognition result of the human face image sequence;
Wherein, the input of the training pattern is followed by convolutional neural networks model, long short-term memory successively to output end Ring neural network model, the first pond layer and Logic Regression Models are built, and the training pattern is by marking classification of expressing one's feelings Sequential frame image set training is obtained.
Optionally, identifying processing unit 203 specifically for:When the human face image sequence includes two frame above facial images When, by first pond layer to each frame facial image of the long short-term memory Recognition with Recurrent Neural Network mode input Face feature vector is unified to carry out dimension-reduction treatment, obtains the face feature vector after dimension-reduction treatment;To the Logic Regression Models Export the face feature vector after the dimension-reduction treatment.
Optionally, the input of the convolutional neural networks model to output end successively by the first convolutional layer, the second pond Layer, the second convolutional layer and the 3rd pond layer building;Identifying processing unit 203 specifically for:To the long short-term memory circulation god The face feature vector obtained after being handled through the 3rd pond layer is exported through network model.
Optionally, image pre-processing unit 202 specifically for:For every frame facial image in each frame facial image Face datection is carried out, human face region is determined;The key feature points in the human face region are detected, and it is special based on the key detected Levy and alignment a little is carried out to corresponding facial image.
Optionally, image pre-processing unit 202 is specifically additionally operable to:By the facial image after alignment according to default mould Plate carries out editing and processing, to obtain the facial image of unified size, wherein, the editing and processing include it is following it is one or two kinds of with On:Shear treatment, scaling processing.
It should be noted that the expression recognition device in the embodiment of the present invention can be integrated in robot, monitoring eventually In end or other terminals.The function of each functional module of the expression recognition device is referred in above method embodiment Description, it implements process and can refer to associated description in above method embodiment, and here is omitted.
Therefore, the expression recognition device in the embodiment of the present invention is based on LSTM-RNN model constructions and trains mould Type, and inputted sequential frame image set (such as video) as the training of the training pattern, the training pattern can be made abundant The multidate information changed using countenance is learnt between neutral expression and the different posture expressive features of identification object automatically Mapping relations, so as to improve the precision of prediction and robustness of the training pattern, and then lift the recognition performance of human face expression.
, can be by it in several embodiments provided herein, it should be understood that disclosed apparatus and method Its mode is realized.
It should be noted that for foregoing each method embodiment, for simplicity description, therefore it is all expressed as a series of Combination of actions, but those skilled in the art should know, the present invention is not limited by described sequence of movement because According to the present invention, some steps can use other orders or carry out simultaneously.Secondly, those skilled in the art should also know Know, embodiment described in this description belongs to preferred embodiment, and involved action and module might not all be this hairs Necessary to bright.
In the above-described embodiments, the description to each embodiment all emphasizes particularly on different fields, and does not have the portion being described in detail in some embodiment Point, it may refer to the associated description of other embodiments.
Above be to a kind of description of facial expression recognizing method and expression recognition device provided by the present invention, it is right In those of ordinary skill in the art, according to the thought of the embodiment of the present invention, can in specific embodiments and applications There is change part, to sum up, this specification content should not be construed as limiting the invention.

Claims (10)

1. a kind of facial expression recognizing method, it is characterised in that including:
Human face image sequence to be identified is obtained, the human face image sequence includes single frames or two frame above facial images;
Each frame facial image in the human face image sequence is pre-processed respectively;
The training pattern that pretreated each frame facial image input has been trained carries out Expression Recognition, obtains the face figure As the Expression Recognition result of sequence;
Wherein, the input of the training pattern circulates god by convolutional neural networks model, long short-term memory successively to output end Built through network model, the first pond layer and Logic Regression Models, and the training pattern is by marking the continuous of classification of expressing one's feelings Two field picture set training is obtained.
2. facial expression recognizing method according to claim 1, it is characterised in that if the human face image sequence includes two Frame above facial image, then, the training pattern that pretreated each frame facial image input has been trained carry out face Identification, including:
Pass through each frame facial image of first pond layer to the long short-term memory Recognition with Recurrent Neural Network mode input Face feature vector it is unified carry out dimension-reduction treatment, obtain the face feature vector after dimension-reduction treatment;
The face feature vector after the dimension-reduction treatment is exported to the Logic Regression Models.
3. facial expression recognizing method according to claim 1 or 2, it is characterised in that the convolutional neural networks model Input to output end successively by the first convolutional layer, the second pond layer, the second convolutional layer and the 3rd pond layer building;
The training pattern that pretreated each frame facial image input has been trained carries out recognition of face, including:To institute State the face feature vector obtained after long short-term memory Recognition with Recurrent Neural Network model output is handled through the 3rd pond layer.
4. facial expression recognizing method according to claim 1 or 2, it is characterised in that described respectively to the face figure As each frame facial image in sequence is pre-processed, including:
Face datection is carried out for every frame facial image in each frame facial image, human face region is determined;
The key feature points in the human face region are detected, and corresponding facial image is entered based on the key feature points detected Row alignment.
5. facial expression recognizing method according to claim 4, it is characterised in that in the detection human face region Key feature points, and alignment is carried out to corresponding facial image based on the key feature points detected, also include afterwards:
Facial image after alignment is subjected to editing and processing according to default template, to obtain the face figure of unified size Picture, wherein, the editing and processing includes following one or more kinds of:Shear treatment, scaling processing.
6. a kind of expression recognition device, it is characterised in that including:
Image acquisition unit, the human face image sequence to be identified for obtaining, the human face image sequence includes single frames or two frames Above facial image;
Image pre-processing unit, for being pre-processed respectively to each frame facial image in the human face image sequence;
Identifying processing unit, the training pattern for pretreated each frame facial image input have been trained carries out expression knowledge Not, the Expression Recognition result of the human face image sequence is obtained;
Wherein, the input of the training pattern circulates god by convolutional neural networks model, long short-term memory successively to output end Built through network model, the first pond layer and Logic Regression Models, and the training pattern is by marking the continuous of classification of expressing one's feelings Two field picture set training is obtained.
7. expression recognition device according to claim 6, it is characterised in that identifying processing unit specifically for:When When the human face image sequence includes two frame above facial images, the long short-term memory is circulated by first pond layer The face feature vector of each frame facial image of neural network model input is unified to carry out dimension-reduction treatment, obtains dimension-reduction treatment Face feature vector afterwards;
The face feature vector after the dimension-reduction treatment is exported to the Logic Regression Models.
8. the expression recognition device according to claim 6 or 7, it is characterised in that the convolutional neural networks model Input to output end successively by the first convolutional layer, the second pond layer, the second convolutional layer and the 3rd pond layer building;
The identifying processing unit specifically for:To the long short-term memory Recognition with Recurrent Neural Network model output through the 3rd pond Change the face feature vector obtained after layer processing.
9. the expression recognition device according to claim 6 or 7, it is characterised in that described image pretreatment unit has Body is used for:Face datection is carried out for every frame facial image in each frame facial image, human face region is determined;Detection is described Key feature points in human face region, and alignment is carried out to corresponding facial image based on the key feature points detected.
10. expression recognition device according to claim 9, it is characterised in that described image pretreatment unit is specific It is additionally operable to:Facial image after alignment is subjected to editing and processing according to default template, to obtain the face of unified size Image, wherein, the editing and processing includes following one or more kinds of:Shear treatment, scaling processing.
CN201610921132.7A 2016-10-21 2016-10-21 Facial expression recognizing method and expression recognition device Pending CN106980811A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610921132.7A CN106980811A (en) 2016-10-21 2016-10-21 Facial expression recognizing method and expression recognition device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610921132.7A CN106980811A (en) 2016-10-21 2016-10-21 Facial expression recognizing method and expression recognition device

Publications (1)

Publication Number Publication Date
CN106980811A true CN106980811A (en) 2017-07-25

Family

ID=59340323

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610921132.7A Pending CN106980811A (en) 2016-10-21 2016-10-21 Facial expression recognizing method and expression recognition device

Country Status (1)

Country Link
CN (1) CN106980811A (en)

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107801096A (en) * 2017-10-30 2018-03-13 广东欧珀移动通信有限公司 Control method, device, terminal device and the storage medium of video playback
CN107832746A (en) * 2017-12-01 2018-03-23 北京小米移动软件有限公司 Expression recognition method and device
CN108062538A (en) * 2017-12-29 2018-05-22 成都智宝大数据科技有限公司 Face identification method and device
CN108304823A (en) * 2018-02-24 2018-07-20 重庆邮电大学 A kind of expression recognition method based on two-fold product CNN and long memory network in short-term
CN108461152A (en) * 2018-01-12 2018-08-28 平安科技(深圳)有限公司 Medical model training method, medical recognition methods, device, equipment and medium
CN108509041A (en) * 2018-03-29 2018-09-07 百度在线网络技术(北京)有限公司 Method and apparatus for executing operation
CN108921024A (en) * 2018-05-31 2018-11-30 东南大学 Expression recognition method based on human face characteristic point information Yu dual network joint training
CN109034143A (en) * 2018-11-01 2018-12-18 云南大学 The micro- expression recognition method of face based on video amplifier and deep learning
CN109325452A (en) * 2018-09-26 2019-02-12 广州大学 A kind of method of face smile expression in detection video
CN109344726A (en) * 2018-09-05 2019-02-15 顺丰科技有限公司 A kind of advertisement placement method and device
CN109697399A (en) * 2017-10-24 2019-04-30 普天信息技术有限公司 A kind of facial expression recognizing method and device
CN109711310A (en) * 2018-12-20 2019-05-03 北京大学 A kind of infant's attachment type automatic Prediction system and its prediction technique
CN109784144A (en) * 2018-11-29 2019-05-21 北京邮电大学 A kind of kinship recognition methods and system
CN109815846A (en) * 2018-12-29 2019-05-28 腾讯科技(深圳)有限公司 Image processing method, device, storage medium and electronic device
CN109829364A (en) * 2018-12-18 2019-05-31 深圳云天励飞技术有限公司 A kind of expression recognition method, device and recommended method, device
US10373332B2 (en) 2017-12-08 2019-08-06 Nvidia Corporation Systems and methods for dynamic facial analysis using a recurrent neural network
CN110887336A (en) * 2018-09-11 2020-03-17 东芝生活电器株式会社 Article taking and placing management system of refrigerator and refrigerator

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104572892A (en) * 2014-12-24 2015-04-29 中国科学院自动化研究所 Text classification method based on cyclic convolution network
CN105139004A (en) * 2015-09-23 2015-12-09 河北工业大学 Face expression identification method based on video sequences
CN105447473A (en) * 2015-12-14 2016-03-30 江苏大学 PCANet-CNN-based arbitrary attitude facial expression recognition method
CN105678250A (en) * 2015-12-31 2016-06-15 北京小孔科技有限公司 Face identification method in video and face identification device in video
CN105845128A (en) * 2016-04-06 2016-08-10 中国科学技术大学 Voice identification efficiency optimization method based on dynamic pruning beam prediction

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104572892A (en) * 2014-12-24 2015-04-29 中国科学院自动化研究所 Text classification method based on cyclic convolution network
CN105139004A (en) * 2015-09-23 2015-12-09 河北工业大学 Face expression identification method based on video sequences
CN105447473A (en) * 2015-12-14 2016-03-30 江苏大学 PCANet-CNN-based arbitrary attitude facial expression recognition method
CN105678250A (en) * 2015-12-31 2016-06-15 北京小孔科技有限公司 Face identification method in video and face identification device in video
CN105845128A (en) * 2016-04-06 2016-08-10 中国科学技术大学 Voice identification efficiency optimization method based on dynamic pruning beam prediction

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
DAE HOE KIM ET.AL: "Micro-Expression Recognition with Expression-State", 《ACM》 *
GRAVES, A., ET.AL,: "Facial expression recognition with recurrent neural networks", 《IN PROCEEDINGS OF THE INTERNATIONAL WORKSHOP ON COGNITION FOR TECHNICAL SYSTEMS》 *
张铎: "《生物识别技术基础》", 31 December 2009, 武汉大学出版社 *
王海宁: "《基于多通道生理信号的情绪识别技术研究》", 31 August 2016, 湖南大学出版社 *
黄福珍,等: "《人脸检测》", 31 December 2006, 上海交通大学出版社 *

Cited By (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109697399A (en) * 2017-10-24 2019-04-30 普天信息技术有限公司 A kind of facial expression recognizing method and device
CN107801096B (en) * 2017-10-30 2020-01-14 Oppo广东移动通信有限公司 Video playing control method and device, terminal equipment and storage medium
CN107801096A (en) * 2017-10-30 2018-03-13 广东欧珀移动通信有限公司 Control method, device, terminal device and the storage medium of video playback
CN107832746A (en) * 2017-12-01 2018-03-23 北京小米移动软件有限公司 Expression recognition method and device
US10373332B2 (en) 2017-12-08 2019-08-06 Nvidia Corporation Systems and methods for dynamic facial analysis using a recurrent neural network
CN108062538A (en) * 2017-12-29 2018-05-22 成都智宝大数据科技有限公司 Face identification method and device
CN108461152A (en) * 2018-01-12 2018-08-28 平安科技(深圳)有限公司 Medical model training method, medical recognition methods, device, equipment and medium
WO2019136806A1 (en) * 2018-01-12 2019-07-18 平安科技(深圳)有限公司 Medical model training method and apparatus, medical identification method and apparatus, device, and medium
CN108304823A (en) * 2018-02-24 2018-07-20 重庆邮电大学 A kind of expression recognition method based on two-fold product CNN and long memory network in short-term
CN108509041A (en) * 2018-03-29 2018-09-07 百度在线网络技术(北京)有限公司 Method and apparatus for executing operation
CN108921024A (en) * 2018-05-31 2018-11-30 东南大学 Expression recognition method based on human face characteristic point information Yu dual network joint training
CN109344726A (en) * 2018-09-05 2019-02-15 顺丰科技有限公司 A kind of advertisement placement method and device
CN110887336A (en) * 2018-09-11 2020-03-17 东芝生活电器株式会社 Article taking and placing management system of refrigerator and refrigerator
CN109325452A (en) * 2018-09-26 2019-02-12 广州大学 A kind of method of face smile expression in detection video
CN109034143A (en) * 2018-11-01 2018-12-18 云南大学 The micro- expression recognition method of face based on video amplifier and deep learning
CN109784144A (en) * 2018-11-29 2019-05-21 北京邮电大学 A kind of kinship recognition methods and system
WO2020125217A1 (en) * 2018-12-18 2020-06-25 深圳云天励飞技术有限公司 Expression recognition method and apparatus and recommendation method and apparatus
CN109829364A (en) * 2018-12-18 2019-05-31 深圳云天励飞技术有限公司 A kind of expression recognition method, device and recommended method, device
CN109711310A (en) * 2018-12-20 2019-05-03 北京大学 A kind of infant's attachment type automatic Prediction system and its prediction technique
CN109815846A (en) * 2018-12-29 2019-05-28 腾讯科技(深圳)有限公司 Image processing method, device, storage medium and electronic device

Similar Documents

Publication Publication Date Title
CN106980811A (en) Facial expression recognizing method and expression recognition device
WO2019174439A1 (en) Image recognition method and apparatus, and terminal and storage medium
Kong et al. Head pose estimation from a 2D face image using 3D face morphing with depth parameters
Murtaza et al. Analysis of face recognition under varying facial expression: a survey.
CN104361316B (en) Dimension emotion recognition method based on multi-scale time sequence modeling
Soo Kim et al. Interpretable 3d human action analysis with temporal convolutional networks
Ma et al. WALKING WALKing walking: Action Recognition from Action Echoes.
Singh et al. Facial expression recognition with convolutional neural networks
Jaiswal et al. Facial emotion detection using deep learning
Cayamcela et al. Fine-tuning a pre-trained convolutional neural network model to translate American sign language in real-time
CN109063626A (en) Dynamic human face recognition methods and device
Manolova et al. Facial expression classification using supervised descent method combined with PCA and SVM
Tur et al. Isolated sign recognition with a siamese neural network of RGB and depth streams
Pourebadi et al. MLP neural network based approach for facial expression analysis
Hsieh et al. Confidence Identification Based on the Combination of Verbal and Non-Verbal factors in Human Robot Interaction
Wang et al. Recurrent convolutional networks based intention recognition for human-robot collaboration tasks
Achkar et al. Ethnicity recognition system using back propagation algorithm of an MLP
Ardizzone et al. Pose classification using support vector machines
Kwon et al. An introduction to face-recognition methods and its implementation in software applications
Bobe et al. Emotion Recognition System for Human-Robot Interface: Comparison of Two Approaches
Ariza et al. Recognition system for facial expression by processing images with deep learning neural network
Iftikhar et al. Detection and classification of facial expressions using artificial neural network
Afroze et al. Classification of attentional focus based on head pose in multi-object scenario
KR20210067815A (en) Method for measuring health condition of user and apparatus therefor
JP5300795B2 (en) Facial expression amplification device, facial expression recognition device, facial expression amplification method, facial expression recognition method, and program

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination