CN112232116A - Facial expression recognition method and device and storage medium - Google Patents
Facial expression recognition method and device and storage medium Download PDFInfo
- Publication number
- CN112232116A CN112232116A CN202010932427.0A CN202010932427A CN112232116A CN 112232116 A CN112232116 A CN 112232116A CN 202010932427 A CN202010932427 A CN 202010932427A CN 112232116 A CN112232116 A CN 112232116A
- Authority
- CN
- China
- Prior art keywords
- image
- neural network
- face detection
- expression recognition
- convolutional neural
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000008921 facial expression Effects 0.000 title claims abstract description 51
- 238000000034 method Methods 0.000 title claims abstract description 49
- 230000014509 gene expression Effects 0.000 claims abstract description 83
- 238000001514 detection method Methods 0.000 claims abstract description 71
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 62
- 238000012549 training Methods 0.000 claims abstract description 28
- 238000012545 processing Methods 0.000 claims abstract description 21
- 238000007781 pre-processing Methods 0.000 claims abstract description 9
- 238000005286 illumination Methods 0.000 claims description 18
- 238000003062 neural network model Methods 0.000 claims description 17
- 238000013528 artificial neural network Methods 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 10
- 238000012937 correction Methods 0.000 claims description 10
- 238000004422 calculation algorithm Methods 0.000 claims description 7
- 230000006870 function Effects 0.000 claims description 7
- 238000010606 normalization Methods 0.000 claims description 7
- 230000009466 transformation Effects 0.000 claims description 7
- 238000011156 evaluation Methods 0.000 claims description 5
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 3
- 210000002569 neuron Anatomy 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000008451 emotion Effects 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 210000004709 eyebrow Anatomy 0.000 description 1
- 238000011478 gradient descent method Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007637 random forest analysis Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000011426 transformation method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Biomedical Technology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a facial expression recognition method, a device and a storage medium, wherein the method comprises the following steps: constructing and training a convolutional neural network for expression recognition, wherein the convolutional neural network is added with a preset network depth; acquiring an image to be recognized, and performing face detection on the image to be recognized to obtain a face detection image; and preprocessing the face detection image, inputting the preprocessed face detection image into the convolutional neural network for expression recognition to perform expression recognition processing, and outputting the expression type in the face detection image. According to the embodiment of the invention, the deepened convolutional neural network is adopted to identify the facial expression of the target face, so that the nonlinear expression is increased, the model expression capability is improved, and more accurate facial expression identification is realized.
Description
Technical Field
The invention relates to the technical field of facial expression recognition, in particular to a facial expression recognition method, a device and a storage medium.
Background
The facial expression recognition technology is used for extracting expression features from a given facial expression image and classifying the expression features into a certain type of specific expression classification. The facial expression recognition has wide application value, and the rapid and accurate facial expression recognition is beneficial to analyzing the emotion of a recognized object and enhancing emotion feedback and the like between the fields of man-machine interaction. The existing facial expression recognition method is mainly based on a random forest algorithm, an expression feature dimension reduction method and the like, and because the data volume is huge in a big data era, more expressions to be recognized are complex in category rules, the recognition operation process of the facial expression is complex, and the recognition accuracy is low.
Accordingly, the prior art is yet to be improved and developed.
Disclosure of Invention
In view of the above disadvantages of the prior art, an object of the present invention is to provide a method, an apparatus and a storage medium for recognizing human facial expressions, which are used to solve the problem of low efficiency and accuracy in recognizing human facial expressions in the prior art.
The technical scheme of the invention is as follows:
a facial expression recognition method comprises the following steps:
constructing and training a convolutional neural network for expression recognition, wherein the convolutional neural network is added with a preset network depth;
acquiring an image to be recognized, and performing face detection on the image to be recognized to obtain a face detection image;
and preprocessing the face detection image, inputting the preprocessed face detection image into the convolutional neural network for expression recognition to perform expression recognition processing, and outputting the expression type in the face detection image.
In the facial expression recognition method, a convolution layer is newly added to the convolutional neural network for expression recognition after an input layer, and the convolution kernel of the newly added convolution layer is 1 x 1.
In the facial expression recognition method, the building and training of the convolutional neural network for expression recognition includes:
obtaining a training sample of known expression categories, wherein the expression categories include happiness, anger, fear, surprise, disgust, sadness and neutrality;
constructing the convolutional neural network for expression recognition, and inputting the training samples of the known expression classes into the convolutional neural network;
and performing error evaluation on the output value of the convolutional neural network through a preset loss function, and performing back propagation and adjustment on the weight parameter of the convolutional neural network according to an error result until the output value of the convolutional neural network reaches an expected value to obtain the trained convolutional neural network for expression recognition.
In the facial expression recognition method, the training sample adopts an FER-2013 data set.
In the facial expression recognition method, the acquiring an image to be recognized, and performing face detection on the image to be recognized to obtain a face detection image includes:
acquiring an image to be recognized, and inputting the image to be recognized into a pre-trained MobileNet cascade neural network model, wherein a Convolution layer of the MobileNet cascade neural network model is Depthwise Separable constraint;
positioning the face position in the image to be recognized through a first neural network in the pre-trained MobileNet cascade neural network model, and cutting the image to be recognized according to a positioning result to obtain a face region image;
and positioning the key points of the human face in the human face region image through a second neural network in the pre-trained MobileNet cascade neural network model, and outputting a human face detection image.
In the facial expression recognition method, the preprocessing the face detection image includes:
the method comprises the steps of predefining key point positions and illumination conditions of a standard face;
aligning the key point position of the current face detection image to the key point position of a standard face through a preset image transformation algorithm to obtain an aligned face detection image;
performing light correction processing on the aligned face detection image according to the illumination condition of the standard face to obtain a face detection image with the illumination condition consistent with that of the standard face;
and sequentially carrying out image graying and image normalization processing on the face detection image subjected to the light ray correction processing.
The invention also provides a facial expression recognition device, which comprises at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the above-described facial expression recognition method.
Another embodiment of the present invention also provides a non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the above-described facial expression recognition method.
Another embodiment of the present invention also provides a computer program product comprising a computer program stored on a non-volatile computer-readable storage medium, the computer program comprising program instructions that, when executed by a processor, cause the processor to perform the above-mentioned facial expression recognition method.
Has the advantages that: compared with the prior art, the embodiment of the invention adopts a deepened convolutional neural network to identify the facial expression of the target face, increases the nonlinear expression, improves the model expression capability and further realizes more accurate facial expression identification.
Drawings
The invention will be further described with reference to the accompanying drawings and examples, in which:
FIG. 1 is a flowchart illustrating a method for recognizing facial expressions according to a preferred embodiment of the present invention;
fig. 2 is a schematic diagram of a hardware structure of a facial expression recognition apparatus according to a preferred embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and effects of the present invention clearer and clearer, the present invention is described in further detail below. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. Embodiments of the present invention will be described below with reference to the accompanying drawings.
Referring to fig. 1, fig. 1 is a flowchart illustrating a facial expression recognition method according to a preferred embodiment of the present invention. As shown in fig. 1, it includes the following steps:
s100, constructing and training a convolutional neural network for expression recognition, wherein the convolutional neural network is added with a preset network depth;
s200, acquiring an image to be recognized, and performing face detection on the image to be recognized to obtain a face detection image;
s300, preprocessing the face detection image, inputting the preprocessed face detection image into the convolutional neural network for expression recognition to perform expression recognition processing, and outputting the expression type in the face detection image.
In the embodiment, a convolutional neural network for expression recognition is constructed and trained, subsequent expression recognition is realized through the convolutional neural network, then an image to be recognized is obtained, face detection is performed on the image to be recognized to obtain a face detection image, the face detection image is preprocessed and then input to the convolutional neural network for expression recognition to perform expression recognition processing, and expression categories in the face detection image are output. Particularly, in the facial expression recognition method provided by the invention, the convolutional neural network increases the preset network depth, that is, a deepened convolutional neural network is adopted to extract and recognize expression features, the receptive field can be effectively increased along with the increase of the network depth, so that the extracted features have more details, more accurate expression recognition is realized, meanwhile, after face detection is carried out to obtain a face detection image, the face detection image is preprocessed and then input to the convolutional neural network to carry out expression recognition, the interference of information such as posture, illumination and the like in the image to be recognized is further eliminated, and the accurate recognition of the expression is ensured.
Specifically, the convolutional neural network for expression recognition newly adds a convolutional layer after an input layer, and the convolution kernel of the newly added convolutional layer is 1 x 1. Namely, the convolutional neural network adopted in the embodiment adds a 1 × 1 convolutional layer after the input layer, and the 1 × 1 convolutional layer increases the nonlinear expression of the input, deepens the network depth, can extract more detailed features, improves the expression capability of the neural network, does not significantly increase the calculated amount, does not affect the original calculation speed, and simultaneously ensures the efficiency and the accuracy of expression recognition.
Further, the building and training of the convolutional neural network for expression recognition comprises:
obtaining a training sample of known expression categories, wherein the expression categories include happiness, anger, fear, surprise, disgust, sadness and neutrality;
constructing the convolutional neural network for expression recognition, and inputting the training samples of the known expression classes into the convolutional neural network;
and performing error evaluation on the output value of the convolutional neural network through a preset loss function, and performing back propagation and adjustment on the weight parameter of the convolutional neural network according to an error result until the output value of the convolutional neural network reaches an expected value to obtain the trained convolutional neural network for expression recognition.
In this embodiment, training data for training a convolutional neural network is obtained first, where the training data is a training sample of a known expression class, where the expression class includes happy, angry, fear, surprise, disgust, sadness, and neutrality, that is, a plurality of expression samples in the seven expression classes are collected in advance, for example, at least 100 expression samples are obtained every expression class, and specifically, the training sample adopts an FER-2013 data set, so that overfitting can be avoided for a large amount of training data, and recognition accuracy is improved; then, constructing the convolutional neural network for expression recognition, inputting the training samples of known expression categories into the convolutional neural network for learning training, namely training the deepened convolutional neural network through the training samples, performing error evaluation on the output value of the convolutional neural network through a preset loss function, adjusting the weight parameters of the convolutional neural network according to the back propagation of error results until the output value of the convolutional neural network reaches an expected value to obtain the trained convolutional neural network for expression recognition, specifically, the preset loss function can perform error evaluation on the output value of the convolutional neural network by adopting, for example, a cross entropy loss function, calculate the value of each node through the forward propagation of input signals, perform the back propagation of errors after calculating the output errors, and adjust the weight parameters of each layer of the convolutional neural network according to an error gradient descent method, after repeated adjustment, the final output of the convolutional neural network reaches an expected value, thereby completing the training of the convolutional neural network to obtain a model finally used for expression recognition,
preferably, the last layer of the convolutional neural network for expression recognition shown in this embodiment uses a classifier to classify facial expressions, and supervised learning is performed on the classifier before classification of facial expressions is performed to obtain classification capability, specifically, the classifier can use a Softmax classifier, and the classification process specifically includes that after expression features are extracted, the classifier uses a plurality of neurons to correspond to different expression categories, each neuron outputs a numerical value with a value between 0 and 1, the numerical value represents the probability that the input sample belongs to the category, the category corresponding to the neuron with the largest output value is selected as the expression classification result of the current input sample, so as to realize accurate expression classification recognition after deepening network expression feature extraction,
further, the acquiring an image to be recognized, and performing face detection on the image to be recognized to obtain a face detection image includes:
acquiring an image to be recognized, and inputting the image to be recognized into a pre-trained MobileNet cascade neural network model, wherein a Convolution layer of the MobileNet cascade neural network model is Depthwise Separable constraint;
positioning the face position in the image to be recognized through a first neural network in the pre-trained MobileNet cascade neural network model, and cutting the image to be recognized according to a positioning result to obtain a face region image;
and positioning the key points of the face in the face region image through a second neural network in the pre-trained MobileNet cascade neural network model, and outputting the face image to be recognized.
In the embodiment, when performing face region detection and face key point detection, a cascade neural network model is trained in advance, wherein the cascade neural network model comprises a first neural network and a second neural network which are respectively used for face region detection and face key point detection, the two neural networks realize face detection from rough to fine in a cascade mode, the two neural networks are respectively trained through a training data set until convergence, and a final cascade neural network model for face detection is obtained, wherein a Convolution layer of the cascade neural network model is a Depthwise Separable Convolution structure in a Mobilene series, namely, a tripthwise and Pointwise Convolution is used for replacing a Sptial Convolution in Pnet, Rnet and Onet, the network computation speed is greatly increased, and an obtained image to be identified is input into the cascade neural network model, positioning the face position in the image to be recognized through a first neural network, cutting the image to be recognized according to the positioning result to obtain a face region image, wherein the face position refers to the information of the position of the face in the image, and usually includes the pixel coordinates of the upper left corner or the central point of the face in the image, the length and the width of the face, and the like, cutting the image to be recognized according to the positioning information, removing the region not including the face to obtain the face region image, then positioning the face key points in the face region image through a second neural network, and outputting the face image to be recognized, wherein the face key points include the coordinate values of eyes, nose tip, mouth corner tip, eyebrows and face contour, the positions of the face key points can be used for indicating the pose and the label of the face, and the face image can be further corrected by using the information so as to extract the face features at the later stage, the accuracy of facial expression recognition is ensured.
Further, the preprocessing the face detection image includes:
the method comprises the steps of predefining key point positions and illumination conditions of a standard face;
aligning the key point position of the current face detection image to the key point position of a standard face through a preset image transformation algorithm to obtain an aligned face detection image;
performing light correction processing on the aligned face detection image according to the illumination condition of the standard face to obtain a face detection image with the illumination condition consistent with that of the standard face;
and sequentially carrying out image graying and image normalization processing on the face detection image subjected to the light ray correction processing.
In this embodiment, before performing expression recognition, a face detection image to be recognized is preprocessed to eliminate the influence of an external environment and a human body pose on expression recognition accuracy, specifically, a key point position and an illumination condition of a standard face are predefined, for example, a key point position and an illumination condition of a certificate photo, and then a preset image transformation algorithm is used to align the key point position of the current face detection image to the key point position of the standard face to obtain an aligned face detection image, thereby achieving the purpose of correcting the face pose, in this embodiment, the image transformation algorithm may be a basic image transformation method or combination such as similarity transformation, affine transformation, or in other embodiments, the pose correction may also be directly realized by the landmark alignment, specifically, the landmark detection is performed on the detected face to obtain a series of landmark points, and calculating an affine matrix H by using the detected landmark points and the landmark points of the standard template posture, and then directly calculating by using the affine matrix H to obtain an aligned image, thereby realizing the correction of the face posture and eliminating the influence of the face posture on the face expression recognition accuracy as much as possible.
Meanwhile, the light correction processing is also carried out on the aligned face detection image according to the illumination condition of the standard face to obtain a face detection image with the illumination condition consistent with that of the standard face, namely, the illumination condition of the aligned face image is converted into the illumination condition of the standard face through the light correction, for example, a gamma value can be used for correcting and adjusting an image pixel value, so that the processed image has proper contrast and the details of the face are clear and visible G. The three-dimensional color image is converted into a two-dimensional gray image represented by a gray value, and the calculation amount of subsequent work can be greatly reduced by carrying out image graying processing, so that the influence of illumination on the image is further reduced, and the processing efficiency is improved; the image normalization is to use a linear normalization algorithm to perform normalization processing on the two-dimensional gray level image to obtain an image with fixed pixels, the statistical distribution of unified image samples can be summarized by performing the image normalization processing, so that the input with different physical meanings and dimensions can be used equally, the influence of different collected environments, such as illumination, equipment performance difference and other factors, on the image of the face detection image to be recognized can be compensated through image preprocessing operation, interference information is removed, the image contrast is improved, and the expression recognition accuracy is further improved.
Another embodiment of the present invention provides a facial expression recognition apparatus, as shown in fig. 2, the apparatus 10 includes:
one or more processors 110 and a memory 120, where one processor 110 is illustrated in fig. 2, the processor 110 and the memory 120 may be connected by a bus or other means, and the connection by the bus is illustrated in fig. 2.
The processor 110 is used to implement various control logic for the device 10, which may be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a single chip, an ARM (Acorn RISC machine) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. Also, the processor 110 may be any conventional processor, microprocessor, or state machine. Processor 110 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
The memory 120 is a non-volatile computer-readable storage medium, and can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions corresponding to the facial expression recognition method in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the apparatus 10, i.e. implements the facial expression recognition method in the above-described method embodiments, by running non-volatile software programs, instructions and units stored in the memory 120.
The memory 120 may include a storage program area and a storage data area, wherein the storage program area may store an application program required for operating the device, at least one function; the storage data area may store data created according to the use of the device 10, and the like. Further, the memory 120 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 120 optionally includes memory located remotely from processor 110, which may be connected to device 10 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
One or more units are stored in the memory 120, which when executed by the one or more processors 110, perform the facial expression recognition method in any of the above-described method embodiments, e.g., performing the above-described method steps S100-S300 in fig. 1.
Embodiments of the present invention provide a non-transitory computer-readable storage medium storing computer-executable instructions for execution by one or more processors, for example, to perform method steps S100-S300 of fig. 1 described above.
By way of example, non-volatile storage media can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which acts as external cache memory. By way of illustration and not limitation, RAM is available in many forms such as Synchronous RAM (SRAM), dynamic RAM, (DRAM), Synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The disclosed memory components or memory of the operating environment described herein are intended to comprise one or more of these and/or any other suitable types of memory.
Another embodiment of the present invention provides a computer program product comprising a computer program stored on a non-volatile computer-readable storage medium, the computer program comprising program instructions which, when executed by a processor, cause the processor to perform the facial expression recognition method of the above-described method embodiment. For example, the method steps S100 to S300 in fig. 1 described above are performed.
In summary, in the facial expression recognition method, apparatus and storage medium disclosed in the present invention, the method includes: constructing and training a convolutional neural network for expression recognition, wherein the convolutional neural network is added with a preset network depth; acquiring an image to be recognized, and performing face detection on the image to be recognized to obtain a face detection image; and preprocessing the face detection image, inputting the preprocessed face detection image into the convolutional neural network for expression recognition to perform expression recognition processing, and outputting the expression type in the face detection image. According to the embodiment of the invention, the deepened convolutional neural network is adopted to identify the facial expression of the target face, so that the nonlinear expression is increased, the model expression capability is improved, and more accurate facial expression identification is realized.
The above-described embodiments are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the embodiment.
Through the above description of the embodiments, those skilled in the art will clearly understand that the embodiments may be implemented by software plus a general hardware platform, and may also be implemented by hardware. With this in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer electronic device (which may be a personal computer, a server, or a network electronic device, etc.) to execute the methods of the various embodiments or some parts of the embodiments.
Conditional language such as "can," "might," or "may" is generally intended to convey that a particular embodiment can include (yet other embodiments do not include) particular features, elements, and/or operations, unless specifically stated otherwise or otherwise understood within the context as used. Thus, such conditional language is not generally intended to imply that features, elements, and/or operations are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without student input or prompting, whether such features, elements, and/or operations are included or are to be performed in any particular embodiment.
What has been described herein in the specification and drawings includes examples capable of providing a facial expression recognition method, apparatus, and storage medium. It will, of course, not be possible to describe every conceivable combination of components and/or methodologies for purposes of describing the various features of the disclosure, but it can be appreciated that many further combinations and permutations of the disclosed features are possible. It is therefore evident that various modifications can be made to the disclosure without departing from the scope or spirit thereof. In addition, or in the alternative, other embodiments of the disclosure may be apparent from consideration of the specification and drawings and from practice of the disclosure as presented herein. It is intended that the examples set forth in this specification and the drawings be considered in all respects as illustrative and not restrictive. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.
Claims (9)
1. A facial expression recognition method is characterized by comprising the following steps:
constructing and training a convolutional neural network for expression recognition, wherein the convolutional neural network is added with a preset network depth;
acquiring an image to be recognized, and performing face detection on the image to be recognized to obtain a face detection image;
and preprocessing the face detection image, inputting the preprocessed face detection image into the convolutional neural network for expression recognition to perform expression recognition processing, and outputting the expression type in the face detection image.
2. The method according to claim 1, wherein the convolutional neural network for expression recognition adds a convolutional layer after the input layer, and the convolutional kernel of the newly added convolutional layer is 1 x 1.
3. The facial expression recognition method of claim 2, wherein the constructing and training of the convolutional neural network for expression recognition comprises:
obtaining a training sample of known expression categories, wherein the expression categories include happiness, anger, fear, surprise, disgust, sadness and neutrality;
constructing the convolutional neural network for expression recognition, and inputting the training samples of the known expression classes into the convolutional neural network;
and performing error evaluation on the output value of the convolutional neural network through a preset loss function, and performing back propagation and adjustment on the weight parameter of the convolutional neural network according to an error result until the output value of the convolutional neural network reaches an expected value to obtain the trained convolutional neural network for expression recognition.
4. The method according to claim 3, wherein the training sample employs FER-2013 data set.
5. The method for recognizing facial expressions according to claim 1, wherein the acquiring an image to be recognized and performing face detection on the image to be recognized to obtain a face detection image comprises:
acquiring an image to be recognized, and inputting the image to be recognized into a pre-trained MobileNet cascade neural network model, wherein a Convolution layer of the MobileNet cascade neural network model is Depthwise Separable constraint;
positioning the face position in the image to be recognized through a first neural network in the pre-trained MobileNet cascade neural network model, and cutting the image to be recognized according to a positioning result to obtain a face region image;
and positioning the key points of the human face in the human face region image through a second neural network in the pre-trained MobileNet cascade neural network model, and outputting a human face detection image.
6. The method of claim 5, wherein the preprocessing the face detection image comprises:
the method comprises the steps of predefining key point positions and illumination conditions of a standard face;
aligning the key point position of the current face detection image to the key point position of a standard face through a preset image transformation algorithm to obtain an aligned face detection image;
performing light correction processing on the aligned face detection image according to the illumination condition of the standard face to obtain a face detection image with the illumination condition consistent with that of the standard face;
and sequentially carrying out image graying and image normalization processing on the face detection image subjected to the light ray correction processing.
7. An apparatus for facial expression recognition, the apparatus comprising at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of facial expression recognition according to any one of claims 1-6.
8. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the method of facial expression recognition of any one of claims 1-6.
9. A computer program product, characterized in that the computer program product comprises a computer program stored on a non-volatile computer-readable storage medium, the computer program comprising program instructions which, when executed by a processor, cause the processor to carry out the method of facial expression recognition according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010932427.0A CN112232116A (en) | 2020-09-08 | 2020-09-08 | Facial expression recognition method and device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010932427.0A CN112232116A (en) | 2020-09-08 | 2020-09-08 | Facial expression recognition method and device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112232116A true CN112232116A (en) | 2021-01-15 |
Family
ID=74116730
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010932427.0A Pending CN112232116A (en) | 2020-09-08 | 2020-09-08 | Facial expression recognition method and device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112232116A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112733803A (en) * | 2021-01-25 | 2021-04-30 | 中国科学院空天信息创新研究院 | Emotion recognition method and system |
| CN112766167A (en) * | 2021-01-20 | 2021-05-07 | 宁夏计算机软件与技术服务有限公司 | Face recognition method based on face feature vector |
| CN112766220A (en) * | 2021-02-01 | 2021-05-07 | 西南大学 | Dual-channel micro-expression recognition method and system, storage medium and computer equipment |
| CN112818772A (en) * | 2021-01-19 | 2021-05-18 | 网易（杭州）网络有限公司 | Facial parameter identification method and device, electronic equipment and storage medium |
| CN112818838A (en) * | 2021-01-29 | 2021-05-18 | 北京嘀嘀无限科技发展有限公司 | Expression recognition method and device and electronic equipment |
| CN113505750A (en) * | 2021-07-28 | 2021-10-15 | 阳光保险集团股份有限公司 | Identification method, device, electronic equipment and computer readable storage medium |
| CN113505746A (en) * | 2021-07-27 | 2021-10-15 | 陕西师范大学 | Fine classification method, device and equipment for micro-expression image and readable storage medium |
| CN113688714A (en) * | 2021-08-18 | 2021-11-23 | 华南师范大学 | Method, device, equipment and storage medium for identifying multi-angle facial expressions |
| CN113688715A (en) * | 2021-08-18 | 2021-11-23 | 山东海量信息技术研究院 | Facial expression recognition method and system |
| CN114333024A (en) * | 2021-12-31 | 2022-04-12 | 郑州工程技术学院 | Method, device, equipment and storage medium for recognizing facial expressions of students based on confrontation training network |
| CN115908999A (en) * | 2022-11-25 | 2023-04-04 | 合肥中科类脑智能技术有限公司 | Method for detecting corrosion of top hardware fitting of power distribution tower, medium and edge terminal equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018128996A1 (en) * | 2017-01-03 | 2018-07-12 | Clipo, Inc. | System and method for facilitating dynamic avatar based on real-time facial expression detection |
| CN108304826A (en) * | 2018-03-01 | 2018-07-20 | 河海大学 | Facial expression recognizing method based on convolutional neural networks |
| CN109993100A (en) * | 2019-03-27 | 2019-07-09 | 南京邮电大学 | The implementation method of facial expression recognition based on further feature cluster |
| CN110414371A (en) * | 2019-07-08 | 2019-11-05 | 西南科技大学 | A kind of real-time face expression recognition method based on multiple dimensioned nuclear convolution neural network |
| CN110633669A (en) * | 2019-09-12 | 2019-12-31 | 华北电力大学（保定） | Mobile terminal face attribute identification method based on deep learning in home environment |
-
2020
- 2020-09-08 CN CN202010932427.0A patent/CN112232116A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018128996A1 (en) * | 2017-01-03 | 2018-07-12 | Clipo, Inc. | System and method for facilitating dynamic avatar based on real-time facial expression detection |
| CN108304826A (en) * | 2018-03-01 | 2018-07-20 | 河海大学 | Facial expression recognizing method based on convolutional neural networks |
| CN109993100A (en) * | 2019-03-27 | 2019-07-09 | 南京邮电大学 | The implementation method of facial expression recognition based on further feature cluster |
| CN110414371A (en) * | 2019-07-08 | 2019-11-05 | 西南科技大学 | A kind of real-time face expression recognition method based on multiple dimensioned nuclear convolution neural network |
| CN110633669A (en) * | 2019-09-12 | 2019-12-31 | 华北电力大学（保定） | Mobile terminal face attribute identification method based on deep learning in home environment |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112818772A (en) * | 2021-01-19 | 2021-05-18 | 网易（杭州）网络有限公司 | Facial parameter identification method and device, electronic equipment and storage medium |
| CN112766167A (en) * | 2021-01-20 | 2021-05-07 | 宁夏计算机软件与技术服务有限公司 | Face recognition method based on face feature vector |
| CN112733803A (en) * | 2021-01-25 | 2021-04-30 | 中国科学院空天信息创新研究院 | Emotion recognition method and system |
| CN112818838A (en) * | 2021-01-29 | 2021-05-18 | 北京嘀嘀无限科技发展有限公司 | Expression recognition method and device and electronic equipment |
| CN112766220A (en) * | 2021-02-01 | 2021-05-07 | 西南大学 | Dual-channel micro-expression recognition method and system, storage medium and computer equipment |
| CN112766220B (en) * | 2021-02-01 | 2023-02-24 | 西南大学 | Dual-channel micro-expression recognition method and system, storage medium and computer equipment |
| CN113505746A (en) * | 2021-07-27 | 2021-10-15 | 陕西师范大学 | Fine classification method, device and equipment for micro-expression image and readable storage medium |
| CN113505750A (en) * | 2021-07-28 | 2021-10-15 | 阳光保险集团股份有限公司 | Identification method, device, electronic equipment and computer readable storage medium |
| CN113688714A (en) * | 2021-08-18 | 2021-11-23 | 华南师范大学 | Method, device, equipment and storage medium for identifying multi-angle facial expressions |
| CN113688715A (en) * | 2021-08-18 | 2021-11-23 | 山东海量信息技术研究院 | Facial expression recognition method and system |
| CN113688714B (en) * | 2021-08-18 | 2023-09-01 | 华南师范大学 | Multi-angle facial expression recognition method, device, equipment and storage medium |
| CN114333024A (en) * | 2021-12-31 | 2022-04-12 | 郑州工程技术学院 | Method, device, equipment and storage medium for recognizing facial expressions of students based on confrontation training network |
| CN114333024B (en) * | 2021-12-31 | 2024-01-26 | 郑州工程技术学院 | Method, device, equipment and storage medium for student facial expression recognition based on countermeasure training network |
| CN115908999A (en) * | 2022-11-25 | 2023-04-04 | 合肥中科类脑智能技术有限公司 | Method for detecting corrosion of top hardware fitting of power distribution tower, medium and edge terminal equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112232116A (en) | Facial expression recognition method and device and storage medium | |
| CN110532900B (en) | Facial expression recognition method based on U-Net and LS-CNN | |
| CN112257503A (en) | Sex age identification method, device and storage medium | |
| Azizah et al. | Deep learning implementation using convolutional neural network in mangosteen surface defect detection | |
| CN106682616B (en) | Method for recognizing neonatal pain expression based on two-channel feature deep learning | |
| Singh et al. | Facial expression recognition with convolutional neural networks | |
| CN109409297B (en) | Identity recognition method based on dual-channel convolutional neural network | |
| CN111191583B (en) | Space target recognition system and method based on convolutional neural network | |
| CN112232117A (en) | Face recognition method, face recognition device and storage medium | |
| CN112257502A (en) | Pedestrian identification and tracking method and device for surveillance video and storage medium | |
| CN106980811A (en) | Facial expression recognizing method and expression recognition device | |
| EP4145345A1 (en) | Data annotation method and device, and fine granularity identification method and device | |
| CN106022273A (en) | Handwritten form identification system of BP neural network based on dynamic sample selection strategy | |
| CN108898138A (en) | Scene text recognition methods based on deep learning | |
| CN107563312A (en) | Facial expression recognizing method | |
| CN110532925B (en) | Driver fatigue detection method based on space-time graph convolutional network | |
| Cayamcela et al. | Fine-tuning a pre-trained convolutional neural network model to translate American sign language in real-time | |
| CN110674777A (en) | Optical character recognition method in patent text scene | |
| CN110956082A (en) | Face key point detection method and detection system based on deep learning | |
| CN105976397A (en) | Target tracking method based on half nonnegative optimization integration learning | |
| CN110852358A (en) | Vehicle type distinguishing method based on deep learning | |
| CN114519401B (en) | Image classification method and device, electronic equipment and storage medium | |
| CN114758399A (en) | Expression control method, device, equipment and storage medium of bionic robot | |
| CN109948569B (en) | Three-dimensional mixed expression recognition method using particle filter framework | |
| Feng et al. | Facial expression recognition based on local features of transfer learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |